Sparsity-based reduction of gate switching in deep neural network accelerators

Abstract

Gate switching in deep learning operations can be reduced based on the sparsity of the input data. The first element of the activation operand and the first element of the weight operand may be stored in an input memory unit associated with a multiplier in the processing element. The multiplier calculates the product of the two elements, and this product may be stored in the multiplier's output memory unit. Gate switching is reduced by avoiding at least one gate switching required for the sum-of-accumulate operation after detecting that the second element of the activation operand or the second element of the weight operand is zero. For example, the input memory unit may not need to be updated. The zero-value data element may be stored in the multiplier's output memory unit and used as the product of the second element of the activation operand and the second element of the weight operand.

Description

[Technical Field]

[0001] Cross-references to related applications This application claims priority to U.S. Patent Application No. 18 / 325,298, filed on 30 May 2023, entitled “Spaciousness-Based Reduction of Gate Switching in Deep Neural Network Accelerators,” which is incorporated herein by reference in its entirety.

[0002] Technical field This disclosure broadly relates to deep neural networks (DNNs), and more specifically to sparsity-based reduction of gate switching in DNN accelerators. [Background technology]

[0003] Due to their ability to achieve high accuracy, DNNs are widely used in a variety of artificial intelligence applications, from computer vision to speech recognition and natural language processing. However, high accuracy comes at the cost of considerable computational expense. DNNs have extremely high computational demands, as each inference can require hundreds of millions of MAC (multiply-accumulate) operations, as well as the reading and writing of large amounts of data. Therefore, techniques are needed to improve the efficiency of DNNs. [Brief explanation of the drawing]

[0004] Embodiments will be readily apparent from the following detailed description in relation to the accompanying drawings. For the sake of this description, similar reference numerals indicate similar structural elements. Embodiments are shown in the figures of the accompanying drawings as examples, not as limitations.

[0005] [Figure 1] Exemplary DNNs in various embodiments are shown.

[0006] [Figure 2] Exemplary convolutions in various embodiments are shown.

[0007] [Figure 3] Block diagrams of DNN accelerators according to various embodiments.

[0008] [Figure 4] Block diagrams of computing blocks according to various embodiments.

[0009] [Figure 5] Shows the sparsity acceleration in MAC operations by processing elements (PEs) according to various embodiments.

[0010] [Figure 6] Shows PEs that can reduce gate switching based on sparsity according to various embodiments.

[0011] [Figure 7] Shows PEs having pipelined multipliers according to various embodiments.

[0012] [[ID=3,4]] [Figure 8] Shows PEs having multipliers and accumulators according to various embodiments.

[0013] [Figure 9] Shows PEs having accumulators with registers configurable based on sparsity according to various embodiments.

[0014] [Figure 10] Shows PEs having adder trees according to various embodiments.

[0015] [Figure 11] [[ID=,53]]Shows a PE array according to various embodiments.

[0016] [Figure 12] Block diagrams of PEs according to various embodiments.

[0017] [Figure 13] This flowchart shows various embodiments of methods for reducing power consumption for DNNs based on sparsity.

[0018] [Figure 14] This is a block diagram of an exemplary computing device in various embodiments. [Modes for carrying out the invention]

[0019] Overview Over the past decade, we have seen a rapid increase in AI-based data processing, particularly DNN-based data processing. DNNs are widely used in the fields of computer vision, speech recognition, image processing, and video processing, primarily due to their ability to achieve accuracy exceeding human levels. Significant improvements in DNN model size and accuracy, coupled with the rapid increase in computing power of execution platforms, have led to the adoption of DNN applications even within resource-constrained mobile and edge devices with limited energy availability.

[0020] A DNN layer may include one or more deep learning operations, such as convolution, pooling, element-wise operations, linear operations, or nonlinear operations. The deep learning operations in a DNN layer may be performed on one or more internal parameters of the DNN layer and on the input data received by the DNN layer. The internal parameters of the DNN layer (e.g., weights) may be determined during the training phase.

[0021] The internal parameters or input data of a DNN layer may be elements of a tensor. A tensor is a data structure that has multiple elements across one or more dimensions. Exemplary tensors include vectors, which are one-dimensional tensors, and matrices, which are two-dimensional tensors. Three-dimensional tensors and even higher-dimensional tensors are also possible. A DNN layer may have an input tensor (also called an "input feature map, IFM") containing one or more input activations (also called "input elements" or "activations"), a weight tensor containing one or more weights, and an output tensor (also called an "output feature map, OFM") containing one or more output activations (also called "output elements" or "activations"). The weight tensor of a convolution may be a kernel, a filter, or a group of filters.

[0022] An accelerator for running a DNN ("DNN accelerator") may include one or more large arrays of PEs that operate concurrently as the layers within the DNN are executed. PEs can perform deep learning operations. For example, for convolution, PEs can perform MAC operations on activations and weights. Input tensors or weight tensors may contain zero-value elements. For example, the training phase may generate zero-value weights. Due to very sparse weights, activations may become sparse in later layers of the DNN after passing through nonlinear activation functions such as rectified linear units (RELs). Network quantization for performing inference on edge devices also results in a large number of zeros in weights and activations. Zero-value activations or weights do not contribute to the accumulation of partial sums during MAC operations. DNN accelerators can leverage the available sparsity in weights and activations to accelerate deep learning operations in DNNs, which can lead to higher speeds or throughput, as well as lower power consumption.

[0023] Embodiments of this disclosure facilitate significant power savings by disabling MAC calculation logic within a DNN accelerator. For example, gate switching in the PE can be reduced based on sparsity in weights or activations without affecting accuracy at the PE output.

[0024] In various embodiments, a DNN accelerator may include one or more computational blocks for performing deep learning operations in a DNN. A computational block may include an array of PEs. A PE may include one or more multipliers, one or more accumulators, and multiple memory units (e.g., registers). One or more PEs within a computational block may be associated with a sparsity module that can accelerate the deep learning operation and reduce the power consumed by the deep learning operation based on sparsity in activations or weights.

[0025] In some embodiments, the multiplier may receive an activation operand containing a sequence of activations and a weight operand containing a sequence of weights. The multiplier may perform a sequence of multiplications, each multiplication being for each activation-weight pair. An activation-weight pair includes an activation in the activation operand and a corresponding weight in the weight operand. The position of the activation in the activation operand may be the same as the position of the weight in the weight operand. In some embodiments, the multiplier may be associated with a clock. Each multiplication may be performed within a single clock cycle. For each multiplication round, a sparsity module may determine whether an activation or weight is zero. The sparsity module may include one or more logical operators, such as a NOR gate.

[0026] In embodiments where the sparsity module determines that neither the activation nor the weight is zero, the activation and weight are stored in the multiplier's input registers and replaced with the activation and weight from the previous round. The multiplier receives the activation and weight from the input registers, calculates the product of the activation and weight, and the product is stored in the multiplier's output register. In embodiments where the sparsity module determines that either the activation or weight is zero, the sparsity module prevents the activation or weight from entering the multiplier's input registers. The activation and weight from the previous round may remain in the input registers. The sparsity module can zero the output of the multiplier. This reduces the number of gate switching required compared to the case where the registers store the activation and weight and the multiplier calculates the product of zeros. In embodiments where the multiplier is pipelined, the sparsity module may delay the zeroing signal by the number of pipes in the multiplier. In addition to zeroing the multiplier's output, or alternatively, the sparsity module can zero the input to an accumulator configured to accumulate the products calculated by the multiplier(s), or disable the accumulator for this round, thereby reducing gate switching in the accumulator.

[0027] Sparsity modules can support integer quantization of floating-point values, such as symmetric quantization and asymmetric quantization. Sparsity modules can also facilitate pattern-based gating, for example, for element-wise operations. Sparsity modules can reduce power consumption for non-zero redundant computations. For example, in some embodiments, sparsity modules may treat denormalized values ​​or values ​​below a threshold number as zero. This disclosure provides a technique that can significantly reduce the power consumption of DNN accelerators for DNN training and inference. Achieving high system performance with lower energy consumption in DNN accelerators can be important for efficient edge inference for various AI applications such as imaging, video, speech, or other applications.

[0028] For explanatory purposes, certain numbers, materials, and configurations are described to provide a full understanding of the exemplary implementation. However, it will be apparent to those skilled in the art that this disclosure may be implemented without such specific details, or / or using only some of the aspects described. On the other hand, well-known features have been omitted or simplified so as not to overshadow the exemplary implementation.

[0029] Furthermore, reference is made to the accompanying drawings, which form part of this application. The drawings illustrate possible embodiments. Other embodiments may be used, and structural or logical modifications may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed as limiting.

[0030] Various actions may be described sequentially as multiple separate actions or operations in the manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be interpreted as meaning that these actions are necessarily order-dependent. In particular, these actions do not have to be performed in the order presented. The described actions may be performed in a different order than in the described embodiments. Various additional actions may be performed, or described actions may be omitted in additional embodiments.

[0031] For the purposes of this disclosure, the phrase "A or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used in relation to measurement ranges, the term "between..." includes both ends of the measurement range.

[0032] The descriptions use phrases such as “in one embodiment” or “in one embodiment,” which may refer to one or more of the same or different embodiments, respectively. Terms such as “equipped,” “included,” and “having” as used in reference to embodiments of this disclosure are synonymous. This disclosure may use descriptions based on perspective views, such as “above,” “below,” “top,” “bottom,” and “side,” to describe various features of the drawings, but these terms are for the sake of discussion only and do not imply any desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” and “third” to describe common subjects simply indicates that different instances of similar subjects are being referred to, and is not intended to imply that the subjects described in this way must be in a given order, temporally, spatially, in rank, or in any other way.

[0033] In the following detailed description, various aspects of the exemplary implementation will be described using terminology commonly used by those skilled in the art to communicate their work to others skilled in the art.

[0034] The terms “substantially,” “near,” “almost,” “near,” and “about” generally refer to being within ±20% of the target value based on specific values ​​as described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to being within ±5 to 20% of the target value based on specific values ​​as described herein or known in the art.

[0035] Furthermore, the terms “equipped,” “containing,” “included,” “having,” “having,” or any other variation thereof are intended to encompass non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that includes a set of elements is not necessarily limited to those elements alone, and may include other elements not explicitly listed or inherent in such a method, process, device, or DNN accelerator. Also, the term “or” refers to an inclusive “or” and not an exclusive “or.”

[0036] The systems, methods, and devices of this disclosure each have several inventive aspects, and no single aspect alone accounts for all the desirable attributes disclosed herein. Details of one or more implementations of the subject matter described herein are given in the following description and accompanying drawings.

[0037] Exemplary DNN Figure 1 shows exemplary DNN 100 in various embodiments. For illustrative purposes, DNN 100 in Figure 1 is a CNN. In other embodiments, DNN 100 may be other types of DNNs. DNN 100 is trained to receive an image and output the classification of objects in the image. In the embodiment of Figure 1, DNN 100 receives an input image 105 containing objects 115, 125, and 135. DNN 100 includes a set of layers including a plurality of convolutional layers 110 (each referred to as “convolutional layer 110”), a plurality of pooling layers 120 (each referred to as “pooling layer 120”), and a plurality of fully connected layers 130 (each referred to as “fully connected layer 130”). In other embodiments, DNN 100 may include fewer, more, or different layers. In DNN 100 inference, the layers of DNN 100 perform tensor computations that include many tensor operations, such as convolution (e.g., multiply-accumulate, MAC), pooling, element-wise operations (e.g., element-wise addition, element-wise multiplication), other types of tensor operations, or any combination thereof.

[0038] The convolutional layer 110 summarizes the presence of features in the input image 105. The convolutional layer 110 functions as a feature extractor. The first layer of the DNN 100 is the convolutional layer 110. In one example, the convolutional layer 110 performs convolution on the input tensor 140 (also called IFM 140) and the filter 150. As shown in Figure 1, the IFM 140 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 140 contains three input channels, each input channel represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix contains seven input elements (also called input points) in each row and seven input elements in each column. The filter 150 is represented by a 3x3x3 three-dimensional matrix. The filter 150 contains three kernels, each of which may correspond to a different input channel of the IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than an IFM. In the embodiment of Figure 1, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel contains three weights in each row and three weights in each column. The weights can be initialized and updated by backpropagation using gradient descent. The magnitude of the weights can indicate the importance of the filter 150 in extracting features from the IFM 140.

[0039] The convolution involves a MAC operation between the input elements in IFM 140 and the weights in filter 150. The convolution may be a standard convolution 163 or a depth-dependent convolution 183. In a standard convolution 163, the entire filter 150 slides across IFM 140. All input channels are combined to produce an output tensor 160 (also called an output feature map (OFM) 160). The OFM 160 is represented by a 5x5 2D matrix. The 5x5 2D matrix contains five output elements (also called output points) in each row and five output elements in each column. For illustrative purposes, the standard convolution includes one filter in the embodiment shown in Figure 1. In embodiments with multiple filters, the standard convolution may produce multiple output channels in OFM 160.

[0040] The multiplication applied between a kernel-sized patch and a kernel in IFM 140 may be a dot product. The dot product is an element-wise multiplication between a kernel-sized patch and the corresponding kernel in IFM 140, which is then summed, always resulting in a single value. Because it gives a single value, this operation is often called a "scalar product." Using a kernel smaller than IFM 140 is intentional, as it allows the same kernel (set of weights) to be multiplied by IFM 140 multiple times at different points on IFM 140. Specifically, the kernel is applied systematically from left to right and top to bottom to each overlapping portion or kernel-sized patch of IFM 140. The result of multiplying a kernel by IFM 141 once is a single value. When a kernel is applied to IFM 140 multiple times, the result of the multiplication is a 2D matrix of output elements. Thus, the 2D output matrix from a standard convolution (i.e., OFM 160) is called OFM.

[0041] In the depth-based convolution 183, the input channels are not combined. Rather, MAC operations are performed on each individual input channel and each individual kernel to generate output channels. As shown in Figure 1, the depth-based convolution 183 generates the depth-based output tensor 180. The depth-based output tensor 180 is represented by a 5x5x3 3D ​​matrix. The depth-based output tensor 180 contains three output channels, each represented by a 5x5 2D matrix. The 5x5 2D matrix contains five output elements in each row and five output elements in each column. Each output channel is the result of MAC operations on the input channels of IFM 140 and the kernels of filter 150. For example, the first output channel (patterned with dots) is the result of a MAC operation between the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal lines) is the result of a MAC operation between the second input channel (patterned with horizontal lines) and the second kernel (patterned with horizontal lines), and the third output channel (patterned with diagonal stripes) is the result of a MAC operation between the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such depth-based convolution, the number of input channels is equal to the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively called depth-based channels. After the depth-based convolution, a point-based convolution 193 is performed on the depth-based output tensor 180 and the 1x1x3 tensor 190 to produce OFM 160.

[0042] OFM 160 is then passed to the next layer in the sequence. In some embodiments, OFM 160 is passed through an activation function. An exemplary activation function is Reu. Elu is a computation that either directly returns the value provided as input, or returns the value zero if the input is less than or equal to zero. The convolutional layer 110 may receive several images as input and compute the convolution of each of them with each of the kernels. This process can be repeated several times. For example, OFM 160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 following the convolutional layer 110 that generates OFM 160 in the sequence). The subsequent convolutional layer 110 performs a convolution on OFM 160 using the new kernel and generates a new feature map. The new feature map may also be normalized and resized. The new feature map may again be kernel-processed by further subsequent convolutional layers 110, and so on.

[0043] In some embodiments, the convolutional layer 110 has four hyperparameters, namely, the number of kernels, a kernel of size F (for example, the kernel has dimensions of F × F × D pixels), S steps in which the window corresponding to the kernel is dragged on the image (for example, a step of 1 means moving the window by 1 pixel at a time), and zero padding P (for example, adding a black contour of P pixels thickness to the input image of the convolutional layer 110). The convolutional layer 110 can perform various types of convolution, such as 2D convolution, dilation or atlas convolution, spatially separable convolution, depth-separable convolution, and transpose convolution. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0044] The pooling layer 120 downsamples the feature map generated by the convolutional layer, for example, by summarizing the presence of features within a patch of the feature map. The pooling layer 120 is placed between two convolutional layers 110, namely, a preceding convolutional layer 110 (a convolutional layer 110 that precedes the pooling layer 120 in the layer sequence) and a succeeding convolutional layer 110 (a convolutional layer 110 that follows the pooling layer 120 in the layer sequence). In some embodiments, the pooling layer 120 is added after the convolutional layer 110, for example, after an activation function (e.g., ReLU) has been applied to OFM 160.

[0045] The pooling layer 120 receives the feature map generated by the preceding convolutional layer 110 and applies a pooling operation to the feature map. The pooling operation reduces the size of the feature map while preserving important characteristics. Thus, the pooling operation improves the efficiency of the DNN and avoids overfitting. The pooling layer 120 can perform the pooling operation through mean pooling (calculating the average value for each patch on the feature map), max pooling (calculating the maximum value for each patch on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of 2 pixels, thereby reducing the size of the feature map by half, for example, the number of pixels or values ​​in the feature map is reduced to a quarter of its original size. In one example, a pooling layer 120 applied to a 6x6 feature map yields a 3x3 output pooled feature map. The output of the pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates separately for each feature map to create a new set of the same number of pooled feature maps.

[0046] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 may or may not be a convolutional layer. The fully connected layer 130 accepts an input operand. The input operand defines the outputs of the convolutional layer 110 and the pooling layer 120 and includes the value of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and activation function to the input operand to generate a vector. The vector can contain as many elements as there are classes, where element i represents the probability that the image belongs to class i. Thus, each element is between 0 and 1, and the sum of all is 1. These probabilities are calculated by the last fully connected layer 130 using the logistic function (binary classification) or the softmax function (multi-class classification) as the activation function.

[0047] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. In the embodiment of Figure 1, N is equal to 3 because there are three objects 115, 125, and 135 in the input image. Each element of the operand represents the probability that the input image 105 belongs to a class. To calculate the probabilities, the fully connected layer 130 multiplies each input element by a weight, sums them up, and then applies an activation function (e.g., logistic for N=2, softmax for N>2). This is equivalent to multiplying the input operands by a matrix containing the weights. In one example, the vector contains three probabilities: a first probability that object 115 is a tree, a second probability that object 125 is a car, and a third probability that object 135 is a person. In other embodiments, where the input image 105 contains different objects or different numbers of objects, the individual values ​​may differ.

[0048] Exemplary convolution Figure 2 shows exemplary convolutions according to various embodiments. The convolution may be performed in a convolutional layer of a DNN, for example, in convolutional layer 110 in Figure 1. The convolutional layer may be a front-end layer. The convolution can be performed on an input tensor 210 and a filter 220 (each referred to as “filter 220”). The result of the convolution is the output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator comprising one or more computation blocks. An example of a DNN accelerator may be the DNN accelerator 300 in Figure 3. An example of a computation block may be the computation block 325 in Figure 3.

[0049] In the embodiment shown in Figure 2, the input tensor 210 includes activations (also called "input activations," "elements," or "input elements") arranged in a 3D matrix. Activations in the input tensor 210 are data points in the input tensor 210. The input tensor 210 has a spatial size H in ×W in ×C in It has, and here H in is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in the column within the 2D matrix for each input channel), and W in is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activations in each row of the 2D matrix for each input channel), and C in is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of input channels). For simplicity and illustration, the input tensor 210 has a spatial size of 7 × 7 × 3, i.e., the input tensor 210 contains 3 input channels, each input channel having a 7 × 7 2D matrix. Each input element in the input tensor 210 can be represented by (X, Y, Z) coordinates. In other embodiments, the height, width, or depth of the input tensor 210 may differ.

[0050] Each filter 220 contains weights arranged in a 3D matrix. The weight values ​​can be determined through training the DNN. Filter 220 has a spatial size H f ×Wf ×C f has. Here, H f is the height of the filter (i.e., the length along the Y-axis indicating the number of weights within each column within each kernel), and W f is the width of the filter (i.e., the length along the X-axis indicating the number of weights in each row within each kernel), and C f is the depth of the filter (i.e., the length along the Z-axis indicating the number of channels). In some embodiments, C f is equal to C in For simplicity and illustrative purposes, each filter 220 in FIG. 2 has a spatial size of 3×3×3, i.e., the filter 220 includes three convolutional kernels having a spatial size of 3×3. In other embodiments, the height, width, or depth of the filter 220 may be different. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 210.

[0051] Activations or weights can take one byte or multiple bytes in memory. The number of bytes for activations or weights can depend on the data format. For example, if an activation or weight has an integer format (e.g., INT8), the activation takes 1 byte. If an activation or weight has a floating-point format (e.g., FP16 or BF16), the activation or weight takes 2 bytes. Other data formats may be used for activations or weights.

[0052] In convolution, each filter 220 slides across the input tensor 210 to generate a 2D matrix for the output channels in the output tensor 230. In the embodiment of FIG. 2, the 2D matrix has a spatial size of 5×5. The output tensor 230 includes activations (also referred to as "output activations", "elements", or "output elements") arranged in a 3D matrix. The activations in the output tensor 230 are data points in the output tensor 230. The output tensor 230 has a spatial size of H out ×W out ×C out has. Here, H outis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in the column within the 2D matrix for each output channel), and W out is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of output activations in a row within the 2D matrix for each output channel), and C out This is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of output channels). out This may be equal to the number of filters 220 in the convolution. out and W out This may depend on the input tensor 210 and the heights and weights of each filter 220.

[0053] As part of the convolution, MAC operations can be performed on the input tensor 210 and the 3x3x3 subtensors 215 within each filter 220 (highlighted with a dot pattern in Figure 2). The result of the MAC operations on the subtensors 215 and one filter 220 is the output activation. In some embodiments (for example, embodiments where the convolution is an integer convolution), the output activation may contain 8 bits, e.g., 1 byte. In other embodiments (for example, embodiments where the convolution is a floating-point convolution), the output activation may contain 2 or more bytes. For example, the output element may contain 2 bytes.

[0054] After the MAC operation is completed on the subtensor 215 and all filters 220, a vector 235 is generated. Vector 235 is highlighted with diagonal lines in Figure 2. Vector 235 contains a sequence of output activations arranged along the Z-axis. Output activations in vector 235 have the same (x,y) coordinates, but each output activation corresponds to a different output channel and has a different Z-coordinate. The dimension of vector 235 along the Z-axis may be equal to the total number of output channels in output tensor 230.

[0055] After vector 235 is generated, further MAC operations are performed to generate additional vectors until output tensor 230 is generated. For example, filter 220 may move along the X or Y axis on input tensor 210, and MAC operations may be performed on filter 220 and another subtensor within input tensor 210 (the subtensor having the same size as filter 220). The amount of movement of filter 220 on input tensor 210 during different computational rounds of the convolution is called the stride size of the convolution. The stride size can be 1 (i.e., the amount of movement of filter 220 is one activation), 2 (i.e., the amount of movement of filter 220 is two activations), and so on. The height and width of output tensor 230 may be determined based on the stride size.

[0056] In some embodiments, MAC operations on a 3×3×3 subtensor (e.g., subtensor 215) and filter 220 may be performed by multiple PEs, such as PE 500 in Figure 5, PE 600 in Figure 6, PE 700 in Figure 7, PE 800 in Figure 8, PE 900 in Figure 9, PE 1000 in Figure 10, or PE 1110 in Figure 11. One or more MAC units may receive an activation operand (e.g., activation operand 217 shown in Figure 2) and a weight operand (e.g., weight operand 227 shown in Figure 2). The activation operand 217 includes a sequence of activations having the same (Y,Z) coordinates but different X coordinates. The weight operand 227 includes a sequence of weights having the same (Y,Z) coordinates but different X coordinates. The length of the activation operand 217 is the same as the length of the weight operand 227. The activations in the activation operand 217 and the weights in the weight operand 227 may be supplied to the PE sequentially. The PE can receive the activation-weight pair at once and multiply the activation by the weight. The position of the activation in the activation operand 217 may coincide with the position of the weight in the weight operand 227.

[0057] Exemplary DNN accelerator Figure 3 is a block diagram of the DNN accelerator 300 in various embodiments. The DNN accelerator 300 can execute a DNN, for example, the DNN 100 in Figure 1. The DNN accelerator 300 includes local memory 410, a DMA (Direct Memory Access) engine 320, and a compute block 330. In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 300. For example, the DNN accelerator 300 may include multiple local memory 410 or multiple DMA engines 320. As another example, the DNN accelerator 300 may include a single compute block 330. Furthermore, functions attributed to the components of the DNN accelerator 300 may be achieved by different components included in the DNN accelerator 300 or by different systems.

[0058] Local memory 410 stores data used by the compute block 330 to perform deep learning operations in the DNN model. Exemplary deep learning operations include convolution (also called “convolution operations”), pooling operations, element-wise operations, other types of deep learning operations, or any combination thereof. Local memory 410 may also be the main memory of the DNN accelerator 300. In some embodiments, local memory 410 includes one or more DRAMs (Dynamic Random Access Memory). For example, local memory 410 may store the input tensor, convolution kernel, or output tensor of a convolutional layer of the DNN, for example, in convolution layer 30. The output tensor may be transmitted from the local memory of the compute block 330 to local memory 410 via the DMA engine 320.

[0059] The DMA engine 320 facilitates data transfer between local memory 410 and the local memories of the compute block 330. For example, the DMA engine 320 can read data from local memory 410 and write data to the local memory of the compute block 330. As another example, the DMA engine 320 can read data from the local memory of the compute block 330 and write data to local memory 410. The DMA engine 320 provides DMA functionality that allows the compute block 330 to initiate data transfer between local memory 410 and the local memories of the compute block 330 and to perform other operations while the data transfer is taking place. In some embodiments, the DMA engine 320 may read a tensor from local memory 410 and modify the tensor in a manner optimized for the compute block 330 before writing the tensor to the local memory of the compute block 330.

[0060] Computation block 330 performs computations for deep learning operations. Computation block 330 may perform operations in the DNN layer, or parts of operations in the DNN layer. Computation block 330 can perform convolutions such as standard convolution (e.g., standard convolution 163 in Figure 1), depth-based convolution (e.g., depth-based convolution 183 in Figure 1), and point-based convolution (e.g., point-based convolution 193 in Figure 1). In some embodiments, computation block 330 receives an input tensor and one or more convolution kernels and performs a convolution using the input tensor and convolution kernels. The result of the convolution may be an output tensor, which can be further computed, for example, by computation block 330 or another computation block. In some embodiments, the operations of the DNN layer may be performed in parallel by multiple computation blocks 330. For example, multiple computation blocks 330 may each perform parts of the workload for convolution. Data may be shared among the computation blocks 330.

[0061] Figure 4 is a block diagram of the compute block 400 according to various embodiments. The compute block 400 may be an example of the compute block 330 in Figure 3. As shown in Figure 4, the compute block 400 includes local memory 410, a read module 420, a write module 430, a PE array 440, and a sparsity module 450. In other embodiments, alternative configurations, different or additional components may be included in the compute block 400. For example, the compute block 400 may include two or more local memory 410s, PE arrays 440s, or sparsity modules 450. Furthermore, functions attributed to the components of the compute block 400 may be achieved by different components included in the compute block 400, other components of the DNN accelerator 300, or different systems. For example, the buffer 445 may be located inside the PE array 440. As another example, at least a portion of the sparsity module 450 may be implemented within the PE array 440.

[0062] The local memory 410 is local to the compute block 400. In the embodiment of Figure 4, the local memory 410 is located inside the compute block 400. In other embodiments, the local memory 410 may be located outside the compute block 400. The local memory 410 and the compute block 400 may be implemented on the same chip. The local memory 410 stores data used for or generated from convolutions, such as input activations, weights, and output activations. In some embodiments, the local memory 410 includes one or more SRAMs (Static Random Access Memory). The local memory 410 may be byte-addressable, with each memory address identifying a single byte (8 bits) of storage. In some embodiments, the local memory 410 may include banks, each bank having a fixed number of bytes, such as 32, 64, etc.

[0063] In some embodiments, the local memory 410 may include data banks. The number of data banks in the local memory 410 may be 128, 256, 512, 1024, 2048, etc. A data bank may include multiple storage units. In one example, a data bank may include 8, 16, 64, or a different number of storage units. A storage unit may have a memory address. In one example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with contiguous memory addresses, i.e., adjacent storage units. For example, one storage unit may store an integer in INT8 format, whereas two storage units may be required to store a number in FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 410 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 410 in multiple read cycles, such as two cycles.

[0064] The read module 420 reads data (e.g., input activations, weights, etc.) from local memory 410 into buffer 445 in the PE array 440. The write module 430 writes data (e.g., output activations, etc.) from buffer 445 to local memory 410. Buffer 445 temporarily stores data being transferred between local memory 410 and the PE array 440. Buffer 445 facilitates data transfer between local memory 410 and the PE array 440 despite the difference between the rate at which data can be received and the rate at which data can be processed. In some embodiments, the storage capacity of buffer 445 may be less than the storage capacity of local memory 410. In one example, buffer 445 contains an array of bytes. The number of bytes in the array can define the width of buffer 445. The width of the buffer may be 16, 32, 64, 128, etc.

[0065] Data from buffer 445 may be loaded into PE array 440, which is used by PE array 440 for MAC calculations. In some embodiments, input activations may be loaded from buffer 445 into an input storage unit in PE array 440. The input storage unit may include one or more register files for storing input activations used for MAC calculations. Weights may be loaded from buffer 445 into a weight storage unit in PE array 440. The weight storage unit may include one or more register files for storing weights used for MAC calculations. The drain module 350 can transfer data generated by PE array 440 to buffer 445. This data may be the result of MAC calculations performed by PE array 440, such as output activations.

[0066] The PE array 440 performs MAC operations in convolution. The PE array 440 may perform other deep learning operations. The PE array 440 may include PEs arranged in columns, or in columns and rows. Each PE can perform MAC operations. In some embodiments, a PE includes one or more multipliers for performing multiplication. A PE may also include one or more adders for performing accumulation. A column of PEs is called a PE column. A PE column may be associated with one or more MAC lanes. A MAC lane is a path for loading data into a MAC column. A MAC lane may also be called a data transmission lane or data load lane. A PE column may have multiple MAC lanes. The load bandwidth of a MAC column is the sum of the load bandwidths of all MAC lanes associated with that MAC column. Using a certain number of MAC lanes, data may be fed simultaneously to the same number of independent PEs. In some embodiments, where a MAC column has four MAC lanes for feeding activations or weights into the MAC column, and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes may have a total load bandwidth of 64 bytes.

[0067] In some embodiments, the PE array 440 may be capable of standard convolution, depth-based convolution, point-based convolution, other types of convolution, or any combination thereof. In standard convolution, the MAC operation may involve accumulation across channels. For example, as opposed to generating an output operand, the PE may accumulate products across different channels to produce a single output point.

[0068] In depth-wise convolution, the PE may perform a MAC operation that includes a sequence of multiplications of the activation operand (e.g., activation operand 217) and the weight operand (e.g., weight operand 227). Each multiplication in the sequence is a multiplication of different activations in the activation operand having different weights in the weight operand. Activations and weights in the same cycle may correspond to the same channel. The sequence of multiplications produces a product operand that includes a sequence of products. The MAC operation may also include a cumulative operation in which multiple product operands are accumulated to produce an output operand of the PE. The PE array 440 can output multiple output operands, each produced by a different PE, at once.

[0069] In some embodiments, the PE may perform multiple rounds of MAC operations for convolution. The data (activations, weights, or both) may be reused within a single round, for example, across different multipliers in the PE, or across different rounds of MAC operations.

[0070] The sparsity module 450 improves the performance of the DNN accelerator 300 and reduces power consumption based on the sparsity of the input data (e.g., activations, weights, etc.) for the deep learning operation. The sparsity module 450 may have sparsity acceleration logic that can identify non-zero activation-weight pairs and skip zero activation-weight pairs. Non-zero activation-weight pairs include non-zero activations and non-zero weights, while zero activation-weight pairs include zero activations or zero weights. The sparsity module 450 can detect sparsity in activations or weights. In situations where the sparsity module detects zero activations or weights, the sparsity module 450 may prevent calculations on the activations or weights. The sparsity module 450 may also prevent activations or weights from entering registers in the PE to reduce the number of gates to switch within the PE.

[0071] The sparsity module 450 can be implemented in hardware, software, firmware, or any combination thereof. In some embodiments, at least a portion of the sparsity module 450 may be located inside the PE. Although Figure 4 shows a single sparsity module 450, the computing block 400 may include multiple sparsity modules 450. In some embodiments, all PEs within the PE array 440 are implemented using sparsity modules 450 to accelerate computations and reduce power consumption in individual PEs. In other embodiments, a subset of the PE array 440 (for example, one or more PE columns within the PE array 440) may be implemented using sparsity modules 450 for accelerated computations within that subset of PEs.

[0072] As shown in Figure 4, the sparsity module 450 includes a sparsity accelerator 460 and a gate switching reducer 470. In other embodiments, alternative configurations, different or additional components may be included in the sparsity module 450. Furthermore, functions attributed to the components of the sparsity module 450 may be achieved by different components included in the sparsity module 450, other components included in the compute block 400, other components of the DNN accelerator 300, or different systems.

[0073] The sparsity accelerator 460 accelerates computations in the PE array 440 based on the sparsity of the computation's input data. In some embodiments (for example, embodiments in which computation block 400 performs a convolutional layer), computations in the PE may be MAC operations on activation operands and weight operands. The activation operand may be part of the input tensor of the convolution. The activation operand includes a sequence of input elements, also known as activations. Activations may come from different input channels. For example, each activation may come from a different input channel than all other activations in the activation operand. The activation operand is associated with an activation bitmap (also called an "activation sparsity vector"), which may be stored in local memory 410. The activation bitmap may indicate the locations of non-zero activations in the activation operand. The activation bitmap may include a sequence of bits, each bit corresponding to a specific activation in the activation operand. The locations of bits in the activation bitmap may coincide with the locations of corresponding activations in the activation operand. Bits in the activation bitmap may be 0 or 1. A bit with a value of 0 indicates that the corresponding activation value is 0, and a bit with a value of 1 indicates that the corresponding activation value is non-zero. In some embodiments, the activation bitmap may be generated during the execution of another DNN layer, for example, a layer placed before a convolutional layer in the DNN.

[0074] A weight operand may be part of the convolution kernel. A weight operand contains a sequence of weights. The values ​​of the weights are determined through training the DNN. The weights in a weight operand may come from different input channels. For example, each weight may come from a different input channel than all the other weights in the weight operand. A weight operand is associated with a weight bitmap (also called an "activation sparsity vector") which may be stored in local memory 410. The weight bitmap can indicate the positions of non-zero weights in the weight operand. The weight bitmap may contain a sequence of bits, where each bit corresponds to a respective weight in the weight operand. The positions of the bits in the weight bitmap may coincide with the positions of the corresponding weights in the weight operand. The bits in the weight bitmap can be 0 or 1. A 0-value bit indicates that the value of the corresponding weight is 0, and a 1-value bit indicates that the value of the corresponding weight is non-zero.

[0075] The sparsity accelerator 460 can receive an activation bitmap and a weight bitmap and generate a combined sparsity bitmap for MAC operations performed by the PE. In some embodiments, the sparsity accelerator 460 generates a combined sparsity bitmap 735 by performing one or more AND operations on the activation bitmap and the weight bitmap. Each bit in the combined sparsity bitmap is the result of an AND operation on the bit in the activation bitmap and the bit in the weight bitmap, i.e., the product of the bit in the activation bitmap and the bit in the weight bitmap. The position of a bit in the combined sparsity bitmap corresponds to the position of a bit in the activation bitmap and the position of a bit in the weight bitmap. The bits in the combined bitmap correspond to activation-weight pairs (activation-weight pairs). A zero bit in the combined sparsity bitmap indicates that at least one of the activations and weights in the pair is zero. A one bit in the combined sparsity bitmap indicates that both the activations and weights in the pair are non-zero. The combined sparsity bitmap can be stored in local memory 410.

[0076] The sparsity accelerator 460 may provide activations and weights to the PE based on a combined sparsity bitmap. For example, the sparsity accelerator 460 may identify one or more non-zero activation-weight pairs from local memory 410 based on a combined sparsity bitmap. Local memory 410 may store activation operands and weight operands in a compressed format such that non-zero activations and non-zero weights are stored, but zero activations and zero weights are not. Non-zero activations of an activation operand may constitute a compressed activation operand. Non-zero weights of a weight operand may constitute a compressed weight operand. For non-zero activation-weight pairs, the sparsity accelerator 460 may determine the location of the activation in the compressed activation operand and the location of the weight in the compressed weight operand based on the activation bitmap, weight bitmap, and combined bitmap. Activations and weights can be read from local memory 410 based on their positions determined by the sparsity accelerator 460.

[0077] In some embodiments, the sparsity accelerator 460 includes sparsity acceleration logic that can compute a position bitmap based on an activation bitmap and a weight bitmap. The sparsity accelerator 460 may determine the position indices of activations and weights based on the position bitmap. In one example, the position index of an activation in a compressed activation operand may be equal to the number of ones in the activation position bitmap generated by the sparsity accelerator 460, and the position index of a weight in a compressed weight operand may be equal to the number of ones in the weight position bitmap generated by the sparsity accelerator 460. The position indices of activations or weights indicate the position of the activation or weight in the compressed activation operand or the compressed weight operand. Based on their position indices, the sparsity accelerator 460 may read the activations and weights from one or more memories.

[0078] The sparsity accelerator 460 can transfer identified non-zero activation-weight pairs to the PE. The sparsity accelerator 460 may skip other activations and weights because they do not contribute to the MAC operation. In some embodiments, local memory 410 may store non-zero activations and weights, but not zero activations or weights. Non-zero activations and weights may be loaded into one or more register files of the PE, from which the sparsity accelerator 460 may retrieve the activations and weights corresponding to 1s in the combined sparsity bitmap. In some embodiments, the total number of 1s in the combined sparsity bitmap is equal to the total number of activation-weight pairs computed by the PE, while the PE computes no other activation-weight pairs. By skipping activation-weight pairs corresponding to zero bits in the combined sparsity bitmap, the PE's computation becomes faster compared to when the PE computes all activation-weight pairs in the activation operand and weight operand.

[0079] The gate switching reducer 470 detects zero-value activations or weights and reduces gate switching in the PE. The gate switching reducer 470 may detect whether an activation or weight is zero-value based on the activation or weight itself, or based on a bitmap. The bitmap may be an activation bitmap, a weight bitmap, or a combined sparsity bitmap, which may be received by the gate switching reducer 470 from the sparsity accelerator 460. In some embodiments, the gate switching reducer 470 includes one or more logical operators, such as logic gates. Examples of logic gates include NOR gates, OR gates, AND gates, and for example, the NOR gate shown in Figures 6 to 10. The gate switching reducer 470 may also include one or more data selectors, such as multiplexers (MUX).

[0080] In some embodiments, the gate switching reducer 470 may receive activations and weights, which are calculated, for example, in the round of operations by the multiplier in the PE. In addition to receiving activations and weights, or instead, the gate switching reducer 470 may receive bits in an activation bitmap representing activations and bits in a weight bitmap representing weights. The gate switching reducer 470 may generate a signal indicating whether the activation or weights are zero. In one example, the gate switching reducer 470 may generate a "0" signal when the activation or weights are zero. In another example, the gate switching reducer 470 may generate a "1" signal when neither the activation nor the weights are zero.

[0081] In other embodiments, the gate switching reducer 470 may determine whether an activation or weight is zero based on bits in a combined sparsity bitmap. The positions of bits in the combined sparsity bitmap may coincide with (for example, be the same as) the positions of activation in the activation operand and weight in the weight operand. The gate switching reducer 470 may determine that an activation or weight is zero when a bit in the combined sparsity bitmap is zero. The gate switching reducer 470 may determine that both an activation and a weight are not zero when a bit in the combined sparsity bitmap is 1.

[0082] After determining that the activation or weight is zero, the gate-switching reducer 470 may prevent the multiplier from processing the activation or weight. In some embodiments, the gate-switching reducer 470 may indicate that the activation and weight enter the multiplier's registers so that the registers can retain the activation and weight from the previous round. This avoids the gate switching required to replace the activation and weight from the previous round with the activation and weight from the current round. The gate-switching reducer 470 may zero out the multiplier's output, for example, by writing a zero-value data element to a register configured to store the multiplier's output. Additionally or alternatively, the gate-switching reducer 470 may zero out the input to an accumulator configured to accumulate the multiplier output, or disable the accumulator for the current round. In some embodiments (for example, embodiments in which the multiplier is pipelined), the gate switching reducer 470 may delay the zeroing signals (for example, a signal to zero out the output of the multiplier, a signal to zero out the input of the accumulator, or a signal to disable the accumulator) by, for example, the number of pipes in the multiplier. The zeroing signals may be stored in a register during the delay.

[0083] In some embodiments, the gate-switching reducer 470 may detect non-zero redundant calculations. Data formats such as floating-point numbers may have other non-zero values ​​that generate non-zero calculations. For example, floating-point numbers may have infinite (INF) and Not-a-Number (NaN) values. In embodiments where the activation or weight is one of these values, the gate-switching reducer 470 may skip these values. The gate-switching reducer 470 may produce NaN / INF values ​​(instead of zero) as the output of the multiplier. In some embodiments, denormalized values ​​may be treated as zero. The gate-switching reducer 470 may determine that the activation or weight having a denormalized value is a zero value.

[0084] In some embodiments (for example, embodiments where the activation or weight is a quantized floating-point number), the gate-switching reducer 470 may determine which quantization method (e.g., symmetric quantization, asymmetric quantization, etc.) was used to quantize the activation or weight. Based on the quantization method, the gate-switching reducer 470 may determine whether the activation or weight is zero. If symmetric quantization is used, zero may be represented as zero. If asymmetric quantization is used, a non-zero value may represent a zero value (e.g., the zero point). The gate-switching reducer 470 may provide configurable inputs for configuring which values ​​are considered zero, and may provide unique values ​​for either the weight or the activation, if necessary.

[0085] In some embodiments, the gate switching reducer 470 may determine that the activation or weight is zero based on the determination that the value of the activation or weight is lower than a threshold number. In embodiments where the activation or weight is greater than a threshold number, the gate switching reducer 470 may determine that both the activation and weight are non-zero. In some embodiments, the gate switching reducer 470 may use the same threshold number for both the activation and the weight. In other embodiments, the threshold number for activation may be different from the threshold number for weight. The threshold number may be configurable.

[0086] In some embodiments, the gate-switching reducer 470 can facilitate pattern-based gating. Some deep learning operations may have a single set of inputs provided to a multi-input PE that are processed by the PE according to a certain pattern. An exemplary pattern may be one data element at a time. The PE can generate multiple outputs from a single set of inputs. Examples of such deep learning operations include element-wise operations, such as element-wise accumulation and element-wise multiplication. The gate-switching reducer 470 can cycle through the patterns (e.g., a walking one for element-wise operations) and be used to provide logic for multiplexing the outputs of multipliers in the PE while keeping the inputs static and minimizing switching activity.

[0087] Exemplary sparsity acceleration in PE Figure 5 shows sparsity acceleration in MAC operations by PE 500 in various embodiments. PE 500 may be a PE within a PE array 440. In the embodiment of Figure 5, PE 500 includes an input register file 510, a weight register file 520, a multiplier 530, an accumulator 540, and an output register file 550. In other embodiments, PE 500 may include fewer, more, or different components. PE 500 is associated with a sparsity module 560, which may be an embodiment of the sparsity accelerator 460 in Figure 4.

[0088] The input register file 510 stores at least a portion of the activation operands. An activation operand includes an input element, also known as a sequence of activations. An activation operand may also be part of an input tensor, for example, the input tensor of a convolutional layer. An activation operand is associated with an activation bitmap 515. The activation bitmap 515 may be stored in the input register file 510, the local memory of the compute block including PE 500, or both. The activation bitmap 515 can indicate the locations of non-zero activations within the activation operand. The activation bitmap 515 includes a sequence of bits, each bit corresponding to a specific activation within the activation operand. In some embodiments, the location of a bit in the activation bitmap 515 coincides with the location of a corresponding activation within the activation operand. For illustrative purposes, the activation bitmap 515 contains 8 bits, and the activation operand contains 8 activations. In other embodiments, the activation bitmap 515 may contain fewer or more bits. As shown in Figure 5, four of the eight bits in the activation bitmap 515 are valued as 0, and the other four bits are valued as 1. The bits valued as 0 indicate that the value of the corresponding activation is 0, and the bits valued as 1 indicate that the value of the corresponding activation is not 0. Therefore, the activation operand includes four zero-value activations and four non-zero-value activations.

[0089] The weight register file 520 stores at least a portion of the weight operands. A weight operand contains a sequence of weights. A weight operand may be part of a filter, for example, a filter in a convolutional layer. A weight operand is associated with a weight bitmap 525. The weight bitmap 525 may be stored in the local memory of the compute block containing the weight register file 520, PE 500, or both. The weight bitmap 525 can indicate the positions of non-zero weights within the weight operand. The weight bitmap 525 contains a sequence of bits, each bit corresponding to each weight within the weight operand. In some embodiments, the positions of the bits in the weight bitmap 525 coincide with the positions of the corresponding weights within the weight operand. For illustrative purposes, the weight bitmap 525 contains 8 bits, and the weight operand contains 8 weights. In other embodiments, the weight bitmap 525 may contain fewer or more bits. As shown in Figure 5, four of the 8 bits in the weight bitmap 525 are 0 and the other four are 1. A bit with a value of 0 indicates that the corresponding weight has a value of 0, and a bit with a value of 1 indicates that the corresponding weight has a value of non-zero. Thus, the weight operand contains four zero-value weights and four non-zero-value weights. The weight bitmap 525 can indicate the positions of the non-zero weights within the weight operand.

[0090] The sparsity module 560 generates a combined sparsity bitmap 535 based on the activation bitmap 515 and the weight bitmap 525. The sparsity module 560 may receive the activation bitmap 515 from the local memory of the compute block, including the input register file 510 or PE 500. The sparsity module 560 may receive the weight bitmap 525 from the weight register file 520 or the local memory of the compute block. In some embodiments, the sparsity module 560 is an AND operator. The sparsity module 560 may generate the combined sparsity bitmap 535 by performing one or more AND operations on the activation bitmap 515 and the weight bitmap 525. Each bit in the combined sparsity bitmap 535 is the result of an AND operation on the bits in the activation bitmap 515 and the bits in the weight bitmap 525. The positions of the bits in the combined sparsity bitmap 535 correspond to the positions of the bits in the activation bitmap 515 and the weight bitmap 525. For example, the first bit in the combined sparsity bitmap 535 is the result of an AND operation on the first bit in the activation bitmap 515 and the first bit in the weight bitmap 525; the second bit in the combined sparsity bitmap 535 is the result of an AND operation on the second bit in the activation bitmap 515 and the second bit in the weight bitmap 525; the third bit in the combined sparsity bitmap 535 is the result of an AND operation on the third bit in the activation bitmap 515 and the third bit in the weight bitmap 525; and so on.

[0091] A bit in the combined sparsity bitmap 535 has a value of 1 when both the corresponding bit in the activation bitmap 515 and the corresponding bit in the weight bitmap 525 have a value of 1. A bit in the combined sparsity bitmap 535 has a value of 0 when at least one of the corresponding bits in the activation bitmap 515 and the corresponding bit in the weight bitmap 525 has a value of 0. As shown in Figure 5, the combined sparsity bitmap 535 contains six zeros and two ones.

[0092] The total number of 1s in the combined sparsity bitmap 535 is equal to the total number of non-zero activation-weight pairs that PE 500 calculates to compute the non-zero partial sum. Other activation-weight pairs are zero-value activation-weight pairs, and since these pairs result in a zero-value partial sum, they can be skipped for computation without any impact on output precision. Thus, the workload of PE 500 in this computation round can be determined based on the total number of 1s in the combined sparsity bitmap 535. The amount of time for computation can also be estimated based on the total number of 1s in the combined sparsity bitmap 535. The more 1s there are in the combined sparsity bitmap 535, the higher the workload of PE 500 and the longer the computation of PE 500.

[0093] In some embodiments, the input register file 510 or the weight register file 520 stores dense data points, for example, non-zero activations or non-zero weights. Sparse data points, for example, zero activations or zero weights, are not stored in the input register file 510 or the weight register file 520. Dense data points may be compressed and held adjacent to each other in the input register file 510 or the weight register file 520. One or more dense data points of an activation operand constitute a compressed activation operand. One or more dense data points of a weight operand constitute a compressed weight operand. A position 1 in the combined sparsity bitmap 535 cannot indicate the position of an activation in a compressed activation operand or the position of a weight in a compressed weight operand. The sparsity module 560 may perform sparsity calculations to determine the positions of activations in a compressed activation operand and the positions of weights in a compressed weight operand. The sparsity module 560 may perform rounds of sparsity calculations for each of the two non-zero activation-weight pairs. In each round of sparsity calculations, the sparsity module 560 may compute an activation position bitmap and a weight position bitmap based on the activation bitmap 515, the weight bitmap 525, and the combined sparsity bitmap 535. The position of an activation in a compressed activation operand may be indicated by the number of 1s in the activation position bitmap, and the position of a weight in a compressed weight operand may be indicated by the number of 1s in the weight position bitmap. In the first round of sparsity calculations, an intermediate bitmap may be determined, which may be used in a second round to identify the next non-zero activation-weight pair.

[0094] The sparsity module 560 can read the activations and weights of non-zero activation-weight pairs from the input register file 510 and the weight register file 520 based on their positions determined through sparsity calculations, and provide the activations and weights to the multiplier 530. The multiplier 530 performs multiplication operations on the activations and weights. For example, the multiplier 530 performs multiplication operations on the activations and weights in each non-zero activation-weight individual pair and outputs a partial sum, i.e., the product of that activation and weight. Since there are two activation-weight pairs, the multiplier 530 can sequentially perform two multiplication operations, for example, based on the position of 1 in the combined sparsity bitmap 535. Without sparsity acceleration, the multiplier 530 would need to perform eight multiplication operations. By reducing the number of multiplication operations from eight to two, the MAC operation in PE 500 is accelerated. Since DNN accelerators typically perform a large number of MAC operations during DNN execution, sparsity acceleration can significantly improve the efficiency and performance of DNN accelerators.

[0095] The accumulator 540 receives two partial sums from the multiplier 530 and accumulates the two partial sums. The result of the accumulation is the internal partial sum at the PE level. The internal partial sum at the PE level may be stored in the output register file 550. In some embodiments, the accumulator 540 receives one or more internal partial sums at the PE level from one or more other PEs. The accumulator 540 can accumulate one or more internal partial sums at the PE level with the internal partial sum at the PE level of PE 500 and store the result of the accumulation (i.e., a multi-PE internal partial sum) in the output register file 550. The one or more other PEs may be in the same column as PE 500 in the PE array. The multi-PE internal partial sum may be a column-level internal partial sum. In some embodiments, the internal partial sum at the PE level of PE 500 or the multi-PE internal partial sum may be sent to one or more other PEs for further accumulation.

[0096] Although Figure 5 shows a single multiplier 530, PE 500 may include multiple multipliers capable of performing multiple multiplication operations simultaneously. These multipliers can be coupled to an internal adder assembly, for example, the internal adder assembly 1240 in Figure 12.

[0097] An example of sparsity-based reduction of gate switching in PE. Figure 6 shows PE 600 in various embodiments that can reduce gate switching based on sparsity. PE 600 may be a PE within the PE array 440 in Figure 4. As shown in Figure 6, PE 600 includes registers 610, 620, 650, multiplier 630, MUX 640, and NOR gates 660, 670, 680. In other embodiments, alternative configurations, different components, or additional components may be included in PE 600. Furthermore, functions attributed to the components of PE 600 may be achieved by different components included in PE 600, other components included in the PE array, or different devices or systems.

[0098] Register 610 is a storage unit for storing activations processed by the multiplier 630. Register 610 can store one activation at a time. In some embodiments, register 610 may be associated with a clock and updated when a new clock cycle begins. For example, an activation written to register 610 in the current clock cycle may be replaced by another activation written to register 610 in the next clock cycle.

[0099] Register 620 is a memory unit for storing weights processed by the multiplier 630. Register 620 may store one weight at a time. In some embodiments, register 620 may be associated with a clock and updated when a new clock cycle begins. For example, a weight written to register 620 in the current clock cycle may be replaced by another weight written to register 620 in the next clock cycle. Registers 610 and 620 are sometimes referred to as the input registers of the multiplier 630.

[0100] For each multiplication round, NOR gates 660, 670, and 680 can detect whether the activation or weight for the current round is zero before the activation and weight are written to registers 610 and 620, for example, while the activation and weight from the previous round are still stored in registers 610 and 620. NOR gate 660 takes the activation from register 610 as input, and NOR gate 670 takes the weight from register 620 as input. A non-zero input results in a "0" output signal for NOR gate 660 or 670. A zero input results in a "1" output signal for NOR gate 660 or 670.

[0101] NOR gate 680 takes the outputs of NOR gates 660 and 670 as inputs. If either input is "1", NOR gate 680 outputs a "0" output signal. The "0" output signal of NOR gate 680 may be a control signal that prevents the activation and weights for this round from being written to registers 610 and 620. This control signal may also activate or deactivate multiplier 630. In some embodiments, registers 610 and 620 are not updated in this round when given the "0" output signal of NOR gate 680. The activation and weights for the previous round, whose product was calculated by multiplier 630 in the previous round, may remain in registers 610 and 620. In this way, gate switching can be reduced or avoided. In some embodiments, multiplier 630 may calculate the product of the activation and weights for the previous round in this round.

[0102] In the embodiment of Figure 6, NOR gates 660, 670, and 680 are used to detect zero-value activations or weights, but in other embodiments, different, fewer, or more logic gates may be used. For example, one or more OR gates, AND gates, or other types of logic gates may be used in addition to, or instead of, the NOR gates 660, 670, and 680. Also, while the embodiment of Figure 6 uses the activation or weight itself for zero detection, in other embodiments, different data may be used to detect zero-value activations or weights. For example, bits in the activation bitmap of the activation operand may be used to determine whether the activation is zero-value. Similarly, bits in the weight bitmap of the weight operand may be used to determine whether the weight is zero-value. As another example, bits in a combined sparsity bitmap may be used to determine whether the activation or weight is zero-value.

[0103] MUX 640 receives the output of multiplier 630 (i.e., the product of activation and weight for the previous round). MUX 640 also receives the output signal of NOR gate 680. MUX 640 may generate a "0" output signal which is written to register 650. Register 650 is the output register of the multiplier. In this way, even though multiplier 630 does not process the activation and weight for this round, the output of multiplier 630 is zeroed out by using MUX 640. In other embodiments, an AND gate may be used instead of MUX 640.

[0104] If both the activation and weights for this round are non-zero, the outputs of NOR gates 660 and 670 are "0", and NOR gate 680 outputs a "1" signal. Registers 610 and 620 are updated so that the activation and weights for the previous round are replaced by the activation and weights for this round. Multiplier 630 calculates the product of the activation and weight for this round, and the product is received by MUX 640. MUX 640 also receives the "1" output signal from NOR gate 680. MUX 640 outputs the product, which is written to register 650.

[0105] Figure 7 shows a PE 700 having a pipelined multiplier 730 in various embodiments. The PE 700 may be a PE within the PE array 440 in Figure 4. As shown in Figure 7, the PE 700 also includes registers 710, 720, 750, 790, 795, MUX 740, and NOR gates 760, 770, 780. In other embodiments, alternative configurations, different components, or additional components may be included in the PE 700. Furthermore, functions attributed to the components of the PE 700 may be achieved by different components included in the PE 700, other components included in the PE array, or different devices or systems.

[0106] Register 710 is a memory unit that stores activations processed by the multiplier 730. Register 710 can store one activation at a time. In some embodiments, register 710 may be associated with a clock and updated when a new clock cycle begins. For example, an activation written to register 710 in the current clock cycle may be replaced by another activation written to register 710 in the next clock cycle.

[0107] Register 720 is a memory unit that stores the weights processed by the multiplier 730. Register 720 can store one weight at a time. In some embodiments, register 720 can be associated with a clock and updated when a new clock cycle begins. For example, a weight written to register 720 in the current clock cycle may be replaced by another weight written to register 720 in the next clock cycle. Registers 710 and 720 are sometimes referred to as the input registers of the multiplier 730.

[0108] For each multiplication round, NOR gates 760, 770, and 780 can detect whether the activation or weight for the current round is zero before the activation and weight are written to registers 710 and 720, for example, while the activation and weight from the previous round are still stored in registers 710 and 720. NOR gate 760 takes the activation from register 710 as input, and NOR gate 770 takes the weight from register 720 as input. A non-zero input results in a "0" output signal for NOR gate 760 or 770. A zero input results in a "1" output signal for NOR gate 760 or 770.

[0109] The NOR gate 780 takes the outputs of NOR gates 760 and 770 as inputs. If either input is "1", the NOR gate 780 outputs a "0" output signal. The "0" output signal of the NOR gate 780 can be a control signal that prevents the activation and weights for this round from being written to registers 710 and 720. This control signal can also activate or deactivate the multiplier 730. In some embodiments, registers 710 and 720 are not updated in this round when the "0" output signal of the NOR gate 780 is given. The activation and weights for the previous round, whose product was calculated by the multiplier 730 in the previous round, may remain in registers 710 and 720. In this way, gate switching can be reduced or avoided.

[0110] In the embodiment of Figure 7, NOR gates 760, 770, and 780 are used to detect zero-value activations or weights, but in other embodiments, different, fewer, or more logic gates may be used. For example, one or more OR gates, AND gates, or other types of logic gates may be used in addition to, or instead of, the NOR gates 760, 770, and 780. Also, while the embodiment of Figure 7 uses the activation or weight itself for zero detection, in other embodiments, different data may be used to detect zero-value activations or weights. For example, bits in the activation bitmap of the activation operand may be used to determine whether the activation is zero-value. Similarly, bits in the weight bitmap of the weight operand may be used to determine whether the weight is zero-value. As another example, bits in a combined sparsity bitmap may be used to determine whether the activation or weight is zero-value.

[0111] The multiplier 730 is pipelined. For simplicity and illustrative purposes, the multiplier 730 has two pipes, represented by rectangles within the multiplier 730 in Figure 7. The two pipes are associated with two registers 790 and 795, respectively. Registers 790 and 795 can delay the zeroing signal from the NOR gate 780 (or from the NOR gates 760, 770, and 780). The zeroing signal from the NOR gate 780 can be delayed by the two pipes. After the pipeline in the multiplier 730 is complete, the MUX 740 receives the "0" output signal from the NOR gate 780 and may output zero, which is written to register 750. Register 750 is the output register of the multiplier. In this way, the output of the multiplier 730 is zeroed by using the MUX 740, without the multiplier 730 having to handle activation and weighting for this round. In other embodiments, an AND gate may be used instead of the MUX 740.

[0112] If both the activation and weights for this round are non-zero, the outputs of NOR gates 760 and 770 are "0", and NOR gate 780 outputs a "1" signal. Registers 710 and 720 are updated so that the activation and weights for the previous round are replaced by the activation and weights for this round. Multiplier 730 calculates the product of the activation and weight for this round, and the product is received by MUX 740. MUX 740 also receives the "1" output signal from NOR gate 780. MUX 740 outputs the product, which is written to register 750.

[0113] Figure 8 shows a PE having a multiplier 830 and an accumulator 845 according to various embodiments. Gate switching and power consumption in PE 800 can be reduced by zeroing the input to the accumulator 845 when the activation or weight in the activation-weight pair is zero. PE 800 may also be a PE within the PE array 440 in Figure 4. As shown in Figure 8, PE 800 also includes registers 810, 820, 850, 890, 895, AND gate 840, and NOR gates 860, 870, 880. In other embodiments, alternative configurations, different components, or additional components may be included in PE 800. Furthermore, functions attributed to the components of PE 800 may be achieved by different components included in PE 800, other components included in the PE array, or different devices or systems.

[0114] Register 810 is a memory unit that stores activations processed by the multiplier 830. Register 810 can store one activation at a time. In some embodiments, register 810 may be associated with a clock and updated when a new clock cycle begins. For example, an activation written to register 810 in the current clock cycle may be replaced by another activation written to register 810 in the next clock cycle.

[0115] Register 820 is a memory unit that stores the weights processed by the multiplier 830. Register 820 may store one weight at a time. In some embodiments, register 820 may be associated with a clock and updated when a new clock cycle begins. For example, a weight written to register 820 in the current clock cycle may be replaced by another weight written to register 820 in the next clock cycle. Registers 810 and 820 are sometimes referred to as the input registers of the multiplier 830.

[0116] For each multiplication round, NOR gates 860, 870, and 880 can detect whether the activation or weight for the current round is zero before the activation and weight are written to registers 810 and 820, for example, while the activation and weight from the previous round are still stored in registers 810 and 820. NOR gate 860 takes the activation from register 810 as input, and NOR gate 870 takes the weight from register 820 as input. A non-zero input results in a "0" output signal for NOR gate 860 or 870. A zero input results in a "1" output signal for NOR gate 860 or 870.

[0117] The NOR gate 880 takes the outputs of NOR gates 860 and 870 as inputs. If either input is "1", the NOR gate 880 outputs a "0" output signal. The "0" output signal of the NOR gate 880 may be a control signal that prevents the activation and weights for this round from being written to registers 810 and 820. This control signal may be used to control the input to the accumulator 845. In some embodiments, registers 810 and 820 are not updated in this round, given the "0" output signal of the NOR gate 880. The activation and weights for the previous round, whose product was calculated by multiplier 830 in the previous round, may remain in registers 810 and 820. In this way, gate switching can be reduced or avoided. The output signal of the NOR gate 880 may be stored in register 890, for example, for synchronization purposes.

[0118] In the embodiment of Figure 8, NOR gates 860, 870, and 880 are used to detect zero-value activations or weights, but in other embodiments, different, fewer, or more logic gates may be used. For example, one or more OR gates, AND gates, or other types of logic gates may be used in addition to, or instead of, the NOR gates 860, 870, and 880. Also, while the embodiment of Figure 8 uses the activation or weight itself for zero detection, in other embodiments, different data may be used to detect zero-value activations or weights. For example, bits in the activation bitmap of the activation operand may be used to determine whether the activation is zero-value. Similarly, bits in the weight bitmap of the weight operand may be used to determine whether the weight is zero-value. As another example, bits in a combined sparsity bitmap may be used to determine whether the activation or weight is zero-value.

[0119] The multiplier 830 may recalculate the product of activation and weight for a previous round, which requires fewer gate switching compared to calculating the product of activation and weight for the current round. The output of the multiplier 830 may be stored in register 850, for example, for synchronization purposes.

[0120] The AND gate 840 receives data from registers 850 and 890 as input. The AND gate 840 performs an AND operation on the input. If the control signal from the NOR gate 880 is a "0" signal, the AND gate 840 outputs a "0" signal. The control signal can zero out the output of the multiplier 830 through the AND operation by the AND gate 840. The accumulator 845 receives a zero signal from the AND gate 840 as input. In this way, the input to the accumulator 845 is zeroed out by using the AND gate 840, without the multiplier 830 having to process the activation and weights for this round. The accumulator 845 may accumulate inputs in other rounds using one or more signals from the AND gate 840. The sum calculated by the accumulator 845 may be a partial sum in a MAC operation.

[0121] If both the activation and weights for this round are non-zero, the outputs of NOR gates 860 and 870 are "0", and NOR gate 880 outputs a "1" signal, which is stored in register 890. Registers 810 and 820 are updated so that the activation and weights for the previous round are replaced by the activation and weights for this round. Multiplier 830 calculates the product of the activation and weight for this round, and this product is stored in register 850. AND gate 840 receives data from registers 850 and 890 and outputs the result of an AND operation between the "1" output signal from NOR gate 880 and the product calculated by multiplier 830. The result is equal to the product calculated by multiplier 830. Accumulator 845 then receives the product calculated by multiplier 830 and may accumulate the product calculated by multiplier 830 along with one or more signals from AND gate 840 in other rounds.

[0122] Figure 9 shows a PE with an accumulator 945 having registers 995 that can be configured based on sparsity, according to various embodiments. Gate switching and power consumption in PE 900 can be reduced by disabling the accumulator 945 when the activation or weight in the activation-weight pair is zero. PE 900 may also be a PE within the PE array 440 in Figure 4. As shown in Figure 9, PE 900 also includes registers 910, 920, 950, 990, 995 and NOR gates 960, 970, 980. In other embodiments, alternative configurations, different components, or additional components may be included in PE 900. Furthermore, functions attributed to the components of PE 900 may be achieved by different components included in PE 900, other components included in the PE array, or different devices or systems.

[0123] Register 910 is a memory unit that stores activations processed by the multiplier 930. Register 910 can store one activation at a time. In some embodiments, register 910 may be associated with a clock and updated when a new clock cycle begins. For example, an activation written to register 910 in the current clock cycle may be replaced by another activation written to register 910 in the next clock cycle.

[0124] Register 920 is a memory unit that stores the weights processed by the multiplier 930. Register 920 can store one weight at a time. In some embodiments, register 920 can be associated with a clock and updated when a new clock cycle begins. For example, a weight written to register 920 in the current clock cycle may be replaced by another weight written to register 920 in the next clock cycle. Registers 910 and 920 are sometimes referred to as the input registers of the multiplier 930.

[0125] For each multiplication round, NOR gates 960, 970, and 980 can detect whether the activation or weight for the current round is zero before the activation and weight are written to registers 910 and 920, for example, while the activation and weight from the previous round are still stored in registers 910 and 920. NOR gate 960 takes the activation from register 910 as input, and NOR gate 970 takes the weight from register 920 as input. A non-zero input results in a "0" output signal for NOR gate 960 or 970. A zero input results in a "1" output signal for NOR gate 960 or 970.

[0126] The NOR gate 980 takes the outputs of NOR gates 960 and 970 as inputs. If either input is "1", the NOR gate 980 outputs a "0" output signal. The "0" output signal of the NOR gate 980 can prevent the writing of activation and weights for this round to registers 910 and 920. In some embodiments, registers 910 and 920 are not updated in this round, given the "0" output signal of the NOR gate 980. The activation and weights for the previous round, whose product was calculated by the multiplier 930 in the previous round, may remain in registers 910 and 920. In this way, gate switching can be reduced or avoided. The output signal of the NOR gate 980 may be stored in register 995 as a control signal that can control, for example, register 990.

[0127] In the embodiment of Figure 9, NOR gates 960, 970, and 980 are used to detect zero-value activations or weights, but in other embodiments, different, fewer, or more logic gates may be used. For example, one or more OR gates, AND gates, or other types of logic gates may be used in addition to, or instead of, the NOR gates 960, 970, and 980. Also, while the embodiment of Figure 9 uses the activation or weight itself for zero detection, in other embodiments, different data may be used to detect zero-value activations or weights. For example, bits in the activation bitmap of the activation operand may be used to determine whether the activation is zero-value. Similarly, bits in the weight bitmap of the weight operand may be used to determine whether the weight is zero-value. As another example, bits in a combined sparsity bitmap may be used to determine whether the activation or weight is zero-value.

[0128] The multiplier 930 may recalculate the product of activation and weight for a previous round, which requires fewer gate switching compared to calculating the product of activation and weight for the current round. The output of the multiplier 930 may be stored in register 950.

[0129] A "0" signal stored in register 990 can prevent register 995 from being updated, meaning that accumulator 945 is disabled for this round and does not have to perform any accumulation in this round, which can reduce gate switching in this round. The signal in register 990 can be a control signal, which may be a clock enable signal or a clock disable signal for register 995. Register 995 is configured to store the output of accumulator 945. In some embodiments (such as those in which multiplier 830 is piped), the disable signal may be delayed by, for example, the number of pipes in multiplier 830. The delay may be facilitated by register 990. For example, the disable signal may be held in register 990 during the delay.

[0130] If both the activation and weights for this round are non-zero, the outputs of NOR gates 960 and 970 are "0", and NOR gate 980 outputs a "1" signal, which is stored in register 990. Registers 910 and 920 are updated so that the activation and weights for the previous round are replaced by the activation and weights for this round. Multiplier 930 calculates the product of the activation and weight for this round, and this product is stored in register 950. Register 990 stores the "1" output signal, which is not a deactivation signal. Accumulator 945 receives the product calculated by multiplier 930 and may accumulate the product calculated by multiplier 930 with one or more signals from MUX 940 for other rounds.

[0131] Figure 10 shows PE 1000 having an adder tree in various embodiments. PE 1000 may be a PE within the PE array 440 in Figure 4. In addition to the adder tree, PE 1000 includes multiplication assemblies 1005A to 1005D (collectively referred to as “multiplication assemblies 1005” or “multiplication assemblies 1005”). In other embodiments, alternative configurations, different or additional components may be included in PE 1000. For example, PE 1000 may include a different number of multiplication assemblies 1005. Different multiplication assemblies 1005 may have different components or configurations. Furthermore, functions attributed to the components of PE 1000 may be achieved by different components included in PE 1000, other components included in the PE array, or different devices or systems.

[0132] The multiplication assembly 1005 calculates the product of activation-weight pairs. The multiplication assembly 1005 sends these products to an adder tree for accumulation. The adder tree includes three accumulators 1070, 1080, and 1090. The adder tree has two hierarchies, with the first hierarchy containing accumulators 1070 and 1080, and the second hierarchy containing accumulator 1090. In other embodiments, the adder tree may contain a different number of accumulators or have a different hierarchical structure.

[0133] The multiplication assembly 1005 includes registers 1010, 1020, and 1050, a multiplier 1030, an AND gate 1040, and a zero detector 1060. The multiplication assembly can detect zero-value activations or weights to accelerate computation in PE 1000 and reduce the power consumed by PE 1000 for computation. Register 1010 is a memory unit that stores activations processed by multiplier 1030. Register 1010 can store one activation at a time. In some embodiments, register 1010 may be associated with a clock and updated when a new clock cycle begins. For example, an activation written to register 1010 in the current clock cycle may be replaced by another activation written to register 1010 in the next clock cycle.

[0134] Register 1020 may be a memory unit that stores the weights processed by the multiplier 1030. Register 1020 may store one weight at a time. In some embodiments, register 1020 may be associated with a clock and updated when a new clock cycle begins. For example, a weight written to register 1020 in the current clock cycle may be replaced by another weight written to register 1020 in the next clock cycle. Registers 1010 and 1020 are sometimes referred to as the input registers of the multiplier 1030.

[0135] The zero detector 1060 detects whether the activation or weight is zero for a round of multiplication. In some embodiments, the zero detector 1060 includes one or more logic operators, such as NOR gates, like the NOR gates 660, 670, and 680 in Figure 6. The zero detector 1060 may output a signal indicating whether the activation or weight is zero. For example, the zero detector may output a "0" output signal in embodiments where the activation or weight is zero, and a "1" output signal in embodiments where neither the activation nor the weight is zero.

[0136] The AND gate 1040 receives the output of the multiplier 1030 and the output of the zero detector 1060. In embodiments where the activation or weights are zero, the AND gate 1040 may output a "0" signal. In embodiments where neither the activation nor the weights are zero, the AND gate 1040 may output the product calculated by the multiplier 1030. The output of the AND gate 1040 may be written to register 1050, which is the output register of the multiplication assembly 1005.

[0137] The data in register 1050 is sent to accumulator 1070 or 1080 coupled to multiplication assembly 1005. Accumulator 1070 may accumulate the outputs of multiplication assemblies 1005A and 1005B. Accumulator 1080 may accumulate the outputs of multiplication assemblies 1005C and 1005D. The outputs of accumulators 1070 and 1080 are provided to accumulator 1090, which accumulates the outputs from the first hierarchy to generate the output of PE 1000. In some embodiments, accumulator 1090 may be associated with a register (not shown in Figure 10) in which the output of accumulator 1090 may be stored. Accumulator 1090 may accumulate its own output with one or more outputs from one or more other PE 1000s. In some embodiments, gate switching in accumulators 1070, 1080, or 1090 can be reduced using the mechanism described above in relation to Figures 8 and 9.

[0138] Exemplary PE array Figure 11 shows PE array 1100 in various embodiments. PE array 1100 may be an embodiment of PE array 440 in Figure 4. PE array 1100 includes a plurality of PEs 1110 (each individually referred to as "PE 1110"). PEs 1110 perform MAC operations. PEs 1110 are sometimes referred to as neurons in a DNN. Each PE 1110 has two input signals 1150 and 1160 and an output signal 1170. Input signal 1150 is at least part of the IFM to the layer. Input signal 1160 is at least part of the filter to the layer. In some embodiments, input signal 1150 of PE 1110 includes one or more activation operands, and input signal 1160 includes one or more weight operands.

[0139] Each PE 1110 performs a MAC operation on input signals 1150 and 1160 and outputs an output signal 1170, which is the result of the MAC operation. Some or all of the input signals 1150 and 1160 and the output signal 1170 may be in integer form such as INT8, or floating-point form such as FP16 or BF16. For simplicity and illustration, all PE 1110 input and output signals have the same reference number, but PE 1110 may receive different input signals from one another and output different output signals. Also, PE 1110 may be different from another PE 1110, for example, it may contain more, fewer, or different components.

[0140] As shown in Figure 11, the PEs 1110 are connected to one another as indicated by the dashed arrows in Figure 11. The output signal 1170 of PE 1110 can be transmitted as an input signal to many other PEs 1110 through the interconnection between PEs 1110 (and potentially returned to itself). In some embodiments, the output signal 1170 of PE 1110 may incorporate the output signals of one or more other PEs 1110 through the cumulative operation of PE 1110 to generate an internal partial sum of the PE array. Further details about PE 1110 will be discussed later in relation to Figure 11B.

[0141] In the embodiment shown in Figure 11, the PEs 1110 are arranged in column 1105 (each referred to as “column 1105”). The inputs and weights of the layer may be distributed to the PEs 1110 based on column 1105. Each column 1105 has a column buffer 1120. The column buffer 1120 temporarily stores the data provided to the PEs 1110 in column 1105. The column buffer 1120 may also store the data output by the last PE 1110 in column 1105. The output of the last PE 1110 may be the sum of the MAC operations of all PEs 1110 in column 1105, which is the column-level internal partial sum of the PE array 1100. In other embodiments, the inputs and weights may be distributed to the PEs 1110 based on rows in the PE array 1100. The PE array 1100 may include row buffers instead of column buffers 1120. The row buffer may store the input signals of the PEs in the corresponding row, or it may store the row-level internal partial sums of the PE array 1100.

[0142] As shown in Figure 11, each column buffer 1120 is associated with a load 1130 and a drain 1140. Data provided to column 1105 is transmitted to column buffer 1120 through the load 1130, for example, through a higher memory hierarchy, such as local memory 410 in Figure 4. Data generated by column 1105 is extracted from column buffer 1120 through drain 1140. In some embodiments, the data extracted from column buffer 1120 is sent through a drain operation to a higher memory hierarchy, such as local memory 410 in Figure 4. In some embodiments, the drain operation is not initiated until all PEs 1110 in column 1105 have completed their MAC operations. Although not shown in Figure 11, one or more columns 1105 may be associated with an external adder assembly.

[0143] Figure 12 is a block diagram of PE 1200 in various embodiments. PE 1200 may be an embodiment of PE 1110 in Figure 11. PE 1200 includes an input register file 1210 (each referred to as “input register file 1210”), a weight register file 1220 (each referred to as “weight register file 1220”), a multiplier 1230 (each referred to as “multiplier 1230”), an internal adder assembly 1240, and an output register file 1250. In other embodiments, PE 1200 may include fewer, more, or different components. For example, PE 1200 may include multiple output register files 1250. As another example, PE 1200 may include a single input register file 1210, a weight register file 1220, or a multiplier 1230. As yet another example, PE 1200 may include an adder instead of the internal adder assembly 1240.

[0144] The input register file 1210 temporarily stores the activation operands for MAC operations performed by the PE 1200. In some embodiments, the input register file 1210 may store a single activation operand at a time. In other embodiments, the input register file 1210 may store multiple activation operands or a portion of activation operands at a time. An activation operand contains multiple input elements (i.e., input elements) in an input tensor. The input elements of an activation operand may be stored sequentially in the input register file 1210 so that the input elements can be processed sequentially. In some embodiments, each input element in an activation operand may come from a different input channel of the input tensor. An activation operand may contain input elements from each of the input channels of the input tensor, and the number of input elements in an activation operand may be equal to the number of input channels. The input elements in an activation operand may have the same XY coordinates, which may be used as the XY coordinates of the activation operand. For example, all input elements of an activation operand may be X0Y0, X0Y1, X1Y1, etc.

[0145] The weight register file 1220 temporarily stores weight operands for MAC operations performed by PE 1200. The weight operands include the weights in the DNN layer's filter. In some embodiments, the weight register file 1220 may store a single weight operand at a time. In other embodiments, the input register file 1210 may store multiple weight operands or a portion of weight operands at once. A weight operand may contain multiple weights. The weights of a weight operand may be stored sequentially in the weight register file 1220, so that the weights can be processed sequentially. In some embodiments, for multiplication operations involving weight operands and activation operands, each weight in the weight operand may correspond to an input element of the activation operand. The number of weights in the weight operand may be equal to the number of input elements in the activation operand.

[0146] In some embodiments, the weight register file 1220 may be the same as or similar to the input register file 1210, and may have the same size, for example. The PE 1200 may contain multiple register files, some of which are designated as the input register file 1210 for storing activation operands, some as the weight register file 1220 for storing weight operands, and some as the output register file 1250 for storing output operands. In other embodiments, register files within the PE 1200 may be designated for other purposes, for example, for storing scale operands used in element-wise addition operations.

[0147] The multiplier 1230 performs multiplication operations on the activation operand and the weight operand. The multiplier 1230 may perform a sequence of multiplication operations on a single activation operand and a single weight operand, generating a product operand containing a sequence of products. Each multiplication operation in the sequence involves multiplying the input element in the activation operand by the weight in the weight operand. In some embodiments, the position (or index) of the input element in the activation operand coincides with the position (or index) of the weight in the weight operand. For example, the first multiplication operation is the multiplication of the first input element in the activation operand by the first weight in the weight operand, the second multiplication operation is the multiplication of the second input element in the activation operand by the second weight in the weight operand, the third multiplication operation is the multiplication of the third input element in the activation operand by the third weight in the weight operand, and so on. The input elements and weights in the same multiplication operation may correspond to channels of the same depth, and their product may also correspond to channels of the same depth.

[0148] Multiple multipliers 1230 may perform multiplication operations simultaneously. These multiplication operations are sometimes referred to as a round of multiplication operations. In a round of multiplication operations by the multipliers 1230, each multiplier 1230 may use different activation operands and different weight operands. Different activation operands or weight operands may be stored in different register files of PE 1200. For example, the first multiplier 1230 uses a first activation operand (for example, stored in the first input register file 1210) and a first weight operand (for example, stored in the first weight register file 1220), the second multiplier 1230 uses a second activation operand (for example, stored in the second input register file 1210) and a second weight operand (for example, stored in the second weight register file 1220), the third multiplier 1230 uses a third activation operand (for example, stored in the third input register file 1210) and a third weight operand (for example, stored in the third weight register file 1220), and so on. For each multiplier 1230, a round of multiplication operations may include multiple cycles. Each cycle includes multiplication operations between the input elements and the weights.

[0149] The multiplier 1230 can perform multiplication operations over multiple rounds. The multiplier 1230 may use the same weight operands in different rounds, but may also use different activation operands. For example, the multiplier 1230 performs a sequence of multiplication operations on a first activation operand stored in a first input register file in the first round, and on a second activation operand stored in a second input register file in the second round. In the second round, a different multiplier 1230 may perform another sequence of multiplication operations using the first activation operand and different weight operands. In this way, the first activation operand is reused in the second round. The first activation operand may be reused further in additional rounds, for example, by an additional multiplier 1230.

[0150] The internal adder assembly 1240 includes one or more adders, i.e., internal adders, within the PE 1200. The internal adder assembly 1240 can perform cumulative operations on two or more product operands from the multiplier 1230 to produce an output operand of the PE 1200. In some embodiments, the internal adders are arranged in a series of hierarchies. Each hierarchy includes one or more internal adders. For the first hierarchy of the internal adder assembly 1240, the internal adders can receive product operands from two or more multipliers 1230 and produce a sum operand through a series of cumulative operations. Each cumulative operation produces the sum of two or more products from different multipliers 1230. The sum operand contains a sequence of sums, each of which is the result of a cumulative operation and corresponds to a channel at each depth. For other hierarchies (one or more) of the internal adder assembly 1240, the internal adders in one hierarchy receive sum operands from preceding hierarchies in the sequence. Each of these numbers may be generated by different internal adders in preceding hierarchies. The ratio of the number of internal adders in one hierarchy to the number of internal adders in a subsequent hierarchy may be 2:1. In some embodiments, the final hierarchy of the internal adder assembly 1240 may include a single internal adder that generates the output operand of PE 1200.

[0151] The output register file 1250 stores the output operands of PE 1200. In some embodiments, the output register file 1250 may store one output operand at a time. In other embodiments, the output register file 1250 may store multiple output operands or a portion of output operands at a time. An output operand includes multiple output elements in the IFM. The output elements of an output operand may be stored sequentially in the output register file 1250 so that the output elements can be processed sequentially. In some embodiments, each output element in an output operand corresponds to a channel at a different depth and is an element of a different output channel of the output channels of the convolution at each depth. The number of output elements in an output operand may be equal to the number of channels at each depth of the convolution at each depth.

[0152] Exemplary methods for reducing DNN power consumption Figure 13 is a flowchart of method 1300 for reducing the power consumption of a DNN based on sparsity, according to various embodiments. Method 1300 can be performed by the sparsity module 450 in Figure 4. Although method 1300 is described with reference to the flowchart shown in Figure 13, many other methods for reducing the power consumption of a DNN based on sparsity may be used as alternatives. For example, the order in which the steps in Figure 13 are performed may be changed. As another example, some of the steps may be modified, deleted, or combined.

[0153] The sparsity module 450 stores the first element of the activation operand of the deep learning operation and the first element of the weight operand of the deep learning operation in one or more first memory units associated with the multiplier (1310). The multiplier calculates the product by multiplying the first element of the activation operation by the first element of the weight operand.

[0154] The sparsity module 450 stores the product in a second memory unit (1320). In some embodiments, the second memory unit is associated with a multiplier to store the output of the multiplier. The second memory unit may store one output of the multiplier at a time. In other embodiments, the second memory unit is associated with an accumulator. The accumulator is configured to accumulate the product calculated by the multiplier or one or more other multipliers. The second memory unit may store inputs to the accumulator. The second memory unit may store one input to the accumulator at a time.

[0155] The sparsity module 450 determines whether the second element of the activation operand or the second element of the weight operand is zero (1330). In some embodiments, the sparsity module 450 performs a logical operation on the second element of the activation operand or the second element of the weight operand to determine whether the second element of the activation operand or the second element of the weight operand is zero. In other embodiments, the sparsity module 450 determines whether the second element of the activation operand or the second element of the weight operand is zero by determining whether the value of the second element of the activation operand or the second element of the weight operand is less than or equal to a threshold.

[0156] In other embodiments, the sparsity module 450 determines whether a second element of an activation operand or a second element of a weight operand is zero based on an activation bitmap or a weight bitmap. The activation bitmap includes a sequence of bits, each corresponding to a respective element of the activation operand, indicating whether each element of the activation operand is zero. The weight bitmap includes a sequence of bits, each corresponding to a respective element of the weight operand, indicating whether each element of the weight operand is zero.

[0157] In some embodiments, the sparsity module 450 performs logical operations on the bits in the activation bitmap and the bits in the weight bitmap. The bits in the activation bitmap correspond to the second element of the activation operand. The bits in the weight bitmap correspond to the second element of the weight operand.

[0158] In some embodiments, the sparsity module 450 calculates a combined bitmap based on the activation bitmap and the weight bitmap. The combined bitmap contains a sequence of bits, each bit being the product of a bit in the activation bitmap and a bit in the weight bitmap. Based on the combined bitmap, the sparsity module 450 determines whether the second element of the activation operand or the second element of the weight operand is zero.

[0159] After determining that the second element of the activation operand or the second element of the weight operand is zero, the sparsity module 450 holds the first element of the activation operand and the first element of the weight operand in one or more first memory units (1340).

[0160] The sparsity module 450 writes zero-value data elements to the second storage unit (1350). In some embodiments, the sparsity module 450 writes zero-value data elements to the second storage unit after the pipeline in the multiplier is complete. In some embodiments, the sparsity module 450 sends the product and zero-value data elements to the accumulator. In other embodiments, the sparsity module 450 sends the product to the accumulator. After determining that the second element of the activation operand or the second element of the weight operand is zero-value, the sparsity module 450 deactivates the accumulator.

[0161] In some embodiments, the first element of the activation operand is placed before the second element of the activation operand. The multiplier calculates the product in the first clock cycle. Zero-value data elements are written to the second storage unit in the second clock cycle following the first clock cycle. The first element of the activation operand and the first element of the weight operand are stored in one or more first storage units in the first and second clock cycles.

[0162] Exemplary computing device Figure 14 is a block diagram of exemplary computing device 1400 according to various embodiments. In some embodiments, computing device 1400 may be used as at least part of the DNN accelerator 300 of Figure 3. Figure 14 shows several components included in computing device 1400, but any one or more of these components may be omitted or duplicated as appropriate depending on the application. In some embodiments, some or all of the components included in computing device 1400 may be mounted on one or more motherboards. In some embodiments, some or all of these components are manufactured on a single system-on-chip (SoC) die. Furthermore, in various embodiments, computing device 1400 may not include one or more of the components shown in Figure 14, but computing device 1400 may include interface circuits for coupling to such one or more components. For example, computing device 1400 may not include a display device 1406, but may include a display device interface circuit (e.g., connector and driver circuit) to which the display device 1406 can be coupled. In another set of examples, the computing device 1400 may not include an audio input device 1418 or an audio output device 1408, but may include an audio input or output device interface circuit (e.g., a connector and support circuit) to which the audio input device 1418 or the audio output device 1408 can be coupled.

[0163] The computing device 1400 may include a processing device 1402 (for example, one or more processing devices). The processing device 1402 processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. The computing device 1400 may also include a memory 1404, which itself may include one or more memory devices such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard drive. In some embodiments, the memory 1404 may include memory that shares a die with the processing device 1402. In some embodiments, the memory 1404 includes one or more non-temporary computer-readable media that store executable instructions for performing operations. The operations are, for example, several operations performed by the method 1300 described above in relation to Figure 13, or by the calculation block 400 (e.g., the sparsity module 450 within the calculation block 400) or PE (e.g., PE 500, PE 600, PE 700, PE 800, PE 900, PE 1000, PE 1110, etc.) described above in relation to Figure 4. Instructions stored in one or more non-temporary computer-readable media may be executed by the processing device 1402.

[0164] In some embodiments, the computing device 1400 may include a communication chip 1412 (for example, one or more communication chips). For example, the communication chip 1412 may be configured to manage wireless communication for transferring data to and from the computing device 1400. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can communicate data using modulated electromagnetic radiation through a non-solid medium. The term does not imply that the devices in question do not contain wires, although in some embodiments they may not contain wires.

[0165] The 1412 communication chip may implement any of several wireless standards or protocols, including, but not limited to, Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), and the Long-Term Evolution (LTE) project (e.g., the Advanced LTE project, the Ultra-Mobile Broadband (UMB) project (also known as "3GPP2")), including any amendments, updates, and / or revisions. IEEE 802.16-compatible broadband radio access (BWA) networks are commonly referred to as WiMAX networks, an acronym representing global interoperability for microwave access, and a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1412 may operate in accordance with Global System for Mobile Communications (GSM®), General-Purpose Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. The communication chip 1412 may operate in accordance with GSM® Evolution Enhanced Data (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1412 may operate in accordance with Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution Data Optimization (EV-DO), and their derivatives, as well as any other radio protocols designated as 3G, 4G, 5G, and later. In other embodiments, the communication chip 1412 may operate in accordance with other radio protocols. The computing device 1400 may include an antenna 1422 for facilitating wireless communication and / or for receiving other wireless communications (such as AM or FM radio transmissions).

[0166] In some embodiments, the communication chip 1412 may manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet®). As described above, the communication chip 1412 may comprise multiple communication chips. For example, the first communication chip 1412 may be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth®, and the second communication chip 1412 may be dedicated to long-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1412 may be dedicated to wireless communications, and the second communication chip 1412 may be dedicated to wired communications.

[0167] The computing device 1400 may include a battery / power supply circuit 1414. The battery / power supply circuit 1414 may include one or more energy storage devices (e.g., a battery or a capacitor) and / or circuits for coupling components of the computing device 1400 to an energy source separate from the computing device 1400 (e.g., AC line power).

[0168] The computing device 1400 may include a display device 1406 (or a corresponding interface circuit as described above). The display device 1406 may include any visual indicator, such as a head-up display, computer monitor, projector, touchscreen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0169] The computing device 1400 may include an audio output device 1408 (or a corresponding interface circuit as described above). The audio output device 1408 may include any device that generates an audible indicator, such as a speaker, headset, or earphones.

[0170] The computing device 1400 may include an audio input device 1418 (or a corresponding interface circuit as described above). The audio input device 1418 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital instrument (for example, an instrument with a Musical Instrument Digital Interface (MIDI) output).

[0171] The computing device 1400 may include a GPS device 1416 (or a corresponding interface circuit as described above). The GPS device 1416 may communicate with a satellite-based system and receive the position of the computing device 1400, as is known in the art.

[0172] The computing device 1400 may include another output device 1410 (or a corresponding interface circuit as described above). Examples of other output devices 1410 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or additional storage devices.

[0173] The computing device 1400 may include another input device 1420 (or a corresponding interface circuit as described above). Examples of other input devices 1420 may include an accelerometer, gyroscope, compass, image acquisition device, cursor control device such as a keyboard or mouse, stylus, touchpad, barcode reader, quick response (QR) code reader, any sensor, or radio frequency identification (RFID) reader.

[0174] The computing device 1400 may have any desired form factor, such as a handheld or mobile computer system (e.g., a mobile phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, PDA (personal digital assistant), ultramobile personal computer, etc.), a desktop computer system, a server or other network-connected computing component, a printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 1400 may be any other electronic device that processes data.

[0175] Selected Examples The following paragraphs provide various examples of embodiments disclosed herein.

[0176] Embodiment 1 provides a method for deep learning, the method comprising the steps of: storing a first element of an activation operand of a deep learning operation and a first element of a weight operand of the deep learning operation in one or more first memory units associated with a multiplier, wherein the multiplier calculates a product by multiplying the first element of the activation operation by the first element of the weight operand; storing the product in a second memory unit; determining whether the second element of the activation operand or the second element of the weight operand is zero; after determining that the second element of the activation operand or the second element of the weight operand is zero, holding the first element of the activation operand and the first element of the weight operand in the one or more first memory units; and writing the zero-value data element to the second memory unit.

[0177] Example 2 provides a method of Example 1 in which determining whether the second element of the activating operand or the second element of the weight operand is zero includes performing a logical operation on the second element of the activating operand or the second element of the weight operand.

[0178] Example 3 provides a method of Example 1 or 2, wherein determining whether a second element of the activation operand or a second element of the weight operand is zero includes determining whether a second element of the activation operand or a second element of the weight operand is zero based on an activation bitmap or a weight bitmap, wherein the activation bitmap includes a sequence of bits, each corresponding to a respective element of the activation operand, indicating whether each of those elements of the activation operand is zero, and the weight bitmap includes a sequence of bits, each corresponding to a respective element of the weight operand, indicating whether each of those elements of the weight operand is zero.

[0179] Embodiment 4 provides a method of Embodiment 3 in which determining whether the second element of the activation operand or the second element of the weight operand is zero includes performing a logical operation on the bits in the activation bitmap and the bits in the weight bitmap, wherein the bits in the activation bitmap correspond to the second element of the activation operand and the bits in the weight bitmap correspond to the second element of the weight operand.

[0180] Example 5 provides a method of Example 3 or 4, wherein determining whether the second element of the activation operand or the second element of the weight operand is zero is a method comprising: calculating a combined bitmap based on the activation bitmap and the weight bitmap, wherein the combined bitmap includes a sequence of bits, each of which is the product of a bit in the activation bitmap and a bit in the weight bitmap; and determining whether the second element of the activation operand or the second element of the weight operand is zero based on the combined bitmap.

[0181] Example 6 provides a method of any of the prior embodiments in which determining whether the second element of the activation operand or the second element of the weight operand is zero includes determining whether the value of the second element of the activation operand or the second element of the weight operand is less than or equal to a threshold.

[0182] Embodiment 7 provides a method of any of the above embodiments, wherein writing the zero-value data element to the second storage unit includes writing the zero-value data element to the second storage unit after the pipeline in the multiplier is completed.

[0183] Example 8 provides a method of any of the above embodiments, wherein, after determining that the second element of the activation operand or the second element of the weight operand is zero, the first element of the activation operand and the first element of the weight operand are held in the one or more first memory units, thereby reducing gate switching for the deep learning operation in response to the determination that the second element of the activation operand or the second element of the weight operand is zero.

[0184] Example 9 provides a method of any of the above embodiments, further comprising: sending the product to an accumulator; and after determining that the second element of the activation operand or the second element of the weight operand is zero, deactivating the accumulator.

[0185] Embodiment 10 provides a method of any of the above embodiments, wherein the first element of the activation operand is placed before the second element of the activation operand, the multiplier calculates the product in a first clock cycle, the zero-value data element is written to the second storage unit in a second clock cycle following the first clock cycle, and the first element of the activation operand and the first element of the weight operand are stored in the one or more first storage units in the first and second clock cycles.

[0186] Embodiment 11 provides one or more non-temporary computer-readable media storing executable instructions for performing operations for in-network computing, the operations comprising: storing a first element of an activation operand of a deep learning operation and a first element of a weight operand of the deep learning operation in one or more first storage units associated with a multiplier, wherein the multiplier calculates a product by multiplying the first element of the activation operand by the first element of the weight operand; storing the product in a second storage unit; determining whether the second element of the activation operand or the second element of the weight operand is zero; after determining that the second element of the activation operand or the second element of the weight operand is zero, holding the first element of the activation operand and the first element of the weight operand in the one or more first storage units; and writing the zero-value data element to the second storage unit.

[0187] Example 12 provides one or more non-temporary computer-readable media of Example 11, wherein determining whether the second element of the activating operand or the second element of the weight operand is zero involves performing a logical operation on the second element of the activating operand or the second element of the weight operand.

[0188] Example 13 provides one or more non-temporary computer-readable media of Example 11 or 12, wherein determining whether a second element of the activation operand or a second element of the weight operand is zero involves determining whether a second element of the activation operand or a second element of the weight operand is zero based on an activation bitmap or a weight bitmap, wherein the activation bitmap includes a sequence of bits, each corresponding to a respective element of the activation operand, indicating whether each of those elements of the activation operand is zero, and the weight bitmap includes a sequence of bits, each corresponding to a respective element of the weight operand, indicating whether each of those elements of the weight operand is zero.

[0189] Example 14 provides one or more non-temporary computer-readable media of any one of Examples 11 to 13, wherein determining whether the second element of the activating operand or the second element of the weight operand is zero includes determining whether the value of the second element of the activating operand or the second element of the weight operand is less than or equal to a threshold.

[0190] Example 15 provides one or more non-temporary computer-readable media of any one of Examples 11 to 14, wherein writing the zero-value data element to the second storage unit includes writing the zero-value data element to the second storage unit after the pipeline in the multiplier is completed.

[0191] Example 16 provides one or more non-temporary computer-readable media of any one of Examples 11 to 15, wherein the operation further includes: transmitting the product to an accumulator; and deactivating the accumulator after determining that the second element of the activation operand or the second element of the weight operand is zero.

[0192] Embodiment 17 provides a device comprising: a computer processor for executing computer program instructions; and non-temporary computer-readable memory storing computer program instructions executable by the computer processor to perform an operation, the operation comprising: storing a first element of an activation operand of a deep learning operation and a first element of a weight operand of the deep learning operation in one or more first storage units associated with a multiplier, wherein the multiplier calculates a product by multiplying the first element of the activation operation by the first element of the weight operand; storing the product in a second storage unit; determining whether the second element of the activation operand or the second element of the weight operand is zero; after determining that the second element of the activation operand or the second element of the weight operand is zero, holding the first element of the activation operand and the first element of the weight operand in the one or more first storage units; and writing the zero-value data element to the second storage unit.

[0193] Example 18 provides an apparatus of Example 17, wherein determining whether the second element of the activating operand or the second element of the weight operand is zero includes performing a logical operation on the second element of the activating operand or the second element of the weight operand.

[0194] Example 19 provides an apparatus of Example 17 or 18, wherein determining whether a second element of the activation operand or a second element of the weight operand is zero includes determining whether a second element of the activation operand or a second element of the weight operand is zero based on an activation bitmap or a weight bitmap, wherein the activation bitmap includes a sequence of bits, each corresponding to a respective element of the activation operand, indicating whether each of those elements of the activation operand is zero, and the weight bitmap includes a sequence of bits, each corresponding to a respective element of the weight operand, indicating whether each of those elements of the weight operand is zero.

[0195] Example 20 provides an apparatus of any one of Examples 17 to 19, wherein the operation further includes: sending the product to an accumulator; and deactivating the accumulator after determining that the second element of the activation operand or the second element of the weight operand is zero.

[0196] The above description of the illustrative implementations of the Disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the Disclosure to the exact form disclosed. While specific implementations of the Disclosure and examples of the Disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the Disclosure, as those skilled in the art will recognize. These modifications may be made to the Disclosure in light of the above detailed description.

Claims

1. A method for deep learning: A step of storing a first element of the activation operand of a deep learning operation and a first element of the weight operand of the deep learning operation in one or more first memory units associated with a multiplier, wherein the multiplier calculates the product by multiplying the first element of the activation operation by the first element of the weight operand; The steps include storing the aforementioned product in a second memory unit; A step of determining whether the second element of the activation operand or the second element of the weight operand is zero; The steps include: determining that the second element of the activation operand or the second element of the weight operand is zero, and then holding the first element of the activation operand and the first element of the weight operand in one or more first storage units; The step of writing a data element with a zero value to the second storage unit and Methods that include...

2. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: This includes performing a logical operation on the second element of the activation operand or the second element of the weight operand. The method according to claim 1.

3. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: The process includes determining whether a second element of the activation operand or a second element of the weight operand is zero, based on an activation bitmap or a weight bitmap. The activation bitmap includes a sequence of bits, each corresponding to each element of the activation operand, indicating whether each element of the activation operand is zero. The weight bitmap includes a sequence of bits, each corresponding to each element of the weight operand, indicating whether each element of the weight operand is zero. The method according to claim 1 or 2.

4. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: The method according to claim 3, comprising performing a logical operation on the bits in the activation bitmap and the bits in the weight bitmap, wherein the bits in the activation bitmap correspond to the second element of the activation operand and the bits in the weight bitmap correspond to the second element of the weight operand.

5. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: A step of calculating a combined bitmap based on the activation bitmap and the weight bitmap, wherein the combined bitmap includes a sequence of bits, each of which is the product of a bit in the activation bitmap and a bit in the weight bitmap; The step of determining whether the second element of the activation operand or the second element of the weight operand is zero, based on the combined bitmap, The method according to claim 3.

6. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: This includes determining whether the value of the second element of the activation operand or the second element of the weight operand is less than or equal to a threshold, The method according to claim 1.

7. Writing the aforementioned zero-value data element to the second storage unit is: The process includes writing the zero-value data element to the second storage unit after the pipeline in the multiplier is completed. The method according to claim 1.

8. After determining that the second element of the activation operand or the second element of the weight operand is zero, the first element of the activation operand and the first element of the weight operand are held in one or more first storage units: The method according to claim 1, comprising reducing gate switching for the deep learning operation in response to determining that the second element of the activation operand or the second element of the weight operand is zero.

9. The product is transmitted to the accumulator; After determining that the second element of the activation operand or the second element of the weight operand is zero, the accumulator is disabled. The method according to claim 1, further comprising:

10. The first element of the activation operand is placed before the second element of the activation operand, The multiplier calculates the product in the first clock cycle. The zero-value data element is written to the second storage unit in the second clock cycle following the first clock cycle. The first element of the activation operand and the first element of the weight operand are stored in one or more first storage units during the first clock cycle and the second clock cycle. The method according to claim 1.

11. One or more non-temporary computer-readable media storing executable instructions for performing an operation for network computing, wherein the operation is: A step of storing a first element of the activation operand of a deep learning operation and a first element of the weight operand of the deep learning operation in one or more first memory units associated with a multiplier, wherein the multiplier calculates the product by multiplying the first element of the activation operand by the first element of the weight operand; The steps include storing the aforementioned product in a second memory unit; A step of determining whether the second element of the activation operand or the second element of the weight operand is zero; The steps include: determining that the second element of the activation operand or the second element of the weight operand is zero, and then holding the first element of the activation operand and the first element of the weight operand in one or more first storage units; The step of writing a data element with a zero value to the second storage unit and One or more non-temporary computer-readable media, including [the specified text].

12. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: This includes performing a logical operation on the second element of the activation operand or the second element of the weight operand. One or more non-temporary computer-readable media according to claim 11.

13. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: The process includes determining whether a second element of the activation operand or a second element of the weight operand is zero, based on an activation bitmap or a weight bitmap. The activation bitmap includes a sequence of bits, each corresponding to each element of the activation operand, indicating whether each element of the activation operand is zero. The weight bitmap includes a sequence of bits, each corresponding to each element of the weight operand, indicating whether each element of the weight operand is zero. One or more non-temporary computer-readable media according to claim 11 or 12.

14. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: One or more non-temporary computer-readable media according to claim 13, comprising performing logical operations on bits in the activation bitmap and bits in the weight bitmap, wherein the bits in the activation bitmap correspond to the second element of the activation operand and the bits in the weight bitmap correspond to the second element of the weight operand.

15. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: A step of calculating a combined bitmap based on the activation bitmap and the weight bitmap, wherein the combined bitmap includes a sequence of bits, each of which is the product of a bit in the activation bitmap and a bit in the weight bitmap; The step of determining whether the second element of the activation operand or the second element of the weight operand is zero, based on the combined bitmap, One or more non-temporary computer-readable media according to claim 13.

16. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: One or more non-temporary computer-readable media according to claim 11, comprising determining whether the value of the second element of the activation operand or the second element of the weight operand is less than or equal to a threshold.

17. Writing the aforementioned zero-value data element to the second storage unit is: The process includes writing the zero-value data element to the second storage unit after the pipeline in the multiplier is completed. One or more non-temporary computer-readable media according to claim 11.

18. After determining that the second element of the activation operand or the second element of the weight operand is zero, the first element of the activation operand and the first element of the weight operand are held in one or more first storage units: One or more non-transient computer-readable media according to claim 11, comprising reducing gate switching for the deep learning operation in response to determining that the second element of the activation operand or the second element of the weight operand is zero.

19. The aforementioned operation further: The product is transmitted to the accumulator; After determining that the second element of the activation operand or the second element of the weight operand is zero, the accumulator is disabled. One or more non-temporary computer-readable media according to claim 11, including the following:

20. The first element of the activation operand is placed before the second element of the activation operand, The multiplier calculates the product in the first clock cycle. The zero-value data element is written to the second storage unit in the second clock cycle following the first clock cycle. The first element of the activation operand and the first element of the weight operand are stored in one or more first storage units during the first clock cycle and the second clock cycle. One or more non-temporary computer-readable media according to claim 11.

21. A computer processor for executing computer program instructions; Non-temporary computer-readable memory storing computer program instructions that can be executed by the computer processor to perform an action, and A device having the following operation: A step of storing a first element of the activation operand of a deep learning operation and a first element of the weight operand of the deep learning operation in one or more first memory units associated with a multiplier, wherein the multiplier calculates the product by multiplying the first element of the activation operation by the first element of the weight operand; The steps include storing the aforementioned product in a second memory unit; A step of determining whether the second element of the activation operand or the second element of the weight operand is zero; The steps include: determining that the second element of the activation operand or the second element of the weight operand is zero, and then holding the first element of the activation operand and the first element of the weight operand in one or more first storage units; The step of writing a data element with a zero value to the second storage unit and A device including a device.

22. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: This includes performing a logical operation on the second element of the activation operand or the second element of the weight operand. The apparatus according to claim 21.

23. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: The process includes determining whether a second element of the activation operand or a second element of the weight operand is zero, based on an activation bitmap or a weight bitmap. The activation bitmap includes a sequence of bits, each corresponding to each element of the activation operand, indicating whether each element of the activation operand is zero. The weight bitmap includes a sequence of bits, each corresponding to each element of the weight operand, indicating whether each element of the weight operand is zero. The apparatus according to claim 21 or 22.

24. The aforementioned operation is: The product is transmitted to the accumulator; After determining that the second element of the activation operand or the second element of the weight operand is zero, the accumulator is disabled. The apparatus according to claim 21, further comprising:

25. Determining whether the second element of the activation operand or the second element of the weight operand is zero means: This includes determining whether the value of the second element of the activation operand or the second element of the weight operand is less than or equal to a threshold, The apparatus according to claim 21.

Images