Performing neural network operation based on spatial similarity in input data

Abstract

Layers in a deep neural network (DNN) may be executed based on spatial similarity in input data to improve computational efficiency. For example, a convolution may be performed based on spatial similarity between activations in the input feature map (IFM). A DNN accelerator may identify an anchor tensor in the IFM and measure similarity between the anchor tensor and one or more other tensors in the IFM. After determining that the similarity between the anchor tensor and another tensor is above a threshold, the DNN accelerator may store the result of a multiply-accumulate (MAC) operation on the anchor tensor as the result of a MAC operation on the other tensor and skip the MAC operation on the other tensor. The IFM may be an output of a previous layer in the DNN. In the previous layer, the DNN accelerator may store the anchor tensor and bypass storing the other tensor.

Description

PERFORMING NEURAL NETWORK OPERATION BASED ON SPATIAL SIMILARITY IN INPUT DATATechnical Field

[0001] This disclosure relates generally to neural networks (also referred to as "deep neural networks" or "DNN"), and more specifically, performing operations in DNNs based on spatial similarity in input data.Background

[0002] DNNs are used extensively for a variety of artificial intelligence applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write. Therefore, techniques to improve efficiency of DNNs are needed.Brief Description of the Drawings

[0003] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0004] FIG. 1 illustrates an example DNN, in accordance with various embodiments.

[0005] FIG. 2 illustrates an example convolution, in accordance with various embodiments.

[0006] FIG. 3 is a block diagram of a DNN system, in accordance with various embodiments.

[0007] FIG. 4 is a block diagram of a DNN module, in accordance with various embodiments.

[0008] FIG. 5 illustrates an example data processing cell, in accordance with various embodiments.

[0009] FIG. 6 illustrates an example data processing unit, in accordance with various embodiments.

[0010] FIG. 7 illustrates an example MAC (multiply-accumulate) unit, in accordance with various embodiments.

[0011] FIG. 8 is a block diagram of a drain module, in accordance with various embodiments.

[0012] FIG. 9 illustrates an example data compression module, in accordance with various embodiments.

[0013] FIG. 10 illustrates an example spatial comparison process, in accordance with various embodiments.

[0014] FIG. 11 illustrates another example spatial comparison process, in accordance with various embodiments.

[0015] FIG. 12A illustrates storage element pointers generated for convolutions executed without spatial similarity, in accordance with various embodiments.

[0016] FIG. 12B illustrates storage element pointers generated for a convolution executed based spatial similarity in input data, in accordance with various embodiments.

[0017] FIG. 13 is a flowchart showing a method of executing a convolution, in accordance with various embodiments.

[0018] FIG. 14 is a block diagram of an example computing device, in accordance with various embodiments.Detailed DescriptionOverview

[0019] The last decade has witnessed a rapid rise in artificial intelligence (Al) based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more deep learning operations (also referred to as "neural network operations"), such as matrix multiplication, convolution, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Input or output data of deep learning operations may be arranged in data structures called tensors. A tensor is a data structure having multiple elements across one or more dimensions. Examples of tensors include vector (which is one-dimensional (ID) tensor), matrix (which is two-dimensional (2D) tensor), three-dimensional (3D) tensors, four-dimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along theaxis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. For a convolution layer, the input tensors include an activation tensor (also referred to as "input feature map (IFM)") including one or more activations (also referred to as "input elements") and a weight tensor. The weight tensor may be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a group of filters (a 4D weight tensor).

[0020] The significant improvements in DNN model size and accuracy coupled with the rapid increase in computing power of execution platforms have led to the adoption of DNN applications even within resource constrained mobile and edge devices that have limited energy availability. Convolutional Neural Networks (CNNs) represent a crucial workload in deep learning, especially in Al applications like object detection, image classification, and image segmentation, and so on. These applications, pivotal in autonomous vehicles, can demand significant computational resources, substantial memory bandwidth, and ample storage for efficient and rapid inference on Al accelerators. The fundamental operation of a convolution is MAC operations between input activations and kernel weights. A CNN may include a large number of independent MAC operations, such as hundreds of millions of MAC operations. Many currently available Al accelerators use sparsity-based approaches to reduce the number of MAC operations and improve computational efficiency of convolution inference. However, these approaches are limited to sparse data and fail to improve computational efficiency non-sparse data.

[0021] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by exploiting spatial similarity in input data of deep learning operations and leveraging the spatial similarity to reduce computational workload and memory storage. Spatial similarity refers to similarity between neighboring segments of the input data. For instance, neighboring pixels in an image may share similar values. When there is sufficient similarity between adjacent data blocks in the input data, computation on one of the data blocks may be performed while computation(s) on the other data block(s) may be skipped. The output of the performed computation may be reused as the output of the unperformed computation(s). That way, the number of computations and memory storage required for executing the deep learning operation are reduced or even minimized for both sparse data and non-sparse data.

[0022] In various embodiments of the present disclosure, spatial correlation between adjacent data blocks in an input tensor of a DNN layer may be determined. The spatial correlation may be determined based on the values of the activations in the data blocks. For instance, a difference score between two data blocks may be determined by comparing norms (such as LI norms, L2 norms (also referred to as "Euclidean norms"), etc.) of the data blocks or performing a XOR operation on the data blocks. An XOR operation may be an exclusive OR logic operation that compares two inputs and generates an output. In embodiments where the difference score is beyond a predetermined threshold, the two data blocks may be treated as being spatially similar. Inference of DNN layers may be accelerated based on such spatial similarity in input data. For example, a convolution may be performed based on spatial similarity between activations in the IFM. A DNN accelerator (also referred to as "Al accelerator") may identify at least one anchor tensor in the IFM and measure similarity between the anchor tensor and one or more other tensors in the IFM. After determining that the similarity between the anchor tensor and another tensor is above a threshold, the DNN accelerator may reuse the result of a MAC operation on the anchor tensor as the result of a MAC operation on the other tensor and skip the MAC operation on the other tensor. The IFM may be an output of a previous layer in the DNN. In the data drain stage of the previous layer, the DNN accelerator may store the anchor tensor and bypass the storage of the other tensor. The DNN accelerator may generate storage element pointer for the other tensor that points to the anchor tensor. Even though execution of convolution based on spatial similarity is described in various sections of the present disclosure, other types of deep learning operation (e.g., matrix multiplications, elementwise multiplication, etc.) may also be executed based on spatial similarity in input data to improve computational efficiency, optimize memory bandwidth, and reduce memory storage requirements and latency.

[0027] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0028] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0029] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0030] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0031] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0032] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0033] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0034] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0035] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.Example DNN

[0036] FIG. 1 illustrates an example DNN 100, in accordance with various embodiments. The DNN 100 may be executed by a DNN accelerator, e.g., the DNN accelerator 302 in FIG. 3. In an example, the DNN 100 may be a convolution-based DNN. In other examples, the DNN 100 may be other types of DNNs. For the purpose of illustration, the DNN 100 includes a sequence of layers comprising a plurality of convolutional layers 110 (individually referred to as "convolutional layer 110"), a plurality of pooling layers 120 (individually referred to as "pooling layer 120"), and a plurality of fully-connected layers 130 (individually referred to as "fully-connected layer 130"). In other embodiments, the DNN 100 may include fewer, more, or different layers. In an execution of the DNN 100, the layers of the DNN 100 execute tensor computation that includes many tensor operations, such as matrix multiplications, convolutions (e.g., multiply-accumulate (MAC) operations, etc.), pooling operations, elementwise operations (e.g., elementwise addition, elementwise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0037] The convolutional layers 110 summarize the presence of features in inputs to the DNN 100. The convolutional layers 110 function as feature extractors. The first layer of the DNN 100 is a convolutional layer 110. In an example, a convolutional layer 110 performs aconvolution on an input tensor 140 (also referred to as IFM 140) and a filter 150. As shown in FIG. 1, the IFM 140 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 140 includes 3 input channels, each of which is represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix includes 7 input elements (also referred to as input points) in each row and 7 input elements in each column. The filter 150 is represented by a 3x3x3 3D matrix. The filter 150 includes 3 kernels, each of which may correspond to a different input channel of the IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM. In the embodiments of FIG. 1, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel includes 3 weights in each row and 3 weights in each column. Weights can be initialized and updated by backpropagation using gradient descent. The magnitudes of the weights can indicate importance of the filter 150 in extracting features from the IFM 140.

[0038] The convolution includes MAC operations with the input elements in the IFM 140 and the weights in the filter 150. The convolution may be a standard convolution 163 or a depthwise convolution 183. In the standard convolution 163, the whole filter 150 slides across the IFM 140. All the input channels are combined to produce an output tensor 160 (also referred to as OFM 160). The OFM 160 is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements (also referred to as output points) in each row and 5 output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG. 1. In embodiments where there are multiple filters, the standard convolution may produce multiple output channels in the OFM 160.

[0039] The multiplication applied between a kernel-sized patch of the IFM 140 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernelsized patch of the IFM 140 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referred to as the "scalar product." Using a kernel smaller than the IFM 140 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 140 multiple times at different points on the IFM 140. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 140, left to right, top to bottom. The result from multiplying the kernel with the IFM 140 one time is a single value. As the kernel is applied multiple times to the IFM 140, the multiplication result is a 2D matrix of output elements. Assuch, the 2D output matrix (i.e., the OFM 160) from the standard convolution 163 is referred to as an OFM.

[0040] In the depthwise convolution 183, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an output channel. As shown in FIG. 1, the depthwise convolution 183 produces a depthwise output tensor 180. The depthwise output tensor 180 is represented by a 5x5x3 3D matrix. The depthwise output tensor 180 includes 3 output channels, each of which is represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements in each row and 5 output elements in each column. Each output channel is a result of MAC operations of an input channel of the IFM 140 and a kernel of the filter 150. For instance, the first output channel (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips), and the third output channel (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 193 is then performed on the depthwise output tensor 180 and a 1x1x3 tensor 190 to produce the OFM 160.

[0041] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is rectified linear unit (ReLU). ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 110 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 160 is passed to the subsequent convolutional layer 110 (i.e., the convolutional layer 110 following the convolutional layer 110 generating the OFM 160 in the sequence). The subsequent convolutional layers 110 perform a convolution on the OFM 160 with new kernels and generate a new feature map. The new feature map may also be normalized andresized. The new feature map can be kernelled again by a further subsequent convolutional layer 110, and so on.

[0042] In some embodiments, a convolutional layer 110 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions FxFxD pixels), the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time), and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 110). The convolutional layers 110 may perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0043] The pooling layers 120 down-sample feature maps generated by the convolutional layers, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 120 is placed between two convolution layers 110: a preceding convolutional layer 110 (the convolution layer 110 preceding the pooling layer 120 in the sequence of layers) and a subsequent convolutional layer 110 (the convolution layer 110 subsequent to the pooling layer 120 in the sequence of layers). In some embodiments, a pooling layer 120 is added after a convolutional layer 110, e.g., after an activation function (e.g., ReLU, etc.) has been applied to the OFM 160.

[0044] A pooling layer 120 receives feature maps generated by the preceding convolution layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics. Accordingly, the pooling operation improves the efficiency of the DNN and avoids over- learning. The pooling layers 120 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map), max pooling (calculating the maximum value for each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 120 applied to a feature map of 6x6 results in an output pooledfeature map of 3x3. The output of the pooling layer 120 is inputted into the subsequent convolution layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

[0045] The fully-connected layers 130 are the last layers of the DNN. The fully-connected layers 130 may be convolutional or not. The fully-connected layers 130 receive an input operand. The input operand defines the output of the convolutional layers 110 and pooling layers 120 and includes the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully-connected layers 130 apply a linear combination and an activation function to the input operand and generate a vector. The vector may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 1, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 130 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function. In some embodiments, the fully-connected layers 130 multiply each input element by weight, make the sum, and then apply an activation function (e.g., logistic if N=2, SoftMax if N>2). This is equivalent to multiplying the input operand by the matrix containing the weights.Example Convolution

[0046] FIG. 2 illustrates an example convolution, in accordance with various embodiments. The convolution may be a deep learning operation in a convolutional layer of a DNN, e.g., a convolutional layer 110 in FIG. 1. The convolution can be executed on an activation tensor 210 and filters 220 (individually referred to as "filter 220"). The filters may constitute a weight tensor of the convolution. The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator. An example of the DNN accelerator may be the DNN accelerator 302 in FIG. 3. For instance, the convolution may be performed by the sparse cell array 370 in the DNN accelerator 302.

[0047] The activation tensor 210 may be computed in a previous layer of the DNN. In some embodiments (e.g., embodiments where the convolutional layer is the first layer of the DNN), the activation tensor 210 may be an image. In the embodiments of FIG. 2, the activation tensor 210 includes activations (also referred to as "input activations," "elements," or "input elements") arranged in a 3D matrix. The activation tensor 210 mayalso be referred to as an input tensor of the convolution. An input element is a data point in the activation tensor 210. The activation tensor 210 has a spatial size HinX WinX Cin, where Hinis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel), Winis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of activations in a row in the 2D matrix of each input channel), and Cinis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of input channels). For the purpose of simplicity and illustration, the activation tensor 210 has a spatial size of 7x7x3, i.e., the activation tensor 210 includes three input channels and each input channel has a 7x7 2D matrix. Each input element in the activation tensor 210 may be represented by a (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the activation tensor 210 may be different.

[0048] Each filter 220 includes weights arranged in a 3D matrix. The values of the weights may be determined through training the DNN. A filter 220 has a spatial sizeX 14^- X C , where H is the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in a column in each kernel), 14^- is the width of the filter (i.e., the length along the X axis, which indicates the number of weights in a row in each kernel), and C is the depth of the filter (i.e., the length along the Z axis, which indicates the number of channels). In some embodiments, equals Cin. For purpose of simplicity and illustration, each filter 220 in FIG. 2 has a spatial size of 2x3x3, i.e., the filter 220 includes 2 convolutional kernels with a spatial size of 2x3. In other embodiments, the height, width, or depth of the filter 220 may be different. The spatial size of the convolutional kernels is smaller than the spatial size of the 2D matrix of each input channel in the activation tensor 210.

[0049] An activation or weight may take one or more bytes in a memory. The number of bytes for an activation or weight may depend on the data format. For example, when the activation or weight has an INT8 format, the activation takes one byte. When the activation or weight has a FP16 format, the activation or weight takes two bytes. Other data formats may be used for activations or weights.

[0050] In the convolution, each filter 220 slides across the activation tensor 210 and generates a 2D matrix for an output channel in the output tensor 230. In the embodimentsof FIG. 2, the 2D matrix has a spatial size of 5x5. The output tensor 230 includes activations (also referred to as "output activations / ' "elements," or "output element") arranged in a 3D matrix. An output activation is a data point in the output tensor 230. The output tensor 230 has a spatial size HoutX WoutX Cout, where Houtis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in a column in the 2D matrix of each output channel), Woutis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 2D matrix of each output channel), and Coutis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels). Coutmay equal the number of filters 220 in the convolution. Houtand Woutmay depend on the heights and weights of the activation tensor 210 and each filter 220. In an example where the kernel size is l l, Houtand VK0Utmay equal to Hinand Win, respectively.

[0051] As a part of the convolution, MAC operations can be performed on a 2x3x3 subtensor 215 (which is highlighted with a dotted pattern in FIG. 2) in the activation tensor 210 and each filter 220. The result of the MAC operations on the subtensor 215 and one filter 220 is an output activation. In some embodiments (e.g., embodiments where the convolution is an integral convolution), an output activation may include 8 bits, e.g., one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), an output activation may include more than one byte. For instance, an output element may include two bytes.

[0052] After the MAC operations on the subtensor 215 and all the filters 220 are finished, a vector 235 is produced. The vector 235 is highlighted with slashes in FIG. 2. The vector 235 includes a sequence of output activations, which are arranged along the Z axis. The output activations in the vector 235 have the same (X, Y) coordinate, but the output activations correspond to different output channels and have different Z coordinates. The dimension of the vector 235 along the Z axis may equal the total number of output channels in the output tensor 230. After the vector 235 is produced, further MAC operations are performed to produce additional vectors till the output tensor 230 is produced.

[0053] In some embodiments, the MAC operations on a 2x3x3 subtensor (e.g., the subtensor 215) and a filter 220 may be performed by a plurality of MAC units. One or more MAC units may receive an input operand (e.g., an activation operand 217 shown in FIG. 2) and a weight operand (e.g., the weight operand 227 shown in FIG. 2). The activationoperand 217 includes a sequence of activations having the same (x, y) coordinate but different z coordinates. The activation operand 217 includes an activation from each of the input channels in the activation tensor 210. The weight operand 227 includes a sequence of weights having the same (x, y) coordinate but different z coordinates. The weight operand 227 includes a weight from each of the channels in the filter 220. Activations in the activation operand 217 and weights in the weight operand 227 may be sequentially fed into a MAC unit. The MAC unit may receive an activation and a weight ("an activation-weight pair") at a time and multiple the activation and the weight. The position of the activation in the activation operand 217 may match the position of the weight in the weight operand 227. The activation and weight may correspond to the same channel.

[0054] Activations or weights may be floating-point numbers. Floating-point numbers may have various data formats, such as FP32, FP16, BF16, and so on. A floating-point number may be a positive or negative number with a decimal point. A floating-point number may be represented by a sequence of bits that includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing an exponent of the floating-point number, and bits representing a mantissa of the floating-point number. The mantissa is the part of a floating-point number that represents the significant digits of that number. The mantissa is multiplied by the base raised to the exponent to give the actual value of the floating-point number.

[0055] In some embodiments, the output activations in the output tensor 230 may be further processed based on one or more activation functions before they are stored or inputted into the next layer of the DNN. The processing based on the one or more activation functions may be at least part of the post processing of the convolution. In some embodiments, the post processing may include one or more other computations, such as offset computation, bias computation, and so on. The results of the post processing may be stored in a local memory of the compute block and be used as input to the next DNN layer. In some embodiments, the input activations in the activation tensor 210 may be results of post processing of the previous DNN layer.Example DNN System

[0056] FIG. 3 is a block diagram of a DNN system 300, in accordance with various embodiments. The whole DNN system 300 or a part of the DNN system 300 may be implemented in one or more computing devices, such as the computing device 1400 in FIG.14. The DNN system 300 can generate and execute DNNs, such as Transformer-based models, convolution-based models, and so on. As shown in FIG. 3, the DNN system 300 includes a DNN module 301 and a DNN accelerator 302. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 300. For instance, the DNN system 300 may include multiple DNN modules or multiple DNN accelerators. Further, functionality attributed to a component of the DNN system 300 may be accomplished by a different component included in the DNN system 300 or a different system. In some embodiments, the DNN module 301 and DNN accelerator 302 may include different types of processing units. In an example, the DNN module 301 may be implemented by one or more central processing units (CPUs). The DNN accelerator 302 may also be referred to as an Al accelerator or an Al processor. The DNN module 301 and DNN accelerator 302 may be implemented in the same chip or separate chips.

[0057] The DNN module 301 facilitates generation and deployment of DNNs. In some embodiments, the DNN module 301 may generate and train DNNs. For instance, the DNN module 301 can define the layered architecture of a DNN. The DNN module 301 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 301 may also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the sparsity level of one or more deep learning tensors for the DNN.

[0058] The DNN module 301 may also compress DNNs, e.g., during or after training. In some embodiments, the DNN module 301 may prune weights in one or more layers of a DNN by changing nonzero valued weight to zeros. The DNN module 301 may prune weights based on a target weight sparsity ratio. A weight sparsity ratio may be the ratio of the number of zero-valued weights to the total number of weights. In an example where the DNN module 301 prunes weight during DNN training, the DNN module 301 may prune weight of a layer to achieve a target sparsity ratio after one or more epochs. The DNN module 301 may prevent the pruned weights from changing values during the rest of the training process. Alternatively, the DNN module 301 may allow the pruned weights to change values so that a pruned, zero-valued weight may have a nonzero value after further training. The DNN module 301 may prune weights of the layer again after one or more additional epochs.

[0059] The DNN module 301 may deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, the DNN module 301 may distributetrained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform tasks (e.g., image classification, motion planning, etc.) for which the DNNs were trained. In other embodiments, the DNN module 301 may facilitate deployment of the DNNs using the DNN accelerator 302. For instance, the DNN module 301 may receive data from a device or system coupled with the DNN system 300 and input the received data (or data generated by the DNN module 301, e.g., based on the received data) into a DNN. The DNN module 301 may generate instructions (e.g., configuration files) that control the operation of the DNN accelerator 302 during the DNN execution. The DNN module 301 may receive an output of the DNN from the DNN accelerator 302. The DNN module 301 may transmit the output of the DNN (or a result of processing the output of the DNN by the DNN module 301) to the device or system.

[0060] The DNN module 301 may control execution processes of trained, compressed, or validated DNNs. The DNN module 301 may function as a deep learning compiler for DNNs executed by the DNN accelerator 302. In some embodiments, the DNN module 301 facilitates exploitation of spatial similarity in input data of deep learning operations to improve computational efficiency of DNN inference. For instance, the DNN module 301 may generate configuration parameters for spatial similarity exploiting and provide the configuration parameters to the DNN accelerator 302. The DNN accelerator 302 may use the configuration parameters to accelerate executions of DNN layers based on spatial similarity in input data. The DNN module 302 may generate configuration parameters offline. In some embodiments, the DNN module 301 may determine configuration parameters for an execution of a DNN before the execution of the DNN is started. Certain aspects of the DNN module 301 are provided below in conjunction with FIG. 4.

[0061] The DNN accelerator 302 executes DNNs provided by the DNN module 301. For instance, the DNN accelerator 302 can execute a DNN by running deep learning operations in the DNN. The process of carrying out a deep learning operation is also referred to as a process of executing the deep learning operation or performing the deep learning operation. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. In some embodiments, the DNN accelerator 302 includes components designed for optimal efficiency in running convolution-based DNNs. As shown in FIG. 3, the DNN accelerator 302 includes a memory 310, a DMA (direct memory access) engine 320, and compute blocks 330 (individually referred to as "compute block 330"). In otherembodiments, alternative configurations, different or additional components may be included in the DNN accelerator 302. For example, the DNN accelerator 302 may include more than one memory 310 or DMA engine 320. As another example, the DNN accelerator 302 may include a single compute block 330. Further, functionality attributed to a component of the DNN accelerator 302 may be accomplished by a different component included in the DNN accelerator 302 or by a different system. A component of the DNN accelerator 302 may be implemented in hardware, software, firmware, or some combination thereof.

[0062] The memory 310 stores data associated with deep learning operations performed by the DNN accelerator 302. In some embodiments, the memory 310 may store data to be used by the compute blocks 330 for DNN execution. The memory 310 may store weights, such as weights of convolutional layers, which are determined by training DNNs. The memory 310 may further store inputs to DNN layers or outputs of DNN layers, such as data generated by the compute blocks 330 from performing deep learning operations in DNNs. Example deep learning operations include convolutions (also referred to as "convolutional operations"), matrix multiplication operations, pooling operations, elementwise operations, activation functions, other types of deep learning operations, or some combination thereof. The memory 310 may be a main memory of the DNN accelerator 302. In some embodiments, the memory 310 includes one or more dynamic random-access memories (DRAMs).

[0063] The DMA engine 320 facilitates data transfer between the memory 310 and local memories of the compute blocks 330. For example, the DMA engine 320 can read data from the memory 310 and write data into a local memory of a compute block 330. As another example, the DMA engine 320 can read data from a local memory of a compute block 330and write data into the memory 310. The DMA engine 320 provides a DMA feature that allows the compute block 330 to initiate data transfer between the memory 310 and the local memories of the compute blocks 330 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA engine 320 may read tensors from the memory 310, modify the tensors in a way that is optimized for the compute block 330 before it writes the tensors into the local memories of the compute blocks 330.

[0064] The compute blocks 330 can perform deep learning operations in DNNs. For instance, a compute block 330 may execute a DNN layer by running one or more deeplearning operations in the DNN layer. A compute block 330 may execute a layer, or a portion of a layer, at a time. The compute blocks 330 may be capable of running various types of deep learning operations, such as convolution, pooling, elementwise operation, linear operation, nonlinear operation, and so on. In an example, a compute block 330 may perform convolutions, e.g., standard convolution or depthwise convolution. In some embodiments, the compute block 330 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the compute block 330 or another compute block 330. In some embodiments, the operations of the DNN layers may be run by multiple compute blocks 330 in parallel. For instance, multiple compute blocks 330 may each perform a portion of a workload for a convolution. Data may be shared between the compute blocks 330. A compute block 330 may also be referred to as a compute tile. In some embodiments, each compute block 330 may be a processing unit.

[0065] A compute block 330 may also referred to as a tile of the DNN accelerator 302. In the embodiments of FIG. 3, each compute block 330 includes a local memory 340, a sparsity mode module 350, a load module 360, a sparse cell array 370 (also referred to as a data processing unit), and a drain module 380. Some or all the components of the compute block 330 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the compute block 330. Further, functionality attributed to a component of the compute block 330 may be accomplished by a different component included in the compute block 330, a different compute block 330, another component of the DNN accelerator 302, or a different system. A component of the compute block 330 may be implemented in hardware, software, firmware, or some combination thereof.

[0066] The local memory 340 is local to the corresponding compute block 330. In the embodiments of FIG. 3, the local memory 340 is inside the compute block 330. In other embodiments, the local memory 340 may be outside the compute block 330. Data in the local memory 340 may be transferred to or from the memory 310, e.g., through the DMA engine 320. In some embodiments, data in the local memory 340 may be transferred to or from the local memory of another compute block 330. The local memory 340 may store data received, used, or generated by the sparsity mode module 350, the load module 360,Y1the sparse cell array 370, or the drain module 380. Examples of the data may include input activations, weights, output activations, sparsity bitmaps, and so on.

[0067] In some embodiments, the local memory 340 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which zero-valued elements (if any) are not removed. A dense tensor may be converted to a sparse tensor by removing one or more zero-valued elements in the dense tensor. A sparse tensor may also be referred to as a compressed tensor or packed tensor. The process of converting a dense tensor to a sparse tensor may be referred to as sparsity encoding. Sparsity encoding may also generate a sparsity tensor. Each element in the sparsity tensor may correspond to a different element in the dense tensor and indicate whether the element in the dense tensor is zero or not. The sparsity tensor may indicate positions of elements of the sparse tensor in the dense tensor. The sparsity tensor may be a sparsity bitmap, each element of which is a bit. A sparse tensor may be converted to a dense tensor through a densifying process, in which one or more zeros may be added to the sparse tensor based on the sparsity tensor.

[0068] In some embodiments, the local memory 340 includes one or more static randomaccess memories (SRAMs). The local memory 340 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 340 may include memory banks. The number of data banks in the local memory 340 may be 16, 64, 128, 356, 512, 1024, 3048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 340 in multiple read cycles, such as two cycles.

[0069] The sparsity mode module 350 determines sparsity modes in which the compute block 330 operates to execute DNN layers. For instance, the sparsity mode module 350 maydetermine whether to accelerate a layer based on weight sparsity, activation sparsity, or both. The sparsity mode module 350 select the sparsity mode for a layer from a group of sparsity modes that includes, for example, combined sparsity mode in which the layer is accelerated based on both weight sparsity and activation sparsity, activation sparsity mode in which the layer is accelerated based on activation sparsity but not based on weight sparsity, weight sparsity mode in which the layer is accelerated based on weight sparsity but not based on activation sparsity, and a dense mode in which the layer is not accelerated based on sparsity. In some embodiments (e.g., embodiments where a layer is executed by multiple compute blocks 330), the sparsity mode module 350 may determine the sparsity mode for all the compute blocks 330 that executes the layer. In some embodiments, the sparsity mode module 350 may receive configuration parameters from the DNN module 301. A configuration parameter may correspond to a layer and indicate whether to accelerate the layer based on weight sparsity. The sparsity mode module 350 may determine the sparsity mode of the layer based on the configuration parameter.

[0070] The load module 360 loads data from the local memory 340 to the sparse cell array 370. The load module 360 may read tensors from the local memory 340. The tensors may include sparse activation tensors, sparse weight tensors, activation sparsity tensors, weight sparsity tensors, and so on. In some embodiments, the load module 360 may load data based on the sparsity mode determined by the sparsity mode module 350. The load module 360 may select different data to transmit to the sparse cell array 370 in different sparsity modes. For instance, the load module 360 may transmit an activation sparsity tensor and a weight sparsity tensor of a layer to the sparse cell array 370 in the combined sparsity mode, while transmit the activation sparsity tensor but not the weight sparsity tensor to the sparse cell array 370 in the activation sparsity mode and transmit the weight sparsity tensor but not the activation sparsity tensor to the sparse cell array 370 in the weight sparsity mode. In the dense mode, the load module 360 does not transmit either the activation sparsity tensor or the weight sparsity tensor to the sparse cell array 370.

[0071] In some embodiments, the load module 360 may process (e.g., densify) data stored in the local memory 340 before providing the data to the sparse cell array 370. In an example, the load module 360, while operating in the weight sparsity mode, may densify sparse activation tensors to generate dense activation tensors based on corresponding activation sparsity tensors. For instance, the load module 360 may add one or more zerosinto a sparse activation tensor based on an activation sparsity tensor associated with the sparse activation tensor to generate the dense activation tensor. The dense activation tensor includes one or more elements than the sparse activation tensor. The additional element(s) are zero-valued. The load module 360 may identify one or more elements in the activation sparsity tensor that correspond to the zero-valued element(s), determine the position of each of the zero-valued element(s) in the dense activation tensor, and insert the zero-valued element(s) into the sparse activation tensor based on the determined positions. After the densification, the load module 360 may transmit the dense activation tensors to the sparse cell array 370. The load module 360 may also transmit corresponding sparse weight tensors and weight sparsity tensors to the sparse cell array 370. Activation sparsity tensor of the dense activation tensors may not be loaded to the sparse cell array 370.

[0072] In another example, the load module 360, while operating in the activation sparsity mode, may densify sparse weight tensors to generate dense weight tensors based on corresponding weight sparsity tensors by inserting zeros into sparse weight tensors. The densification of sparse weight tensors may be similar to the densification of sparse activation tensors described above. After the densification, the load module 360 may transmit the dense weight tensors to the sparse cell array 370. The load module 360 may also transmit corresponding sparse activation tensors and activation sparsity tensors to the sparse cell array 370. Weight sparsity tensor of the dense weight tensors may not be loaded to the sparse cell array 370.

[0073] In yet another example, the load module 360, while operating in the dense mode, may densify both sparse weight tensors and sparse activation tensors. The load module 360 may generate the input tensor and weight tensor of the layer and transmit the tensors to the sparse cell array 370 for executing the layer without sparsity acceleration.

[0074] The sparse cell array 370 may include one or more data processing cells. Each data processing cell may include one or more MAC units that can perform MAC operations. The MAC units in a data processing cell may be arranged in an array that includes rows and columns. The data processing cells may be arranged in one or more rows and one or more columns in the sparse cell array 370. All the MAC units in the sparse cell array 370 may constitute a bigger array that includes more rows and columns. In some embodiments (e.g., embodiments where the compute block 330 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand.The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN. The weights in the weight operand may be in different input channels.

[0075] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the load module 360, into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0076] In some embodiments, the sparse cell array 370 may be capable of depthwise convolution, standard convolution, or both. In a depthwise convolution, an MAC unit may perform an MAC operation that includes a sequence of multiplications for an input operand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activation and weight in the same cycle may correspond to the same channel. The sequence of multiplication produces a product operand that includes a sequence of products. The MAC operation may also include accumulations in which multiple product operands are accumulated to produce an output operand of the MAC unit. The sparse cell array 370 may output multiple output operands at a time, each of which is generated by a different MAC unit. In a standard convolution, MAC operations may include accumulations across the channels. For instance, as opposed to generating an output operand, a MAC unit may accumulate products across different channels to generate a single output point.

[0077] In some embodiments, the sparse cell array 370 may perform MAC operations in quantized deep learning operations, such as MAC operations in a quantized convolution. In some embodiments, an MAC unit in the sparse cell array 370 may receive quantized activation and quantized weights and compute a quantized MAC result. The quantized MAC result may be a quantized value in an integer format and may be the output of the MAC unit. In some embodiments, the MAC unit may also include a quantization multiplier that can multiply a quantization scale with the quantized MAC result, and the output of the MAC unit may be a real value in a floating-point format. The MAC unit may include no quantization subtractors as zero-point offsetting is not needed for the MAC operations in quantized deep learning operations.

[0078] In some embodiments, the sparse cell array 370 may include sparsity acceleration logic for facilitating sparsity acceleration. For instance, each data processing cell in the sparse cell array 370 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, a sparsity module accelerates computations in the sparse cell array 370 based on sparsity in activations, sparsity in weights, or both. The sparsity module may include a storage unit that stores a sparsity tensor, which may be loaded to the storage unit by the load module 360. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0079] An activation sparsity tensor may be the sparsity tensor of an activation tensor and has the same number of elements as the activation tensor. An element in the activation sparsity tensor may indicate whether the corresponding element in the activation tensor is zero or not. For instance, a zero-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is zero. A one-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is nonzero. A weight sparsity tensor may be the sparsity tensor of a weight tensor and has the same number of elements as the weight tensor. An element in the weight sparsity tensor may indicate whether the corresponding element in the weight tensor is zero or not. For instance, a zero-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is zero. A one-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is nonzero. The sparsitymodule may generate a combined sparsity tensor using an activation sparsity tensor and a weight sparsity tensor. For instance, the sparsity module may multiply an element of the activation sparsity tensor with a corresponding element of the weight sparsity tensor to compute an element of the combined sparsity tensor. The positions of the three elements in their corresponding sparsity tensors may match. In some embodiments, each element in a sparsity tensor may be a bit, and the sparsity tensor may be referred to as a sparsity bitmap.

[0080] The sparsity module may use the sparsity tensor to identify activations and weights to be used in MAC operations by the MAC units. In an embodiment where the sparse cell array 370 operates in the combined sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a combined sparsity tensor. In an embodiment where the sparse cell array 370 operates in the activation sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of an activation sparsity tensor. In an embodiment where the sparse cell array 370 operates in the weight sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a weight sparsity tensor. The sparsity module may be bypassed in the dense mode as no sparsity acceleration would be conducted.

[0081] The drain module 380 drains data from the sparse cell array 370 and writes the data to the local memory 340. The data may be outputs of MAC operations performed by MAC units in the sparse cell array 370. The drain module 380 may facilitate compression of data computed by the sparse cell array 370 based on spatial similarity in the data. In some embodiments, the drain module 380 may identify data blocks that are spatially similar. For instance, the drain module 380 may identify data blocks in an output tensor of a DNN layer that are adjacent to each other in the output tensor and have similar activations.

[0082] In some embodiments, the drain module 380 may exploit spatial similarity within separate segments of the output tensor. The drain module 380 may identify multiple data blocks from a single segment. Each data block is a different portion of the segment. The data blocks are spatially adjacent to each other as they are all in the same segment. The drain module 380 may measure similarities between these data blocks based on values of the activations in these data blocks. A data block is a tensor, which is a subtensor of a segment of the output tensor. The size of a data block ("block size") may be predetermined, e.g., by the DNN module 301.

[0083] In some embodiments, the drain module 380 may determine a difference score indicating a measurement of the difference between two data blocks. The drain module 380 may compare the difference score with a threshold score and determine that the two data blocks are similar when the difference score is below the threshold score. In other embodiments, the drain module 380 may determine a similarity score indicating a measurement of the similarity between two data blocks. The drain module 380 may compare the similarity score with a threshold score and determine that the two data blocks are similar when the similarity score is above the threshold score. The threshold score may be predetermined, e.g., by the DNN module 301.

[0084] In some embodiments, the drain module 380 may select one or more data blocks in a segment as anchor(s). The drain module 380 may compare each anchor with the other data blocks in the segment to detect spatial similarity in the segment. The drain module 380 may detect spatial similarity in different segments separately. For instance, the drain module 380 would not compare a data block in a segment with any data block in a different segment.

[0085] After the drain module 380 determines that a data block is spatially similar to an anchor, the drain module 380 may write the anchor into the memory 340 and bypass writing the data block into the memory 340. The drain module 380 may also generate storage element pointers for all the data blocks. The data block, which is not written into the memory 340, may have a storage element pointer pointing to the anchor. When the next layer is executed, the anchor would be read from the memory 340, e.g., by the load module 360, and used for computations, e.g., by the sparse cell array 370, in the next layer. The data block would be skipped by the load module 360 and the sparse cell array 370. The result of the computation on the anchor would be reused as result of the computation on the skipped data block. A single segment may have multiple skipped data blocks.

[0086] In some embodiments, the drain module 380 may drain data on a cell level. For each data processing cell, the drain module 380 may drain outputs of MAC units in the data processing cell based on a row index or column index of each MAC unit. For instance, the drain module 380 may use a sequence of cycles to drain data from a data processing cell. The drain module 380 may drain the output of some of the MAC units in each cycle. The sequence of the cycles may be configured based on a configuration parameter indicating the operation mode of the load module 360.

[0087] In some embodiments, the drain module 380 may determine whether to drain the output of an MAC unit based on the column index of the MAC unit when the load module operates in the activation sparsity mode versus based on the row index of the MAC unit when the load module operates in the weight sparsity mode. For instance, for MAC operations where the load module 360 operates in the activation sparsity mode, the drain module 380 may drain the output of a different MAC column in each cycle. The sequence of cycles may start with the first MAC column (e.g., the MAC column on the left side of the data processing cell) and end with the last MAC column (e.g., the MAC column on the right side of the data processing cell). For MAC operations where the load module 360 operates in the weight sparsity mode, the drain module 380 may drain the output of a different MAC row in each cycle. The sequence of cycles may start with the first MAC row (e.g., the MAC row at the top of the data processing cell) and end with the last MAC row (e.g., the MAC column at the bottom of the data processing cell). In other embodiments, the drain module 380 may determine whether to drain the output of an MAC unit based on the row index of the MAC unit when the load module operates in the activation sparsity mode versus based on the column index of the MAC unit when the load module operates in the weight sparsity mode.

[0088] The drain module 380 may also include sparsity encoding logic that can convert outputs of the sparse cell array 370 from a dense format to a sparse format. For instance, the drain module 380 may be implemented with one or more sparsity encoders. A sparsity encoder converts dense data to compressed data based on sparsity in the dense data. For instance, the sparsity encoder may remove zeros in an activation tensor computed by the sparse cell array 370 to convert the activation tensor to a compressed activation tensor. The sparsity encoder may also generate sparsity tensors, including activation sparsity tensors.

[0089] In some embodiments, the data drained from the sparse cell array 370 may be at least part of an output tensor (e.g., the output tensor 230 in FIG. 2) of a deep learning operation. The sparsity encoder may generate a compressed version of the output tensor. The sparsity encoder may identify every zero-valued activation in the output tensor and remove these activations from the output tensor to generate a compressed activation tensor (aka "sparse activation tensor"). The sparsity encoder may also generate one or more sparsity tensors for the output tensor. A sparsity tensor may correspond to a portion of the output tensor (e.g., the vector 235 in FIG. 2). The sparsity tensor may include sparsityelements (e.g., bits), each of which corresponds to a different activation in the vector and indicates whether the corresponding activation is zeroed or not.

[0090] The drain module 380 may write the compressed activation tensor and the one or more sparsity tensors into the local memory 340. The sparse activation tensor and the one or more sparsity tensors may be further loaded to the memory 310, e.g., through the DMA engine 320. Additionally or alternatively, the sparse activation tensor and the one or more sparsity tensors may be loaded by the load module 360 to the sparse cell array for further computation, e.g., for performing a deep learning operation in the next layer. Certain aspects of the drain module 380 are described below in conjunction with FIG. 8.

[0091] FIG. 4 is a block diagram of a DNN module 400, in accordance with various embodiments. The DNN module 400 facilitates transformation of matrix multiplications to convolutions. The DNN module 400 may be an embodiment of the DNN module 301 in FIG. 3. As shown in FIG. 4, the DNN module 400 includes an interface module 410, a training module 420, a compressing module 430, a validating module 440, and a datastore 450. In other embodiments, alternative configurations, different or additional components may be included in the DNN module 400. Further, functionality attributed to a component of the DNN module 400 may be accomplished by a different component included in the DNN module 400 or a different module or system.

[0092] The interface module 410 facilitates communications of the DNN module 400 with other modules or systems. For example, the interface module 410 establishes communications between the DNN module 400 with an external database to receive data that can be used to train DNNs or input into DNNs to perform tasks. As another example, the interface module 410 supports the DNN module 400 to distribute DNNs to other systems, e.g., computing devices configured to apply DNNs to perform tasks.

[0093] The training module 420 trains DNNs by using a training dataset. The training module 420 forms the training dataset. In an embodiment where the training module 420 trains an DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe ground-truth classifications of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a part of the training dataset may be used to initially train the DNN, and the rest of the training dataset may be held back as a validation subset used by the validating module 440 to validate performance of a trainedDNN. The portion of the training dataset not including the tuning subset and the validation subset may be used to train the DNN.

[0094] The training module 420 also determines hyperparameters for training the DNN. Hyperparameters are variables specifying the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., weights of filters). In some embodiments, hyperparameters include variables determining the architecture of the DNN, such as number of hidden layers, etc. Hyperparameters also include variables which determine how the DNN is trained, such as batch size, number of epochs, etc. A batch size defines the number of training samples to work through before updating the parameters of the DNN. The batch size is the same as or smallerthan the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backwards through the entire network. The number of epochs defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each training sample in the training dataset has had an opportunity to update the parameters inside the DNN. An epoch may include one or more batches. The number of epochs may be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0095] The training module 420 defines the architecture of the DNN, e.g., based on some of the hyperparameters. The architecture of the DNN includes an input layer, an output layer, and a plurality of hidden layers. The input layer of an DNN may include tensors (e.g., a multidimensional array) specifying attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layers are layers between the input layer and output layer. The hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully-connected layers, normalization layers, SoftMax or logistic layers, and so on. The convolutional layers of the DNN abstract the input image to a feature map that is represented by a tensor specifying the feature map height, the feature map width, and the feature map channels (e.g., red, green, blue images include 3 channels). A pooling layer is used to reduce the spatial volume of input image after convolution. It is used between two convolution layers. A fully-connected layer involvesweights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

[0096] In the process of defining the architecture of the DNN, the training module 420 also adds an activation function to a hidden layer or the output layer. An activation function of a layer transforms the weighted sum of the input of the layer to an output of the layer. The activation function may be, for example, a ReLU activation function, a tangent activation function, or other types of activation functions.

[0097] After the training module 420 defines the architecture of the DNN, the training module 420 inputs a training dataset into the DNN. The training dataset includes a plurality of training samples. An example of a training sample includes an object in an image and a ground-truth label of the object. The training module 420 modifies the parameters inside the DNN ("internal parameters of the DNN") to minimize the error between labels of the training objects that are generated by the DNN and the ground-truth labels of the objects. The internal parameters include weights of filters in the convolutional layers of the DNN. In some embodiments, the training module 420 uses a cost function to minimize the error.

[0098] The training module 420 may train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm will work through the entire training dataset. One epoch means that each sample in the training dataset has had an opportunity to update internal parameters of the DNN. After the training module 420 finishes the predetermined number of epochs, the training module 420 may stop updating the parameters in the DNN. The DNN having the updated parameters is referred to as a trained DNN.

[0099] The compressing module 430 compresses DNNs. For instance, the compressing module 430 facilitates compressing DNN layers based on spatial similarity in input data. The compressing module 430 may determine configuration parameters for compression based on spatial similarity. In some embodiments, the compressing module 430 generates a configuration parameter indicating the number of layers to be compressed in a DNN. When there are more compressed layers, the DNN inference would be more efficient, more memory storage would be saved, but the DNN accuracy may be lower. The compressing module 430 may also select which layers to be compressed. For instance, the compressing module 430 may select one or more layers that have more computations or have a larger size (e.g., more internal parameters) than other layers in the DNN.

[0100] In some embodiments, the compressing module 430 determines a threshold score that would be used to determine whether data blocks are similar. When the difference score between two data blocks is below the threshold score or the similarity score between two data blocks are above the threshold score, the two data blocks would be considered spatially similar. Further, the compressing module 430 may determine a block size. The block size may be the spatial size of a unit data block in one or more dimensions. Data blocks may be extracted from the input tensor of a DNN layer based on the block size for spatial similarity exploitation. The block size may indicate the number of reused / skipped activations. As the block size increases, more values in the input tensor may be skipped from storage and computation, and more values in the input tensor may be reused. The accuracy of the DNN layer (or the entire DNN) may reduce as the block size increases.

[0101] In some embodiments, the compressing module 430 may determine the number of compressed layers, threshold score, or block size based on one or more factors. Examples of the factors include a target accuracy of the DNN, a target compression ratio, amount of computational resources that can be allocated to the DNN inference, estimated amount of computation resources needed for the DNN inference, other factors, or some combination thereof.

[0102] In some embodiments, the compressing module 430 may add pruning operations to DNN layers to reduce computational complexity or memory usage. A pruning operation may prune weight tensors of a DNN layer by changing one or more nonzero valued weights of the layer to zeros. The modification may be done before, during, or after training. Weights may be pruned during training, during inference, or a combination of both. The compressing module 430 may determine a sparsity ratio for a DNN layer. The sparsity ratio may be a ratio of the number of zero-valued weight to the total number of weights in the layer. The compressing module 430 may perform the pruning operation till the sparsity ratio of the DNN layer meets a target sparsity ration, such as 10%, 20%, 30%, 40%, 50%, and so on.

[0103] In some embodiments, the compressing module 430 may select one or more layers in a DNN and modify each selected layer with a pruning operation. For instance, the compressing module 430 may select computationally complex layers, such as layers with large filters. For a pruning operation of a layer or of a type of layer, the compressing module 430 may determine a weight threshold that would not cause a loss of the accuracy of the DNN to exceed an accuracy loss constraint. A pruning operation may modify weights havingabsolute values above the weight threshold to zeros and leave the other weights unchanged. The weight pruning can reduce memory storage as zero-valued weights may not be stored. Also, the number of operations in the layer can be reduced as computations on zero-valued weights can be skipped without impacting the output of the layer. In some embodiments, the compressing module 430 may also measure energy saving, final DNN accuracy, or layer-wise sparsity caused by pruning operations.

[0104] After compressing a DNN, the compressing module 430 may fine tune the DNN, e.g., through a retraining process. The compressing module 430 may fine tunes DNNs after weights are pruned. In some embodiments, the fine-tuning process is a retraining or further training process. For instance, after weights in a DNN are pruned, the compressing module 430 may furthertrain the DNN by inputting a training dataset into the DNN. The values of the unpruned weights in the DNN may be modified based on outputs of the DNN and ground-truth labels of the training samples in the training dataset. In some embodiments, the values of the pruned weights (i.e., zero) are not changed during the fine-tuning process. For instance, the compressing module 430 may place a mask over a pruned weight block and the mask can prevent values in the pruned weight blocks from being changed during the fine-tuning process. In other embodiments, the values of all weights, including the pruned weights, may be changed during the fine-tuning process. After one or more cycles of retraining and weight changing by the compressing module 430, the compressing module 430 may perform a new pruning process, e.g., by selecting weight blocks and pruning the selected weight blocks. In some embodiments, the weight pruning process may be repeated multiple times before the fine-tuning process is done.

[0105] In some embodiments, the number of epochs in the fine-tuning process may be different from the number of epochs in the training process in which the pre-pruning values of the weights are determined. For instance, the fine-tuning process may have less epochs than the training process. In an example, the number of epochs in the fine-tuning process may be relatively small, such as 2, 3, 4, 5, and so on.

[0106] The validating module 440 verifies accuracy of trained or compressed DNNs. In some embodiments, the validating module 440 inputs samples in a validation dataset into a trained DNN and uses the outputs of the DNN to determine the model accuracy. In some embodiments, a validation dataset may be formed of some or all the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples,other than those in the training sets. In some embodiments, the validating module 440 may determine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN. The validating module 440 may use the following metrics to determine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN), where precision may be how many the DNN correctly predicted (TP or true positives) out of the total it predicted (TP + FP or false positives), and recall may be how many the DNN correctly predicted (TP) out of the total number of objects that did have the property in question (TP + FN or false negatives). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single measure.

[0107] The validating module 440 may compare the accuracy score with a threshold score. In an example where the validating module 440 determines that the accuracy score of the DNN is less than the threshold score, the validating module 440 instructs the training module 420 to re-train the DNN. In one embodiment, the training module 420 may iteratively re-train the DNN until the occurrence of a stopping condition, such as the accuracy measurement indication that the DNN may be sufficiently accurate, or a number of training rounds having taken place.

[0108] The datastore 450 stores data received, generated, used, or otherwise associated with the DNN module 400. For example, the datastore 450 stores the datasets used by the training module 420 and validating module 440. The datastore 450 may also store data generated by the training module 420 and validating module 440, such as the hyperparameters for training DNNs, internal parameters of trained DNNs (e.g., weights, etc.), data for sparsity acceleration (e.g., sparsity bitmap, etc.), and so on. The datastore 450 may store configuration parameters or other data generated by the compressing module 430. In the embodiment of FIG. 4, the datastore 450 is a component of the DNN module 400. In other embodiments, the datastore 450 may be external to the DNN module 400 and communicate with the DNN module 400 through a network.

[0109] FIG. 5 illustrates an example data processing cell 500, in accordance with various embodiments. The data processing cell 500 may be a sparse cell in a sparse cell array, e.g., the sparse cell array 370 in FIG. 3. The data processing cell 500 includes 16 MAC units 510 (individually referred to as "MAC unit 510"), which constitutes a MAC array having four rows and four columns. The MAC array has a spatial shape of 5x4, meaning the height of the MAC array is four and the width of the MAC array is also 5. The data processing cell 500 alsoincludes 16 weight register files 520 (individually referred to as "weight register file 520"), 16 activation register files 530 (individually referred to as "activation register file 530"), four row buffers 540 (individually referred to as "row buffer 540"), and sparsity modules 560 (individually referred to as "sparsity module 560"). In other embodiments, the data processing cell 500 may include fewer, more, or different components. For example, the data processing cell 500 may include a different number of MAC units 510, weight register files 520, activation register files 530, row buffers 540, or sparsity modules 560. As another example, the data processing cell 500 may include column buffers in lieu of or in addition to the row buffers 540. Also, the shape (e.g., the height or width) of the MAC array may be different.

[0110] The MAC units 510 are configured to perform MAC operations. Each MAC unit 510 may include one or more multipliers and one or more adders. A multiplier may multiply an activation with a weight at a time to compute a product. In some embodiments (e.g., embodiments where the MAC unit 510 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. An adder may accumulate products computed by the multipliers. Even though not shown in FIG. 5, the data processing cell may include an adder tree including a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units 510. The number of adders in the first tier may be half of the number of the MAC units 510, and each adder may accumulate the outputs of two MAC units 510. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the data processing cell 500.

[0111] The weight register files 520 store weights to be processed in MAC operations. In the embodiments of FIG. 5, four weight register files 520 are grouped into a storage set that stores data to be used by a column of MAC units 510. There are four storage sets corresponding to the four columns of MAC units 510. In some embodiments, a weight register file 520 may correspond to a MAC unit 510 and store data to be processed by the MAC unit. In some embodiments, all the 16 weight register files 520 constitute a weight storage unit.

[0112] The activation register files 530 stores activations to be processed in MAC operations. In the embodiments of FIG. 5, four activation register files 530 are grouped into a storage set that stores data to be used by a row of MAC units 510. There are four storage sets corresponding to the four rows of MAC units 510. In some embodiments, an activation register file 530 may correspond to a MAC unit 510 and store data to be processed by the MAC unit. In some embodiments, all the 16 activation register files 530 constitute an activation storage unit. The row buffers 540 store outputs of the MAC units 510. Each row buffer 540 may drain outputs of a single row of MAC units 510.

[0113] The sparsity module 560 facilitates dynamic sparsity-based acceleration in the data processing cell 500. In the embodiments of FIG. 5, each sparsity module 560 includes a sparsity tensor storage unit 565 and a control logic 567. The sparsity tensor storage unit 565 stores combined sparsity tensors. A combined sparsity tensor stored in the sparsity tensor storage unit 565 may correspond to an activation tensor and a weight tensor. A nonzero element in the combined sparsity tensor may correspond to a nonzero activation-weight pair that includes a nonzero activation and a nonzero weight. The position of the nonzero activation in the activation tensor may match the position of the nonzero weight in the weight tensor. The product of the nonzero activation and nonzero weight would be nonzero.

[0114] The control logic 567 may control transmission of activations and weights stored from the weight register files 520 and the activation register files 530 to the MAC units 510 based on sparsity tensors. For instance, the control logic 567 may select a subset of the weights stored in the weight register files 520 and select a subset of activations stored in the activation register files 530 based on a combined sparsity tensor. The selected weights and activations constitute nonzero activation-weight pairs. The control logic 567 may transmit the selected weights and activations to the MAC units 510 for performing MAC operations. The other weights stored in the weight register files 520 and the other activations stored in the activation register files 530 are skipped from computation. In the embodiments of FIG. 5, each sparsity module 560 controls sparsity acceleration in a respective MAC unit 510. As the sparsity acceleration is either based on both weight sparsity and activation sparsity, 16 sparsity modules 560 are used for acceleration computations in the 16 MAC units 510.

[0115] As shown in FIG. 5, the data processing cell 500 is associated with multiplexers (MUXs) 503, 504, 505, and 506. In other embodiments, the data processing cell 500 may beassociated with a different number of MUXs or other devices. The MUX 503 facilitates loading weights, e.g., from the local memory 340, into the weight register files 520. The MUX 504 facilitates loading activations, e.g., from the local memory 340, into the activation register files 530. The MUX 505 facilitates loading sparsity tensors into the sparsity tensor storage unit 565. The MUX 506 may be a drain MUX that can facilitate draining outputs of the MAC units 510, e.g., to the local memory 340.

[0116] In some embodiments, the data processing cell 500 may also execute matrix multiplications converted from Fourier transform operations. For an example Fourier transform operation, the MAC units 510 may perform MAC operations in the two sequences of matrix multiplications converted from the Fourier transform operation. The weight register files 520 may be used to store data points in transformation tensor of the Fourier transform operation. The activation register file 530 may be used to store data points in the input tensor of the Fourier transform operation. The row buffers 540 may store data points in the output tensor of the Fourier transform operation.

[0117] FIG. 6 illustrates a data processing unit 600, in accordance with various embodiments. The data processing unit 600 may be an example of the sparse cell array 370 in FIG. 3. In FIG. 6, the data processing unit 600 includes data processing cells 610 (individually referred to as "data processing cell 610") arranged in four columns and four rows, an activation memory 620, and a weight memory 630. The data processing unit 600 may also be referred to as a data processing unit. In other embodiments, the data processing unit 600 may include fewer, more, or different components. For instance, the data processing unit 600 may include a different number of columns, rows, or data processing cells 610.

[0118] Each data processing cell 610 may perform sparsity accelerated MAC operations. The data processing cells 610 may facilitate dynamic sparsity mode. For instance, the sparsity modes of a data processing cell 610 may be dynamically changed between a combined sparsity mode, an activation sparsity mode, a weight sparsity mode, and a dense mode. An embodiment of a data processing cell 610 may be the data processing cell 500 in FIG. 5. The activation memory 620 stores activations, such as activations in input tensors of deep learning operations. Activations may be loaded from the activation memory 620 to data processing cells 610. The weight memory 630 stores weights, such as weights in filters of deep learning operations. Weights may be loaded from the weight memory 630 to dataprocessing cells 610. The activation memory 620 or weight memory 630 may be a buffer. In other embodiments, the data processing unit 600 may include a dense data memory and a sparse data memory in lieu of the activation memory 620 and weight memory 630. The dense data memory may store dense tensors, e.g., dense tensors generated by the load module 360. The sparse data memory may store sparse tensors.

[0119] The data processing unit 600 may also execute matrix multiplications in Fourier transform operations. The activation memory 620 may be used to store input tensors of the Fourier transform operations. The weight memory 630 may be used to store transformation matrices of the Fourier transform operations.

[0120] FIG. 7 illustrates an example MAC unit 700, in accordance with various embodiments. The MAC unit 700 may be a unit component of a MAC array in a data processing cell, e.g., the data processing cell 500. In the embodiments of FIG. 7, the MAC unit 700 is coupled to an activation register file 710, a weight register file 720, an output register file 750, and a sparsity accelerator 760. The MAC unit 700 includes a multiplier 730 and an adder 740. In other embodiments, the MAC unit 700 may include fewer, more, or different components. The multiplier 730 and adder 740 may constitute an MAC unit. The activation register file 710 may be an example of the activation register files 530 in FIG. 5. The weight register file 720 may be an example of the weight register files 520 in FIG. 5.

[0121] The activation register file 710 stores an activation operand, which may be a context. The weight register file 720 stores a weight operand. The activation operand and weight operand may be loaded from a memory (e.g., the memory 340) into the activation register file 710 and the weight register file 720. The sparsity accelerator 760 receives a sparsity bitmap 715 that corresponds to the sparse tensor in the weight register file 720. The sparsity bitmap 715 may be a combined sparsity bitmap when the MAC unit 700 operates in a combined sparsity mode. The sparsity bitmap 715 may be an activation sparsity bitmap when the MAC unit 700 operates in an activation sparsity mode. The sparsity bitmap 715 may be a weight sparsity bitmap when the MAC unit 700 operates in a weight sparsity mode. The sparsity bitmap 715 may have the same size (e.g., the same number of elements) as or a larger size than the activation operand or the weight operand.

[0122] Using the sparsity bitmap 715, the sparsity accelerator 760 selects four activations from the activation register file 710 and selects four weights from the weight register file 720. The sparsity accelerator 760 transmits the selected activations and weights to themultiplier 730. These selected data elements correspond to the nonzero valued elements of the sparsity bitmap 715. The four selected activations and the four selected weights may constitute four activation-weight pairs. The multiplier 730 may compute a product based on each activation-weight pair and therefore, compute four products in total. The four products may be provided to the adder 740. Even though FIG. 7 shows a single multiplier 730, the MAC unit 700 may include multiple multipliers that can perform multiple multiplication operations at the same time.

[0123] The adder 740 accumulates the four products and computes a unit-level internal partial sum. The four unselected elements of the dense tensor are not processed to save power and time, which would not impact the value of the unit-level internal partial sum. For instance, when the dense tensor is a dense activation tensor, the weights corresponding to the unselected activations are zeros so the products of the unselected activations and the weights would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. Similarly, when the dense tensor is a dense weight tensor, the activations corresponding to the unselected weights are zeros so the products of the unselected weights and the activations would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. In other embodiments, the MAC unit 700 may operate in a dense mode in which the sparsity bitmap 715 is not used and the sparsity accelerator 760 is inactive. The MAC unit 700 may process all the activations in the activation operand and all the weights in the weight operand.

[0124] The unit-level internal partial sum may be stored in the output register file 750. In some embodiments, the unit-level internal partial sum may be used multiple times. For instance, the activation operand may represent N data blocks in the input tensor of the convolution, where N is an integer greater than 1. Instead of processing all the N data blocks to compute N unit-level internal partial sums, the unit-level internal partial sum is computed once and used N times in the convolutional layers as N unit-level internal partial sums.

[0125] In some embodiments, the adder 740 receives one or more unit-level internal partial sums from one or more other MAC units, e.g., through a MUX 735. The adder 740 can accumulate the one or more unit-level internal partial sums with the unit-level internal partial sum of the MAC unit 700 and store the result of the accumulation (i.e., a multi-PEinternal partial sum) in the output register file 750. The one or more other MAC units may be in the same column as the MAC unit 700 in a sparse cell. The multi-unit internal partial sum may be a column-level internal partial sum. In some embodiments, the unit-level internal partial sum of the MAC unit 700 or the multi-unit internal partial sum may be sent to one or more other MAC units for further accumulation.

[0126] FIG. 8 is a block diagram of a drain module, in accordance with various embodiments. The drain module 800 extracts output activations computed by sparse cells (e.g., sparse cells in the sparse cell array 370) and writes the output activations into memories (e.g., the local memory 340). The drain module 800 may be an example of the drain module 380 in FIG. 3. As shown in FIG. 8, the drain module 800 includes post processing engines 810 (individually referred to as "post processing engine 810"), circular buffers 820 (individually referred to as "circular buffer 820"), a drain staging buffer 830, a global drain module 840, drain banks 850 (individually referred to as "drain bank 850"), data compression modules 860 (individually referred to as "data compression module 860"), a write module 870, and a write buffer 880. In other embodiments, alternative configurations, different or additional components may be included in the drain module 800. Further, functionality attributed to a component of the drain module 800 may be accomplished by a different component included in the drain module 800 or a different module or system, such as the load module 360.

[0127] The post processing engines 810 process outputs of a sparse cell array, e.g., the sparse cell array 370. In some embodiments, a post processing engine 810 computes activation functions. The post processing engine 810 may receive outputs of the sparse cell array 370 as inputs to the activation functions. In addition or alternative to activation functions, the post processing engine 810 may perform other types of post processing on outputs of the sparse cell array 370. For instance, the post processing engine 810 may apply a bias on an output of the sparse cell array 370. The post processing engine 810 may transmit the results of the post processing to the circular buffers 820. The output data stored in the circular buffers 820 may be further transmitted and written into the drain staging buffer 830.

[0128] The global drain module 840 may select activations stored in the drain staging buffer 830. In some embodiments, the global drain module 840 selects activations in a predetermined manner, e.g., a lxlxOC manner, where OC stands for output channel. Insome embodiments, the global drain module 840 selects one of a predetermined number of entries of the drain staging buffer 830. The predetermined number may be the number of MAC units in a column of the sparse cell. In other embodiments, the global drain module 840 may select a predetermined amount of data, e.g., 16 bytes, 32 bytes, and so on. After the entries are selected, the global drain module 840 may select one or more drain banks 850 and multicast the selected entries to the selected drain bank(s) 850. In an example, the global drain module 840 may be associated with 16 drain banks 850 in four groups. Each group may include 4 drain banks 850. The global drain module 840 may assign the right rotate value specific to each drain bank 850 to align and concatenate the consecutive OCs in a single drain bank 850. The global drain module 840 may further write the correct set of bytes in the selected line of the drain staging buffer 830 to the drain bank 850.

[0129] In some embodiments, a single drain bank 850 may store an activation vector including activations having the same (OX, OY) coordinate but different OCs. The activations of the activation vector may be arranged in sequence in accordance with their OCs. For instance, the OC coordinate of the first activation in the activation vector may be in 0, the OC coordinate of the second activation may be in 1, the OC coordinate of the third activation may be in 2, and so on. Different drain banks 850 may store different activation vectors. The activations in different drain banks 850 may have different OX coordinates or different OY coordinates.

[0130] The data compression modules 860 compress output data of DNN layers. In some embodiments, the data compression modules 860 may receive an output tensor (e.g., the output tensor 230 in FIG. 2) of a layer, e.g., from the global drain module 840. The data compression modules 860 may detect spatial similarity of data within various segments of the output tensor and compress the output tensor based on the detected spatial similarity. For instance, one or more data compression module 860 may identify multiple data blocks in each segment. In some embodiments, a data block may be an MAC context (also referred to as "context") that includes activations to be processed by a MAC unit for performing a convolution. In an example, the context may be a vector including activations with the same (OX, OY) coordinate but different OCs. In other examples, a data block may be a tensor of a different dimension or shape. The data compression module(s) 860 may extract data blocks from segments of the output tensor based on a configuration parameter that indicates the size of a unit data block.

[0131] A data compression module 860 may determine whether a context is spatially similar to any of the other contexts in the segment of the output tensor. For instance, the data compression module 860 may determine a difference score that measures the difference between the context and another context. The data compression module 860 may determine the difference score by determining the difference between norms (e.g., LI norms, L2 norms, etc.) of the two contexts or by applying an XOR operator on the two contexts. The data compression module 860 may further compare the difference score with a threshold score, which may be encoded by another configuration parameter. The data compression module 860 may determine that the two contexts are spatially similar when the difference score is below the threshold score; otherwise, the data compression module 860 may determine that the two contexts are spatially different.

[0132] In some embodiments, the data compression module 860 may identify one context in a segment of the output tensor as the anchor context of the segment. The data compression module 860 may compare each of the other contexts in the segment with the anchor to detect spatial similarity. In other embodiments, the data compression module 860 may identify multiple anchor contexts or use all the contexts in the segment as anchor contexts. For example, the data compression module 860 may compare multiple contexts with each of the other contexts in the segment to detect spatial similarity. As another example, the data compression module 860 may detect spatial similarity in every unique pair of contexts in the segment.

[0133] After the data compression module 860 determines that a context is spatially similar to the other context, the data compression module 860 may compress the segment by populating the activations in the context to the other context. The original activations in the other context would be replaced with the activations in the context. One of the contexts, as opposed to both contexts, may be stored and processed for executing the next DNN layer. The data compression module 860 may also generate storage element pointers for the contexts in the output tensor, including unstored contexts that are not stored. The storage element pointer of an unstored context may point to a context that is spatially similar to the unstored context.

[0134] The data compression modules 860 may also compress output tensors based on sparsity or activation threshold. In some embodiments, the data compression modules 860 may compress an output tensor based on sparsity or activation threshold after the datacompression module 860 compresses the output tensor on spatial similarity. The data compression modules 860 may compare the absolute value of each activation with the activation threshold. The data compression modules 860 may remove any activations whose absolute value is no greater than the activation threshold from the activation vector to generate a compressed activation vector. In some embodiments, the activation threshold may be zero, which would make the compression sparsity-based compression. In other embodiments, the activation threshold may be a value greater than zero. The removed activations may not be stored in the write buffer 880 or the local memory to save bandwidth and memory usage.

[0135] In some embodiments, a data compression module 860 may also generate one or more sparsity tensors for the output tensor. The sparsity tensor may include sparsity elements, each of which corresponds to a different activation in an activation vector and indicates whether the corresponding activation is removed or not. In some embodiments, the sparsity tensor may be a sparsity bitmap, and a sparsity element in the sparsity bitmap may be a bit. A zero bit may indicate that the corresponding activation is removed and not in the compressed activation vector, while a one bit may indicate that the corresponding activation is not removed and is in the compressed activation vector.

[0136] The write module 870 determines memory addresses of output activations and writes the output activations into a memory based on the memory addresses. An example of the memory may be the local memory 340. In some embodiments, the write module 870 determines memory address of activations in the compressed activation vectors. The write module 870 may avoid the determination of memory addresses for activations removed by the data compression module 860. The write module 870 may use the position of an activation in the output tensor of the deep learning operation to generate a memory address for the activation. For instance, the write module 870 may compute the 3D coordinate (e.g., a (OX, OY, OC) coordinate) of the activation. The write module 870 may identify the location of any (OX, OY, OC) coordinate of the output tensor in the memory.

[0137] In some embodiments (e.g., embodiments where the data compression module 860 compresses the output tensor), the write module 870 may write compressed activation vectors generated by the data compression module 860 into the memory. The write module 870 may skip activations removed by the data compression module 860 in the compression process. The write module 870 may also write sparsity tensors generated by the datacompression module 860 into the memory or a separate memory. In some embodiments, the write module 870 may determine memory addresses of sparsity tensors associated with the output tensor and write the sparsity tensors to the memory based on the memory addresses.

[0138] To write an activation vector or sparsity tensor into the memory, the write module 870 may generate a write request that includes the memory address(se) of the activation vector or sparsity tensor and transmit the write request to the memory. The memory, after receiving the write request, may process the write request and store the activation vector or sparsity tensor in one or more data write operations. The write buffer 880 may store the activation vector, sparsity tensor, or the write request while the write request or one or more previous write requests are being processed by the memory.

[0139] FIG. 9 illustrates an example data compression module 900, in accordance with various embodiments. The data compression module 900 may compress data computed in DNN layers. The data compression module 900 may be an example of the data compression modules 860 in FIG. 8. In some embodiments, the data compression module 900 may be arranged in the drain path of a DNN accelerator. The data compression module 900 may receive output data of a DNN layer and compress the output data before the output data is stored in a memory or is processed in the next DNN layer. The data compression module 900 may compress the output data based on spatial similarity and sparsity in the output data.

[0140] As shown in FIG. 9, the data compression module 900 includes a spatial similarity encoder 910 and a sparsity encoder 920. The spatial similarity encoder 910 includes an XOR operator 930, a comparator 940, and an address generator 950. The sparsity encoder 920 includes comparators 960 (individually referred to as "comparator 960") and a compression packer 970. In other embodiments, alternative configurations, different or additional components may be included in the compression module 900. For instance, the spatial similarity encoder 910 may include multiple XOR operators, comparators, or address generators. Further, functionality attributed to a component of the compression module 900 may be accomplished by a different component included in the compression module 900 or a different system.

[0141] The spatial similarity encoder 910 may compress an output tensor of a DNN layer based on spatial similarity. In some embodiments, the spatial similarity encoder 910 maypartition an output tensor of a DNN layer into segments. The segments may have the same spatial shape. A segment may be a tensor itself, which is a subtensor of the output tensor. The spatial similarity encoder 910 may detect spatial similarity in the tensors. Each tensor may be analyzed separately. For instance, the spatial similarity encoder 910 may detect spatial similarity between activations in the same tensor and may not detect similarity between activations in different tensors. The XOR operator 930 may receive two or more data blocks in the same tensor and determine a difference score that indicates the difference between the data blocks. The data blocks are subtensors. These subtensors are adjacent to each other as they are located in the same segment of the output tensor. In some embodiments, a subtensor may be a vector that includes activations from different channels. The subtensor may have less channels than the tensor. The number of channels in the subtensor may be a predetermined parameter, such as a configuration parameter determined by the DNN module. The XOR operator 930 may transmit the difference score to the comparator 940.

[0142] The comparator 940 may compare the difference score with a threshold score. When the difference score is below the threshold score, the two subtensors are determined to be spatially similar. The address generator 950 may determine a memory address where one of the two subtensors would be stored. The address generator 950 may also generate storage element pointers for the two subtensors. Both storage element pointers may point to the memory address. That way, memory storage for the other subtensor can be saved. Even though FIG. 9 shows a single XOR operator 930 and a single comparator 940, the spatial similarity encoder 910 may include multiple XOR operators or comparators that can detect similarity between multiple pairs of subtensor at the same time. The address generator 950 may generate a pointer table for each tensor in the output tensor. The pointer table may include storage element pointers for all the subtensors in the tensor.

[0143] The sparsity encoder 920 may further compress the output tensor based on sparsity. In some embodiments, the sparsity encoder 920 receives data already compressed by the spatial similarity encoder 910. Each comparator 960 may receive a different activation and compare the activation with a predetermined value. The predetermined value may be zero in some embodiments. In other embodiments, the predetermined value may be nonzero. The comparator 960 may change the value of an activation having a value lower than the predetermined value to zero and output zero. For an activation having a value not lowerthan the predetermined value, the comparator 960 may output the activation as is. In some embodiments, each comparator 960 may also output a bit indicating whether the data element output from the comparator is zero or not. In an example, when the output is zero, the bit is zero; when the output is nonzero, the bit is one. The data elements and bits are provided to the compression packer 970.

[0144] The compression packer 970 may generate a compressed tensor that includes the nonzero data elements output from the comparators 960. All the data elements in the compressed tensor may be nonzero. The compressed tensor may have a smaller size (e.g., less data elements) than the output tensor. The compression packer 970 also generates a sparsity tensor that indicates positions of the data elements of the compressed tensor in the output tensor. The sparsity tensor may have the same size as the output tensor.Example Spatial Similarity Exploitation

[0145] FIG. 10 illustrates an example spatial comparison process, in accordance with various embodiments. The spatial comparison process in FIG. 10 is spatial comparison with a single anchor. The spatial comparison process may be performed by the drain module 380 in FIG.3. In the spatial comparison process, the spatial correlation between the anchor context and each of the other contexts in a tensor 1010 is determined. The tensor 1010 may be a segment of an activation tensor of a convolutional layer of a DNN and may be computed in a previous layer of the DNN. The tensor 1010 includes four subtensors 1011, 1012, 1013, and 1014 with position indices of X0Y0, X1Y0, X0Y1, and X1Y1, respectively. The subtensor 1011 with the position index of X0Y0 is used as the anchor. The subtensors 1011, 1012, 1013, and 1014 may be extracted from the tensor 1010 based on a block size of 1 X 1 X 8. The subtensors 1011, 1012, 1013, and 1014 each have 8 elements that may correspond to eight input channels of the convolution. The block size may be determined before the execution of the DNN is started. In some embodiments, the size of the subtensors 1011, 1012, 1013, and 1014 may be encoded by a configuration parameter generated by the DNN module 301.

[0146] A spatial similarity encoder 1000 detects spatial similarity within the tensor 1010 and compresses the tensor 1010 based on the detected spatial similarity. In the embodiments of FIG. 10, the spatial similarity encoder 1000 includes three XOR operators 1020 (individually referred to as "XOR operator 1020") and three comparators 1030 (individually referred to as "comparator 1030"). In other embodiments, the spatial similarity encoder 1000 may include fewer, more, or different components. The spatial similarity encoder 1000 receives the foursubtensors 1011, 1012, 1013, and 1014. The subtensor 1011, which is the anchor, is sent to all the three XOR operators 1020. Each XOR operator 1020 also receives a different one of the other three subtensor 1012, 1013, and 1014. The XOR operators 1020 determines difference scores that indicate similarity between the anchor subtensor 1011 and each of the other three subtensors 1012, 1013, and 1014. As shown in FIG. 10, the difference score for the subtensor 1012 is 2, the difference score for the subtensor 1013 is 1, and the difference score for the subtensor 1014 is 4.

[0147] The comparators 1030 compares the difference scores with a threshold score. The difference score may be encoded by another configuration parameter generated by the DNN module 301. In an example, the threshold score is 3. As the difference scores for the subtensor 1012 and the subtensor 1013 are below 3, the subtensor 1012 and the subtensor 1013 are determined to be spatially similar as the anchor subtensor 1011. As the difference score for the subtensor 1014 is above 3, the subtensor 1014 is determined to be spatially different from the anchor subtensor 1011. Accordingly, the values in the subtensor 1012 and the subtensor 1013 may be replaced by the values in the anchor subtensor 1011, while the subtensor 1014 is not changed.

[0148] The tensor 1010 is compressed, which generates a tensor 1040. As three subtensors of the tensor 1040 have the same values, it may be sufficient to store and perform a MAC operation on the subtensor 1011. The subtensor 1012 and the subtensor 1013 may not be stored or processed. The result of the MAC operation on the subtensor 1011 may be reused twice as the results of MAC operations on the subtensor 1012 and the subtensor 1013. Storage element pointers may be generated for the four subtensors with the storage element pointers for the subtensor 1012 and the subtensor 1013 pointing to the storage element(s) storing the subtensor 1011.

[0149] FIG. 11 illustrates another example spatial comparison process, in accordance with various embodiments. The spatial comparison process may be performed by the drain module 380 in FIG. 3. Different from the spatial comparison process in FIG. 10, the spatial comparison process in FIG. 11 does not use a single anchor subtensor. The spatial comparison process in FIG. 11 is conducted by a spatial similarity encoder 1100 on a tensor 1110. The tensor 1110 may be a portion of an activation tensor of a convolutional layer of a DNN and may be computed in a previous layer of the DNN. The tensor 1110 includes four subtensors with position index of X0Y0, X1Y0, X0Y1, and X1Y1, respectively. The subtensors1111, 1112, 1113, and 1114 may be extracted from the tensor 1110 based on a block size of 1 x 1 x 8. The subtensors 1111, 1112, 1113, and 1114 each have 8 elements that may correspond to eight input channels of the convolution. The block size may be determined before the execution of the DNN is started. In some embodiments, the size of the subtensors 1111, 1112, 1113, and 1114 may be encoded by a configuration parameter generated by the DNN module 301.

[0150] The spatial similarity encoder 1100 detects spatial similarity within the tensor 1110 and compresses the tensor 1110 based on the detected spatial similarity. In the spatial comparison process in FIG. 11, the spatial similarity encoder 1100 may identify the maximum number of spatial similarities among the four subtensors to minimize memory allocation. As shown in FIG. 11, the spatial similarity encoder 1100 includes four XOR operators 1120A-1120D (collectively referred to as "XOR operators 1120" or "XOR operator 1120") and four comparators 1130 (individually referred to as "comparator 1130"). In other embodiments, the spatial similarity encoder 1100 may include fewer, more, or different components. The spatial similarity encoder 1100 receives the four subtensors 1111, 1112, 1113, and 1114.

[0151] The XOR operator 1120A receives the subtensor 1111 and the subtensor 1112 and computes a difference score of 3. The difference score indicates the difference between the subtensor 1111 and the subtensor 1112. The XOR operator 1120B receives the subtensor 1111 and the subtensor 1113 and computes a difference score of 1. The difference score indicates the difference between the subtensor 1111 and the subtensor 1113. The XOR operator 1120C receives the subtensor 1111 and the subtensor 1114 and computes a difference score of 4. The difference score indicates the difference between the subtensor1111 and the subtensor 1114. The XOR operator 1120D receives the subtensor 1112 and the subtensor 1114 and computes a difference score of 1. The difference score indicates the difference between the subtensor 1112 and the subtensor 1114. In other embodiments, the spatial similarity encoder 1100 may also determine the difference between the subtensor1112 and the subtensor 1113 or the difference between the subtensor 1113 and the subtensor 1114.

[0152] The comparators 1130 compares the difference scores with a threshold score. In an example, the threshold score is 2. The spatial similarity encoder 1100 may determine that the subtensor 1111 has one spatially similar subtensor, i.e., the subtensor 1113, as thedifference score for the two subtensors is below 2. The spatial similarity encoder 1100 may also determine that the subtensor 1112 also has one spatially similar subtensor, i.e., the subtensor 1114, as the difference score for the two subtensors is below 2. The spatial similarity encoder 1100 may then transform the tensor 1110 into a tensor 1140 that has two unique subtensors, i.e., the subtensors 1111 and 1112. The subtensor 1113 and the subtensor 1114 may not be stored. Also, MAC operations on the subtensor 1113 and the subtensor 1114 may be skipped. The result of the MAC operation on the subtensor 1111 may be reused as the result of MAC operation on the subtensor 1113. The result of the MAC operation on the subtensor 1112 may be reused as the result of MAC operation on the subtensor 1114. Storage element pointers may be generated for the four subtensors with the storage element pointer for the subtensor 1113 pointing to the storage element(s) storing the subtensor 1111 and with the storage element pointer for the subtensor 1114 pointing to the storage element(s) storing the subtensor 1112.

[0153] As more spatial similarities are detected in the tensor 1110, more computational resources can be saved. In some embodiments, up to 75% of the memory storage for the tensor 1110 may be saved when all the four subtensors are determined to be spatially similar. When no spatial similarity is detected in the tensor 1110, the storage overhead may remain the same and read / write of all the four subtensors would be required for performing the convolution. The size of the vectors and the threshold score may be programmable. For instance, the size of the vectors or the threshold score may be determined based on a target accuracy of the convolution, a computational resource saving target, and so on. Even though the block size in FIG. 10 and FIG. 11 is 1 X 1 X 8, the block size may be different in other embodiments.

[0154] FIG. 12A illustrates storage element pointers generated for convolutions 1210 and 1220 executed without spatial similarity, in accordance with various embodiments. For the purpose of illustration and simplicity, the convolutions 1210 and 1220 have an activation tensor. The activation tensor may be a 3D tensor with a size of 6 X 6 X Cin. A single input channel of the activation tensor is an activation matrix 1201 with a size of 6 X 6. Each activation in the activation matrix 1201 may represent a context that includes a vector with activations in different input channels. All the activations in the context may have the same XY coordinate, which may be a position index of the context. In some embodiments, 3 x 3 x Cinmay be the unit segment of the activation tensor in which spatial similarity isexploited. The activation tensor may be partitioned into four tensors, each of which has a size of 3 x 3 x Cin. The analysis of spatial similarity may be conducted separately on these four tensors.

[0155] A matrix 1202 represents one of the four tenors. Each activation in the matrix 1202 corresponds to a unique data block. Each data block may be a vector that includes multiple input channels and may have an XY coordinate indicating the position of the data block in the activation tensor. Thus, the matrix 1202 includes 9 data blocks. Taking the matrix 1202 for example, FIG. 12A shows generation of storge element pointers for the matrix 1202. A pointer table 1203 is generated for the convolution 1210. SE in FIG. 12A standards for storage element. The pointer table 1203 includes 9 storage element pointers corresponding to the 9 data blocks. Each data block is represented by a data point in the matrix 1202. Each storage element pointer points to the storage element where the corresponding data block is stored. In the pointer table 1203, each storage element corresponds to a position index of the corresponding data block. The position index is a XY coordinate, which indicates the position of the data block in the activation matrix 1201. For instance, X0Y0 means the activation is in the first row and the first column of the activation matrix 1201, X1Y0 means the activation is in the second row and the first column of the activation matrix 1201, X0Y1 means the activation is in the first row and the second column of the activation matrix 1201, and X1Y1 means the activation is in the second row and the second column of the activation matrix 1201. For the convolution 1210, all the activations in the matrix 1202 are stored so the storage element pointers point to 9 different storage elements.

[0156] The convolution 1220 is performed with programmable thresholding. In an example where the threshold is programmed to 2, activations having absolute values lower than 2 would be changed to zero to introduce sparsity. As shown in FIG. 12A, the matrix 1202 is converted into a new matrix 1204 by changing 7 of the 9 activations to zero. The pointer table 1203 may be generated for the convolution 1220 as well with some storage element pointers point to storage elements storing zeros. In some embodiments, the zero-valued activations in the matrix 1204 are not stored so that two storage element pointers are needed for the two non-zero-valued activations. Multiplications of the zero-valued activations with weights may be skipped as the products would be zeros and would not impact the result of the convolution. As the values of the activations are changed, the programmable thresholding method can affect accuracy of the result of the convolution,which may reduce the influence of neighboring points on the final output. Even though retraining could potentially recover lost accuracy, it can increase the time and cost for edge inference DNN accelerators.

[0157] FIG. 12B illustrates storage element pointers generated for a convolution 1230 executed based spatial similarity in input data, in accordance with various embodiments. For the purpose of illustration, the input data of the convolution 1230 is the same as the convolutions 1210 and 1220, which may also be represented by the activation matrix 1201. Similarity between the 9 data blocks may be measured, e.g., based on L2 norms of the data blocks. In an example, all the 9 data blocks are determined to be similar. Based on the determination, the matrix 1202 is compressed into a single point X0Y0, represented by a new matrix 1205. A pointer table 1206 is generated for the convolution 1230 accordingly. As shown in FIG. 12B, the pointer table 1206 includes 9 storage element pointers pointing to the same storage element, which stores the X0Y0 data block. The X0Y0 data block is stored while the storage of the other 8 data blocks is skipped. Also, the computation on the X0Y0 would be conducted but the computations on the other 8 data blocks would be skipped.

[0158] Compared with the convolution 1210, the convolution 1230 can have better computational efficiency and memory bandwidth but less memory storage requirement. Also, compared with the convolution 1220, the convolution 1230 can have better accuracy as the compression is based on spatial similarity. The convolution 1230 can have better computational efficiency and memory bandwidth but less memory storage requirement than the convolution 1220, as the compression for the convolution 1230 can be used for both sparse data and non-sparse data.Example Method of Executing Convolutions

[0159] FIG. 13 is a flowchart showing a method 1300 of executing a convolution, in accordance with various embodiments. The method 1300 may be performed by the DNN accelerator 302 in FIG. 3. Although the method 1300 is described with reference to the flowchart illustrated in FIG. 13, many other methods for executing convolutions may alternatively be used. For example, the order of execution of the steps in FIG. 13 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0160] The DNN accelerator 302 identifies 1310 a first tensor and a second tensor from an activation tensor of a convolution in a layer of a neural network, wherein the first tensor and the second tensor are different portions of the activation tensor. In someembodiments, the activation tensor is an IFM of the convolution. In some embodiments, the first tensor is adjacent to the second tensor in the activation tensor. In some embodiments, the DNN accelerator 302 partitions the activation tensor into segments. The DNN accelerator 302 identifies 1310 the first tensor and the second tensor from the same segment. In some embodiments, the DNN accelerator 302 computes the activation tensor in a previous layer of the neural network. After computing the activation tensor, the DNN accelerator 302 stores one or more activations in the first tensor and bypasses storing any activations in the second tensor.

[0161] In some embodiments, the DNN accelerator 302 generates a first storage element pointer for the first tensor. The first storage element pointer points to a storage element in the memory where the one or more activations in the first tensor are stored. The DNN accelerator 302 generates a second storage element for the second tensor. The second storage element pointer points to the storage element. In some embodiments, the DNN accelerator 302 partitions the activation tensor into tensor groups. Each tensor group comprises a predetermined number of tensors that are different positions of the activation tensor. The DNN accelerator 302 identifies the first tensor and the second tensor from a tensor group.

[0162] The DNN accelerator 302 computes 1320 a data element by performing a MAC operation on the first tensor and one or more weights of the convolution. In some embodiments, the first tensor is a vector that comprises activations in different input channels of the convolution. The MAC operation is performed by a MAC unit in the DNN accelerator 302 on the first tensor and a weight operand including weights in the input channels of the convolution.

[0163] The DNN accelerator 302 determines 1330 whether a difference score is below a threshold, the difference score indicating a measurement of similarity between the first tensor and the second tensor. In some embodiments, the DNN accelerator 302 determines the difference score by determining a difference between a norm of at least part of the first tensor and a norm of at least part of the second tensor. In some embodiments, the DNN accelerator 302 determines the difference score by applying a XOR operator on the first tensor and the second tensor. In some embodiments, the DNN accelerator 302 determines that the difference score is below the threshold before computing the data element.

[0164] In some embodiments, the first tensor or the second tensor has a dimension equal to a predetermined number of input channels. The predetermined number is smallerthan the total number of input channels in the activation tensor. In some embodiments, the predetermined number is determined for the neural network before an execution of the neural network is started, the execution of the neural network comprises the MAC operation.

[0165] In some embodiments, the DNN accelerator 302 identifies a first subtensor from the first tensor. The first subtensor is a portion of the first tensor. The DNN accelerator 302 also identifies a second subtensor from the second tensor. The second subtensor is a portion of the second tensor. The DNN accelerator 302 determines the difference score by measuring a difference between the first subtensor and the second subtensor. In some embodiments, the first tensor or the second tensor has a dimension equal to a total number of input channels in the convolution. The first subtensor or the second subtensor has a corresponding dimension equal to a predetermined number of input channels. The predetermined number is smaller than the total number.

[0166] The DNN accelerator 302 stores 1340 the data element as a result of the MAC operation on the first tensor and as a result of a MAC operation on the second tensor, in response to determining that the difference score is below the threshold. The data element is used as the result of the MAC operation on the first tensor and reused as the result of the MAC operation on the second tensor. The MAC operation on the second tensor is bypassed. In some embodiments, the DNN accelerator 302 identifies a third tensor from the same segment of the activation tensor. The DNN accelerator 302 determines that a difference score for the third tensor is below the threshold. The difference score for the third tensor indicates a measurement of similarity between the first tensor and the third tensor. The DNN accelerator 302 reuses the data element as a result of a MAC operation on the third tensor.Example Computing Device

[0167] FIG. 14 is a block diagram of an example computing device 1400, in accordance with various embodiments. In some embodiments, the computing device 1400 can be used as at least part of the DNN system 300. A number of components are illustrated in FIG. 14 as included in the computing device 1400, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all ofthe components included in the computing device 1400 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1400 may not include one or more of the components illustrated in FIG. 14, but the computing device 1400 may include interface circuitry for coupling to the one or more components. For example, the computing device 1400 may not include a display device 1406, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1406 may be coupled. In another set of examples, the computing device 1400 may not include an audio input device 1418 or an audio output device 1408 but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 1418 or audio output device 1408 may be coupled.

[0168] The computing device 1400 may include a processing device 1402 (e.g., one or more processing devices). The processing device 1402 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 1400 may include a memory 1404, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 1404 may include memory that shares a die with the processing device 1402. In some embodiments, the memory 1404 includes one or more non-transitory computer- readable media storing instructions executable to perform operations for executing convolutions (e.g., the method 1300 described in conjunction with FIG. 13) or some operations performed by the DNN system 300. The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1402.

[0169] In some embodiments, the computing device 1400 may include a communication chip 1412 (e.g., one or more communication chips). For example, the communication chip 1412 may be configured for managing wireless communications for the transfer of data to and from the computing device 1400. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiationthrough a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0170] The communication chip 1412 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1412 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 1412 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1412 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced CordlessTelecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 1412 may operate in accordance with other wireless protocols in other embodiments. The computing device 1400 may include an antenna 1422 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0171] In some embodiments, the communication chip 1412 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 1412 may include multiple communication chips. For instance, a first communication chip 1412 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1412 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. Insome embodiments, a first communication chip 1412 may be dedicated to wireless communications, and a second communication chip 1412 may be dedicated to wired communications.

[0172] The computing device 1400 may include battery / power circuitry 1414. The battery / power circuitry 1414 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1400 to an energy source separate from the computing device 1400 (e.g., AC line power).

[0173] The computing device 1400 may include a display device 1406 (or corresponding interface circuitry, as discussed above). The display device 1406 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0174] The computing device 1400 may include an audio output device 1408 (or corresponding interface circuitry, as discussed above). The audio output device 1408 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0175] The computing device 1400 may include an audio input device 1418 (or corresponding interface circuitry, as discussed above). The audio input device 1418 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0176] The computing device 1400 may include a GPS device 1416 (or corresponding interface circuitry, as discussed above). The GPS device 1416 may be in communication with a satellite-based system and may receive a location of the computing device 1400, as known in the art.

[0177] The computing device 1400 may include another output device 1410 (or corresponding interface circuitry, as discussed above). Examples of the other output device 1410 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0178] The computing device 1400 may include another input device 1420 (or corresponding interface circuitry, as discussed above). Examples of the other input device 1420 may include an accelerometer, a gyroscope, a compass, an image capture device, akeyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0179] The computing device 1400 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1400 may be any other electronic device that processes data.Select Examples

[0180] The following paragraphs provide various examples of the embodiments disclosed herein.

[0181] Example 1 provides a method, including identifying a first tensor and a second tensor from an activation tensor of a convolution in a layer of a neural network, in which the first tensor and the second tensor are different portions of the activation tensor; computing a data element by performing a MAC operation on the first tensor and one or more weights of the convolution; determining whether a difference score is below a threshold, the difference score indicating a measurement of similarity between the first tensor and the second tensor; and in response to determining that the difference score is below the threshold, storing the data element as a result of the MAC operation on the first tensor and as a result of a MAC operation on the second tensor.

[0182] Example 2 provides the method of example 1, in which identifying the first tensor and the second tensor from the activation tensor includes partitioning the activation tensor into segments; and identifying the first tensor and the second tensor from a same segment of the activation tensor.

[0183] Example 3 provides the method of example 2, further including identifying a third tensor from the same segment of the activation tensor; determining that a difference score for the third tensor is below the threshold, the difference score for the third tensorindicating a measurement of similarity between the first tensor and the third tensor; and storing the data element as a result of a MAC operation on the third tensor.

[0184] Example 4 provides the method of any one of examples 1-3, in which the first tensor or the second tensor has a dimension equal to a predetermined number of input channels, and the predetermined number is smaller than a total number of input channels in the activation tensor.

[0185] Example 5 provides the method of example 4, in which the predetermined number is determined for the neural network before an execution of the neural network is started, the execution of the neural network includes the MAC operation.

[0186] Example 6 provides the method of any one of examples 1-5, in which determining whether the difference score is below the threshold includes determining whether the difference score is below the threshold before computing the data element.

[0187] Example 7 provides the method of any one of examples 1-6, further including computing the activation tensor in a previous layer of the neural network; and after computing the activation tensor, storing one or more activations in the first tensor and bypassing storing any activations in the second tensor.

[0188] Example 8 provides the method of example 7, further including generating a first storage element pointer for the first tensor, the first storage element pointer pointing to a storage element in a memory where the one or more activations in the first tensor are stored; and generating a second storage element for the second tensor, the second storage element pointer pointing to the storage element.

[0189] Example 9 provides the method of any one of examples 1-8, in which determining whether the difference score is below the threshold includes determining a difference between a norm of at least part of the first tensor and a norm of at least part of the second tensor.

[0190] Example 10 provides the method of any one of examples 1-9, in which determining whether the difference score is below the threshold includes applying a XOR operator on the first tensor and the second tensor.

[0191] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including identifying a first tensor and a second tensor from an activation tensor of a convolution in a layer of a neural network, in which the first tensor and the second tensor are different portions of theactivation tensor; computing a data element by performing a MAC operation on the first tensor and one or more weights of the convolution; determining whether a difference score is below a threshold, the difference score indicating a measurement of similarity between the first tensor and the second tensor; and in response to determining that the difference score is below the threshold, storing the data element as a result of the MAC operation on the first tensor and as a result of a MAC operation on the second tensor.

[0192] Example 12 provides the one or more non-transitory computer-readable media of example 11, in which identifying the first tensor and the second tensor from the activation tensor includes partitioning the activation tensor into segments; and identifying the first tensor and the second tensor from a same segment of the activation tensor.

[0193] Example 13 provides the one or more non-transitory computer-readable media of example 11 or 12, in which the first tensor or the second tensor has a dimension equal to a predetermined number of input channels, and the predetermined number is smaller than a total number of input channels in the activation tensor.

[0194] Example 14 provides the one or more non-transitory computer-readable media of any one of examples 11-13, in which determining whetherthe difference score is below the threshold includes determining whether the difference score is below the threshold before computing the data element.

[0195] Example 15 provides the one or more non-transitory computer-readable media of any one of examples 11-14, in which the operations further include computing the activation tensor in a previous layer of the neural network; and after computing the activation tensor, storing one or more activations in the first tensor and bypassing storing any activations in the second tensor.

[0196] Example 16 provides the one or more non-transitory computer-readable media of example 15, in which the operations further include generating a first storage element pointer for the first tensor, the first storage element pointer pointing to a storage element in a memory where the one or more activations in the first tensor are stored; and generating a second storage element for the second tensor, the second storage element pointer pointing to the storage element.

[0197] Example 17 provides the one or more non-transitory computer-readable media of any one of examples 11-16, in which determining whetherthe difference score is below the threshold includes applying a XOR operator on the first tensor and the second tensor.

[0198] Example 18 provides an apparatus, including a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations including identifying a first tensor and a second tensor from an activation tensor of a convolution in a layer of a neural network, in which the first tensor and the second tensor are different portions of the activation tensor, computing a data element by performing a MAC operation on the first tensor and one or more weights of the convolution, determining whether a difference score is below a threshold, the difference score indicating a measurement of similarity between the first tensor and the second tensor, and in response to determining that the difference score is below the threshold, storing the data element as a result of the MAC operation on the first tensor and as a result of a MAC operation on the second tensor..

[0199] Example 19 provides the apparatus of example 18, in which identifying the first tensor and the second tensor from the activation tensor includes partitioning the activation tensor into segments; and identifying the first tensor and the second tensor from a same segment of the activation tensor.

[0200] Example 20 provides the apparatus of example 18 or 19, in which the operations further include computing the activation tensor in a previous layer of the neural network; and after computing the activation tensor, storing one or more activations in the first tensor and bypassing storing any activations in the second tensor.

[0201] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims1. A method, comprising: identifying a first tensor and a second tensor from an activation tensor of a convolution in a layer of a neural network, wherein the first tensor and the second tensor are different portions of the activation tensor; computing a data element by performing a multiply-accumulate (MAC) operation on the first tensor and one or more weights of the convolution; determining whether a difference score is below a threshold, the difference score indicating a measurement of similarity between the first tensor and the second tensor; and in response to determining that the difference score is below the threshold, storing the data element as a result of the MAC operation on the first tensor and as a result of a MAC operation on the second tensor.

2. The method of claim 1, wherein identifying the first tensor and the second tensor from the activation tensor comprises: partitioning the activation tensor into segments; and identifying the first tensor and the second tensor from a same segment of the activation tensor.

3. The method of claim 2, further comprising: identifying a third tensor from the same segment of the activation tensor; determining that a difference score for the third tensor is below the threshold, the difference score for the third tensor indicating a measurement of similarity between the first tensor and the third tensor; and storing the data element as a result of a MAC operation on the third tensor.

4. The method of claim 1, wherein the first tensor orthe second tensor has a dimension equal to a predetermined number of input channels, and the predetermined number is smaller than a total number of input channels in the activation tensor.

5. The method of claim 4, wherein the predetermined number is determined for the neural network before an execution of the neural network is started, the execution of the neural network comprises the MAC operation.

6. The method of claim 1, wherein determining whether the difference score is below the threshold comprises: determining whether the difference score is below the threshold before computing the data element.

7. The method of claim 1, further comprising: computing the activation tensor in a previous layer of the neural network; and after computing the activation tensor, storing one or more activations in the first tensor and bypassing storing any activations in the second tensor.

8. The method of claim 7, further comprising: generating a first storage element pointer for the first tensor, the first storage element pointer pointing to a storage element in a memory where the one or more activations in the first tensor are stored; and generating a second storage element for the second tensor, the second storage element pointer pointing to the storage element.

9. The method of claim 1, wherein determining whether the difference score is below the threshold comprises: determining a difference between a norm of at least part of the first tensor and a norm of at least part of the second tensor.

10. The method of claim 1, wherein determining whether the difference score is below the threshold comprises: applying a XOR operator on the first tensor and the second tensor.

11. One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:identifying a first tensor and a second tensor from an activation tensor of a convolution in a layer of a neural network, wherein the first tensor and the second tensor are different portions of the activation tensor; computing a data element by performing a multiply-accumulate (MAC) operation on the first tensor and one or more weights of the convolution; determining whether a difference score is below a threshold, the difference score indicating a measurement of similarity between the first tensor and the second tensor; and in response to determining that the difference score is below the threshold, storing the data element as a result of the MAC operation on the first tensor and as a result of a MAC operation on the second tensor.

12. The one or more non-transitory computer-readable media of claim 11, wherein identifying the first tensor and the second tensor from the activation tensor comprises: partitioning the activation tensor into segments; and identifying the first tensor and the second tensor from a same segment of the activation tensor.

13. The one or more non-transitory computer-readable media of claim 11, wherein the first tensor or the second tensor has a dimension equal to a predetermined number of input channels, and the predetermined number is smaller than a total number of input channels in the activation tensor.

14. The one or more non-transitory computer-readable media of claim 11, wherein determining whether the difference score is below the threshold comprises: determining whether the difference score is below the threshold before computing the data element.

15. The one or more non-transitory computer-readable media of claim 11, wherein the operations further comprise: computing the activation tensor in a previous layer of the neural network; and after computing the activation tensor, storing one or more activations in the first tensor and bypassing storing any activations in the second tensor.

16. The one or more non-transitory computer-readable media of claim 15, wherein the operations further comprise: generating a first storage element pointer for the first tensor, the first storage element pointer pointing to a storage element in a memory where the one or more activations in the first tensor are stored; and generating a second storage element for the second tensor, the second storage element pointer pointing to the storage element.

17. The one or more non-transitory computer-readable media of claim 11, wherein determining whether the difference score is below the threshold comprises: applying a XOR operator on the first tensor and the second tensor.

18. An apparatus, comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising: identifying a first tensor and a second tensor from an activation tensor of a convolution in a layer of a neural network, wherein the first tensor and the second tensor are different portions of the activation tensor, computing a data element by performing a multiply-accumulate (MAC) operation on the first tensor and one or more weights of the convolution, determining whether a difference score is below a threshold, the difference score indicating a measurement of similarity between the first tensor and the second tensor, and in response to determining that the difference score is below the threshold, storing the data element as a result of the MAC operation on the first tensor and as a result of a MAC operation on the second tensor.

19. The apparatus of claim 18, wherein identifying the first tensor and the second tensor from the activation tensor comprises: partitioning the activation tensor into segments; andidentifying the first tensor and the second tensor from a same segment of the activation tensor.

20. The apparatus of claim 18, wherein the operations further comprise: computing the activation tensor in a previous layer of the neural network; and after computing the activation tensor, storing one or more activations in the first tensor and bypassing storing any activations in the second tensor.

Images