Dynamic decompression for channel separable operations in neural networks

Abstract

The computation block can dynamically decompress compressed data to perform channel-separable operations. The compressed data contains one or more non-zero data elements. The compressed data may be stored in a data store along with a sparsity bitmap of the input operand containing the compressed data. The decompression module may determine whether the input operand contains any zero data elements, for example, by determining whether the sparsity bitmap contains zero bits. After determining that the sparsity bitmap contains zero bits, the decompression module inserts the zero data elements into the compressed data based on the bit positions in the sparsity bitmap, generates the decompressed data, and updates the sparsity bitmap so that all bits are 1. The decompressed high-density data is sent to one or more processing elements (PEs) in the computation block to compute an output operand based on the decompressed high-density data.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority of U.S. Patent Application No. 18 / 184,921, entitled "Dynamic Decompression for Channel - Separable Operations in Neural Networks", filed on March 16, 2023, which is hereby incorporated by reference in its entirety.

[0002] This disclosure generally relates to neural networks and, more specifically, to dynamic decompression for channel - separable operations in deep neural networks (DNNs).

Background Art

[0003] DNNs are widely used in various artificial intelligence applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, as a trade - off for high accuracy, they require a great deal of computational cost. DNNs have very high computing requirements because each inference may require hundreds of millions of MAC (multiply - accumulate) operations as well as large amounts of data for reading and writing. Therefore, techniques for improving the efficiency of DNNs are needed.

[0004] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are shown by way of example and not by way of limitation in the figures of the accompanying drawings.

Brief Description of the Drawings

[0005] [Figure 1] A diagram showing an exemplary DNN according to various embodiments.

[0006] [Figure 2] A block diagram of a DNN accelerator according to various embodiments.

[0007] [Figure 3] These are block diagrams of computational blocks in various embodiments.

[0008] [Figure 4] This figure shows a data store implemented with a decompression module in various embodiments.

[0009] [Figure 5] This figure shows a dynamic decompression process in a decompression module comprising two extension units, according to various embodiments.

[0010] [Figure 6] This figure shows the inputs and outputs of a decompression module for four load rounds in various embodiments.

[0011] [Figure 7] This figure shows a DNN that can be performed without dynamic decompression using various embodiments.

[0012] [Figure 8] This figure shows a DNN that is executed with dynamic decompression using various embodiments.

[0013] [Figure 9] This figure shows exemplary standard convolutions in various embodiments.

[0014] [Figure 10] This figure shows exemplary depth unit convolutions according to various embodiments.

[0015] [Figure 11] This figure shows exemplary group convolutions according to various embodiments.

[0016] [Figure 12] This figure shows exemplary depth unit convolutions in a processing element (PE) according to various embodiments.

[0017] [Figure 13] This figure shows exemplary channel separable pooling operations in PE according to various embodiments.

[0018] [Figure 14] This figure shows another exemplary channel separable pooling operation in PE according to various embodiments.

[0019] [Figure 15] This figure shows exemplary channel-separable element-level addition in PE according to various embodiments.

[0020] [Figure 16] This figure shows another exemplary channel-separable element-unit addition in PE according to various embodiments.

[0021] [Figure 17] This figure shows exemplary channel-separable element-level multiplication in PE according to various embodiments.

[0022] [Figure 18] This figure shows PE arrays according to various embodiments.

[0023] [Figure 19] These are block diagrams of PE in various embodiments.

[0024] [Figure 20] This flowchart shows a method for dynamic decompression for channel separable calculations according to various embodiments.

[0025] [Figure 21] This is a block diagram of an exemplary computing device in various embodiments. [Modes for carrying out the invention]

[0026] Outline Over the past decade, AI-based data processing, particularly that based on DNNs (Deep Neural Networks), has grown rapidly. DNNs are widely used in the fields of computer vision, speech recognition, image processing, and video processing, primarily due to their ability to achieve accuracy beyond human levels. Coupled with the rapid increase in computing power of execution platforms, DNN model size and accuracy have improved significantly, leading to the adoption of DNN applications even within resource-constrained mobile and edge devices with limited energy availability.

[0027] A DNN layer may include one or more deep learning operations, such as convolution, pooling, element-wise operations, linear operations, and nonlinear operations. Deep learning operations in a DNN may be performed on one or more internal parameters of the DNN (e.g., weights) and one or more activation values, which are determined during the training phase. Activation values ​​may be data points (also called "data elements" or "elements"). The activation values ​​or weights of a DNN layer may be elements of the DNN layer's tensor. A tensor is a data structure having multiple elements across one or more dimensions. Exemplary tensors include vectors, which are one-dimensional tensors, and matrices, which are two-dimensional tensors. Three-dimensional tensors and even higher-dimensional tensors may also exist. A DNN layer may have an input tensor (also called an "input feature map (IFM)") containing one or more input activation values ​​(also called "input elements"), and a weight tensor containing one or more weights. Weights are elements within the weight tensor. The weight tensor of a convolution may be a kernel, a filter, or a group of filters. The combination of input activation values ​​and weights is sometimes called the input data for a DNN layer. The output data for a DNN layer may be an output tensor (also called an "output feature map (OFM)") containing one or more output activation values ​​(also called "output elements").

[0028] An input tensor may contain one or more input channels. For example, a three-dimensional input tensor may contain input channels arranged along the Z-axis, each of which may contain a two-dimensional matrix in the XY plane. For each pair of (X,Y) coordinates, the input tensor may contain a sequence of data elements, each in a different input channel. An output tensor may contain one or more output channels. For example, a three-dimensional output tensor may contain output channels arranged along the Z-axis, each of which may contain a two-dimensional matrix in the XY plane. For each pair of (X,Y) coordinates, the output tensor may contain a sequence of data elements, each in a different output channel.

[0029] The input data for a DNN layer may be sparse data, meaning that at least one element in the input tensor or weight tensor has a zero value. For example, some weights determined during the training phase may have zero values. Sparse weights can be passed through a nonlinear activation function, such as a normalized linear activation function (ReLU), so that the activation values ​​become sparse in later layers of the DNN. Furthermore, quantizing the network for inference on edge devices may result in a high proportion of zeros in the weights and activation values. Zero-value weights and activation values ​​(collectively called zero-value input data) do not contribute to the output of a channel-inseparable operation. A channel-inseparable operation is a deep learning operation in which a single data element in an output tensor is computed based on data elements from multiple input channels (e.g., data elements from all input channels of the input tensor). For example, all data elements in the input tensor with the same (X,Y) coordinates are used to compute a single data element in the output tensor. An example of a channel-inseparable operation is standard convolution, where the partial sum of all input channels is accumulated into a single data element during the MAC operation.

[0030] Sparse DNN accelerators can accelerate the non-channel separable computation layers that form the backbone of many DNNs by leveraging the sparsity (i.e., the presence of zero values) in the input data of these layers. Sparse DNN accelerators can achieve significant sparsity acceleration by skipping zeros in computation. In addition, these DNN accelerators can achieve memory traffic reduction by performing zero-value compression, leveraging the underlying data sparsity. Zero-value compression prevents zero-value input data from being stored or processed. Therefore, less data is loaded from memory and processed during computation. This results in significant memory and computational energy savings, leading to a significant performance improvement in sparse DNN accelerators for non-channel separable computation layers.

[0031] However, most DNNs may also include other types of layers that have channel-separable operations. Channel-separable operations are deep learning operations in which a single data element in an output tensor is computed based on one or more data elements from a subset of input channels in an input tensor. The subset may include one or more input channels (but not all input channels) in the input tensor. Examples of channel-separable operations include depth-based convolution, group convolution (e.g., MobileNet, DenseNet, ResNet, ResNext, etc.), element-wise addition, element-wise multiplication, and channel-separable pooling operations. In the case of channel-separable operations, zero-value input data can contribute to the output, and avoiding zero-value input data can impair the precision of the output. Therefore, these layers cannot leverage the underlying sparsity of the data for acceleration and require that the input data be stored and loaded in a decompressed format, i.e., that zero-value elements are stored and loaded.

[0032] As a result, during the execution of a DNN with channel-separable operations, currently available DNN accelerators must switch between compressed and decompressed modes for the input data storage based on the type of deep learning operations in the next layer. For DNNs with a large number of layers that have channel-separable operations, this can have a significant negative impact on overall energy consumption and degrade the performance per watt of these accelerators. In addition, the complexity of determining which nodes should be in compressed mode and which should be in decompressed mode can make it difficult to adopt sparsity acceleration.

[0033] Embodiments of the present disclosure may improve at least some of the challenges and problems described above by dynamically decompressing sparse data compressed for channel-separable operations in a DNN. Dynamic decompression can facilitate memory and computational energy savings based on sparsity without impairing the accuracy of the output of channel-separable operations.

[0034] In various embodiments of this disclosure, a DNN accelerator may include one or more computational blocks that perform various layers within a DNN. A DNN layer may have one or more deep learning operations, such as convolution, pooling, element-wise operations, linear operations, and nonlinear operations. A computational block may include memory, a datastore, a PE array, a decompression module, and a compression module. Memory may store input and output data for one or more deep learning operations performed by the computational block. Memory may be on-chip memory, such as SRAM (Static Random Access Memory). A datastore may function as a buffer, and input data may be loaded from memory into the datastore before being sent to the PE array for computation. Output data generated by the PE array may be stored in the datastore before being loaded into memory.

[0035] The computation block can perform a variety of deep learning operations, including both channel-separable and channel-inseparable operations. For channel-inseparable operations, the computation block can facilitate memory savings by storing compressed input data in a data store. Compressed input data includes non-zero elements and excludes zero elements. The decompression module can dynamically decompress the compressed input data by inserting zero values ​​into the compressed input data and provide the decompressed input data to the PE array for computation. For example, the decompression module may determine whether an input operand contains zero data points based on a sparsity bitmap of the input operand. The sparsity bitmap contains a sequence of bits, each indicating whether each element of the input operand has a zero or non-zero value. A zero bit may indicate that the corresponding element has a zero value, and a one bit may indicate that the corresponding element has a non-zero value. After determining that an input operand contains zero data points, the decompression module inserts the zero data points into the compressed data. The decompression module may determine the location to insert zero-value data points based on the position of the corresponding bits in the sparsity bitmap.

[0036] In some embodiments, dynamic decompression may include dynamic densification. The decompression module may modify one or more bits in the sparsity bitmap of the input operand so that all bits in the sparsity bitmap become 1. In this way, all elements in the decompressed data, including zero-value and non-zero-value elements, are treated as high-density data and processed by the PE array.

[0037] The PE array computes output operands based on the decompressed data. The output operands may be stored in a datastore. In some embodiments (e.g., embodiments in which the output operands contain at least one zero-value element), the compute block can further facilitate memory savings by compressing the output operands before loading them into memory. For example, a compression module may generate a sparsity bitmap of the output operands to prevent zero-value elements in the output operands from being written to memory. In this way, less data is stored in memory. The output operands may be used as input data (or part of input data) for another deep learning operation, such as a deep learning operation in the next layer of the DNN.

[0038] The dynamic decompression in this disclosure overcomes the requirement of storing zero-value input elements in memory and loading zero-value input elements from memory for channel separable operations, despite the interdependence between sparsity acceleration logic and sparse compression of data in sparse DNN accelerators. Compared to currently available DNN accelerators that typically store input data in a decompressed format, the DNN accelerator in this disclosure can save memory storage and bandwidth and achieve higher performance per watt.

[0039] For explanatory purposes and to provide a complete understanding of the exemplary embodiments, certain numbers, materials, and configurations are described. However, it will be apparent to those skilled in the art that this disclosure may be carried out without specific details and / or only in some of the embodiments described herein. In other examples, well-known features are omitted or simplified so as not to obscure the exemplary embodiments.

[0040] Furthermore, reference is made to the accompanying drawings, which illustrate embodiments that may be implemented and which form part of this specification. It should be understood that other embodiments may be utilized and structural or logical modifications may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed as restrictive.

[0041] Various operations may be described as multiple separate actions or operations in the manner most useful for understanding the claimed subject matter. However, the order of the description should not be interpreted as meaning that these operations necessarily depend on the order. In particular, these operations do not have to be performed in the order presented. The described operations may be performed in a different order than in the described embodiments. In additional embodiments, various additional operations may be performed, or the described operations may be omitted.

[0042] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used in relation to a measurement range, the term "between" includes both ends of the measurement range.

[0043] The descriptions use phrases such as “in one embodiment” or “in one embodiment,” each of which may refer to one or more of the same or different embodiments. Terms such as “comprising,” “including,” and “having” as used in reference to embodiments of this disclosure are synonymous. This disclosure may use descriptions based on viewpoints such as “top,” “bottom,” “summit,” “bottom,” and “side,” but these terms are merely for the purpose of facilitating description and do not imply any desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” and “third” to describe common objects simply indicates that different instances of similar objects are being referred to, and is not intended to mean that the objects thus described must be in a given sequence in time, space, in ranking, or in any other way.

[0044] In the following detailed description, various aspects of the exemplary embodiments will be described using terminology commonly used by those skilled in the art to convey the content of their research to others skilled in the art.

[0045] The terms “substantially,” “close,” “approximately,” “near,” and “about” generally refer to being within + / - 20% of a target value based on a specific value input operand, as described herein or as known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to being within + / - 5 to 20% of a target value based on a specific value input operand, as described herein or as known in the art.

[0046] In addition, the terms “comprise,” “comprising,” “include,” “including,” “have,” “having,” or any other variations thereof are intended to encompass non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that includes a list of elements is not necessarily limited to those elements alone, but may include other elements that are not explicitly listed or that are specific to such a method, process, device, or DNN accelerator. Also, the term “or” refers to an inclusive “or” and not an exclusive “or.”

[0047] Each of the systems, methods, and devices disclosed herein has several innovative aspects, and no single one alone is solely responsible for all the desirable attributes disclosed herein. Details of one or more embodiments of the subject matter described herein are given in the following description and accompanying drawings.

[0048] Exemplary DNN Figure 1 shows an exemplary DNN100 in various embodiments. For illustrative purposes, the DNN100 in Figure 1 is a CNN. In other embodiments, the DNN100 may be a different type of DNN. The DNN100 is trained to receive an image and output the classification of objects in the image. In the embodiment of Figure 1, the DNN100 receives an input image 105 containing objects 115, 125, and 135. The DNN100 includes a sequence of layers, each containing a plurality of convolutional layers 110 (each referred to as “convolutional layer 110”), a plurality of pooling layers 120 (each referred to as “pooling layer 120”), and a plurality of fully connected layers 130 (each referred to as “fully connected layer 130”). In other embodiments, the DNN100 may contain fewer, more, or different layers. The layers of the DNN100 have deep learning operations such as convolution (e.g., standard convolution, depth-based convolution, point-based convolution, group convolution), deconvolution, pooling operations, element-based operations (e.g., element-based addition, element-based multiplication), linear operations, nonlinear operations, other types of deep learning operations, or any combination thereof.

[0049] The convolutional layer 110 summarizes the presence of features in the input image 105. The convolutional layer 110 functions as a feature extractor. The first layer of the DNN 100 is the convolutional layer 110. In one example, the convolutional layer 110 performs convolution on the input tensor 140 (also called IFM140) and the filter 150. As shown in Figure 1, the IFM140 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM140 contains three input channels, each represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix contains seven input elements (also called input points) in each row and seven input elements in each column. The filter 150 is represented by a 3x3x3 three-dimensional matrix. The filter 150 contains three kernels, each of which may correspond to a different input channel of the IFM140. A kernel is a 2D matrix of weights, with the weights arranged in the columns and rows. A kernel can be smaller than an IFM. In the embodiment shown in Figure 1, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel contains three weights in each row and three weights in each column. The weights can be initialized and updated by backpropagation using gradient descent. The magnitude of the weights can indicate the importance of the filter 150 in extracting features from the IFM 140.

[0050] The convolution involves a MAC operation using the input elements in IFM140 and the weights in filter 150. The convolution may be a standard convolution 163 or a unit depth convolution 183. In the standard convolution 163, the entire filter 150 slides across IFM140. All input channels are combined to produce an output tensor 160 (also called OFM160). OFM160 is represented by a 5x5 2D matrix. The 5x5 2D matrix contains five output elements (also called output points) in each row and five output elements in each column. For illustrative purposes, the standard convolution includes one filter in the embodiment shown in Figure 1. In embodiments with multiple filters, the standard convolution may produce multiple output channels in OFM160.

[0051] The multiplication applied between a kernel-sized patch and a kernel in IFM140 may also be an inner product. The inner product is an element-wise multiplication between a kernel-sized patch and the corresponding kernel in IFM140, which is then summed up, always yielding a single value. Because a single value is obtained, this operation is often called a "scalar product." Using kernels smaller than IFM140 is intentional because it allows the same kernel (set of weights) to be multiplied multiple times by IFM140 at different points on the IFM140. Specifically, the kernel is applied systematically from left to right and top to bottom to each overlapping portion or kernel-sized patch of IFM140. The result of multiplying a kernel by IFM140 once is a single value. When a kernel is applied to IFM140 multiple times, the result of the multiplication is a 2D matrix of output elements. Thus, the 2D output matrix from the standard convolution 163 (i.e., OFM160) is called OFM.

[0052] In depth-unit convolution 183, the input channels are not combined. Rather, MAC operations are performed on each individual input channel and each individual kernel to generate output channels. As shown in Figure 1, depth-unit convolution 183 generates depth-unit output tensor 180. The depth-unit output tensor 180 is represented by a 5x5x3 3D ​​matrix. The depth-unit output tensor 180 contains three output channels, each represented by a 5x5 2D matrix. The 5x5 2D matrix contains five output elements in each row and five output elements in each column. Each output channel is the result of MAC operations on one input channel of IFM 140 and the kernel of filter 150. For example, the first output channel (patterned with dots) is the result of a MAC operation on the first input channel (patterned with dots) and the first kernel (patterned with dots); the second output channel (patterned with horizontal strips) is the result of a MAC operation on the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips); and the third output channel (patterned with diagonal stripes) is the result of a MAC operation on the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a unit-depth convolution, the number of input channels is equal to the number of output channels, and each output channel corresponds to a different input channel. The input and output channels are collectively called unit-depth channels. After the unit-depth convolution, a unit-point convolution 193 is then performed on the unit-depth output tensor 180 and the 1×1×3 tensor 190 to generate OFM 160.

[0053] Subsequently, OFM160 is passed to the next layer in the sequence. In some embodiments, OFM160 is passed through an activation function. An exemplary activation function is ReLU. ReLU is a computation that directly returns the value provided as input, or returns zero if the input is zero or lower. The convolutional layer 110 may take several images as input and compute the convolution of each of them with each of the kernels. This process can be repeated several times. For example, OFM160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 following the convolutional layer 110 that produced OFM160 in the sequence). The subsequent convolutional layer 110 performs a convolution on OFM160 with a new kernel and generates a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelized again by further subsequent convolutional layers 110, and so on.

[0054] In some embodiments, the convolutional layer 110 has four hyperparameters, namely, the number of kernels, the size of the kernel F (e.g., the kernel has dimensions of F × F × D pixels), the S steps in which the window corresponding to the kernel is dragged on the image (e.g., 1 step means moving the window one pixel at a time), and zero padding P (e.g., adding a black contour of P pixel thickness to the input image of the convolutional layer 110). The convolutional layer 110 can perform various types of convolution, such as 2D convolution, extended convolution or atlas convolution, spatially separable convolution, depth unit separable convolution, and transposed convolution. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0055] The pooling layer 120 downsamples the feature map generated by the convolutional layer, for example, by summarizing the presence of features within a patch of the feature map. The pooling layer 120 is placed between two convolutional layers 110, namely, a preceding convolutional layer 110 (a convolutional layer 110 that precedes the pooling layer 120 in the layer sequence) and a succeeding convolutional layer 110 (a convolutional layer 110 that follows the pooling layer 120 in the layer sequence). In some embodiments, the pooling layer 120 is added after the convolutional layer 110, for example, after an activation function (e.g., ReLU) has been applied to OFM 160.

[0056] The pooling layer 120 receives the feature map generated by the preceding convolutional layer 110 and applies a pooling operation to the feature map. The pooling operation reduces the size of the feature map while preserving its important properties. Thus, the pooling operation improves the efficiency of the DNN and avoids overfitting. The pooling layer 120 can perform the pooling operation by mean pooling (calculating the average value of each patch on the feature map), max pooling (calculating the maximum value of each patch on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of 2 pixels, and as a result, the pooling operation reduces the size of the feature map by a factor of 2, for example, reducing the number of pixels or values ​​in the feature map to 1 / 4 of its size. In one example, a pooling layer 120 applied to a 6x6 feature map results in a 3x3 output pooled feature map. The output of the pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, the pooling layer 120 manipulates each feature map separately to create a new set of the same number of pooled feature maps.

[0057] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 may or may not be a convolutional layer. The fully connected layer 130 accepts an input operand. The input operand defines the outputs of the convolutional layer 110 and the pooling layer 120 and includes the value of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and activation function to the input operand to generate a vector. The vector may contain as many elements as there are classes, where element i represents the probability that the image belongs to class i. Thus, each element is between 0 and 1, and the sum of all is 1. These probabilities are computed in the last fully connected layer 130 by using either the logistic function (binary classification) or the softmax function (multiclass classification) as the activation function.

[0058] In some embodiments, the fully connected layer 130 classifies the input image 105 and returns an operand of size N, where N is the number of classes in the image classification problem. In the embodiment of Figure 1, N is equal to 3 because the input image has three objects 115, 125, and 135. Each element of the operand represents the probability that the input image 105 belongs to one of the classes. To calculate the probabilities, the fully connected layer 130 multiplies each input element by a weight, creates a sum, and then applies an activation function (e.g., logistic for N=2, softmax for N>2). This is equivalent to multiplying the input operands by a matrix containing the weights. In one example, the vector contains three probabilities: a first probability that object 115 is a tree, a second probability that object 125 is a car, and a third probability that object 135 is a person. In other embodiments, where the input image 105 contains different objects or a different number of objects, the individual values ​​may differ.

[0059] Exemplary DNN accelerator Figure 2 is a block diagram of the DNN accelerator 200 in various embodiments. The DNN accelerator 200 can execute a DNN, for example, the DNN 100 in Figure 1. The DNN accelerator 200 includes memory 210, a DMA (Direct Memory Access) engine 220, and a compute block 230 (each referred to as “Compute Block 230”). In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 200. For example, the DNN accelerator 200 may include more than one memory 210 or more than one DMA engine 220. As another example, the DNN accelerator 200 may include a single compute block 230. Furthermore, functions attributable to the components of the DNN accelerator 200 may be achieved by different components or different systems included in the DNN accelerator 200.

[0060] Memory 210 stores data associated with the DNN executed by the DNN accelerator 200. Memory 210 may store data processed or computed by the compute blocks 230. For example, memory 210 may store the internal parameters of the DNN (e.g., weights). As another example, memory 210 may store input and output data for deep learning operations performed by one or more of the compute blocks 230. Input data may be sent from memory 210 to the compute blocks 230 via the DMA engine 220. Output data may be sent from the compute blocks 230 to memory 210 via the DMA engine 220. In some embodiments, memory 210 may be the main memory of the DNN accelerator 200. Memory 210 may include one or more DRAMs (Dynamic Random Access Memory).

[0061] The DMA engine 220 facilitates data transfer between memory 210 and the local memory of the compute block 230. For example, the DMA engine 220 can read data from memory 210 and write data to the local memory of the compute block 230. As another example, the DMA engine 220 can read data from the local memory of the compute block 230 and write data to memory 210. The DMA engine 220 provides DMA functionality that allows the compute block 230 to initiate data transfer between memory 210 and its local memory and to perform other operations while the data transfer is being performed. In some embodiments, the DMA engine 220 may read a tensor from memory 210 and modify the tensor in a manner optimized for the compute block 230 before writing the tensor to the local memory of the compute block 230.

[0062] Computation block 230 performs calculations for deep learning operations. Computation block 230 may perform deep learning operations on the DNN layer. Deep learning operations may be non-channel separable operations or channel separable operations. Examples of deep learning operations may include standard convolutions (e.g., standard convolution 163 in Figure 1, standard convolution 900 in Figure 9, etc.), depth-unit convolutions (e.g., depth-unit convolution 183 in Figure 1, depth-unit convolution 1000 in Figure 10, depth convolution in Figure 12, etc.), point-unit convolutions (e.g., point-unit convolution 193 in Figure 1, etc.), group convolutions (e.g., group convolution 1100 in Figure 11, etc.), deconvolution, pooling operations (e.g., channel-separable pooling operations in Figures 13 and 14, etc.), element-unit operations (e.g., element-unit addition in Figures 15 and 16, element-unit multiplication in Figure 17, etc.), linear operations, nonlinear operations, other types of deep learning operations, or any combination thereof.

[0063] In some embodiments, multiple computation blocks 230 may be performed in parallel to execute deep learning operations. For example, each computation block 230 may process different portions of the input data for a deep learning operation and generate different portions of the output data for that deep learning operation. In some embodiments, the output of a deep learning operation performed by a computation block 230 may be used as input for another deep learning operation performed by the same computation block 230 or one or more other computation blocks 230.

[0064] Computation block 230 can perform both channel-inseparable and channel-separable operations. Computation block 230 may also be performed with dynamic decompression, so that regardless of whether the deep learning operation is channel-inseparable or not, computation block 230 can store the compressed data in its local memory and load the compressed data from local memory into a buffer. The compressed data may be generated by removing zero values ​​from the input or output data of the deep learning operation. Specific embodiments of computation block 230 are described below in reference to Figure 3.

[0065] Figure 3 is a block diagram of the compute block 300 in various embodiments. The compute block 300 may be an example of the compute block 230 in Figure 2. As shown in Figure 3, the compute block 300 includes local memory 310, data stores 320 and 350, a decompression module 330, a PE array 340, and a compression module 360. In other embodiments, alternative configurations, different or additional components may be included in the compute block 300. For example, data stores 320 and 350 may be implemented as a single data store. As another example, the compute block 300 may include more than one local memory 310, PE array 340, decompression module 330, or compression module 360. Furthermore, functions attributable to the components of the compute block 300 may be achieved by different components included in the compute block 300, other components of the DNN accelerator 200, or different systems.

[0066] The local memory 310 is local to the compute block 300. In the embodiment shown in Figure 3, the local memory 310 is located within the compute block 300. In other embodiments, the local memory 310 may be located outside the compute block 300. The local memory 310 and the compute block 300 can be implemented on the same chip. In some embodiments, the local memory 310 includes one or more SRAMs. The local memory 310 can be byte-addressable, with each memory address specifying a single byte (8 bits) of storage. In some embodiments, the local memory 310 may include banks, each bank having a fixed number of bytes, such as 22 or 64.

[0067] Local memory 310 may store input data (e.g., input tensors, filters, etc.) and output data (e.g., output tensors, etc.) of deep learning operations performed by the computation block 300. Tensors may include elements arranged in vectors, 2D matrices, or 3D matrices. In embodiments where the tensor is a 3D matrix, the position of an element in the tensor may be represented by (X,Y,Z) coordinates. The Z axis of the 3D matrix may correspond to the channels of the DNN layer, and the Z coordinate of an element may indicate which channel the element is located in. The data stored in local memory 310 may be in a compressed format. For example, in the case of a tensor containing one or more non-zero elements and one or more zero elements, local memory 310 may store one or more non-zero elements and not one or more zero elements. Local memory 310 may also store other data associated with the deep learning operations performed by the computation block 300, such as sparsity bitmaps that can be used to accelerate the deep learning operations.

[0068] A sparsity bitmap can be associated with operands of a deep learning operation. An operand may be at least a portion of the tensor of a DNN layer. In some embodiments, elements within an operand may have the same Z coordinate, i.e., elements are in the same channel. For example, considering a convolutional layer, the input operand may include one or more input activation values ​​in the convolution's input tensor, the weight operand may include one or more weights in the convolution's filter, and the output operand may include one or more output activation values ​​in the convolution's output tensor. The input operand or weight operand may be processed by a PE array 340 (e.g., one or more PEs in the PE array 340) to compute the output operand. The sparsity bitmap of an operand may contain one or more bits, each corresponding to each element in the operand and indicating whether each element is zero or non-zero. In one example, a zero bit indicates that the corresponding element is zero, and a one bit indicates that the corresponding element is non-zero.

[0069] The data store 320 stores data used by the PE array 340 to perform deep learning operations. The data store 320 can function as one or more buffers between the local memory 310 and the PE array 340. Data in the data store 320 may be loaded from the local memory 310 and sent to the PE array 340 for computation. In some embodiments, the data store 320 includes one or more data banks. A data bank may include a sequence of storage units. A storage unit may store a portion of the data in the data bank. In some embodiments, a storage unit may have a fixed storage size, for example, 32, 64, or 126 bytes. The number of storage units in the data store 320 can be 8, 16, 32, 64, etc.

[0070] A storage unit may act as a buffer for one PE at a time. Data in a storage unit may be supplied to one or more PEs for a computation cycle of the PEs. For different computation cycles, a storage unit may be a buffer for different PEs. Data in a storage unit may be supplied to the PE array 340 via MAC lanes. A MAC lane is a path for loading data into the PE array 340 or a portion of the PE array 340, such as a PE column within the PE array 340. MAC lanes are sometimes also called data transmission lanes or data load lanes. A PE array 340 (or a PE column) may have multiple MAC lanes. The load bandwidth of a PE array 340 (or a PE column) is the aggregate of the load bandwidths of all MAC lanes associated with the PE array 340 (or a PE column). In an example where a PE array 340 (or a PE column within a PE array 340) has four MAC lanes, and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total load bandwidth of 64 bytes. N MAC lanes (where N is an integer) can be used to supply data to N PEs simultaneously. In some embodiments (for example, embodiments where all PE columns have separate MAC lanes), data in a storage unit may be broadcast to multiple PE columns via the MAC lanes of these PE columns. In embodiments where all PE columns have more than one separate MAC lanes, data in more than one storage unit can be broadcast to multiple PE columns. In an example where each PE column has four MAC lanes, data in four storage units can be broadcast to multiple PE columns.

[0071] In some embodiments, the data store 320 may store at least a portion of the input tensors or weight tensors of the DNN layer. The storage unit may store at least a portion of the operands (e.g., input operands or weight operands). The storage unit may also store a sparsity bitmap of the operands. In some embodiments (e.g., embodiments in which local memory 310 stores input data in a compressed format), the input data in the data store 320 is in a compressed format. For example, the data store 320 stores non-zero activation values ​​or weights, but zero activation values ​​or weights are not stored in the data store 320. The compressed data in the data store 320 can be decompressed by the decompression module 330 before being supplied to the PE array 340. Specific embodiments of the data store 320 are described below in reference to Figure 4.

[0072] The decompression module 330 decompresses data from the data store 320. In some embodiments, the decompression module 330 may receive compressed data from the data store 320, for example, from the storage units of the data store 320. The compressed data may be one or more non-zero elements in the operands of a deep learning operation (e.g., input operands or weight operands). In embodiments where the operands include one or more zero-value elements, the compressed data does not include any zero-value elements. The decompression module 330 may also receive a sparsity bitmap of the operands from the data store 320.

[0073] In some embodiments, the decompression module 330 determines whether the operand contains zero-value elements not stored in the data store 320, based on the sparsity bitmap of the operand. The decompression module 330 may determine whether there are zeros in the sparsity bitmap. In response to determining that there are zero-value bits in the sparsity bitmap, the decompression module 330 determines that there are zero-value elements in the operand. The decompression module 330 may determine that the operand contains one or more zero-value elements. After such a determination, the decompression module 330 may insert one or more zero-value elements into the compressed data, resulting in decompressed data. The decompressed data may contain all elements in the operand, including both zero-value and non-zero-value elements.

[0074] In some embodiments, elements within an operand may be arranged in a sequence, for example, in a sequence of channels within a tensor. Bits in a sparsity map may also be arranged in a sequence. The positions of bits in a sparsity map may coincide with (for example, be the same as) the positions of corresponding elements in an operand. The decompression module 330 may determine the positions to insert zero-value elements into the compressed data based on the positions of corresponding bits in the sparsity bitmap. The corresponding bits are zero-value bits, and based on this, the decompression module 330 determines that the operand contains zero-value elements. The positions of zero-value elements in the decompression data (or within the operand) may be the same as the positions of corresponding bits in the sparsity bitmap.

[0075] Although not shown in Figure 3, the decompression module 330 may include or otherwise be associated with one or more sparsity accelerators. The sparsity accelerators can accelerate calculations within the PE array 340 (e.g., calculations for non-separable channel operations) based on the sparsity in the input data. In some embodiments, the sparsity accelerators can accelerate calculations within a single PE. In other embodiments, the sparsity accelerators can accelerate calculations across multiple PEs, e.g., one or more PE columns or the entire PE array 340.

[0076] In some embodiments (for example, an embodiment in which the computation block 300 performs a convolutional layer), the decompression module 330 may generate input operands and weight operands by decompressing data from the data store 320. The input operands may be part of the input tensor of the convolution. The input operands include a sequence of input elements. The activation values ​​may come from different input channels. For example, each activation value comes from a different input channel than all other activation values ​​in the input operand. The input operands are associated with an input bitmap that may be received by the decompression module 330 from the data store 320. The input bitmap may indicate the positions of non-zero activation values ​​in the input operand. The input bitmap may include a sequence of bits, each corresponding to a respective activation value in the input operand. The positions of the bits in the input bitmap may coincide with the positions of the corresponding activation values ​​in the input operand. The bits in the input bitmap may be zero or one. A zero-value bit indicates that the value of the corresponding activation value is zero, and a one-value bit indicates that the value of the corresponding activation value is non-zero. In some embodiments, the input bitmap may be generated during the execution of another DNN layer, for example, a layer placed before a convolutional layer in the DNN.

[0077] A weight operand may be part of the convolution kernel. A weight operand contains a sequence of weights. The weight values ​​are determined by training the DNN. The weights in a weight operand may come from different input channels. For example, each weight may come from a different input channel than all the other weights in the weight operand. A weight operand is associated with a weight bitmap that can be received from the data store 320 by the decompression module 330. The weight bitmap can indicate the positions of non-zero weights in the weight operand. The weight bitmap may contain a sequence of bits, each corresponding to a specific weight in the weight operand. The positions of the bits in the weight bitmap may coincide with the positions of the corresponding weights in the weight operand. The bits in the weight bitmap can be zero or one. A zero-value bit indicates that the value of the corresponding weight is zero, and a one-value bit indicates that the value of the corresponding weight is non-zero.

[0078] A sparsity accelerator can generate a combined bitmap for MAC operations based on an input bitmap and a weight bitmap. In some embodiments, the sparsity accelerator generates a combined sparsity bitmap by performing one or more AND operations on the input bitmap and the weight bitmap. Each bit in the combined sparsity bitmap is the result of an AND operation on the bits in the input bitmap and the bits in the weight bitmap, i.e., the product of the bits in the input bitmap and the bits in the weight bitmap. The position of a bit in the combined sparsity bitmap coincides with (e.g., is the same as) the position of a bit in the input bitmap and the position of a bit in the weight bitmap. Bits in the combined bitmap correspond to activation value and weight pairs (activation value / weight pairs). A zero bit in the combined bitmap indicates that at least one of the activation value and weight in the pair is zero. A one bit in the combined bitmap indicates that both the activation value and weight in the pair are non-zero.

[0079] A sparsity accelerator can provide activation / weight pairs to the PE based on the combined bitmap. For example, a sparsity accelerator can identify activation / weight pairs corresponding to 1s in the combined bitmap and transfer these activation / weight pairs to the PE. The sparsity accelerator may skip other activation / weight pairs because they do not contribute to the result of the MAC operation in a non-separable channel convolution (e.g., standard convolution). The total number of 1s in the combined bitmap may be equal to the total number of activation / weight pairs that will be computed by the PE. By skipping activation / weight pairs corresponding to zero bits in the combined bitmap, the PE's computation becomes faster compared to a PE that computes all activation / weight pairs in the input operands and weight operands.

[0080] However, such sparsity acceleration may not be applicable to channel-separable convolutions (e.g., unit depth convolution, group convolution, etc.) because activation / weight pairs corresponding to zeros in the combined bitmap may contribute to the MAC operation result, and skipping these activation / weight pairs may impair the accuracy of the result. To avoid the risks of channel-separable operations, the decompression module 330 may update the operand's sparsity bitmap. The updated sparsity bitmap may contain 1s and not zeros. In embodiments where the operand is an input operand or a weight operand, the decompression module 330 may update the input sparsity bitmap, the weight sparsity bitmap, the combined sparsity bitmap, or any combination thereof. By updating the sparsity bitmap, the decompression module 330 can increase the density of zero-value elements, for example, by changing the corresponding bits in the sparsity bitmap from zero to one, so that the sparsity accelerator does not prevent zero-value elements from being sent to the PE array 340.

[0081] In some embodiments, the decompression module 330 may have a decompression mode and a bypass mode. In decompression mode, the decompression module 330 may decompress data from the data store 320 and send the decompressed data to the PE array 340. In bypass mode, the decompression module 330 may send data from the data store 320 to the PE array 340 without decompression so that the PE array 340 receives the compressed data. For example, in embodiments where the compute block 300 performs non-channel separable calculations, the decompression module 330 may be set to decompression mode. For example, in embodiments where the compute block 300 performs channel separable calculations, the decompression module 330 may be set to bypass mode.

[0082] The decompression module 330 can be implemented in the data store 320. For example, the decompression module 330 can be implemented in a storage unit of the data store 320 and decompress the data in the storage unit before the data is supplied to the PE array 340. Although Figure 3 shows one decompression module 330, the compute block 300 may contain more than one decompression module 330. For example, the compute block 300 may contain a separate decompression module 330 for each storage unit in the data store 320. Decompression by the decompression module 330 can be dynamic. For example, decompression is performed when data is loaded from the data store 320 to the PE array 340. In such dynamic decompression, zero-value elements in the decompressed data are not stored in the data store 320. Rather, zero-value elements can be provided directly to the PE array 340 after being generated by the decompression module 330. Thus, decompression does not impair memory savings in the data store 320 or local memory 310.

[0083] The PE array 340 performs calculations for performing deep learning operations, including channel-separable and channel-inseparable operations. The PE array 340 may include PEs arranged in columns, or PEs arranged in columns and rows. In some embodiments, a PE includes one or more multipliers for performing multiplication. A PE may also include one or more adders for performing accumulation. Columns of a PE are called PE columns. As described above, a PE column may be associated with one or more MAC lanes for receiving data from the data store 320. In some embodiments, a PE may perform multiple rounds of calculations (e.g., MAC operations) for deep learning operations. Data (activation values, weights, or both) may be reused within a single round, for example, across different multipliers in a PE, or across different rounds of MAC operations. Specific aspects of the PE array 340 or components within a PE are described below in reference to Figures 12-19.

[0084] The PE array 340 generates output data through computations performed in the PE array 340. The output data may be at least a portion of the output tensor of the deep learning operations in the DNN layer. The output data may also be input data. The output data may include one or more output operands. In some embodiments, the output is sparse data in a decompression format and may include one or more zero-value elements. A sparsity bitmap of the output operands may also be stored in the data store 350. In some embodiments, the sparsity bitmap may be generated by the decompression module 330 or a sparsity accelerator, for example, before the output operands are computed. In other embodiments, the sparsity bitmap may be generated by the compression module 360 ​​after the output operands have been computed.

[0085] The compression module 360 ​​compresses the output data in the data store 350. In some embodiments, the compression module 360 ​​generates compressed data by removing zero values ​​from the output data. The compression module 360 ​​may also generate one or more sparsity bitmaps for the output data. The sparsity bitmap may contain a sequence of bits, each indicating whether each element in the output operand is zero or not. The compressed data and sparsity bitmaps may be written to local memory 310. In some embodiments, the compressed data and sparsity bitmaps may be used to perform another deep learning operation, for example, a deep learning operation of the next DNN layer which can use the compressed data as input data.

[0086] Exemplary Dynamic Decompression Figure 4 shows a data store 400 implemented with a decompression module 420 in various embodiments. The decompression module 420 is referred to individually as decompression module 420. Decompression module 420 may be an embodiment of decompression module 330 in Figure 3. Data store 400 may be an embodiment of data store 320 in Figure 3. The data store 400 includes a plurality of storage units 410 (each referred to individually as “storage unit 410”). Each storage unit 410 is coupled with a decompression module 420. The decompression module 420 may be located in or adjacent to a storage unit 410. In other embodiments, the plurality of storage units 410 may share one decompression module 420. Figure 4 also shows a PE array 401 including PE columns 405A to 405N (collectively referred to as “PE columns 405” or “PE column 405”). Each PE column 405 includes multiple PEs 430 (each referred to as "PE430") and four data transfer lanes 440 (each referred to as "data transfer lane 440"). In other embodiments, the PE array 401 may include a different number of PE columns 405, or each PE column may have a different number of data transfer lanes 440. The PE array 401 may be an embodiment of the PE array 340 in Figure 3.

[0087] In some embodiments, the storage unit 410 may load compressed data and sparsity bitmaps from memory, such as local memory 310. The compressed data includes non-zero elements of the operands but does not include zero elements. In one example, the storage unit 410 may store compressed data and its sparsity bitmap at a time for one operand. After the compressed data and sparsity bitmap have been fetched into the PE array 401, the storage unit 410 may store compressed data and its sparsity bitmap for a new operand. The storage unit 410 may have a storage capacity greater than or equal to the sum of the storage size of the operand and the storage size of the sparsity bitmap. An operand may have a predetermined storage size, for example, a predetermined number of bytes. A sparsity bitmap may have a predetermined storage size, for example, a predetermined number of bits. The predetermined number of bytes or bits in an operand or sparsity bitmap may be, for example, 8, 16, 32, 64, 128, etc.

[0088] The compressed data in datastore 400 is distributed to PE array 401 for computation in PE 430. In the data distribution process, the decompression module 420 of storage unit 410 can form operands based on the compressed data and sparsity bitmap from storage unit 410. For example, the decompression module 420 may, for example, identify one or more zeros in the sparsity bitmap and then insert one or more zero-value elements into the compressed data. After insertion, the operand is regenerated and includes one or more zero-value elements and all non-zero-value elements stored in storage unit 410. The total number of elements in the operand may be equal to the number of bits in the sparsity bitmap. In some embodiments (e.g., embodiments where a sparsity accelerator is available to accelerate PE computation), the decompression module 420 may also set all bits in the sparsity bitmap to 1. In this way, all elements in the operand are considered high-density data by the sparsity accelerator and fetched into the PE column. The elements in the operand may correspond to different channels in the tensor of the DNN layer executed by PE array 401. For example, all elements may have the same (X,Y) coordinates, but their Z coordinates may be different.

[0089] In some embodiments (for example, embodiments in which the PE array 401 performs channel-inseparable operations), the decompression module 420 of the storage unit may be disabled. For example, the decompression module 420 may operate in bypass mode. When the decompression module 420 is disabled, the decompression module 420 does not decompress the compressed data or modify the sparsity bitmap from the storage unit 410. The compressed data is provided to the PE array 401, which processes non-zero elements in the compressed data, but skips zero elements in the operands.

[0090] Data (for example, decompressed data in embodiments where the decompression module 420 is enabled, or compressed data in embodiments where the decompression module 420 is disabled) is fetched to the PE array 401 via the data transfer lane 440. For illustrative purposes, each PE column 405 has four data transfer lanes 440 and can receive data from four storage units 410 in one cycle. As shown in Figure 4, each data transfer lane 440 can facilitate the transfer of data from different storage units 410. Also, data within the same four storage units 410 can be broadcast to some or all of the PE column 405 in one cycle. Figure 4 shows that data from the top four storage units 410 is fetched to all PE columns 405 simultaneously. Thus, data can be reused in all PE columns 405, improving the efficiency of the data store 400.

[0091] Figure 5 shows a dynamic decompression process in a decompression module comprising two extension units 510 and 520 according to various embodiments. Although not shown in Figure 5, the decompression module may include other components. The decompression module may also include one or more extension units. The decompression module in Figure 5 may be an embodiment of the decompression module 330 in Figure 3 or the decompression module 420 in Figure 4.

[0092] The extension units 510 and 520 receive a compressed data stream, represented as cdata in Figure 5. The compressed data stream contains non-zero elements of operands, such as input operands or weight operands. The compressed data stream can be formed by removing zero elements from the operands. The extension units 510 and 520 extend the compressed data stream by inserting non-zero elements into the compressed data stream and regenerating the operands based on a sparsity bitmap of the operands.

[0093] In the embodiment of Figure 5, the compressed data stream has a storage size limit of 128 bits (i.e., 16 bytes), represented by [127:0] in Figure 5. The data in the compressed data stream may be less than 16 bytes. The compressed data stream contains non-zero value elements of the operands. The sparsity bitmap contains 16 bits, which is represented by [15:0] in Figure 5. The sparsity bitmap is divided into two bitstreams, the first bitstream containing the first 8 bits in the sparsity bitmap (i.e., [7:0]), and the second bitstream containing the other 8 bits in the sparsity bitmap (i.e., [15:8]). Similarly, the compressed data stream is divided into two datastreams, the first datastream containing the first 64 bits (i.e., [63:0]), and the second datastream containing the other 64 bits (i.e., [127:64]). The extension unit 510 extends the first data stream based on the first bitstream and outputs a first decompressed data stream containing 64 bits represented by ucdata[63:0] in Figure 15. The extension unit 520 extends the second data stream based on the second bitstream and outputs a second decompressed data stream containing 64 bits represented by ucdata[127:64] in Figure 15. The first and second decompressed data streams constitute a decompressed data stream that may contain all elements of an operand, including zero and non-zero elements.

[0094] The extension unit 510 may also set all bits in the first bitstream, for example, by changing zeros in the first bitstream to ones. Similarly, the extension unit 520 may also set all bits in the second bitstream, for example, by changing zeros in the second bitstream to ones. The decompression module can output a new sparsity bitmap containing 16 ones.

[0095] Figure 6 shows the inputs 610 and outputs 620 of a decompression module for four load rounds according to various embodiments. The decompression module in Figure 6 may be an embodiment of the decompression module 330 in Figure 3, an embodiment of the decompression module 420 in Figure 4, or an embodiment of the decompression module in Figure 5.

[0096] The input 610 includes compressed data 630 and a sparsity bitmap 640. The output includes decompressed data 650 and a sparsity bitmap 660. The decompressed data 650 can be generated by the decompression module by inserting zeros into the compressed data 630 based on zeros in the sparsity bitmap 640. The sparsity bitmap 660 can be generated by the decompression module by setting all bits in the sparsity bitmap 640. The decompressed data 650 can be loaded into one or more PEs that perform deep learning operations using the decompressed data 650 as input data.

[0097] Figure 7 shows DNN700 performed without dynamic decompression in various embodiments. One embodiment of DNN700 may be DNN100 in Figure 1. DNN700 includes multiple layers with non-channel separable operations, which are represented by circles with N in Figure 7. DNN700 also includes multiple layers with channel separable operations, which are represented by circles with S in Figure 7. Figure 7 also shows the data transferred between layers. Rectangles with C represent compressed data streams that do not contain zero-value elements. Rectangles with D represent decompressed data streams that contain zero-value elements.

[0098] As shown in Figure 7, the data stream transferred to the layer with non-channel separable operations is in a compressed format. However, the data stream transferred to the layer with channel separable operations is in a decompressed format. Decompressed data requires more storage in memory compared to compressed data. Also, transferring decompressed data consumes more bandwidth.

[0099] Figure 8 shows DNN800 running with dynamic decompression in various embodiments. One embodiment of DNN800 may be DNN100 in Figure 1. DNN800 includes multiple layers with non-separable channel operations, which are represented by circles with N in Figure 8. DNN800 also includes multiple layers with separable channel operations, which are represented by circles with S in Figure 8. Each layer of DNN800 is the same as each layer of DNN700. Figure 8 also shows the data transferred between layers. A rectangle with C represents a compressed data stream that does not contain zero-value elements.

[0100] As shown in Figure 8, the data streams transferred to all layers in the DNN800 are in a compressed format. The DNN800 includes the same layers as the DNN700, but dynamic decompression allows the compressed data streams to be transferred to layers with channel separable operations. Unlike the embodiment in Figure 7, where the decompressed data stream is stored and loaded from memory, in the embodiment in Figure 8, the decompressed data stream is neither stored nor loaded from memory. Rather, the compressed data stream can be stored and loaded from memory, and then dynamically converted to a decompressed data stream when the compressed data stream is fetched from the buffer to the PE. Thus, dynamic decompression can save memory and bandwidth consumption.

[0101] Complex channel separation impossible calculation Figure 9 shows exemplary standard convolutions 900 in various embodiments. A standard convolution 900 may be a convolution in a convolutional layer of a DNN, for example, convolutional layer 110 in Figure 1. A standard convolution 900 may be an example of a standard convolution 163 in Figure 1. A standard convolution 900 is performed on input data including an input tensor 910 and a filter 920. The result of a standard convolution 900 is an output tensor 930. In some embodiments, a standard convolution 900 is performed by a DNN accelerator including one or more computation blocks. An example of a DNN accelerator may be the DNN accelerator 200 in Figure 2. An example of a computation block may be the computation block 300 in Figure 3.

[0102] In the embodiment shown in Figure 9, the input tensor 910 includes activation values ​​(also called "input activation values," "elements," or "input elements") arranged in a 3D matrix. The activation values ​​in the input tensor 910 are the data points in the input tensor 910. The input tensor 910 has a spatial size H in ×W in ×C in It has, and here H in is the height of the 3D matrix (i.e., the length along the Y-axis, which indicates the number of activation values ​​in the column of the 2D matrix for each input channel), W in is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activation values ​​in the row of the 2D matrix for each input channel), C in is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of input channels). For the purpose of simplification and illustration, H in and W in Both are 7, meaning the input tensor 910 contains a 7x7 2D matrix for each input channel. Each input element in the input tensor 910 can be represented by (X,Y,Z) coordinates. In other embodiments, the height, width, or depth of the input tensor 910 may be different.

[0103] Filter 920 contains weights arranged in a 3D matrix. The weight values ​​can be determined by training a DNN. Filter 920 has a spatial size Hf ×W f ×C in having, where H f is the height of the filter (i.e., the length along the Y-axis and indicates the number of weights within a column in each kernel), W f is the width of the filter (i.e., the length along the X-axis and indicates the number of weights in a row within each kernel), C in is the depth of the filter (i.e., the length along the Z-axis and indicates the number of input channels). For purposes of simplification and illustration, filter 920 has a 3×3 kernel for each input channel. In other embodiments, the height, width, or depth of filter 920 may be different. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix of each input channel within input tensor 910.

[0104] Activation values or weights may occupy 1 byte or more in memory. The number of bytes of activation values or weights may depend on the data format. For example, if the activation values or weights have an integer format (e.g., INT8), the activation value occupies 1 byte. If the activation values or weights have a floating-point format (e.g., FP16 or BF16), the activation values or weights occupy 2 bytes. Other data formats may be used for activation values or weights.

[0105] In standard convolution 900, filter 920 slides over input tensor 910 to generate a 2D matrix, i.e., output tensor 930. In the embodiment of FIG. 9, output tensor 930 has a 5×5 spatial size. Output tensor 930 has a single output channel. This is because standard convolution 900 is a non-channel-separable operation where input channels are combined into one output channel and are no longer separated.

[0106] As part of the standard convolution 900, a MAC operation can be performed on a 3x3x3 subtensor 915 (highlighted by a dot pattern in Figure 9) and a filter 920 within the input tensor 910. The subtensor 915 and the filter 920 have the same spatial size. The result of the MAC operation on the subtensor 915 and one filter 920 is the output activation value 935, which is highlighted by a dot pattern in Figure 9. In some embodiments (e.g., embodiments where the convolution is an integer convolution), the output activation value 935 may contain 8 bits, e.g., 1 byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), the output activation value 935 may contain more than one byte. For example, the output activation value 935 may contain 2 bytes.

[0107] After the output activation value 935 is generated, further MAC operations are performed to generate additional output activation values ​​until the entire output tensor 930 is generated. For example, the filter 920 may move along the X or Y axis on the input tensor 910, and MAC operations may be performed on the filter 920 and another subtensor within the input tensor 910 (the subtensor having the same size as the filter 920). The amount the filter 920 moves on the input tensor 910 during different computational rounds of the convolution is called the stride size of the convolution. The stride size may be 1 (i.e., the amount the filter 920 moves is one activation value), 2 (i.e., the amount the filter 920 moves is two activation values), and so on. The height and width of the output tensor 930 may be determined based on the stride size.

[0108] In some embodiments, MAC operations on a 3×3×3 subtensor (e.g., subtensor 915) and filter 920 may be performed by multiple PEs, such as PEs in a PE array 340. One or more PEs may accept an input operand (e.g., input operand 917 shown in Figure 9) and a weight operand (e.g., weight operand 927 shown in Figure 9). The input operand 917 includes a sequence of activation values ​​that have the same (X,Y) coordinates but different Z coordinates. Similarly, the weight operand 927 includes a sequence of weights that have the same (X,Y) coordinates but different Z coordinates. The length of the input operand 917 may be the same as the length of the weight operand 927. The activation values ​​in the input operand 917 and the weights in the weight operand 927 may be supplied to the PE sequentially. The PE may accept activation value / weight pairs at once and multiply the activation value and weight. The positions of the activation values ​​in the input operand 917 may coincide with the positions of the weights in the weight operand 927.

[0109] The input operand 917 or weight operand 927 may be sparse, meaning it may contain one or more zero values. In some embodiments, the PE does not receive or process any activation value / weight pairs where the activation value or weight is zero. By skipping such activation value / weight pairs, computation in the PE can be accelerated without compromising the precision of the output, since the input channel is not separable in the output tensor 930. Figure 9 shows one filter 920 and one output channel, but in other embodiments, multiple filters 920 may be used in the standard convolution 900, resulting in multiple output channels. The number of output channels may be equal to the number of filters 920 used in the standard convolution 900.

[0110] Exemplary channel separable operation Figure 10 shows exemplary unit-depth convolutions 1000 according to various embodiments. A unit-depth convolution 1000 may be a convolution in a convolutional layer of a DNN, for example, convolutional layer 110 in Figure 1. A unit-depth convolution 1000 may be an example of a unit-depth convolution 183 in Figure 1. A unit-depth convolution 1000 is performed on input data including an input tensor 1010 and a filter 1020. The result of a unit-depth convolution 1000 is an output tensor 1030. In some embodiments, a unit-depth convolution 1000 is performed by a DNN accelerator including one or more computation blocks. An example of a DNN accelerator may be the DNN accelerator 200 in Figure 2. An example of a computation block may be the computation block 300 in Figure 3.

[0111] In the embodiment shown in Figure 10, the input tensor 1010 includes activation values ​​(also called "input activation values," "elements," or "input elements") arranged in a 3D matrix. The activation values ​​in the input tensor 1010 are the data points in the input tensor 1010. The input tensor 1010 has a spatial size H in ×W in ×C in It has, and here H in is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activation values ​​in the column of the 10D matrix for each input channel), W in is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activation values ​​in the row of the 10D matrix for each input channel), C in is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of input channels). For the purpose of simplification and illustration, H in and W in Both are 7, meaning that the input tensor 1010 contains a 7x7 2D matrix for each input channel. Each input element in the input tensor 1010 can be represented by (X,Y,Z) coordinates. In other embodiments, the height, width, or depth of the input tensor 1010 may be different.

[0112] Filter 1020 contains weights arranged in a 3D matrix. The weight values ​​can be determined by training a DNN. Filter 1020 has a spatial size H f ×W f ×C in It has, and here H f is the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in the column within each kernel), W f is the width of the filter (i.e., the length along the X-axis, which indicates the number of row weights in each kernel), C in is the depth of the filter (i.e., the length along the Z-axis, which indicates the number of input channels). For the purposes of simplification and illustration, filter 1020 has a 3x3 kernel for each input channel. In other embodiments, the height, width, or depth of filter 1020 may be different. The spatial size of the convolution kernel is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 1010.

[0113] Activation values ​​or weights may occupy one byte or more in memory. The number of bytes for an activation value or weight may depend on the data format. For example, if the activation value or weight has an integer format (e.g., INT8), it will occupy one byte. If the activation value or weight has a floating-point format (e.g., FP16 or BF16), it will occupy two bytes. Other data formats may be used for activation values ​​or weights.

[0114] In a unit depth convolution 1000, the filter 1020 slides across the input tensor 1010 to generate a 3D matrix, i.e., the output tensor 1030. In the embodiment of Figure 10, the output tensor 1030 is H out ×W out ×C in It has a spatial size of H out is the height of the 3D matrix (i.e., the length along the Y-axis, which indicates the number of activation values ​​in the column of the 2D matrix for each input channel), W inis the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activation values ​​in the row of the 2D matrix for each input channel), C in is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of input channels). Unlike standard convolution 900, unit depth convolution 1000 is a channel-separable operation because the input channels are still separated in the output tensor 1030.

[0115] As part of a depth unit convolution 1000, a MAC operation can be performed on a 3x3x3 subtensor 1015 (highlighted by the dot pattern in Figure 10) and a filter 1020 within the input tensor 1010. The subtensor 1015 and the filter 1020 have the same spatial size. The result of the MAC operation on the subtensor 1015 and the filter 1020 is a vector 1035, which is highlighted by the dot pattern in Figure 10 in the output tensor 1030. Vector 1035 contains a sequence of output activation values, each in a different input channel. Vector 1035 may also be an output operand.

[0116] After vector 1035 is generated, further MAC operations are performed to generate additional output operands until the entire output tensor 1030 is generated. For example, filter 1020 may move along the X or Y axis on input tensor 1010, and MAC operations may be performed on filter 1020 and another subtensor within input tensor 1010 (the subtensor having the same size as filter 1020). The amount of movement of filter 1020 on input tensor 1010 during different computational rounds of the convolution is called the stride size of the convolution. The stride size may be 1 (i.e., the amount of movement of filter 1020 is one activation value), 2 (i.e., the amount of movement of filter 1020 is two activation values), and so on. The height and width of output tensor 1030 may be determined based on the stride size.

[0117] In some embodiments, MAC operations on a 3×3×3 subtensor (e.g., subtensor 1015) and filter 1020 may be performed by multiple PEs, such as PEs in a PE array 340. One or more PEs may accept an input operand (e.g., input operand 1017 shown in Figure 10) and a weight operand (e.g., weight operand 1027 shown in Figure 10). The input operand 1017 contains a sequence of activation values ​​that have the same (X,Y) coordinates but different Z coordinates. Similarly, the weight operand 1027 contains a sequence of weights that have the same (X,Y) coordinates but different Z coordinates. The length of the input operand 1017 may be the same as the length of the weight operand 1027. The activation values ​​in the input operand 1017 and the weights in the weight operand 1027 may be supplied sequentially to the PE. The PE may accept activation value / weight pairs at a time and multiply the activation value and weight. The positions of the activation values ​​in the input operand 1017 may coincide with the positions of the weights in the weight operand 1027. Since depth unit convolution 1000 is a channel-separable operation, zero-value activation values ​​or weights should not be skipped as they may contribute to the output.

[0118] Figure 11 shows exemplary group convolutions 1100 according to various embodiments. The group convolution 1100 may be a convolution in a convolutional layer of a DNN, for example, convolutional layer 110 in Figure 1. The group convolution 1100 is performed on input data including input tensor 1110 and filters 1125 and 1127. The result of the group convolution 1100 is output tensor 1130. In some embodiments, the group convolution 1100 is performed by a DNN accelerator including one or more computation blocks. An example of a DNN accelerator may be the DNN accelerator 200 in Figure 2. An example of a computation block may be the computation block 300 in Figure 3.

[0119] In the embodiment shown in Figure 11, the input tensor 1110 includes activation values ​​(also called "input activation values," "elements," or "input elements") arranged in a 3D matrix. The activation values ​​in the input tensor 1110 are data points in the input tensor 1110. The input tensor 1110 has a spatial size H in ×W in ×C in It has, and here H in is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activation values ​​in the column of the 11D matrix for each input channel), W in is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activation values ​​in the row of the 11D matrix for each input channel), C in is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of input channels). For the purpose of simplification and illustration, H in and W in Both are 7, meaning that the input tensor 1110 contains a 7x7 2D matrix for each input channel. Each input element in the input tensor 1110 can be represented by (X,Y,Z) coordinates. In other embodiments, the height, width, or depth of the input tensor 1110 may be different.

[0120] The group convolution 1100 has a group size of 2, which means that the group convolution 1100 contains two convolutions. The input tensor 1110 is such that each of them is H in ×W in ×C in It is divided into two subtensors 1115 and 1117 having a spatial size of / 2. One convolution (the first convolution) lies on subtensor 1115 and filter 1125 (each referred to as "filter 1125"). The other convolution (the second convolution) lies on subtensor 1117 and filter 1127 (each referred to as "filter 1127"). In the embodiment shown in Figure 11, C out / 2 filters 1125 and C out There are two filters 1127, and here C outindicates the number of output channels in the output tensor 1130. Each filter 1125 or 1127 is H f ×W f ×C in It has a spatial size of H f is the height of filter 1125 or 1127 (i.e., the length along the Y axis, which indicates the number of weights in the column within each kernel), W f is the width of filter 1125 or 1127 (i.e., the length along the X-axis, which indicates the number of row weights in each kernel), C in is the depth of filter 1125 or 1127 (i.e., the length along the Z-axis, indicating the number of input channels). The spatial size of the convolution kernel is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 1110. For the purposes of simplification and illustration, filter 1125 or 1127 has a 3x3 kernel for each input channel. In other embodiments, the height, width, or depth of filter 1125 or 1127 may differ. Also, filter 1127 may have different weights than filter 1125.

[0121] Activation values ​​or weights may occupy one byte or more in memory. The number of bytes for an activation value or weight may depend on the data format. For example, if the activation value or weight has an integer format (e.g., INT8), it will occupy one byte. If the activation value or weight has a floating-point format (e.g., FP16 or BF16), it will occupy two bytes. Other data formats may be used for activation values ​​or weights.

[0122] In the first convolution of the group convolution 1100, each filter 1125 slides across the subtensor 1115 to generate a 3D matrix, i.e., a subtensor 1135 in the output tensor 1130. In the embodiment of Figure 11, the subtensor 1135 is H out ×W out ×C out It has a spatial size of / 2, where H outis the height of the 3D matrix (i.e., the length along the Y-axis, which indicates the number of activation values ​​in the column of the 2D matrix for each input channel), W out is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activation values ​​in the row of the 2D matrix for each input channel), C out is the depth of the output tensor 1130 (i.e., the length along the Z axis, which indicates the number of output channels), C out / 2 is the depth of subtensor 1135.

[0123] In the second convolution of the group convolution 1100, each filter 1127 slides across the subtensor 1117 to generate a 3D matrix, i.e., a subtensor 1137 in the output tensor 1130. In the embodiment of Figure 11, the subtensor 1137 is H out ×W out ×C out It has a spatial size of / 2, where H out is the height of the 3D matrix (i.e., the length along the Y-axis, which indicates the number of activation values ​​in the column of the 2D matrix for each input channel), W out is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activation values ​​in the row of the 2D matrix for each input channel), C out is the depth of the output tensor 1130 (i.e., the length along the Z axis, which indicates the number of output channels), C out / 2 is the depth of subtensor 1137.

[0124] Since the input tensor 1110 is divided into multiple subtensors (i.e., subtensors 1115 and 1117 in Figure 11), and subtensors 1115 and 1117 are used in separate convolutions, a single output activation value in the output tensor 1130 is half of the input channel in the input tensor 1110, i.e., C in C in It is calculated based on the input activation values ​​from the two input channels. For each pair of (X,Y) coordinates, the input activation value (the total number of which is C) is calculated. inThe two output activation values ​​are calculated using ( ). In an embodiment where the group convolution 1110 has a group size of N (where N is an integer greater than 2), C has the same (X,Y) coordinates. in Using n input activation values, N output activation values ​​can be calculated. Thus, the group convolution 1100 is a channel separable operation. The group convolution 1100 may be performed by a computation block capable of performing a standard convolution or a depth unit convolution. In some embodiments, the first and second convolutions are depth unit convolutions, respectively.

[0125] Figure 12 shows exemplary depth unit convolutions in PE1200 according to various embodiments. PE1200 may be one embodiment of PE within PE array 340. For illustrative purposes, the PE1200 includes an internal adder assembly 1240 containing four input register files 1210 (collectively referred to as "input register files 1210" or "input register file 1210"), four weight register files 1220 (collectively referred to as "weight register files 1220" or "weight register file 1220"), four multipliers 1230A-1230D (collectively referred to as "multipliers 1230" or "multiplier 1230"), three internal adders 1245A-1245C (collectively referred to as "internal adders 1245" or "internal adder 1245"), and an output register file 1250. In other embodiments, the PE1200 may include a different number of input register files 1210, weight register files 1220, multipliers 1230, internal adders 1245, or output register files 1250.

[0126] In the embodiment shown in Figure 12, each input register file 1210 stores input operands containing 16 input elements IF0 to IF15 for 16 depth unit channels. Each weight register file 1220 stores weight operands containing 16 weights FL0 to FL15. Each multiplier 1230 receives input operands from the input register file 1210 and weights from the weight register file 1220. The multiplier 1230 performs a multiplication operation for 16 cycles. In each cycle, the multiplier 1230 multiplies the input elements and weights to produce a product. The multiplier 1230 processes the input elements and weights sequentially based on their positions in the input operands and weight operands. For example, the multiplier 1230 multiplies IF0 and FL0 in the first cycle, IF1 and FL1 in the second cycle, and continues until it finishes multiplying IF15 and FL15 in the 16th cycle. The multipliers 1230 may operate simultaneously. In the embodiment shown in Figure 12, all input register files 1210 and all weight register files 1220 store data, and all multipliers 1230 are active. In other embodiments, one or more of the input register files 1210 or weight register files 1220 may be empty, and one or more of the multipliers 1230 may be inactive.

[0127] The product generated by the multiplier 1230 is fed to the internal adder assembly 1240. The internal adder assembly 1240 performs an in-row reduction. As shown in Figure 12, the internal adder 1245 is arranged in two layers within the internal adder assembly 1240, where internal adders 1245A and 1245B are in the first layer and internal adder 1245C is in the second layer. The internal adder 1245A receives the products from the multipliers 1230A and 1230B and performs an accumulation operation on these products. In some embodiments, the internal adder 1245A performs an accumulation operation for 16 cycles, each corresponding to a different depth unit channel and being an accumulation of the product of the corresponding depth unit channel. For example, in the first cycle, the internal adder 1245A accumulates the product of IF0 and FL0 from the multiplier 1230A and the product of IF0 and FL0 from the multiplier 1230B. In the second cycle of the cumulative operation, internal adder 1245A accumulates the product of IF1 and FL1 from multiplier 1230A, and the product of IF1 and FL1 from multiplier 1230B, and so on. Similarly, internal adder 1245B may receive the product from multipliers 1230C and 1230D and perform a 16-cycle cumulative operation on these products. Internal adders 1245A and 1245B may operate simultaneously. Internal adder assembly 1240 may perform an intra-row reduction within PE1200 during depth unit convolution.

[0128] The sum generated by internal adders 1245A and 1245B is fed to internal adder 1245C. In some embodiments, internal adder 1245C performs a 16-cycle cumulative operation, each corresponding to a different depth unit channel, which for the corresponding depth unit channel is the cumulative sum of internal adders 1245A and 1245B. Internal adder 1245C outputs an output operand stored in the output register file 1250 of PE1200. The output operand contains 16 output elements OF0 to OF16. The output operand may be part of the OFM of a depth unit convolution. Each output element may correspond to a different depth unit channel.

[0129] Through cumulative operations by internal adders 1245A to 1245C, the internal adder assembly 1240 performs an intra-row reduction of the kernel, i.e., an intra-row reduction. In the example where the depth unit convolution is 3 × 3s1 (for example, the depth unit convolution described above in relation to Figures 6 and 7), the internal adder assembly 1240 can perform a 3-point reduction within a row of the 3 × 3 kernel, producing a sum equal to X0Y0 × FX0FY0 + X1Y0 × FX1FY0 + X2Y0 × FX2FY0 for each of the 16 depth unit channels.

[0130] In some embodiments, the size of the output element can be 1 byte, and the output register file 1250 has a storage capacity of 16 bytes or more. Since the output register file 1250 can store 16 output elements at once, the PE 1200 can receive 16 depth unit channels and compute and store 16 output elements without having to perform any reduction in the Z direction. This is advantageous over conventional DNN accelerators that process one output element at a time within a single PE while consuming all input channels associated with the generation of that output element by distributing the input channels across multiple multipliers. Such DNN accelerators may work well for standard convolutions, but are inefficient for depth unit convolutions because the number of input channels that need to be accumulated in a depth unit convolution is 1 (a depth unit convolution does not involve accumulation across multiple input channels), and therefore, typically only one multiplier is active at a time.

[0131] In addition to more efficient unit-depth convolution, the PE1200 can also perform standard convolution. For example, one or more of the internal adders 1245 may perform accumulation across 16 channels to produce a single output point. In some embodiments, the PE1200 may have a unit-depth convolution mode and a standard convolution mode. The PE1200 performs unit-depth convolution when in unit-depth convolution mode and standard convolution when in standard convolution mode.

[0132] In addition to intra-row reduction, depth unit convolution may also include inter-row reduction across PEs within a PE column. As mentioned above, such inter-row reduction can be performed by using an external adder assembly.

[0133] In addition to unit-depth and group convolutions, pooling layers may also have channel-separable operations. As described above, pooling operations can downsample feature maps without reducing the number of channels. In some embodiments, a pooling layer receives the output tensor of a convolutional layer as its input tensor. A pooling operation is performed on the input tensor to reduce the size of the input tensor and generate the output tensor of the pooling layer. A channel-separable pooling operation may be performed on an input operand containing multiple unit-depth channels. The input operand may be the output operand of a unit-depth convolution, for example, one of the unit-depth convolutions described above. The pooling operation is channel-separable, meaning that a pooling operation can be performed separately on the input array for each unit-depth channel. For example, for each unit-depth channel, output elements are generated from X-dimensional and Y-dimensional windows. The input elements may be organized in a similar manner to unit-depth convolutions, having different X coordinates across different input register files in a PE and different Y coordinates across different PEs. A series of separable channels, one of which is evaluated per cycle, can occupy consecutive register file entries.

[0134] Figure 13 shows exemplary channel separable pooling operations within PE1300 in various embodiments. PE1300 may be one embodiment of PE within PE array 340. The channel separable pooling operations in the embodiment of Figure 13 can determine values ​​from fixed windows of X and Y dimensions across the entire tensor volume without changing the Z dimension. As shown in Figure 13, PE1300 includes an internal pooling assembly 1310, input register files 1330A-1330D (collectively referred to as "input register files 1330" or "input register file 1330"), and an output register file 1340.

[0135] Each input register file 1330 stores an input operand containing 14 input elements IF0 to IF13. Each input element corresponds to a different depth unit channel. The 14 input elements of each input operand can then be supplied to the internal pooling assembly 1310. Each input element or weight can be stored in the storage unit of the corresponding register file. The size of the storage unit may be 1 byte. The input elements or weights may be integers in the data format, for example, INT8.

[0136] The internal pooling assembly 1310 performs a pooling operation on the input operands from the input register file 1330. In one embodiment, the pooling operation is a maximum pooling operation, in which the internal pooling assembly 1310 may take the maximum value from a fixed window of X and Y dimensions across the entire tensor volume without changing the Z dimension. In another embodiment, the pooling operation is a mean pooling operation, in which the internal pooling assembly 1310 may determine the average of the fixed window of X and Y dimensions across the entire tensor volume without changing the Z dimension. In other embodiments, the internal pooling assembly 1310 may perform other types of pooling operations.

[0137] The internal pooling assembly 1310 includes internal pooling operators 1320A to 1320C (collectively referred to as "internal pooling operators 1320" or "internal pooling operators 1320"). The internal pooling operators 1320 are arranged in two layers. The first layer includes internal pooling operators 1320A and 1320B. The second layer includes internal pooling operator 1320C. Each of the internal pooling operators 1320 in the first layer receives two input operands from two input register files 1330. For example, internal pooling operator 1320A receives input operands from input register files 1310A and 1310B. Internal pooling operator 1320A performs a pooling operation for 13 cycles. In each cycle, the internal pooling operator 1320A performs a pooling operation on the input elements from input register file 1310A and input elements from input register file 1310B. For example, the internal pooling operator 1320A selects the input element with the larger value or determines the average value of two input elements. The two input elements used in each cycle correspond to the same depth unit channel. Thus, the internal pooling operator 1320A generates an output operand containing 13 elements, each corresponding to a different depth unit channel.

[0138] Similarly, the internal pooling operator 1320B receives input operands from input register files 1310C and 1310D, performs a 13-cycle pooling operation on the two input operands, each cycle including pooling operations on input elements from input register file 1310A and input elements from input register file 1310B. The internal pooling operator 1320B generates an output operand containing 13 elements.

[0139] The output operands of internal pooling operators 1320A and 1320B are provided to internal pooling operator 1320C as two input operands. Internal pooling operator 1320C performs a pooling operation for 13 cycles on the two input operands. In each cycle, internal pooling operator 1320C compares the input elements from internal pooling operator 1320A and internal pooling operator 1320B and may select the input element with the larger value or determine the average of the two input elements. Internal pooling operator 1320B generates an output operand containing 14 elements OF0 to OF13, each corresponding to a depth unit channel. Internal pooling assembly 1310 reduces the four input operands in input register file 1330 to one output operand in output register file 1340.

[0140] Figure 14 shows another exemplary channel separable pooling operation in PE1400 according to various embodiments. PE1400 may be one embodiment of PE within PE array 340. The channel separable pooling operation in the embodiment of Figure 14 can determine values ​​from a fixed window of X and Y dimensions across the entire tensor volume without changing the Z dimension. As shown in Figure 14, PE1400 includes an internal pooling operator 1410, input register files 1430A-1430D (collectively referred to as "input register files 1430" or "input register file 1430"), and two output register files 1440.

[0141] In the embodiment shown in Figure 14, two input register files 1430 store input operands containing 16 input elements IF0 to IF15, i.e., four input register files 1430 store two input operands. Each input element corresponds to a different depth unit channel. The 16 input elements of each input operand can be sequentially supplied to the internal pooling operator 1410, for example, via a linking module. Each input element or weight can be stored in two storage units of the corresponding register file. The size of the storage unit may be 1 byte. The input elements or weights may be, for example, the number of FPs in the data format FP16 or BF16.

[0142] The internal pooling operator 1410 performs a pooling operation on two input operands from the input register file 1430. In one embodiment, the pooling operation is a max pooling operation, in which the internal pooling operator 1410 may take the maximum value from a fixed window of X and Y dimensions across the entire tensor volume without changing the Z dimension. In another embodiment, the pooling operation is a mean pooling operation, in which the internal pooling operator 1410 may determine the average of the fixed window of X and Y dimensions across the entire tensor volume without changing the Z dimension. In other embodiments, the internal pooling operator 1410 may perform other types of pooling operations. For example, the internal pooling operator 1420 performs a pooling operation for 16 cycles. In each cycle, the internal pooling operator 1420A performs a pooling operation on the input elements of a first input operand, which are from input register files 1410A and 1410B, and on the input elements of a second input operand, which are from input register files 1410C and 1410D. The internal pooling operator 1420 may select the input element with the larger value, or it may determine the average value of two input elements. The two input elements used in each cycle correspond to the same depth unit channel. Thus, the internal pooling operator 1420A generates an output operand containing 16 elements OF0 to OF15, each corresponding to a different depth unit channel. The output operand can be stored in the output register file 1440.

[0143] In some embodiments (for example, embodiments where channel separable pooling is average pooling), the PE used for channel separable pooling may be an embodiment of the PE that can be used for depth unit convolution. For example, a multiplier in the PE can multiply each input element of the input operands by 1, so the product is the input elements. An internal adder assembly in the PE may perform an accumulation operation on the product produced by the multiplier in the PE. A divider, which may be in or outside the PE, may perform a division operation on the output of the internal adder assembly, for example, by dividing each output element from the internal adder assembly by a predetermined number. The predetermined number may be the number of input operands received by the internal adder assembly.

[0144] Figure 15 shows exemplary channel-separable element-by-element addition in PE1500 according to various embodiments. PE1500 may be one embodiment of PE within PE array 340. The element-by-element addition operation can take two input tensors and perform vector addition or vector addition after initial scalar multiplication. The dimensions of the two input tensors may be identical. Separate scale values ​​can be applied to one or both input tensors. The channel-separable element-by-element addition illustrated in Figure 15 involves a scale value. The size of the scale value is 8 bits, i.e., 1 byte, which is the same as the size of the input element. PE1500 has the same or similar components as PE which can be used for depth-unit convolution. As shown in Figure 15, the PE1500 includes an internal adder assembly 1540 containing four input register files 1510 (collectively referred to as "input register files 1510" or "input register file 1510"), four scale register files 1520 (collectively referred to as "scale register files 1520" or "scale register file 1520"), four multipliers 1530A-1530D (collectively referred to as "multipliers 1530" or "multiplier 1530"), three internal adders 1545A-1545C (collectively referred to as "internal adders 1545" or "internal adder 1545"), and an output register file 1550. In other embodiments, the PE1500 may include a different number of input register files 1510, a scale register file 1520, a multiplier 1530, an internal adder 1545, or an output register file 1550.

[0145] Input register file 1510A stores the first input operand from one of the two input tensors. Input register file 1510C stores the second input operand, which is from the other of the two input tensors. Each input operand contains 16 input elements IF0~IF15, each corresponding to a different depth unit channel. Input register files 1510B and 1510D are empty. Scale register files 1520A and 1520C each store a vector of 16 scale values, i.e., SV0~SV15. The scale values ​​can be one or more fixed values ​​that can be determined by training the DNN.

[0146] Multiplier 1530A performs a multiplication operation on the first input operand and a vector of scale values ​​from scale register file 1520A. Similarly, multiplier 1530B performs a multiplication operation on the second input operand and a vector of scale values ​​from scale register file 1520B. Multipliers 1530B and 1530D are inactive.

[0147] The product generated by the multiplication operation is supplied to the internal adder assembly 1540. Since multipliers 1530B and 1530D are inactive, the values ​​supplied from multipliers 1530B and 1530D to the internal adder assembly 1540 may be zero. The internal adder assembly 1540 includes internal adders 1545A to 1545C, each capable of performing channel-separable cumulative operations similar to the cumulative operation of internal adder 1045 described above in relation to Figure 10. The internal adder assembly 1040 outputs an output operand containing 16 output elements OF0 to OF15. The output operand is stored in the output register file 1550.

[0148] In embodiments where element-level addition does not involve a scale value, the values ​​stored in scale register files 1520A and 1520C can be set to 1, and as a result, the outputs of multipliers 1530A and 1530C become the input operands themselves.

[0149] Figure 16 shows another exemplary channel-separable element-unit addition in PE1600 according to various embodiments. PE1600 may be an embodiment of PE within PE array 340. Channel-separable element-unit addition involves a scale value. The size of the scale value is 16 bits, i.e., 2 bytes, which is twice the size of the input element. PE1600 has the same or similar components as PE which can be used for depth-unit convolution. As shown in Figure 16, the PE1600 includes four input register files 1610 (collectively referred to as "input register files 1610" or "input register file 1610"), four scale register files 1620 (collectively referred to as "scale register files 1620" or "scale register file 1620"), four multipliers 1630A-1630D (collectively referred to as "multipliers 1630" or "multiplier 1630"), three internal adders 1645A-1645C (collectively referred to as "internal adders 1645" or "internal adder 1645"), and an internal adder assembly 1640 containing two bit shifters 1643A and 1643B, and an output register file 1650. In other embodiments, the PE1600 may include a different number of input register files 1610, a scale register file 1620, a multiplier 1630, an internal adder 1645, or an output register file 1650.

[0150] In Figure 16, each input register file 1610 stores an input operand. Input register files 1610A and 1610B store the same input operand (e.g., the first input operand from one of the two input tensors), and input register files 1610A and 1610B store the same input operand (e.g., the second input operand from the other of the two input tensors). Scale register files 1620A and 1620B each store half of the scale vector. Scale register file 1620A stores the lower bytes SV0 to SV7, and scale register file 1620B stores the upper bytes SV8 to SV15. Similarly, scale register files 1620C and 1620D each store half of another scale vector. Scale register file 1620C stores the lower bytes SV0 to SV7, and scale register file 1620D stores the upper bytes SV8 to SV15.

[0151] Multiplier 1630A performs a multiplication operation on the first input operand from input register file 1610A and the first half of the first scale vector from scale register file 1620A. Multiplier 1630B performs a multiplication operation on the first input operand from input register file 1610B and the second half of the first scale vector from scale register file 1620B. Similarly, multiplier 1630C performs a multiplication operation on the second input operand from input register file 1610C and the first half of the second scale vector from scale register file 1620C, and multiplier 1630D performs a multiplication operation on the second input operand from input register file 1610D and the second half of the second scale vector from scale register file 1620D.

[0152] The products generated by the four multipliers 1630 are fed to the internal adder assembly 1640. The products from multipliers 1630A and 1630C are fed directly to internal adders 1645A and 1645B, respectively. The products from multipliers 1630B and 1630D are first fed to bit shifters 1643A and 1643B, respectively. Bit shifter 1643A can change the position of the product from multiplier 1630B, and these products are then combined with the product from multiplier 1630A by internal adder 1645A. Similarly, bit shifter 1643B can change the position of the product from multiplier 1630D, and these products are then combined with the product from multiplier 1630C by internal adder 1645B. Next, the sum from internal adders 1645A and 1645B is then provided to internal adder 1645C, which generates an output operand containing 16 output elements OF0 to OF15. The output operand is stored in the output register file 1650.

[0153] Figure 17 shows exemplary channel-separable element-level multiplication in PE1700 according to various embodiments. PE1700 may be one embodiment of PE within PE array 340. Channel-separable element-level multiplication is performed on two input tensors, which may be from two DNN layers. The two input tensors may have the same dimension. The result of channel-separable element-level multiplication may be a new tensor (also called an output tensor) having the same dimension as the input tensors. The output tensor contains multiple scalar values. Each scalar value may be the product of a first scale value in the first input tensor and a second scalar value in the second input tensor.

[0154] As shown in Figure 17, the PE1710 includes four first input register files 1713A-1713D (collectively referred to as "first input register files 1713" or "first input register file 1713"), four second input register files 1715A-1715D (collectively referred to as "second input register files 1715" or "second input register file 1715"), and four multipliers 1717A-1717D (collectively referred to as "multipliers 1717" or "multiplier 1717"). Although not shown in Figure 17, the PE1710 may also include other components such as an internal adder assembly and output register files. The internal adder assembly does not necessarily have to be used for channel-separable element-level multiplication. Furthermore, PE1710 may include a different number of first or second input register files. PE1710 may be the same as or similar to the PE used to perform depth unit convolution.

[0155] The first and second input tensors may be loaded separately into the first input register file 1713 and the second input register file 1715, respectively. As shown in Figure 17, each first input register file 1713 stores the first input operands, which may be part of the first input tensor. Each second input register file 1715 stores the second input operands, which may be part of the second input tensor. Each multiplier 1717 performs a multiplication operation, for example, 16 sequential multiplication cycles, on the first and second input operands. Each cycle may be the multiplication of the input elements in the first input operand and the input elements in the second input operand. The two input elements may correspond to the same depth unit channel. The product produced by multiplying each pair of input elements may be the output element of an output operand that can be written to the output register file of PE1710. Within the PE1710, the presence of multiple register files 1713 or 1715 and multiple multipliers 1717 for each input tensor allows the PE1710 to implement N parallel contexts, where N is an integer equal to the number of multipliers 1717 (N=4 in Figure 17). A context may refer to the individual partial sums of different output elements. This is possible because the PE architecture bypasses internal adder assembly and allows contexts to be written in parallel to the output register files. Furthermore, channel-separable element-wise multiplication may not require an external adder.

[0156] The PE1710 is more advantageous compared to conventional element-wise multiplication, which generates a single context per clock cycle. In conventional element-wise multiplication, subsequent channels can be fed in parallel to different multipliers to generate a single context. The channels are then reduced through an adder before being written to the output register file. The result accumulated across channels passing through the adder produces an incorrect element-wise multiplication result, and therefore only one multiplier can be used per PE. In contrast, in the embodiment of Figure 17, four multipliers 1717 can be utilized within the PE1710, resulting in four times the throughput of conventional element-wise multiplication.

[0157] Figure 18 shows PE array 1800 in various embodiments. PE array 1800 may be one embodiment of PE array 340 in Figure 3 or one embodiment of PE array 401 in Figure 4. PE array 1800 includes a plurality of PE1810s (each individually referred to as "PE1810"). PE1810s can perform MAC operations in convolution. PE1810s can also perform other types of deep learning operations. PE1810s are sometimes referred to as neurons in a DNN. PE1810s may be examples such as PE430 in Figure 4, PE1200 in Figure 12, PE1300 in Figure 13, PE1400 in Figure 14, PE1500 in Figure 15, PE1600 in Figure 16, or PE1700 in Figure 17. Each PE1810 has two input signals 1850 and 1860 and an output signal 1870. The input signal 1850 is at least part of the IFM for the layer. The input signal 1860 is at least part of the filter for the layer. In some embodiments, the input signal 1850 of PE1810 includes one or more input operands, and the input signal 1860 includes one or more weight operands.

[0158] Each PE1810 performs a MAC operation on input signals 1850 and 1860 and outputs an output signal 1870, which is the result of the MAC operation. Some or all of the input signals 1850 and 1860 and the output signal 1870 may be in integer format such as INT8, or floating-point format such as FP16 or BF16. For the purposes of simplification and illustration, all PE1810 input and output signals have the same reference number, but PE1810s may receive different input signals and output different output signals from one another. Also, a PE1810 may differ from another PE1810 in that it contains, for example, more, fewer, or different components.

[0159] As shown in Figure 18, the PE1810s are connected to one another as indicated by the dashed arrows in Figure 18. The output signal 1870 of PE1810 may be transmitted as an input signal to many other PE1810s via the interconnection between the PE1810s (and possibly return to them). In some embodiments, the output signal 1870 of PE1810 may incorporate the output signals of one or more other PE1810s via the cumulative operation of PE1810 to generate an internal partial sum of the PE array. Further details regarding the PE1810s are described below in relation to Figure 18.

[0160] In the embodiment of Figure 18, the PE1810s are located in column 1805 (individually referred to as “column 1805”). The inputs and weights of the layer can be distributed to the PE1810s based on column 1805. Each column 1805 has a column buffer 1820. The column buffer 1820 temporarily stores the data provided to the PE1810s in column 1805. The column buffer 1820 can also store the data output by the last PE1810 in column 1805. The output of the last PE1810 can be the sum of the MAC operations of all PE1810s in column 1805, which is the column-level internal partial sum of the PE array 1800. In other embodiments, the inputs and weights may be distributed to the PE1810s based on rows in the PE array 1800. The PE array 1800 may include row buffers instead of column buffers 1820. The row buffer may store the input signals of the corresponding row's PE, or it may store the row-level internal partial sums of the PE array 1800.

[0161] As shown in Figure 18, each column buffer 1820 is associated with a load 1830 and a drain 1840. Data provided to column 1805 is sent to column buffer 1820 via load 1830, for example, through a higher memory hierarchy, such as local memory 310 in Figure 3. Data generated in column 1805 is extracted from column buffer 1820 via drain 1840. Column buffer 1820 may be a data store (or part of a data store), such as data store 330 or 350 in Figure 3. In some embodiments, the data extracted from column buffer 1820 is sent via drain operation to a higher memory hierarchy, such as memory 410 in Figure 4. In some embodiments, the drain operation does not begin until all PEs 1810 in column 1805 have completed their MAC operations. Although not shown in Figure 18, one or more columns 1805 may be associated with an external adder assembly.

[0162] Figure 19 is a block diagram of PE1900 in various embodiments. PE1900 may be an embodiment of PE1810 in Figure 18. PE1900 includes an input register file 1910 (each referred to as "input register file 1910"), a weight register file 1920 (each referred to as "weight register file 1920"), a multiplier 1930 (each referred to as "multiplier 1930"), an internal adder assembly 1940, and an output register file 1950. In other embodiments, PE1900 may include fewer, more, or different components. For example, PE1900 may include multiple output register files 1950. As yet another example, PE1900 may include a single input register file 1910, a weight register file 1920, or a multiplier 1930. As yet another example, PE1900 may include an adder instead of the internal adder assembly 1940.

[0163] The input register file 1910 temporarily stores input operands for MAC operations performed by PE1900. In some embodiments, the input register file 1910 may store a single input operand at a time. In other embodiments, the input register file 1910 may store multiple input operands or a portion of input operands at a time. An input operand contains multiple input elements (i.e., input elements) in an input tensor. The input elements of an input operand may be stored sequentially in the input register file 1910 so that the input elements can be processed sequentially. In some embodiments, each input element in an input operand may come from a different input channel of the input tensor. An input operand may contain input elements from each of the input channels of the input tensor, and the number of input elements in an input operand may be equal to the number of input channels. The input elements in an input operand may have the same XY coordinates, which may be used as the XY coordinates of the input operand. For example, all input elements of an input operand may be X0Y0, X0Y1, X1Y1, etc.

[0164] The weight register file 1920 temporarily stores weight operands for MAC operations by PE1900. The weight operands include the weights within the DNN layer's filter. In some embodiments, the weight register file 1920 may store a single weight operand at a time. In other embodiments, the input register file 1910 may store multiple weight operands or a portion of weight operands at a time. A weight operand may contain multiple weights. The weights of a weight operand may be stored sequentially in the weight register file 1920 so that the weights can be processed sequentially. In some embodiments, for multiplication operations involving weight operands and input operands, each weight in the weight operand may correspond to an input element of the input operand. The number of weights in the weight operand may be equal to the number of input elements in the input operand.

[0165] In some embodiments, the weight register file 1920 may be the same as or similar to the input register file 1910, for example, having the same size. The PE 1900 may contain multiple register files, some of which are designated as the input register file 1910 for storing input operands, some as the weight register file 1920 for storing weight operands, and some as the output register file 1950 for storing output operands. In other embodiments, register files within the PE 1900 may be designated for other purposes, such as storing scale operands used in element-wise addition operations.

[0166] The multiplier 1930 performs multiplication operations on the input operands and weight operands. The multiplier 1930 may perform a sequence of multiplication operations on a single input operand and a single weight operand, generating a product operand containing a sequence of products. Each multiplication operation in the sequence involves multiplying the input element in the input operand and the weight in the weight operand. In some embodiments, the position (or index) of the input element in the input operand coincides with the position (or index) of the weight in the weight operand. For example, the first multiplication operation is the multiplication of the first input element in the input operand and the first weight in the weight operand; the second multiplication operation is the multiplication of the second input element in the input operand and the second weight in the weight operand; the third multiplication operation is the multiplication of the third input element in the input operand and the third weight in the weight operand, and so on. The input elements and weights in the same multiplication operation may correspond to the same depth unit channel, and their product may also correspond to the same depth unit channel.

[0167] Multiple multipliers 1930 may perform multiplication operations simultaneously. These multiplication operations are sometimes referred to as a round of multiplication operations. In a round of multiplication operations by multipliers 1930, each multiplier 1930 may use different input operands and different weight operands. Different input operands or weight operands may be stored in different register files of PE1900. For example, the first multiplier 1930 uses a first input operand (e.g., stored in the first input register file 1910) and a first weight operand (e.g., stored in the first weight register file 1920), the second multiplier 1930 uses a second input operand (e.g., stored in the second input register file 1910) and a second weight operand (e.g., stored in the second weight register file 1920), the third multiplier 1930 uses a third input operand (e.g., stored in the third input register file 1910) and a third weight operand (e.g., stored in the third weight register file 1920), and so on. For each multiplier 1930, a round of multiplication operations may include multiple cycles. Each cycle includes the multiplication operation of the input elements and weights.

[0168] A multiplier 1930 can perform multiplication operations over multiple rounds. A multiplier 1930 may use the same weight operands but different input operands in different rounds. For example, a multiplier 1930 performs a sequence of multiplication operations on a first input operand stored in a first input register file in the first round, and on a second input operand stored in a second input register file in the second round. In the second round, a different multiplier 1930 may perform another sequence of multiplication operations using the first input operand and different weight operands. In this way, the first input operand is reused in the second round. The first input operand may be reused further in additional rounds, for example, by an additional multiplier 1930.

[0169] An internal adder assembly 1940 includes one or more adders, i.e., internal adders, within the PE 1900. The internal adder assembly 1940 can perform cumulative operations on two or more product operands from the multiplier 1930 to produce an output operand of the PE 1900. In some embodiments, the internal adders are arranged in a sequence of hierarchies. A hierarchy includes one or more internal adders. For a first hierarchy of the internal adder assembly 1940, an internal adder may receive product operands from two or more multipliers 1930 and produce a sum operand by a sequence of cumulative operations. Each cumulative operation produces the sum of two or more products, each from a different multiplier 1930. The sum operand includes a sequence of sums, each of which is the result of a cumulative operation and corresponds to a depth unit channel. For other hierarchies of the internal adder assembly 1940, the internal adders in the hierarchy receive sum operands from the preceding hierarchy in the sequence. Each of these numbers may be produced by a different internal adder in the preceding hierarchy. The ratio of the number of internal adders in one tier to the number of internal adders in a subsequent tier may be 2:1. In some embodiments, the final tier of the internal adder assembly 1940 may include a single internal adder that generates the output operand of PE1900.

[0170] The output register file 1950 stores the output operands of PE1900. In some embodiments, the output register file 1950 may store one output operand at a time. In other embodiments, the output register file 1950 may store multiple output operands or a portion of output operands at a time. An output operand includes multiple output elements within the IFM. The output elements of an output operand may be stored sequentially in the output register file 1950 so that the output elements can be processed sequentially. In some embodiments, each output element in an output operand corresponds to a different depth unit channel and is an element of a different output channel of the depth unit convolution's output channels. The number of output elements in an output operand may be equal to the number of depth unit channels in the depth unit convolution.

[0171] Exemplary Method of Dynamic Decompression for Channel-Separable Operations Figure 20 is a flowchart illustrating method 2000 of dynamic decompression for channel-separable operations according to various embodiments. Method 2000 may be performed by one or more components of the computation block 300 in Figure 3 to perform layers of a DNN. The layers include one or more channel-separable operations, such as depth unit convolution, group convolution, element unit operation, and channel-separable pooling operation. Although method 2000 is described with reference to the flowchart illustrated in Figure 20, many other methods for dynamic decompression for channel-separable operations may be used as alternatives. For example, the execution order of the steps in Figure 20 may be changed. As another example, some of the steps may be modified, eliminated, or combined.

[0172] Computation block 300 stores the compressed data in a data store (2010). The compressed data includes one or more non-zero data points, which are a subset of the input operands of a layer in the DNN. An input operand includes multiple data points. In some embodiments, the layer is selected from the group consisting of depth unit convolutional layers, group convolutional layers, element unit layers, and pooling layers.

[0173] Computation block 300 determines whether the input operand contains any zero-value data points based on the sparsity bitmap of the input operand (2020). The sparsity bitmap contains multiple bits. Each bit corresponds to each data point in the input operand and indicates whether each data point is zero or non-zero. In some embodiments, the input operand is part of the layer's IFM. The IFM contains multiple channels. Multiple data points in the input operand are located in different channels among the multiple channels.

[0174] In some embodiments, the calculation block 300 determines whether the input operand contains any zero-value data points by determining whether any bit in the sparsity bitmap is zero. In some embodiments, multiple bits in the sparsity bitmap are in sequence. The calculation block 300 may also determine the location of zero-value data points in the input operand based on the location of bits in the sparsity bitmap.

[0175] After determining that the input operand contains zero-value data points, the computation block 300 generates decompressed data by inserting the zero-value data points into the compressed data based on their positions in the input operand (2030). In some embodiments, the computation block 300 also generates a new sparsity bitmap for the decompressed data. The new sparsity bitmap contains multiple bits, each bit having a value of 1. The new sparsity bitmap can facilitate the densification of zero-value data points so that the PE can handle the zero-value data points, even though it is implemented in sparsity acceleration logic.

[0176] The calculation block 300 sends the decompression data to the PE (2040), which is configured to calculate an output operand based on the decompression data. In some embodiments, the output operand includes data points in two or more channels. The two or more channels may be some or all of the channels in the input operand.

[0177] In some embodiments, the compute block 300 stores the output operands in a data store. The compute block 300 also writes a subset of the output operands from the data store to memory, such as local memory 310. The subset of output operands includes one or more non-zero data points within the output operands.

[0178] In some embodiments, the computation block 300 generates a new sparsity bitmap for the output operand. The new sparsity bitmap contains multiple bits, each corresponding to a data point in the output operand and indicating whether each data point in the output operand is zero or non-zero.

[0179] Exemplary computing device Figure 21 is a block diagram of an exemplary computing device 2100 according to various embodiments. In some embodiments, the computing device 2100 may be used as at least part of the DNN accelerator 200 of Figure 2. Several components included in the computing device 2100 are shown in Figure 21, but one or more of these components may be omitted or duplicated as appropriate for the application. In some embodiments, some or all of the components included in the computing device 2100 may be mounted on one or more motherboards. In some embodiments, some or all of these components are manufactured on a single system-on-chip (SoC) die. Furthermore, in various embodiments, the computing device 2100 may not include one or more of the components shown in Figure 21, but the computing device 2100 may include interface circuits for coupling to one or more components. For example, the computing device 2100 may not include a display device 2106, but it may include a display device interface circuit (e.g., a connector and driver circuit) to which the display device 2106 can be coupled. In another example set, the computing device 2100 may not include an audio input device 2118 or an audio output device 2108, but may include an audio input or output device interface circuit (e.g., a connector and support circuit) to which the audio input device 2118 or the audio output device 2108 can be coupled.

[0180] The computing device 2100 may include a processing device 2102 (e.g., one or more processing devices). The processing device 2102 processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. The computing device 2100 may include a memory 2104 which itself may include one or more memory devices such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard drives. In some embodiments, the memory 2104 may include memory that shares a die with the processing device 2102. In some embodiments, the memory 2104 includes one or more non-temporary computer-readable media that store executable instructions for performing operations for scheduling computations in a DNN, such as the method 2000 described above in relation to Figure 20, several operations performed by the computation block 300 described above in relation to Figure 3, or several operations performed by the decompression module 330 described above in relation to Figure 3. Instructions stored in one or more non-temporary computer-readable media may be executed by the processing device 2102.

[0181] In some embodiments, the computing device 2100 may include a communication chip 2112 (e.g., one or more communication chips). For example, the communication chip 2112 may be configured to manage wireless communication for data transfer to and from the computing device 2100. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can communicate data using modulated electromagnetic radiation over a non-solid medium. The term does not mean that the associated device is wire-free, although in some embodiments it may be wire-free.

[0182] The communication chip 2112 may implement any of several wireless standards or protocols, including, but not limited to, Wi-Fi® (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Institute of Electrical and Electronics Engineers (IEEE) standards including the Long-Term Evolution (LTE) project, and any modifications, updates, and / or revisions (e.g., the Advanced LTE project, the Ultra-Mobile Broadband (UMB) project (also known as "3GPP®2")). IEEE 802.16-compatible broadband wireless access (BWA) networks are commonly referred to as WiMAX® networks, where WiMAX® is an acronym for Worldwide Interoperability for Microwave Access and is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The communication chip 2112 may operate in accordance with Global Mobile Communications System (GSM®), General Purpose Packet Radio Service (GPRS), Universal Mobile Communications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. The communication chip 2112 may operate in accordance with GSM® Evolution Extension Data (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 2112 may operate in accordance with Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Extended Cordless Telecommunications (DECT), Evolution Data Optimized (EV-DO), and their derivatives, as well as any other radio protocols designated as 3G, 4G, 5G, and later. In other embodiments, the communication chip 2112 may operate in accordance with other radio protocols.The computing device 2100 may include an antenna 2122 for facilitating wireless communication and / or for receiving other wireless communications (such as AM or FM radio transmissions).

[0183] In some embodiments, the communication chip 2112 may manage wired communications such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet®). As described above, the communication chip 2112 may comprise multiple communication chips. For example, the first communication chip 2112 may be dedicated to short-range wireless communications such as Wi-Fi® or Bluetooth®, and the second communication chip 2112 may be dedicated to long-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX®, LTE, or EV-DO. In some embodiments, the first communication chip 2112 may be dedicated to wireless communications, and the second communication chip 2112 may be dedicated to wired communications.

[0184] The computing device 2100 may include a battery / power supply circuit 2114. The battery / power supply circuit 2114 may include a circuit for coupling one or more energy storage devices (e.g., batteries or capacitors) and / or components of the computing device 2100 to an energy source separate from the computing device 2100 (e.g., an AC line power supply).

[0185] The computing device 2100 may include a display device 2106 (or a corresponding interface circuit as described above). The display device 2106 may include any visual indicator, such as a head-up display, computer monitor, projector, touchscreen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0186] The computing device 2100 may include an audio output device 2108 (or a corresponding interface circuit as described above). The audio output device 2108 may include any device that generates an audible indicator, such as a speaker, headset, or earphones.

[0187] The computing device 2100 may include an audio input device 2118 (or a corresponding interface circuit as described above). The audio input device 2118 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital instrument (e.g., an instrument with a Musical Instrument Digital Interface (MIDI) output).

[0188] The computing device 2100 may include a GPS device 2116 (or a corresponding interface circuit as described above). The GPS device 2116 can communicate with a satellite-based system, as is known in the art, and can receive the position of the computing device 2100.

[0189] The computing device 2100 may include another output device 2110 (or a corresponding interface circuit as described above). Examples of other output devices 2110 may include an audio codec, a video codec, a printer, a wired or wireless transmitter, or additional storage devices for providing information to other devices.

[0190] The computing device 2100 may include another input device 2120 (or a corresponding interface circuit as described above). Examples of other input devices 2120 may include an accelerometer, gyroscope, compass, image capture device, keyboard, cursor control device such as a mouse, stylus, touchpad, barcode reader, quick response (QR) code reader, any sensor, or radio frequency identification (RFID) reader.

[0191] The computing device 2100 may have any desired form factor, such as a handheld computer system or mobile computer system (e.g., a mobile phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, PDA® (Personal Digital Assistant), ultramobile personal computer, etc.), a desktop computer system, a server or other network computing component, a printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 2100 may be any other electronic device that processes data.

[0192] Example Selection The following paragraphs provide various examples of embodiments disclosed herein.

[0193] Example 1 provides a method for executing a DNN layer, the method comprising: storing compressed data in a datastore, the compressed data comprising one or more non-zero data elements which are a subset of the layer's input operands; determining whether an input operand contains zero data elements based on a sparsity bitmap of the input operand, where the input operand comprises multiple data elements and the sparsity bitmap comprises multiple bits, each bit corresponding to each data element in the input operand and indicating whether each data element is zero or non-zero; generating decompressed data by inserting zero data elements into the compressed data based on the location of the zero data elements in the input operand, after determining that the input operand contains zero data elements; and sending the decompressed data to a PE, where the PE is configured to compute output operands based on the decompressed data.

[0194] Example 2 provides the method of Example 1, further comprising the step of generating a new sparsity bitmap for decompressed data, the new sparsity bitmap comprising a plurality of bits, each having a value of 1.

[0195] Example 3 provides the method according to Example 1 or 2, wherein the layer is selected from the group consisting of depth unit convolutional layers, group convolutional layers, element unit layers, and pooling layers.

[0196] Example 4 provides a method according to any one of Examples 1 to 3, wherein the input operand is part of a layer's IFM, the IFM includes multiple channels, and the multiple data elements in the input operand are in different channels among the multiple channels.

[0197] Example 5 provides the method of Example 4, wherein the output operand includes data elements in two or more of the different channels.

[0198] Example 6 provides a method of any one of the preceding examples, further comprising the steps of: storing output operands in a data store; and writing a subset of output operands from the data store to memory, wherein the subset of output operands includes one or more non-zero data elements within the output operands.

[0199] Example 7 provides a method of any one of the preceding examples, further comprising the step of generating a new sparsity bitmap for an output operand, the new sparsity bitmap containing multiple bits, each bit corresponding to each data element in the output operand and indicating whether each data element in the output operand is zero or non-zero.

[0200] Example 8 provides a method according to any one of the preceding examples, wherein the step of determining whether an input operand contains zero-value data elements based on the sparsity bitmap of the input operand includes the step of determining whether a bit in the sparsity bitmap is zero.

[0201] Example 9 provides a method of any one of the preceding examples, further comprising the step of determining the position of a zero-value data element in an input operand based on the position of a bit in a sparsity bitmap corresponding to a zero-value data element, wherein the multiple bits in the sparsity bitmap are in sequence.

[0202] Example 10 provides a method of any one of the preceding examples, further comprising the step of storing a sparsity bitmap in a data store, and having the step of determining whether an input operand contains zero-value data elements after the compressed data and sparsity bitmap have been stored in the data store.

[0203] Example 11 provides a compute block configured to run a layer of a DNN, the compute block comprising: a data store configured to store compressed data in a data store, the compressed data comprising one or more non-zero data elements which are a subset of the input operands of the layer; a densification module configured to determine whether an input operand contains zero data elements based on a sparsity bitmap of the input operand, where the input operand comprises multiple data elements and the sparsity bitmap comprises multiple bits, each bit corresponding to each data element in the input operand and indicating whether each data element is zero or non-zero; and after determining that the input operand contains zero data elements, the densification module is configured to generate decompressed data by inserting zero data elements into the compressed data based on the location of the zero data elements in the input operand; and a PE configured to compute an output operand based on the decompressed data.

[0204] Example 12 provides the computation block described in Example 11, wherein the densification module generates a new sparsity bitmap for the decompressed data, and the new sparsity bitmap is further configured to contain a plurality of bits, each having a value of 1.

[0205] Example 13 provides the computational block described in Example 11 or 12, wherein the layers are selected from the group consisting of depth unit convolutional layers, group convolutional layers, element unit layers, and pooling layers.

[0206] Example 14 provides a computation block described in any one of Examples 11 to 13, wherein the input operand is part of a layer's IFM, the IFM contains multiple channels, and the multiple data elements in the input operand reside in different channels among the multiple channels.

[0207] Example 15 provides the computation block described in Example 14, wherein the output operand includes data elements in two or more different channels.

[0208] Example 16 provides a computation block as described in any one of Examples 11 to 15, wherein the data store is further configured to store output operands, the computation block further comprises memory, a subset of output operands is written from the data store to memory, and the subset of output operands contains one or more non-zero data elements within the output operands.

[0209] Example 17 provides a computation block as described in any one of Examples 11 to 16, further comprising a compression module configured to generate a new sparsity bitmap for an output operand, the new sparsity bitmap containing multiple bits, each bit corresponding to a data element in the output operand and indicating whether each data element in the output operand is zero or non-zero.

[0210] Example 18 provides a computation block according to any one of Examples 11 to 17, wherein the densification module is configured to determine whether an input operand contains zero-value data elements based on the sparsity bitmap of the input operand by determining whether the bits in the sparsity bitmap are zero.

[0211] Example 19 provides a computation block described in any one of Examples 11 to 18, wherein the densification module is further configured to determine the position of zero-value data elements in the input operand based on the position of bits in a sparsity bitmap corresponding to zero-value data elements, and the multiple bits in the sparsity bitmap are sequenced.

[0212] Example 20 provides a compute block described in any one of Examples 11 to 19, wherein the datastore has multiple storage units, and the densification module resides in one of the multiple storage units.

[0213] Example 21 provides one or more non-temporary computer-readable media that store executable instructions for performing operations to run a layer of a DNN, the operations comprising: a step of storing compressed data in a datastore, the compressed data comprising one or more non-zero data elements which are a subset of the input operands of the layer; a step of determining whether an input operand contains zero data elements based on a sparsity bitmap of the input operand, the input operand comprising multiple data elements and the sparsity bitmap comprising multiple bits, each bit corresponding to each data element in the input operand and indicating whether each data element is zero or non-zero; a step of generating decompressed data by inserting zero data elements into the compressed data based on the location of the zero data elements in the input operand, after determining that the input operand contains zero data elements; and a step of sending the decompressed data to a PE, the PE configured to compute an output operand based on the decompressed data.

[0214] Example 22 provides one or more non-temporary computer-readable media as described in Example 21, wherein the operation comprises a step of generating a new sparsity bitmap for decompressed data, the new sparsity bitmap comprising a plurality of bits, each having a value of 1.

[0215] Example 23 provides one or more non-temporary computer-readable media as described in Example 21 or 22, wherein the input operand is part of a layered IFM, the IFM includes multiple channels, and the multiple data elements in the input operand are in different channels of the multiple channels; and the output operand includes data elements in two or more of the different channels.

[0216] Example 24 provides one or more non-temporary computer-readable media as described in any one of Examples 21 to 23, wherein the operation further comprises a step of generating a new sparsity bitmap for an output operand, the new sparsity bitmap comprising a plurality of bits, each bit corresponding to each data element in the output operand and indicating whether each data element in the output operand is zero or non-zero.

[0217] Example 25 provides one or more non-temporary computer-readable media according to any one of Examples 21 to 24, wherein the procedure for determining whether an input operand contains zero-value data elements, based on the sparsity bitmap of the input operand, includes a procedure for determining whether a bit in the sparsity bitmap is zero.

[0218] The above description of the exemplary embodiments of the Disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the Disclosure to the exact form disclosed. Specific embodiments and examples of the Disclosure are described herein for illustrative purposes only, but various equivalent modifications are possible within the scope of the Disclosure, as will be apparent to those skilled in the art. These modifications may be made to the Disclosure in light of the above detailed description. (Other possible items) (Item 1) A method for executing layers of a deep neural network (DNN), wherein the method is: In the step of storing compressed data in a data store, the compressed data includes one or more non-zero data elements which are a subset of the input operands of the layer, and the input operands include multiple data elements; The step of determining whether the input operand contains any zero-value data element based on the sparsity bitmap of the input operand, wherein the sparsity bitmap contains a plurality of bits, each bit corresponding to each data element in the input operand and indicating whether each data element is zero or non-zero; A step of generating decompressed data by determining that the input operand contains zero-value data elements, and then inserting the zero-value data elements into the compressed data based on their positions within the input operand; and In the step of transmitting the decompression data to a processing element, the processing element is configured to calculate an output operand based on the decompression data. A method that includes [a certain feature]. (Item 2) The method described above is The step of generating a new sparsity bitmap for the decompressed data, wherein the new sparsity bitmap includes one or more bits, each having a value of 1. The method described in item 1, further comprising the features described in item 1. (Item 3) The method according to item 1 or 2, wherein the layer is selected from the group consisting of depth unit convolution layers, group convolution layers, element unit layers, and pooling layers. (Item 4) The method according to any one of items 1 to 3, wherein the input operand is part of the input feature map of the layer, the input feature map includes one or more channels, and the one or more data elements in the input operand are in different channels among the one or more channels. (Item 5) The method according to item 4, wherein the output operand includes data elements in two or more of the different channels. (Item 6) The method described above is The step of storing the output operand in the data store; and In the step of writing a subset of the output operands from the data store to memory, the subset of the output operands includes one or more non-zero data elements within the output operands. The method described in any one of items 1 to 5, further comprising: (Item 7) The method described above is A step of generating a new sparsity bitmap for the output operand, wherein the new sparsity bitmap includes one or more bits, each of which corresponds to each data element in the output operand and indicates whether each data element in the output operand is zero or non-zero. The method according to any one of items 1 to 6, further comprising: (Item 8) The step of determining whether the input operand includes the zero-value data element based on the sparsity bitmap of the input operand is: The method according to any one of items 1 to 7, further comprising the step of determining whether or not a bit in the sparsity bitmap is zero. (Item 9) The method described above is The step further comprises determining the position of the zero-value data element in the input operand based on the position of the bit in the sparsity bitmap corresponding to the zero-value data element, The method according to any one of items 1 to 8, wherein the plurality of bits in the sparsity bitmap are sequenced. (Item 10) The method described above is The step further comprises storing the sparsity bitmap in the data store, The method according to any one of items 1 to 9, wherein the step of determining whether the input operand includes the zero-value data element is the step of determining whether the input operand includes the zero-value data element after the compressed data and the sparsity bitmap have been stored in the data store. (Item 11) A DNN accelerator configured to run layers of a deep neural network (DNN), wherein the computation block is: A data store configured to store compressed data, wherein the compressed data includes one or more non-zero data elements that are a subset of the input operands of the layer; A high-density module, Whether the input operand contains any zero-value data element is determined based on the sparsity bitmap of the input operand, wherein the input operand contains multiple data elements, the sparsity bitmap contains multiple bits, each bit corresponding to each data element in the input operand, and indicating whether each data element is zero or non-zero. After determining that the input operand contains zero-value data elements, decompressed data is generated by inserting the zero-value data elements into the compressed data based on their positions within the input operand. High-density modules configured as follows; and A processing element configured to calculate an output operand based on the decompressed data. A DNN accelerator equipped with [this feature]. (Item 12) The aforementioned high-density module A new sparsity bitmap for the decompressed data is generated, and the new sparsity bitmap includes a plurality of bits, each having a value of 1. The DNN accelerator described in item 11 is further configured as follows. (Item 13) The DNN accelerator according to item 11 or 12, wherein the layer is selected from the group consisting of depth unit convolutional layers, group convolutional layers, element unit layers, and pooling layers. (Item 14) A DNN accelerator according to any one of items 11 to 13, wherein the input operand is part of the input feature map of the layer, the input feature map includes a plurality of channels, and the plurality of data elements in the input operand are in different channels among the plurality of channels. (Item 15) The DNN accelerator according to item 14, wherein the output operand includes data elements in two or more of the different channels. (Item 16) One or more non-temporary computer-readable media storing executable instructions for performing operations to execute layers of a deep neural network (DNN), wherein the operations are: A procedure for storing compressed data in a data store, wherein the compressed data includes one or more non-zero data elements that are a subset of the input operands of the layer; A procedure for determining whether the input operand contains any zero-value data element based on the sparsity bitmap of the input operand, wherein the input operand contains multiple data elements, the sparsity bitmap contains multiple bits, each bit corresponding to each data element in the input operand, and indicating whether each data element is zero or non-zero; A procedure for generating decompressed data by determining that the input operand contains zero-value data elements, and then inserting the zero-value data elements into the compressed data based on their positions within the input operand; and A procedure for transmitting the decompression data to a processing element, wherein the processing element is configured to calculate an output operand based on the decompression data. One or more non-temporary computer-readable media comprising: (Item 17) The aforementioned operation, A procedure for generating a new sparsity bitmap for the decompressed data, wherein the new sparsity bitmap includes a plurality of bits, each having a value of 1. One or more non-temporary computer-readable media as described in item 16, comprising: (Item 18) One or more non-temporary computer-readable media according to item 16 or 17, wherein the layer is selected from the group consisting of depth unit convolutional layers, group convolutional layers, element unit layers, and pooling layers. (Item 19) The input operand is part of the input feature map of the layer, the input feature map includes one or more channels, and the one or more data elements in the input operand are in different channels of the one or more channels, one or more non-temporary computer-readable media as described in any one of items 16 to 18. (Item 20) One or more non-temporary computer-readable media as described in item 19, wherein the output operand includes data elements in two or more of the different channels. (Item 21) The aforementioned operation, A procedure for storing the output operand in the data store; and A procedure for writing a subset of the output operands from the data store to memory, wherein the subset of the output operands includes one or more non-zero data elements within the output operands. One or more non-temporary computer-readable media as described in any one of items 16 to 20, further comprising: (Item 22) The aforementioned operation, A procedure for generating a new sparsity bitmap for the output operand, wherein the new sparsity bitmap includes one or more bits, each of which corresponds to each data element in the output operand and indicates whether each data element in the output operand is zero or non-zero. One or more non-temporary computer-readable media as described in any one of items 16 to 21, further comprising: (Item 23) A procedure for determining whether the input operand contains the zero-value data element based on the sparsity bitmap of the input operand is: Procedure for determining whether a bit in the sparsity bitmap is zero or not. One or more non-temporary computer-readable media as described in any one of items 16 to 22, having the following characteristics: (Item 24) The aforementioned operation, A procedure for determining the position of the zero-value data element in the input operand based on the position of the bit in the sparsity bitmap corresponding to the zero-value data element. Furthermore, The plurality of bits in the sparsity bitmap are in sequence. One or more non-temporary computer-readable media as described in any one of items 16 to 23. (Item 25) The aforementioned operation, Procedure for storing the sparsity bitmap in the data store. Furthermore, The procedure for determining whether the input operand includes the zero-value data element comprises the procedure for determining whether the input operand includes the zero-value data element after the compressed data and the sparsity bitmap have been stored in the data store. One or more non-temporary computer-readable media as described in any one of items 16 to 24.

Claims

1. A method for executing layers of a deep neural network (DNN), wherein the method is: In the step of storing compressed data in a data store, the compressed data includes one or more non-zero data elements which are a subset of the input operands of the layer, and the input operands include a plurality of data elements; The step of determining whether the input operand contains any zero-value data elements based on the sparsity bitmap of the input operand, wherein the sparsity bitmap contains a plurality of bits, each bit corresponding to each data element in the input operand, and indicating whether each data element is zero or non-zero; A step of generating decompressed data by determining that the input operand contains zero-value data elements, and then inserting the zero-value data elements into the compressed data based on their positions within the input operand; and In the step of transmitting the decompression data to a processing element, the processing element is configured to calculate an output operand based on the decompression data. A method that includes [a certain feature].

2. The method described above is The step of generating a new sparsity bitmap for the decompressed data, wherein the new sparsity bitmap includes one or more bits, each having a value of 1. The method according to claim 1, further comprising:

3. The method according to claim 1, wherein the layer is selected from the group consisting of depth unit convolution layers, group convolution layers, element unit layers, and pooling layers.

4. The method according to claim 1, wherein the input operand is part of the input feature map of the layer, the input feature map includes one or more channels, and the one or more data elements in the input operand are in different channels among the one or more channels.

5. The method according to claim 4, wherein the output operand includes data elements in two or more of the different channels.

6. The method described above is The step of storing the output operand in the data store; and In the step of writing a subset of the output operands from the data store to memory, the subset of the output operands includes one or more non-zero data elements within the output operands. The method according to claim 1, further comprising:

7. The method described above is A step of generating a new sparsity bitmap for the output operand, wherein the new sparsity bitmap includes one or more bits, each of which corresponds to each data element in the output operand and indicates whether each data element in the output operand is zero or non-zero. The method according to claim 1, further comprising:

8. The step of determining whether the input operand includes the zero-value data element based on the sparsity bitmap of the input operand is: The method according to claim 1, further comprising the step of determining whether or not a bit in the sparsity bitmap is zero.

9. The method described above is The step further comprises determining the position of the zero-value data element in the input operand based on the position of the bit in the sparsity bitmap corresponding to the zero-value data element, The method according to claim 1, wherein the plurality of bits in the sparsity bitmap are in sequence.

10. The method described above is The step further comprises storing the sparsity bitmap in the data store, The method according to any one of claims 1 to 9, wherein the step of determining whether the input operand includes the zero-value data element is the step of determining whether the input operand includes the zero-value data element after the compressed data and the sparsity bitmap have been stored in the data store.

11. A DNN accelerator configured to execute layers of a deep neural network (DNN), wherein the DNN accelerator is A data store configured to store compressed data, wherein the compressed data includes one or more non-zero data elements which are a subset of the input operands of the layer; A high-density module, Whether the input operand contains any zero-value data element is determined based on the sparsity bitmap of the input operand, wherein the input operand contains multiple data elements, the sparsity bitmap contains multiple bits, each bit corresponding to each data element in the input operand, and indicating whether each data element is zero or non-zero. After determining that the input operand contains zero-value data elements, decompressed data is generated by inserting the zero-value data elements into the compressed data based on their positions within the input operand. A high-density module configured as follows; and A processing element configured to calculate an output operand based on the decompressed data. A DNN accelerator equipped with [this feature].

12. The aforementioned high-density module A new sparsity bitmap for the decompressed data is generated, and the new sparsity bitmap includes a plurality of bits, each having a value of 1. The DNN accelerator according to claim 11, further configured as follows.

13. The DNN accelerator according to claim 11, wherein the layer is selected from the group consisting of a depth unit convolutional layer, a group convolutional layer, an element unit layer, and a pooling layer.

14. The DNN accelerator according to any one of claims 11 to 13, wherein the input operand is part of the input feature map of the layer, the input feature map includes a plurality of channels, and the plurality of data elements in the input operand are in different channels among the plurality of channels.

15. The DNN accelerator according to claim 14, wherein the output operand includes data elements in two or more of the different channels.

16. A computer program comprising executable instructions for causing a computer's processing circuit to perform operations for executing layers of a deep neural network (DNN), wherein the operations are: A procedure for storing compressed data in a data store, wherein the compressed data includes one or more non-zero data elements which are a subset of the input operands of the layer; A procedure for determining whether the input operand contains any zero-value data element based on the sparsity bitmap of the input operand, wherein the input operand contains a plurality of data elements, the sparsity bitmap contains a plurality of bits, each bit corresponding to each data element in the input operand and indicating whether each data element is zero or non-zero; A procedure for generating decompressed data by determining that the input operand contains zero-value data elements, and then inserting the zero-value data elements into the compressed data based on their positions within the input operand; and A procedure for transmitting the decompression data to a processing element, wherein the processing element is configured to calculate an output operand based on the decompression data. A computer program that includes the following features.

17. The aforementioned operation, A procedure for generating a new sparsity bitmap for the decompressed data, wherein the new sparsity bitmap includes a plurality of bits, each having a value of 1. The computer program according to claim 16, comprising:

18. The computer program according to claim 16, wherein the layer is selected from the group consisting of depth unit convolution layers, group convolution layers, element unit layers, and pooling layers.

19. The computer program according to claim 16, wherein the input operand is part of the input feature map of the layer, the input feature map includes one or more channels, and the one or more data elements in the input operand are in different channels among the one or more channels.

20. The computer program according to claim 19, wherein the output operand includes data elements in two or more of the different channels.

21. The aforementioned operation, A procedure for storing the output operand in the data store; and A procedure for writing a subset of the output operands from the data store to memory, wherein the subset of the output operands includes one or more non-zero data elements within the output operands. The computer program according to claim 16, further comprising:

22. The aforementioned operation, A procedure for generating a new sparsity bitmap for the output operand, wherein the new sparsity bitmap includes one or more bits, each of which corresponds to each data element in the output operand and indicates whether each data element in the output operand is zero or non-zero. The computer program according to claim 16, further comprising:

23. A procedure for determining whether the input operand contains the zero-value data element based on the sparsity bitmap of the input operand is: Procedure for determining whether a bit in the sparsity bitmap is zero or not. A computer program according to claim 16, having the following characteristics.

24. The aforementioned operation, A procedure for determining the position of the zero-value data element in the input operand based on the position of the bit in the sparsity bitmap corresponding to the zero-value data element. Furthermore, The plurality of bits in the sparsity bitmap are in sequence. The computer program according to claim 16.

25. The aforementioned operation, Procedure for storing the sparsity bitmap in the data store. Furthermore, The procedure for determining whether the input operand includes the zero-value data element comprises the procedure for determining whether the input operand includes the zero-value data element after the compressed data and the sparsity bitmap have been stored in the data store. The computer program according to claim 16.

26. A computer-readable recording medium for storing a computer program according to any one of claims 16 to 25.

Images