Neural network accelerator with sparsity logic supporting various sparsity patterns and data precisions

Abstract

A data processing unit (DPU) can accelerate neural network layers based on various sparsity patterns. The DPU can decompress a compressed graph to generate a sparsity graph that indicates a structured sparsity of weights and provide the sparsity graph to a control unit in the DPU. The control unit can input the sparsity graph to a lookup table that associates the sparsity graph with one or more read indices. The lookup table can be configured for one structured sparsity pattern, but the control unit can use the lookup table for multiple structured sparsity patterns. The control unit can determine memory addresses based on each read index, read activations or weights from a data storage unit based on the memory addresses, and pass the activations / weights to a multiply-accumulate unit for computation. The control unit can gate the activations / weight passing based on unstructured sparsity of the activations. The DPU can support various data precisions.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 640464, filed April 30, 2024, entitled “Flexible Acceleration of Structured Sparsity and Combination of Structured and Unstructured Sparsity,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to neural networks (also known as “deep neural networks” or “DNNs”), and more specifically, to DNN accelerators having sparse logic that supports a variety of sparsity patterns and data precision. Background Technology

[0003] Because of their high accuracy, deep neural networks (DNNs) are widely used in a wide range of artificial intelligence applications, from computer vision to speech recognition and natural language processing. However, this high accuracy comes at the cost of enormous computational costs. DNNs have extremely high computational requirements due to the potentially large amount of computation and the vast amounts of data that need to be read and written. Therefore, techniques to improve the efficiency of DNNs are needed. Attached Figure Description

[0004] The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For ease of description, similar reference numerals denote similar structural elements. In these figures, the embodiments are shown as examples rather than as limitations.

[0005] Figure 1 Example DNNs according to various embodiments are shown.

[0006] Figure 2 Example convolutions according to various embodiments are shown.

[0007] Figures 3A-3D Various sparsity patterns of neural network tensors according to various embodiments are shown.

[0008] Figure 4 This is a block diagram of a DNN system according to various embodiments.

[0009] Figure 5 This is a block diagram of a DNN module according to various embodiments.

[0010] Figure 6 Compression of sparsity graphs according to various embodiments is shown.

[0011] Figure 7 The reduction in overhead from sparse graph compression according to various embodiments is illustrated.

[0012] Figure 8AAn example weight load path is shown in accordance with various embodiments.

[0013] Figure 8B An example activation load path is shown in accordance with various embodiments.

[0014] Figure 9 An example sparse unit is shown in accordance with various embodiments.

[0015] Figure 10 An example sparse unit array is shown in accordance with various embodiments.

[0016] Figure 11A and 11B A lookup table (LUT) in a sparsity module is shown in accordance with various embodiments.

[0017] Figure 12 Generating a memory address from a sparsity map by using a LUT is shown in accordance with various embodiments.

[0018] Figure 13A and 13B A read port is shown in accordance with various embodiments.

[0019] Figure 14A and 14B Loading activations and weights into a multiply-accumulate (MAC) unit is shown in accordance with various embodiments.

[0020] Figure 15 An example control and data flow for structured sparsity acceleration is shown in accordance with various embodiments.

[0021] Figure 16 Another example control and data flow for structured sparsity acceleration is shown in accordance with various embodiments.

[0022] Figure 17 A data path in a sparse unit operating in floating point mode is shown in accordance with various embodiments.

[0023] Figure 18 An example LUT supporting multiple structured sparsity modes is shown in accordance with various embodiments.

[0024] Figure 19 Another example LUT supporting multiple structured sparsity modes is shown in accordance with various embodiments.

[0025] Figure 20 A hybrid sparsity mode for a sparse unit is shown in accordance with various embodiments.

[0026] Figure 21 An example control and data flow for a hybrid sparsity mode in a sparse unit is shown in accordance with various embodiments.

[0027] Figure 22 is a block diagram of an ejection module according to various embodiments.

[0028] Figure 23 An example sparse encoder is shown according to various embodiments.

[0029] Figure 24 is a flowchart showing a method of performing a DNN layer according to various embodiments.

[0030] Figure 25 is a block diagram of an example computing device according to various embodiments. DETAILED DESCRIPTION

[0031] Overview The past decade has witnessed a rapid growth in data processing based on artificial intelligence (AI), in particular, based on DNNs. DNNs are widely used in the fields of computer vision, speech recognition, image and video processing, which is mainly attributed to their ability to achieve a level of accuracy beyond human level. A DNN typically includes a series of layers. A DNN layer can include one or more deep learning operations (also referred to as “neural network operations”), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax operation, pooling, element-wise operation, linear operation, non-linear operation, and so on.

[0032] Input or output data of a deep learning operation can be arranged in a data structure called a tensor. A tensor is a data structure with multiple elements in one or more dimensions. Examples of tensors include a vector (which is a one-dimensional (ID) tensor), a matrix (which is a two-dimensional (2D) tensor), a three-dimensional (3D) tensor, a four-dimensional (4D) tensor, and even higher dimensional tensors. The dimensions of a tensor can correspond to axes, e.g., axes in a coordinate system. A dimension can be measured by the number of data points along an axis. The dimensions of a tensor can define the shape of the tensor. A DNN layer can receive one or more input tensors and compute an output tensor from the one or more input tensors. Take a convolution layer as an example, an input tensor includes an activation tensor (also referred to as an “input feature map (IFM)” or “input activation tensor”) containing one or more activations (also referred to as “input elements”) and a weight tensor. The weight tensor can be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a filter bank (a 4D weight tensor). Convolution can be performed on the input activation tensor and the weight tensor in the convolution layer to compute an output activation tensor.

[0033] A 3D tensor can have three dimensions corresponding to X, Y, and Z axes in 3D space, respectively. Data elements (e.g., activations, weights, etc.) in a 3D tensor can be arranged along these axes. Each data element can be represented by an (x, y, z) coordinate, which can indicate the data element’s position in the 3D tensor. The shape and spatial size of a 3D tensor can be defined by the lengths in the three dimensions, which can be the number of data elements along the three axes. In one example, a 3D tensor with a 3 x 4 x 5 spatial size can have a total of three elements along the X axis, a total of four elements along the Y axis, and a total of five elements along the Z axis. In some embodiments, the Z axis can correspond to channels (e.g., input channels, output channels, etc.). The length along the Z axis can indicate the total number of channels in the tensor. (x, y) can represent a spatial point in the 3D tensor. Similarly, a 2D tensor can have two dimensions corresponding to X and Y axes in 2D space, respectively.

[0034] The significant increase in DNN model size and accuracy, combined with the rapid growth in computing power of execution platforms, has led to the adoption of DNN applications even in resource-constrained mobile and edge devices, which have limited energy availability. DNN models can be executed by DNN accelerators, e.g., for training or inference. A DNN accelerator can be or include one or more data processing units (DPUs). A DPU can include processing elements (PEs) capable of performing neural network operations in a neural network layer. Many neural network operations include multiply-accumulate (MAC) operations. For example, the basic operation of a convolutional layer involves MAC operations between input activations and kernel weights.

[0035] Neural network operations can be accelerated based on the sparsity of data. Sparsity refers to the presence of less relevant or less important elements. In many applications, the less relevant or less important elements are zero elements. For example, the execution of a convolutional layer can be accelerated based on the sparsity of input activations (“activation sparsity”) or the sparsity of kernel weights (“weight sparsity”) by skipping the computation of zero activations or weights. These zeros can be safely ignored during output computation, which not only helps to save energy during computation, but also speeds up the processing of a neural network layer. Many convolutional neural networks (CNNs) employ a rectified linear unit (ReLU) function as a nonlinear activation function. Since ReLU clamps all negative values to zero, it can introduce a large number of zeros in output activations, which is a major source of activation sparsity. On the weight side, sparsity can be introduced after training by pruning small magnitude values and replacing them with zeros. Techniques such as certain types of regularization can also cause weight values to approach zero during training.

[0036] Deep learning models are constantly evolving and have undergone tremendous changes. Many newer and more accurate deep learning models, such as transformers, have evolved from ReLU-based activation functions, resulting in almost negligible activation sparsity. The amount of unstructured sparsity in the weights of many models has also decreased due to current practices of pruning and fine-tuning of models along with the use of smaller floating point precision (e.g., FP8, FP4, MXFP, etc.) instead of integer precision types. While unstructured sparsity can still occur due to quantization, it can still be difficult to extract computational acceleration (and improvement in TOPS) from it because there is typically no control over the location of zeros, which frequently results in the slowest PE effect for lockstep-based DNN accelerators.

[0037] In view of this, many DNN acceleration approaches require training or fine-tuning of weights to ensure regular structured sparsity patterns, which guarantees significant benefits from sparsity acceleration. Such approaches can be facilitated by lightweight sparsity acceleration logic in DNN accelerators. They can also mitigate the slowest PE effect resulting from the presence of random sparsity in the underlying weights or activations. Many currently available DNN accelerators employ these approaches and are typically designed to support fixed structured sparsity ratios and fixed data precisions. However, these DNN accelerators cannot efficiently accelerate neural network operations with various sparsity patterns or data precisions.

[0038] Embodiments of the present disclosure can improve at least some of the above challenges and issues by providing a configurable data store to a DNN accelerator. An example DNN accelerator can include LUTs configured for different structured sparsity patterns. Data elements (e.g., non-zero data elements) can be loaded from the data store to a PE based on the output of the LUTs. When there is a combination of structured sparsity and unstructured sparsity, data loading can be further gated based on the unstructured sparsity. LUTs configured for a structured sparsity pattern and a data precision can be used for one or more other structured sparsity patterns or data precisions.

[0039] In various embodiments of the present disclosure, a DNN accelerator includes one or more DPUs that can perform operations in a DNN. A DPU can include a compute unit, a local memory, a load module, and a drain module. The compute unit can include PEs for performing computations in neural network operations. The local memory stores data used and generated by the compute unit. The load module loads data (e.g., input of a neural network layer) from the local memory to the compute unit. The load module can also be referred to as an input delivery unit. The drain module drains data (e.g., output of a neural network layer) from the compute unit to the memory. The drain module can also be referred to as an output delivery unit. A DPU can operate in a sparse mode in which neural network operations can be accelerated based on sparsity or a dense mode in which neural network operations are not accelerated based on sparsity.

[0040] In the sparse mode, a DPU can accelerate execution of a neural network layer based on structured sparsity of weights and unstructured sparsity of activations. A structured sparsity pattern can be represented by a structured sparsity ratio N:M that indicates that for every M consecutive data elements, N data elements are non-zero. N and M can be integers. Examples of structured sparsity ratios can include 1:4, 2:4, 3:4, 4:8, and so on. A neural network tensor (e.g., a kernel, a filter, etc.) can be divided into data blocks, each data block including M consecutive data elements of the neural network tensor. A data block can also be referred to as a sub-tensor of the neural network tensor. The positions of non-zero elements in different data blocks can be different. A data block can have a sparsity map that indicates the positions of non-zero elements or zero elements. A sparsity map can include a sequence of sparsity elements (each sparsity element corresponding to a respective data element in a data block) and have values indicating whether the respective data element is zero. In an example in which a sparsity element is a bit, the sparsity map is a sparsity bitmap. A structured sparsity pattern can correspond to a set of sparsity maps. For example, a structured sparsity pattern 2:4 of weights can have six sparsity bitmaps: 0011, 0110, 1100, 0101, 1010, and 1001, where a 1 bit corresponds to a non-zero weight and a 0 bit corresponds to a zero weight.

[0041] A sparsity map can be compressed into a compressed map to reduce overhead. A compressed map can include fewer elements than a corresponding sparsity map. For example, a sparsity bitmap having 4 bits can be compressed into a compressed map having 3 bits. Compression of a sparsity map can be performed by a DNN module associated with a DNN accelerator. A sparsity map can be compressed offline (e.g., before a DNN execution starts). A load module in a DPU can decompress a compressed map to recover a sparsity map and provide the sparsity map to the DPU. The load module can also load activations and weights from the local memory into a data storage unit in the DPU.

[0042] The control unit in the DPU can pass activations or weights from the data storage unit into the PEs based on the structured sparsity in the weights. The control unit can include one or more LUTs. The LUTs can be configured for a structured sparsity pattern and map the sparsity pattern of the structured sparsity pattern to read port indices (“read indices”). The read indices can represent read ports. Upon receiving the sparsity pattern, the control unit can determine one or more read indices using the LUTs that include the sparsity pattern. The control unit can also determine memory addresses from the read indices and read the activations or weights stored at the memory addresses. The LUTs configured for a structured sparsity pattern can be used for one or more other structured sparsity patterns. For example, a LUT configured for structured sparsity pattern 1 :4 can be used for structured sparsity pattern 2:4 by converting a 2:4 sparsity pattern into two 1 :4 sparsity patterns. The LUT can also be used for structured sparsity pattern 3:4 by converting a 3:4 sparsity pattern into two 3:4 sparsity patterns. The LUT can also be used for structured sparsity pattern 2:8 by splitting a 2:8 sparsity pattern into two 4-bit sparsity patterns. The control unit can also load based on unstructured sparsity gating data in the activations. For example, the control unit can prevent passing activation-weight pairs to any PE when the activation is zero. The LUTs in the control unit can include sparsity patterns corresponding to the dense mode of the DPU, such that the control unit can facilitate DPU operation in the dense mode. Dense mode operation can take (M / N) times as many cycles to compute the same workload as compared to sparse mode.

[0043] The control unit can also support various data precisions, including integer precisions (e.g., INT8) and floating point precisions (e.g., FP16, BP16, etc.). In an example, the control unit can pass data elements (e.g., activations or weights) having integer precisions to the PEs through a single data path, while passing data elements having floating point precisions to the PEs using multiple data paths. For example, a floating point data element can be split into two parts: a first part including the low byte(s) of the data element and a second part including the high byte(s) of the data element. The control unit can pass the first part to the PEs through a first data path and the second part to the PEs through a second data path.

[0044] By supporting flexible structured sparsity, the present disclosure provides a method that can reduce the overall sparsity bitmask vector overhead for smaller block (M) sizes due to a limited number of sparsity patterns. For example, for 2:4 structured sparsity, the sparsity bitmap overhead can be reduced by 25% compared to full unstructured sparsity that requires one bit per byte. The method in the present disclosure can further reduce the overhead and exploit additional power savings by exploiting unstructured sparsity in activations. Exploiting weight sparsity can result in memory savings and compute acceleration, while exploiting activation sparsity can result in memory savings and power savings due to MAC gating within the PE. The present disclosure provides low-overhead flexible structured sparsity acceleration, as well as a combination of structured and unstructured sparsity that can improve DNN accelerator performance. By supporting multiple N:M sparsity ratios for both integer and floating point operations, the DNN accelerator can achieve significantly higher TOPS, TOPS / W, and TOPS / mm2 compared to many currently available DNN accelerators, such as dense or unstructured sparsity based DNN accelerators. The additional flexibility of supporting multiple N:M sparsity ratios is introduced with a much smaller overhead. The present disclosure provides a DNN accelerator that is not only high performance but also energy efficient. Such a DNN accelerator can play a key role in edge inference for various AI applications, including imaging, video, speech, or other types of AI applications.

[0045] For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. Persons skilled in the relevant art will appreciate that the disclosure can be practiced without the specific details

[0046] Furthermore, reference is made to the accompanying drawings that form a part of this disclosure, in which are shown, by way of illustration, embodiments that can be implemented. It is to be understood that other embodiments can be utilized and structural or logical changes can be made without departing from the scope of the present disclosure. The following detailed description, therefore, is not to be taken in a limiting sense, as the scope of the disclosure is defined by the appended claims.

[0047] Various operations can be described as multiple discrete actions or operations in turn, in a manner that can most closely correspond to the drawings and description of the disclosure. However, the order of description is not to be construed as limiting, and any number of the described operations can be reordered, combined, sub-divided, eliminated, or added. Furthermore, description of an operation as a single operation does not imply that the individual operations of any such single operation are to be performed concurrently or by a single device.

[0048] For purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The term "between" when used in reference to a range of measurements includes the end points of the range of measurements.

[0049] This description uses the phrases "in an embodiment," or "in embodiments," which can each refer to one or more of the same or different embodiments. As used with respect to embodiments of the present disclosure, the terms "including," "containing," "having," and the like are synonymous, and are used inclusively, in a manner that also allows for items to be added. The present disclosure can use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" that can be used to explain the various features illustrated in the drawings by way of example. These terms are used herein for the purpose of explanation only and not of limitation, and do not imply orientation in use. The drawings are not necessarily drawn to scale. Except where otherwise noted, the use of ordinal adjectives such as "first," "second," and "third," etc., to describe a common object indicates different instances of the same object, and does not imply that the chronological, spatial, hierarchical, or any other order of sequence is intended or required.

[0050] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly used in the art to convey the substance of their work to others skilled in the art.

[0051] As described herein or known in the art, the terms "substantially," "near," "approximately," "close," and "about" generally mean within + / - 20% of the target value. Similarly, as described herein or known in the art, the terms indicating the orientation of various elements, such as "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally mean within + / - 5-20% of the target value.

[0052] Furthermore, the terms "comprise," "include," "have," or any other variant thereof are intended to cover non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but can include other elements not expressly listed or inherent to such method, process, apparatus, or DNN accelerator. Further, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or."

[0053] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for the overall invention disclosed herein. The details of one or more implementations of the subject matter described in this specification are set forth in the description below and the attached drawings.

[0054] Example DNN Figure 1 An example DNN 100 is shown in accordance with various embodiments. The DNN 100 can be executed by a DNN accelerator, such as the DNN accelerator 402 in Figure 4 An example DNN 100 is shown in accordance with various embodiments. The DNN 100 can be executed by a DNN accelerator, such as the DNN accelerator 402 in

[0055] The convolutional layers 110 summarize the presence of features in the input to the DNN 100. The convolutional layers 110 serve as feature extractors. The first layer of the DNN 100 is a convolutional layer 110. In an example, the convolutional layer 110 performs a convolution on an input tensor 140 (also referred to as an IFM 140) and a filter 150. As shown in Figure 1 The IFM 140 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 140 includes 3 input channels, each represented by a 7x7 two-dimensional (2D) matrix. The 7x7 2D matrix includes 7 input elements (also referred to as input points) in each row and 7 input elements in each column. The filter 150 is represented by a 3x3x3 3D matrix. The filter 150 includes 3 kernels, each of which can correspond to a different input channel of the IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged by column and row. A kernel can be smaller than the IFM. In an embodiment, each kernel is represented by a 3x3 2D matrix. The 3x3 kernel includes 3 weights in each row and 3 weights in each column. The weights can be initialized and updated through backpropagation using gradient descent. The magnitude of the weights can indicate the importance of the filter 150 in extracting features from the IFM 140. Figure 1

[0056] ​Convolution involves a MAC operation with the input elements in IFM 140 and the weights in filter 150. The convolution can be a standard convolution 163 or a depth-direction convolution 183. In a standard convolution 163, the entire filter 150 slides through IFM 140. All input channels are combined to produce an output tensor 160 (also called OFM 160). OFM 160 is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements (also called output points) in each row and 5 output elements in each column. For illustrative purposes, in Figure 1 In one embodiment, the standard convolution includes a filter. In embodiments with multiple filters, the standard convolution can produce multiple output channels in the OFM 160.

[0057] The multiplication applied between a kernel-sized block and a kernel of IFM 140 can be a dot product. A dot product is an element-wise multiplication between a kernel-sized block of IFM 140 and its corresponding kernel, then summed, always resulting in a single value. Because it yields a single value, this operation is often called a "scalar product." Using a kernel smaller than IFM 140 is intentional because it allows the same kernel (set of weights) to be multiplied multiple times at different points on IFM 140. Specifically, the kernel is systematically applied to each overlapping portion or kernel-sized block of IFM 140 from left to right and top to bottom. Multiplying a kernel by IFM 140 once results in a single value. Since the kernel is applied multiple times to IFM 140, the result of the multiplication is a 2D matrix of output elements. Thus, the 2D output matrix from a standard convolution 163 (i.e., OFM 160) is called OFM.

[0058] In depthwise convolution 183, the input channels are not combined. Instead, the MAC operation is performed on separate input channels and separate kernels, producing an output channel. For example... Figure 1As shown in the middle, the depthwise convolution 183 produces a depthwise output tensor 180. The depthwise output tensor 180 is represented by a 5x5x3 3D matrix. The depthwise output tensor 180 includes 3 output channels, each represented by a 5x5 2D matrix. The 5x5 2D matrix includes 5 output elements in each row and 5 output elements in each column. Each output channel is the result of MAC operations of an input channel of the IFM 140 and a kernel of the filter 150. For example, the first output channel (patterened with dots) is the result of MAC operations of the first input channel (patterened with dots) and the first kernel (patterened with dots), the second output channel (patterened with horizontal bars) is the result of MAC operations of the second input channel (patterened with horizontal bars) and the second kernel (patterened with horizontal bars), and the third output channel (patterened with diagonal bars) is the result of MAC operations of the third input channel (patterened with diagonal bars) and the third kernel (patterened with diagonal bars). In such a depthwise convolution, the number of input channels is equal to the number of output channels, and each output channel corresponds to a different input channel. The input channels and the output channels are collectively referred to as depthwise channels. After the depthwise convolution, a pointwise convolution 193 is then performed on the depthwise output tensor 180 and a 1x1x3 tensor 190 to produce the OFM 160.

[0059] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is ReLU. ReLU is a computation that returns the value provided as input directly, or a value of zero if the input is zero or less. The convolutional layer 110 can receive several images as input and compute the convolution of each image with each kernel. This process can be repeated several times. For example, the OFM 160 is passed to a subsequent convolutional layer 110 (i.e., a convolutional layer 110 that follows the convolutional layer 110 that generated the OFM 160 in the sequence). The subsequent convolutional layer 110 performs a convolution on the OFM 160 with new kernels and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be kernelized again by further subsequent convolutional layers 110, and so on.

[0060] In some embodiments, convolutional layer 110 has four hyperparameters: number of kernels, kernel size F (e.g., a kernel has a dimension of F x F x D pixels), S steps that the window corresponding to the kernel is dragged over the image (e.g., one step means moving the window by one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thickness to the input image of convolutional layer 110). Convolutional layer 110 can perform various types of convolutions, such as two-dimensional convolution, dilated or contracting convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. DNN 100 includes 16 convolutional layers 110. In other embodiments, DNN 100 can include a different number of convolutional layers.

[0061] Pooling layer 120 down-samples the feature maps generated by the convolutional layers, for example, by summarizing the presence of features in blocks of the feature maps. Pooling layer 120 is located between two convolutional layers 110: a preceding convolutional layer 110 (the convolutional layer 110 that precedes pooling layer 120 in the layer sequence) and a subsequent convolutional layer 110 (the convolutional layer 110 that follows pooling layer 120 in the layer sequence). In some embodiments, pooling layer 120 is added after convolutional layer 110, for example, after an activation function (e.g., ReLU, etc.) has been applied to OFM 160.

[0062] Pooling layer 120 receives the feature maps generated by the preceding convolutional layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics. Thus, the pooling operation improves the efficiency of the DNN and avoids overfitting. Pooling layer 120 can perform the pooling operation by average pooling (computing the average value of each block on the feature map), max pooling (computing the maximum value of each block of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is applied in 2 x 2 pixels with a stride of two pixels, such that the pooling operation reduces the size of the feature map by a factor of two, e.g., the number of pixels or values in the feature map is reduced to one quarter of the size. In an example, a pooling layer 120 applied to a 6 x 6 feature map results in a 3 x 3 output pooled feature map. The output of pooling layer 120 is input to the subsequent convolutional layer 110 for further feature extraction. In some embodiments, pooling layer 120 operates on each feature map individually to create a new set of the same number of pooled feature maps.

[0063] The fully connected layer 130 is the last layer of the DNN. The fully connected layer 130 can or can not be convolutional. The fully connected layer 130 receives an input operand. The input operand defines the outputs of the convolutional layers 110 and the pooling layers 120 and includes the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully connected layer 130 applies a linear combination and an activation function to the input operand and generates a vector. The vector can contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Thus, each element is between 0 and 1 and the sum of all is equal to 1. These probabilities are computed by the last fully connected layer 130 by using a logistic function (for binary classification) or a SoftMax function (for multi-class classification) as the activation function. In some embodiments, the fully connected layer 130 multiplies each input element by a weight, sums, and then applies an activation function (e.g., logistic function if N = 2, SoftMax function if N > 2). This is equivalent to multiplying the input operand by a matrix containing the weights.

[0064] Figure 2 An example convolution is shown in accordance with various embodiments. The convolution can be a deep learning operation in a convolutional layer of a DNN (e.g., the convolutional layer 110 in Figure 1 Figure 4 The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator. An example of a DNN accelerator can be the DNN accelerator 402 in

[0065] The activation tensor 210 can be computed in a previous layer of the DNN. In some embodiments (e.g., embodiments in which the convolutional layer is the first layer of the DNN), the activation tensor 210 can be an image. In Figure 2 In embodiments in which the activation tensor 210 includes activations (also referred to as “input activations,” “elements,” or “input elements”) arranged in a 3D matrix. The activation tensor 210 can also be referred to as an input tensor of the convolution. An input element is a data point in the activation tensor 210. The activation tensor 210 has a spatial size H in × W in × C in where H in is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix for each input channel), W in ​is the width of the 3D matrix (i.e., the length along the X-axis, which indicates the number of activations in a row in the 2D matrix for each input channel), and C in is the depth of the 3D matrix (i.e., the length along the Z-axis, which indicates the number of input channels). For simplicity and illustration purposes, the activation tensor 210 has a spatial size of 7 x 7 x 3, i.e., the activation tensor 210 includes three input channels, and each input channel has a 7 x 7 2D matrix. Each input element in the activation tensor 210 can be represented by an (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the activation tensor 210 can be different.

[0066] Each filter 220 includes weights arranged in a 3D matrix. The values of the weights can be determined by training the DNN. The filter 220 has a spatial size of H f x W f x C f where H f is the height of the filter (i.e., the length along the Y-axis, which indicates the number of weights in a column in each kernel), W f is the width of the filter (i.e., the length along the X-axis, which indicates the number of weights in a row in each kernel), and C f is the depth of the filter (i.e., the length along the Z-axis, which indicates the number of channels). In some embodiments, C f is equal to C in For simplicity and illustration purposes, Figure 2 Each filter 220 in the set of filters 220 has a spatial size of 2 x 3 x 3, i.e., the filter 220 includes 2 convolutional kernels with a spatial size of 2 x 3. In other embodiments, the height, width, or depth of the filter 220 can be different. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix for each input channel in the activation tensor 210.

[0067] An activation or a weight can occupy one or more bytes in memory. The number of bytes for an activation or a weight can depend on the data format. For example, when an activation or a weight has an INT8 format, the activation occupies one byte. When an activation or a weight has an FP16 format, the activation or the weight occupies two bytes. Other data formats can be used for an activation or a weight.

[0068] In a convolution, each filter 220 slides through the activation tensor 210 and generates a 2D matrix for an output channel in an output tensor 230. In Figure 2In some embodiments, the 2D matrix has a spatial size of 5 x 5. The output tensor 230 includes activations (also referred to as “output activations,” “elements,” or “output elements”) arranged in a 3D matrix. An output activation is a data point in the output tensor 230. The output tensor 230 has a spatial size of H out x W out x C out where H out is a height of the 3D matrix (i.e., a length along the Y-axis that indicates a number of output activations in a column in the 2D matrix for each output channel), W out is a width of the 3D matrix (i.e., a length along the X-axis that indicates a number of output activations in a row in the 2D matrix for each output channel), and C out is a depth of the 3D matrix (i.e., a length along the Z-axis that indicates a number of output channels). C out may equal a number of filters 220 in the convolution. H out and W out may depend on each filter 220 and a height and a weight of the activation tensor 210. In an example in which a kernel size is 1 x 1, H out and W out may equal H in and W in respectively.

[0069] As part of the convolution, a MAC operation can be performed on each filter 220 and the activation tensor 210 on a 2 x 3 x 3 sub-tensor 215 (which is highlighted in dashed lines in Figure 2 The result of the MAC operation on the sub-tensor 215 and one filter 220 is an output activation. In some embodiments (e.g., embodiments in which the convolution is an integral convolution), the output activation can include 8 bits, such as one byte. In other embodiments (e.g., embodiments in which the convolution is a floating point convolution), the output activation can include more than one byte. For example, the output element can include two bytes.

[0070] After the MAC operations on the sub-tensor 215 and all filters 220 are completed, a vector 235 is produced. The vector 235 is arranged in a 3D matrix in the output tensor 230. Figure 2The point-wise pattern is highlighted in bold. Vector 235 includes a sequence of output activations arranged along the Z axis. The output activations in vector 235 have the same (x, y) coordinates, but the output activations correspond to different output channels and have different Z coordinates. The dimension of vector 235 along the Z axis can be equal to the total number of output channels in output tensor 230. After vector 235 is produced, further MAC operations are performed to produce additional vectors until output tensor 230 is produced. In some embodiments, the MAC operations are performed by MAC units 370. In some embodiments, the MAC operations are performed by a single MAC unit that is shared among the MAC units 370. In some embodiments, the MAC operations are performed by a single MAC unit that is shared among the MAC units 370 and other MAC units in the system. Figure 2 In embodiments in which output tensor 230 is computed in Z-major format, a vector adjacent to vector 235 along the X axis can be computed immediately after vector 235 when output tensor 230 is computed in ZXY format. A vector adjacent to vector 235 along the Y axis can be computed immediately after vector 235 when output tensor 230 is computed in ZYX format. Output tensor 230 can be permuted, e.g., by permutation module 390, and stored in memory (e.g., local memory 440) in X-major format or Y-major format.

[0071] In some embodiments, the MAC operations on 3x3x3 sub-tensors (e.g., sub-tensor 215) and filter 220 can be performed by multiple MAC units. One or more MAC units can receive input operands (e.g., activation operands 217 shown in Figure 2 Figure 2 In some embodiments, the MAC operations on 3x3x3 sub-tensors (e.g., sub-tensor 215) and filter 220 can be performed by multiple MAC units. One or more MAC units can receive input operands (e.g., activation operands 217 shown in

[0072] The activations or weights can be floating-point numbers. The floating-point numbers can have various data formats, such as FP32, FP16, BF16, etc. The floating-point numbers can be positive or negative numbers with a decimal point. The floating-point numbers can be represented by a bit sequence that includes one or more bits representing the sign (e.g., positive or negative) of the floating-point number, bits representing the exponent of the floating-point number, and bits representing the mantissa of the floating-point number. The mantissa is the part of the floating-point number that represents the significant digits of the floating-point number. The mantissa is multiplied by the base raised to the power of the exponent to get the actual value of the floating-point number.

[0073] ​In some embodiments, the output activations in the output tensor 230 can be further processed based on one or more activation functions before being written to memory or input to the next layer of the DNN. The processing based on one or more activation functions can be at least part of the post-processing of the convolution. In some embodiments, the post-processing can include one or more other computations, such as offset computation, bias computation, etc. The result of the post-processing can be stored in the local memory of the compute block and used as input to the next layer of the DNN. In some embodiments, the input activations in the activation tensor 210 can be the result of the post-processing of the previous layer of the DNN.

[0074] Figures 3A-3D Various sparsity patterns of a neural network tensor are shown in accordance with various embodiments. The neural network tensor can be an input activation tensor (e.g., the activation tensor 210), a weight tensor (e.g., the filter 220), an output activation tensor (e.g., the output tensor 230), or a sub-tensor thereof. The sparsity pattern can be a structured sparsity pattern with a fixed sparsity ratio of N:M, or an unstructured sparsity pattern. The structured sparsity pattern has a fixed number of N non-zero elements within each unit block of data with a fixed number of M additional elements. The unstructured sparsity pattern can have zero elements at random locations.

[0075] Figure 3A A tensor 310 with structured sparsity 2:4 is shown. For illustration and simplicity purposes, the tensor 310 is a 2D tensor with a spatial size of 8x8. Each data element in the tensor 310 is represented by a square with a dashed boundary. The squares representing non-zero data elements are filled with a dotted pattern. The squares representing zero data elements are empty. The tensor 310 is divided into 16 sub-tensors 315 (individually referred to as “sub-tensor 315”), each with a spatial size of 4x1. Each sub-tensor 315 is represented by a square with a solid boundary. Each sub-tensor 315 has a total of four elements, two of which are non-zero elements and the other two are zero elements. The locations of the non-zero elements or zero elements in the sub-tensors 315 can be different.

[0076] Figure 3BTensor 320 with structured sparsity of 4:8 is shown. For illustrative and simplification purposes, tensor 320 is a 2D tensor with a spatial size of 8×8. Each data element in tensor 320 is represented by a square with dashed borders. Squares representing non-zero data elements are filled with a dotted pattern. Squares representing non-zero data elements are empty. Tensor 320 is divided into eight sub-tensors 325 (referred to separately as "sub-tensors 325"), each with a spatial size of 8×1. Each sub-tensor 325 is represented by a square with solid borders. Each sub-tensor 325 has a total of eight elements, four of which are non-zero elements and the other four are zero elements. The positions of the non-zero or zero elements in sub-tensors 325 can vary.

[0077] Figure 3C Tensor 330 with structured sparsity 2:8 is shown. For illustrative and simplification purposes, tensor 330 is a 2D tensor with a spatial size of 8×8. Each data element in tensor 330 is represented by a square with dashed borders. Squares representing non-zero data elements are filled with a dotted pattern. Squares representing non-zero data elements are empty. Tensor 330 is divided into 16 sub-tensors 335 (referred to separately as "sub-tensors 335"), each with a spatial size of 8×1. Each sub-tensor 335 is represented by a square with solid borders. Each sub-tensor 335 has a total of four elements, two of which are non-zero elements, and the other six are zero elements. The positions of non-zero or zero elements in sub-tensors 335 can be different. Although... Figures 3A-3C Three structured sparsity patterns are shown, but neural network tensors can have other structured sparsity patterns. In some embodiments, tensors 310, 320, or 330 can be weight tensors, and the structured sparsity pattern can be achieved by performing structure-sparseness-based pruning on the weight tensors.

[0078] Figure 3D Tensor 340 with unstructured sparsity is shown. For illustrative and simplification purposes, tensor 340 is a 2D tensor with a spatial size of 8×8. Each data element in tensor 340 is represented by a square with dashed borders. Squares representing non-zero data elements are filled with a dotted pattern. Squares representing non-zero data elements are empty. Figure 3D As shown, the positions of zero data elements are random. Tensor 340 does not have any structured sparsity pattern. In some embodiments, tensor 340 may be an activation tensor, which may be the input or output of a neural network layer.

[0079] Example DNN system Figure 4is a block diagram of a DNN system 400 according to various embodiments. The entire DNN system 400 or a portion of the DNN system 400 can be implemented in one or more computing devices, such as the computing device 1700 in FIG. Figure 17 The DNN system 400 can generate and execute DNNs, such as transformer-based models, convolution-based models, and the like. As shown in FIG. Figure 4 The DNN system 400 includes a DNN module 401 and a DNN accelerator 402. In other embodiments, alternative configurations, different, or additional components can be included in the DNN system 400. For example, the DNN system 400 can include multiple DNN modules or multiple DNN accelerators. Moreover, the functionality attributed to the components of the DNN system 400 can be implemented by different components or different systems included in the DNN system 400. In some embodiments, the DNN module 401 and the DNN accelerator 402 can include different types of processing elements. In an example, the DNN module 401 can be implemented by one or more central processing units (CPUs). The DNN accelerator 402 can also be referred to as an AI accelerator or an AI processor. The DNN module 401 and the DNN accelerator 402 can be implemented in the same chip or separate chips.

[0080] The DNN module 401 facilitates the generation and deployment of DNNs. In some embodiments, the DNN module 401 can generate and train DNNs. For example, the DNN module 401 can define the hierarchical architecture of a DNN. The DNN module 401 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 401 can also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the level of sparsity of one or more deep learning tensors of the DNN.

[0081] The DNN module 401 can also compress the DNN, for example, during or after training. In some embodiments, the DNN module 401 can prune weights in one or more layers of the DNN by changing non-zero valued weights to zero. The DNN module 401 can prune the weights based on a target weight sparsity ratio. The weight sparsity ratio can be the ratio of the number of zero-valued weights to the total number of weights. In examples where the DNN module 401 prunes weights during DNN training, the DNN module 401 can prune the weights of a layer after one or more epochs to achieve the target sparsity ratio. The DNN module 401 can prevent the pruned weights from changing values during the remaining training process. Alternatively, the DNN module 401 can allow the pruned weights to change values such that the pruned zero weights can have non-zero values after further training. The DNN module 401 can prune the weights of the layer again after one or more additional epochs.

[0082] The DNN module 401 can deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, the DNN module 401 can distribute trained, compressed, or validated DNNs to devices or systems for which the DNNs were trained, which can use the DNNs to perform tasks (e.g., image classification, motion planning, etc.). In other embodiments, the DNN module 401 can use the DNN accelerator 402 to facilitate deployment of DNNs. For example, the DNN module 401 can receive data from a device or system coupled with the DNN system 400 and input the received data (or data generated by the DNN module 401, e.g., based on the received data) into a DNN. The DNN module 401 can generate instructions (e.g., configuration descriptors) that control operation of the DNN accelerator 402 during execution of the DNN. The DNN module 401 can receive output of the DNN from the DNN accelerator 402. The DNN module 401 can transmit the output of the DNN (or results of processing the output of the DNN by the DNN module 401) to the device or system. In some embodiments, the DNN module 401 can control the execution process of a trained, compressed, or validated DNN. The DNN module 401 can act as a compiler for a DNN executed by the DNN accelerator 402. The DNN module 401 can perform compilation of a DNN and generate configuration descriptors or configuration parameters based on which the DNN can be executed. Details of the DNN module 401 are described below in connection with FIGS. 5-7. Figure 5 Certain aspects of the DNN module 401 are provided.

[0083] The DNN accelerator 402 executes DNNs provided by the DNN module 401. For example, the DNN accelerator 402 can execute a DNN by performing neural network operations in the DNN. The process of performing a neural network operation is also referred to as running or conducting the process of the neural network operation. The execution of a DNN can be for training the DNN or for using the DNN to perform an AI task. As described below, the DNN accelerator 402 can include a memory 410, a DMA (direct memory access) engine 420, and a DPU 430 (referred to individually as “DPU 430”). In other embodiments, alternative configurations, different, or additional components can be included in the DNN accelerator 402. For example, the DNN accelerator 402 can include more than one memory 410 or DMA engine 420. As another example, the DNN accelerator 402 can include a single DPU 430. Furthermore, the functionality attributed to the components of the DNN accelerator 402 can be accomplished by different components included in the DNN accelerator 402 or by different systems. The components of the DNN accelerator 402 can be implemented in hardware, software, firmware, or some combination thereof. Figure 4

[0084] ​Memory 410 stores data associated with neural network operations performed by DNN accelerator 402. In some embodiments, memory 410 can store data to be used by DPU 430 for DNN execution. Memory 410 can store weights, such as weights of a convolutional layer determined by training a DNN. Memory 410 can further store inputs to a DNN layer or outputs of a DNN layer, such as data generated by DPU 430 from performing a deep learning operation in a DNN. Example neural network operations include a convolution (also referred to as a “convolution operation”), a layer normalization operation, a SoftMax operation, a matrix multiplication operation, a pooling operation, an element-wise operation, an activation function, other types of deep learning operations, or some combination thereof. Memory 410 can also store configuration descriptors, including configuration descriptors that configure data storage devices in DNN accelerator 402. Memory 410 can be a main memory of DNN accelerator 402. In some embodiments, memory 410 includes one or more dynamic random access memories (DRAMs).

[0085] DMA engine 420 facilitates data transfers between memory 410 and local memory of DPU 430. For example, DMA engine 420 can read data from memory 410 and write the data to local memory of DPU 430. As another example, DMA engine 420 can read data from local memory of DPU 430 and write the data to memory 410. DMA engine 420 provides DMA features that allow DPU 430 to initiate data transfers between memory 410 and local memory of DPU 430 and perform other operations while the data transfers are in progress. In some embodiments, DMA engine 420 can read a tensor from memory 410, modify the tensor in a manner optimized for DPU 430 before DMA engine 420 writes the tensor to local memory of DPU 430.

[0086] DPU 430 performs neural network operations in a DNN. DPU 430 can also be referred to as a neural processing element, a compute block, or a tile. In some embodiments, DPU 430 can execute a DNN layer by running one or more neural network operations in the DNN layer. For example, DPU 430 can compute an output of a DNN layer from an input of the DNN layer. In some embodiments, DPU 430 can also use internal parameters (e.g., weights) of the DNN layer to compute the output. DPU 430 can execute one layer or a portion of a layer at a time. In some embodiments, operations of a DNN layer can be run in parallel by multiple DPU 430. For example, multiple DPU 430 can each execute a portion of a workload of a deep learning operation. Data can be shared among DPU 430. DPU 430 can also be referred to as a neural processing unit, a compute block, or a compute tile.

[0087] DPU 430 is capable of running various types of neural network operations, such as convolution, layer normalization, SoftMax operation, pooling, element-wise operation, linear operation, non-linear operation, and the like. Deep learning operations performed by DPU 430 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, DPU 430 receives an input tensor and one or more convolution kernels and performs a convolution with the input tensor and the convolution kernels. The result of the convolution can be an output tensor, which can be further computed, e.g., by DPU 430 or another DPU 430.

[0088] In Figure 4 embodiments, each DPU 430 includes a local memory 440, a load module 450, a compute unit 460 including a processing engine 470 and a post-processing engine 480, and an ejection module 490. Some or all of the components of DPU 430 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components can be included in DPU 430. Moreover, the functionality attributed to the components of DPU 430 can be implemented by different components included in DPU 430, a different DPU 430, another component of DNN accelerator 402, or a different system. The components of DPU 430 can be implemented in hardware, software, firmware, or some combination thereof.

[0089] Local memory 440 is local to the corresponding DPU 430. In Figure 4 embodiments, local memory 440 is internal to DPU 430. In other embodiments, local memory 440 can be external to DPU 430. Data in local memory 440 can be transferred to or from memory 410, e.g., by DMA engine 420. In some embodiments, data in local memory 440 can be transferred to or from the local memory of another DPU 430. Local memory 440 can store data received by, used by, or generated by load module 450, processing engine 470, post-processing engine 480, or ejection module 490. Examples of data can include input activations, weights, output activations, sparsity bitmaps, and the like.

[0090] In some embodiments, local memory 440 can store tensors to be processed by processing engine 470 or post-processing engine 480. The tensors can be input tensors of a deep learning operation. Local memory 440 can also store tensors generated by processing engine 470 or post-processing engine 480. The tensors can be output tensors of a deep learning operation. The layout of data points of a tensor in local memory 440 can depend on the format in which the tensor is stored. In some embodiments, local memory 440 can store tensors in various formats, including Z-major format, X-major format, and Y-major format. For tensors having Z-major format, local memory 440 can store data points having the same (x, y) coordinates contiguously. For example, data points having the same (x, y) coordinates can be stored at a series of memory addresses in local memory 440. For tensors having ZXY format or ZYX format, local memory 440 can store data points having the same (x, y) coordinates contiguously. For example, data points having the same (x, y) coordinates can be stored at a series of memory addresses in local memory 440. For tensors having X-major format, local memory 440 can store data points having the same (y, z) coordinates contiguously. For tensors having Y-major format, local memory 440 can store data points having the same (x, z) coordinates contiguously.

[0091] In some embodiments, local memory 440 can store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and the like. A dense tensor can be a tensor in which zero elements, if any, have not been removed. A dense tensor can be converted to a sparse tensor by removing one or more zero elements in the dense tensor. A sparse tensor can also be referred to as a compressed tensor or a packed tensor.

[0092] Local memory 440 can also store a sparsity map. In some embodiments, the sparsity map is generated in a sparsity encoding process in which a dense tensor is compressed to a sparse tensor. Each sparsity element in the sparsity map can correspond to a different element in the dense tensor and indicate whether the element in the dense tensor is zero. The sparsity map can indicate the location of elements of the sparse tensor in the dense tensor. The sparsity map can be a sparsity bitmap, each element of which is a bit. In some embodiments, local memory 440 can store a compressed version of the sparsity map, i.e., a compressed map.

[0093] In some embodiments, local memory 440 includes one or more static random access memories (SRAMs). Local memory 440 can be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, local memory 440 can include memory banks. The number of data banks in local memory 440 can be 16, 64, 128, 256, 512, 1024, 2048, or other number. A memory bank can include a number of memory locations. In an example, a data bank can include 8, 16, 64, or a different number of memory locations. A memory bank or memory location in a memory bank can have a memory address. In an example, a memory location can store a single byte, and data larger than a single byte can be stored in memory locations with contiguous memory addresses, i.e., adjacent memory locations. For example, a memory location can store an integer in INT8 format, while storing a number in FP16 or BF16 format with 16 bits can require two memory locations. In some embodiments, 16 bits can be transferred from local memory 440 in a single read cycle. In other embodiments, 16 bits can be transferred from local memory 440 in multiple read cycles, e.g., two cycles.

[0094] Load module 450 loads data from local memory 440 to compute unit 460. Load module 450 can read tensors from local memory 440. The tensors can include activation tensors, weight tensors, activation sparsity maps, weight sparsity maps, combined sparsity maps, and the like. In some embodiments, load module 450 can load data based on the operational mode of DPU 430. Load module 450 can select different data to transfer to compute unit 460 in different sparsity modes. For example, load module 450 can transfer activation sparsity maps and weight sparsity maps of a layer to compute unit 460 in combined sparsity mode, transfer activation sparsity maps but not weight sparsity maps to compute unit 460 in activation sparsity mode, and transfer weight sparsity maps but not activation sparsity maps to compute unit 460 in weight sparsity mode. In dense mode, load module 450 does not transfer activation sparsity maps or weight sparsity maps to compute unit 460.

[0095] In some embodiments, load module 450 can process (e.g., densify) data stored in local memory 440 and then provide the data to compute unit 460. In some embodiments, load module 450 can convert compressed maps read from local memory 440 to sparsity maps and transfer the sparsity maps but not the compressed maps to compute unit 460. In some embodiments, load module can densify a sparse tensor by adding one or more zeros to the sparse tensor based on one or more sparsity maps.

[0096] In an example, when operating in the weight sparsity mode, the loading module 450 can densify the sparse activation tensor based on the corresponding activation sparsity map to generate a dense activation tensor. For example, the loading module 450 can add one or more zeros to the sparse activation tensor based on the activation sparsity map associated with the sparse activation tensor to generate the dense activation tensor. The dense activation tensor includes one or more elements compared to the sparse activation tensor. The value of the additional element(s) is zero. The loading module 450 can identify one or more elements in the activation sparsity map that correspond to the zero element(s), determine the location of each of the zero-value element(s) in the dense activation tensor, and insert the zero element(s) into the sparse activation tensor based on the determined locations. After the densification, the loading module 450 can communicate the dense activation tensor to the compute unit 460. The loading module 450 can also communicate the corresponding sparse weight tensor and the weight sparsity map to the compute unit 460. The activation sparsity map of the dense activation tensor can not be loaded to the compute unit 460.

[0097] In another example, when operating in the activation sparsity mode, the loading module 450 can densify the sparse weight tensor based on the corresponding weight sparsity map by inserting zeros into the sparse weight tensor to generate a dense weight tensor. The densification of the sparse weight tensor can be similar to the densification of the sparse activation tensor described above. After the densification, the loading module 450 can communicate the dense weight tensor to the compute unit 460. The loading module 450 can also communicate the corresponding sparse activation tensor and the activation sparsity map to the compute unit 460. The weight sparsity map of the dense weight tensor can not be loaded to the compute unit 460.

[0098] In yet another example, when operating in the dense mode, the loading module 450 can densify both the sparse weight tensor and the sparse activation tensor. The loading module 450 can generate the input tensor and the weight tensor for the layer and communicate these tensors to the processing engine 470 for execution of the layer without sparsity acceleration. The following describes certain aspects of the loading module 450. Figure 8A and 8B In some embodiments, the loading module 450 can be configured to operate in the weight sparsity mode, the activation sparsity mode, and the dense mode. In the weight sparsity mode, the loading module 450 can densify the sparse weight tensor based on the weight sparsity map to generate a dense weight tensor. In the activation sparsity mode, the loading module 450 can densify the sparse activation tensor based on the activation sparsity map to generate a dense activation tensor. In the dense mode, the loading module 450 can generate the input tensor and the weight tensor for the layer and communicate these

[0099] The processing engine 470 performs operations in the DNN. The processing engine 470 can accelerate neural network operations based on the sparsity of the data. In some embodiments, the processing engine 470 can operate in a dense mode without performing sparsity acceleration. The processing engine 470 can include one or more processing units. In some embodiments, the processing units can be arranged in one or more rows and one or more columns in the processing engine 470. Each processing unit can include a PE that can be arranged in an array including rows and columns. All of the PEs in the processing engine 470 can constitute a larger array including more rows and columns.

[0100] One example PE can be or can include one or more MAC units capable of performing MAC operations. In some embodiments (e.g., embodiments in which DPU 430 performs a convolution layer), the computation in a MAC unit can be a MAC operation on an activation operand and a weight operand. The activation operand can be an activation tensor, which can include one or more activations in an input tensor of the convolution. Different activations can be in different input channels. The weight operand can be a weight tensor, which can include one or more weights in a filter of the convolution. The values of the weights are determined by training the DNN. The weights in the weight operand can be in different input channels.

[0101] In some embodiments, a MAC unit includes one or more multipliers for performing multiplications. A MAC unit can also include one or more adders for performing accumulations. A column of MAC units is referred to as a MAC column. A MAC column can be associated with one or more MAC channels. A MAC channel is a path for loading data into a MAC column, e.g., by loading module 450. A MAC channel can also be referred to as a data channel or a data path. A MAC column can have multiple MAC channels. The loading bandwidth of a MAC column is the aggregate of the loading bandwidths of all MAC channels associated with the MAC column. With a certain number of MAC channels, data can be fed into the same number of independent MAC units simultaneously. In some embodiments, a MAC column has four MAC channels for feeding activations or weights into the MAC column, and each MAC channel can have a bandwidth of 16 bytes, which four MAC channels can have a total loading bandwidth of 64 bytes.

[0102] In some embodiments, processing engine 470 is capable of depthwise convolution, standard convolution, or both. In depthwise convolution, a MAC unit can perform a MAC operation that includes a sequence of multiplications of an input operand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operand and a different weight in the weight operand. The activation and the weight in the same cycle can correspond to the same channel. The sequence of multiplications produces a product operand that includes a sequence of products. The MAC operation can also include an accumulation in which multiple product operands are accumulated to produce an output operand of the MAC unit. Processing engine 470 can output multiple output operands at a time, each generated by a different MAC unit. In standard convolution, a MAC operation can include an accumulation across channels. For example, instead of generating an output operand, a MAC unit can accumulate products over different channels to generate a single output point.

[0103] In some embodiments, the processing engine 470 can perform MAC operations in quantized deep learning operations, such as MAC operations in quantized convolutions. In some embodiments, a MAC unit in the processing engine 470 can receive quantized activations and quantized weights and compute a quantized MAC result. The quantized MAC result can be a quantized value in integer format, and can be the output of the MAC unit. In some embodiments, the MAC unit can also include a quantized multiplier that can multiply a quantization scale with the quantized MAC result, and the output of the MAC unit can be a real value in floating point format. The MAC unit can not include a quantized subtractor because MAC operations in quantized deep learning operations do not require zero point offset.

[0104] In some embodiments, the processing engine 470 can include sparsity acceleration logic to facilitate computation acceleration in the processing engine 470 based on sparsity. For example, the processing engine 470 can include one or more sparsity modules. In an example, each MAC column or each MAC row can have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In another example, each MAC unit can have a corresponding sparsity module to accelerate MAC operations in the MAC unit. In some embodiments, the sparsity modules accelerate computations in the processing engine 470 based on sparsity of activations, sparsity of weights, or a combination of both.

[0105] The activation sparsity map can be a sparsity map of the activation tensor and have the same number of sparsity elements as the activation tensor. A sparsity element in the activation sparsity map can indicate whether a corresponding element in the activation tensor is zero. For example, a zero-valued sparsity element in the activation sparsity map can indicate that the corresponding element in the activation tensor is zero. A 1-valued sparsity element in the activation sparsity map can indicate that the corresponding element in the activation tensor is non-zero. The weight sparsity map can be a sparsity map of the weight tensor and have the same number of elements as the weight tensor. A sparsity element in the weight sparsity map can indicate whether a corresponding element in the weight tensor is zero. For example, a zero-valued sparsity element in the weight sparsity map can indicate that the corresponding element in the weight tensor is zero. A 1-valued sparsity element in the weight sparsity map can indicate that the corresponding element in the weight tensor is non-zero. In some embodiments, each sparsity element in a sparsity map can be a bit, and the sparsity map can be referred to as a sparsity bitmap.

[0106] The sparsity module can use one or more sparsity maps to communicate activations and weights to the MAC units. For example, the sparsity module can use a sparsity map to read activations or weights from a data storage unit associated with a MAC unit and communicate these activations or weights to the MAC unit. In some embodiments, the sparsity module can input a sparsity map to a LUT and use the output of the LUT to determine a memory address for an activation or weight. The sparsity module can include LUTs configured for various structured sparsity patterns. The sparsity module can also facilitate acceleration based on a combination of structured sparsity (e.g., structured sparsity of weights) and unstructured sparsity (e.g., unstructured sparsity of activations). In some embodiments, the sparsity module can support various data precisions, such as integer precision, floating point precision, and the like. The following describes certain aspects of the sparsity module 430 in conjunction with Figure 9 and 10 Certain aspects of the processing engine 470 are described below.

[0107] The post-processing engine 480 processes the output of the processing engine 470. The post-processing engine 480 can include one or more post-processing elements. In some embodiments, the post-processing elements in the post-processing engine 480 can be arranged in an array having rows and columns. In some embodiments, the post-processing engine 480 computes an activation function. The post-processing engine 480 can receive the output of the processing engine 470 as input to the activation function. In addition to or in lieu of the activation function, the post-processing engine 480 can perform other types of post-processing on the output of the processing engine 470. For example, the post-processing engine 480 can apply a bias to the output of the processing engine 470. In some embodiments, the post-processing engine 480 can be bypassed for certain DNN layers.

[0108] The drain module 490 drains data from the compute units 460 and writes the data to the local memory 440. In some embodiments, the drain module 490 can drain data on a unit level. For each processing unit, the drain module 490 can drain the output of the PEs in the processing unit based on the row index or column index of each PE. For example, the drain module 490 can drain data from the processing units using a series of cycles. The drain module 490 can drain the output of some PEs in each cycle. The series of cycles can be configured based on a configuration descriptor that indicates the operational mode of the load module 450. In some embodiments, the drain module 490 can perform parallel reads or parallel writes. For example, the drain module 490 can write data to multiple memory banks in parallel or read data from multiple memory banks in parallel.

[0109] The drain module 490 can process the drained data and then write the data to the local storage 440. In some embodiments, the drain module 490 can compress the data, for example, based on the sparsity of the data. The drain module 490 can also generate one or more sparsity maps that indicate the sparsity of the data. The drain module 490 can include sparsity encoding logic that can convert the data from a dense format to a sparse format. For example, the drain module 490 can be implemented with one or more sparsity encoders. The sparsity encoders convert dense data to compressed data based on the sparsity of the dense data. The data received by the drain module 490 from the compute unit 460 can be at least a portion of an output tensor of a DNN layer (e.g., the output tensor 230 in Figure 2 The sparsity encoders can remove one or more zeros in the output tensor to convert the output tensor to a compressed activation tensor (“sparse activation tensor”). The sparsity encoders can also generate at least one sparsity map that indicates the sparsity of the output tensor. The sparsity map can correspond to at least a portion of the output tensor (e.g., the vector 235 in Figure 2 The sparsity map can include sparsity elements (e.g., bits), each of which corresponds to a different activation in the output tensor and indicates whether the corresponding activation is zero. The drain module 490 can have a data storage unit that stores the sparsity data. Certain aspects of the processing engine 460 are described below in connection with Figure 23 and 24

[0110] Figure 5 is a block diagram of a DNN module 500 according to various embodiments. The DNN module 500 can be an embodiment of the DNN module 401 in Figure 4 As shown in Figure 5 The DNN module 500 includes an interface module 510, a training module 520, a compression module 530, a validation module 540, a compiler 550, and a data storage device 560. In other embodiments, alternative configurations, different, or additional components can be included in the DNN module 500. Moreover, the functionality attributed to the components of the DNN module 500 can be implemented by different components or different modules or systems included in the DNN module 500.

[0111] ​The interface module 510 facilitates communication of the DNN module 500 with other modules or systems. For example, the interface module 510 establishes communication between the DNN module 500 and an external database to receive data that can be used to train a DNN or input to a DNN to perform a task. As another example, the interface module 510 communicates a configuration descriptor to the DNN accelerator 402 for configuring components of the DNN accelerator 402 for execution of a DNN. As yet another example, the interface module 510 supports the DNN module 500 to distribute a DNN to other systems, such as computing devices configured to apply the DNN to perform a task.

[0112] The training module 520 trains the DNN by using a training dataset. The training module 520 forms the training dataset. In an example where the training module 520 trains the DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe ground-truth classifications of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a portion of the training dataset can be used to initially train the DNN, while the remaining portion of the training dataset can be reserved as a validation subset used by the validation module 540 to validate performance of the trained DNN. The portion of the training dataset that does not include the adjustment subset and the validation subset can be used to train the DNN.

[0113] The training module 520 also determines hyperparameters for training the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters are different from parameters within the DNN (e.g., weights of filters). In some embodiments, the hyperparameters include variables that determine the DNN architecture, such as the number of hidden layers, etc. The hyperparameters also include variables that determine how to train the DNN, such as batch size, number of epochs, etc. The batch size defines the number of training samples to be solved before updating the parameters of the DNN. The batch size is equal to or less than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines the number of times the entire training dataset is passed back and forth through the network. The number of epochs defines the number of times a deep learning algorithm solves the entire training dataset. One epoch means that each training sample in the training dataset has a chance to update the parameters within the DNN. An epoch can include one or more batches. The number of epochs can be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0114] The training module 520 defines the architecture of the DNN, for example, based on some hyperparameters. The architecture of the DNN includes an input layer, an output layer, and multiple hidden layers. The input layer of the DNN can include a tensor (e.g., a multi-dimensional array) that specifies attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits that specifies the color of a pixel in the input image). The output layer includes labels of the objects in the input layer. The hidden layers are layers between the input layer and the output layer. The hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully connected layers, normalization layers, SoftMax or logistic layers, etc. The convolutional layers of the DNN abstract the input image into a feature map, which is represented by a tensor that specifies the height of the feature map, the width of the feature map, and the channels of the feature map (e.g., a red, green, blue image includes 3 channels). The pooling layer is used to reduce the spatial volume of the input image after convolution. It is used between two convolutional layers. The fully connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

[0115] In the process of defining the architecture of the DNN, the training module 520 also adds activation functions to the hidden layers or the output layer. The activation function of a layer transforms the weighted sum of the inputs of that layer to the output of that layer. The activation function can be, for example, a ReLU activation function, a tanh activation function, or other types of activation functions.

[0116] After the training module 520 defines the architecture of the DNN, the training module 520 inputs a training dataset to the DNN. The training dataset includes multiple training samples. An example of a training sample includes an object in an image and a ground truth label of the object. The training module 520 modifies the parameters inside the DNN (“internal parameters of the DNN”) to minimize the error between the labels of the training objects generated by the DNN and the ground truth labels of the objects. The internal parameters include the weights of the filters in the convolutional layers of the DNN. In some embodiments, the training module 520 uses a cost function to minimize the error.

[0117] The training module 520 can train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times the deep learning algorithm will solve the entire training dataset. One epoch means that every sample in the training dataset has a chance to update the internal parameters of the DNN. After the training module 520 completes the predetermined number of epochs, the training module 520 can stop updating the parameters in the DNN. The DNN with the updated parameters is referred to as a trained DNN.

[0118] The compression module 530 compresses the DNN. For example, the compression module 530 can add a pruning operation to a DNN layer to reduce computational complexity or memory usage. The pruning operation can prune a weight tensor of the DNN layer by changing one or more non-zero valued weights of the layer to zero. The modification can be made before, during, or after training. The weights can be pruned during training, during inference, or a combination of both. The compression module 530 can determine a sparsity ratio of a DNN layer. The sparsity ratio can be a ratio of a number of zero-valued weights to a total number of weights in the layer. The compression module 530 can perform the pruning operation until the sparsity ratio of the DNN layer meets a target sparsity ratio, such as 10%, 20%, 30%, 40%, 50%, and the like.

[0119] In some embodiments, the compression module 530 can select a structured sparsity pattern for a DNN layer and prune the weights of the DNN layer to reach the structured sparsity pattern. The structured sparsity pattern can be represented by a structured sparsity ratio N:M. During the pruning process, the compression module 530 can divide the kernel into weight blocks, each weight block including M consecutive weights. For each weight block, the compression module 530 can select N elements and change the values of the unselected element(s) in the weight block to zero. The compression module 530 can generate a sparsity map indicating the sparsity of the weights. In some embodiments, the compression module 530 can generate a sparsity map for each weight block. The sparsity map can include M sparsity elements corresponding to the M weights in the weight block. Each sparsity element can indicate whether the corresponding weight is zero. In some embodiments, the compression module 530 can compress the sparsity map to generate a compressed map. The compressed map has fewer elements compared to the sparsity map from which the compressed map is generated. The compression module 530 can write the sparsity map or the compressed map to the memory 410 or the local memory 440.

[0120] In some embodiments, the compression module 530 can select one or more layers in the DNN and modify each selected layer with a pruning operation. For example, the compression module 530 can select computationally complex layers, such as layers with large filters. For a pruning operation of a layer or a class of layers, the compression module 530 can determine a weight threshold that will not cause a loss of accuracy of the DNN to exceed an accuracy loss constraint. The pruning operation can modify weights with an absolute value higher than the weight threshold to zero and leave other weights unchanged. The weight pruning can reduce memory storage because zero-valued weights can not be stored. In addition, the number of operations in the layer can be reduced because calculations on zero-valued weights can be skipped without affecting the output of the layer. In some embodiments, the compression module 530 can also measure energy savings, final DNN accuracy, or layer-wise sparsity caused by the pruning operation.

[0121] After compressing the DNN, the compression module 530 can fine-tune the DNN, for example, through a retraining process. The compression module 530 can fine-tune the DNN after pruning the weights. In some embodiments, the fine-tuning process is a retraining or further training process. For example, after pruning the weights in the DNN, the compression module 530 can further train the DNN by inputting a training dataset into the DNN. The values ​​of the unpruned weights in the DNN can be modified based on the output of the DNN and the true labels of the training samples in the training dataset. In some embodiments, the values ​​of the pruned weights (i.e., zero) remain unchanged during the fine-tuning process. For example, the compression module 530 can place a mask on the pruned weight block, and this mask can prevent the values ​​in the pruned weight block from being changed during the fine-tuning process. In other embodiments, the values ​​of all weights, including the pruned weights, can be changed during fine-tuning. After one or more cycles of retraining and weight changes by the compression module 530, the compression module 530 can perform a new pruning process, for example, by selecting a weight block and pruning the selected weight block. In some embodiments, the weight pruning process can be repeated multiple times before the fine-tuning process is completed. In some embodiments, the number of rounds in the fine-tuning process may differ from the number of rounds in the training process used to determine the pre-pruned weight values. For example, the fine-tuning process may have fewer rounds than the training process. In the examples, the number of rounds in the fine-tuning process may be relatively small, such as 2, 3, 4, 5, etc.

[0122] Validation module 540 verifies the accuracy of the trained or compressed DNN. In some embodiments, validation module 540 inputs samples from a validation dataset into the trained DNN and uses the DNN's output to determine model accuracy. In some embodiments, the validation dataset may consist of some or all of the samples from the training dataset. Additionally or alternatively, the validation dataset may include additional samples in addition to those from the training set. In some embodiments, validation module 540 may determine a precision score that measures the precision, recall, or a combination of precision and recall of the DNN. Validation module 540 may use the following metrics to determine the precision score: Precision = TP / (TP+FP) and Recall = TP / (TP+FN), where precision can be how many of its predictions (TP or true positives) out of the total number of objects (TP+FP or false positives) are correctly predicted (TP), and recall can be how many of its predictions (TP+FN or false negatives) out of the total number of objects with the considered attributes are correctly predicted (TP). The F-score (F-score = 2*PR / (P+R)) unifies precision and recall into a single metric.

[0123] The verification module 540 can compare the accuracy score with a threshold score. In an example where the verification module 540 determines that the accuracy score of the DNN is less than the threshold score, the verification module 540 instructs the training module 520 to retrain the DNN. In one embodiment, the training module 520 can iteratively retrain the DNN until a stopping condition is met, such as an accuracy measurement indicating that the DNN may be accurate enough, or multiple training epochs have already occurred.

[0124] Compiler 550 compiles information from the DNN into executable instructions, which can be executed, for example, by DNN accelerator 302, to perform neural network operations within the DNN. In some embodiments, compiler 405 can generate a graph representing the DNN. This graph can include nodes and edges. Nodes can represent specific neural network operations in the DNN. Edges can connect two nodes and represent connections between two corresponding neural network operations. In an example, an edge can encode a tensor flowing from one neural network operation to another. The tensor can be the output tensor of the first neural network operation and the input tensor of the second neural network operation. Edges can encode one or more properties of the tensor, such as size, shape, storage format, etc. Compiler 550 can use this graph to generate an executable DNN. For example, the compiler can generate computer program instructions (e.g., compiler descriptors) for executing the DNN. These instructions can be stored in registers associated with components of DNN accelerator 302.

[0125] The compiler 550 can also generate configuration descriptors specifying the operating modes of components in the DNN accelerator 402, such as one or more configuration descriptors specifying whether the DPU 430 operates in dense or sparse mode. In dense mode, the DPU 430 can execute the DNN layer without any sparsity acceleration. For example, in dense mode, the DPU 430 may not skip the computation of zero-value data elements. In sparse mode, the DPU 430 can accelerate the execution of the DNN layer by skipping the computation of zero-value data elements based on sparsity. The compiler 550 can determine whether to accelerate the layer based on weight sparsity, activation sparsity, or both. The compiler 550 can select a sparsity mode for the layer from a set of sparsity modes, including, for example, a combined sparsity mode that accelerates the layer based on both weight sparsity and activation sparsity, an activation sparsity mode that accelerates the layer based on activation sparsity but not weight sparsity, a weight sparsity mode that accelerates the layer based on weight sparsity but not activation sparsity, and a dense mode that does not accelerate the layer based on sparsity. In some embodiments (e.g., embodiments where the layer is executed by multiple DPUs 430), compiler 550 can determine a sparse pattern for all DPUs 430 executing the layer. Configuration descriptors can be provided to the DPUs 430, for example, through interface module 510.

[0126] The compiler 550 can also generate configuration parameters that facilitate data reading or writing, such as configuration parameters indicating the number of data elements to be processed (e.g., the number of data elements in the chip), configuration parameters indicating the memory address where input data elements can be obtained, configuration parameters indicating the memory address where output data elements can be stored, configuration parameters indicating the memory address where another configuration parameter can be stored, and so on.

[0127] Data storage device 560 stores data received, generated, used, or otherwise associated with DNN module 500. For example, data storage device 560 stores datasets used by training module 520 and validation module 540. Data storage device 560 may also store data generated by training module 520 and validation module 540, such as hyperparameters used to train the DNN, intrinsic parameters of the trained DNN (e.g., weights), data used for sparsity acceleration (e.g., sparse bitmaps), etc. Data storage device 560 may store configuration descriptors, configuration parameters, or other data generated by compression module 530. Data storage device 560 may include one or more memories. Figure 5 In one embodiment, the data storage device 560 is a component of the DNN module 500. In other embodiments, the data storage device 560 may be external to the DNN module 500 and communicate with the DNN module 500 via a network.

[0128] Figure 6 Compressed sparsity graphs 610A-610E according to various embodiments are shown. Sparsity graphs 610A-610E are for a structured sparsity pattern of 2:4. Figure 6 In the embodiments described, sparse graphs 610A-610E are sparse bitmaps, where each bit 0 corresponds to zero weight and each bit 1 corresponds to non-zero weight. For the structured sparse pattern 2:4, there are six different combinations of zero and non-zero weights, represented by sparse graphs 610A-610E respectively. Sparse graphs 610A-610E are compressed to generate compressed graphs 620A-620E. Figure 6 As shown, each of the compressed graphs 620A-620E has three bits, while each of the sparsity graphs 610A-610E has four bits. Compression can be performed by the DNN module 401. The compressed graphs 620A-620E can later be used to accelerate computation in DNN layers, for example, in the sparse mode of the DPU performing the DNN layers. Two additional compressed graphs 620G and 620H are also generated, which can be used to facilitate other operating modes of the DPU.

[0129] Figure 7 The reduction in overhead from sparse graph compression according to various embodiments is illustrated. Figure 7Memory layout 710 and another memory layout 720 are shown. Memory layout 710 is a layout that stores eight sparse graphs of a structured sparse pattern 2:4. Memory layout 720 is a layout that stores eight compressed graphs. The compressed graphs can be generated from the sparse graphs. Figure 7 As shown, memory layout 720 requires 25% less storage space compared to memory layout 710. Therefore, compression of the sparse graph can reduce overhead for sparsity acceleration of DNN execution.

[0130] Figure 8A An example weight loading path 800 according to various embodiments is shown. The weight loading path 800 may be located between local memory 440 and computing unit 460. The weight loading path 800 may be internal to or associated with loading module 450. Loading module 450 may densify a compressed graph 810 and a sparse weight tensor 820 on the weight loading path 800. The compressed graph 810 may be generated by compressing the sparsity graph of the weight tensor. The sparse weight tensor 820 may be generated by removing zero weights from the weight tensor.

[0131] During the densification process, loading module 450 converts the compressed graph 810 into a sparse graph 830. The sparse graph 830 can be a sparse graph from which the compressed graph 810 was generated. Loading module 450 also converts the sparse weight tensor 820 into a dense weight tensor 840. The dense weight tensor 840 can be a weight tensor from which the sparse weight tensor 820 was generated. Loading module 450 can generate the dense weight tensor 840 based on the sparse graph 830. For example, loading module 450 can determine the position of zeros based on the sparse graph 830 and insert zeros into the sparse weight tensor 820 at the determined positions to generate the dense weight tensor 840. Loading module 450 can then transmit the sparse graph 830 and the dense weight tensor 840 to processing engine 470. In some embodiments, the loading module 450 may skip the densification of the sparse weight tensor 820 and transmit the sparse weight tensor 820 instead of the dense weight tensor 840 to the processing engine 470.

[0132] Figure 8BAn example activation loading path 805 according to various embodiments is shown. Activation loading path 805 may be located between local memory 440 and computing unit 460. Activation loading path 805 may be internal to or associated with loading module 450. Loading module 450 may use a sparsity graph 815 to densify a sparse activation tensor 825 on activation loading path 805. Sparse activation tensor 825 may be generated by removing zero activations from the activation tensor. Sparsity graph 815 may be generated, for example, by a sparsity encoder in exhaust module 490 during the compression of the activation tensor. During densification, loading module 450 may determine the positions of zeros based on sparsity graph 815 and insert zeros into sparse activation tensor 825 at the determined positions to generate dense activation tensor 845. Loading module 450 may transmit sparse graph 815 and dense activation tensor 845 to processing engine 470.

[0133] Figure 9 Example sparse unit 900 according to various embodiments is shown. Sparse unit 900 can be used in a processing engine (e.g., Figure 4 The sparse unit 900 includes 16 MAC units 910 (referred to individually as "MAC unit 910") arranged in four rows and four columns, 16 weight register files 920 (referred to individually as "weight register file 920"), 16 weight sparsity register files 925 (referred to individually as "weight sparsity register file 925"), 16 activation register files 930 (referred to individually as "activation register file 930"), 16 activation sparsity register files 935 (referred to individually as "activation sparsity register file 935"), four row buffers 940 (referred to individually as "row buffer 940"), a transpose module 950, and a control unit 960. In other embodiments, the sparse unit 900 may include fewer, more, or different components. For example, the sparse unit may include different numbers of MAC units 910, weight register file 920, weight sparsity register file 925, activation register file 930, activation sparsity register file 935, row buffer 940 or sparsity module 965.

[0134] MAC unit 910 is configured to perform MAC operations. Each MAC unit 910 may include one or more multipliers and one or more adders. A multiplier may multiply an activation by a weight to compute a product each time. In some embodiments (e.g., embodiments where MAC unit 910 includes multiple multipliers), the multipliers may operate simultaneously in one cycle to process multiple activation-weight pairs and compute multiple products. Adders may accumulate the products computed by the multipliers. In an example, MAC unit 910 may include a tree of multipliers and adders. The adder tree may include multiple levels. A first level may receive the outputs of the multipliers. The number of adders in the first level may be half the number of multipliers, and each adder may accumulate the outputs of two multipliers. A second level may receive the outputs of the adders in the first level. The number of adders in the second level may be half the number of adders in the first level, and each adder in the second level may accumulate the outputs of two adders in the first level. The adder tree may include one or more other levels. The final level may include a single adder that sums the outputs of the adders in the penultimate level to compute a partial sum for MAC unit 910.

[0135] The weight register file 920 stores the weights to be processed in the MAC operation performed by the MAC unit 910. Figure 9 In some embodiments, four weight register files 920 are grouped into a storage set that stores data to be used by a column of MAC units 910. There are four storage sets corresponding to the four columns of MAC units 910. In some embodiments, the weight register files 920 may correspond to MAC units 910 and store data to be processed by the MAC units. In some embodiments, all 16 weight register files 920 constitute a weight storage unit. A weight sparsity register file 925 stores a weight sparsity graph that can be used to accelerate computation in the MAC units 910. In some embodiments, four weight sparsity register files 925 are grouped into a sparse storage set that stores a sparsity graph used to accelerate computation in a column of MAC units 910. There are four sparse storage sets corresponding to the four columns of MAC units 910. In some embodiments, the weight sparsity register files 925 may correspond to MAC units 910 and store a weight sparsity graph used to accelerate computation in the MAC units 910. In some embodiments, all 16 weight sparsity register files 925 may constitute a sparse storage unit.

[0136] The activation register file 930 stores the activation that will be processed during MAC operations. Figure 9In some embodiments, four active register files 930 are grouped into a storage set that stores data to be used by a row of MAC units 910. There are four storage sets corresponding to the four rows of MAC units 910. In some embodiments, the active register files 930 may correspond to MAC units 910 and store data to be processed by the MAC units. In some embodiments, all 16 active register files 930 constitute an active storage unit. An active sparsity register file 935 stores an active sparsity graph that can be used to accelerate computation in the MAC units 910. In some embodiments, the four active sparsity register files 935 may be grouped into a sparse storage set that stores a sparsity graph used to accelerate computation in a column of MAC units 910. There are four sparse storage sets corresponding to the four columns of MAC units 910. In some embodiments, the active sparsity register files 935 may correspond to MAC units 910 and store a weighted sparsity graph used to accelerate computation in the MAC units 910. In some embodiments, all 16 active sparsity register files 935 may constitute a sparse storage unit.

[0137] Row buffers 940 store the output of MAC units 910. Each row buffer 940 can discharge the output of a single row of MAC units 910. Data stored in the row buffers 940, such as output operands, can be further passed to transpose module 950. Transpose module 950 can operate in either active sparsity mode or weighted sparsity mode. In some embodiments, transpose module 950 can transpose output operands in one of the two sparsity modes and keep output operands unchanged in the other sparsity mode.

[0138] Control unit 960 controls the operation of MAC unit 910. In some embodiments, control unit 960 can control sparsity-based acceleration in sparse unit 900. Control unit 960 can control the transfer of activations and weights to MAC unit 910 based on the operating mode of sparse unit 900 (e.g., sparse mode or dense mode) or the sparsity mode of neural network layers. Figure 9In one embodiment, the control unit 960 includes 16 sparsity modules 965 (referred to individually as "sparse modules 965"), each controlling sparsity-based acceleration in one of the 16 MAC units 910. The sparsity modules 965 can control the transfer of activations and weights from corresponding weight register files 920 and corresponding activation register files 930 to the corresponding MAC units 910 based on one or more sparsity graphs. For example, the sparsity module 965 can select a subset of weights stored in the weight register file 920 and a subset of activations stored in the activation register file 930 based on a combined sparsity graph. The selected weights and activations form non-zero activation-weight pairs. The sparsity module 965 can then transfer the selected weights and activations to the MAC units 910 for performing MAC operations. Other weights stored in the weight register file 920 and other activations stored in the activation register file 930 are skipped from computation. Figure 9 In one embodiment, each sparsity module 965 can control the operation of the corresponding MAC unit 910, such as sparsity acceleration in the MAC unit 910.

[0139] In some embodiments, the sparsity module 965 may include one or more LUTs and an address generator. The LUT can associate a sparse graph with a read index. The sparsity module 965 can input a sparse graph into the LUT, and the LUT outputs a corresponding read index. The LUT can be configured for a specific structured sparse pattern, and the sparse graphs in the LUT can all have a structured sparse pattern. In some embodiments, the sparsity module 965 can use the LUT for one or more other structured sparse patterns. For example, the sparsity module 965 can convert a sparse graph with other structured sparse patterns into a sparse graph in the LUT. In some embodiments, the sparsity module 965 may include multiple LUTs configured for different structured sparse patterns. The address generator can generate a memory address from the read index output by the LUT. The memory address is an address in the activation register file 930 or the weight register file 920. The sparsity module can read an activation or weight from the activation register file 930 or the weight register file 920 based on the memory address and pass the activation or weight to the MAC unit 910.

[0140] In some embodiments, the sparsity module 965 may further include gating logic for gating the transmission of activations or weights to the MAC unit 910. In an example, the structured sparsity pattern(s) supported by the LUT(s) may be the sparsity pattern(s) of the weights(s). The sparsity module 965 may use the gating logic to utilize the unstructured sparsity of activations. The sparsity module 965 may read the activation sparsity graph from the activation sparsity register file 935. For each sparsity element indicating a zero activation, the sparsity module 965 may prevent the activation or corresponding weight from being transmitted to the MAC unit 910. In some embodiments, the sparsity module 965 may read the activation or weight but not transmit it to the MAC unit 910. In other embodiments, the sparsity module 965 may not even read the activation or weight. The multiplication of activations and weights is then skipped. The sparsity module 965 may send zeros to the adder tree in the MAC unit 910 as the product of activations and weights.

[0141] In some embodiments, the sparsity module 965 can support various data precisions. For example, the sparsity module 965 can support integer precision and floating-point precision. For higher data precision, the sparsity module 965 can divide data elements into multiple fragments and distribute these fragments to different data paths. Some aspects of the control unit 960 or the sparsity module 965 are described below with reference to Figures 11-21.

[0142] Figure 10 An example sparse cell array 1000 according to various embodiments is shown. The sparse cell array 1000 can be... Figure 4 An example of the processing engine 470. In Figure 10 In this embodiment, the sparse cell array 1000 includes sparse cells 1010 (referred to individually as "sparse cells 1010") arranged in four columns and four rows, an activation memory 1020, and a weight memory 1030. In other embodiments, the sparse cell array 1000 may include fewer, more, or different components. For example, the sparse cell array 1000 may include different numbers of columns, rows, or sparse cells 1010.

[0143] Each sparse unit 1010 can perform sparsity-accelerated MAC operations. Figure 9The sparse unit 1000 is 900. An activation memory 1020 stores activations, such as activations in the input tensor of a deep learning operation. Activations can be loaded from the activation memory 1020 into the sparse unit 1010. A weight memory 1030 stores weights. Weights can be loaded from the weight memory 1030 into the sparse unit 1010. The activation memory 1020 or the weight memory 1030 can be a buffer. In other embodiments, instead of the activation memory 1020 and the weight memory 1030, the sparse unit array 1000 may include a dense data memory and a sparse data memory. The dense data memory may store dense tensors, such as dense tensors generated by the loading module 360. The sparse data memory may store sparse tensors.

[0144] Figure 11A and 11B A LUT 1100 in a sparsity module according to various embodiments is shown. The LUT 1100 is referred to solely as "LUT 1100". Figure 11A In some embodiments, LUT 1100 is configured for different structured sparsity patterns, such as different structured sparsity patterns for weights. For example, one LUT 1100 is configured for structured sparsity pattern 2:4, while another LUT 1100 is configured for structured sparsity pattern 1:4. Additionally or alternatively, the sparsity module may include various numbers of LUTs configured for various structured sparsity patterns. The sparsity module may be... Figure 9 An example of the sparsity module 965 is provided. In some embodiments, the sparsity module can control the operation of at least one MAC unit.

[0145] A LUT 1100 configured for a specific structured sparse pattern can map the sparse graph of the structured sparse pattern to read indices. Taking a LUT 1100 configured for structured sparse pattern 2:4 as an example, the LUT 1100 has a sparse graph portion 1110 and a read index portion 1120. The sparse graph portion 1110 has a total of eight sparse graphs, with the top six targeting structured sparse pattern 2:4. Each of these six sparse graphs corresponds to a set of two read indices in the read index portion 1120. These two read indices can correspond to memory addresses storing two non-zero elements in the register file associated with the MAC unit. Figure 11A As shown, sparse graph 1101 is input to LUT 1100. Sparse graph 1101 can be a weight sparse graph of a weight block. Sparse graph 1101 indicates that the weight block has a total of four weights, with the first and third weights being non-zero weights, while the second and fourth weights are zero weights. LUT 1100 outputs a read index 1102, which is mapped to sparse graph 1101 in LUT 1100.

[0146] In addition to the six sparse graphs for the structured sparse pattern 2:4, the LUT 1100 includes a sparse graph with four 0-bit bits and a sparse graph with four 1-bit bits. These two sparse graphs do not have the structured sparse pattern 2:4. They can be used to control other operating modes of the MAC unit. For example, the sparse graph with four 0-bit bits can be used to skip computations in the MAC unit. The LUT 1100 may not output a read index, and the sparse module may not pass activations or weights to the MAC unit. The sparse graph with four 1-bit bits can be used to enable the MAC unit to operate in dense mode. Figure 11B As shown, for dense mode, the sparsity module can input sparse graph 1103 into LUT 1100. Sparse graph 1103 has four 1 bits and no 0 bits. The LUT 1100 output includes read index 1104 with four read indices, which can correspond to the memory addresses of four data elements, allowing the sparsity module to pass all four data elements to the MAC unit, although these four data elements may include one or more zeros.

[0147] Even when LUT 1100 is configured for a specific structured sparsity pattern, it can also be used for one or more other structured sparsity patterns. In the example, LUT 1100 configured for structured sparsity pattern 1:4 can be used for structured sparsity patterns 2:4 or 3:4. In another example, LUT 1100 configured for structured sparsity pattern 2:4 can be used for structured sparsity pattern 4:8. The following is in conjunction with... Figure 18 and 19 This provides certain aspects of LUTs that support various structured sparsity patterns.

[0148] Figure 12 The generation of memory addresses from a sparse graph using LUT 1210 is illustrated according to various embodiments. The memory addresses can be generated by a sparsity module including LUT 1210. The LUT can be an example of LUT 1100 configured for structured sparse pattern 2:4 as shown in Figure 11. For illustrative and simplification purposes, Figure 12Two data paths, 1220 and 1230, are shown. Data paths 1220 and 1230 can be used to transfer data from memory addresses to the MAC unit. Four sparse graphs 1201A-1201D are input to LUT 1210 via data path 1220. LUT 1210 outputs four read index groups 1202A-1202D, each corresponding to one of sparse graphs 1201A-1201D and including two read indices. Four sparse graphs 1201E-1201H are input to LUT 1210 via data path 1230. LUT 1210 outputs four read index groups 1202E-1202H, each corresponding to one of sparse graphs 1201E-1201H and including two read indices.

[0149] In some embodiments, each read index is one bit, and each read index group has two bits. The sparsity module can generate a memory address from the read index group by converting a 2-bit read index into a 5-bit address by appending fixed bits based on the read index. The 5-bit address consists of three fixed bits and two read index bits. The 5-bit address can be a memory address storing two data elements. For example, for each read index group, the sparsity module can append the same bits to each read index in that read index group. Figure 12 As shown, the sparsity module can append 000 to each read index in read index group 1202A, 001 to each read index in read index group 1202B, 010 to each read index in read index group 1202C, 011 to each read index in read index group 1202D, 100 to each read index in read index group 1202E, 101 to each read index in read index group 1202F, 110 to each read index in read index group 1202G, and 111 to each read index in read index group 1202H. The sparsity module can read data elements from a data storage unit (e.g., a register file) associated with the MAC unit based on the generated memory address. Data elements can be sent to the MAC unit via data path 1220 or data path 1230.

[0150] Figure 13A and 13B Read ports 1310 and 1320 according to various embodiments are shown. Figure 13A The read port 1310 is shown, which can be used in many currently available DNN accelerators. Figure 13ARead port 1320 is shown, which can be used in the DNN accelerator of this disclosure. Read ports 1310 and 1320 can be read ports of a data storage unit (e.g., a register file) associated with a MAC unit and can facilitate data reading from the data storage unit to the MAC unit. Read ports 1310 and 1320 can each include a multiplexer and can each be coupled to a data path.

[0151] like Figure 13A As shown, read port 1310 is located in data storage unit 1315 comprising 16 memory entries. One memory entry can be one byte. The 16 memory entries can be four data blocks. Each memory entry is represented by a box. These four data blocks are represented by different patterns. Read port 1310 can access all 16 memory entries at a time. In some embodiments, read port 1310 can be coupled to multiple data paths, and 16 memory entries can be accessed for each data path.

[0152] exist Figure 13B In this context, read port 1320 is located in data storage unit 1325, which includes 16 memory entries. One memory entry can be one byte. The 16 memory entries can be four data blocks. Each memory entry is represented by a frame. These four data blocks are represented by different patterns. Unlike read port 1310, Figure 13B The read port 1320 can access up to four memory entries at a time. These four memory entries can form a single data block stored in the data storage unit. Each memory entry is represented by a box. Figure 13B Four data blocks represented by different modes are shown. Figure 13B An example is shown where read port 1320 can access three memory entries because access to the first memory entry in the data block can be avoided, since the first memory entry is zero and does not need to be passed to the MAC unit.

[0153] Compared to read port 1310, read port 1320 is optimized for structured sparsity. Dedicated data storage cells can be created where the read port has access to 4 memory entries at a granularity (or M in the structured sparsity ratio), instead of being able to access all memory entries stored in the data storage cell. This limits the data storage cell read multiplexer, resulting in significant benefits in terms of area and power.

[0154] Figure 14A and 14B The diagram illustrates the loading of activation and weights into MAC unit 1410 according to various embodiments. MAC unit 1410 may be... Figure 9 Example of MAC unit 910 in the example. Figure 14A The diagram shows that the sparsity module associated with MAC unit 1410 will activate the MAC unit 1410. Figure 14B The diagram illustrates the loading of weights into the MAC unit 1410 using a sparsity module. The sparsity module can be... Figure 9 Example of sparse module 965 in the example.

[0155] exist Figure 14A In this configuration, MAC unit 1410 is associated with data path 1420 and another data path 1430. Data path 1420 receives four sparse graphs 1401A-1401D. Data path 1420 also receives four sparse graphs 1401E-1401H. Sparse graphs 1401A-1401H can be weighted sparse graphs. For illustrative purposes, sparse graphs 1401A-1401H have a structured sparse pattern of 2:4. The sparsity module is based on sparse graphs 1401A-1401H, for example, by using the combination described above. Figure 12 The described memory address generation method accesses the active register file 1440 and reads memory entries 1402A-1402H and 1403A-1403H from the active register file 1440. The active register file 1440 can be... Figure 9 An example of activation register file 930 is shown. Each sparsity graph can correspond to two memory entries. For example, sparsity graph 1401A is used to locate memory entries 1402A and 1403A. Each memory entry can be a single activation. These 16 activations are transferred to MAC unit 1410 via a set of multiplexers. By using sparsity graphs 1401A-1401H, the sparsity module can identify activations corresponding to non-zero weights. The sparsity module can skip reading activations corresponding to zero weights, even though these activations could be stored in activation register file 1440, because the computation of these activations can be skipped without affecting the output of the MAC operation, since the product of the activation and zero weight will be zero.

[0156] exist Figure 14B In this process, non-zero weights are stored in weight register file 1450. Weight register file 1450 can be... Figure 9 Example of weight register file 920 in [the context of the file]. Figure 14B In this embodiment, the weight register file 1450 stores sparse weight tensors, meaning that all weights stored in the weight register file 1450 are non-zero, and zero weights are not stored. Figure 14BAs shown, the weight register file 1450 can be a simple latch or register array with full MAC length read and write operations. All weights stored in the weight register file 1450 at once can be read and provided to the MAC unit 1410 at once. In other embodiments, the weight register file 1450 can store dense weight tensors, meaning that zero weights are also stored in the weight register file 1450. It can be used... Figure 14A The loading method shown reads weights from the weight register stack 1450.

[0157] MAC unit 1410 includes 16 multipliers 1413 (referred to individually as "multipliers 1413") and an adder tree 1415. Each multiplier 1413 receives activation and non-zero weights one at a time and multiplies the activation and non-zero weights. The products calculated by multipliers 1413 are accumulated by adder tree 1415, which outputs the internal partial sum of MAC unit 1410. The internal partial sum is stored in output register file 1460. Output register file 1460 may receive the internal partial sum of another MAC unit, which may be accumulated with the internal partial sum of MAC unit 1410 of adder tree 1415 to calculate a partial sum. In some embodiments, MAC unit 1410, activation register file 1440, weight register file 1450, and output register file 1460 may constitute a processing element. Although in Figure 14A and 14B In this embodiment, MAC unit 1410 has 16 multipliers, but in other embodiments, MAC unit may include a different number of multipliers. Furthermore, MAC unit may be associated with a different number of data paths.

[0158] Figure 15 Example control and data flows for structured sparsity acceleration are illustrated according to various embodiments. This flow can be implemented in a DPU (e.g., a DPU). Figure 4 In the DPU 430, the stream begins with eight compressed graphs 1501 (referred to individually as "compressed graph 1501"). Each compressed graph 1501 is converted into a sparse graph 1502, resulting in eight sparse graphs 1502. The decompression of the compressed graphs 1501 can be performed by a loading module in the DPU (e.g., ...). Figure 4 The loading module 450 in the middle is completed.

[0159] Sparse graph 1502 is input to LUT 1510. LUT 1510 outputs a 2-bit read index 1503 for each sparse graph 1502. Eight read indices 1503 are output from LUT 1510. The read indices 1503 are provided to address generator 1520, which generates memory addresses from the read indices 1503. LUT 1510 and address generator 1520 can be located in the control unit of the DPU, for example in... Figure 9 In the control unit 960, activation and weight are read from the data storage unit 1530 based on the address generated by the address generator 1520. The data storage unit 1530 may include an activation register file and a weight register file.

[0160] The activation and weight are passed to eight multipliers 1540, each receiving one activation and one corresponding weight. The product calculated by the multipliers 1540 is accumulated by the adder tree 1550, and the sum is stored in the data storage unit 1560. The data storage unit 1560 may be an output register file. The data storage unit 1530, multipliers 1540, adder tree 1550, and data storage unit 1560 may reside in a MAC unit. The sum may be passed to another MAC unit for further computation. Additionally or alternatively, the data storage unit 1560 may receive a sum from another MAC unit. The two sums may be accumulated by the adder tree 1550 to compute a new sum.

[0161] Figure 16 Another example of control and data flow for accelerating structured sparsity, according to various embodiments, is shown. Figure 16 The flow shown is for a structured sparsity pattern of 1:4. This flow can be in a DPU, for example, in... Figure 4 In the DPU430, the stream begins with four compressed graphs 1601 (referred to individually as "compressed graph 1601"). Each compressed graph 1601 is transformed into a sparse graph 1602, resulting in four sparse graphs 1602 with a structured sparse pattern of 1:4. The decompression of the compressed graphs 1601 can be performed by a loading module in the DPU (e.g., ...). Figure 4 The loading module 450 in the middle is completed.

[0162] Sparse graph 1602 is input to LUT 1610. LUT 1610 outputs a 1-bit read index 1603 for each sparse graph 1602. Four read indices 1603 are output from LUT 1610. The read indices 1603 are provided to address generator 1620, which generates memory addresses from the read indices 1603. LUT 1610 and address generator 1620 can be located in the control unit of the DPU, for example in... Figure 9In the control unit 960, activation and weight are read from the data storage unit 1630 based on the address generated by the address generator 1620. The data storage unit 1630 may include an activation register file and a weight register file.

[0163] The activation and weight are passed to four multipliers 1640, each receiving an activation and a corresponding weight. The product calculated by multipliers 1640 is accumulated by adder tree 1650, and the sum is stored in data storage unit 1660. Data storage unit 1660 may be an output register file. Data storage unit 1630, multipliers 1640, adder tree 1650, and data storage unit 1660 may reside in a MAC unit. The sum may be passed to another MAC unit for further computation. Additionally or alternatively, data storage unit 1660 may receive a sum from another MAC unit. The two sums may be accumulated by adder tree 1650 to compute a new sum.

[0164] Figure 17 Data paths 1710 and 1720 in a sparse cell operating in floating-point mode according to various embodiments are shown. Examples of sparse cells may include... Figure 9 The sparse unit 900 is described above. In some embodiments, the data paths 1710 and 1720 of the sparse unit or other components (e.g., control units) are configured for integer precision, such as INT8. The sparse unit can also operate in floating-point mode when the data (e.g., activations or weights) has floating-point precision, supporting structured sparsity acceleration. Figure 17 An example with FP16 precision is shown, so the data element is stored as two bytes: a high-order byte and a low-order byte. The data element can be split into two-byte operands. Data path 1710 can be used for the high-order byte, while data path 1720 can be used for the low-order byte. An example of data path 1710 could be data path 1420 in Figure 14, and an example of data path 1720 could be data path 1430. For example... Figure 14A As shown, data elements can be read and processed within the sparse cell array. In some embodiments, the splitting of data elements can be performed by the loading module 450. One or more additional multiplexers can be added at the interface of the loading module 450 to facilitate the splitting. Because this is done at the loading module 450, rather than as a repetitive function within the sparse cell array, overhead can be kept to a minimum.

[0165] Figure 18 An example LUT 1800 supporting multiple structured sparsity patterns according to various embodiments is shown. For simplicity and illustrative purposes, the LUT 1800 is configured for structured sparsity pattern 2:4. Figure 18An example of a LUT used for a structured sparse pattern 4:8 is shown. LUT 1800 receives a sparse graph 1810 and another sparse graph 1820, both of which have a structured sparse pattern 4:8. Because 2:4 is a subset of 4:8, LUT 1800 can work with a structured sparse pattern 4:8. Sparse graph 1810 can be split into two 2:4 sparse graphs, 1813 and 1815. Sparse graph 1820 can be split into two 2:4 sparse graphs, 1823 and 1825. The four 2:4 sparse graphs are input to LUT 1800, whose output corresponds to four read index groups for eight non-zero data elements. Each of the eight non-zero data elements can be provided to multiplier 1830 via multiplexer 1840. A multiplexer design for passing data elements in a structured sparse pattern 2:4 can also work with a structured sparse pattern 4:8.

[0166] In some embodiments, the LUT 1800 can support structured sparse patterns with different sizes M, where M is the size of the data blocks in the structured sparse pattern. One structured sparse pattern can be a subset of another structured sparse pattern. The LUT 1800 can be configured to accommodate all sizes M for a minimum M.

[0167] Figure 19 Another example LUT 1900 supporting multiple structured sparsity patterns according to various embodiments is shown. For simplicity and illustration purposes, LUT 1900 is configured for structured sparsity pattern 1:4. Figure 19An example of using a LUT for a structured sparse pattern 2:4 is shown. LUT 1900 receives four sparse graphs 1910A-1910D with a structured sparse pattern 2:4. Sparse graph 1910A is transformed into two sparse graphs 1913A and 1915A. Similarly, 1910B is transformed into sparse graphs 1913B and 1915B, 1910C is transformed into sparse graphs 1913C and 1915C, and 1910D is transformed into sparse graphs 1913D and 1915D. Sparse graphs 1913A-1913D and 1915A-1915D both have a 1:4 structured sparse pattern and are therefore compatible with LUT 1900. Sparse graphs 1913A-1913D and 1915A-1915D are input to LUT 1900 in two cycles. Sparse graph 1913A-1913D is input to LUT 1900 in cycle 0, while sparse graph 1915A-1915D is input to LUT 1900 in cycle 1. In each cycle, LUT 1900 outputs four read indices, which are used to pass four activations to four multipliers 1930 via four multiplexers 1940, respectively. The computation takes two cycles to complete. LUT 1900 can also be used for structured sparse pattern 3:4, and the computation will take three cycles to complete because each sparse graph with structured sparse pattern 3:4 can be transformed into three sparse graphs with structured sparse pattern 1:4.

[0168] Figure 20 Hybrid sparsity patterns of sparse units according to various embodiments are illustrated. For illustrative and simplification purposes, Figure 20 Not all components of a sparse unit are shown. Examples of sparse units could be... Figure 9 The sparse unit 900 in the model. In the hybrid sparsity mode, the execution of DNN layers can be accelerated based on the structured sparsity of weights and the unstructured sparsity of activations.

[0169] exist Figure 20 In embodiments, the control unit (e.g., Figure 9 The control unit 960 in the unit uses a weighted sparsity graph 2001 and an activation sparsity graph 2002 to accelerate computation in the sparse unit. The weighted sparsity graph 2001 comprises four sparse graphs with a structured sparsity pattern of 2:4, which can be used by employing a LUT (e.g., Figure 15The memory address is generated using LUT 1510 in the data storage unit 2010. The memory address can be used to read activations or weights from the data storage unit 2010. The data storage unit 2010 can be a register file. As a supplement to the weight sparsity graph 2001, the activation sparsity graph 2002 is used to gating data reads from the data storage unit 2010. The activation sparsity graph 2002 indicates the unstructured sparsity of activations. For each zero bit in the activation sparsity graph 2002, the control unit can generate a signal that can skip the read and multiplication of the corresponding activation or weight. This signal can also drive zeros into the adder tree 2020 as the product of activations and weights.

[0170] Ungated activations and weights read using the weighted sparsity graph 2001 are loaded into multiplier 2030 (referred to as "multiplier 2030") via read port 2040 (referred to as "read port 2040" alone). Figure 20 A product graph 2003 is shown, indicating the sparsity of the data input into the adder tree. Product graph 2003 has eight elements corresponding to eight non-zero weights as indicated by weight sparsity graph 2001. The third, fifth, and eighth elements in product graph 2003 are zero because they correspond to zero activations, as indicated by activation sparsity graph 2002.

[0171] Figure 21 Example control and data flows in sparse units according to various embodiments are shown. For illustrative and simplification purposes, Figure 20 Not all components of a sparse unit are shown. Examples of sparse units could be... Figure 9 The sparse unit 900. In Figure 20 In this process, eight sparse graphs 2101 (referred to individually as "sparse graph 2101") are input into LUT 2110. The sparse graphs 2101 can be decompressed from the compressed graph ( Figure 20 (Not shown in the diagram) is used to generate the LUT 2110 output read index 2102 (solely referred to as "read index 2102"). Read index 2102 has two bits. In some embodiments, each bit may be referred to as a read index, and read index 2102 may also be referred to as a read index group. Read index 2102 is input to address generator 2120 for generating memory addresses. LUT 2110 and address generator 2120 may be in the control unit of the sparse cell. The control unit may also include gating logic ( Figure 21 (Not shown in the image).

[0172] and Figure 15 and 16 The flows shown are different. Figure 21 The stream then goes to activate the sparse storage device 2130. Activating the sparse storage device 2130 can be... Figure 9 An example of an active sparse register file 935 is shown. Active sparse storage device 2130 can store one or more active sparse graphs, such as active sparse graph 2002. Gating logic can use the active sparse graph to gating one or more read ports of data storage unit 2140 and prevent read address generator 2120 from having already determined one or more activations or weights for the memory address. For example, the gating logic can gating a read port corresponding to zero activation as indicated by the active sparse graph. Other activations or weights can be read from data storage unit 2140 using the memory address determined by address generator 2120.

[0173] These activations and weights are provided to multiplier 2150 (referred to separately as "multiplier 2150"). Due to the gating mechanism, one or more multipliers 2150 may not receive any activations or weights. A multiplexer 2160 (referred to separately as "multiplier 2160") is used. Each multiplexer 2160 is coupled to multiplier 2150 and selects either the output or zero of multiplier 2150. Multiplexer 2160 can select zero when the gating mechanism is triggered by activation sparsity. The value selected by multiplexer 2160 is provided to adder tree 2170 for accumulation. The sum calculated by adder tree 2170 is stored in data storage unit 2180. Data storage unit 2180 may be an output register file. Activation sparsity storage device 2130, data storage unit 2140, multiplier 2150, multiplexer 2160, adder tree 2170, and data storage unit 2180 may be located in a MAC unit. The sums can be transferred to another MAC unit for further computation. Alternatively, data storage unit 2180 can receive the sums from another MAC unit. The two sums can be accumulated by adder tree 2170 to compute a new sum.

[0174] Figure 22 This is a block diagram of a discharge module 2200 according to various embodiments. The discharge module 2200 extracts output elements calculated by sparse units (e.g., sparse unit 900, sparse unit 1010, etc.) and writes the output elements to a memory (e.g., local memory 440). The discharge module 2200 may be... Figure 4 Example of module 490 in the middle. Figure 22As shown, the discharge module 2200 includes a recirculation buffer 2220 (referred to solely as "recirculation buffer 2220"), a discharge grading buffer 2230, a global discharge module 2240, a discharge library 2250 (referred to solely as "discharge library 2250"), a sparse encoder 2260 (referred to solely as "sparse encoder 2260"), a write module 2270, and a write buffer 2280. In other embodiments, alternative configurations, different, or additional components may be included in the discharge module 2200. Furthermore, the functionality of the components belonging to the discharge module 2200 may be implemented by different components, modules, or systems included in the discharge module 2200.

[0175] The circular buffer 2220 can temporarily store data calculated by the computing unit 460. In some embodiments, each circular buffer 2220 corresponds to a PE column and stores the output of that PE column. The output data stored in the circular buffer 2220 can be further transmitted and written to the discharge hierarchical buffer 2230.

[0176] The global eject module 2240 can select output elements stored in the eject hierarchy buffer 2230. In some embodiments, the global eject module 2240 selects output elements in a predetermined manner (e.g., a 1×1×Output Channel (OC) manner). In some embodiments, the global eject module 2240 selects one of a predetermined number of entries from the eject hierarchy buffer 2230. This predetermined number may be the number of MAC cells in a column of sparse cells. In other embodiments, the global eject module 2240 can select a predetermined amount of data, such as 16 bytes, 32 bytes, etc. After selecting an entry, the global eject module 2240 can select one or more eject libraries 2250 and multicast the selected entry to the selected eject library(s) 2250(s). In the example, there may be 16 eject libraries 2250 arranged in four groups. Each group may include 4 eject libraries 2250. The global eject module 2240 may assign a right rotation value specific to each eject library 2250 to align and cascade consecutive OCs in a single eject library 2250. The global eject module 2240 can further write the correct set of bytes from the selected line of the eject hierarchy buffer 2230 into the eject library 2250.

[0177] In some embodiments, a single discharge cassette 2250 can store an output vector comprising output elements having the same (x, y) coordinates but different OCs. The output elements of the output vector can be arranged sequentially according to their OCs. For example, the OC coordinate of the first output element in the output vector can be 0, the OC coordinate of the second output element can be 1, the OC coordinate of the third output element can be 2, and so on. Different discharge cassettes 2250 can store different output vectors. Output elements in different discharge cassettes 2250 can have different x-coordinates or different y-coordinates.

[0178] The sparse encoder 2260 transforms dense data into compressed data based on the sparsity in the dense data. In some embodiments, the sparse encoder 2260 may receive output elements of the layer (e.g., from the global exhaust module 2240) for example. Figure 2 The output tensor 230 is used. Output elements can be arranged in an output vector. The sparse encoder 2260 can generate compressed versions of one or more output vectors. In some embodiments, the sparse encoder 2260 can compress the output vector based on a threshold. The sparse encoder 2260 can compare the absolute value of each output element with a threshold. The sparse encoder 2260 can remove any output element whose absolute value is not greater than the threshold from the output vector to generate a compressed output vector. The threshold can be zero. The removed output elements may not be stored in the write buffer 2280 or local memory to save bandwidth and memory usage.

[0179] In some embodiments, the sparse encoder 2260 may also generate one or more sparsity maps of the output vector. The sparsity map may include sparsity elements, each corresponding to a different output element in the output vector and indicating whether the corresponding output element has been removed. In some embodiments, the sparsity map may be a sparse bitmap, and the sparsity elements in the sparse bitmap may be bits. Bit 0 may indicate that the corresponding output element has been removed and is not in the compressed output vector, while bit 1 may indicate that the corresponding output element has not been removed and is in the compressed output vector.

[0180] In some embodiments, the sparse encoder 2260 can encode sparsity at the context level. The context can be a portion of the output tensor generated by an array of sparse units. In an example, the context can be an output vector comprising output elements having the same (X, Y) coordinates but different Z coordinates. This context can be processed in the next DNN layer (e.g., by one or more MAC units). For a given context, the sparse encoder 2260 can read multiple lines from the data volume before emitting N bytes (where N is an integer, such as 16, 32, 64, etc.) of a single line, depending on the sparsity level. Since elements in the context stream may pass through multiple rounds, the sparse encoder 2260 can store data indicating the state of the context (“context state”) in a buffer and retrieve the context state from the buffer later. The context state can include a compressed context, a sparsity graph of the context, output elements in the compressed context, and a count of lines.

[0181] The write module 2270 determines the memory address of the output element and writes the output element to memory based on the memory address. An example of memory could be... Figure 4 The local memory 440 is used. In some embodiments (e.g., embodiments where the sparse encoder 2260 compresses the output tensor), the write module 2270 may write the compressed output vector generated by the sparse encoder 2260 to the memory. The write module 2270 may skip output elements removed by the sparse encoder 2260 during compression. The write module 2270 may also write the sparsity graph generated by the sparse encoder 2260 to this memory or a separate memory. In some embodiments, the write module 2270 may determine the memory address of the sparsity graph associated with the output tensor and write the sparsity graph to the memory based on the memory address.

[0182] To write an output vector or sparse graph to memory, write module 2270 can generate a write request that includes one or more memory addresses of the output vector or sparse graph, and transmit the write request to memory. Upon receiving the write request, memory can process the write request and store the output vector or sparse graph in one or more data write operations. While memory is processing a write request or one or more previous write requests, write buffer 2280 can store the output vector, sparse graph, or write request.

[0183] Figure 23 An example sparse encoder 2300 according to various embodiments is shown. The sparse encoder 2300 can be... Figure 22 An example of a sparse encoder 2260. (e.g.) Figure 23As shown, the sparse encoder 2300 includes a comparator 2310 (referred to separately as "comparator 2310") and a compression packer 2320. For illustrative purposes, Figure 23 Sixteen comparators 2310 are shown. In other embodiments, the sparse encoder 2300 may include a different number of comparators 2310.

[0184] exist Figure 23 In the embodiments, the sparse encoder 2300 receives an activation tensor 2301, the activation tensor 2301 including... Figure 23 The 16 activations are represented as D0-D15. The activation tensor can be an output operand computed in a sparse unit (e.g., sparse unit 900). Each comparator 2310 receives different activations in the activation tensor 2301 and compresses the activation with a predetermined value. In some embodiments, the predetermined value can be zero. In other embodiments, the predetermined value can be non-zero. Comparator 2310 can change the value of an activation with a value lower than the predetermined value to zero and output zero. For activations with a value not lower than the predetermined value, comparator 2310 can output the activation as is. The data elements output from comparator 2310 are C0-C15. Each comparator 2310 can also output a bit indicating whether the data element output from the comparator is zero. In the example, the bit is zero when the output is zero; the bit is 1 when the output is non-zero. These bits are... Figure 23 The data elements are represented as B0-B15. The data elements and bits are provided to the compression packer 2320.

[0185] The compression packer 2320 can generate a compressed activation vector 2302, which includes non-zero data elements output from the comparator 2310. All data elements in the compressed activation vector 2302 can be non-zero. The compressed activation vector 2302 can have a smaller size (e.g., fewer data elements) than the activation tensor 2301. The compression packer 2320 also generates a sparsity graph 2303, which indicates the positions of the data elements of the compressed activation vector 2302 within the activation tensor 2301. In some embodiments, the compression packer 2320 uses bits B0-B15 to generate the sparsity graph 2303. The sparsity graph 2303 can have the same size as the activation tensor 2301.

[0186] Example methods for executing DNN layers Figure 24 This is a flowchart illustrating a method for executing a DNN layer according to various embodiments. Method 2400 can be executed by the DPU 430 in Figure 3. Although method 2400 is referred to Figure 24 The flowchart shown illustrates this, but alternatively, many other methods for accelerating DNN layers can be used. For example, the method can be modified... Figure 24The execution order of the steps. As another example, some steps can be changed, deleted, or combined.

[0187] The DPU 430 generates a sparse graph of 2410 weight vectors by decompressing the compressed graph. The weight vectors comprise M weights from the neural network layers. The sparse graph indicates the sparsity pattern of the weight vectors, which includes M elements. The compressed graph contains fewer elements than the sparse graph. In some embodiments, the sparsity pattern is a structured sparsity pattern, where N weights within every M consecutive weights in the kernel of the neural network layer are non-zero.

[0188] The DPU 430 stores the activations of 2420 neural network layers in its data storage unit.

[0189] The DPU 430 uses a sparse graph to generate one or more memory addresses. In some embodiments, the DPU 430 feeds the sparse graph into a lookup table. The lookup table associates the sparse graph with one or more read port indices and outputs one or more read port indices. The DPU 430 generates one or more memory addresses from the one or more read port indices. In some embodiments, the DPU 430 selects a lookup table from a plurality of LUTs. The plurality of LUTs are configured for different structured sparsity patterns. In some embodiments, the DPU 430 generates a first memory address by appending one or more bits to a first read port index. The DPU 430 also generates a second memory address by appending one or more bits to a second read port index.

[0190] In some embodiments, DPU 430 transforms a sparse graph into other sparse graphs with different sparsity patterns. DPU 430 uses a lookup table that associates read port indices with other sparse graphs to determine the read port indices. DPU 430 generates one or more memory addresses based on the read port indices. In some embodiments, DPU 430 transforms the sparse graph by splitting it into other sparse graphs. Each of these other sparse graphs is a part of the original sparse graph.

[0191] Based on one or more memory addresses, the DPU 430 reads one or more activations from the data storage unit 2440 to the PE. In some embodiments, the one or more activations are N activations. In some embodiments, multiple weights, including one or more weights, are stored in an additional data storage unit. The DPU 430 reads multiple weights from the additional data storage unit based on the number of multipliers in the PE.

[0192] The DPU 430 performs 2450 computations on one or more activations and weights in one or more weight vectors via the PE to execute a neural network layer. In some embodiments, the PE performs computations in a sparse mode of the device including the PE. The lookup table includes different sparsity graphs corresponding to the dense mode of the device. In some embodiments, the weight vector includes non-zero weights in addition to one or more weights. Based on the zero activations in the activation vector corresponding to the weight vector, the DPU 430 prevents non-zero weights from being passed to the PE.

[0193] Example computing device Figure 25 This is a block diagram of an example computing device 2500 according to various embodiments. In some embodiments, the computing device 2500 may be used as at least a portion of a DNN system 400. Multiple components are... Figure 25 The components are shown as included in computing device 2500, but any one or more of these components may be omitted or copied to suit the application. In some embodiments, some or all of the components included in computing device 2500 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. Additionally, in various embodiments, computing device 2500 may not include... Figure 25 The computing device 2500 may include one or more components as shown, but may include interface circuitry for coupling to one or more components. For example, the computing device 2500 may not include the display device 2506, but may include display device interface circuitry (e.g., connector and driver circuitry) to which the display device 2506 may be coupled. In another set of examples, the computing device 2500 may not include the audio input device 2518 or the audio output device 2508, but may include audio input or output device interface circuitry (e.g., connector and support circuitry) to which the audio input device 2518 or the audio output device 2508 may be coupled.

[0194] Computing device 2500 may include processing device 2502 (e.g., one or more processing devices). Processing device 2502 processes electronic data from registers and / or memory to transform the electronic data into other electronic data that can be stored in registers and / or memory. Computing device 2500 may include memory 2504, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drive. In some embodiments, memory 2504 may include memory sharing a die with processing device 2502. In some embodiments, memory 2504 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for executing a DNN layer (e.g., combining...). Figure 24 The described method 2400) or by Figure 4 Some operations are performed by the DNN system 400 (e.g., DPU 430). Instructions stored in one or more non-transitory computer-readable media can be executed by the processing device 2502.

[0195] In some embodiments, computing device 2500 may include communication chip 2512 (e.g., one or more communication chips). For example, communication chip 2512 may be configured to manage wireless communication for transmitting data to and from computing device 2500. The term "wireless" and its derivatives can be used to describe circuits, apparatuses, systems, methods, techniques, communication channels, etc., that can transmit data using modulated electromagnetic radiation over a non-solid medium. This term does not imply that associated apparatuses do not contain any wiring, although in some embodiments they may not contain any wiring.

[0196] The 2512 communication chip can implement any of the following wireless standards or protocols, including but not limited to: Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.11 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), Long Term Evolution (LTE) projects along with any amendments, updates, and / or revisions (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). IEEE 802.16 compliant Broadband Wireless Access (BWA) networks are often referred to as WiMAX networks, an acronym for Global Microwave Access Interoperability, which is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The 2512 communication chip can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication chip 2512 can operate according to Enhanced GSM Data Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication chip 2512 can operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolved Data Optimization (EV-DO) and its derivative protocols, and any other wireless protocol designated as 3G, 4G, 5G, and above. In other embodiments, communication chip 2512 can operate according to other wireless protocols. Computing device 2500 may include antenna 2522 to facilitate wireless communication and / or receive other wireless communications (e.g., AM or FM radio transmissions).

[0197] In some embodiments, the communication chip 2512 can manage wired communications, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 2512 may include multiple communication chips. For example, a first communication chip 2512 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, while a second communication chip 2512 may be dedicated to longer-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, the first communication chip 2512 may be dedicated to wireless communications, while the second communication chip 2512 may be dedicated to wired communications.

[0198] The computing device 2500 may include a battery / power circuit 2514. The battery / power circuit 2514 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 2500 to a power source (e.g., AC line power) that is separate from the computing device 2500.

[0199] The computing device 2500 may include a display device 2506 (or a corresponding interface circuit as described above). The display device 2506 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0200] The computing device 2500 may include an audio output device 2508 (or a corresponding interface circuit as described above). For example, the audio output device 2508 may include any device that generates auditory indicators, such as a speaker, headphones, or earphones.

[0201] The computing device 2500 may include an audio input device 2518 (or a corresponding interface circuit as described above). The audio input device 2518 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital musical instrument (e.g., a musical instrument with a Musical Instrument Digital Interface (MIDI) output).

[0202] The computing device 2500 may include a GPS device 2516 (or a corresponding interface circuit as described above). As is known in the art, the GPS device 2516 can communicate with a satellite-based system and can receive the location of the computing device 2500.

[0203] The computing device 2500 may include another output device 2510 (or a corresponding interface circuit as described above). Examples of the other output device 2510 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0204] The computing device 2500 may include another input device 2520 (or a corresponding interface circuit as described above). Examples of the other input device 2520 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a quick-response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0205] The computing device 2500 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), ultra-mobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 2500 can be any other electronic device that processes data.

[0206] Select Example The following paragraphs provide various examples of the embodiments disclosed herein.

[0207] Example 1 provides an apparatus comprising: a data storage unit configured to store activations of a neural network layer; a sparsity logic unit configured to: receive a sparse graph indicating a sparse pattern of a weight vector, the weight vector including M weights of the neural network layer, the sparse graph comprising M elements and generated by decompressing a compressed graph having fewer elements than the sparse graph; generate one or more memory addresses using the sparse graph; and read one or more activations from the data storage unit based on the one or more memory addresses; and a PE configured to perform computations on the one or more activations and one or more weights of the weight vector for executing the neural network layer.

[0208] Example 2 provides the device of Example 1, wherein the sparsity logic unit includes a lookup table that associates the sparsity graph with one or more read port indices, and the sparsity logic unit is further configured to determine the one or more memory addresses by converting the one or more read port indices into the one or more memory addresses.

[0209] Example 3 provides the device of Example 2, wherein the sparsity logic unit includes an additional lookup table, and the lookup table and the additional lookup table are configured for different structured sparsity patterns of weights.

[0210] Example 4 provides a device from Example 2 or 3, wherein the PE is configured to perform the computation in the sparse mode of the device, and the lookup table includes different sparsity graphs corresponding to the dense mode of the device.

[0211] Example 5 provides a device from any of Examples 2-4, wherein converting the one or more read port indices into the one or more memory addresses includes: generating a first memory address by appending one or more bits to a first read port index; and generating a second memory address by appending the one or more bits to a second read port index.

[0212] Example 6 provides a device for any one of Examples 1-5, wherein the sparsity pattern is a structured sparsity pattern, in which N weights within every M consecutive weights in the kernel of the neural network layer are non-zero, and the one or more activations are N activations.

[0213] Example 7 provides a device of any one of Examples 1-6, further comprising: an additional data storage unit configured to store the one or more weights, wherein the sparse logic unit is further configured to read the one or more weights from the additional data storage unit based on the number of multipliers in the PE.

[0214] Example 8 provides a device of any one of Examples 1-7, wherein the sparsity logic unit is configured to generate the one or more memory addresses by: converting the sparsity graph into other sparsity graphs with different sparsity patterns; determining the read port index using a lookup table that associates the read port index with the other sparsity graphs; and generating the one or more memory addresses based on the read port index.

[0215] Example 9 provides the device of Example 8, wherein transforming the sparse graph into other sparse graphs includes: dividing the sparse graph into other sparse graphs, wherein each of the other sparse graphs is a part of the sparse graph.

[0216] Example 10 provides a device from any of Examples 1-9, wherein, in addition to the one or more weights, the weight vector includes non-zero weights, and the sparsity logic unit is configured to prevent the non-zero weights from being passed to the PE based on zero activations in the activation vector corresponding to the weight vector.

[0217] Example 11 provides an apparatus comprising: a data loading unit configured to generate a sparse graph of weight vectors by decompressing a compressed graph, the weight vectors including weights of a number M neural network layers, the sparse graph indicating a sparse pattern of the weight vectors including M elements, the compressed graph including fewer elements than the sparse graph; and a processing engine comprising: a data storage unit configured to store activations of the neural network layers; a sparse logic unit configured to: receive the sparse graph from the data loading unit, generate one or more memory addresses using the sparse graph, and read one or more activations from the data storage unit based on the one or more memory addresses; and a PE configured to perform computations on the one or more activations and one or more weights of the weight vectors for executing the neural network layers.

[0218] Example 12 provides the device of Example 11, wherein the data loading unit is further configured to: read the activation of the neural network layer from memory; and write the activation of the neural network layer to the data storage unit.

[0219] Example 13 provides a device of Example 11 or 12, wherein the sparsity logic unit includes a lookup table that associates the sparsity graph with one or more read port indices, and the sparsity logic unit is further configured to determine the one or more memory addresses by converting the one or more read port indices into the one or more memory addresses.

[0220] Example 14 provides a device of any one of Examples 11-13, wherein the sparsity logic unit is configured to generate the one or more memory addresses by: converting the sparsity graph into other sparsity graphs with different sparsity patterns; determining the read port index using a lookup table that associates the read port index with the other sparsity graphs; and generating the one or more memory addresses based on the read port index.

[0221] Example 15 provides a device of any one of Examples 11-14, wherein, in addition to the one or more weights, the weight vector includes non-zero weights, and the sparsity logic unit is configured to prevent the non-zero weights from being passed to the PE based on zero activations in the activation vector corresponding to the weight vector.

[0222] Example 16 provides a method comprising: generating a sparse graph of weight vectors by decompressing a compressed graph, the weight vectors including weights of M neural network layers, the sparse graph indicating a sparse pattern of the weight vectors including M elements, the compressed graph including fewer elements than the sparse graph; storing activations of the neural network layers in a data storage unit; using the sparse graph to generate one or more memory addresses; reading one or more activations from the data storage unit into a PE based on the one or more memory addresses; and using the PE to compute calculations on the one or more activations and one or more weights of the weight vector for executing the neural network layer.

[0223] Example 17 provides the method of Example 16, wherein generating the one or more memory addresses includes: inputting the sparsity graph into a lookup table that associates the sparsity graph with one or more read port indices and outputs the one or more read port indices; and generating the one or more memory addresses from the one or more read port indices.

[0224] Example 18 provides the method of Example 17, wherein generating the one or more memory addresses further includes: selecting the lookup table from a plurality of LUTs configured for different structured sparsity patterns.

[0225] Example 19 provides the method of Example 17 or 18, wherein generating the one or more memory addresses includes: generating a first memory address by appending one or more bits to a first read port index; and generating a second memory address by appending the one or more bits to a second read port index.

[0226] Example 20 provides a method of any one of Examples 17-19, wherein the device including the processing element has a dense mode and a sparse mode, the processing element is configured to perform the computation in the sparse mode of the device, and the lookup table includes different sparsity graphs corresponding to the dense mode of the device.

[0227] Example 21 provides a method of any one of Examples 16-20, wherein the sparsity pattern is a structured sparsity pattern, in which N weights within every M consecutive weights in the kernel of the neural network layer are nonzero, and the one or more activations are N activations.

[0228] Example 22 provides a method of any one of Examples 16-21, wherein the one or more weights are stored in an additional data storage unit, and the method further includes: reading the one or more weights from the additional data storage unit based on the number of multipliers in the PE.

[0229] Example 23 provides a method for any of Examples 16-22, wherein generating the one or more memory addresses includes: converting the sparse graph into other sparse graphs with different sparse patterns; determining the read port index using a lookup table that associates the read port index with the other sparse graphs; and generating the one or more memory addresses based on the read port index.

[0230] Example 24 provides the method of Example 23, wherein transforming the sparse graph into other sparse graphs includes: dividing the sparse graph into other sparse graphs, wherein each of the other sparse graphs is a part of the sparse graph.

[0231] Example 25 provides a method of any one of Examples 16-24, wherein, in addition to the one or more weights, the weight vector also includes non-zero weights, and the method further includes: preventing the non-zero weights from being passed to the PE based on zero activations in the activation vector corresponding to the weight vector.

[0232] Example 26 provides a non-transitory computer-readable medium that stores one or more executable instructions to perform the methods of any one of Examples 16-25.

[0233] Example 27 provides an apparatus including: a computer processor for executing computer program instructions; and a non-transitory computer-readable storage medium for storing the computer program instructions, which are executable by the computer processor to perform the method of any one of Examples 16-25.

[0234] The above description of the implementations illustrated in this disclosure, including those described in the abstract, is not intended to be exhaustive or to limit this disclosure to its precise form. While specific implementations and examples of this disclosure have been described herein for illustrative purposes, various equivalent modifications are possible within the scope of this disclosure, as will be recognized by those skilled in the art. These modifications can be made to this disclosure based on the detailed description above.

Claims

1. An apparatus comprising: The data storage unit is configured to store the activation of neural network layers; Sparse logic units are configured as follows: The system receives a sparse graph indicating the sparse pattern of a weight vector, the weight vector comprising M weights of the neural network layers, and the sparse graph comprising M elements and generated by decompressing a compressed graph having fewer elements than the sparse graph. One or more memory addresses are generated using the sparse graph, and Based on the one or more memory addresses, one or more activations are read from the data storage unit; as well as A processing element is configured to perform computations on the one or more activations and one or more weights in the weight vector for executing the neural network layer.

2. The device according to claim 1, wherein, The sparsity logic unit includes a lookup table that associates the sparsity graph with one or more read port indices, and the sparsity logic unit is further configured to determine the one or more memory addresses by converting the one or more read port indices into the one or more memory addresses.

3. The device according to claim 2, wherein, The sparse logic unit includes an additional lookup table, and the lookup table and the additional lookup table are configured for different structured sparsity patterns of weights.

4. The device according to claim 2 or 3, wherein, The device has a dense mode and a sparse mode, the processing element is configured to perform the computation in the sparse mode of the device, and the lookup table includes different sparsity graphs corresponding to the dense mode of the device.

5. The device according to any one of claims 2-4, wherein, Converting the one or more read port indices into the one or more memory addresses includes: A first memory address is generated by appending one or more bits to the first read port index; and A second memory address is generated by appending one or more bits to the second read port index.

6. The device according to any one of claims 1-5, wherein, The sparsity pattern is a structured sparsity pattern, in which N weights within every M consecutive weights in the kernel of the neural network layer are non-zero, and the one or more activations are N activations.

7. The device according to any one of claims 1-6, further comprising: Additional data storage units are configured to store the one or more weights. The sparse logic unit is further configured to read the one or more weights from the additional data storage unit based on the number of multipliers in the processing element.

8. The device according to any one of claims 1-7, wherein, The sparse logic unit is configured to generate the one or more memory addresses by: The sparse graph is then transformed into other sparse graphs with different sparsity patterns. The read port index is determined using a lookup table that associates the read port index with the other sparse graphs; as well as The one or more memory addresses are generated based on the read port index.

9. The device according to claim 8, wherein, Transforming the sparse graph into other sparse graphs includes: The sparse graph is divided into the other sparse graphs, each of which is a part of the sparse graph.

10. The device according to any one of claims 1-9, wherein, In addition to the one or more weights, the weight vector also includes non-zero weights, and the sparse logic unit is configured to prevent the non-zero weights from being passed to the processing element based on zero activations in the activation vector corresponding to the weight vector.

11. An apparatus comprising: A data loading unit is configured to generate a sparse graph of weight vectors by decompressing a compressed graph, the weight vectors including weights of M neural network layers, the sparse graph indicating a sparse pattern of the weight vectors including M elements, and the compressed graph including fewer elements than the sparse graph. as well as Processing engine, including: Data storage units are configured to store the activations of the neural network layers. Sparse logic units are configured as follows: The sparsity graph is received from the data loading unit. One or more memory addresses are generated using the sparse graph, and Based on the one or more memory addresses, read one or more activations from the data storage unit, and A processing element is configured to perform computations on the one or more activations and one or more weights in the weight vector for executing the neural network layer.

12. The device according to claim 11, wherein, The data loading unit is further configured to: Read the activation of the neural network layer from memory; and The activation of the neural network layer is written to the data storage unit.

13. The device according to claim 11 or 12, wherein, The sparsity logic unit includes a lookup table that associates the sparsity graph with one or more read port indices, and the sparsity logic unit is further configured to determine the one or more memory addresses by converting the one or more read port indices into the one or more memory addresses.

14. The device according to any one of claims 11-13, wherein, The sparse logic unit is configured to generate the one or more memory addresses by: The sparse graph is then transformed into other sparse graphs with different sparsity patterns. The read port index is determined using a lookup table that associates the read port index with the other sparse graphs; as well as The one or more memory addresses are generated based on the read port index.

15. The device according to any one of claims 11-14, wherein, In addition to the one or more weights, the weight vector also includes non-zero weights, and the sparse logic unit is configured to prevent the non-zero weights from being passed to the processing element based on zero activations in the activation vector corresponding to the weight vector.

16. A method comprising: A sparse graph of weight vectors is generated by decompressing a compressed graph, the weight vectors comprising weights of M neural network layers, the sparse graph indicating the sparsity pattern of the weight vectors comprising M elements, and the compressed graph comprising fewer elements than the sparse graph. The activation of the neural network layer is stored in the data storage unit; Use the sparse graph to generate one or more memory addresses; Based on the one or more memory addresses, one or more activations are read from the data storage unit into the processing element; as well as The processing element calculates one or more activations and one or more weights in the weight vector to execute the neural network layer.

17. The method according to claim 16, wherein, Generating the one or more memory addresses includes: The sparse graph is input into a lookup table, which associates the sparse graph with one or more read port indices and outputs the one or more read port indices; and The one or more memory addresses are generated from the one or more read port indices.

18. The method according to claim 17, wherein, Generating the one or more memory addresses further includes: The lookup table is selected from a plurality of lookup tables configured for different structured sparsity patterns.

19. The method according to claim 17 or 18, wherein, Generating the one or more memory addresses includes: A first memory address is generated by appending one or more bits to the first read port index; and A second memory address is generated by appending one or more bits to the second read port index.

20. The method according to any one of claims 17-19, wherein, The device including the processing element has a dense mode and a sparse mode, the processing element is configured to perform the computation in the sparse mode of the device, and the lookup table includes different sparsity graphs corresponding to the dense mode of the device.

21. The method according to any one of claims 16-20, wherein, The sparsity pattern is a structured sparsity pattern, in which N weights within every M consecutive weights in the kernel of the neural network layer are non-zero, and the one or more activations are N activations.

22. The method according to any one of claims 16-21, wherein, The one or more weights are stored in an additional data storage unit, and the method further includes: The one or more weights are read from the additional data storage unit based on the number of multipliers in the processing element.

23. The method according to any one of claims 16-22, wherein, Generating the one or more memory addresses includes: The sparse graph is then transformed into other sparse graphs with different sparsity patterns. The read port index is determined using a lookup table that associates the read port index with the other sparse graphs; and The one or more memory addresses are generated based on the read port index.

24. The method according to claim 23, wherein, Transforming the sparse graph into other sparse graphs includes: The sparse graph is divided into the other sparse graphs, each of which is a part of the sparse graph.

25. The method according to any one of claims 16-24, wherein, In addition to the one or more weights mentioned above, the weight vector also includes non-zero weights, and the method further includes: Based on zero activation in the activation vector corresponding to the weight vector, the non-zero weights are prevented from being passed to the processing element.

Images