Neural network accelerator with configurable data storage

Abstract

A data processing unit (DPU) in a neural network accelerator can include compute units, memory, and data transfer units. The compute units can perform computations in a neural network layer. The memory stores data used and generated by the compute units. The transfer units can transfer data between the memory and the compute units, and can include one or more configurable data stores that can be configured to store different types of data for different modes of operation of the DPU. An example configurable data store can store sparsity data when the DPU is operating in one mode, and input or output data for a neural network layer when the DPU is operating in another mode. The sparsity data can indicate sparsity of the input or output data for a neural network layer, and can be used to accelerate execution of the neural network layer or a next layer.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 640,074, filed April 29, 2024, entitled “Reconfigurable Storage for Cross-Modal Functionality and Performance Uplift in Deep Neural Networks,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to neural networks (also known as “deep neural networks” or “DNNs”), and more specifically, to DNN accelerators having configurable data storage. Background Technology

[0003] Because of their high accuracy, deep neural networks (DNNs) are widely used in various artificial intelligence applications, ranging from computer vision to speech recognition and natural language processing. However, this high accuracy comes at the cost of significant computational cost. DNNs are extremely computationally demanding due to the potentially large number of operations and data read / write operations. Therefore, techniques to improve the efficiency of DNNs are needed. Attached Figure Description

[0004] The various embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For ease of description, similar reference numerals denote similar structural elements. In the accompanying figures, embodiments are shown by way of example rather than limitation.

[0005] Figure 1 Example DNNs according to various embodiments are shown.

[0006] Figure 2 Example convolutions according to various embodiments are shown.

[0007] Figures 3A to 3F Various storage formats for tensors according to various embodiments are shown.

[0008] Figure 4 This is a block diagram of a DNN system according to various embodiments.

[0009] Figure 5 Example sparse units according to various embodiments are shown.

[0010] Figure 6 Example sparse cell arrays according to various embodiments are shown.

[0011] Figure 7Example processing elements (PEs) according to various embodiments are shown.

[0012] Figure 8 Example input transmission units (IDUs) according to various embodiments are shown.

[0013] Figure 9 Example output transmission units (ODUs) according to various embodiments are shown.

[0014] Figure 10 It is shown that the sparse storage in the ODU according to various embodiments can be reconfigured for staging.

[0015] Figures 11A to 11C It is shown that the sparse storage in the ODU according to various embodiments can be reconfigured for reordering.

[0016] Figure 12 It is shown that the sparse storage in the ODU according to various embodiments can be reconfigured for reordering and transposition.

[0017] Figure 13 It is shown that the replacement storage in the ODU according to various embodiments can be reconfigured for temporary storage.

[0018] Figure 14 It is shown that the write storage in the ODU according to various embodiments can be reconfigured as transaction first-in-first-out (FIFO).

[0019] Figures 15A to 15L The process of permuting tensors in a transit storage unit according to various embodiments is illustrated.

[0020] Figure 16 This is a block diagram of a DNN module according to various embodiments.

[0021] Figure 17 This is a block diagram of an example computing device according to various embodiments. Detailed Implementation

[0022] Overview The past decade has witnessed the rapid rise of data processing technologies based on artificial intelligence (AI), particularly those based on deep neural networks (DNNs). DNNs are widely used in computer vision, speech recognition, image and video processing primarily due to their ability to achieve accuracy surpassing human levels. DNNs typically consist of sequences of layers. A DNN layer can include one or more deep learning operations (also known as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax operations, pooling, element-wise operations, linear operations, non-linear operations, and more.

[0023] The input or output data for deep learning operations can be arranged in a data structure called a tensor. A tensor is a data structure with multiple elements in one or more dimensions. Example tensors include vectors (which are one-dimensional (1D) tensors), matrices (which are two-dimensional (2D) tensors), three-dimensional (3D) tensors, four-dimensional (4D) tensors, and even tensors of higher dimensions. The dimensions of a tensor can correspond to axes, such as axes in a coordinate system. Dimensions can be measured by the number of data points along an axis. The dimensions of a tensor can define the shape of the tensor. A DNN layer can receive one or more input tensors and compute an output tensor based on one or more input tensors. Taking a convolutional layer as an example, the input tensors include activation tensors (also called "input feature maps (IFM)" or "input activation tensors") and weight tensors, with the activation tensors containing one or more activation values ​​(also called "input elements"). Weight tensors can be kernels (2D weight tensors), filters (3D weight tensors), or a set of filters (4D weight tensors). Convolutions can be performed on the input activation tensor and weight tensor to compute the output activation tensor in the convolutional layer.

[0024] The significant improvements in DNN model size and accuracy, coupled with the rapid increase in the computing power of execution platforms, have enabled DNN applications to be used even in resource-constrained mobile and edge devices with limited energy supplies. DNN models can be executed by DNN accelerators, for example, for training or inference. A DNN accelerator can be one or more data processing units (DPUs) or include one or more data processing units (DPUs). A DPU can also be called a neural processing unit or computational block. A DPU may include PEs capable of performing neural network operations.

[0025] The execution of many DNN models can be improved through tensor permutation. Tensor permutation can be used to improve the utilization of the DPU. In an example where the DPU is designed to be fully utilized when the minimum dimension of the input tensor is 4×4×1, the utilization will be 25% when the input tensor is 1×4×4. By permuting the input tensor to 4×4×1, 100% utilization can be achieved, which can significantly improve the efficiency and performance of the DPU. Alternatively or additionally, tensor permutation may be necessary for some DNN models (e.g., Transformer-based networks). In the example, tensor permutation is used to transform 2D tensors (e.g., spatially sized as batch size × sequence length) into the input embedding blocks of a Transformer-based network. (2D tensors) are converted to 3D tensors (e.g., spatial size is batch size × sequence length × embedding dimension). (3D tensors). Tensor permutation is also used in encoder blocks of Transformer-based networks.

[0026] A 3D tensor can have three dimensions, corresponding to the X, Y, and Z axes in 3D space. Data elements (e.g., activation values, weights, etc.) in a 3D tensor can be arranged along these axes. Each data element can be represented by (x, y, z) coordinates, which indicate the position of the data element in the 3D tensor. The shape and spatial size of a 3D tensor can be defined by the lengths of its three dimensions, which can be the number of data elements along the three axes. In the example, a 3D tensor with a spatial size of 3×4×5 can have a total of three elements along the X-axis, four elements along the Y-axis, and five elements along the Z-axis. In some embodiments, the Z-axis can correspond to channels (e.g., input channels, output channels, etc.). The length along the Z-axis can indicate the total number of channels in the tensor. (x, y) can represent a spatial point in the 3D tensor.

[0027] Tensors in DNNs can be stored or transmitted in various formats, such as X-major (e.g., XYZ or XZY format), Y-major (e.g., YXZ or YZX format), and Z-major (e.g., ZXY or ZYX format). The format of a tensor defines the order in which data points in the tensor are stored, written, or retrieved. The first character indicates the contiguous dimension of the data points in memory. The second character indicates the dimension of the data points that can be accessed after accessing contiguous data points in memory. The third character indicates the dimension of the data points that can be accessed after the data points in the dimension represented by the second character have been exhausted. For example, in the ZXY format, the access order starts from the Z dimension, then moves to the X dimension, and finally to the Y dimension. Data points in a tensor are contiguous in memory along the Z dimension, meaning that data points with the same (x,y) coordinates are contiguous in memory. Tensors can be read from memory in other formats through tensor permutations. The output tensors of neural network operations can be stored in Z-major order format because the DPU can accumulate partial products along the channels or the Z-axis. The Z-major order format of a tensor can be converted to X-major order format by X-major order permutation, or to Y-major order format by Y-major order permutation.

[0028] In addition to one or more computation units, the DPU in a DNN accelerator may also include one or more data transfer units. These data transfer units can transfer data between the computation unit(s) and memory (e.g., static random access memory (SRAM)). For example, the DPU may include a loading unit for loading data into the computation unit(s) and an ejection unit for ejecting data from the computation unit(s). The data transfer units may include one or more data storage units for tensor computations. For example, tensor permutation can be performed by writing an incoming tensor to memory at one or more desired locations, then reading it from the stored tensor(s) in the desired order and writing it back to memory. There may also be one or more data storage units that can store sparse data, which can indicate the sparsity of the inputs or outputs of neural network layers, and can be used to accelerate computations in the computation unit(s). Sparse data may include sparse tensors, such as sparse bitmaps. These data storage units may be used in some functional modes (also called "operating modes") of the DPU, but not in others. For example, when the output of (one or more) computation units is already in Z-major order, the storage used for permutation in the Z-major order functional mode of the DPU may not be used. Furthermore, when the computation performed by (one or more) computation units does not involve sparsity acceleration, the sparsity storage in the dense mode of the DPU may not be used. For functional modes that do not use storage, there may be wasted area, which could potentially harm the performance of the DNN accelerator.

[0029] Embodiments of this disclosure can improve at least some of the challenges and problems described above by providing a DNN accelerator with configurable data storage. Example configurable data storage can be configured to have different storage capabilities for different operating modes of the DNN accelerator. For instance, configurable data storage can be configured to store different types of data for different operating modes, which can improve the area efficiency and performance of the DNN accelerator.

[0030] In various embodiments of this disclosure, the DNN accelerator includes one or more DPUs capable of performing operations in a DNN. The DPU may include a computation unit, local memory, an IDU, and an ODU. The computation unit may include a PE for performing computations in the neural network operation. The local memory stores data used and generated by the computation unit. The IDU transfers data (e.g., inputs to neural network layers) from the local memory to the computation unit. The IDU may also be referred to as an input delivery module or a loading module. The ODU transfers data (e.g., outputs to neural network layers) from the computation unit to the memory. The ODU may also be referred to as an output delivery module or an output module. The DPU may have different operating modes. For example, the DPU may have: a sparse mode, in which computations in the neural network operation can be accelerated by skipping computations performed on zero-value data elements in the input based on the sparsity of the input to the neural network operation; and a dense mode, in which there is no sparsity acceleration, and the computation unit can process all data elements in the input to the neural network operation. Additionally or alternatively, the DPU may have operating modes corresponding to different data formats, such as Z-major order mode, X-major order mode, and Y-major order mode.

[0031] An IDU or ODU may include one or more configurable data stores. An IDU may include sparse storage (e.g., data random access memory (RAM)) configured to store sparse data (e.g., sparse bitmaps) when the DPU operates in sparse mode, but configured to store input data for neural network operations in dense mode. The sparse storage in the IDU can be used as a response FIFO buffer in dense mode. An ODU may include sparse storage (e.g., data RAM) configured to store sparse data (e.g., sparse bitmaps) when the DPU operates in sparse mode, but configured to store output data for neural network operations in dense mode. The sparse storage in the ODU can be used as a staging buffer in dense Z-master order mode. In dense X-master or Y-master order modes, the sparse storage can be used to facilitate tensor permutations. An ODU may also include permutation storage that performs tensor permutations in X-master or Y-master order modes, but serves as a staging buffer in Z-master order mode. The ODU may also include a write combination buffer, which can be configured once to combine small write transactions and once to be used as a FIFO buffer.

[0032] This disclosure provides data storage in a DNN accelerator that can be reconfigured by utilizing unused data storage in certain operating modes of the DNN accelerator to extend existing data storage or provide new data storage capabilities. With this reconfigurable and versatile data storage, the DNN accelerator of this disclosure can achieve better area efficiency and performance compared to currently available DNN accelerators.

[0033] For illustrative purposes, specific figures, materials, and configurations have been set forth to provide a thorough understanding of the illustrative implementation. However, it will be apparent to those skilled in the art that this disclosure may be practiced without specific details, and / or may be practiced using only some of the aspects described. In other instances, well-known features have been omitted or simplified so as not to obscure the illustrative embodiments.

[0034] Furthermore, reference has been made to the accompanying drawings, which form part of this disclosure, and practical embodiments are illustrated in the drawings by way of illustration. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed as limiting.

[0035] Various operations can be described sequentially as a plurality of discrete actions or operations in a manner most conducive to understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations may not be performed in the order presented. The described operations may be performed in a different order than in the described embodiments. Various additional operations may be performed, or the described operations may be omitted in additional embodiments.

[0036] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" refers to (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" refers to (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used to refer to a measurement range, the term "between" includes the endpoints of the measurement range.

[0037] This description uses the phrases "in one embodiment" or "in an embodiment," both of which can refer to one or more of the same or different embodiments. Terms such as "comprising," "including," "having," etc., used with respect to embodiments of this disclosure are synonyms. This disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to interpret various features of the drawings; however, these terms are merely for ease of discussion and do not imply any desired or required direction. The drawings are not necessarily drawn to scale. Unless otherwise stated, the use of ordinal adjectives such as "first," "second," and "third" to describe common objects indicates only different instances of the similar objects referred to and is not intended to imply that the objects described must be arranged in a given order, whether temporally, spatially, in rank, or otherwise.

[0038] In the following detailed description, terms commonly used by those skilled in the art will be used to describe various aspects of the illustrative implementations in order to convey the substance of their work to others skilled in the art.

[0039] The terms “substantially,” “close to,” “approximately,” “near,” and “about” generally refer to within + / -20% of the target value as described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to within + / -5-20% of the target value based on the description herein or known in the art.

[0040] Furthermore, the terms “comprising,” “including,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such a method, process, apparatus, or DNN accelerator. Additionally, the term “or” refers to inclusive “or” rather than exclusive “or.”

[0041] The systems, methods, and apparatuses disclosed herein are innovative in several ways, but none of them alone is responsible for all the desired properties disclosed herein. Details of one or more implementations of the subject matter described herein are set forth in the following description and figures.

[0042] Example DNN Figure 1 An example DNN 100 according to various embodiments is shown. The DNN 100 can be constructed using a DNN accelerator (e.g., Figure 4 The execution is performed by a DNN accelerator 402. In one example, DNN 100 may be a convolution-based DNN. In other examples, DNN 100 may be other types of DNNs. For illustrative purposes, DNN 100 includes a sequence of layers comprising multiple convolutional layers 110 (referred to individually as "convolutional layer 110"), multiple pooling layers 120 (referred to individually as "pooling layer 120"), and multiple fully connected layers 130 (referred to individually as "fully connected layer 130"). In other embodiments, DNN 100 may include fewer, more, or different layers. During the execution of DNN 100, the layer execution of the DNN includes tensor computations of a variety of tensor operations, such as matrix multiplication, convolution (e.g., multiply-accumulate (MAC) operation, pooling operation, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or certain combinations of these operations.

[0043] Convolutional layer 110 summarizes the presence of features input to DNN 100. Convolutional layer 110 acts as a feature extractor. The first layer of DNN 100 is convolutional layer 110. In the example, convolutional layer 110 performs a convolution operation on the input tensor 140 (also known as IFM 140) and filter 150. Figure 1 As shown, the IFM 140 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 140 includes three input channels, each represented by a 7×7 two-dimensional (2D) matrix. Each row of the 7×7 2D matrix contains 7 input elements (also called input points), and each column contains 7 input elements. The filter 150 is represented by a 3×3×3 3D matrix. The filter 150 includes three kernels, each corresponding to a different input channel of the IFM 140. The kernel is a 2D matrix of weights, where the weights are arranged by column and row. The kernel can be smaller than the IFM. Figure 1 In this embodiment, each kernel is represented by a 3×3 2D matrix. Each row of the 3×3 kernel contains 3 weights, and each column also contains 3 weights. The weights can be initialized and updated using gradient descent via backpropagation. The magnitude of the weights can represent the importance of filter 150 in extracting features from IFM 140.

[0044] The convolution involves a MAC operation on the input elements in IFM 140 and the weights in filter 150. The convolution can be either a standard convolution 163 or a depthwise convolution 183. In a standard convolution 163, the entire filter 150 slides over IFM 140. All input channels are combined to produce an output tensor 160 (also called an output feature map (OFM) 160). OFM 160 is represented by a 5×5 2D matrix. Each row of the 5×5 2D matrix contains 5 output elements (also called output points), and each column also contains 5 output elements. For illustration, in... Figure 1 In one embodiment, the standard convolution includes a filter. In an embodiment with multiple filters, the standard convolution can produce multiple output channels in the OFM 160.

[0045] The multiplication applied between a local patch of IFM 140 kernel size and the kernel can be a dot product. A dot product is an element-wise multiplication between a local patch of IFM 140 kernel size and the corresponding kernel, then summed, always producing a single value. Because it produces a single value, this operation is often called a "scalar product." Using a kernel smaller than IFM 140 is intentional because it allows the same kernel (a set of weights) to be multiplied multiple times by IFM 140 at different points on IFM 140. Specifically, the kernel is systematically applied from left to right and top to bottom to each overlapping portion or local patch of IFM 140 kernel size. Multiplying the kernel by IFM 140 once results in a single value. Since the kernel is applied multiple times to IFM 140, the result of the multiplication is the output element of a 2D matrix. Thus, the 2D output matrix from a standard convolution 163 (i.e., OFM 160) is called OFM.

[0046] In depthwise convolution 183, the input channels are not combined. Instead, a MAC operation is performed on each individual input channel and individual kernel, producing the output channel. For example... Figure 1 As shown, depthwise convolution 183 produces a depth output tensor 180. The depth output tensor 180 is represented by a 5×5×3 3D matrix. The depth output tensor 180 includes three output channels, each represented by a 5×5 2D matrix. Each row of the 5×5 2D matrix contains 5 output elements, and each column also contains 5 output elements. Each output channel is the result of a MAC operation performed on the input channels of IFM 140 and the kernel of filter 150. For example, the first output channel (dot pattern) is the result of a MAC operation on the first input channel (dot pattern) and the first kernel (dot pattern); the second output channel (horizontal stripe pattern) is the result of a MAC operation on the second input channel (horizontal stripe pattern) and the second kernel (horizontal stripe pattern); and the third output channel (diagonal stripe pattern) is the result of a MAC operation on the third input channel (diagonal stripe pattern) and the third kernel (diagonal stripe pattern). In such depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are collectively referred to as depth channels. After depthwise convolution, pointwise convolution 193 is performed on the depth output tensor 180 and the 1×1×3 tensor 190 to produce OFM 160.

[0047] OFM 160 is then passed to the next layer in the sequence. In some embodiments, OFM 160 is passed through an activation function. An example activation function is the Rectified Linear Unit (ReLU). ReLU is a computation that directly returns the value provided as input, or returns 0 if the input is 0 or less. Convolutional layer 110 may receive several images as input and compute the convolution of each of them with each kernel. This process may be repeated several times. For example, OFM 160 is passed to a subsequent convolutional layer 110 (i.e., the convolutional layer 110 in the sequence that produces OFM 160). The subsequent convolutional layer 110 performs convolution on OFM 160 with a new kernel and generates a new feature map. The new feature map may also be normalized and resized. The new feature map may be kernelized again by further subsequent convolutional layers 110, and so on.

[0048] In some embodiments, the convolutional layer 110 has four hyperparameters: the number of kernels, the kernel size (e.g., the kernel size is F×F×D pixels), the stride S of dragging the window corresponding to the kernel on the image (e.g., a stride of 1 means moving the window one pixel at a time), and zero padding P (e.g., adding a black outline of P pixels thickness to the input image of the convolutional layer 110). The convolutional layer 110 can perform various types of convolution, such as 2D convolution, dilated or dilated convolution, spatially separable convolution, depthwise separable convolution, transposed convolution, etc. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0049] Pooling layer 120 downsamples the feature map generated by the convolutional layer, for example, by downsampling the presence of features in local blocks that summarize the feature map. Pooling layer 120 is positioned between two convolutional layers 110: a pre-convolutional layer 110 (the convolutional layer 110 preceding pooling layer 120 in the layer sequence) and a post-convolutional layer 110 (the convolutional layer 110 following pooling layer 120 in the layer sequence). In some embodiments, pooling layer 120 is added after convolutional layer 110, for example, after an activation function (e.g., ReLU, etc.) has been applied to OFM 160.

[0050] Pooling layer 120 receives feature maps generated by the preceding convolutional layer 110 and applies pooling operations to these feature maps. Pooling operations reduce the size of the feature maps while preserving their important characteristics. Therefore, pooling operations improve the efficiency of the DNN and avoid overlearning. Pooling layer 120 can perform pooling operations using average pooling (calculating the average value of each local block on the feature map), max pooling (calculating the maximum value of each local block on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is applied with a stride of 2 pixels (2×2 pixels), thereby reducing the size of the feature map by a factor of 2, for example, reducing the number of pixels or values ​​in the feature map to one-quarter of its original size. In one example, pooling layer 120 applied to a 6×6 feature map produces a 3×3 output pooled feature map. The output of pooling layer 120 is fed into the subsequent convolutional layer 110 for further feature extraction. In some embodiments, pooling layer 120 operates on each feature map individually to create a new set of the same number of pooled feature maps.

[0051] Fully connected layer 130 is the last layer of the DNN. Fully connected layer 130 may or may not be convolutional. Fully connected layer 130 receives input operands. The input operands define the outputs of convolutional layer 110 and pooling layer 120 and include the values ​​of the final feature map generated by the last pooling layer 120 in the sequence. Fully connected layer 130 applies a linear combination and activation function to the input operands and generates a vector. This vector can contain as many elements as there are classes: element i represents the probability that an image belongs to class i. Therefore, each element is between 0 and 1, and the sum of all elements is 1. These probabilities are calculated by the last fully connected layer 130 using either a logistic function (for binary classification) or a SoftMax function (for multi-class classification) as the activation function. In some embodiments, fully connected layer 130 multiplies each input element by a weight, sums the results, and then applies an activation function (e.g., logistic if N=2, SoftMax if N>2). This is equivalent to multiplying the input operands by a matrix containing the weights.

[0052] Figure 2 Example convolutions according to various embodiments are shown. This convolution can be a convolutional layer in a DNN (e.g., ...). Figure 1 The deep learning operation is performed in the convolutional layer 110. This convolution can be performed on activation tensor 210 and filter 220 (referred to individually as "filter 220"). The filters can form the weight tensor of the convolution. The result of the convolution is output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator. Examples of DNN accelerators could be... Figure 4 The DNN accelerator 402 is used in the DNN accelerator 402. For example, convolution can be performed by one or more DPUs 430 in the DNN accelerator 402.

[0053] The activation tensor 210 can be computed in previous layers of the DNN. In some embodiments (e.g., where the convolutional layer is the first layer of the DNN), the activation tensor 210 can be an image. Figure 2 In this embodiment, the activation tensor 210 includes activation values ​​(also referred to as "input activation values," "elements," or "input elements") arranged in a 3D matrix. The activation tensor 210 can also be referred to as the input tensor of the convolution. Input elements are data points in the activation tensor 210. The activation tensor 210 has a spatial size. ,in It is the height of the 3D matrix (i.e., the length along the Y-axis, representing the number of activation values ​​in the column of the 3D matrix for each input channel). It is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of activation values ​​in the rows of the 2D matrix for each input channel). This is the depth of the 3D matrix (i.e., the length along the Z-axis, representing the number of input channels). For simplicity and illustration, activation tensor 210 has a spatial size of 7×7×3, meaning activation tensor 210 includes three input channels, each with a 7×7 2D matrix. Each input element in activation tensor 210 can be represented by (X, Y, Z) coordinates. In other embodiments, the height, width, or depth of activation tensor 210 may be different.

[0054] Each filter 220 includes weights arranged in a 3D matrix. The values ​​of the weights can be determined by training a DNN. Filter 220 has a spatial size. ,in It is the height of the filter (i.e., the length along the Y-axis, representing the number of weights in each column of the core). It is the width of the filter (i.e., the length along the X-axis, representing the number of weights in each row of the core). This is the depth of the filter (i.e., the length along the Z-axis, representing the number of channels). In some embodiments, equal For the sake of simplicity and explanation, Figure 2 Each filter 220 in the configuration has a spatial size of 2×3×3, meaning that filter 220 includes two convolutional kernels with a spatial size of 2×3. In other embodiments, the height, width, or depth of filter 220 may vary. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix of each input channel in activation tensor 210.

[0055] Activation values ​​or weights can occupy one or more bytes in memory. The number of bytes for activation values ​​or weights can depend on the data format. For example, when activation values ​​or weights are in INT8 format, the activation value occupies one byte. When activation values ​​or weights are in FP16 format, the activation value or weight occupies two bytes. Other data formats can be used for activation values ​​or weights.

[0056] In the convolution, each filter 220 slides over the activation tensor 210 and generates a 2D matrix for the output channels in the output tensor 230. Figure 2 In this embodiment, the 2D matrix has a spatial size of 5×5. The output tensor 230 includes activation values ​​(also referred to as "output activation values," "elements," or "output elements") arranged in a 3D matrix. Output activation values ​​are data points in the output tensor 230. The output tensor 230 has a spatial size... ,in It is the height of the 3D matrix (i.e., the length along the Y-axis, representing the number of output activation values ​​in the columns of the 2D matrix for each output channel). It is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of output activation values ​​in the rows of the 2D matrix for each output channel). It is the depth of the 3D matrix (i.e., the length along the Z-axis, representing the number of output channels). It can be equal to the number of filters 220 in the convolution. and This can depend on the activation tensor 210 and the height and width of each filter 220. In the example with a kernel size of 1×1, and They can be equal to respectively and .

[0057] As part of the convolution, the 3×3×3 sub-tensor 215 of the activation tensor 210 (in Figure 2 (Highlighted with a dotted pattern) and each filter 220 performs a MAC operation. The result of performing a MAC operation on subtensor 215 and filter 220 is the output activation value. In some embodiments (e.g., embodiments where the convolution is an integer convolution), the output activation value may include 8 bits, such as one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), the output activation value may include more than one byte. For example, the output element may include two bytes.

[0058] After completing the MAC operations on subtensor 215 and all filters 220, vector 235 is produced. Vector 235 is... Figure 2The vector 235 is highlighted using a dotted pattern. It comprises a sequence of output activation values ​​arranged along the Z-axis. The output activation values ​​in vector 235 have the same (x, y) coordinates, but they correspond to different output channels and have different Z-coordinates. The Z-axis dimension of vector 235 can be equal to the total number of output channels in output tensor 230. After generating vector 235, further MAC operations are performed to generate additional vectors until output tensor 230 is produced. Figure 2 In this embodiment, the output tensor 230 is computed in Z-major order. When the output tensor 230 is computed in ZXY format, vectors adjacent to vector 235 along the X-axis can be computed immediately after vector 235. When the output tensor 230 is computed in ZYX format, vectors adjacent to vector 235 along the Y-axis can be computed immediately after vector 235. The output tensor 230 can be permuted, for example by the expulsion module 390, and stored in memory (e.g., local memory 440) in X-major order or Y-major order.

[0059] In some embodiments, multiple MAC units can perform MAC operations on a 3×3×3 subtensor (e.g., subtensor 215) and filter 220. One or more MAC units can receive input operands (e.g., ... Figure 2 The activation operand 217 shown in the figure) and the weight operand (e.g., Figure 2 The weight operand 227 is shown in Figure 210. The activation operand 217 includes a sequence of activation values ​​with the same (x, y) coordinates but different z coordinates. The activation operand 217 includes activation values ​​from each input channel in the activation tensor 210. The weight operand 227 includes a sequence of weights with the same (x, y) coordinates but different z coordinates. The weight operand 227 includes weights from each channel in the filter 220. The activation values ​​in the activation operand 217 and the weights in the weight operand 227 can be sequentially input into the MAC unit. The MAC unit can receive the activation value and weight (“activation value-weight pair”) at once and multiply the activation value by the weight. The position of the activation value in the activation operand 217 can be matched with the position of the weight in the weight operand 227. The activation value and weight can correspond to the same channel.

[0060] Activation values ​​or weights can be floating-point numbers. Floating-point numbers can have various data formats, such as FP32, FP16, BF16, etc. Floating-point numbers are positive or negative numbers with a decimal point. Floating-point numbers can be represented by a bit sequence, which includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing the exponent, and bits representing the mantissa. The mantissa is the portion of the floating-point number that represents its significant digits. Multiplying the mantissa by the exponent of the base yields the actual value of the floating-point number.

[0061] In some embodiments, the output activation values ​​in the output tensor 230 may be further processed based on one or more activation functions before being written to memory or input to the next layer of the DNN. This processing based on the one or more activation functions may be at least part of the post-processing of the convolution. In some embodiments, the post-processing may include one or more other computations, such as offset computation, bias computation, etc. The result of the post-processing may be stored in the local memory of the computation block and used as input to the next DNN layer. In some embodiments, the input activation values ​​in the activation tensor 210 may be the result of post-processing of a previous DNN layer.

[0062] Figures 3A to 3F Various storage formats of tensors according to various embodiments are illustrated. For illustrative purposes, tensors have... The spatial size is defined as eight data points per vector along the X-axis (e.g., per row), eight data points per vector along the Y-axis (e.g., per column), and eight data points per vector along the Z-axis. In other embodiments, the tensor may have other shapes or spatial sizes.

[0063] Figure 3A The tensor is represented in ZYX format. Eight data points with coordinates (0, 0, 0-7) (highlighted with a denser dot pattern) can be stored in memory first. For example, these data points are stored as consecutive bytes in local memory 440. These eight data points are followed by data points with coordinates (0, 1, 0-7), then data points with coordinates (0, 2, 0-7), then data points with coordinates (0, 3, 0-7), and so on up to data points with coordinates (0, 7, 0-7). Fifty-six data points with coordinates (0, 1-7, 0-7) are highlighted with a sparser dot pattern. 0-7 represents all integers from 0 to 7. 1-7 represents all integers from 1 to 7. Then, 64 data points with coordinates (1, 0-7, 0-7) are stored. This continues until all data points of the tensor have been stored.

[0064] Figure 3B The tensor is presented in ZXY format. Using ZXY format, eight data points with coordinates (0, 0, 0-7) (highlighted with a denser dot pattern) can be stored in memory first, followed by data points with coordinates (1, 0, 0-7), then (2, 0, 0-7), then (3, 0, 0-7), and so on up to (7, 0, 0-7). Fifty-six data points with coordinates (1-7, 0, 0-7) are highlighted with a sparser dot pattern. Then, 64 data points with coordinates (0-7, 1, 0-7) are stored. This continues until all data points of the tensor are stored.

[0065] Figure 3C The tensor is represented in XYZ format. Eight data points with coordinates (0-7, 0, 0) (highlighted with a denser dot pattern) are stored in memory first, followed by data points with coordinates (0-7, 1, 0), then (0-7, 2, 0), then (0-7, 3, 0), and so on up to (0-7, 7, 0). Fifty-six data points with coordinates (0-7, 1-7, 0) are highlighted with a sparser dot pattern. Then, 64 data points with coordinates (0-7, 0-7, 1) are stored. This process continues until all data points of the tensor are stored.

[0066] Figure 3D The tensor is presented in XZY format. Using XZY format, eight data points with coordinates (0-7, 0, 0) (highlighted with a denser dot pattern) are stored in memory first, followed by data points with coordinates (0-7, 0, 1), then (0-7, 0, 2), then (0-7, 0, 3), and so on up to (0-7, 0, 7). Fifty-six data points with coordinates (0-7, 0, 1-7) are highlighted with a sparser dot pattern. Then, 64 data points with coordinates (0-7, 1, 0-7) are stored. This process continues until all data points of the tensor are stored.

[0067] Figure 3E The tensor is represented in YZX format. Eight data points with coordinates (0, 0-7, 0) (highlighted with a denser dot pattern) are stored in memory first, followed by data points with coordinates (0, 0-7, 1), then (0, 0-7, 2), then (0, 0-7, 3), and so on up to (0, 0-7, 7). Fifty-six data points with coordinates (0, 0-7, 1-7) are highlighted with a sparser dot pattern. Then, 64 data points with coordinates (1, 0-7, 0-7) are stored. This process continues until all data points of the tensor are stored.

[0068] Figure 3FThe tensor is represented in YXZ format. Eight data points with coordinates (0, 0-7, 0) (highlighted with a denser dot pattern) are stored in memory first, followed by data points with coordinates (1, 0-7, 0), then (2, 0-7, 0), then (3, 0-7, 0), and so on up to (7, 0-7, 0). Fifty-six data points with coordinates (1-7, 0-7, 0) are highlighted with a sparser dot pattern. Then, 64 data points with coordinates (0-7, 0-7, 1) are stored. This process continues until all data points of the tensor are stored.

[0069] In some embodiments, Figures 3A to 3E The format shown can be changed to another format via tensor permutation. Tensor permutation will change the layout of the 256 data points in memory, such as the order in which the data points are arranged in memory, but will not change the value of any data point. Tensor permutation can be performed by ODU 490 before ODU 490 writes the data points to memory.

[0070] Example DNN system Figure 4 This is a block diagram of a DNN system 400 according to various embodiments. The entire DNN system 400 or a portion thereof may be implemented in one or more computing devices, such as... Figure 17 The computing device 1700 in the system. The DNN system 400 can generate and execute DNNs, such as Transformer-based models, convolution-based models, etc. Figure 4 As shown, the DNN system 400 includes a DNN module 401 and a DNN accelerator 402. In other embodiments, the DNN system 400 may include alternative configurations, different, or additional components. For example, the DNN system 400 may include multiple DNN modules or multiple DNN accelerators. Furthermore, the functionality implemented by the components of the DNN system 400 may be performed by another component included in the DNN system 400 or by other systems. In some embodiments, the DNN module 401 and the DNN accelerator 402 may include different types of processing units. In the example, the DNN module 401 may be implemented by one or more central processing units (CPUs). The DNN accelerator 402 may also be referred to as an AI accelerator or an AI processor. The DNN module 401 and the DNN accelerator 402 may be implemented on the same chip or on different chips.

[0071] DNN module 401 facilitates the generation and deployment of DNNs. In some embodiments, DNN module 401 can generate and train DNNs. For example, DNN module 401 can define the hierarchical architecture of the DNN. DNN module 401 can also determine the intrinsic parameters of the DNN through the DNN training process. DNN module 401 can also determine one or more hyperparameters that define how the DNN is trained. Example hyperparameters are sparsity ratios that define the sparsity level of one or more deep learning tensors of the DNN.

[0072] DNN module 401 can also compress the DNN, for example, during or after training. In some embodiments, DNN module 401 can prune weights in one or more layers of the DNN by changing non-zero value weights to zero. DNN module 401 can prune weights based on a target weight sparsity ratio. The weight sparsity ratio can be the ratio of the number of zero-value weights to the total number of weights. In the example where DNN module 401 prunes weights during DNN training, DNN module 401 can prune the weights of a layer to achieve the target sparsity ratio after one or more epochs. DNN module 401 can prevent pruned weights from changing their values ​​for the remainder of the training process. Alternatively, DNN module 401 can allow pruned weights to change their values ​​such that pruned zero-value weights may have non-zero values ​​after further training. DNN module 401 can prune the weights of the layer again after one or more additional epochs.

[0073] DNN module 401 can deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, DNN module 401 can distribute trained, compressed, or validated DNNs to devices or systems that can use the DNNs to perform tasks for which the DNNs were trained (e.g., image classification, motion planning, etc.). In other embodiments, DNN module 401 can use DNN accelerator 402 to facilitate DNN deployment. For example, DNN module 401 can receive data from devices or systems coupled to DNN system 300 and input the received data (or data generated by DNN module 401 (e.g., based on the received data)) into the DNN. DNN module 401 can generate instructions (e.g., configuration descriptors) to control the operation of DNN accelerator 402 during DNN execution. For example, DNN module 401 can generate configuration descriptors that can be used to configure data storage in DNN accelerator 402. DNN module 401 can receive the output of the DNN from DNN accelerator 402. DNN module 401 can transmit the output of the DNN (or the result of processing the output of the DNN) to the device or system. In some embodiments, DNN module 401 can control the execution process of a trained, compressed, or validated DNN. DNN module 401 can act as a compiler for the DNN executed by DNN accelerator 402. DNN module 401 can compile the DNN and generate configuration descriptors or configuration parameters, based on which the DNN can be executed. Certain aspects of DNN module 401 will be described below. Figure 16 supply.

[0074] DNN accelerator 402 executes the DNN provided by DNN module 401. For example, DNN accelerator 402 can execute the DNN by running neural network operations within the DNN. The process of executing neural network operations is also called running neural network operations or implementing neural network operations. The execution of the DNN can be for training the DNN or for performing AI tasks using the DNN. Figure 4 As shown, the DNN accelerator 402 includes a memory 410, a DMA (Direct Memory Access) engine 420, and a DPU 430 (referred to separately as "DPU 430"). In other embodiments, the DNN accelerator 402 may include alternative configurations, different, or additional elements. For example, the DNN accelerator 402 may include more than one memory 410 or DMA engine 420. As another example, the DNN accelerator 402 may include a single DPU 430. Furthermore, the functionality implemented by the components of the DNN accelerator 402 may be performed by another component included in the DNN accelerator 402 or by other systems. The components of the DNN accelerator 402 may be implemented in hardware, software, firmware, or some combination thereof.

[0075] Memory 410 stores data associated with neural network operations performed by the DNN accelerator 402. In some embodiments, memory 410 may store data for DPU 430 to use for DNN execution. Memory 410 may store weights, such as the weights of convolutional layers, which are determined by training the DNN. Memory 410 may also store inputs or outputs of DNN layers, such as data generated by DPU 430 by performing deep learning operations in the DNN. Example neural network operations include convolution (also known as a “convolution operation”), layer normalization, SoftMax, matrix multiplication, pooling, element-wise operations, activation functions, other types of deep learning operations, or some combination thereof. Memory 410 may also store configuration descriptors, including configuration descriptors for configuring data storage in the DNN accelerator 402. Memory 410 may be the main memory of the DNN accelerator 402. In some embodiments, memory 410 includes one or more dynamic random access memories (DRAM).

[0076] DMA engine 420 facilitates data transfer between memory 410 and the local memory of DPU 430. For example, DMA engine 420 can read data from memory 410 and write data to the local memory of DPU 430. As another example, DMA engine 420 can read data from the local memory of DPU 430 and write data to memory 410. DMA engine 420 provides DMA functionality, allowing DPU 430 to initiate data transfers between memory 410 and the local memory of DPU 430, and to perform other operations while the data transfer is in progress. In some embodiments, DMA engine 420 can read tensors from memory 410, modify the tensors in a manner optimized for DPU 430, and then write the tensors to the local memory of DPU 430.

[0077] DPU 430 executes neural network operations in a DNN. For example, DPU 430 can execute a DNN layer by running one or more neural network operations within the DNN layer. DPU 430 can compute the output of the DNN layer based on the input of the DNN layer. In some embodiments, DPU 430 can also use the internal parameters of the DNN layer (e.g., weights) to compute the output. DPU 430 can execute one layer or a portion of a layer at a time. In some embodiments, the operations of the DNN layer can be run in parallel by multiple DPUs 430. For example, multiple DPUs 430 can each execute a portion of the deep learning workload. Data can be shared among DPUs 430. DPU 430 may also be referred to as a neural processing unit, computation block, or computation slice.

[0078] The DPU 430 can perform various types of neural network operations, such as convolution, layer normalization, SoftMax operations, pooling, element-wise operations, linear operations, and non-linear operations. The deep learning operations performed by the DPU 430 include tensor operations, i.e., operations that take tensors as input or output. In the example, the DPU 430 receives an input tensor and one or more convolutional kernels, and performs convolution using the input tensor and the kernels. The result of the convolution can be an output tensor, which can be further computed, for example, by this DPU 430 or another DPU 430.

[0079] exist Figure 4 In one embodiment, each DPU 430 includes local memory 440, IDU 450, computing units 460 (including processing engine 470 and post-processing engine 480), and ODU 490. Some or all of the components of the DPU 430 may be implemented on the same chip. In other embodiments, the DPU 430 may include alternative configurations, different or additional elements. Furthermore, the functionality implemented by the components of the DPU 430 may be performed by another component included in the DPU 430, another DPU 430, another component of the DNN accelerator 402, or other systems. The components of the DPU 430 may be implemented in hardware, software, firmware, or some combination thereof.

[0080] Local memory 440 is located locally within the corresponding DPU 430. Figure 4 In one embodiment, local memory 440 is internal to DPU 430. In other embodiments, local memory 440 may be external to DPU 430. Data in local memory 440 may be transferred to or from memory 410 (e.g., via DMA engine 420). In some embodiments, data in local memory 440 may be transferred to or from the local memory of another DPU 430. Local memory 440 may store data received, used, or generated by IDU 450, processing engine 470, post-processing engine 480, or ODU 490. Examples of data may include input activation values, weights, output activation values, sparse bitmaps, etc.

[0081] In some embodiments, local memory 440 may store tensors to be processed by processing engine 470 or post-processing engine 480. The tensors may be input tensors for deep learning operations. Local memory 440 may also store tensors generated by processing engine 470 or post-processing engine 480. These tensors may be output tensors for deep learning operations. The layout of the data points of the tensors in local memory 440 may depend on the format in which the tensors are stored. In some embodiments, local memory 440 may store tensors in various formats, including Z-major order, X-major order, and Y-major order. For Z-major order tensors, local memory 440 may store data points with the same (x, y) coordinates contiguously. For example, data points with the same (x, y) coordinates may be stored in a series of memory addresses in local memory 440. For ZXY or ZYX format tensors, local memory 440 may store data points with the same (x, y) coordinates contiguously. For example, data points with the same (x, y) coordinates may be stored in a series of memory addresses in local memory 440. For tensors in X-major order, local memory 440 can store data points with the same (y, z) coordinates contiguously. For tensors in Y-major order, local memory 440 can store data points with the same (x, z) coordinates contiguously.

[0082] In one example, the tensor is in ZXY format. Using ZXY format, data points with coordinates (0, 0, 0-C) are stored in memory first, followed by data points with coordinates (1, 0, 0-C), then data points with coordinates (2, 0, 0-C), and so on. 0-C represent all integers from 0 to C, where C is the largest coordinate index of the tensor in the Z dimension. After all coordinates in the X dimension are exhausted, the layout continues until the y coordinate is 1. In the example where the tensor is in ZYX format, data points with coordinates (0, 0, 0-C) are stored in memory first, followed by data points with coordinates (0, 1, 0-C), then data points with coordinates (0, 2, 0-C), and so on. After all coordinates in the Y dimension are exhausted, the layout continues until the x coordinate is 1.

[0083] In the example with the XYZ tensor format, data points with coordinates (0-W, 0, 0) are stored in memory first, followed by data points with coordinates (0-W, 1, 0), then data points with coordinates (0-W, 2, 0), and so on. 0-W represent all integers from 0 to W, where W represents the maximum coordinate index of the tensor in the X dimension. After all coordinates in the Y dimension are exhausted, the layout continues until the z-coordinate is 1. In the example with the XZY tensor format, data points with coordinates (0-W, 0, 0) are stored in memory first, followed by data points with coordinates (0-W, 0, 1), then data points with coordinates (0-W, 0, 2), and so on. After all coordinates in the Z dimension are exhausted, the layout continues until the y-coordinate is 1.

[0084] In the example with a YXZ tensor, data points with coordinates (0, 0-H, 0) are stored in memory first, followed by data points with coordinates (1, 0-H, 0), then data points with coordinates (2, 0-H, 0), and so on. 0-H represent all integers from 0 to H, where H represents the maximum coordinate index in the X dimension of the tensor. After all coordinates in the X dimension are exhausted, the layout continues until the z-coordinate is 1. In the example with a YZX tensor, data points with coordinates (0, 0-H, 0) are stored in memory first, followed by data points with coordinates (0, 0-H, 1), then data points with coordinates (0, 0-H, 2), and so on. After all coordinates in the Z dimension are exhausted, the layout continues until the x-coordinate is 1.

[0085] In some embodiments, local memory 440 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), etc. A dense tensor may be a tensor from which zero-value elements (if any) have not been removed. A dense tensor can be converted into a sparse tensor by removing one or more zero-value elements from it. A sparse tensor may also be called a compressed tensor or a packed tensor. The process of converting a dense tensor into a sparse tensor can be called sparsity encoding. Sparsity encoding can also generate sparse tensors. Each element in a sparse tensor may correspond to a different element in a dense tensor and indicate whether that element in the dense tensor is zero. A sparse tensor may indicate the position of an element of the sparse tensor within the dense tensor. A sparse tensor may be a sparse bitmap, where each element is a bit. A sparse tensor can be transformed into a dense tensor through a compaction process, in which one or more zeros can be added to the sparse tensor based on its sparsity.

[0086] In some embodiments, local memory 440 includes one or more SRAMs. Local memory 440 may be byte-addressable, and each memory address identifies one storage byte (eight bits). In some embodiments, local memory 440 may include a memory bank / data bank. The number of memory banks in local memory 440 may be 16, 64, 128, 256, 512, 1024, 2048, or other numbers. A memory bank may include multiple memory cells. In the example, a memory bank may include 8, 16, 64, or other numbers of memory cells. A memory bank or a memory cell within a memory bank may have a memory address. In the example, a memory cell may store a single byte, and data larger than a single byte may be stored in memory cells with contiguous memory addresses (i.e., adjacent memory cells). For example, a memory cell may store an integer in INT8 format, while storing a number in FP16 or BF16 format (with 16 bits) may require two memory cells. In some embodiments, 16 bits may be transferred from local memory 440 in a single read cycle. In other embodiments, 16 bits can be transferred from local memory 440 over multiple read cycles (e.g., two cycles).

[0087] IDU 450 loads data from local memory 440 to computation unit 460. IDU 450 can read tensors from local memory 440. Tensors may include activation tensors, weight tensors, activation sparsity tensors, weight sparsity tensors, combined sparsity tensors, etc. In some embodiments, IDU 450 may load data based on the operating mode of DPU 430. IDU 450 may select different data transmissions to computation unit 460 in different sparsity modes. For example, IDU 450 may transmit the activation sparsity tensor and weight sparsity tensor of the layer to computation unit 460 in combined sparsity mode, transmit the activation sparsity tensor to computation unit 460 but not the weight sparsity tensor in activation sparsity mode, and transmit the weight sparsity tensor to computation unit 460 but not the activation sparsity tensor in weight sparsity mode. In dense mode, IDU 450 neither transfers the activation sparsity tensor nor the weight sparsity tensor to computation unit 460.

[0088] In some embodiments, IDU 450 may process (e.g., densify) data stored in local memory 440 before providing data to computation unit 460. In an example, when operating in weighted sparse mode, IDU 450 may densify a sparse activation tensor based on the corresponding activation sparsity tensor to generate a dense activation tensor. For example, IDU 450 may add one or more zeros to the sparse activation tensor based on the activation sparsity tensor associated with the sparse activation tensor to generate a dense activation tensor. The dense activation tensor contains one or more additional elements than the sparse activation tensor. The additional elements are zero values. IDU 450 may identify one or more elements in the activation sparsity tensor corresponding to the zero-value elements, determine the position of each of the zero-value elements in the dense activation tensor, and insert the zero-value elements into the sparse activation tensor based on the determined positions. After densification, IDU 450 may transfer the dense activation tensor to computation unit 460. IDU 450 can also transfer the corresponding sparse weight tensor and weight sparsity tensor to computation unit 460. The activation sparsity tensor of the dense activation tensor may not be loaded into computation unit 460.

[0089] In another example, when operating in active sparse mode, IDU 450 can densify the sparse weight tensor to generate a dense weight tensor by inserting zeros into the sparse weight tensor based on the corresponding weight sparsity tensor. The densification of the sparse weight tensor can be similar to the densification of the sparse activation tensor described above. After densification, IDU 450 can transfer the dense weight tensor to computation unit 460. IDU 450 can also transfer the corresponding sparse activation tensor and activation sparsity tensor to computation unit 460. The weight sparsity tensor of the dense weight tensor may not be loaded into computation unit 460.

[0090] In yet another example, when operating in dense mode, IDU 450 can densify both the sparse weight tensor and the sparse activation tensor. IDU 450 can generate the input tensor and weight tensor of the layer and pass these tensors to the processing engine 470 to execute the layer without requiring sparsity acceleration.

[0091] The IDU 450 may include one or more sparse data stores for storing sparse data, such as sparse tensors. The sparse data stores can be configured to have different functionalities for different operating modes of the DPU 430. For example, when the DPU 430 operates in dense mode, there may be no sparse data to store in the sparse data stores. The sparse data stores can also be configured to store other types of data, such as activation values ​​or weights. Some aspects of the IDU 450 will be discussed below. Figure 8 Describe it.

[0092] Processing engine 470 performs operations in the DNN. Processing engine 470 can accelerate neural network operations based on the sparsity of the data. In some embodiments, processing engine 470 can operate in a dense mode without sparsity acceleration. Processing engine 470 may include one or more processing units. In some embodiments, processing units may be arranged in processing engine 470 in one or more rows and one or more columns. Each processing unit may include a physical process (PE), which may be arranged in an array including rows and columns. All PEs in processing engine 470 may form a larger array including more rows and columns.

[0093] Example PEs may be or may include one or more MAC units capable of performing MAC operations. In some embodiments (e.g., embodiments where the DPU 430 performs convolutional layers), computations in a MAC unit may be MAC operations on activation operands and weight operands. Activation operands may be activation tensors, which may include one or more activation values ​​from the input tensor of the convolution. Different activation values ​​may be in different input channels. Weight operands may be weight tensors, which may include one or more weights from the filters of the convolution. The values ​​of the weights are determined by training the DNN. The weights in the weight operands may be in different input channels.

[0094] In some embodiments, a MAC unit includes one or more multipliers for performing multiplication. A MAC unit may also include one or more accumulators (“adders”) for performing accumulation. A column of MAC units is called a MAC column. A MAC column may be associated with one or more MAC lanes. A MAC lane is a path used to load data (e.g., by IDU 450) into a MAC column. A MAC lane may also be referred to as a data transfer lane or a data loading lane. A MAC column may have multiple MAC lanes. The loading bandwidth of a MAC column is the sum of the loading bandwidths of all MAC lanes associated with that MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments, a MAC column has four MAC lanes for feeding activation values ​​or weights into the MAC column, and each MAC lane may have 16 bytes of bandwidth, resulting in a total loading bandwidth of 64 bytes for the four MAC lanes.

[0095] In some embodiments, the processing engine 470 is capable of performing depthwise convolution, standard convolution, or both. In depthwise convolution, the MAC unit can perform a MAC operation, which includes a series of multiplications of input operands and weight operands. Each multiplication in this sequence (also called a period) is a multiplication of different activation values ​​in the input operands with different weights in the weight operands. Activations and weights in the same period can correspond to the same channel. The sequence of multiplications produces a product operand that includes a series of products. The MAC operation can also include accumulation, where multiple product operands are accumulated to produce the output operand of the MAC unit. The processing engine 470 can output multiple output operands at once, each output operand generated by a different MAC unit. In standard convolution, the MAC operation can include accumulation across channels. For example, instead of generating output operands, the MAC unit can accumulate products across different channels to generate a single output point.

[0096] In some embodiments, the processing engine 470 can perform MAC operations within quantized deep learning operations, such as MAC operations in quantized convolution. In some embodiments, the MAC unit in the processing engine 470 can receive quantized activation values ​​and quantized weights, and compute a quantized MAC result. The quantized MAC result can be a quantized value in integer format and can be the output of the MAC unit. In some embodiments, the MAC unit can further include a quantization multiplier that multiplies the quantization scale by the quantized MAC result, and the output of the MAC unit can be a real value in floating-point format. The MAC unit may not include a quantization subtractor because MAC operations in quantized deep learning operations do not require zero-point offsets.

[0097] In some embodiments, processing engine 470 may include sparsity acceleration logic to facilitate sparsity acceleration. For example, each processing unit in processing engine 470 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, the sparsity module accelerates computation in processing engine 470 based on the sparsity in activation values, the sparsity in weights, or both. The sparsity module may include a storage unit for storing a sparsity tensor, which may be loaded into the storage unit by IDU 450. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0098] An activation sparsity tensor can be a sparse tensor of an activation tensor and has the same number of elements as the activation tensor. Elements in an activation sparsity tensor can indicate whether the corresponding element in the activation tensor is zero. For example, a zero value in an activation sparsity tensor indicates that the corresponding element in the activation tensor is zero. A one value in an activation sparsity tensor indicates that the corresponding element in the activation tensor is non-zero. A weight sparsity tensor can be a sparse tensor of a weight tensor and has the same number of elements as the weight tensor. Elements in a weight sparsity tensor can indicate whether the corresponding element in the weight tensor is zero. For example, a zero value in a weight sparsity tensor indicates that the corresponding element in the weight tensor is zero. A one value in a weight sparsity tensor indicates that the corresponding element in the weight tensor is non-zero. The sparsity module can use activation sparsity tensors and weight sparsity tensors to generate combined sparsity tensors. For example, the sparsity module can multiply the elements of the activation sparsity tensor by the corresponding elements of the weight sparsity tensor to calculate the elements of the combined sparsity tensor. The positions of these three elements can be matched within their respective sparse tensors. In some embodiments, each element in a sparse tensor can be a bit, and the sparse tensor can be called a sparse bitmap.

[0099] The sparsity module can use a sparsity tensor to identify the activation values ​​and weights to be used by the MAC unit for MAC operations. In an embodiment where the processing engine 470 operates in a combined sparse mode, the sparsity module can identify the activation values ​​and weights corresponding to the non-zero value elements of the combined sparse tensor. In an embodiment where the processing engine 470 operates in an activation sparse mode, the sparsity module can identify the activation values ​​and weights corresponding to the non-zero value elements of the activation sparse tensor. In an embodiment where the processing engine 470 operates in a weighted sparse mode, the sparsity module can identify the activation values ​​and weights corresponding to the non-zero value elements of the weighted sparse tensor. In dense mode, the sparsity module can be bypassed because no sparsity acceleration is performed.

[0100] Post-processing engine 480 processes the output of processing engine 470. Post-processing engine 480 may include one or more post-processing elements. In some embodiments, the post-processing elements in post-processing engine 480 may be arranged in an array having rows and columns. In some embodiments, post-processing engine 480 computes an activation function. Post-processing engine 480 may receive the output of processing engine 470 as input to the activation function. In addition to or as an alternative to the activation function, post-processing engine 480 may perform other types of post-processing on the output of processing engine 470. For example, post-processing engine 480 may apply a bias to the output of processing engine 470. In some embodiments, post-processing engine 480 may be bypassed for certain DNN layers.

[0101] ODU 490 evicts data from computing unit 460 and writes data to local memory 440. In some embodiments, ODU 490 can evict data at the unit level. For each processing unit, ODU 490 can evict the output of a PE in the processing unit based on the row index or column index of each PE. For example, ODU 490 can evict data from the processing unit using a periodic sequence. ODU 490 can evict the output of some PEs in each cycle. This periodic sequence can be configured based on a configuration descriptor indicating the operating mode of IDU 450. In some embodiments, ODU 490 can perform parallel reads or parallel writes. For example, ODU 490 can write data to multiple memory banks in parallel, or read data from multiple memory banks in parallel.

[0102] ODU 490 can process the effluent data before writing it to local memory 440. In some embodiments, ODU 490 can compress the data, for example, based on the sparsity of the data. ODU 490 can also generate sparse data indicating the sparsity of the data. ODU 490 may include sparsity encoding logic that can convert data from a dense format to a sparse format. For example, ODU 490 can be implemented with one or more sparsity encoders. The sparsity encoder converts dense data into compressed data based on the sparsity of the dense data. The data received by ODU 490 from computing unit 460 may be the output tensor of a DNN layer (e.g., Figure 2 The output tensor (230) is at least a portion of the output tensor. The sparse encoder can remove one or more zeros from the output tensor to convert it into a compressed activation tensor (“sparse activation tensor”). The sparse encoder can also generate at least one sparse tensor indicating the sparsity of the output tensor. This sparse tensor can also be called an activation sparse tensor. The activation sparse tensor can correspond to at least a portion of the output tensor of the DNN layer (e.g., ...). Figure 2 (Vector 235 in the output tensor). The activation sparsity tensor may include sparsity elements (e.g., bits), each element corresponding to a different activation value in the output tensor and indicating whether the corresponding activation value is zero. The ODU 490 may have a data storage unit for storing sparse data.

[0103] In addition to compression or as an alternative, ODU 490 can perform tensor permutation to change the storage format of tensors. For example, ODU 490 can permutate tensors before writing them to local memory 440, allowing ODU 490 to write the tensors to local memory 440 in a new format. In some embodiments, ODU 490 can perform tensor permutation to change a Z-major order format to an X-major order format or a Y-major order format. For example, tensors ejected by ODU 490 from processing engine 470 or post-processing engine 480 may be in Z-major order format. ODU 490 can change the Z-major order format to an X-major order or Y-major order format and write the tensors to local memory 440 in X-major order or Y-major order format. In some embodiments, ODU 490 includes intermediate storage to facilitate tensor permutation. For example, ODU 490 may include one or more storage banks. The storage banks may have a matrix structure with rows and columns. ODU 490 may write data to the storage banks in a different manner than reading data from the storage banks for tensor permutation. For example, the ODU 490 can write tensors to memory column-wise but read them from memory row-wise. As another example, the ODU 490 can write tensors to memory row-wise but read them from memory column-wise. The ODU 490 can read another tensor while writing a tensor. For example, the ODU 490 can read data points of a tensor from a row of memory and write data points of another tensor to that row of memory in the same cycle.

[0104] The ODU 490 may include one or more write modules that write data to local memory 440. Example write modules may include write storage, such as a write combination buffer (WCB), which can combine smaller write transactions into larger write transactions. Data storage in the ODU 490, such as sparse storage, permutation storage, or write storage, can be configured to perform different functions for different operating modes of the DPU 430. Certain aspects of the ODU 490 will be discussed below. Figure 9-14 Describe it.

[0105] Figure 5 Example sparse unit 500 according to various embodiments is shown. Sparse unit 500 may be a processing engine (e.g., Figure 4The sparse unit 500 is a processing unit within the processing engine 470. The sparse unit 500 includes 16 MAC units 510 (referred to individually as "MAC units 510") forming a MAC array with four rows and four columns. The MAC array has a 5x4 spatial shape, meaning the height of the MAC array is 4 and the width of the MAC array is 5. The sparse unit 500 also includes 16 weight register files 520 (referred to individually as "weight register files 520"), 16 activation register files 530 (referred to individually as "activation register files 530"), four row buffers 540 (referred to individually as "row buffers 540"), and a sparsity module 560 (referred to individually as "sparse module 560"). In other embodiments, the sparse unit 500 may include fewer, more, or different components. For example, the sparse unit 500 may include other numbers of MAC units 510, weight register files 520, activation register files 530, row buffers 540, or sparsity modules 560. As another example, sparse cell 500 may include column buffers in place of or as a supplement to row buffer 540. Furthermore, the shape (e.g., height or width) of the MAC array may vary.

[0106] MAC unit 510 is configured to perform MAC operations. Each MAC unit 510 may include one or more multipliers and one or more adders. A multiplier may multiply an activation value by a weight at a time to compute a product. In some embodiments (e.g., embodiments where MAC unit 510 includes multiple multipliers), these multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. Adders may accumulate the products computed by the multipliers. Although... Figure 5 Not shown, but the sparse unit may include an adder tree comprising multiple adder levels. A first level may receive the outputs of multiple MAC units 510. The number of adders in the first level may be half the number of MAC units 510, and each adder may sum the outputs of two MAC units 510. A second level may receive the outputs of the adders from the first level. The number of adders in the second level may be half the number of adders in the first level, and each adder in the second level may sum the outputs of two adders from the first level. The adder tree may include one or more other levels. The final level may include a single adder that sums the outputs of the adders from the penultimate level to compute a partial sum for the sparse unit 500.

[0107] Weight register file 520 stores the weights to be processed in MAC operations. Figure 5In one embodiment, four weight register files 520 are grouped into a storage set that stores data to be used by a column of MAC units 510. There are four storage sets corresponding to four columns of MAC units 510. In some embodiments, the weight register files 520 may correspond to MAC units 510 and store data to be processed by those MAC units. In some embodiments, all 16 weight register files 520 constitute a weight storage unit.

[0108] Activation register file 530 stores the activation values ​​to be processed during MAC operations. Figure 5 In this embodiment, four activation register files 530 are grouped into storage sets that store data to be used by a row MAC unit 510. There are four storage sets corresponding to four rows of MAC units 510. In some embodiments, the activation register files 530 may correspond to MAC units 510 and store data to be processed by those MAC units. In some embodiments, all 16 activation register files 530 constitute an activation storage unit. Row buffers 540 store the output of MAC units 510. Each row buffer 540 may discharge the output of one row of MAC units 510.

[0109] The sparsity module 560 facilitates dynamic, sparsity-based acceleration within the sparse unit 500. Figure 5 In the embodiments, each sparsity module 560 includes a sparsity tensor storage unit 565 and control logic 567. The sparsity tensor storage unit 565 stores combined sparse tensors. The combined sparse tensors stored in the sparse tensor storage unit 565 can correspond to activation tensors and weight tensors. Non-zero elements in the combined sparse tensors can correspond to non-zero activation-weight pairs, including non-zero activation values ​​and non-zero weights. The position of a non-zero activation value in the activation tensor can match the position of a non-zero weight in the weight tensor. The product of a non-zero activation value and a non-zero weight will be non-zero.

[0110] Control logic 567 can control the transfer of activation values ​​and weights stored in weight register file 520 and activation register file 530 to MAC unit 510 based on a sparsity tensor. For example, control logic 567 can select a subset of weights stored in weight register file 520 and a subset of activation values ​​stored in activation register file 530 based on a combined sparsity tensor. The selected weights and activation values ​​constitute non-zero activation value-weight pairs. Control logic 567 can transfer the selected weights and activation values ​​to MAC unit 510 to perform a MAC operation. Other weights stored in weight register file 520 and other activation values ​​stored in activation register file 530 are skipped from computation. Figure 5In this embodiment, each sparsity module 560 controls the sparsity acceleration in the corresponding MAC unit 510. Since the sparsity acceleration is based on both weight sparsity and activation sparsity, 16 sparsity modules 560 are used for accelerated computation in 16 MAC units 510.

[0111] like Figure 5 As shown, sparse unit 500 is associated with multiplexers (MUX) 503, 504, 505, and 506. In other embodiments, sparse unit 500 may be associated with other numbers of MUXs or other devices. MUX 503 facilitates loading weights (e.g., from local memory 440) into weight register file 520. MUX 504 facilitates loading activation values ​​(e.g., from local memory 440) into activation register file 530. MUX 505 facilitates loading sparsity tensors into sparse tensor storage unit 565. MUX 506 may be an exhaust MUX that facilitates exhausting the output of MAC unit 510 (e.g., to local memory 440).

[0112] In some embodiments, the sparse unit 500 can also perform matrix multiplication derived from the Fourier transform operation. For the example Fourier transform operation, the MAC unit 510 can perform the MAC operation on two matrix multiplication sequences derived from the Fourier transform operation. The weight register file 520 can be used to store data points in the transform tensor of the Fourier transform operation. The activation register file 530 can be used to store data points in the input tensor of the Fourier transform operation. The row buffer 540 can store data points in the output tensor of the Fourier transform operation.

[0113] Figure 6 A sparse cell array 600 according to various embodiments is illustrated. The sparse cell array 600 may be... Figure 4 An example of the processing engine 470. Figure 6 In this embodiment, the sparse cell array 600 includes sparse cells 610 (referred to individually as "sparse cells 610") arranged in four columns and four rows, an activation memory 620, and a weight memory 630. In other embodiments, the sparse cell array 600 may include fewer, more, or different components. For example, the sparse cell array 600 may include other numbers of columns, rows, or sparse cells 610.

[0114] Each sparse unit 610 can perform sparsity-accelerated MAC operations. The sparse unit 610 can facilitate dynamic sparse patterns. For example, the sparse pattern of the sparse unit 610 can dynamically change between combined sparse patterns, active sparse patterns, weighted sparse patterns, and dense patterns. Embodiments of the sparse unit 610 may include... Figure 5The sparse unit array 600 comprises sparse units 500. An activation memory 620 stores activation values, such as activation values ​​in the input tensor of a deep learning operation. Activation values ​​can be loaded from the activation memory 620 into the sparse unit 610. A weight memory 630 stores weights, such as weights in a filter of a deep learning operation. Weights can be loaded from the weight memory 630 into the sparse unit 610. The activation memory 620 or the weight memory 630 may be a buffer. In other embodiments, the sparse unit array 600 may include a dense data memory and a sparse data memory instead of the activation memory 620 and the weight memory 630. The dense data memory may store dense tensors, such as dense tensors generated by the loading module 360. The sparse data memory may store sparse tensors.

[0115] The sparse cell array 600 can also perform matrix multiplication in the Fourier transform operation. The activation memory 620 can be used to store the input tensors of the Fourier transform operation. The weight memory 630 can be used to store the transform matrix of the Fourier transform operation.

[0116] Figure 7 Example PE 700 according to various embodiments is shown. PE 700 may be a unit component of a processing unit (e.g., a processing unit in processing engine 370). Figure 7 In one embodiment, PE 700 includes a MAC unit 705, an activation register file 710, a weight register file 720, an output register file 750, and a sparsity accelerator 760. MAC unit 705 includes a multiplier 730 and an adder 740. In other embodiments, PE 700 may include fewer, more, or different components.

[0117] Activation register file 710 stores activation operands, which can be context. Activation register file 710 can be... Figure 5 Example of activation register file 530. Weight register file 720 stores weight operands. Weight register file 720 can be... Figure 5An example of a weight register file 520. Activation operands and weight operands can be loaded from memory (e.g., memory 340) into activation register file 710 and weight register file 720, respectively. The sparsity accelerator 760 receives a sparse bitmap 715 corresponding to the sparse tensors in weight register file 720. When MAC unit 705 operates in combined sparse mode, sparse bitmap 715 can be a combined sparse bitmap. When MAC unit 705 operates in activation sparse mode, sparse bitmap 715 can be an activation sparse bitmap. When MAC unit 705 operates in weighted sparse mode, sparse bitmap 715 can be a weighted sparse bitmap. Sparse bitmap 715 can have the same size (e.g., the same number of elements) or larger than the activation operands or weight operands.

[0118] Using a sparse bitmap 715, the sparse accelerator 760 selects four activation values ​​from the activation register file 710 and four weights from the weight register file 720. The sparse accelerator 760 transmits the selected activation values ​​and weights to the multiplier 730. These selected data elements correspond to non-zero value elements in the sparse bitmap 715. The four selected activation values ​​and four selected weights can form four activation-weight pairs. The multiplier 730 can compute a product based on each activation-weight pair, thus computing a total of four products. These four products can be provided to the adder 740. Although... Figure 7 A multiplier 730 is shown, but the MAC unit 705 may include multiple multipliers that can perform multiple multiplication operations simultaneously.

[0119] Adder 740 accumulates the four products and computes the cell-level internal partial sum. The four unselected elements of the dense tensor are not processed to save power and time, and this does not affect the value of the cell-level internal partial sum. For example, when the dense tensor is a dense activation tensor, the weights corresponding to the unselected activation values ​​are zero, so the product of the unselected activation values ​​and the weights will be zero, and will not contribute to the cell-level internal partial sum or other partial sums computed by the sparse unit. Similarly, when the dense tensor is a dense weight tensor, the activation values ​​corresponding to the unselected weights are zero, so the product of the unselected weights and the activation values ​​will be zero, and will not contribute to the cell-level internal partial sum or other partial sums computed by the sparse unit. In other embodiments, MAC unit 705 can operate in a dense mode, in which the sparsity bitmap 715 is not used and the sparsity accelerator 760 is inactive. MAC unit 705 can process all activation values ​​in the activation operand and all weights in the weight operand.

[0120] The cell-level internal partial sums can be stored in the output register file 750. In some embodiments, the cell-level internal partial sums can be used multiple times. For example, the activation operand can represent N data blocks in the input tensor of the convolution, where N is an integer greater than 1. Instead of processing all N data blocks to compute N cell-level internal partial sums, the cell-level internal partial sums are computed once and used N times as N cell-level internal partial sums in the convolutional layer.

[0121] In some embodiments, PE 700 receives one or more internal portions of PE level from one or more other PEs. Adder 740 or accumulator ( Figure 7 (Not shown in the image) The sum of one or more PE-level internal portions can be accumulated with the sum of the PE-level internal portions of PE 700, and the accumulated result (i.e., the multi-PE internal portion sum) is stored in the output register file 750. One or more other PEs may be located in the same column as PE 700 in sparse cells. The multi-cell internal portion sum may be a column-level internal portion sum. In some embodiments, the PE-level internal portion sum or the multi-cell internal portion sum of PE 700 may be sent to one or more other PEs for further accumulation.

[0122] Figure 8 An example IDU 800 according to various embodiments is shown. IDU 800 may be a data transmission unit within a DPU. An IDU may be... Figure 4 Example of IDU 450. IDU 800 is communicatively coupled to memory port 801 and compute unit 802. Memory port 801 provides an interface through which memory can be accessed for data reading or writing. In some embodiments, memory port 801 may be a common memory port (CMX), which may be shared by IDU 800 with ODU in DPU or other components of DPU. IDU 800 can load data from memory to compute unit 802 through memory port 801. In some embodiments, memory port 801 may be a component of memory. Memory may be... Figure 4 An example of local memory 440. Computation unit 802 may be... Figure 4 Example of a computing unit 460. (e.g., ...) Figure 8 As shown, IDU 800 includes an arbitrator 810, a reader 820, a storage unit 830, a response FIFO 840, a response data FIFO 845, a command FIFO 850, a configurable storage unit 860, a MUX 870, control logic 880, and a data loader 890. In other embodiments, IDU 800 may include fewer, more, or different components. For example, IDU 800 may include multiple response data FIFOs.

[0123] Arbitrator 810 can arbitrate multiple data read requests (“read requests”). A read request is a request to read data (e.g., activation values, weights, sparse bitmaps, etc.) stored in memory. A read request may include information indicating the memory address where the data is stored. Memory port 801 can be shared among multiple data requests, which may lead to memory bank conflicts; for example, multiple data requests may request data stored in the same memory bank. Such conflicts can be resolved by arbitrator 810. Arbitrator 810 can schedule these read requests, for example, by determining the order in which read requests should be processed or transmitted to memory port 801. Reader 820 issues a read request.

[0124] Storage unit 830, response FIFO 840, response data FIFO 845, command FIFO 850, and configurable storage unit 860 can constitute the internal storage of IDU 800. Internal storage may need to meet DNN performance targets. For example, without internal storage, IDU 800 may not be able to keep the pipeline to memory busy or absorb stalls, resulting in underutilization of available bandwidth. In some embodiments, IDU 800 may use separate ports for activation values, weights, and sparsity bitmaps. This may cause activation values, weights, and the sparsity bitmap to arrive at different times. Computation unit 802 will need activation values ​​aligned with weights to perform computations. In sparse mode, computation unit 802 will need activation values, weights, and the sparsity bitmap aligned. With internal storage, IDU 800 can store received data in its internal storage while waiting for other data, and send all data to computation unit 802 after all data has been received. In some embodiments, while the computing unit 802 is performing calculations on previously loaded data, the internal storage of the IDU 800 stores data for the next calculation to be performed by the computing unit 802.

[0125] Storage unit 830 can store read requests. In some embodiments, storage unit 830 is RAM. When arbitrator 810 determines that a read request can be processed, reader 820 can obtain the read request from storage unit 830 and transmit the read request to arbitrator 810. Command FIFO 850 can store commands from reader 820, such as control signals, metadata, etc. Command FIFO 850 may include sparse storage that can store commands for sparse data.

[0126] The response FIFO 840 and response data FIFO 845 receive responses to read requests from the arbitrator 810. A response to a read request may include one or more response signals indicating that the corresponding read request is valid and a valid response has been issued. The response signals may be stored in the response FIFO 840. The response may also include data read from memory, and this data may be stored in the response data FIFO 845. In some embodiments, the response data FIFO 845 stores activation values ​​and weights.

[0127] The configurable storage unit 860 can be configured to perform different functions for different operating modes of the DPU. The configurable storage unit 860 can be configured by a configuration descriptor, which can be configured by a DNN module (e.g., Figure 4 The data is generated by the DNN module 401 in the memory. In some embodiments (e.g., embodiments where the DPU operates in sparse mode), the data received by the IDU 800 from the memory port 801 may be sparse data. The configurable storage unit 860 is configured to store sparse data. The sparse data may include one or more sparse tensors, such as a sparse bitmap. In some embodiments (e.g., embodiments where the DPU operates in dense mode), the configurable storage unit 860 may have other functions. The IDU 800 may not request any sparse data from memory. Instead of storing sparse data or remaining empty, the configurable storage unit 860 may be used as an additional response data FIFO to expand the response data FIFO 845, such as... Figure 8 As shown by the dashed lines, IDU 800 can store more response data at once. For example, configurable storage unit 860 can increase the FIFO depth used to store response data, for example, from depth 20 to depth 52. In some embodiments, the response data FIFO depth can also be further expanded by sparse storage in command FIFO 850, for example, from depth 52 to depth 68. The increase in depth helps absorb additional pauses. In dense mode, configurable storage unit 860 may not receive sparse data. In some embodiments, configurable storage unit 860 is RAM.

[0128] MUX 870 receives data from response data FIFO 845 and data from configurable storage unit 860 as input signals, and selects one of the input signals as the output of MUX 870. Control logic 880 receives commands stored in command FIFO 850 and uses these commands to control data transfer to computing unit 802. Data loader 890 receives output data from MUX 870, which may be data stored in response data FIFO 845 or data stored in configurable storage unit 860. Data loader 890 may perform one or more operations on the data. Operations may include, for example, generating data tables, data transformation, data unpacking, or other types of operations. Data loader 890 sends the data to computing unit 802.

[0129] Figure 9 An example ODU 900 according to various embodiments is shown. The ODU 900 can be a data transfer unit in a DPU. The ODU 900 facilitates the transfer of data computed by the computing unit from the computing unit to memory. The ODU 900 can transform data by performing data compression, tensor permutation, etc. The ODU 900 can be... Figure 4 An example of ODU 490. (e.g.) Figure 9 As shown, ODU 900 includes a temporary buffer 910, a transposeable register file (TRF) 920 (referred to separately as "TRF 920"), a compression module 930, a data shifter 940, a data memory 945, a data buffer 947, a sparsity shifter 950, a sparsity memory 955, a sparsity buffer 957, a WCB 960, a data memory 965, a CMX data buffer 967, another WCB 970, another sparsity memory 975, and a CMX sparsity buffer 977. In other embodiments, ODU 900 may include fewer, more, or different components.

[0130] Temporary buffer 910 temporarily stores data received from the computation unit. For example, temporary buffer 910 can store data while TRF 920 is performing a permutation on previously received data. In the example, the data could be the output activation values ​​of a neural network layer, which can be used as input activation values ​​for the next neural network layer. Data can be read from temporary buffer 910 and written to TRF 920. TRF 920 is a store that facilitates tensor permutations. For example, TRF 920 can transpose the X and Z coordinates and write the data in X-major order, even though it receives the data in Z-major order. As another example, TRF 920 can transpose the Y and Z coordinates and write the data in Y-major order, even though it receives the data in Z-major order.

[0131] In some embodiments, each TRF 920 may include storage elements arranged in an array having rows and columns. Each storage element may store a single entry at a time. An entry may be a data point in a tensor. The TRF 920 may alternate the direction of writing and reading data points of the tensor to change the storage format of the tensor. A tensor may be a matrix having rows and columns. The number of data points in each row of a tensor may not be greater than the number of storage elements in a row or column of a TRF 920. Furthermore, the number of data points in each column of a tensor may not be greater than the number of storage elements in a row or column of a TRF 920. In one example, data may be written to the TRF 920 column-wise but read from the TRF 920 row-wise to permutate the tensor. In another example, data may be written to the TRF 920 row-wise but read from the TRF 920 column-wise to permutate the tensor.

[0132] Compression module 930 can compress data read from TRF 920. For example, compression module 930 compresses data based on sparsity. In some embodiments, compression module 930 can compress data by removing data elements whose values ​​are not greater than a threshold. The threshold may be, for example, zero. Compression module 930 may include one or more comparators, each of which can compare the value of a data element with the threshold. After determining that a data element is not greater than the threshold, compression module 930 can remove the data element. Compression module 930 can also generate sparse data indicating the sparsity of data received from TRF 920. For example, compression module 930 can generate a sparse tensor. The sparse tensor may include a sequence of sparse data elements, each sparse data element corresponding to a data element received from TRF 920 and indicating whether the data element is zero. The sparse tensor may correspond to an activation tensor or a weight tensor, and the position of the sparse data element in the sparse tensor may match the position of the corresponding data element in the activation tensor or weight tensor. The example sparse tensor is a sparse bitmap, where sparse bit elements are single bits. A zero bit indicates that the corresponding data element is zero, while a single bit indicates that the corresponding data element is not zero.

[0133] Compressed data (e.g., compressed activation tensors) is transferred to data shifter 940. Data shifter 940 can shift data elements received from compression module 930. In some embodiments, the data elements all have non-zero values. The compressed data can be stored in data memory 945 before or after the shift. The compressed and shifted data is transferred to data buffer 947. Sparse data is transferred to sparse shifter 950. Sparse shifter 950 can shift sparse data elements received from compression module 930. The sparse data can be stored in sparse memory 955 before or after the shift. The compressed and shifted sparse data is transferred to sparse buffer 957.

[0134] WCB 960 can combine smaller write transactions used to write data stored in data buffer 947 to form a larger write transaction. In the example, WCB 960 can combine two 16-byte transactions into a 32-byte transaction. The smaller or larger write transactions can be stored in data memory 965. The larger write transaction generated by WCB 960 is stored in CMX data buffer 967, from which the write transaction is sent to memory. WCB 970 can combine smaller write transactions used to write sparse data stored in sparsity buffer 957 to form a larger write transaction. The smaller or larger write transactions can be stored in sparse memory 975. The larger write transaction generated by WCB 970 is stored in CMX sparse buffer 977, from which the write transaction is sent to memory.

[0135] In some embodiments, one or more TRF 920, sparse memory 955, sparse memory 975, WCB 960, or WCB 970 may be configurable such that they can be configured to have different functions in different operating modes of the DPU. Certain aspects of configurable storage in the ODU will be discussed below. Figure 10-13 Describe it.

[0136] Figure 10 This illustrates that the sparse storage in the ODU 900, according to various embodiments, can be reconfigured for temporary storage. Figure 10 In the embodiments, sparse memory 955 and sparse memory 975 are configurable. For example, when a computing unit (or another computing unit) is to operate in sparse mode to execute the next neural network layer, sparse memory 955 and sparse memory 975 are configured to store sparse data. When a computing unit (or another computing unit) is to operate in dense mode to execute the next neural network layer, sparse memory 955 and sparse memory 975 are configured to function as a temporary buffer 910, such as... Figure 10 As shown by the dashed lines in the diagram. Sparse memory 955 and sparse memory 975 can also store data received from the computing unit and expand the storage capacity of temporary buffer 910.

[0137] Sparse memory 955 and sparse memory 975 can be configured by one or more configuration descriptors generated for the DNN. The configuration descriptors can be generated offline by the DNN module 401, for example, before DNN execution begins. Although in Figure 10Both sparse memory 955 and sparse memory 975 are configurable, but in other embodiments, sparse memory 955 or sparse memory 975 may not be configurable.

[0138] Figures 11A to 11C This illustrates that the sparse storage in the ODU 900, according to various embodiments, can be reconfigured for reordering. Figures 11A to 11C In the embodiments, sparse memory 955 and sparse memory 975 are configurable. For example, when a computing unit (or another computing unit) is to operate in sparse mode to execute the next neural network layer, sparse memory 955 and sparse memory 975 are configured to store sparse data. When a computing unit (or another computing unit) is to operate in dense mode to execute the next neural network layer, sparse memory 955 and sparse memory 975 are configured to serve as a reordering buffer 1100, such as... Figure 11A The dashed lines in the diagram illustrate this. In the example, the dense mode is a dense X-major order mode, where the next neural network layer will be executed with input activation values ​​in X-major order format, and the next neural network layer will be executed without sparsity acceleration.

[0139] The reordering buffer 1100 is located between the temporary buffer 910 and the TRF 920 on the data path in ODU 900. Sparse memories 955 and 975 can change the order of stored or transmitted data elements (e.g., output activation values) to facilitate tensor permutations in the TRF 920. In the example, data elements can be stored in the temporary buffer in a first order, which can be the same order in which data elements are output from the computation unit. Data elements can be written to the reordering buffer 1100 in the first order but read from it in a second order. The second order differs from the first order. For example, the first order can follow a Z-major order format, while the second order can follow an X-major order format.

[0140] Figure 11B An example of read and write transactions in reorder buffer 1100 and TRF 920 is shown for a first order following the Z-major order format and a second order following the X-major order format. Figure 11B The "W" in the text represents writing. Figure 11B In this context, "R" stands for read. Figure 11B Each column in the table represents four cycles. Each cycle can be used to transmit a fixed number of bits, such as 8 bits, or 1 byte. The ODU 900 can receive 4 X bytes (by Z=0:15, i.e., 16 consecutive Z bytes) in Z=0:15 (i.e., 16 consecutive Z bytes). Figure 11B The cells with dotted patterns are represented in the middle), followed by 4 X bytes in Z=16:31 (i.e., 16 consecutive Z bytes). Figure 11BThe cells with horizontal stripe patterns are represented by the following 4 X bytes in Z=32:47 (i.e., 16 consecutive Z bytes). Figure 11B The cells are represented by a diamond grid pattern), followed by 4 X bytes in Z=48:63 (i.e., 16 consecutive Z bytes). Figure 11B (Represented by cells with diagonal stripe patterns). These 16 X bytes are... Figure 11B The four different pattern representations are: dot pattern, horizontal stripe pattern, diamond grid pattern, and diagonal stripe pattern. These bytes are written to the temporary buffer 910 in the same order, and then to the reorder buffer 1100. Figure 11B Four such groups of 16 bytes are shown.

[0141] Bytes are read from the reordering buffer 1100 in a different order. The reordering buffer 1100 can collect 16 consecutive X bytes from each Z set and present them to the TRF 920. (As...) Figure 11B As shown, 16 consecutive X bytes are read from the reordering buffer 1100 and written to the TRF 920. With the reordering capabilities of the sparse memory 955 and sparse memory 975, fewer TRFs may be required. Figure 11C As shown, for 64 consecutive Z bytes, four TRFs 1110A-1110D would be needed because each group of four X bytes in 16 consecutive Z bytes would require one TRF. With the help of the reordering buffer 1100, one TRF is sufficient because 16 consecutive X bytes can be read from the reordering buffer and written to a single TRF.

[0142] Sparse memory 955 and sparse memory 975 can be configured by one or more configuration descriptors generated for the DNN. The configuration descriptors can be generated offline by the DNN module 401, for example, before DNN execution begins. Although in Figures 11A to 11C In one embodiment, both sparse memory 955 and sparse memory 975 are reconfigured as reordering buffers, but in other embodiments, one of sparse memory 955 and sparse memory 975 may not be reconfigured.

[0143] Figure 12 This illustrates that the sparse storage in the ODU 900, according to various embodiments, can be reconfigured for reordering and transposition. Figure 12In the embodiments, sparse memory 955 and sparse memory 975 are configurable. For example, when a computing unit (or another computing unit) is to operate in sparse mode to execute the next neural network layer, sparse memory 955 and sparse memory 975 are configured to store sparse data. When a computing unit (or another computing unit) is to operate in dense mode to execute the next neural network layer, sparse memory 955 and sparse memory 975 are configured to act as buffer 1200, such as... Figure 12 As shown by the dashed line in the image.

[0144] Buffer 1200 can reorder and transpose data. In the example, the dense pattern is a dense Y-major order pattern; for example, the next neural network layer will be executed with input activation values ​​in Y-major order format, and the next neural network layer will be executed without sparsity acceleration. The ODU 900 may need to perform Y permutation on the data received from the computation unit. Y permutation can be done using buffer 1200 and TRF 920 in the ODU 900. For Y permutation, consecutive bytes in Y need to be collected while the computation unit sends consecutive bytes in Z. Buffer 1200 can hold incoming 1x4x16 tensors until there are consecutive bytes in 16 Y dimensions. For 16x16x16 tensors, the entire tensor needs to be stored before any data can be written. This is likely because Y is not the fastest changing dimension in ZXY / ZYX order (i.e., Z-major order). The entire tensor can be stored in buffer 1200. By reading the tensor and transposing it using TRF 920, ODU 900 can achieve near-Z-major order performance in Y-major order permutations.

[0145] Figure 13 The permutation storage in the ODU 900, according to various embodiments, is shown to be reconfigurable for temporary storage. Figure 13 In the embodiments described, TRF 920 is configurable. For example, TRF 920 is configured as a permutation tensor when a computation unit (or another computation unit) is to operate in X-major or Y-major mode to execute the next neural network layer. When a computation unit (or another computation unit) is to operate in Z-major mode to execute the next neural network layer, TRF 920 is configured as an expanded temporary buffer 910, forming an expanded temporary buffer 1300 with the temporary buffer 910. TRF 920 can be configured by one or more configuration descriptors generated for the DNN. The configuration descriptors can be generated offline by the DNN module 401, for example, before the DNN execution begins. Although in Figure 13 All TRF 920s are configurable, but in other embodiments, one or more of the TRF 920s may not be configurable.

[0146] Figure 14 The write storage in the ODU 900, according to various embodiments, can be reconfigured as a transactional FIFO 1400. Figure 13 In the embodiments described, WCB 960 and WCB 970 are configurable. For example, WCB 960 and WCB 970 are configured to buffer write transactions before they are sent to memory. As described above, WCB 960 and WCB 970 can combine smaller write transactions into larger ones. In some embodiments, combining smaller transactions may not offer much advantage. In these cases, WCB 960 and WCB 970 are reused as transaction FIFO 1400. Transaction FIFO 1400 can buffer transactions before they are sent to a memory port (e.g., CMX). This can be advantageous because CMX may sometimes be unprepared to accept transactions, for example, due to other accesses from the read side. To prevent these pauses from propagating throughout the DPU pipeline and degrading overall performance, transaction FIFO 1400 can absorb some or all of the impact of the pauses.

[0147] The WCB 960 and WCB 970 can be configured by one or more configuration descriptors generated for the DNN. These configuration descriptors can be generated offline by the DNN module 401, for example, before DNN execution begins. Although in Figure 13 Both WCB 960 and WCB 970 are configurable, but in other embodiments, WCB 960 or WCB 970 may not be configurable.

[0148] Figures 15A to 15L The process of permuting tensors in a transit storage unit is illustrated according to various embodiments. The tensor permutation process can be performed by an ODU in a DNN accelerator, for example... Figure 4 ODU 490 or Figure 9 The ODU 900 is used in this context. The intermediate storage unit can be a TRF. A tensor permutation process can be used to change the Z-major order format to an X-major order format. For illustrative purposes, the tensor has a length of 8 in the X-dimension and a length of 8 in the Z-dimension. In other embodiments, the tensor may have other shapes or sizes.

[0149] Figures 15A to 15D The process of writing tensors into intermediate storage units is shown. Figure 15A A transit storage unit is shown before any data is written to it. The transit storage unit has a matrix or grid structure comprising 64 storage elements arranged in an 8x8 grid. Each storage element in the transit storage unit is represented by a square. In some embodiments, each storage element may include a fixed number of triggers for storing a fixed number of bits. The fixed number may be 4, 8, 16, 32, etc. Examples of storage elements include... Figure 14 The trigger group in the memory. In some embodiments, each storage element can store one entry at a time. An entry may include a single data point. The intermediate storage unit can be used to store data of various data types, such as INT8, FP16, BF16, FP32, etc. The intermediate storage unit can be... Figure 9 Example of TRF 920. In some embodiments, the intermediate storage unit is a component of the discharge module that performs tensor permutations.

[0150] exist Figure 15B In the process, a vector of 8 data points is written to a column of the intermediate storage unit. These 8 data points are... Figure 15B The vector is represented by a dotted square. This vector can be in the Z dimension of a tensor. For example, the eight data points can have the same (x, y) coordinates but different z coordinates. In some embodiments, these eight data points can be computed by performing a neural network operation or a portion of a neural network operation. The vector is written to the rightmost column of the intermediate memory unit. After writing the vector, one column of the intermediate memory unit stores data, and seven columns are empty. These eight data points can be written within one clock cycle of the intermediate memory unit. This clock cycle can be the first clock cycle of a first clock cycle sequence.

[0151] exist Figure 15C In the next step, another vector of 8 data points is written to another column of the intermediate memory unit. This vector is written to the second rightmost column of the intermediate memory unit. After writing this vector, two columns of the intermediate memory unit store data, and six columns are empty. These 8 data points can be written in the second clock cycle of the first clock cycle sequence.

[0152] Despite Figure 15C Not shown in the diagram, but six additional vectors (each with 8 data points) are written to the other columns of the intermediate memory cell until the intermediate memory cell is full. Each additional vector can be written in a subsequent clock cycle of the clock cycle sequence. Figure 15D As shown, all columns of the intermediate storage unit store data. The writing of 64 data points is performed column-wise and takes 8 clock cycles of the first clock cycle sequence. Each column of the intermediate storage unit stores a vector in the Z dimension, and each row of the intermediate storage unit stores a vector in the X dimension.

[0153] exist Figure 15EIn the first clock cycle of the second clock cycle sequence, data stored in the top row of the intermediate memory unit (i.e., a vector in the X dimension) is read from the intermediate memory unit. This vector can be written to memory, such as local memory 440, by the ejection module. In the same clock cycle, a new vector of 8 data points (represented by squares with a diamond grid) is written to the top row of the intermediate memory unit. This new vector can be part of a new tensor that is different from the tensor written to the intermediate memory unit in the first clock cycle sequence. This new vector can be in the Z dimension. For illustrative purposes, the tensor written to the intermediate memory unit in the first clock cycle sequence is called the first tensor, and the new tensor is called the second tensor.

[0154] exist Figure 15F In the second clock cycle of the second clock cycle sequence, data stored in the second top row of the intermediate storage unit (i.e., another vector of the first tensor in the X dimension) is read from the intermediate storage unit. This vector can be written to local memory 440. In the same cycle, another vector from the second tensor is written to the second top row of the intermediate storage unit. This row-by-row read / write operation continues.

[0155] Figure 15G The layout of the transit memory cell is shown after the seventh clock cycle of the second clock cycle sequence. Data stored in the first seven rows of the transit memory cell (i.e., data in the first tensor) is read, and data in the second tensor is written to these rows.

[0156] exist Figure 15H In the second clock cycle sequence, during the last clock cycle, data stored in the bottom row of the intermediate memory cell is read from the intermediate memory cell. In the same cycle, another vector from the second tensor is written to the bottom row of the intermediate memory cell. The second clock cycle sequence is then complete, meaning the reading of the first tensor and the writing of the second tensor are finished.

[0157] Because the first tensor is written column-wise (each column stores a vector in the Z dimension) but read row-wise (each row stores a vector in the X dimension), this alternating write / read pattern of the first tensor can change its Z-major order format to X-format. Furthermore, compared to currently available DNN accelerators that cannot write tensors before completing the reading of the previous tensor, the flushing process can be more efficient because the second tensor is written in the same clock cycle as the first tensor.

[0158] Figures 15I-15L The third clock cycle sequence is shown. Figure 15IIn the first cycle of the third clock cycle sequence, the data in the rightmost column of the intermediate storage unit (i.e., the vector of the second tensor in the X dimension) is read. In the same clock cycle, the vector of 8 data points from the third tensor is written to the rightmost column of the intermediate storage unit. The data points of the third tensor are represented by dotted squares.

[0159] exist Figure 15I In the second cycle of the third clock cycle sequence, data from the second rightmost column of the intermediate storage unit (i.e., another vector of the second tensor in the X dimension) is read. In the same clock cycle, a vector of another 8 data points from the third tensor is written to the second rightmost column of the intermediate storage unit. This column-wise read / write process continues.

[0160] exist Figure 15K In the process, seven columns of data are read from the intermediate storage unit. Furthermore, the seven vectors of the third tensor are written to these seven columns of the intermediate storage unit. Figure 15L In the last clock cycle of the third clock cycle sequence, data is read from the last column of the intermediate memory unit. Additionally, in the same cycle, the eighth vector of the third tensor is written to the last column of the intermediate memory unit.

[0161] Because the second tensor is written row-wise (each column stores a vector in the Z dimension) but read column-wise (each row stores a vector in the X dimension), this alternating write / read pattern can change the Z-major order format of the second tensor to the X-major order format. Furthermore, compared to currently available DNN accelerators that cannot write tensors before completing the reading of the previous tensor, the ejection process can be more efficient because the third tensor is written in the same clock cycle as the second tensor.

[0162] Despite Figures 15A to 15L Not shown, but alternating read / write operations can continue. In some embodiments, the first, second, or third tensor can be a sub-tensor of the output tensor of a neural network operation computed by a computation engine (e.g., processing engine 370 or post-processing engine 380). The output tensor can be an input tensor of another neural network operation (e.g., a neural network operation in the next layer of a DNN). The first, second, and third tensors can be written from the intermediate storage unit to the local memory 440. Figures 15A to 15LThe process described in the text can be used to permutate the output tensor so that the order in which data points are stored in local memory 440 differs from the order in which the data points are output from the computation engine. This process can be used to change the Z-major order format to the X-major order format, change the X-major order format to the Z-major order format, or implement other types of tensor permutations. Tensor permutations can be part of a neural network operation. Additionally or alternatively, tensor permutations can improve the utilization of components in the computation engine to execute the next layer of the DNN, which uses the first, second, or third tensor as input data.

[0163] Figure 16 This is a block diagram of a DNN module 1600 according to various embodiments. The DNN module 1600 can be... Figure 4 An example of the DNN module 401. (See example...) Figure 16 As shown, the DNN module 1600 includes an interface module 1610, a training module 1620, a compression module 1630, a validation module 1640, a compiler 1650, and a data repository 1660. In other embodiments, the DNN module 1600 may include alternative configurations, different or additional elements. Furthermore, the functionality implemented by the components of the DNN module 1600 may be performed by other components included in the DNN module 1600 or by other modules or systems.

[0164] Interface module 1610 facilitates communication between DNN module 1600 and other modules or systems. For example, interface module 1610 establishes communication between DNN module 1600 and an external database to receive data that can be used to train the DNN or input into the DNN to perform a task. As another example, interface module 1610 sends a configuration descriptor to DNN accelerator 402 to configure components of DNN accelerator 402 for DNN execution. As yet another example, interface module 1610 supports DNN module 1600 in distributing the DNN to other systems (e.g., computing devices configured to apply the DNN to perform tasks).

[0165] Training module 1620 trains the DNN using a training dataset. Training module 1620 forms the training dataset. In the example where training module 1620 trains the DNN to recognize objects in an image, the training dataset includes training images and training labels. The training labels describe the ground-truth classification of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a portion of the training dataset can be used for initial DNN training, and the remainder of the training dataset can be reserved as a validation subset, which is used by validation module 1640 to validate the performance of the trained DNN. A portion of the training dataset excluding the tuning and validation subsets can be used to train the DNN.

[0166] The training module 1620 also determines the hyperparameters used to train the DNN. Hyperparameters are variables that specify the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., filter weights). In some embodiments, hyperparameters include variables that determine the DNN architecture, such as the number of hidden layers. Hyperparameters also include variables that determine how the DNN is trained, such as batch size, number of epochs, etc. The batch size defines the number of training samples to be processed before updating the DNN parameters. The batch size is equal to or less than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines the number of forward and backward passes through the entire network on the entire training dataset. The number of epochs defines the number of times the deep learning algorithm processes the entire training dataset. An epoch means that each training sample in the training dataset has a chance to update the parameters inside the DNN. An epoch can include one or more batches. The number of epochs can be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0167] The training module 1620 defines the architecture of the DNN, for example, based on some hyperparameters. The DNN architecture includes an input layer, an output layer, and multiple hidden layers. The input layer of the DNN can include tensors specifying attributes of the input image (e.g., a multidimensional array), such as the height, width, and depth of the input image (e.g., specifying the number of bits in the pixel colors of the input image). The output layer includes labels for the objects in the input layer. Hidden layers are layers between the input and output layers. Hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully connected layers, normalization layers, SoftMax, or logistic layers. The convolutional layers of the DNN abstract the input image into a feature map, which is represented by a tensor specifying the feature map height, width, and number of channels (e.g., red, green, and blue images have three channels). Pooling layers are used to reduce the spatial volume of the input image after convolution. They are used between two convolutional layers. Fully connected layers involve weights, biases, and neurons. They connect neurons in one layer to neurons in another layer. It is used to classify images between different categories through training.

[0168] In defining the DNN architecture, the training module 1620 also adds activation functions to the hidden or output layers. The activation function of a layer transforms the weighted sum of the layer's inputs into the layer's output. Activation functions can be, for example, ReLU, tangent, or other types of activation functions.

[0169] After defining the architecture of the DNN in training module 1620, training module 1620 feeds a training dataset into the DNN. The training dataset includes multiple training samples. Examples of training samples include objects in images and the ground truth labels of those objects. Training module 1620 modifies the parameters internal to the DNN (“DNN internal parameters”) to minimize the error between the labels of the training objects generated by the DNN and the ground truth labels of the objects. Internal parameters include the weights of the filters in the convolutional layers of the DNN. In some embodiments, training module 1620 uses a cost function to minimize the error.

[0170] Training module 1620 can train the DNN for a predetermined number of rounds. The number of rounds is a hyperparameter that defines how many times the deep learning algorithm will process the entire training dataset. One round means that each sample in the training dataset has a chance to update the internal parameters of the DNN. After training module 1620 completes the predetermined number of rounds, it can stop updating the parameters in the DNN. The DNN with updated parameters is called a trained DNN.

[0171] Compression module 1630 compresses the DNN. For example, compression module 1630 can add pruning operations to the DNN layer to reduce computational complexity or memory usage. Pruning operations can prune the weight tensor of a DNN layer by changing one or more non-zero value weights to zero. This modification can be performed before, during, or after training. Weights can be pruned during training, during inference, or a combination of both. Compression module 1630 can determine the sparsity ratio of the DNN layer. The sparsity ratio can be the ratio of the number of zero-value weights to the total number of weights in the layer. Compression module 1630 can perform pruning operations until the sparsity ratio of the DNN layer reaches a target sparsity ratio, such as 10%, 20%, 30%, 40%, 50%, etc.

[0172] In some embodiments, the compression module 1630 can select one or more layers in the DNN and modify each selected layer through a pruning operation. For example, the compression module 1630 can select computationally complex layers, such as layers with large filters. For a pruning operation on a layer or a class of layers, the compression module 1630 can determine a weight threshold that will not cause the DNN's accuracy loss to exceed the accuracy loss constraint. The pruning operation can modify weights with absolute values ​​higher than the weight threshold to zero, while other weights remain unchanged. Weight pruning can reduce memory storage because zero-value weights can be left unstored. Furthermore, the number of operations in a layer can be reduced because the computation of zero-value weights can be skipped without affecting the layer's output. In some embodiments, the compression module 1630 can also measure the energy savings, final DNN accuracy, or layer-by-layer sparsity caused by the pruning operation.

[0173] After compressing the DNN, the compression module 1630 can fine-tune the DNN, for example, through a retraining process. The compression module 1630 fine-tunes the DNN after the weights have been pruned. In some embodiments, the fine-tuning process is a retraining or further training process. For example, after the weights in the DNN have been pruned, the compression module 1630 can further train the DNN by inputting a training dataset into the DNN. The values ​​of the unpruned weights in the DNN can be modified based on the output of the DNN and the true labels of the training samples in the training dataset. In some embodiments, the values ​​of the pruned weights (i.e., zero) do not change during the fine-tuning process. For example, the compression module 1630 can place a mask on the pruned weight block to prevent the values ​​in the pruned weight block from being changed during the fine-tuning process. In other embodiments, the values ​​of all weights (including pruned weights) may be changed during the fine-tuning process. After one or more retraining and weight-changing cycles by the compression module 1630, the compression module 1630 can perform a new pruning process, for example, by selecting a weight block and pruning the selected weight block. In some embodiments, the weight pruning process may be repeated multiple times before the fine-tuning process is complete.

[0174] In some embodiments, the number of rounds in the fine-tuning process may differ from the number of rounds in the training process (where the values ​​of the weights before pruning are determined). For example, the number of rounds in the fine-tuning process may be fewer than the number of rounds in the training process. In the examples, the number of rounds in the fine-tuning process may be relatively small, such as 2, 3, 4, 5, etc.

[0175] The validation module 1640 validates the accuracy of the trained or compressed DNN. In some embodiments, the validation module 1640 feeds samples from the validation dataset into the trained DNN and uses the DNN's output to determine the model's accuracy. In some embodiments, the validation dataset may consist of some or all of the samples from the training dataset. Alternatively, the validation dataset may include additional samples outside the training set. In some embodiments, the validation module 1640 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The validation module 1640 may use the following metrics to determine the accuracy score: precision = TP / (TP + FP) and recall = TP / (TP + FN), where precision can be the number of correctly predicted DNN instances (TP or true positives) divided by the total number of predicted instances (TP + FP or false positives), and recall can be the number of correctly predicted DNN instances (TP) divided by the total number of objects with the attribute in question (TP + FN or false negatives). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single metric.

[0176] The verification module 1640 can compare the accuracy score with a threshold score. In an example where the verification module 1640 determines that the accuracy score of the DNN is lower than the threshold score, the verification module 1640 instructs the training module 1620 to retrain the DNN. In one embodiment, the training module 1620 can iteratively retrain the DNN until a stopping condition occurs, such as an accuracy measurement indicating that the DNN may be accurate enough or that multiple rounds of training have already been performed.

[0177] Compiler 1650 compiles information from the DNN into executable instructions that can be executed, for example, by DNN accelerator 302, to perform neural network operations within the DNN. In some embodiments, compiler 405 can generate a graph representing the DNN. This graph can include nodes and edges. Nodes can represent specific neural network operations in the DNN. Edges can connect two nodes and represent a connection between two corresponding neural network operations. In an example, an edge can encode a tensor flowing from one neural network operation to another. This tensor can be the output tensor of the first neural network operation and the input tensor of the second neural network operation. Edges can encode one or more properties of the tensor, such as size, shape, storage format, etc. Compiler 1650 can use this graph to generate an executable DNN. For example, the compiler can generate computer program instructions (e.g., compiler descriptors) for executing the DNN. The instructions can be stored in registers associated with components of the DNN accelerator 302.

[0178] The compiler 1650 can also generate configuration descriptors specifying the operating modes of components in the DNN accelerator 402, such as one or more configuration descriptors specifying whether the DPU 430 operates in dense or sparse mode. In dense mode, the DPU 430 can execute the DNN layer without any sparsity acceleration. For example, the DPU 430 in dense mode may not skip computations on zero-valued data elements. In sparse mode, the DPU 430 can accelerate the execution of the DNN layer based on sparsity by skipping computations on zero-valued data elements. The compiler 1650 can determine whether to accelerate the layer based on weight sparsity, activation sparsity, or both. The compiler 1650 can select a sparse mode for a layer from a set of sparse modes, including, for example, a combined sparse mode (where the layer is accelerated based on both weight sparsity and activation sparsity), an activation sparsity mode (where the layer is accelerated based on activation sparsity but not weight sparsity), a weight sparsity mode (where the layer is accelerated based on weight sparsity but not activation sparsity), and a dense mode (where the layer is not accelerated based on sparsity). In some embodiments (e.g., embodiments where the layer is executed by multiple DPUs 430), the compiler 1650 can determine the sparse mode for all DPUs 430 executing the layer.

[0179] Additionally or alternatively, compiler 1650 may determine whether DPU 430 operates in Z-major order, X-major order, or Y-major order mode when executing DNN layers. Compiler 1650 may determine the tensor storage format of the DNN layer, which may be the format of the input tensors of the DNN layer, and select the corresponding mode as the operating mode of DPU 430. In some embodiments, after selecting the operating mode of DPU 430, compiler 1650 may generate configuration descriptors that configure the components of DPU 430 to operate in the selected operating mode. Configuration descriptors include configuration descriptors that can configure data storage in DNN accelerator 402. For example, compiler 1650 may generate configuration descriptors that configure data storage in IDU 450 or ODU 490 to adopt new functionality when DPU 430 operates in a mode that may differ from its original functionality. The new functionality may be the functionality of another data storage in IDU 450 or ODU 490. In some embodiments, compiler 1650 may generate a configuration descriptor for configuring data storage based on the operating mode of DPU 430 (e.g., based on a configuration descriptor specifying the operating mode). The configuration descriptor may be provided to DPU 430, for example, through interface module 1610.

[0180] The compiler 1650 can also generate configuration parameters that facilitate data reading or writing, such as configuration parameters indicating the number of data elements to be processed (e.g., the number of data elements in the chip), configuration parameters indicating the memory address where input data elements can be obtained, configuration parameters indicating the memory address where output data elements can be stored, configuration parameters indicating the memory address where other configuration parameters can be stored, etc.

[0181] Data repository 1660 stores data received, generated, used, or otherwise associated with DNN module 1600. For example, data repository 1660 stores datasets used by training module 1620 and validation module 1640. Data repository 1660 may also store data generated by training module 1620 and validation module 1640, such as hyperparameters used to train the DNN, intrinsic parameters of the trained DNN (e.g., weights), data used for sparsity acceleration (e.g., sparse bitmaps), etc. Data repository 1660 may store configuration descriptors, configuration parameters, or other data generated by compression module 1630. Data repository 1660 may include one or more memories. Figure 16 In one embodiment, the data repository 1660 is a component of the DNN module 1600. In other embodiments, the data repository 1660 may be external to the DNN module 1600 and communicate with the DNN module 1600 via a network.

[0182] Example computing device Figure 17 This is a block diagram of an example computing device 1700 according to various embodiments. In some embodiments, the computing device 1700 may be used as at least a portion of a DNN system 400. Multiple components are... Figure 17 The components are shown as being included in computing device 1700, but any one or more of these components may be omitted or copied to suit the application. In some embodiments, some or all of the components included in computing device 1700 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured on a single system-on-a-chip (SoC) die. Furthermore, in various embodiments, computing device 1700 may not include... Figure 17The computing device 1700 may include one or more of the components shown, but may include interface circuitry for coupling to said one or more components. For example, the computing device 1700 may not include display device 1706, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which display device 1706 may be coupled. In another set of examples, the computing device 1700 may not include audio input device 1718 or audio output device 1708, but may include audio input or output device interface circuitry (e.g., connectors and support circuitry) to which audio input device 1718 or audio output device 1708 may be coupled.

[0183] Computing device 1700 may include processing device 1702 (e.g., one or more processing devices). Processing device 1702 processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory. Computing device 1700 may include memory 1704, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drive. In some embodiments, memory 1704 may include memory sharing a die with processing device 1702. In some embodiments, memory 1704 includes one or more non-transitory computer-readable media storing instructions executable to perform operations performed by one or more components of DNN system 400 (e.g., DNN module 401, etc.). Instructions stored in one or more non-transitory computer-readable media can be executed by processing device 1702.

[0184] In some embodiments, computing device 1700 may include communication chip 1712 (e.g., one or more communication chips). For example, communication chip 1712 may be configured to manage wireless communication for transmitting data to and from computing device 1700. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, technologies, communication channels, etc., which can transmit data through a non-solid medium using modulated electromagnetic radiation. This term does not imply that the associated device does not contain any wires, however, in some embodiments they may not.

[0185] The 1712 communication chip can implement any of many wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, such as Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), the Long Term Evolution (LTE) project, and any amendments, updates, and / or revisions (e.g., the Improved LTE project, the Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compatible with IEEE 802.16 are often referred to as WiMAX networks, an abbreviation for Global Microwave Access Interoperability, which is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. The communication chip 1712 can operate according to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. The communication chip 1712 can also operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1712 can operate according to Code-Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunication (DECT), Evolution-Data Optimized (EV-DO) and its derivatives, as well as any other wireless protocol specified as 3G, 4G, 5G, etc. In other embodiments, the communication chip 1712 can operate according to other wireless protocols.The computing device 1700 may include an antenna 1722 to facilitate wireless communication and / or receive other wireless communications (e.g., AM or FM radio transmissions).

[0186] In some embodiments, the communication chip 1712 can manage wired communications such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1712 may include multiple communication chips. For example, a first communication chip 1712 may be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1712 may be dedicated to long-range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1712 may be dedicated to wireless communications, and the second communication chip 1712 may be dedicated to wired communications.

[0187] Computing device 1700 may include battery / power circuitry 1714. Battery / power circuitry 1714 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of computing device 1700 to a power source (e.g., AC line power) that is separate from computing device 1700.

[0188] The computing device 1700 may include a display device 1706 (or the corresponding interface circuitry described above). For example, the display device 1706 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0189] The computing device 1700 may include an audio output device 1708 (or a corresponding interface circuit as described above). For example, the audio output device 1708 may include any device that generates audible indicators, such as a speaker, headphones, or earphones.

[0190] The computing device 1700 may include an audio input device 1718 (or a corresponding interface circuit as described above). The audio input device 1718 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital musical instrument (e.g., a musical instrument with a Musical Instrument Digital Interface (MIDI) output).

[0191] The computing device 1700 may include a GPS device 1716 (or a corresponding interface circuit as described above). As is known in the art, the GPS device 1716 can communicate with a satellite-based system and can receive the location of the computing device 1700.

[0192] The computing device 1700 may include other output devices 1710 (or corresponding interface circuitry as described above). Examples of other output devices 1710 may include audio codecs, video codecs, printers, wired or wireless transmitters for providing information to other devices, or additional storage devices.

[0193] The computing device 1700 may include other input devices 1720 (or corresponding interface circuitry as described above). Examples of other input devices 1720 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0194] The computing device 1700 can have any desired form factor, such as a handheld or mobile computer system (e.g., a mobile phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), ultraportable personal computer, etc.), desktop computer system, server or other networked computing component, printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 1700 can be any other electronic device that processes data.

[0195] Select Example The following paragraphs provide various examples of the embodiments disclosed herein.

[0196] Example 1 provides an apparatus comprising: a writing module configured to write data to a memory, wherein the data is at least a portion of output data of a neural network layer, the output data being computed by a computing unit performing one or more computations in the neural network layer; and a data storage unit configured to: store sparse data indicating the sparsity of the output data in a first operating mode of the apparatus, and store at least a portion of the data in a second operating mode of the apparatus, wherein the second operating mode differs from the first operating mode.

[0197] Example 2 provides the apparatus of Example 1, wherein, in the first operating mode, the data includes all data elements in the output data, and in the second operating mode, the data is generated by removing one or more data elements from the output data.

[0198] Example 3 provides the apparatus of Example 2, wherein one or more computations performed by the computing unit for the next neural network layer are accelerated in the first operating mode by skipping at least a portion of the one or more computations based on the sparsity data.

[0199] Example 4 provides the apparatus of Example 2 or 3, and further includes a compression module configured to: remove one or more data elements from the output data to generate the data in the second operating mode; and generate the sparse data.

[0200] Example 5 provides an apparatus of any one of Examples 1-4, wherein the apparatus further includes a transit storage unit configured to change the order of data elements in the data before the data is written to the memory.

[0201] Example 6 provides the apparatus of Example 5, wherein the data storage unit is configured in a second operating mode of the apparatus to: store the data elements in a first order in which the data elements are received by the apparatus from the computing unit; and after storing the data elements in the first order, transfer the data elements to the intermediate storage unit in the first order, wherein the intermediate storage unit is configured to change the first order to a second order different from the first order.

[0202] Example 7 provides the apparatus of Example 5, wherein the data storage unit is further configured in a second operating mode of the apparatus to: store the data elements in a first order in which the data elements are received by the apparatus from the computing unit; after storing the data elements in the first order, change the first order of the data elements to a second order; and transfer the data elements to the transit storage unit in the second order, wherein the transit storage unit is configured to change the second order to a third order different from the first order and the second order.

[0203] Example 8 provides an apparatus of any one of Examples 5-7, wherein the intermediate storage unit includes a transposeable register file and is configured to change the order of the data elements by writing the data elements to the transposeable register file in one of a column-wise and a row-wise manner, the transposeable register file including storage elements arranged in columns and rows; and reading the data elements from the transposeable register file in another of the column-wise and the row-wise manner.

[0204] Example 9 provides an apparatus of any one of Examples 1-8, wherein the write module includes a write combination buffer configured to: receive a data transfer transaction for writing the data to the memory; in a first operating mode of the write module, combine the data transfer transactions to generate a combined data transfer transaction, and write the data to the memory by executing the combined data transfer transaction; and in a second operating mode of the write module, store the data transfer transaction, and write the data to the memory by executing the data transfer transaction in the order in which the data transfer transactions were received.

[0205] Example 10 provides an apparatus of any one of Examples 1-9, wherein the sparse data comprises a sequence of sparse data elements, the sparse data elements corresponding to data elements in the output data and indicating whether the data elements are zero.

[0206] Example 11 provides an apparatus comprising: a reading module configured to read data from a memory, wherein the data is at least a portion of an input tensor of a neural network layer and will be used by a computing unit to perform one or more computations in the neural network layer; and a data storage unit configured to: store sparse data to be used by the computing unit to accelerate the one or more computations in a first operating mode of the apparatus, and store one or more data elements of the data before the one or more data elements are provided to the computing unit in a second operating mode of the apparatus, wherein the second operating mode differs from the first operating mode.

[0207] Example 12 provides the apparatus of Example 11, wherein the data storage unit is further configured in a second operating mode of the apparatus to: receive the one or more data elements by sequentially receiving responses to one or more data read requests issued by the read module; and transmit the one or more data elements in the specified order.

[0208] Example 13 provides an apparatus of Example 11 or 12, wherein in the first operating mode, all data elements in the data are non-zero data elements, and in the second operating mode, the data elements in the data are zero.

[0209] Example 14 provides an apparatus of any of Examples 11-13, wherein the sparse data comprises a sequence of sparse data elements, each sparse data element corresponding to a data element in the input tensor and indicating whether the data element is zero.

[0210] Example 15 provides an apparatus comprising: a computing unit configured to perform one or more computations in a neural network layer having inputs and outputs; and a data transfer unit configured to transfer data in the inputs or outputs of the neural network layer between a memory and the computing unit, the data transfer unit including a data storage unit configured to: store sparse data indicating at least a portion of the sparsity of the inputs or outputs of the neural network layer in a first operating mode of the apparatus, and store at least a portion of the data in a second operating mode of the apparatus, wherein the second operating mode differs from the first operating mode.

[0211] Example 16 provides an apparatus of Example 15, wherein the first operating mode is a sparse mode, in which the one or more computations performed by the computing unit are accelerated by skipping at least a portion of the one or more computations based on the sparse data.

[0212] Example 17 provides an apparatus of Example 15 or 16, wherein the data transfer unit is configured to transfer data from the memory to the computing unit, the data being at least a portion of the input to the neural network layer, and the apparatus further includes an additional data transfer unit configured to transfer at least a portion of the output of the neural network layer from the computing unit to the memory.

[0213] Example 18 provides an apparatus of Example 17, wherein the additional data transmission unit includes an additional data storage unit configured to: store sparse data indicating at least a portion of the sparsity of the output of the neural network layer in a first operating mode of the apparatus; and store at least a portion of the output of the neural network layer in a second operating mode of the apparatus.

[0214] Example 19 provides an apparatus of Example 15 or 16, wherein the data transfer unit is configured to transfer data from the computing unit to the memory, the data being at least a portion of the output of the neural network layer, and the apparatus further includes an additional data transfer unit configured to transfer at least a portion of the input of the neural network layer from the memory to the computing unit.

[0215] Example 20 provides an apparatus of Example 19, wherein the additional data transmission unit includes an additional data storage unit configured to: store sparse data indicating the sparsity of at least a portion of the input to the neural network layer in a first operating mode of the apparatus; and store at least a portion of the input to the neural network layer in a second operating mode of the apparatus.

[0216] Example 21 provides an apparatus of any one of Examples 19 or 20, wherein the data transmission unit further includes another storage unit configured to change the order of data elements before the data elements of the output of the neural network layer are written to the memory in a first operating mode and a second operating mode of the apparatus.

[0217] Example 22 provides an apparatus of Example 21, wherein the data storage unit is configured in a second operating mode of the apparatus to: store the data elements in a first order in which the data elements are received by the apparatus from the computing unit; and after storing the data elements in the first order, transfer the data elements to the other storage unit in the first order, wherein the other storage unit is configured to change the first order to a second order different from the first order.

[0218] Example 23 provides the apparatus of Example 21, wherein the data storage unit is further configured in a second operating mode of the apparatus to: store the data elements in a first order in which the data elements are received by the apparatus from the computing unit; after storing the data elements in the first order, change the first order of the data elements to a second order; and transfer the data elements to the other storage unit in the second order, wherein the other storage unit is configured to change the second order to a third order different from the first order and the second order.

[0219] Example 24 provides an apparatus of any one of Examples 19-23, wherein the data transfer unit further includes a write combination buffer configured to: receive a data transfer transaction for writing the data to the memory; in a first operating mode of the write combination buffer, combine the data transfer transactions to generate a combined data transfer transaction, and write the data to the memory by executing the combined data transfer transaction; and in a second operating mode of the write combination buffer, store the data transfer transaction, and write the data to the memory by executing the data transfer transaction in the order in which the data transfer transactions were received.

[0220] Example 25 provides an apparatus of any one of Examples 15-24, wherein the apparatus further includes the memory.

[0221] The foregoing description of illustrative implementations of this disclosure, including the description in the abstract, is not intended to be exhaustive or to limit this disclosure to the precise form disclosed. Although specific implementations and examples of this disclosure have been described herein for illustrative purposes, various equivalent modifications can be made within the scope of this disclosure, as will be recognized by those skilled in the art. These modifications can be made to this disclosure based on the above detailed description.

Claims

1. An apparatus comprising: A writing module is configured to write data to a memory, wherein the data is at least a portion of the output data of a neural network layer, the output data being calculated by a computing unit performing one or more computations in the neural network layer; as well as The data storage unit is configured as follows: In the first operating mode of the device, sparse data indicating the sparsity of the output data is stored, and In a second operating mode of the device, at least a portion of the data is stored, wherein the second operating mode is different from the first operating mode.

2. The apparatus according to claim 1, wherein, In the first operating mode, the data includes all data elements in the output data, and in the second operating mode, the data is generated by removing one or more data elements from the output data.

3. The apparatus according to claim 2, wherein, One or more computations performed by the computing unit for the next neural network layer are accelerated in the first operating mode by skipping at least a portion of the one or more computations based on the sparsity data.

4. The apparatus according to claim 2 or 3, further comprising a compression module, the compression module being configured to operate in the second operating mode: Remove one or more data elements from the output data to generate the data; and The sparse data is generated based on the output data.

5. The apparatus according to any one of claims 1-4, wherein the apparatus further comprises a transfer storage unit configured to change the order of data elements in the data before the data is written to the memory.

6. The apparatus of claim 5, wherein the data storage unit is configured in the second operating mode of the apparatus as follows: The data elements are stored in a first order in which they are received from the computing unit by the device; and After storing the data elements in the first order, the data elements are transferred to the intermediate storage unit in the first order. The intermediate storage unit is configured to change the first order to a second order that is different from the first order.

7. The apparatus according to claim 5, wherein the data storage unit is further configured in the second operating mode of the apparatus to: The data elements are stored in a first order in which they are received from the computing unit by the device. After storing the data elements in the first order, the first order of the data elements is changed to a second order; and The data elements are transferred to the intermediate storage unit in the second order. The intermediate storage unit is configured to change the second order to a third order that is different from the first order and the second order.

8. The apparatus according to any one of claims 5-7, wherein the intermediate storage unit comprises a transposeable register file, and the intermediate storage unit is configured to change the order of the data elements by: The data elements are written to the transposeable register file in one of two modes: column-wise and row-wise. The transposeable register file includes storage elements arranged in both columns and rows. The data element is read from the transposeable register file using either the column-based or row-based method.

9. The apparatus according to any one of claims 1-8, wherein the write module includes a write combination buffer, the write combination buffer being configured to: Receive a data transfer transaction for writing the data into the memory; In the first operating mode of the write module, the data transmission transactions are combined to generate a combined data transmission transaction, and the data is written to the memory by executing the combined data transmission transaction; and In the second operating mode of the write module, the data transmission transaction is stored, and the data is written to the memory by executing the data transmission transaction in the order in which the data transmission transaction is received.

10. The apparatus according to any one of claims 1-9, wherein the sparse data comprises a sequence of sparse data elements, the sparse data elements corresponding to data elements in the output data and indicating whether the data elements are zero.

11. An apparatus comprising: A read module is configured to read data from memory, wherein the data is at least a portion of the input tensor of a neural network layer, and the data will be used by a computing unit to perform one or more computations in the neural network layer; as well as The data storage unit is configured as follows: In the first operating mode of the device, sparse data that will be used by the computing unit to accelerate the one or more computations is stored; as well as In a second operating mode of the device, one or more data elements are stored in the data before the one or more data elements are provided to the computing unit, wherein the second operating mode is different from the first operating mode.

12. The apparatus of claim 11, wherein the data storage unit is further configured in the second operating mode of the apparatus to: The one or more data elements are received by sequentially receiving responses to one or more data read requests issued by the read module; and Transmit the one or more data elements in the stated order.

13. The apparatus according to claim 11 or 12, wherein in the first operating mode, all data elements in the data are non-zero data elements, and in the second operating mode, the data elements in the data are zero.

14. The apparatus according to any one of claims 11-13, wherein the sparse data comprises a sequence of sparse data elements, the sparse data elements corresponding to data elements in the input tensor and indicating whether the data elements are zero.

15. An apparatus comprising: A computing unit is configured to perform one or more computations in a neural network layer having inputs and outputs; as well as A data transmission unit is configured to transmit data from the input or output of the neural network layer between the memory and the computing unit. The data transmission unit includes a data storage unit configured to: In the first operating mode of the device, sparse data indicating the sparsity of at least a portion of the inputs or outputs of the neural network layer is stored, and In a second operating mode of the device, at least a portion of the data is stored, wherein the second operating mode is different from the first operating mode.

16. The apparatus of claim 15, wherein the first operating mode is a sparse mode, wherein the one or more computations performed by the computing unit are accelerated by skipping at least a portion of the one or more computations based on the sparse data.

17. The apparatus of claim 15 or 16, wherein the data transfer unit is configured to transfer data from the memory to the computing unit, the data being at least a portion of the input of the neural network layer, and the apparatus further comprises an additional data transfer unit configured to transfer at least a portion of the output of the neural network layer from the computing unit to the memory.

18. The apparatus of claim 17, wherein the additional data transmission unit includes an additional data storage unit, the additional data storage unit being configured to: In a first operating mode of the device, sparse data indicating the sparsity of at least a portion of the output of the neural network layer is stored; and In the second operating mode of the device, at least a portion of the output of the neural network layer is stored.

19. The apparatus of claim 15 or 16, wherein the data transfer unit is configured to transfer data from the computing unit to the memory, the data being at least a portion of the output of the neural network layer, and the apparatus further comprises an additional data transfer unit configured to transfer at least a portion of the input of the neural network layer from the memory to the computing unit.

20. The apparatus of claim 19, wherein the additional data transmission unit includes an additional data storage unit, the additional data storage unit being configured to: In a first operating mode of the device, sparse data indicating the sparsity of at least a portion of the input to the neural network layer is stored; and In the second operating mode of the device, at least a portion of the input to the neural network layer is stored.

21. The apparatus according to any one of claims 19 or 20, wherein the data transmission unit further comprises another storage unit configured to change the order of data elements before data elements of the output of the neural network layer are written to the memory in a first operating mode and a second operating mode of the apparatus.

22. The apparatus of claim 21, wherein the data storage unit is configured in the second operating mode of the apparatus as follows: The data elements are stored in a first order in which they are received from the computing unit by the device; and After storing the data elements in the first order, the data elements are transferred to the other storage unit in the first order. The other storage unit is configured to change the first order to a second order that is different from the first order.

23. The apparatus of claim 21, wherein the data storage unit is further configured in the second operating mode of the apparatus to: The data elements are stored in a first order in which they are received from the computing unit by the device. After storing the data elements in the first order, the first order of the data elements is changed to a second order; and The data elements are transferred to the other storage unit in the second order. The other storage unit is configured to change the second order to a third order that is different from the first order and the second order.

24. The apparatus according to any one of claims 19-23, wherein the data transmission unit further comprises a write combination buffer, the write combination buffer being configured to: Receive a data transfer transaction for writing the data into the memory; In the first operating mode of the write combination buffer, the data transfer transactions are combined to generate a combined data transfer transaction, and the data is written to the memory by executing the combined data transfer transaction; and In the second operating mode of the write combination buffer, the data transfer transaction is stored, and the data is written to the memory by executing the data transfer transaction in the order in which the data transfer transaction is received.

25. The apparatus according to any one of claims 15-24, wherein the apparatus further comprises the memory.

Images