Multifunctional memory in neural network accelerator

Abstract

A deep neural network (DNN) accelerator may include one or more tiles, each of which has a local memory. The local memory may include memory cuts, each of which has one or more memory banks. The local memory can facilitate address swizzling transformations. The local memory may receive a request for transferring data associated with a computation in a DNN. The request may include information indicating a first memory address that points to a first memory cut in the local memory. The local memory may transform the first memory address to a second memory address that corresponds to a second memory cut in the local memory. The data associated with the computation in the neural network is to be read from or written to the second memory cut. Memory banks may include state machines that can facilitate memory scrubbing. The local memory may also detect errors in data transfer requests.

Description

MULTIFUNCTIONAL MEMORY IN NEURAL NETWORK ACCELERATOR Technical Field

[0001] This disclosure relates generally to neural networks (also referred to as “deep neural networks” or “DNN”), and more specifically, multifunctional memories in DNN accelerators. Background

[0002] DNNs are used extensively for a variety of artificial intelligence (AI) applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write. Therefore, techniques to improve efficiency of DNNs are needed. Brief Description of the Drawings

[0003] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0004] FIG.1 illustrates an example DNN, in accordance with various embodiments.

[0005] FIG.2 illustrates an example convolution, in accordance with various embodiments.

[0006] FIG.3 is a block diagram of a DNN system, in accordance with various embodiments.

[0007] FIG.4 is a block diagram of a DNN module, in accordance with various embodiments.

[0008] FIG.5 illustrates an example sparse cell, in accordance with various embodiments.

[0009] FIG.6 illustrates an example sparse cell array, in accordance with various embodiments.

[0010] FIG.7 illustrates an example processing element (PE), in accordance with various embodiments.

[0011] FIG.8 illustrates memory access collision, in accordance with various embodiments.

[0012] FIG.9 illustrates an example memory, in accordance with various embodiments.

[0013] FIG.10 illustrates an example control space module, in accordance with various embodiments.

[0014] FIG.11 illustrates data allocation in a memory without address swizzle, in accordance with various embodiments.

[0015] FIG.12 illustrates data allocation in a memory with address swizzle, in accordance with various embodiments.

[0016] FIG.13 illustrates an example input address, in accordance with various embodiments.

[0017] FIG.14 illustrates an example swizzled address, in accordance with various embodiments.

[0018] FIG.15 illustrates extracting swizzle seeds from an input address for address swizzing transformation, in accordance with various embodiments.

[0019] FIGS.16A-16F illustrate determination of swizzled memory cut select bits, in accordance with various embodiments.

[0020] FIG.17 illustrates memory scrubber handshake waves, in accordance with various embodiments.

[0021] FIG.18 illustrates an example process of error detection in a memory, in accordance with various embodiments.

[0022] FIG.19 is a flowchart of a method for executing a DNN, in accordance with various embodiments.

[0023] FIG.20 is a block diagram of an example computing device, in accordance with various embodiments. Detailed Description Overview

[0024] The last decade has witnessed a rapid rise in AI based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more deep learning operations (also referred to as “neural network operations”), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on.

[0025] Input or output data of deep learning operations may be arranged in data structures called tensors. A tensor is a data structure having multiple elements across one or more dimensions. Examples of tensors include vector (which is one-dimensional (1D) tensor), matrix (which is two-dimensional (2D) tensor), three-dimensional (3D) tensors, four- dimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along the axis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. Taking a convolutional layer for example, the input tensors include an activation tensor (also referred to as “input feature map (IFM)” or “input activation tensor”) including one or more activations (also referred to as “input elements”) and a weight tensor. The weight tensor may be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a group of filters (a 4D weight tensor). A convolution may be performed on the input activation tensor and weight tensor to compute an output activation tensor in the convolutional layer.

[0026] Tensors in DNNs can be saved in X-major (e.g., XYZ or XZY format), Y-major formats (e.g., YXZ or YZX format), or Z-major formats (e.g., ZXY or ZYX format). The format of a tensor may define the order in which the data points in the tensor are stored, written, or read. The first character may represent the dimension in which data points are contiguous in memory. The second character may represent the dimension in which data points can be accessed after the contiguous data points are accessed in memory. The third character may represent the dimension in which data points are accessed after the data points in the dimension represented by the second character are exhausted. Taking the ZXY format for example, the access order first starts in the Z dimension, then moves to the X dimension, and finally moves to the Y dimension. Data points in the tensor are contiguous in memory in the Z dimension, meaning data points having the same (x, y) coordinates are contiguous in memory. Using tensor permutation, the tensor may be read from memory in a different format.

[0027] The significant improvements in DNN model size and accuracy coupled with the rapid increase in computing power of execution platforms have led to the adoption of DNN applications even within resource constrained mobile and edge devices that have limited energy availability. DNN models may be executed, e.g., for training or inference, by DNNaccelerators. A DNN accelerator may be or include one or more data processing units. A data processing unit may also be referred to as a compute block or compute tile. A data processing unit may include PEs that can carry out neural network operations.

[0028] DNN accelerators are typically designed to speed up the process of carrying out DNNs. The process of carrying out DNNs is a process of executing DNNs that includes executing the layers or neural network operations in the DNNs. Some DNN accelerators are also referred to as neural processing units (NPUs), AI processors, or AI processing units. An example DNN accelerator includes a data processing unit (DPU) providing computation capacity, a memory providing data storage capacity, and a direct memory access (DMA) module allowing pipelined movement of parameters and data between the memory and a memory local to the DPU (“local memory”). The DPU is a compute unit, which may also be referred to as “compute block” or “compute tile.” Some DNN accelerators may have multiple DPUs, i.e., multiple tiles. A DPU may include an array of PEs, a data signal processor (DSP), a local memory, other components, or some combination thereof.

[0029] Local memory may be built out of multiple memory banks, such as static random- access memory banks. The memory banks are connected to the access ports of agents (e.g., PEs, DSP, DMA module, etc.) using one or more interconnects, which may be interconnect fabric made of multiple links with assigned bandwidth that agents can utilize to read data from and write data to the local memory. The data may include parameters, input data of neural network operations, output data of neural network operations, and so on. Memory banks can be organized in series without any additional grouping or hierarchy that is aware of the inherent nature of the access pattern. In some currently available interconnect topologies, memory grouping is introduced to enable higher computation efficiency by better PE utilization and less starvation. In addition, interconnect can facilitate accessing other local memories in the DNN accelerator where tiling of compute / memory resources is employed. Tiles may be aggregated together to split workload, and broadcasting or multicasting capability in Cross-Tile Fabric may be available to allow agents to write data into multiple local memories of different tiles in a single access cycle.

[0030] Usage of generic and directly addressable storage is usually not optimal due to the inherent nature of AI workloads and characteristics of traffic patterns generated by these computational engines. For example, the number of agent ports connected to a local memory may be significantly smaller than the number of memory cuts. A memory cut is partof the local memory and may include one or more memory banks in the local memory. A memory cut may be a memory instance. The ratio between agent ports and memory cuts can be high. Such a high ratio can become a performance threat for high-bandwidth, low- latency memory as contention rate to resources is high when many or all agents are active. It can lead to access collision or complex arbitration. Many currently available solutions for this issue rely on Time-division multiplexing (TDM). The arbitration strategy that is based on TDM cycles through all memory cuts, selecting a new memory instance every single time whereby the address increments by the memory cut width. The cyclic nature of accesses can provide an organic solution for memory contention. When multiple requests are competing for the same memory instance, the winner can move on and get access to the memory line decreasing the number of contending parties. In DNN accelerators, access patterns are usually nonlinear and highly application dependent and as such this solution has limited benefit.

[0031] Some currently available solutions employ an address interleaving technique to optimize transfer rate between agents and memory banks. For instance, linearly increasing memory address can be sweeping first through memory banks and then through words within memory banks. Interleaving can enable efficient use of memory by allowing multiple agents ports target contiguous memory location simultaneously as those are directed to different memory banks. In an example where port #0 accesses local memory location 0 while port #1 accesses local memory location 1, interconnect can direct them respectively to Bank #0 / Word #0 and Bank #1 / Word #0 so that the two requests can proceed in parallel.

[0032] Some currently available solutions overcome contention between multiple ports accessing same resource (e.g., the same memory cut) by introducing temporal arbitration in the PE array. These solutions can temporally map ports to different shared resources so maximum performance in terms of bandwidth usage can be achieved. The ports may be data readers or writers. Data required by the agents can be mapped to a linear address space. Discrete addresses can be mapped to the same shared memory resource. This one- to-many mapping of addresses to shared memory resource occurs since the number of memory cuts may be much smaller than the size of the access address space. Scenarios like this can create performance penalty due to access collisions per transaction cycle. Address swizzling is used to maintain two separate address spaces for the memory cuts and theports. All request addresses can undergo a transformation by the PE array before entering on-chip memory with the goal of reducing the probability of access collision.

[0033] However, usage of address swizzling temporal arbitration within the PE array has a downside as it is exclusively crafted for it. It prevents other functions of the DNN accelerator from operating unless the data can be un-swizzled. Specifically, the DSP and DMA module do not understand address swizzled format and as such the PE’s address swizzle feature must be disabled when other components of the DNN accelerator are consuming or producing data. In addition, this feature can add extra complexity into compiler as weights and parameters need to be swizzled ahead of time for the DMA module to transparently bring it from off-chip to on-chip memory in the address swizzled data format without native support of it inside the DMA module. As the PEs array is the component that can consume and produce data written using swizzled address scheme, it can significantly limit usage of this feature. Introducing similar feature in other components of the DNN accelerator has the disadvantage of introducing control software complexity as solution becomes disaggregated across multiple components and must be coordinated by software.

[0034] DNN accelerator may perform application switch to yield for higher priority task or to avoid starvation of another task which has been scheduled to execute on it. During application switch, the local memory can be erased for security reason to ensure input data, output data, or parameters are not available to other processes. This usually needs to be done quickly to reduce switching overhead. In currently available DNN accelerators, the DMA module is used to perform local memory cleanup. The DMA module can support write tasks, also referred as “Memset.” When “Memset” operation is used, the DMA module can generate stream of writes with linear address pattern which sets the local memory using data pattern provided into the DMA module through register setup or task descriptor.

[0035] However, memory scrubbing driven by the DMA module can be power inefficient and highly time-consuming operation and impact overall application switch time. It can serially set pattern (word by word) in each local memory location by generating series of memory write requests (e.g., one word at a time). The lack of power efficiency can come from utilizing local memory fabric to carry same data pattern to millions of memory words using standard write requests which lights up many logics cell within it. Some currently available memory solutions handle all access exceptions, such as like accesses to non-implemented regions ornon-supported multicast or broadcast operations (e.g., when workload tiling is employed), by quietly dropping writes and returning 0’s for read requests. The inability for the control firmware to have good visibility into memory access exceptions can have a debuggability issue.

[0036] Requests into non-implemented memory address ranges or non-supported multicast or broadcast operation due to bugs or security attacks can happen. When not captured, such exceptions can impact results or hang design without easy way to root cause. For example, DNN run can provide invalid results without any indication of issue when parameter writes are quietly dropped, or data reads has provided response with 0’s in data payload. When such issues arise, it can significantly increase debug time and slow down the control firmware and compiler development time.

[0037] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing multifunctional memory devices that can be used as local memories of DPUs in DNN accelerators. An example multifunctional memory in the present disclosure can facilitate address swizzling, memory scrubbing, error detection, other functions, or some combination thereof. Such multifunctional memories can improve performance, power efficiency, and debuggability of DNN accelerators.

[0038] In various embodiments of the present disclosure, a DNN accelerator may include one or more tiles, each of which has a local memory. The local memory may include memory instances (e.g., memory cuts), each of which has one or more memory banks. The local memory can perform temporal arbitration through address swizzling. In an example, the local memory may receive a request for transferring data associated with a computation in a DNN. The local memory may receive the request from an agent, e.g., a PE array, DSP, DMA module, etc. The request may include information indicating a first memory address that points to a first memory cut in the local memory. The local memory may transform the first memory address to a second memory address that corresponds to a second memory cut in the local memory. The data associated with the computation in the neural network is to be read from or written to the second memory cut. The transformation of the memory address can prevent the first memory cut from receiving multiple (or too many) access requests in the same transaction cycle and therefore can avoid access collision. By implementing the address swizzling function in the local memory, network-level performance of DNN accelerators in the present disclosure can be improved due to datamovement improvement across all agents connected to the local memory, as opposed to being limited locally to the PE array in some currently available DNN accelerators.

[0039] The local memory may also facilitate memory scrubbing, e.g., by using scrubbers implemented in memory banks. A scrubber may be a state machine. In an example, each memory bank in the local memory may include a state machine that can perform memory scrubbing. The state machines in different memory banks may perform memory scrubbing in parallel. The state machines may be controlled by the same configuration space register. When a data transfer request is received by a memory bank during memory scrubbing, the request may be temporally stored in a first-in-fist-out (FIFO) buffer and would be processed after the memory scrubbing is complete. Embedding the memory scrubbing solution within the local memory can significantly improves wipe-out time and application switch performance and provides a more efficient utilization of computation resources, compared with currently available DNN accelerators in which memory scrubbing is done by the DMA module. The present disclosure can provide significant improvement in switch time.

[0040] The local memory may also be implemented with debuggability. For instance, the local memory may detect errors in data transfer requests, such as errors related to address, errors related to data broadcasting or multicasting, errors related to tile access, other types of errors, or some combination thereof. An error may be an exception. Different from currently available memories in DNN accelerators that have limited debuggability support, the present disclosure integrates an exception handling capability into the local memory through error request sink, which can improve debuggability by indicating to software access errors with details required for effective debug (e.g., violating request, violation code, etc.).

[0041] Compared with currently available approaches for address swizzing, memory scrubbing, and debugging, the present disclosure integrates these functions into local memories in DNN accelerators can improve performance of DNN accelerators without additional impact to power and silicon area. It can lead to less wasted compute cycles and energy savings.

[0042] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practicedwith only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0043] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0044] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0045] For the purposes of the present disclosure, the phrase “A or B” or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase “A, B, or C” or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0046] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives “first,” “second,” and “third,” etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0047] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0048] The terms “substantially,” “close,” “approximately,” “near,” and “about,” generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0049] In addition, the terms “comprise,” “comprising,” “include,” “including,” “have,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term “or” refers to an inclusive “or” and not to an exclusive “or.”

[0050] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings. Example DNN

[0051] FIG.1 illustrates an example DNN 100, in accordance with various embodiments. The DNN 100 may be executed by a DNN accelerator, e.g., the DNN accelerator 302 in FIG.3. In an example, the DNN 100 may be a convolution-based DNN. In other examples, the DNN 100 may be other types of DNNs. For the purpose of illustration, the DNN 100 includes a sequence of layers comprising a plurality of convolutional layers 110 (individually referred to as “convolutional layer 110”), a plurality of pooling layers 120 (individually referred to as “pooling layer 120”), and a plurality of fully-connected layers 130 (individually referred to as “fully-connected layer 130”). In other embodiments, the DNN 100 may include fewer, more, or different layers. In an execution of the DNN 100, the layers of the DNN 100 execute tensor computation that includes many tensor operations, such as matrix multiplications, convolutions (e.g., multiply-accumulate (MAC) operations, etc.), pooling operations, elementwise operations (e.g., elementwise addition, elementwise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0052] The convolutional layers 110 summarize the presence of features in inputs to the DNN 100. The convolutional layers 110 function as feature extractors. The first layer of the DNN 100 is a convolutional layer 110. In an example, a convolutional layer 110 performs a convolution on an input tensor 140 (also referred to as IFM 140) and a filter 150. As shown in FIG.1, the IFM 140 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 140 includes 3 input channels, each of which is represented by a 7×7 two-dimensional (2D) matrix. The 7×72D matrix includes 7 input elements (also referred to as input points) in each row and 7 input elements in each column. The filter 150 is represented by a 3×3×33D matrix. The filter 150 includes 3 kernels, each of which may correspond to a different input channel of the IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM. In the embodiments of FIG.1, each kernel is represented by a 3×32D matrix. The 3×3 kernel includes 3 weights in each row and 3 weights in each column. Weights can be initialized and updated by backpropagation using gradient descent. The magnitudes of the weights can indicate imdata portance of the filter 150 in extracting features from the IFM 140.

[0053] The convolution includes MAC operations with the input elements in the IFM 140 and the weights in the filter 150. The convolution may be a standard convolution 163 or a depthwise convolution 183. In the standard convolution 163, the whole filter 150 slides across the IFM 140. All the input channels are combined to produce an output tensor 160 (also referred to as OFM 160). The OFM 160 is represented by a 5×52D matrix. The 5×52D matrix includes 5 output elements (also referred to as output points) in each row and 5 output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG.1. In embodiments where there are multiple filters, the standard convolution may produce multiple output channels in the OFM 160.

[0054] The multiplication applied between a kernel-sized patch of the IFM 140 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernel- sized patch of the IFM 140 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referred to as the “scalar product.” Using a kernel smaller than the IFM 140 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 140 multiple times at different points on the IFM 140. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 140, left to right, top to bottom. The result frommultiplying the kernel with the IFM 140 one time is a single value. As the kernel is applied multiple times to the IFM 140, the multiplication result is a 2D matrix of output elements. As such, the 2D output matrix (i.e., the OFM 160) from the standard convolution 163 is referred to as an OFM.

[0055] In the depthwise convolution 183, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an output channel. As shown in FIG.1, the depthwise convolution 183 produces a depthwise output tensor 180. The depthwise output tensor 180 is represented by a 5×5×3 3D matrix. The depthwise output tensor 180 includes 3 output channels, each of which is represented by a 5×52D matrix. The 5×52D matrix includes 5 output elements in each row and 5 output elements in each column. Each output channel is a result of MAC operations of an input channel of the IFM 140 and a kernel of the filter 150. For instance, the first output channel (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots), the second output channel (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips), and the third output channel (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 193 is then performed on the depthwise output tensor 180 and a 1×1×3 tensor 190 to produce the OFM 160.

[0056] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is rectified linear unit (ReLU). ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 110 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 160 is passed to the subsequent convolutional layer 110 (i.e., the convolutional layer 110 following the convolutional layer 110 generating the OFM 160 in the sequence). Thesubsequent convolutional layers 110 perform a convolution on the OFM 160 with new kernels and generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelled again by a further subsequent convolutional layer 110, and so on.

[0057] In some embodiments, a convolutional layer 110 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions F×F×D pixels), the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time), and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 110). The convolutional layers 110 may perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0058] The pooling layers 120 down-sample feature maps generated by the convolutional layers, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 120 is placed between two convolution layers 110: a preceding convolutional layer 110 (the convolution layer 110 preceding the pooling layer 120 in the sequence of layers) and a subsequent convolutional layer 110 (the convolution layer 110 subsequent to the pooling layer 120 in the sequence of layers). In some embodiments, a pooling layer 120 is added after a convolutional layer 110, e.g., after an activation function (e.g., ReLU, etc.) has been applied to the OFM 160.

[0059] A pooling layer 120 receives feature maps generated by the preceding convolution layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their imdata portant characteristics. Accordingly, the pooling operation improves the efficiency of the DNN and avoids over- learning. The pooling layers 120 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map), max pooling (calculating the maximum value for each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. In various embodiments, the pooling operation is 2×2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., thenumber of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 120 applied to a feature map of 6×6 results in an output pooled feature map of 3×3. The output of the pooling layer 120 is inputted into the subsequent convolution layer 110 for further feature extraction. In some embodiments, the pooling layer 120 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

[0060] The fully-connected layers 130 are the last layers of the DNN. The fully-connected layers 130 may be convolutional or not. The fully-connected layers 130 receive an input operand. The input operand defines the output of the convolutional layers 110 and pooling layers 120 and includes the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully-connected layers 130 apply a linear combination and an activation function to the input operand and generate a vector. The vector may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 1, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 130 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function. In some embodiments, the fully-connected layers 130 multiply each input element by weight, make the sum, and then apply an activation function (e.g., logistic if N=2, SoftMax if N>2). This is equivalent to multiplying the input operand by the matrix containing the weights.

[0061] FIG.2 illustrates an example convolution, in accordance with various embodiments. The convolution may be a deep learning operation in a convolutional layer of a DNN, e.g., a convolutional layer 110 in FIG.1. The convolution can be executed on an activation tensor 210 and filters 220 (individually referred to as “filter 220”). The filters may constitute a weight tensor of the convolution. The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator. An example of the DNN accelerator may be the DNN accelerator 302 in FIG.3. For instance, the convolution may be performed by one or more DPUs 330 in the DNN accelerator 302.

[0062] The activation tensor 210 may be computed in a previous layer of the DNN. In some embodiments (e.g., embodiments where the convolutional layer is the first layer of the DNN), the activation tensor 210 may be an image. In the embodiments of FIG.2, the activation tensor 210 includes activations (also referred to as “input activations,”“elements,” or “input elements”) arranged in a 3D matrix. The activation tensor 210 may also be referred to as an input tensor of the convolution. An input element is a data point inthe activation tensor 210. The activation tensor 210 has a spatial size ^^^ × ^^^ × ^^^,where ^^^is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel), ^^^is the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of activationsin a row in the 2D matrix of each input channel), and ^^^is the depth of matrix (i.e., the length along the Z axis, which indicates the number of input channels). For the purpose of simplicity and illustration, the activation tensor 210 has a spatial size of 7×7×3, i.e., the activation tensor 210 includes three input channels and each input channel has a 7×72D matrix. Each input element in the activation tensor 210 may be represented by a (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the activation tensor 210 may be different.

[0063] Each filter 220 includes weights arranged in a 3D matrix. The values of the weightsmay be determined through training the DNN. A filter 220 has a spatial size ^^ × ^^ × ^^,where ^^is the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in a column in each kernel), ^^is the width of the filter (i.e., the length along the X axis, which indicates the number of weights in a row in each kernel), and ^^is the depth of the filter (i.e., the length along the Z axis, which indicates the number of channels). In some embodiments, ^^equals ^^^. For purpose of simplicity and illustration, each filter 220 in FIG.2 has a spatial size of 2×3×3, i.e., the filter 220 includes 2 convolutional kernels with a spatial size of 2×3. In other embodiments, the height, width, or depth of the filter 220 may be different. The spatial size of the convolutional kernels is smaller than the spatial size of the 2D matrix of each input channel in the activation tensor 210.

[0064] An activation or weight may take one or more bytes in a memory. The number of bytes for an activation or weight may depend on the data format. For example, when the activation or weight has an INT8 format, the activation takes one byte. When the activation or weight has a FP16 format, the activation or weight takes two bytes. Other data formats may be used for activations or weights.

[0065] In the convolution, each filter 220 slides across the activation tensor 210 and generates a 2D matrix for an output channel in the output tensor 230. In the embodiments of FIG.2, the 2D matrix has a spatial size of 5×5. The output tensor 230 includes activations (also referred to as “output activations,” “elements,” or “output element”) arranged in a 3D matrix. An output activation is a data point in the output tensor 230. The output tensor 230has a spatial size ^^^^ × ^^^^ × ^^^^, where ^^^^ is the height of the 3D matrix (i.e., thelength along the Y axis, which indicates the number of output activations in a column in the 2D matrix of each output channel), ^^^^is the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 2D matrix of each output channel), and ^^^^is the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels). ^^^^may equal the number of filters 220 in the convolution. ^^^^and ^^^^may depend on the heights and weights of the activation tensor 210 and each filter 220. In an example where the kernel size is 1×1, ^^^^and ^^^^may equal to ^^^and ^^^, respectively.

[0066] As a part of the convolution, MAC operations can be performed on a 2×3×3 subtensor 215 (which is highlighted with a dotted pattern in FIG.2) in the activation tensor 210 and each filter 220. The result of the MAC operations on the subtensor 215 and one filter 220 is an output activation. In some embodiments (e.g., embodiments where the convolution is an integral convolution), an output activation may include 8 bits, e.g., one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), an output activation may include more than one byte. For instance, an output element may include two bytes.

[0067] After the MAC operations on the subtensor 215 and all the filters 220 are finished, a vector 235 is produced. The vector 235 is highlighted with a dotted pattern in FIG.2. The vector 235 includes a sequence of output activations, which are arranged along the Z axis. The output activations in the vector 235 have the same (x, y) coordinate, but the output activations correspond to different output channels and have different Z coordinates. The dimension of the vector 235 along the Z axis may equal the total number of output channels in the output tensor 230. After the vector 235 is produced, further MAC operations are performed to produce additional vectors till the output tensor 230 is produced. In the embodiments of FIG.2, the output tensor 230 is computed in a Z-major format. When the output tensor 230 is computed in the ZXY format, the vector that is adjacent to the vector235 along the X axis may be computed right after the vector 235. When the output tensor 230 is computed in the ZYX format, the vector that is adjacent to the vector 235 along the Y axis may be computed right after the vector 235. The output tensor 230 may be permuted, e.g., by the drain module 390, and stored in a memory (e.g., the local memory 340) in an X- major format or Y-major format.

[0068] In some embodiments, the MAC operations on a 3×3×3 subtensor (e.g., the subtensor 215) and a filter 220 may be performed by a plurality of MAC units. One or more MAC units may receive an input operand (e.g., an activation operand 217 shown in FIG.2) and a weight operand (e.g., the weight operand 227 shown in FIG.2). The activation operand 217 includes a sequence of activations having the same (x, y) coordinate but different z coordinates. The activation operand 217 includes an activation from each of the input channels in the activation tensor 210. The weight operand 227 includes a sequence of weights having the same (x, y) coordinate but different z coordinates. The weight operand 227 includes a weight from each of the channels in the filter 220. Activations in the activation operand 217 and weights in the weight operand 227 may be sequentially fed into a MAC unit. The MAC unit may receive an activation and a weight (“an activation-weight pair”) at a time and multiple the activation and the weight. The position of the activation in the activation operand 217 may match the position of the weight in the weight operand 227. The activation and weight may correspond to the same channel.

[0069] Activations or weights may be floating-point numbers. Floating-point numbers may have various data formats, such as FP32, FP16, BF16, and so on. A floating-point number may be a positive or negative number with a decimal point. A floating-point number may be represented by a sequence of bits that includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing an exponent of the floating-point number, and bits representing a mantissa of the floating-point number. The mantissa is the part of a floating-point number that represents the significant digits of that number. The mantissa is multiplied by the base raised to the exponent to give the actual value of the floating-point number.

[0070] In some embodiments, the output activations in the output tensor 230 may be further processed based on one or more activation functions before they are written into the memory or inputted into the next layer of the DNN. The processing based on the one or more activation functions may be at least part of the post processing of the convolution. Insome embodiments, the post processing may include one or more other computations, such as offset computation, bias computation, and so on. The results of the post processing may be stored in a local memory of the compute block and be used as input to the next DNN layer. In some embodiments, the input activations in the activation tensor 210 may be results of post processing of the previous DNN layer. Example DNN System

[0071] FIG.3 is a block diagram of a DNN system 300, in accordance with various embodiments. The whole DNN system 300 or a part of the DNN system 300 may be implemented in one or more computing devices, such as the computing device 2000 in FIG. 20. The DNN system 300 can generate and execute DNNs, such as Transformer-based models, convolution-based models, and so on. As shown in FIG.3, the DNN system 300 includes a DNN module 301 and a DNN accelerator 302. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 300. For instance, the DNN system 300 may include multiple DNN modules or multiple DNN accelerators. Further, functionality attributed to a component of the DNN system 300 may be accomplished by a different component included in the DNN system 300 or a different system. In some embodiments, the DNN module 301 and DNN accelerator 302 may include different types of processing units. In an example, the DNN module 301 may be implemented by one or more central processing units (CPUs). The DNN module 301 and DNN accelerator 302 may be implemented in the same chip or separate chips.

[0072] The DNN module 301 facilitates generation and deployment of DNNs. In some embodiments, the DNN module 301 may generate and train DNNs. For instance, the DNN module 301 can define the layered architecture of a DNN. The DNN module 301 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 301 may also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the sparsity level of one or more deep learning tensors for the DNN.

[0073] The DNN module 301 may also compress DNNs, e.g., during or after training. In some embodiments, the DNN module 301 may prune weights in one or more layers of a DNN by changing nonzero valued weight to zeros. The DNN module 301 may prune weights based on a target weight sparsity ratio. A weight sparsity ratio may be the ratio of the number of zero-valued weights to the total number of weights. In an example where the DNN module301 prunes weight during DNN training, the DNN module 301 may prune weight of a layer to achieve a target sparsity ratio after one or more epochs. The DNN module 301 may prevent the pruned weights from changing values during the rest of the training process. Alternatively, the DNN module 301 may allow the pruned weights to change values so that a pruned, zero-valued weight may have a nonzero value after further training. The DNN module 301 may prune weights of the layer again after one or more additional epochs.

[0074] The DNN module 301 may deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, the DNN module 301 may distribute trained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform tasks (e.g., image classification, motion planning, etc.) for which the DNNs were trained. In other embodiments, the DNN module 301 may facilitate deployment of the DNNs using the DNN accelerator 302. For instance, the DNN module 301 may receive data from a device or system coupled with the DNN system 300 and input the received data (or data generated by the DNN module 301, e.g., based on the received data) into a DNN. The DNN module 301 may receive an output of the DNN from the DNN accelerator 302. The DNN module 301 may transmit the output of the DNN (or a result of processing the output of the DNN by the DNN module 301) to the device or system. In some embodiments, the DNN module 301 may control execution processes of trained, compressed, or validated DNNs. The DNN module 301 may function as a complier for DNNs executed by the DNN accelerator 302. The DNN module 301 may perform compilation of DNNs and generate compilation descriptors, based on which the DNNs may be executed.

[0075] The DNN module 301 may control the operation of the DNN accelerator 302 during the DNN execution. For instance, the DNN module 301 may generate registers based on which components of the DNN accelerator 302 operate to execute DNNs. In some embodiments, the DNN module 301 may generate configuration space registers that control operations in memories in the DNN accelerator 302. A configuration space may include all possible configurations or states of one or more memories in the DNN accelerator 302. The configuration space registers may be used to configure the memories during DNN execution. Certain aspects of the DNN module 301 are provided below in conjunction with FIG.4.

[0076] The DNN accelerator 302 executes DNNs provided by the DNN module 301. For instance, the DNN accelerator 302 can execute a DNN by running neural network operationsin the DNN. The process of carrying out a neural network operation is also referred to as a process of executing the neural network operation or a process of performing the neural network operation. The execution of the DNN may be for training the DNN or for using the DNN to perform AI tasks. As shown in FIG.3, the DNN accelerator 302 includes a memory 310, a DMA module 320, and DPUs 330 (individually referred to as “DPU 330”). In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 302. For example, the DNN accelerator 302 may include more than one memory 310 or DMA module 320. As another example, the DNN accelerator 302 may include a single DPU 330. Further, functionality attributed to a component of the DNN accelerator 302 may be accomplished by a different component included in the DNN accelerator 302 or by a different system. A component of the DNN accelerator 302 may be implemented in hardware, software, firmware, or some combination thereof.

[0077] The memory 310 stores data associated with neural network operations performed by the DNN accelerator 302. In some embodiments, the memory 310 may store data to be used by the DPUs 330 for DNN execution. The memory 310 may store weights, such as weights of convolutional layers, which are determined by training DNNs. The memory 310 may further store inputs to DNN layers or outputs of DNN layers, such as data generated by the DPUs 330 from performing neural network operations in DNNs. Example neural network operations include convolutions (also referred to as “convolutional operations”), layer normalization operations, SoftMax operations, matrix multiplication operations, pooling operations, elementwise operations, activation functions, other types of neural network operations, or some combination thereof. The memory 310 may also store configuration descriptors or parameters generated by the DNN module 301. The memory 310 may be a main memory of the DNN accelerator 302. In some embodiments, the memory 310 includes one or more dynamic random-access memories (DRAMs).

[0078] The DMA module 320 facilitates data transfer between the memory 310 and local memories of the DPUs 330. For example, the DMA module 320 can read data from the memory 310 and write data into a local memory of a DPU 330. As another example, the DMA module 320 can read data from a local memory of a DPU 330and write data into the memory 310. The DMA module 320 provides a DMA feature that allows the DPU 330 to initiate data transfer between the memory 310 and the local memories of the DPUs 330 and to perform other operations while the data transfer is being conducted. In someembodiments, the DMA module 320 may read tensors from the memory 310, modify the tensors in a way that is optimized for the DPU 330 before it writes the tensors into the local memories of the DPUs 330.

[0079] The DPUs 330 execute workloads in DNN execution. The DPUs 330 may perform neural network operations in DNNs for DNN training or DNN inference. For instance, a DPU 330 may execute a DNN layer by running one or more neural network operations in the DNN layer. A DPU 330 may execute a layer, or a data portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple DPUs 330 in parallel. For instance, multiple DPUs 330 may each perform a data portion of a workload for a neural network operation. Data may be shared between the DPUs 330. The DPUs 330 may be capable of running various types of neural network operations, such as convolution, layer normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Neural network operations performed by the DPUs 330 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, the DPU 330 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the DPU 330 or another DPU 330.

[0080] In the embodiments of FIG.3, each DPU 330 includes a local memory 340, a processing engine 350, a post-processing engine 360, a DSP array 370, and a data transfer module 380. The local memory 340 can provide data storage capacity. The processing engine 350, post-processing engine 360, and DSP array 370 can provide computing capacity. The data transfer module 380 can transfer data between the local memory 340 and any of the processing engine 350, post-processing engine 360, and DSP array 370. Some or all the components of the DPU 330 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the DPU 330. Further, functionality attributed to a component of the DPU 330 may be accomplished by a different component included in the DPU 330, a different DPU 330, another component of the DNN accelerator 302, or a different system. A component of the DPU 330 may be implemented in hardware, software, firmware, or some combination thereof.

[0081] Each local memory 340 may be local to the corresponding DPU 330. In the embodiments of FIG.3, the local memory 340 is inside the DPU 330. In other embodiments,the local memory 340 may be outside the DPU 330. In some embodiments, the local memory 340 includes one or more SRAMs. In some embodiments, the local memory 340 may be an on-chip memory, while the memory 310 may be an off-chip memory. Data in the local memory 340 may be transferred to or from the memory 310, e.g., through the DMA module 320. In some embodiments, data in the local memory 340 may be transferred to or from the local memory of another DPU 330.

[0082] The local memory 340 may store data generated or used for controlling DNN execution. For instance, the local memory 340 may store data generated by the DNN module 301 for controlling one or more components of the DNN accelerator 301. Data for controlling one or more components of the local memory 340 (e.g., data for controlling address swizzling transformation, error detection, scrubbing processes, etc.) may be stored in in the local memory 340. The local memory 340 may also store data processed or computed in DNNs. For instance, the local memory 340 may store data received, processed, or computed by the processing engine 350, the post-processing engine 360, or the DSP array 370. Examples of the data may include input activations, weights, output activations, sparsity bitmaps, and so on.

[0083] In some embodiments, the local memory 340 may include memory cuts. A memory cut includes one or more memory banks. In some embodiments, the number of memory banks in the local memory 340 may be 16, 64, 128, 356, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a memory bank may include 8, 16, 64, or a different number of storage units. The local memory 340 may be byte- addressable, and each memory address identifies a single byte (eight bits) of storage. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 340 in multiple read cycles, such as two cycles.

[0084] In some embodiments, each memory bank may be coupled with a separate agent port. The agent port may connect one or more agents with the corresponding memory bankso that the agent(s) can read data from the memory bank or write data to the memory bank. The agent(s) may send data transfer requests to the port. A data transfer request is a request for transferring data to or from the local memory 340. The request may include information indicating a memory address where the data is to be read from or written to. The request may also include the data itself. The data path between the port and the memory bank may include an input bridge module and an interconnect. The input bridge module may process data transfer requests received by the port. For instance, the input bridge module may perform address swizzing transformation, which is a process of transforming addresses in data transfer requests received from agents to swizzled addresses to be used for data transfer. The input bridge module may also detect whether data transfer requests have any errors. After being processed by the input bridge module, the data transfer requests may be transmitted to the memory bank through the interconnect.

[0085] In some embodiments, each memory bank may include a scrubbing module, which is also referred to as a scrubber. The scrubber may include a state machine that controls scrubbing processes in the memory bank. A scrubbing process may include reading data from the memory bank, correcting errors in the data (e.g., bit errors), and writing the corrected data back into the memory bank. The data may be written back to the same address. Scrubbers in different memory banks (or even all the memory banks) may be controlled by a configuration space register. The scrubbers may perform scrubbing processes in parallel. In some embodiments, the configuration space register may include different bits for controlling scrubbing processes in local memories of different tiles.

[0086] In some embodiments, the local memory 340 may store tensors to be processed by the processing engine 350 or the post-processing engine 360. The tensors may be input tensors of neural network operations. The local memory 340 may also store tensors generated by the processing engine 350 or the post-processing engine 360. The tensors may be output tensors of neural network operations. The layout of data points of a tensor in the local memory 340 may depend on the format in which the tensor is stored. In some embodiments, the local memory 340 may store tensors in various formats, including Z- major format, X-major format, and Y-major format. For a tensor with Z-major format, the local memory 340 may store data points having the same (x, y) coordinate contiguously. For instance, the data points having the same (x, y) coordinate may be stored at a sequence of memory addresses in the local memory 340. For a tensor with the ZXY format or ZYX format,the local memory 340 may store data points having the same (x, y) coordinate contiguously. For instance, the data points having the same (x, y) coordinate may be stored at a sequence of memory addresses in the local memory 340. For a tensor with X-major format, the local memory 340 may store data points having the same (y, z) coordinate contiguously. For a tensor with Y-major format, the local memory 340 may store data points having the same (x, z) coordinate contiguously.

[0087] In some embodiments, the local memory 340 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which zero-valued elements (if any) are not removed. A dense tensor may be converted to a sparse tensor by removing one or more zero-valued elements in the dense tensor. A sparse tensor may also be referred to as a compressed tensor or packed tensor. The process of converting a dense tensor to a sparse tensor may be referred to as sparsity encoding. Sparsity encoding may also generate a sparsity tensor. Each element in the sparsity tensor may correspond to a different element in the dense tensor and indicate whether the element in the dense tensor is zero or not. The sparsity tensor may indicate positions of elements of the sparse tensor in the dense tensor. The sparsity tensor may be a sparsity bitmap, each element of which is a bit. A sparse tensor may be converted to a dense tensor through a densifying process, in which one or more zeros may be added to the sparse tensor based on the sparsity tensor.

[0088] The processing engine 350 performs computations in neural network operations. An example PE may include one or more MAC units that can perform MAC operations. In some embodiments (e.g., embodiments where the DPU 330 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN. The weights in the weight operand may be in different input channels.

[0089] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators (“adders”) forperforming accumulations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0090] In some embodiments, the processing engine 350 may be capable of depthwise convolution, standard convolution, or both. In a depthwise convolution, an MAC unit may perform an MAC operation that includes a sequence of multiplications for an input operand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activation and weight in the same cycle may correspond to the same channel. The sequence of multiplication produces a product operand that includes a sequence of products. The MAC operation may also include accumulations in which multiple product operands are accumulated to produce an output operand of the MAC unit. The processing engine 350 may output multiple output operands at a time, each of which is generated by a different MAC unit. In a standard convolution, MAC operations may include accumulations across the channels. For instance, as opposed to generating an output operand, a MAC unit may accumulate products across different channels to generate a single output point.

[0091] In some embodiments, the processing engine 350 may perform MAC operations in quantized neural network operations, such as MAC operations in a quantized convolution. In some embodiments, an MAC unit in the processing engine 350 may receive quantized activation and quantized weights and compute a quantized MAC result. The quantized MAC result may be a quantized value in an integer format and may be the output of the MAC unit. In some embodiments, the MAC unit may also include a quantization multiplier that can multiply a quantization scale with the quantized MAC result, and the output of the MAC unit may be a real value in a floating-point format. The MAC unit may include noquantization subtractors as zero-point offsetting is not needed for the MAC operations in quantized neural network operations.

[0092] The processing engine 350 may accelerate neural network operations based on sparsity in data. In some embodiments, the processing engine 350 may operate in a dense mode in which sparsity acceleration is not performed. The processing engine 350 may include one or more processing cells. In some embodiments, the processing cells may be arranged in one or more rows and one or more columns in the processing engine 350. Each processing cell may include PEs that may be arranged in an array that includes rows and columns. All the PEs in the processing engine 350 may constitute a bigger array that includes more rows and columns.

[0093] In some embodiments, the processing engine 350 may include sparsity acceleration logic for facilitating sparsity acceleration. For instance, each processing cell in the processing engine 350 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, a sparsity module accelerates computations in the processing engine 350 based on sparsity in activations, sparsity in weights, or both. The sparsity module may include a storage unit that stores a sparsity tensor. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0094] An activation sparsity tensor may be the sparsity tensor of an activation tensor and has the same number of elements as the activation tensor. An element in the activation sparsity tensor may indicate whether the corresponding element in the activation tensor is zero or not. For instance, a zero-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is zero. A one-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is nonzero. A weight sparsity tensor may be the sparsity tensor of a weight tensor and has the same number of elements as the weight tensor. An element in the weight sparsity tensor may indicate whether the corresponding element in the weight tensor is zero or not. For instance, a zero-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is zero. A one-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is nonzero. The sparsity module may generate a combined sparsity tensor using an activation sparsity tensor and aweight sparsity tensor. For instance, the sparsity module may multiply an element of the activation sparsity tensor with a corresponding element of the weight sparsity tensor to compute an element of the combined sparsity tensor. The positions of the three elements in their corresponding sparsity tensors may match. In some embodiments, each element in a sparsity tensor may be a bit, and the sparsity tensor may be referred to as a sparsity bitmap.

[0095] The sparsity module may use the sparsity tensor to identify activations and weights to be used in MAC operations by the MAC units. In an embodiment where the processing engine 350 operates in the combined sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a combined sparsity tensor. In an embodiment where the processing engine 350 operates in the activation sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of an activation sparsity tensor. In an embodiment where the processing engine 350 operates in the weight sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a weight sparsity tensor. The sparsity module may be bypassed in the dense mode as no sparsity acceleration would be conducted.

[0096] The post-processing engine 360 processes outputs of the processing engine 350. The post-processing engine 360 may include one or more post-processing elements. In some embodiments, the post-processing elements in the post-processing engine 360 may be arranged in an array that has rows and columns. In some embodiments, the post-processing engine 360 computes activation functions. The post-processing engine 360 may receive outputs of the processing engine 350 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 360 may perform other types of post processing on outputs of the processing engine 350. For instance, the post- processing engine 360 may apply a bias on an output of the processing engine 350. In some embodiments, the post-processing engine 360 may be bypassed for certain neural network operations.

[0097] The DSP array 370 includes one or more DSPs. A DSP may be a Streaming Hybrid Architecture Vector Engine (SHAVE) DSP. In some embodiments, a DSP can be pipelined along with the inference pipeline and the DMA engine, thereby enabling parallel computations to improve overall performance. In some embodiments, the DSP array may perform some computations in a neural network operation, while other computations in theneural network operation may be performed by the processing engine 350 or the post- processing engine 360.

[0098] The data transfer module 380 facilitates data transfer within the DPU 330. For instance, the data transfer module 380 may load data from the local memory 340 to the processing engine 350, the post-processing engine 360, or the DSP array 370. The data transfer module 380 may also drain data computed in the processing engine 350, the post- processing engine 360, or the DSP array 370 and write the data to the local memory 340. The data transfer module 380 may perform data read and write operations. In some embodiments, the data transfer module 380 may generate or transmit data transfer request from the processing engine 350, the post-processing engine 360, or the DSP array 370 to the local memory 340. A data transfer request may be a read request or write request. The data transfer request may include information indicating an address where the data is to be read from or written to. In some embodiments, the address corresponds to a memory cut in the local memory 340. The address may be referred to as an input address from the perspective of the local memory 340 and may be transformed into a swizzled address by the local memory 340.

[0099] The data transfer module 380 may include a buffer. The buffer may temporarily store data received by the data transfer module 380. For instance, data computed by the processing engine 350, the post-processing engine 360, or the DSP array 370 may be stored in the buffer before being processed by the data transfer module 380. In some embodiments, the buffer includes one or more circular buffers. In some embodiments, the data transfer module 380 may include multiple buffers, each of which may be used for storing data computed in a portion of the processing engine 350, the post-processing engine 360, or the DSP array 370. The portion of the processing engine 350, the post-processing engine 360, or the DSP array 370 may be a column or a row.

[0100] In some embodiments, the data transfer module 380 may include a sparsity encoder. The sparsity encoder may compress output data of DNN layers based on sparsity in the output data. In some embodiments, the sparsity encoder may receive an output tensor (e.g., the output tensor 230 in FIG.2) of a layer. The sparsity encoder may remove zero- valued data points from a tensor to generate a compressed tensor. In some embodiments, the sparsity encoder may also generate one or more sparsity tensors for the output tensor. The sparsity tensor may include sparsity elements, each of which corresponds to a differentdata point in the tensor and indicates whether the corresponding data point is removed or not. In some embodiments, the sparsity tensor may be a sparsity bitmap, and a sparsity element in the sparsity bitmap may be a bit. A zero bit may indicate that the corresponding data point is removed and not in the compressed tensor, while a one bit may indicate that the corresponding data point is not removed and is in the compressed tensor.

[0101] FIG.4 is a block diagram of a DNN module 400, in accordance with various embodiments. The DNN module 400 may be an embodiment of the DNN module 301 in FIG. 3. As shown in FIG.4, the DNN module 400 includes an interface module 410, a training module 420, a compressing module 430, a validating module 440, a compiler 450, and a datastore 460. In other embodiments, alternative configurations, different or additional components may be included in the DNN module 400. Further, functionality attributed to a component of the DNN module 400 may be accomplished by a different component included in the DNN module 400 or a different module or system.

[0102] The interface module 410 facilitates communications of the DNN module 400 with other modules or systems. For example, the interface module 410 establishes communications between the DNN module 400 with an external database to receive data that can be used to train DNNs or input into DNNs to perform tasks. As another example, the interface module 410 may distribute trained DNNs to other systems, e.g., computing devices configured to apply DNNs to perform tasks.

[0103] The training module 420 trains DNNs by using a training dataset. The training module 420 forms the training dataset. In an example where the training module 420 trains an DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe ground-truth classifications of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a part of the training dataset may be used to initially train the DNN, and the rest of the training dataset may be held back as a validation subset used by the validating module 440 to validate performance of a trained DNN. The data portion of the training dataset not including the tuning subset and the validation subset may be used to train the DNN.

[0104] The training module 420 also determines hyperparameters for training the DNN. Hyperparameters are variables specifying the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., weights of filters). In some embodiments,hyperparameters include variables determining the architecture of the DNN, such as number of hidden layers, etc. Hyperparameters also include variables which determine how the DNN is trained, such as batch size, number of epochs, etc. A batch size defines the number of training samples to work through before updating the parameters of the DNN. The batch size is the same as or smaller than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backwards through the entire network. The number of epochs defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each training sample in the training dataset has had an opportunity to update the parameters inside the DNN. An epoch may include one or more batches. The number of epochs may be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0105] The training module 420 defines the architecture of the DNN, e.g., based on some of the hyperparameters. The architecture of the DNN includes an input layer, an output layer, and a plurality of hidden layers. The input layer of an DNN may include tensors (e.g., a multidimensional array) specifying attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layers are layers between the input layer and output layer. The hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully-connected layers, normalization layers, SoftMax or logistic layers, and so on. The convolutional layers of the DNN abstract the input image to a feature map that is represented by a tensor specifying the feature map height, the feature map width, and the feature map channels (e.g., red, green, blue images include 3 channels). A pooling layer is used to reduce the spatial volume of input image after convolution. It is used between two convolution layers. A fully-connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

[0106] In the process of defining the architecture of the DNN, the training module 420 also adds an activation function to a hidden layer or the output layer. An activation function of a layer transforms the weighted sum of the input of the layer to an output of the layer. Theactivation function may be, for example, a ReLU activation function, a tangent activation function, or other types of activation functions.

[0107] After the training module 420 defines the architecture of the DNN, the training module 420 inputs a training dataset into the DNN. The training dataset includes a plurality of training samples. An example of a training sample includes an object in an image and a ground-truth label of the object. The training module 420 modifies the parameters inside the DNN (“internal parameters of the DNN”) to minimize the error between labels of the training objects that are generated by the DNN and the ground-truth labels of the objects. The internal parameters include weights of filters in the convolutional layers of the DNN. In some embodiments, the training module 420 uses a cost function to minimize the error.

[0108] The training module 420 may train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm will work through the entire training dataset. One epoch means that each sample in the training dataset has had an opportunity to update internal parameters of the DNN. After the training module 420 finishes the predetermined number of epochs, the training module 420 may stop updating the parameters in the DNN. The DNN having the updated parameters is referred to as a trained DNN.

[0109] The compressing module 430 compresses DNNs. For instance, the compressing module 430 may add pruning operations to DNN layers to reduce computational complexity or memory usage. A pruning operation may prune weight tensors of a DNN layer by changing one or more nonzero valued weights of the layer to zeros. The modification may be done before, during, or after training. Weights may be pruned during training, during inference, or a combination of both. The compressing module 430 may determine a sparsity ratio for a DNN layer. The sparsity ratio may be a ratio of the number of zero-valued weight to the total number of weights in the layer. The compressing module 430 may perform the pruning operation till the sparsity ratio of the DNN layer meets a target sparsity ration, such as 10%, 20%, 30%, 40%, 50%, and so on.

[0110] In some embodiments, the compressing module 430 may select one or more layers in a DNN and modify each selected layer with a pruning operation. For instance, the compressing module 430 may select computationally complex layers, such as layers with large filters. For a pruning operation of a layer or of a type of layer, the compressing module 430 may determine a weight threshold that would not cause a loss of the accuracy of theDNN to exceed an accuracy loss constraint. A pruning operation may modify weights having absolute values above the weight threshold to zeros and leave the other weights unchanged. The weight pruning can reduce memory storage as zero-valued weights may not be stored. Also, the number of operations in the layer can be reduced as computations on zero-valued weights can be skipped without impacting the output of the layer. In some embodiments, the compressing module 430 may also measure energy saving, final DNN accuracy, or layer-wise sparsity caused by pruning operations.

[0111] After compressing a DNN, the compressing module 430 may fine tune the DNN, e.g., through a retraining process. The compressing module 430 may fine tunes DNNs after weights are pruned. In some embodiments, the fine-tuning process is a retraining or further training process. For instance, after weights in a DNN are pruned, the compressing module 430 may further train the DNN by inputting a training dataset into the DNN. The values of the unpruned weights in the DNN may be modified based on outputs of the DNN and ground-truth labels of the training samples in the training dataset. In some embodiments, the values of the pruned weights (i.e., zero) are not changed during the fine-tuning process. For instance, the compressing module 430 may place a mask over a pruned weight block and the mask can prevent values in the pruned weight blocks from being changed during the fine-tuning process. In other embodiments, the values of all weights, including the pruned weights, may be changed during the fine-tuning process. After one or more cycles of retraining and weight changing by the compressing module 430, the compressing module 430 may perform a new pruning process, e.g., by selecting weight blocks and pruning the selected weight blocks. In some embodiments, the weight pruning process may be repeated multiple times before the fine-tuning process is done. In some embodiments, the number of epochs in the fine-tuning process may be different from the number of epochs in the training process in which the pre-pruning values of the weights are determined. For instance, the fine-tuning process may have less epochs than the training process. In an example, the number of epochs in the fine-tuning process may be relatively small, such as 2, 3, 4, 5, and so on.

[0112] The validating module 440 verifies accuracy of trained or compressed DNNs. In some embodiments, the validating module 440 inputs samples in a validation dataset into a trained DNN and uses the outputs of the DNN to determine the model accuracy. In some embodiments, a validation dataset may be formed of some or all the samples in the trainingdataset. Additionally or alternatively, the validation dataset includes additional samples, other than those in the training sets. In some embodiments, the validating module 440 may determine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN. The validating module 440 may use the following metrics to determine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN), where precision may be how many the DNN correctly predicted (TP or true positives) out of the total it predicted (TP + FP or false positives), and recall may be how many the DNN correctly predicted (TP) out of the total number of objects that did have the property in question (TP + FN or false negatives). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single measure.

[0113] The validating module 440 may compare the accuracy score with a threshold score. In an example where the validating module 440 determines that the accuracy score of the DNN is less than the threshold score, the validating module 440 instructs the training module 420 to re-train the DNN. In one embodiment, the training module 420 may iteratively re-train the DNN until the occurrence of a stopping condition, such as the accuracy measurement indication that the DNN may be sufficiently accurate, or a number of training rounds having taken place.

[0114] The compiler 450 compiles information of DNNs to executable instructions that can be executed, e.g., by the DNN accelerator 302, to carry out neural network operations in DNNs. In some embodiments, the compiler 405 may generate a graph representing a DNN. The graph may include nodes and edges. A node may represent a specific neural network operation in the DNN. An edge may connect two nodes and represent a connection between the two corresponding neural network operations. In an example, an edge may encode a tensor that flows from one of the neural network operations to the other neural network operation. The tensor may be an output tensor of the first neural network operation and an input tensor of the second neural network operation. The edge may encode one or more attributes of the tensor, such as size, shape, storage format, and so on. The compiler 450 may use the graph to generate executable DNNs. For instance, the compiler may generate computer program instructions (e.g., compilation descriptors) for executing DNNs. The instructions may be stored in registers associated with components of the DNN accelerator 302.

[0115] In some embodiments, the compiler 450 may generate instructions associated with operation in the local memory 340, e.g., address swizzling transformation, error detection, memory scrubbing, and so on. In an example, the compiler 450 may determine swizzling parameters for address swizzling transformations in the local memory 340. The compiler 450 may determine a swizzling parameter offline, e.g., before the execution of the corresponding neural network operation is started or even before the execution of the DNN is started. In another example, the compiler 450 may generate one or more control parameters for controlling scrubbing operations in the local memory 340. The control parameters may be stored in one or more control space registers of the local memory 340.

[0116] The datastore 460 stores data received, generated, used, or otherwise associated with the DNN module 400. For example, the datastore 460 stores the datasets used by the training module 420 and validating module 440. The datastore 460 may also store data generated by the training module 420 and validating module 440, such as the hyperparameters for training DNNs, internal parameters of trained DNNs (e.g., weights, etc.), data for sparsity acceleration (e.g., sparsity bitmap, etc.), and so on. The datastore 460 may store control data (e.g., control parameters, etc.), compilation descriptors, or other data generated by the compressing module 430. The datastore 460 may include one or more memories. In the embodiment of FIG.4, the datastore 460 is a component of the DNN module 400. In other embodiments, the datastore 460 may be external to the DNN module 400 and communicate with the DNN module 400 through a network.

[0117] FIG.5 illustrates an example sparse cell 500, in accordance with various embodiments. The sparse cell 500 may be a processing cell in a processing engine, e.g., the processing engine 350 in FIG.3. The sparse cell 500 includes 16 MAC units 510 (individually referred to as “MAC unit 510”), which constitutes a MAC array having four rows and four columns. The MAC array has a spatial shape of 5x4, meaning the height of the MAC array is four and the width of the MAC array is also 5. The sparse cell 500 also includes 16 weight register files 520 (individually referred to as “weight register file 520”), 16 activation register files 530 (individually referred to as “activation register file 530”), four row buffers 540 (individually referred to as “row buffer 540”), and sparsity modules 560 (individually referred to as “sparsity module 560”). In other embodiments, the sparse cell 500 may include fewer, more, or different components. For example, the sparse cell 500 may include a different number of MAC units 510, weight register files 520, activation register files 530,row buffers 540, or sparsity modules 560. As another example, the sparse cell 500 may include column buffers in lieu of or in addition to the row buffers 540. Also, the shape (e.g., the height or width) of the MAC array may be different.

[0118] The MAC units 510 are configured to perform MAC operations. Each MAC unit 510 may include one or more multipliers and one or more adders. A multiplier may multiply an activation with a weight at a time to compute a product. In some embodiments (e.g., embodiments where the MAC unit 510 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. An adder may accumulate products computed by the multipliers. Even though not shown in FIG.5, the sparse cell may include an adder tree including a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units 510. The number of adders in the first tier may be half of the number of the MAC units 510, and each adder may accumulate the outputs of two MAC units 510. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the sparse cell 500.

[0119] The weight register files 520 store weights to be processed in MAC operations. In the embodiments of FIG.5, four weight register files 520 are grouped into a storage set that stores data to be used by a column of MAC units 510. There are four storage sets corresponding to the four columns of MAC units 510. In some embodiments, a weight register file 520 may correspond to a MAC unit 510 and store data to be processed by the MAC unit. In some embodiments, all the 16 weight register files 520 constitute a weight storage unit.

[0120] The activation register files 530 stores activations to be processed in MAC operations. In the embodiments of FIG.5, four activation register files 530 are grouped into a storage set that stores data to be used by a row of MAC units 510. There are four storage sets corresponding to the four rows of MAC units 510. In some embodiments, an activation register file 530 may correspond to a MAC unit 510 and store data to be processed by the MAC unit. In some embodiments, all the 16 activation register files 530 constitute anactivation storage unit. The row buffers 540 store outputs of the MAC units 510. Each row buffer 540 may drain outputs of a single row of MAC units 510.

[0121] The sparsity module 560 facilitates dynamic sparsity-based acceleration in the sparse cell 500. In the embodiments of FIG.5, each sparsity module 560 includes a sparsity tensor storage unit 565 and a control logic 567. The sparsity tensor storage unit 565 stores combined sparsity tensors. A combined sparsity tensor stored in the sparsity tensor storage unit 565 may correspond to an activation tensor and a weight tensor. A nonzero element in the combined sparsity tensor may correspond to a nonzero activation-weight pair that includes a nonzero activation and a nonzero weight. The position of the nonzero activation in the activation tensor may match the position of the nonzero weight in the weight tensor. The product of the nonzero activation and nonzero weight would be nonzero.

[0122] The control logic 567 may control transmission of activations and weights stored from the weight register files 520 and the activation register files 530 to the MAC units 510 based on sparsity tensors. For instance, the control logic 567 may select a subset of the weights stored in the weight register files 520 and select a subset of activations stored in the activation register files 530 based on a combined sparsity tensor. The selected weights and activations constitute nonzero activation-weight pairs. The control logic 567 may transmit the selected weights and activations to the MAC units 510 for performing MAC operations. The other weights stored in the weight register files 520 and the other activations stored in the activation register files 530 are skipped from computation. In the embodiments of FIG. 5, each sparsity module 560 controls sparsity acceleration in a respective MAC unit 510. As the sparsity acceleration is either based on both weight sparsity and activation sparsity, 16 sparsity modules 560 are used for acceleration computations in the 16 MAC units 510.

[0123] As shown in FIG.5, the sparse cell 500 is associated with multiplexers (MUXs) 503, 504, 505, and 506. In other embodiments, the sparse cell 500 may be associated with a different number of MUXs or other devices. The MUX 503 facilitates loading weights, e.g., from the local memory 340, into the weight register files 520. The MUX 504 facilitates loading activations, e.g., from the local memory 340, into the activation register files 530. The MUX 505 facilitates loading sparsity tensors into the sparsity tensor storage unit 565. The MUX 506 may be a drain MUX that can facilitate draining outputs of the MAC units 510, e.g., to the local memory 340.

[0124] In some embodiments, the sparse cell 500 may also execute matrix multiplications converted from Fourier transform operations. For an example Fourier transform operation, the MAC units 510 may perform MAC operations in the two sequences of matrix multiplications converted from the Fourier transform operation. The weight register files 520 may be used to store data points in transformation tensor of the Fourier transform operation. The activation register file 530 may be used to store data points in the input tensor of the Fourier transform operation. The row buffers 540 may store data points in the output tensor of the Fourier transform operation.

[0125] FIG.6 illustrates a sparse cell array 600, in accordance with various embodiments. The sparse cell array 600 may be an example of the processing engine 350 in FIG.3. In FIG. 6, the sparse cell array 600 includes sparse cells 610 (individually referred to as “sparse cell 610”) arranged in four columns and four rows, an activation memory 620, and a weight memory 630. In other embodiments, the sparse cell array 600 may include fewer, more, or different components. For instance, the sparse cell array 600 may include a different number of columns, rows, or sparse cells 610.

[0126] Each sparse cell 610 may perform sparsity accelerated MAC operations. The sparse cells 610 may facilitate dynamic sparsity mode. For instance, the sparsity modes of a sparse cell 610 may be dynamically changed between a combined sparsity mode, an activation sparsity mode, a weight sparsity mode, and a dense mode. An embodiment of a sparse cell 610 may be the sparse cell 500 in FIG.5. The activation memory 620 stores activations, such as activations in input tensors of neural network operations. Activations may be loaded from the activation memory 620 to sparse cells 610. The weight memory 630 stores weights, such as weights in filters of neural network operations. Weights may be loaded from the weight memory 630 to sparse cells 610. The activation memory 620 or weight memory 630 may be a buffer. In other embodiments, the sparse cell array 600 may include a dense data memory and a sparse data memory in lieu of the activation memory 620 and weight memory 630. The dense data memory may store dense tensors. The sparse data memory may store sparse tensors.

[0127] The sparse cell array 600 may also execute matrix multiplications in Fourier transform operations. The activation memory 620 may be used to store input tensors of the Fourier transform operations. The weight memory 630 may be used to store transformation matrices of the Fourier transform operations.

[0128] FIG.7 illustrates an example PE 700, in accordance with various embodiments. The PE 700 may be a unit component of a processing cell, e.g., a processing cell in the processing engine 350. In the embodiments of FIG.7, the PE 700 includes an MAC unit 705, an activation register file 710, a weight register file 720, an output register file 750, and a sparsity accelerator 760. The MAC unit 705 includes a multiplier 730 and an adder 740. In other embodiments, the PE 700 may include fewer, more, or different components.

[0129] The activation register file 710 stores an activation operand, which may be a context. The activation register file 710 may be an example of the activation register files 530 in FIG. 5. The weight register file 720 stores a weight operand. The weight register file 720 may be an example of the weight register files 520 in FIG.5. The activation operand and weight operand may be loaded from a memory (e.g., the memory 340) into the activation register file 710 and the weight register file 720, respectively. The sparsity accelerator 760 receives a sparsity bitmap 715 that corresponds to the sparse tensor in the weight register file 720. The sparsity bitmap 715 may be a combined sparsity bitmap when the MAC unit 705 operates in a combined sparsity mode. The sparsity bitmap 715 may be an activation sparsity bitmap when the MAC unit 705 operates in an activation sparsity mode. The sparsity bitmap 715 may be a weight sparsity bitmap when the MAC unit 705 operates in a weight sparsity mode. The sparsity bitmap 715 may have the same size (e.g., the same number of elements) as or a larger size than the activation operand or the weight operand.

[0130] Using the sparsity bitmap 715, the sparsity accelerator 760 selects four activations from the activation register file 710 and selects four weights from the weight register file 720. The sparsity accelerator 760 transmits the selected activations and weights to the multiplier 730. These selected data elements correspond to the nonzero valued elements of the sparsity bitmap 715. The four selected activations and the four selected weights may constitute four activation-weight pairs. The multiplier 730 may compute a product based on each activation-weight pair and therefore, compute four products in total. The four products may be provided to the adder 740. Even though FIG.7 shows a single multiplier 730, the MAC unit 705 may include multiple multipliers that can perform multiple multiplication operations at the same time.

[0131] The adder 740 accumulates the four products and computes a unit-level internal partial sum. The four unselected elements of the dense tensor are not processed to save power and time, which would not impact the value of the unit-level internal partial sum. Forinstance, when the dense tensor is a dense activation tensor, the weights corresponding to the unselected activations are zeros so the products of the unselected activations and the weights would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. Similarly, when the dense tensor is a dense weight tensor, the activations corresponding to the unselected weights are zeros so the products of the unselected weights and the activations would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. In other embodiments, the MAC unit 705 may operate in a dense mode in which the sparsity bitmap 715 is not used and the sparsity accelerator 760 is inactive. The MAC unit 705 may process all the activations in the activation operand and all the weights in the weight operand.

[0132] The unit-level internal partial sum may be stored in the output register file 750. In some embodiments, the unit-level internal partial sum may be used multiple times. For instance, the activation operand may represent ^ data blocks in the input tensor of the convolution, where ^ is an integer greater than 1. Instead of processing all the ^ data blocks to compute ^ unit-level internal partial sums, the unit-level internal partial sum is computed once and used ^ times in the convolutional layers as ^ unit-level internal partial sums.

[0133] In some embodiments, the PE 700 receives one or more PE-level internal partial sums from one or more other PEs. The adder 740 or an accumulator (not shown in FIG.7) can accumulate the one or more PE-level internal partial sums with the PE-level internal partial sum of the PE 700 and store the result of the accumulation (i.e., a multi-PE internal partial sum) in the output register file 750. The one or more other PEs may be in the same column as the PE 700 in a sparse cell. The multi-unit internal partial sum may be a column- level internal partial sum. In some embodiments, the PE-level internal partial sum of the PE 700 or the multi-unit internal partial sum may be sent to one or more other PEs for further accumulation.

[0134] FIG.8 illustrates memory access collision, in accordance with various embodiments. A local memory 810 receives data transfer requests from agents 820. The local memory 810 includes memory cuts 830, individually referred to as “memory cut 830.” The agents 820 may include PEs, post-processing elements, or DSPs. The data transfer requests from the agents 820 are received through ports A0-Am, as shown in FIG.8. The ports may be datareaders and writers. The data required by the agents 820 may be mapped to a linear address space. As shown in FIG.8, discrete addresses are mapped to the same memory cut 830. This one-to-many mapping of addresses can occur when the number of memory cuts 830 in the local memory 810 is smaller than the size of the access address space. As the same memory cut 830 receives multiple data transfer requests in the same transaction cycle, access collision occurs. Access collision can cause performance penalty. It can take multiple cycles for the memory cut 830 to process all the data transfer requests as the data transfer requests are serviced serially into a single memory cut while the other memory cuts are unused. The memory access collision can be avoided through address swizzling.

[0135] FIG.9 illustrates an example memory 900, in accordance with various embodiments. The memory 900 may be used as a multifunctional memory in a tile of a DNN accelerator. The memory 900 may be an example of the local memory 340 in FIG.3. As shown in FIG.9, the memory 900 includes agent ports 905 (individually referred to as “agent port 905”), input bridge modules 910 (individually referred to as “input bridge module 910”), interconnects 920 (individually referred to as “interconnect 920”), memory banks 930 (individually referred to as “memory bank 930”), and a scrubber control module 940. In other embodiments, alternative configurations, different or additional components may be included in the memory 900. Further, functionality attributed to a component of the memory 900 may be accomplished by a different component included in the memory 900, a different memory, or a different module. One or more components of the memory 900may be implemented in hardware, software, firmware, or some combination thereof.

[0136] The agent ports 905 facilitate communication of the memory 900 with agents. In some embodiments, the agent ports 905 receive data transfer requests from agents. A data transfer request may be a read request or a write request. A read request may be a request to read data from a memory bank 930. A read request may include information indicating one or more memory addresses from which the data is to be read. A write request may be a request to write data to a memory bank 930. A write request may include information indicating one or more memory addresses to which the data is to be written. A write request may also include the data that is to be written to the memory 900. The agent ports 905 may receive multiple data transfer requests in the same transaction cycle. These data transfer requests may come from different agents or the same agent.

[0137] In some embodiments, a data transfer request may be a broadcast or multicast request. A broadcast or multicast request may be a request for writing data to multiple memories, e.g., to the memory 900 and to one or more other memories. These memories may be in different DPUs. Alternatively, a broadcast or multicast request may be a request for reading data from multiple memories, e.g., from the memory 900 and from one or more other memories. These memories may be in different DPUs.

[0138] The input bridge module 910 processes data transfer requests received by the agent ports 905. Each input bridge module 910 includes an error detector 913, an address decoder 915, and an address swizzler 917. The error detector 913 detects errors in data transfer requests. An error may be an exception in the data transfer request. In some embodiments, the error detector 913 may determine whether a memory address of a data transfer request falls into a predetermined range. The predetermined range may be the range of memory addresses of the memory 900 or the range of multiple memories that include the memory 900. These memories may be local memories of different DPUs.

[0139] The address decoder 915 may select memory cuts or memory banks based on data transfer requests. For instance, the address decoder 915 may receive memory addresses in data transfer requests as inputs and outputs selection signals. The address decoder 915 may process a memory address to determine which memory bank 930 or memory cut is to be accessed. The address decoder 915 may output a selection signal that identifies the memory bank 930 or memory cut. In some embodiments, the address decoder 915 may identify a memory cell or a storage unit based on the memory address. The memory cell or storage unit may be a portion of a memory bank 930.

[0140] The address swizzler 917 performs temporal arbitration of data transfer requests through address sizzling. The address swizzler 917 may transform memory addresses in data transfer requests. For instance, the address swizzler 917 may change a memory address pointing to a memory cut to a different memory address pointing to a different memory cut. In some embodiments, the address swizzler 917 may perform temporal arbitration in a manner to make sure data transfer requests are shared across most or all the memory cuts of the memory 900 to minimize or even eliminate oversubscribing. With the address swizzling, memory access collision (e.g., the access collision illustrated in FIG.8) can be avoided.

[0141] In some embodiments, the address swizzler 917 may use a swizzling parameter to transform addresses. The swizzling parameter may be determined before the neural network operation is started. For instance, the swizzling parameter may be determined offline by the DNN module 301. In some embodiments, the address swizzler 917 may use the same swizzling parameter to transform addresses associated with requests from different agents. It can be a significant simplification compared with per agent address swizzle control where the control software usually needs to independently enable feature in each agent. In case feature is not supported by one of the agents, the control software must build-in fine grain control to disable this feature for certain DNNs layers where producer or consumer of data is agent without address swizzle support. It can add great complexity to software architecture and increase development time. In some embodiments, address swizzling transformation by the address swizzler 917 can be transparent to the related agent(s).

[0142] In some embodiments, the error detector 913, address decoder 915, and address swizzler 917 may process data transfer requests sequentially. For instance, the error detector 913 may detect whether a data transfer request has any error. After the error detector 913 detects no error in the data transfer request, the address decoder 915 may process the data transfer request, followed by the address swizzler 917. When the error detector 913 detects one or more errors in the data transfer request, the address decoder 915 or address swizzler 917 may be bypassed. The data transfer request may not be transmitted to the interconnects 920 or any memory banks 930.

[0143] After being processed by the input bridge module 910, data transfer requests may be forwarded to the interconnects 920. In some embodiments, the interconnects 920 includes one or more interconnect fabric through which data transfer requests may be transmitted from the input bridge module 910 to the memory banks 930. A memory bank 930 may be selected for a data transfer request based on the swizzled address of the data transfer request.

[0144] Each memory bank 930 includes a scrubbing module 932, a MUX 934, an error correcting code (ECC) module 936, and memory cells 938. The scrubbing module 932 facilitates scrubbing processes in the memory bank 930. The scrubbing module 932 may be a state machine, e.g., a finite state machine (FSM). The state machine may have an ON state and an OFF state, which indicates a scrubbing operation is being performed or not,respectively. The MUX 934 receives a signal from the scrubbing module 932 and a signal from the interconnects 920. In some embodiments, the signal from the scrubbing module 932 may indicate whether scrubbing operation is being performed. The signal from the interconnects 920 may be data transfer requests. The MUX 934 may select one of the signals. In an example where a scrubbing operation is being performed, the MUX 934 may not select any data transfer requests received from the interconnects 920. The data transfer requests may be staged in a buffer, which may be a FIFO buffer and may be processed after the scrubbing operation is complete. The ECC module 936 may correct errors in data stored in the memory bank 930 during scrubbing processes. In some embodiments, the ECC module 936 may correct bit errors in the data. The memory cells 938 store data. The memory cells 938 may store data corrected by the ECC module 936 or data in data transfer requests.

[0145] The scrubber control module 940 controls the scrubbing module 932 in each memory bank 930. The scrubber control module 940 may include registers that store instructions controlling the state of the scrubbing module 932. In some embodiments, scrubbing modules in multiple or even all the memory banks 930 may be controlled using the same register. The register may have separate bits for separate tiles to facilitate independent control of scrubbing processes in these tiles. The scrubber control module 940 may control the scrubbing modules to start or end scrubbing processes in parallel.

[0146] FIG.10 illustrates an example control space module 1000, in accordance with various embodiments. The control space module 1000 may be in a multifunctional memory, e.g., the memory 900 in FIG.9. In some embodiments, the control space module 1000 may switch an input line to multiple separate output lines. In some embodiments, the control space module 1000 may receive signals from the interconnects 920 or send signals to the scrubber control module 940. The control space module 1000 may include a demultiplexer. In an example, the control space module 1000 is a control space demultiplexer.

[0147] In some embodiments, the control space module 1000 may receive data transfer requests from the input bridge module 910 through the interconnects 920. The control space module 1000 may perform address decoding on the data transfer requests. The control space module 1000 may also determine whether the decoded address is a valid control space address. For instance, the control space module 1830 may determine whether the request transaction targets any of the control space registers 1020 coupled with thecontrol space module 1830. The control space registers 1020 are individually referred to as “control space register 1020” and collectively referred to as a control space register set. In an example embodiment, the data in a data transfer request may be data used for controlling DNN execution (e.g., for controlling one or more components of the DNN accelerator 301), the address decoded from the data transfer request may fall into a control space address range, and the control space module 1000 may determine that the decoded address is a valid control space address. The control space module 1000 may further transmit the data to the corresponding control space register(s) 1020 where the data is stored.

[0148] In some embodiments, e.g., embodiment where the control space module 1000 determines that a data transfer request fails to target any of the control space registers 1020, the control space module 1000 may forward the data transfer request to the error reporting module 1010. The error reporting module 1010 may report an error, e.g., to a control module controlling the DNN execution by the DNN accelerator 302. The control module may be the DNN module 301. The control module may suspend (e.g., pause or cancel) the DNN execution. In some embodiments, the control module may suspect the execution of a neural network operation in the DNN but may allow one or more other neural network operations in the DNNs to be executed. Example Address Swizzling

[0149] FIG.11 illustrates data allocation in a memory 1100 without address swizzle, in accordance with various embodiments. The memory 1100 may be a local memory of a DPU. In some embodiments, data (e.g., weights, activations, etc.) may be stored in a packed format to utilized storage elements with fixed sizes. For the purpose of illustration and simplicity, FIG.11 shows four storage elements 1110A-1110D, collectively referred to as “storage elements 1110” or “storage element 1110.” The four storage elements 1110 may constitute at least part of the memory 1100. Each storage element 1110 has a width that is equal to eight memory cuts. The four storage elements 1110 may correspond to an address space of the memory 1100, such as an address space in hexadecimal, e.g., from address 0×00000 to 0×00E00. Each address in the address space may correspond to a portion of each of the storage elements 1110. In an example, the first row of each of the storage elements 1110 may correspond to address 0×0000, the second row may correspond to 0×0200, the third row may correspond to 0×0400, this continues till the last row that maycorrespond to 0×00E00. In other embodiments, the storage elements 1110 may constitute a different address space. Also, a storage element 1110 may have a different size.

[0150] Data fill level in the storage elements 1110 can vary and can be less than 100% due to sparsity in the data. For the purpose of illustration and simplicity, 50% fill level inside storage element is used in FIG.11. Data is stored in a subset of memory cuts: 0-3, 8-11, 16- 19 and 24-27, while the other memory cuts (i.e., 4-7, 12-15, 20-23, and 28-31) are unused. This creates uneven usage of memory cuts as traffic is directed into half of the memory cuts. The used memory cuts are represented by boxes highlighted with patterns in FIG.11, while the unused memory cuts are represented by blank boxes.

[0151] FIG.12 illustrates data allocation in a memory 1200 with address swizzling, in accordance with various embodiments. The memory 1200 may be a local memory of a DPU, e.g., the local memory 340. For the purpose of illustration and simplicity, FIG.12 also shows four storage elements 1210A-1210D, collectively referred to as “storage elements 1210” or “storage element 1210.” The four storage elements 1210 may constitute at least part of the memory. Each storage element 1210 has a width that is equal to eight memory cuts. The four storage elements 1210 may correspond to an address space, such as an address space in hexadecimal, e.g., from address 0×00000 to 0×00E00. Each address in the address space may correspond to a portion of each of the memory cuts 0-31. In an example, the first row of each of the storage elements 1210 may correspond to address 0×0000, the second row may correspond to 0×0200, the third row may correspond to 0×0400, this continues till the last row that may correspond to 0×00E00. In other embodiments, the storage elements 1210 may constitute a different address space. Also, a storage element 1210 or a memory cut may have a different size. In an example, a memory cut may be 256-bit wide and 2048- word deep.

[0152] The data allocation in the memory 1200 may be a result of an address swizzing transformation. In some embodiments, the data stored in the memory 1200 may be the same as the data stored in the memory 1100. Due to the address swizzing transformation, the allocation of the data in the memory cuts in the embodiments of FIG.12 is different from that in the embodiments of FIG.11. As shown in FIG.12, all the memory cuts 0-32 store data. Compared with the data allocation in FIG.11, the data allocation in FIG.12 is more even among the memory cuts. Such an even usage of the memory cuts can reduce contention rate and increase overall performance of the DPU.

[0153] FIG.13 illustrates an example input address, in accordance with various embodiments. The input address may be a memory address indicated in a data transfer request from an agent. In some embodiments, the input address may include a plurality of bits. A first subset of bits in the input address may be memory cut word address bits. A second subset of bits in the input address may be input memory cut select bits. A third subset of bits in the input address may be memory word alignment bits. The number of input memory cut select bits may equal log^^N^, where N may be the number of memory cuts in the memory. For the purpose of illustration and simplicity, the input address in FIG. 13 has 21 bits, represented by [20:0]. The number of memory cuts N is 32. Each memory cut may have a width of 256 bits and a depth of 2048 words. As shown in FIG.13, 5 bits in the input address are input memory cut select bits.11 bits are input memory cut word address bits.5 bits indicate input memory word alignment.

[0154] FIG.14 illustrates an example swizzled address, in accordance with various embodiments. The swizzled address may be determined by performing an address swizzing transformation on an input address, e.g., the input address in FIG.13. The swizzled address may point to a different memory cut from the input address. A memory cut may be selected using the swizzled address. Data may be read from or written to the selected memory cut. The swizzled address includes a plurality of bits. A first subset of bits in the swizzled address may be the memory cut word address bits in the input address. A second subset of bits in the swizzled address may be swizzled memory cut select bits, which may be determined using the input memory cut select bits in the input address. A third subset of bits in the swizzled address may be the memory word alignment bits in the input address. The number of swizzled memory cut select bits may equal log^^N^, where N may be the number of memory cuts in the memory. In some embodiments, the total number of bits in the swizzled address may be the same as the total number of bits in the input address.

[0155] For the purpose of illustration and simplicity, the swizzled address in FIG.14 has 21 bits, represented by [20:0]. The number of memory cuts N is 32. Each memory cut may have a width of 256 bits and a depth of 2048 words. As shown in FIG.14, 5 bits in the swizzled address are swizzled memory cut select bits.11 bits are memory cut word address bits.5 bits indicate memory word alignment.

[0156] FIG.15 illustrates extracting swizzle seeds from an input address for address swizzing transformation, in accordance with various embodiments. The address swizzlingtransformation may be a process of transforming the input address (e.g., the input address in FIG.13) to a swizzled address (e.g., the swizzled address in FIG.14). The swizzled address may point to a different memory cut from the input address. Data may be read from or written to the memory cut corresponding to the swizzled address as opposed to the memory cut corresponding to the input address.

[0157] The address swizzling transformation includes converting input memory cut select bits in the input address to swizzle memory cut select bits in the swizzled address by using a swizzle seed. In some embodiments, swizzled memory cut select bits in the swizzled address may be determined based on input memory cut select bits in the input address using the algorithm below:^^^^^^^^_^^^^^^_ !"_^^^^ "_#^"^ = ^%&!"_^^^^^^_ !"_^^^^ "_#^"^ + ^^^^^^^^ ^^^^≪ ^ log^^N^ − ^^^^^^^_#^"^^^For the purpose of illustration, the input memory cut word address bits in FIG.15 are bits [9:5].

[0158] The swizzle seed may be a bit field that includes one or more swizzle bits in the input address. In some embodiments, the one or more swizzle bits are one or more bits adjacent to the input memory cut select bits in the input address. In some embodiments, the width of the swizzle seed, i.e., the number of swizzle bits in the swizzle seed, may be indicated by a predetermined parameter, which may also be referred to as a swizzling parameter. The value of the swizzling parameter may be the width of the swizzle seed, such as 0, 1, 2, 3, and so on. In some embodiments, the swizzling parameter may be determined offline, e.g., before the neural network execution is started. The value of the swizzling parameter may be in a range from 0 to log^^N^. The range may include 0 and log^^N^.

[0159] FIG.15 shows the swizzle seeds for five different settings. The bits in the swizzle seeds are represented by boxes highlighted with a dot pattern. As shown in FIG.15, when the swizzling parameter has a value of 0, no swizzle bit is extracted from the input address. When the swizzling parameter has a value of 1, bit

[0010] is extracted from the input address as the single swizzle bit. When the swizzling parameter has a value of 2, bits [11:10] are extracted as swizzle bits. When the swizzling parameter has a value of 3, bits [12:10] are extracted as swizzle bits. When the swizzling parameter has a value of 4, bits [13:10] are extracted as swizzle bits. When the swizzling parameter has a value of 5, bits [14:10] are extracted as swizzle bits.

[0160] FIGS.16A-16F illustrate determination of swizzled memory cut select bits, in accordance with various embodiments. Each of FIGS.16A-16F shows a table that includes a row indicating input memory cut select bits, a row indicating extract swizzling bits, and a row indicating swizzling memory cut select bits determined from the input memory cut select bits and the swizzling bits. For the purpose of illustration and simplicity, the input memory cut select bits in each of FIGS.16A-16F are bits [9:5]. In other embodiments, the input memory cut select bits may be different bits in the input address.

[0161] In FIG.16A, the swizzling parameter is 0, meaning no swizzling bits are extracted or used. The swizzing memory cut select bits are the same as the input memory cut select bits. No address swizzing transformation is performed. Address swizzing transformation is performed in each of FIGS.16B-16F with different swizzling parameters. In FIG.16B, the swizzling parameter is 1, bit

[0010] is extracted as the swizzling bit and is accumulated with bit [9] in the input address to compute bit [9] in the swizzled address. Bits [8:5] in the input address are used as bits [8:5] in the swizzled address.

[0162] In FIG.16C, the swizzling parameter is 2. Bit

[0010] is extracted as a swizzling bit and is accumulated with bit [8] in the input address to compute bit [8] in the swizzled address. Also, bit

[0011] is extracted as another swizzling bit and is accumulated with bit [9] in the input address to compute bit [9] in the swizzled address. Bits [7:5] in the input address are used as bits [7:5] in the swizzled address.

[0163] In FIG.16D, the swizzling parameter is 3. Bit

[0010] is extracted as a swizzling bit and is accumulated with bit [7] in the input address to compute bit [7] in the swizzled address. Bit

[0011] is extracted as another swizzling bit and is accumulated with bit [8] in the input address to compute bit [8] in the swizzled address. Also, bit

[0012] is extracted as yet another swizzling bit and is accumulated with bit [9] in the input address to compute bit [9] in the swizzled address. Bits [6:5] in the input address are used as bits [6:5] in the swizzled address.

[0164] In FIG.16E, the swizzling parameter is 4. Bits [13:10] are extracted as the swizzling bits. Bit

[0013] is accumulated with bit [9] in the input address to compute bit [9] in the swizzled address. Bit

[0012] is accumulated with bit [8] in the input address to compute bit [8] in the swizzled address. Bit

[0011] is accumulated with bit [7] in the input address to compute bit [7] in the swizzled address. Bit

[0010] is accumulated with bit [6] in the input address to compute bit [6] in the swizzled address. Bit [5] in the input address is used as bit [5] in the swizzled address.

[0165] In FIG.16F, the swizzling parameter is 4. Bits [14:10] are extracted as the swizzling bits. Bit

[0014] is accumulated with bit [9] in the input address to compute bit [9] in the swizzled address. Bit

[0013] is accumulated with bit [8] in the input address to compute bit [8] in the swizzled address. Bit

[0012] is accumulated with bit [7] in the input address to compute bit [7] in the swizzled address. Bit

[0011] is accumulated with bit [6] in the input address to compute bit [6] in the swizzled address. Bit

[0010] is accumulated with bit [5] in the input address to compute bit [5] in the swizzled address. Example Memory Bank Scrubbing

[0166] FIG.17 illustrates memory scrubber handshake waves, in accordance with various embodiments. In some embodiments, a scrubber may be controlled through configuration space registers (CSRs) in a scrubber control module and follow the handshake shown in FIG. 17. For instance, a “Scrubber Go” register may control one or more scrubbing state machines in the memory. Scrubbing may be initiated on rising edge of this register. In embodiments where there are multiple compute tiles, this register may have separate bits for different tiles to allow independent control of scrubbing processes in these tiles. On a rising edge of the “Scrubber Go” register, scrubbers in some or even all memory banks may start scrubbing processes in parallel. During the scrubbing processes, incoming traffic (e.g., data transfer requests coming to the memory banks) may be staged in a request FIFO buffer. In some embodiments, one or more scrubbers may generate a stream of writes to a memory cut with data pattern provided through the “Scrubber MSET DATA” register shown in FIG.17. During scrubbing time, “Scrubber FSM Busy” signals in the memory banks may be asserted and OR combined version of it (“SCRUBBER BUSY ALL BANKS Ored”) may be monitored by the scrubber control module to observe when the scrubbing process is finished in all the memory banks. When that happens, the scrubber control module may de- assert the “Scrubber Go” register which may be observed by software through register polling or interrupt.

[0167] Additional registers in the scrubber control module may be available to throttle down rate of writes to limit peak power or to skip address range inside the memory from being scrubbed. Throttling can be achieved through setting window in which single write happens and can be set between single write per clock cycle and single write per eight clock cycles. Example Error Detection

[0168] FIG.18 illustrates an example process of error detection in a memory, in accordance with various embodiments. The process involves an input bridge module 1810, a memory bank 1820 in a target memory, a control space module 1830, a control space register set 1840, and an error reporting module 1850. The input bridge module 1810 may be an example of the input bridge module 910 in FIG.9. The memory bank 1820 may be an example of a memory bank 930 in FIG.9. The control space module 1830 may be an example of the control space module 1000 in FIG.10. The control space register set 1840 may be an example of the control space register set in FIG.10.

[0169] The input bridge module 1810 receives a request from an agent. The request may be a data transfer request for transferring data associated with one or more computations in a DNN. The data may be one or more control parameters for controlling the DNN execution. The data may alternatively be data processed or computed in the DNN. The request may be to read the data from or write the data to the target memory, e.g., one or more configuration registers or memory banks in the target memory. The input bridge module 1810 determines whether an address decoded from the request is in an address range of a control space. In embodiments where the input bridge module 1810 determines that the memory address of the request is in the control space address range, the input bridge module 1810 may forward the request to the control space module 1830. No additional address decoding or swizzling transformation may be performed by the input bridge module 1810.

[0170] The control space module 1830 may perform additional address decoding to determine whether the request has a valid control space address. For instance, the control space module 1830 may determine whether the request transaction targets any control space register in the control space register set 1840. In embodiments where the control space module 1830 determines that the request does not have any valid control space address, an error of the request falling outside address range is detected. In embodiments where the control space module 1830 determines that the request has a valid control space address, the control space register set 1840 may process the request, e.g., by reading data from one or more control space registers or writing data to one or more control space registers.

[0171] In embodiments where the input bridge module 1810 determines that the address decoded from the request is not in the control space address range, the input bridgemodule 1810 may perform additional error checks alongside address decoding and address swizzling transformation. The input bridge module 1810 determines whether an address decoded from the request is in a memory address space. The memory address space may include addresses in the memory banks of the target memory. In some embodiments, the memory address space may also include address in memory banks of a remote memory. The remote memory may be in a different tile from the target memory. The input bridge module 1810 may perform the error detection to ensure that the request targets an implemented memory bank or word within the memory bank. In embodiments where the input bridge module 1810 determines that the address decoded from the request is not in the memory address space, an error of the request falling outside address range is detected.

[0172] In embodiments where the input bridge module 1810 determines that the address decoded from the request is in the memory address space, the input bridge module 1810 may further determine whether the request facilitates a valid multicast / broadcast operation after determining that the request falls in the memory address space. In embodiments where the input bridge module 1810 determines that the request fails to facilitate a valid multicast / broadcast operation, an error of invalid operation is detected.

[0173] In embodiments where the input bridge module 1810 determines that the request facilitates a valid multicast / broadcast operation, the input bridge module 1810 may further determine whether the target tile is isolated. The target tile may be the tile where the target memory is located. The target tile may be a remote tile to the agent, i.e., a tile other than the tile where the agent is located. The input bridge module 1810 may ensure that the target tile is implemented and that the multicast / broadcast operation is supported. In embodiments where the input bridge module 1810 determines that the target tile is not isolated, the request may be forward to the memory bank in the tile. The memory bank may process the request. Data may be read from or written to the memory bank. In embodiments where the input bridge module 1810 determines that the target tile is isolated, an error relating to isolated tile access is detected.

[0174] In embodiments where an error is detected, the request may be directed into the error reporting module 1850. In some embodiments, the error reporting module 1850 may also receive information indicating a cause of the error. The error reporting module 1850 may report the error. In some embodiments, the error reporting module 1850 may send a response, which indicates the detection of the error, to the agent from which the request isreceived. The error reporting module 1850 may communicate with the control module of the DNN accelerator to pause the computation being performed by the agent or one or more other agents. In some embodiments, the error reporting module 1850 may send information indicating an error exit code, the request, or the agent’s identity into an event debug queue that can be accessed by the control module. Example Method of DNN Execution

[0175] FIG.19 is a flowchart of a method for executing a DNN, in accordance with various embodiments. The method 1900 may be performed by the local memory 340 in FIG.3. Although the method 1900 is described with reference to the flowchart illustrated in FIG. 19, many other methods for executing DNNs may alternatively be used. For example, the order of execution of the steps in FIG.19 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0176] The local memory 340 receives 1910 a request for transferring data associated with a computation in the neural network. The request comprises information indicating a first memory address that points to a first memory cut in a memory. For instance, the first memory address is a memory address in the first memory cut. The first memory cut comprising one or more memory banks.

[0177] The local memory 340 transforms 1920 the first memory address to a second memory address. The second memory address points to a second memory cut in the memory. For instance, the second memory address is a memory address in the second memory cut. The second memory cut comprises one or more memory banks. In some embodiments, the memory has a total number, N, of memory cuts. The local memory 340 transforms the first memory address to the second memory address by transforming the first memory address to the second memory address based on N. In some embodiments, the local memory 340 transforms the first memory address to the second memory address by transforming the first memory address to the second memory address based on a swizzling parameter. The swizzling parameter is determined before the computation in the neural network starts. In some embodiments, the swizzling parameter is determined by a compiler.

[0178] The local memory 340 transmits 1930 the data to the second memory cut. In some embodiments, the data is transmitted to the second memory cut through one or more interconnects. In some embodiments, the local memory 340 detects whether the requestcomprises any error by determining whether the first memory address or the second memory address is in a predetermined address range. In some embodiments, the predetermined address range corresponds to a memory address space that includes addresses of the memory and addresses of an additional memory. The memory and the additional memory are in separate tiles of a DNN accelerator that executes the DNN. In some embodiments, the local memory 340 detects whether the request is associated with a valid data broadcast operation for broadcasting data to the separate tiles. In some embodiments, the computation in the DNN is suspended in response to detection of an error in the request.

[0179] In some embodiments, the local memory 340 performs a memory scrubbing process on data stored in the one or more memory banks of the first memory cut or the second memory cut. In some embodiments, the memory scrubbing process is performed by a state memory in the first memory cut or the second memory cut. In some embodiments, the memory has multiple state machines embedded in different memory cuts or different memory banks. These state machines may perform memory scrubbing processes in parallel. In some embodiments, these state machines are controlled by a configuration space register. In some embodiments, the configuration space register may control multiple local memories, such as local memories of in different DPUs. In some embodiments, a request for transferring data to or from a memory bank is received during the memory scrubbing process. The request is staged in a first in, first out queue associated with the memory bank. Example Computing Device

[0180] FIG.20 is a block diagram of an example computing device 2000, in accordance with various embodiments. In some embodiments, the computing device 2000 can be used as at least part of the DNN system 300. A number of components are illustrated in FIG.20 as included in the computing device 2000, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 2000 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 2000 may not include one or more of the components illustrated in FIG.20, but the computing device 2000 may include interface circuitry for coupling to the one or more components. For example, the computing device 2000 may not include a display device2006, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 2006 may be coupled. In another set of examples, the computing device 2000 may not include an audio input device 2018 or an audio output device 2008 but may include audio input or output device interface circuitry to which an audio input device 2018 or audio output device 2008 may be coupled.

[0181] The computing device 2000 may include a processing device 2002 (e.g., one or more processing devices). The processing device 2002 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 2000 may include a memory 2004, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 2004 may include memory that shares a die with the processing device 2002. In some embodiments, the memory 2004 includes one or more non-transitory computer- readable media storing instructions executable to perform operations for DNN execution (e.g., the method 1900 described in conjunction with FIG.19) or some operations performed by one or more components of the DNN system 300 (e.g., the DNN module 301, the local memory 340, etc.). The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 2002.

[0182] In some embodiments, the computing device 2000 may include a communication chip 2012 (e.g., one or more communication chips). For example, the communication chip 2012 may be configured for managing wireless communications for the transfer of data to and from the computing device 2000. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0183] The communication chip 2012 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project(also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 2012 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 2012 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 2012 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 2012 may operate in accordance with other wireless protocols in other embodiments. The computing device 2000 may include an antenna 2022 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0184] In some embodiments, the communication chip 2012 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 2012 may include multiple communication chips. For instance, a first communication chip 2012 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 2012 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 2012 may be dedicated to wireless communications, and a second communication chip 2012 may be dedicated to wired communications.

[0185] The computing device 2000 may include battery / power circuitry 2014. The battery / power circuitry 2014 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 2000 to an energy source separate from the computing device 2000 (e.g., AC line power).

[0186] The computing device 2000 may include a display device 2006 (or corresponding interface circuitry, as discussed above). The display device 2006 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0187] The computing device 2000 may include an audio output device 2008 (or corresponding interface circuitry, as discussed above). The audio output device 2008 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0188] The computing device 2000 may include an audio input device 2018 (or corresponding interface circuitry, as discussed above). The audio input device 2018 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0189] The computing device 2000 may include a GPS device 2016 (or corresponding interface circuitry, as discussed above). The GPS device 2016 may be in communication with a satellite-based system and may receive a location of the computing device 2000, as known in the art.

[0190] The computing device 2000 may include another output device 2010 (or corresponding interface circuitry, as discussed above). Examples of the other output device 2010 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0191] The computing device 2000 may include another input device 2020 (or corresponding interface circuitry, as discussed above). Examples of the other input device 2020 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0192] The computing device 2000 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), adesktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 2000 may be any other electronic device that processes data. Select Examples

[0193] The following paragraphs provide various examples of the embodiments disclosed herein.

[0194] Example 1 provides a memory, including a plurality of memory cuts, a memory cut including one or more memory banks in the memory; a data port to receive a request for transferring data associated with a computation in a neural network, the request including information indicating a first memory address that points to a first memory cut of the plurality of memory cuts; and an input bridge module to transform the first memory address to a second memory address, the second memory address pointing to a second memory cut of the plurality of memory cuts, in which the data associated with the computation in the neural network is to be read from or written to the second memory cut.

[0195] Example 2 provides the memory of example 1, in which the plurality of memory cuts has a total number, N, of memory cuts, and the input bridge module is to transform the first memory address to the second memory address based on N.

[0196] Example 3 provides the memory of example 1 or 2, further including an interconnect between the input bridge module and the second memory cut, in which the data associated with the computation in the neural network is to be read from or written to the second memory cut through the interconnect.

[0197] Example 4 provides the memory of any one of examples 1-3, in which the input bridge module is to transform the first memory address to the second memory address based on a configuration parameter, and the configuration parameter is determined before the computation in the neural network starts.

[0198] Example 5 provides the memory of example 4, further including one or more other data ports; and one or more other input bridge modules to transform one or more other memory addresses based on the configuration parameter, in which each of the one or more other input bridge modules is coupled to a different one of the one or more other data ports.

[0199] Example 6 provides the memory of any one of examples 1-5, in which a memory bank includes a state machine, and the state machine is to start a memory scrubbing process on data stored in the memory bank.

[0200] Example 7 provides the memory of example 6, in which another memory bank includes another state machine, and the another state machine to start another memory scrubbing process in parallel with the memory scrubbing process started by the state machine.

[0201] Example 8 provides the memory of example 7, in which the state machine and the another state machine are both controlled by a configuration space register.

[0202] Example 9 provides the memory of example 8, in which the memory is in a first tile of a neural network accelerator that executes the neural network, another memory is in a second tile of the neural network accelerator, and the configuration space register includes different bits for the memory and the other memory.

[0203] Example 10 provides the memory of any one of examples 6-9, in which a request for transferring data to or from the memory bank is received during the memory scrubbing process, and the request is staged in a first in, first out queue associated with the memory bank.

[0204] Example 11 provides the memory of any one of examples 1-10, in which the input bridge module is further to detect whether the request includes any error.

[0205] Example 12 provides the memory of example 11, in which the input bridge module is further to detect whether the request includes any error by determining whether the first memory address or the second memory address is in a predetermined address range.

[0206] Example 13 provides the memory of example 12, in which the predetermined address range corresponds to a memory address space that includes addresses of the memory and addresses of an additional memory, and the memory and the additional memory are in separate tiles of a neural network accelerator that executes the neural network.

[0207] Example 14 provides the memory of example 13, in which the input bridge module is further to detect whether the request is associated with a valid data broadcast operation for broadcasting data to the separate tiles.

[0208] Example 15 provides the memory of any one of examples 11-14, in which the computation in the neural network is suspended in response to the input bridge module detecting an error in the request.

[0209] Example 16 provides an apparatus, including one or more PEs to perform a computation in a neural network; and a memory, the memory including a plurality of memory cuts, a memory cut including one or more memory banks in the memory, a data port to receive a request for transferring data associated with the computation in the neural network, the request including information indicating a first memory address that points to a first memory cut of the plurality of memory cuts, and an input bridge module to transform the first memory address to a second memory address, the second memory address pointing to a second memory cut of the plurality of memory cuts, in which the data associated with the computation in the neural network is to be read from or written to the second memory cut.

[0210] Example 17 provides the apparatus of example 16, in which the apparatus is coupled to a neural network compiler, the input bridge module is to transform the first memory address to the second memory address based on a configuration parameter, and the configuration parameter is generated by the neural network compiler before the computation in the neural network starts.

[0211] Example 18 provides the apparatus of example 16 or 17, in which a memory bank of the first memory cut or the second memory cut includes a state machine, and the state machine is to start a memory scrubbing process on data stored in the memory bank based on a configuration space register.

[0212] Example 19 provides the apparatus of example 18, in which the one or more PEs and the memory are in a first tile of the apparatus, the apparatus further includes a second tile, and the configuration space register includes different bits for the memory and a memory in the second tile.

[0213] Example 20 provides the apparatus of any one of examples 16-19, in which the input bridge module is further to detect whether the request includes any error by determining whether the first memory address or the second memory address is in a predetermined address range.

[0214] Example 21 provides a method of executing a neural network, including receiving, by a memory, a request for transferring data associated with a computation in the neuralnetwork, the request including information indicating a first memory address that points to a first memory cut in a memory, the first memory cut including one or more memory banks; transforming, by the memory, the first memory address to a second memory address, the second memory address pointing to a second memory cut in the memory, the second memory cut including one or more memory banks; and transmitting, by the memory, the data to the second memory cut.

[0215] Example 22 provides the method of example 21, in which the memory has a total number, N, of memory cuts, and transforming the first memory address to the second memory address includes transforming the first memory address to the second memory address based on N.

[0216] Example 23 provides the method of example 21 or 22, in which transforming the first memory address to the second memory address includes transforming the first memory address to the second memory address based on a configuration parameter, and the configuration parameter is determined before the computation in the neural network starts.

[0217] Example 24 provides the method of any one of examples 21-23, further including performing, by the memory, a memory scrubbing process on data stored in the one or more memory banks of the first memory cut or the second memory cut.

[0218] Example 25 provides the method of any one of examples 21-24, further including detecting, by the memory, whether the request includes any error by determining whether the first memory address or the second memory address is in a predetermined address range.

[0219] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims 1. A computer memory, comprising: a plurality of memory cuts, a memory cut comprising one or more memory banks; a data port to receive a request for transferring data associated with a computation in a neural network, the request comprising information indicating a first memory address that points to a first memory cut of the plurality of memory cuts; and an input bridge module to transform the first memory address to a second memory address, the second memory address pointing to a second memory cut of the plurality of memory cuts, wherein the data associated with the computation in the neural network is to be read from or written to the second memory cut.

2. The computer memory of claim 1, wherein the plurality of memory cuts has a total number, N, of memory cuts, and the input bridge module is to transform the first memory address to the second memory address based on N.

3. The computer memory of claim 1 or 2, further comprising: an interconnect between the input bridge module and the second memory cut, wherein the data associated with the computation in the neural network is to be read from or written to the second memory cut through the interconnect.

4. The memory of any one of claims 1-3, wherein the input bridge module is to transform the first memory address to the second memory address based on a configuration parameter, and the configuration parameter is determined before the computation in the neural network starts.

5. The computer memory of claim 4, further comprising: one or more other data ports; and one or more other input bridge modules to transform one or more other memory addresses based on the configuration parameter,wherein each of the one or more other input bridge modules is coupled to a different one of the one or more other data ports.

6. The memory of any one of claims 1-5, wherein a memory bank comprises a state machine, and the state machine is to start a memory scrubbing process on data stored in the memory bank.

7. The memory of claim 6, wherein another memory bank comprises another state machine, and the another state machine to start another memory scrubbing process in parallel with the memory scrubbing process started by the state machine.

8. The computer memory of claim 7, wherein the state machine and the another state machine are both controlled by a configuration space register.

9. The computer memory of claim 8, wherein the memory is in a first tile of a neural network accelerator that executes the neural network, another memory is in a second tile of the neural network accelerator, and the configuration space register has different bits for the memory and the another memory.

10. The memory of any one of claims 6-9, wherein a request for transferring data to or from the memory bank is received during the memory scrubbing process, and the request is staged in a first in, first out queue associated with the memory bank.

11. The memory of any one of claims 1-10, wherein the input bridge module is further to detect whether the request comprises any error.

12. The computer memory of claim 11, wherein the input bridge module is further to detect whether the request comprises any error by determining whether the first memory address or the second memory address is in a predetermined address range.

13. The computer memory of claim 12, wherein the predetermined address range corresponds to a memory address space that includes addresses of the memory andaddresses of an additional memory, and the memory and the additional memory are in separate tiles of a neural network accelerator that executes the neural network.

14. The computer memory of claim 13, wherein the input bridge module is further to detect whether the request is associated with a valid data broadcast operation for broadcasting data to the separate tiles.

15. The memory of any one of claims 11-14, wherein the computation in the neural network is suspended in response to the input bridge module detecting an error in the request.

16. An apparatus, comprising: one or more processing elements to perform a computation in a neural network; and a memory, the memory comprising: a plurality of memory cuts, a memory cut comprising one or more memory banks in the memory, a data port to receive a request for transferring data associated with the computation in the neural network, the request comprising information indicating a first memory address that points to a first memory cut of the plurality of memory cuts, and an input bridge module to transform the first memory address to a second memory address, the second memory address pointing to a second memory cut of the plurality of memory cuts, wherein the data associated with the computation in the neural network is to be read from or written to the second memory cut.

17. The apparatus of claim 16, wherein the apparatus is coupled to a neural network compiler, the input bridge module is to transform the first memory address to the second memory address based on a configuration parameter, and the configuration parameter is generated by the neural network compiler before the computation in the neural network starts.

18. The apparatus of claim 16 or 17, wherein a memory bank of the first memory cut or the second memory cut comprises a state machine, and the state machine is to start a memory scrubbing process on data stored in the memory bank based on a configuration space register.

19. The apparatus of claim 18, wherein the one or more processing elements and the memory are in a first tile of the apparatus, the apparatus further comprises a second tile, and the configuration space register has different bits for the memory and a memory in the second tile.

20. The apparatus of any one of claims 16-19, wherein the input bridge module is further to detect whether the request comprises any error by determining whether the first memory address or the second memory address is in a predetermined address range.

21. A method of executing a neural network, comprising: receiving, by a memory, a request for transferring data associated with a computation in the neural network, the request comprising information indicating a first memory address that points to a first memory cut in a memory, the first memory cut comprising one or more memory banks; transforming, by the memory, the first memory address to a second memory address, the second memory address pointing to a second memory cut in the memory, the second memory cut comprising one or more memory banks; and transmitting, by the memory, the data to the second memory cut.

22. The method of claim 21, wherein the memory has a total number, N, of memory cuts, and transforming the first memory address to the second memory address comprises transforming the first memory address to the second memory address based on N.

23. The method of claim 21 or 22, wherein transforming the first memory address to the second memory address comprises transforming the first memory address to the second memory address based on a configuration parameter, and the configuration parameter is determined before the computation in the neural network starts.

24. The method of any one of claims 21-23, further comprising: performing, by the memory, a memory scrubbing process on data stored in the one or more memory banks of the first memory cut or the second memory cut.

25. The method of any one of claims 21-24, further comprising: detecting, by the memory, whether the request comprises any error by determining whether the first memory address or the second memory address is in a predetermined address range.

Images