Neural network accelerator with self-clock gating

Abstract

Operations of a digital signal processor (DSP) may be synchronized with a clock signal provided by a DSP clock coupled with a self-clock gate that can gate the DSP clock from providing the clock signal to deactivate the DSP. The self-clock gate may be controlled by a gate controller, which is communicatively coupled with the DSP as well as one or more event detectors. Each event detector may send a signal to the gate controller after it detects a trigger event. The trigger event may be an event associated with a hardware barrier blocking the task from being performed, a first-in-first-out (FIFO) unit dispatching the task, memory, configuration register, etc. The gate controller may use the signals from the event detectors to determine whether to enable or disable the self-clock gate. When the self-clock gate is enabled, execution of instructions by the DSP is prevented to save power.

Description

NEURAL NETWORK ACCELERATOR WITH SELF-CLOCK GATINGCross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 662,710, filed June 21, 2024, and entitled "SELF-CLOCKING GATING HARDWARE OPTIMIZATION TO IMPROVE DYNAMIC POWER OF NEURAL NETWORK ACCELERATOR," which is incorporated by reference in its entirety.Technical Field

[0002] This disclosure relates generally to neural network (also referred to as "deep neural network" or "DNN"), and more specifically, DNN accelerators with self-clock gating, such as self-clock gating for digital signal processors (DSPs).Background

[0003] DNNs are used extensively for a variety of artificial intelligence applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write.Therefore, techniques to improve efficiency of DNNs are needed.Brief Description of the Drawings

[0004] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0005] FIG. 1 illustrates an example transformer model, in accordance with various embodiments.

[0006] FIG. 2 illustrates computations in an example attention layer, in accordance with various embodiments.

[0007] FIG. 3 illustrates an example convolution, in accordance with various embodiments.

[0008] FIG. 4 illustrates partition of a workload of executing a DNN layer, in accordance with various embodiments.

[0009] FIG. 5 is a block diagram of a DNN system, in accordance with various embodiments.

[0010] FIG. 6 illustrates an example DNN execution sequence, in accordance with various embodiments.

[0011] FIG. 7 illustrates clock gating timing during DNN executions, in accordance with various embodiments.

[0012] FIG. 8 illustrates self-clock gating timing during DNN executions, in accordance with various embodiments.

[0013] FIG. 9 illustrates an example circuitry for DSP self-clock gating, in accordance with various embodiments.

[0014] FIG. 10 illustrates a flow graph of an example self-clock gating handshake of a DSP, in accordance with various embodiments.

[0015] FIG. 11 illustrates timing of various signals involved in self-clock gating of a DSP, in accordance with various embodiments.

[0016] FIG. 12 illustrates an example sparse cell, in accordance with various embodiments.

[0017] FIG. 13 is a flowchart of a method 1300 for DSP self-clock gating, in accordance with various embodiments.

[0018] FIG. 14 is a block diagram of an example computing device, in accordance with various embodiments.Detailed Description

[0019] The last decade has witnessed a rapid rise in artificial intelligence (Al) based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more deep learning operations (also referred to as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on.

[0020] Input or output data of deep learning operations may be arranged in data structures called tensors. A tensor is a data structure having multiple elements across one or moredimensions. Examples of tensors include vector (which is one-dimensional (ID) tensor), matrix (which is two-dimensional (2D) tensor), three-dimensional (3D) tensors, fourdimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along the axis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. Taking a convolutional layer for example, the input tensors include an activation tensor (also referred to as "input feature map (IFM)" or "input activation tensor") including one or more activations (also referred to as "input elements") and a weight tensor. The weight tensor may be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a group of filters (a 4D weight tensor). A convolution may be performed on the input activation tensor and weight tensor to compute an output activation tensor in the convolutional layer.

[0021] The significant improvements in DNN model size and accuracy coupled with the rapid increase in computing power of execution platforms have led to the adoption of DNN applications even within resource constrained mobile and edge devices that have limited energy availability. In edge and client devices, efficient use of limited memory resources and reducing memory traffic can be critical for achieving performance and power goals for a given memory budget.

[0022] DNN models can be executed, e.g., for training or inference, by DNN accelerators. A DNN accelerator may include an array of processing elements (PEs)), array of DSPs, on-chip- memory (e.g., static random-access memory (SRAM)), direct memory access (DMA) module, control module, and microcontroller (uC) for scheduling and accelerator management. The PEs may be the computational units responsible for performing the core operations involved in DNN executions, specifically designed to efficiently execute operations such as matrix multiplication, convolution, activation function and pooling operation. The DSPs can also provide computational capacity. Because of the general capability of DSPs, layers which are not supported by PEs can be executed on them. The on-chip memory may provide a low- power, low-latency, and high bandwidth storage capacity. The DMA module can facilitate pipelined movement of data between off-chip and on-chip memory. The control module can provide hardware-based task synchronization.

[0023] Clock gating is used in synchronous digital circuits to reduce power consumption. Clock gating can reduce dynamic power dissipation by selectively disabling clock signals to inactive parts of the design, minimizing unnecessary switching activity, glitching power and reducing the capacitive load on the clock network. A DSP clock can be fully controllable by higher level firmware and once enabled during start of network. It usually remains open for the entire DNN execution time. One reason is because the DNN execution is usually dictated by producer-consumer dependencies between computational tasks performed by compute engines (e.g., DSP array, PEs Array, etc.) but the firmware typically has limited visibility into the execution schedule of the compute engines. Furthermore, any interaction with firmware while neural network is executing can have a significant impact on the performance and utilization of compute resources. It is because the time required for firmware to act is usually remarkably high and during that period, the agents of the compute engines (e.g., PEs, DSP cores, etc.) are idle and not utilized.

[0024] Clock gating is usually utilized to reduce power consumed by a DSP when a reduced set of functionalities of the DSP is used. It can be effective to save power consumption. However, in the case of management kernels, trunk clock tree and clocks connected to components essential for instruction (such as fetching, decoding, execution and external signal observing) still need to be enabled. As described above, the solution of using high level firmware to control clock has major drawback as the clock cannot be disabled when DSP kernels are waiting for external dependencies from other blocks. The reason for it is that the firmware usually has high latency to react to changing producer-consumer dependencies which in the end can lead to reduced compute resource utilization and drop in effective performance of the DNN accelerator. Feature level internal to DSP architecture clock gating is effective to a limited degree as internal components essential to DSP execution still need to be enabled.

[0025] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing DNN accelerators with DSPs having selfclock gating features. For instance, a DSP may be associated with a self-clock gate that can provide clock gating in addition to the clock gating provided by the firmware. The self-clock gate may be controlled based on visibility into the execution schedule of the DSP, such as the schedule of the DSP performing computational tasks with producer-consumer dependencies.

[0026] In various embodiments of the present disclosure, a DNN accelerator may include a circuit that facilitates DSP self-clock gating. A DSP may have a clock that provides a clock signal. Operations of the DSP, such as execution of instruction may be synchronized with the clock signal. For instance, when the clock signal is on, the DSP is active and can execute instruction; but when the clock signal is off, the DSP becomes inactive and stops executing instructions. The DSP clock may be coupled with a self-clock gate that can gate the DSP clock from providing the clock signal to deactivate the DSP when the DSP has no computational task to perform, e.g., when the next computational task for the DSP is being blocked by a barrier due to producer-consumer dependency. The self-clock gate may be controlled by a gate controller, which is communicatively coupled with the DSP as well as one or more event detectors. Each event detector may send a signal to the gate controller after it detects a trigger event. The trigger event may be an event associated with other hardware resources in the DNN accelerator, such as hardware barrier used to block computational task from being performed before the completion of the corresponding producer task is complete, task first-in-first-out (FIFO) unit that dispatches computational tasks to DSPs, memory, cache configuration registers, and so on. The gate controller may use the signals from the event detectors to determine whether to enable or disable the self-clock gate.

[0027] In an example, a firmware may enable the DSP clock, which then generates the clock signal. After the clock signal is started, the DSP is active and may execute instructions. The DSP may then send a gate enable signal to the gate controller. In response to receiving the gate enable signal, the self-clock gate may determine whether it is safe to enable the selfclock gate. For instance, the gate controller may determine whether there is any external request to be finished by the DSP. The self-clock gate may determine whether there is any pending memory request to be finished by the DSP or whether an interface of a configuration space register associated with the DSP is active. When there is no pending memory request and the interface is inactive, the gate controller may determine that it is safe to enable the self-clock gate. The gate controller then enables the self-clock gate and commands the self-clock gate to gate the DSP clock. The gated DSP clock stops the clock signal and the DSP becomes inactive. The DSP clock may be awakened by the gate controller when a trigger event is detected. For instance, the gate controller may receive a wake signal from an event detector indicating that a trigger even has occurred. The trigger event may be that the barrier is lifted or that a task is dispatched to the DSP. The event detector may beprogrammed by the DSP in a way that the event detector sends out the wake signal when an event meeting one or more criteria is detected. After receiving the wake signal, the gate controller may send a clock awaken signal to the DSP. The DSP clock may be activated in response to the clock awaken signal and resume the clock signal. After the clock signal is resumed, the DSP may start executing instructions again. The DSP may clear the gate enable signal in response to the awaken signal. The clock gate may be disabled by writing to a GPO (global port outputs) pin of the DSP. This process may be a cycle of handshake. The DSP may repeat this cycle during the execution of a DNN. After the execution is complete (or after all the tasks assigned to the DSP are complete), the firmware may stop the DSP clock, after which the DSP may stop executing instructions.

[0028] With the self-clock gating mechanism, the power consumption of DSP can be minimized or even eliminated while DSPs are waiting inside management kernel for new task to be enqueued into task FIFO or barrier dependency to be met. Therefore, the power consumption of DNN accelerators for DNN inference can be significantly reduced. Compared with currently available solutions, the present disclosure provides a more advantageous approach that can lead to power efficient and high-performance DNN inference. This advantage can be very important for many Al applications, especially Al applications run on edge devices.

[0029] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0030] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0031] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.However, the order of description should not be construed as to imply that these operationsare necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0032] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0033] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0034] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0035] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0036] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expresslylisted or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0037] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0038] FIG. 1 illustrates an example transformer model 100, in accordance with various embodiments. The transformer model 100 may transform input sequences into output sequences. In some embodiments, the transformer model 100 is a DNN that can learn context and meaning by tracking relationships in sequential data, such as sequential words in a sentence, sequential audio signals, sequential images, and so on. In an example, the transformer model 100 may be at least part of a large language model (LLM). The transformer model 100 includes an encoder block 110, a decoder block 120, and a head block 130. In other embodiment, different or additional components may be included in the transformer model 100. Further, functionality attributed to a component of the transformer model 100 may be accomplished by a different component included in the transformer model 100 or a different model or module.

[0039] The encoder block 110 receives input sequences and generates matrix representations of the input sequences. In the embodiments of FIG. 1, the encoder block 110 receives an input 101 and generates an encoder output 102. The input 101 may be an input prompt. In some embodiments, the input 101 may include one or more input tokens, such as words, phrases, sentences, images, audio signals, other types of input tokens, or some combination thereof. In an example, the input 101 may include a prompt received from a user of the transformer model 100. The prompt may include a question or request made by the user. A word in the prompt may be an input token. The encoder output 102 may include one or more vectors that are contextualized representations of the input 101. Each vector in the encoder output 102 may represent a token in the input 101 with contextual understanding.

[0040] The encoder block 110 includes an embedding layer 113, a positional encoding layer 115, and a plurality of layers 140 (individually referred to as "layer 140"). In other embodiments, the encoder block 110 may have different, fewer, or more components. Also, the arrangement of the components in the encoder block 110 may be different from thearrangement shown in FIG. 1. For the purpose of illustration, the encoder block 110 has N layers in FIG. 1, where N is an integer. Each layer 140 may include one or more neural network operations. The layers 140 may transform a sequence of embeddings into a representation that encapsulates the learned information from the input 101. Different layers 140 may have different internal parameters, e.g., different weights, bias, or other types of internal parameters. In some embodiments, the layers 140 have identical components. The components in a layer 140 may be layers and may also be referred to as sub-layers of the layer 140. As shown in FIG. 1, a layer 140 includes four sub-layers: a multihead attention (MHA) layer 141, an add & norm layer 142, a feed forward layer 143, and another add & norm layer 144.

[0041] The decoder block 120 iteratively generates outputs 103 using encoded representations generated by the encoder block 110. The decoder block 120 includes an embedding layer 123, a positional encoding layer 125, and a plurality of layers 150 (individually referred to as "layer 150"). For the purpose of illustration, the decoder block 120 has N layers in FIG. 1, where N is an integer. In the embodiments of FIG. 1, the number of layers 150 in the decoder block 120 is the same as the number of layers 140 in the encoder block 110. In other embodiments, the number of layers 150 in the decoder block 120 may be different from the number of layers 140 in the encoder block 110. Each layer 150 may include one or more neural network operations. Different layers 150 may have different internal parameters. In some embodiments, the layers 150 may have identical components. The components in a layer 150 may be layers and may also be referred to as sub-layers of the layer 150. As shown in FIG. 1, a layer 150 includes six sub-layers: an MHA layer 151, an add & norm layer 152, an encoder-decoder attention layer 153, another add & norm layer 154, a feed forward layer 155, and another add & norm layer 156.

[0042] In some embodiments, a sequence of inference stages is performed in the decoder block 120 using encoder outputs, e.g., the encoder output 102. A matrix may be predicted through each inference stage. The outputs 103 may include a plurality of matrices. Each matrix may be further processed in the head block 130 to predict a token. The plurality of matrices may be used to predict a sequence of tokens. For the first inference stage, the decoder block 120 may receive one or more start tokens as input tokens and compute a first matrix from the input tokens and the output of the encoder block 110. The first matrix may be used by the head block 130 to predict a first token. The predicted token may be used as anew input token, in addition to the start token(s), in the second inference stage. Similarly, a second token may be predicted through the second inference stage and may be used in the third inference stage. This iteration may continue till all the inference stages are complete.

[0043] The head block 130 receives the output of the decoder block 120 and processes it in a linear layer 143 and a SoftMax layer 145. A linear operation may be performed on the output of the decoder block 120 in the linear layer 143. The linear operation may include a multiplication of the output of the decoder block 120 with a weight matrix. The output of the linear layer 143 may be a vector. In some embodiments, the head block 130 may function as a classifier. The number of data elements in the vector computed in the linear layer 143 may depend on the number of classes involved. In an example where there are M classes, where M is an integer, the vector computed in the linear layer 143 may have M data elements representing the prediction for the M classes, respectively.

[0044] The output of the linear layer 143 may be input into the SoftMax layer 145. A SoftMax function may be applied on the output of the linear layer 143 to compute probability scores. A probability score may have a value in the range from 0 to 1. In some embodiments, a probability value is computed for each data element in the vector computed in the linear layer 143. The highest one of the probability scores may be the key. The corresponding index of the key may point to the token that the transformer model 100 predicts as the next in the sequence. The final output of the transformer model 100 may be the sequence of predicted tokens. In some embodiments, the head block 130 may be a language modeling head.

[0045] An embedding layer (e.g., the embedding layer 113 or the embedding layer 123) converts an input of the embedding layer (e.g., the input 101 or the outputs 103) into one or more embeddings. An embedding may be a vector, which is also referred to as an embedding vector or a vector embedding. The vector embedding may include a sequence of data elements. In some embodiments, the embedding layer 113 may generate a plurality of embeddings, each of which may be converted from a different input token in the input 101. The embeddings may capture the semantic meaning of the tokens in the input 101. The embeddings may be numerical representations that capture the relationships or meanings of words, phrases, or other data types. In an example where the input 101 is a prompt including a sequence of words, the embedding layer 113 may generate an embedding from each word in the input 101. The embedding layer 123 in the decoder block 120 maygenerate a plurality of embeddings from tokens received by the decoder block 120 in a similar manner as the embedding layer 113.

[0046] A positional encoding layer (e.g., the positional encoding layer 115 or the positional encoding layer 125) performs positional encoding on embeddings generated in the corresponding embedding layer. In some embodiments, the positional encoding layer may apply one or more positional encoding vectors (e.g., a positional encoding vector 104 or positional encoding vector 105) on vector embeddings from the corresponding embedding layer to generate new vector embeddings that represents the embeddings with positional context. The positional encoding vector may encode information about the position of the embedding in a sequence of embeddings. In some embodiments, the positional encoding layer performs an addition operation on a positional encoding vector and a vector embedding. The addition operation may be elementwise addition. The positional encoding layer may output an embedding matrix that includes the vector embeddings computed in the positional encoding layer.

[0047] An MHA layer (e.g., the MHA layer 141, the MHA layer 151, or the MHA layer 153) may implement a multi-head attention mechanism, which may be a multi-head selfattention mechanism or a multi-head cross-attention mechanism. In some embodiments, the MHA layer 141 or the MHA layer 151 may implement a self-attention mechanism. For self-attention, the queries, keys, and values may come from the same place. For instance, for the MHA layer 141, the queries, keys, and values may all come from the positional encoding layer 115. For the MHA layer 151, the queries, keys, and values may all come from the positional encoding layer 125. The self-attention mechanism may enable the transformer model 100 to relate each token with other tokens. The MHA layer may compute attention scores from embeddings generated in the corresponding positional encoding layer. In some embodiments, the MHA layer may receive one or more queries, one or more keys, and one or more values. In some embodiments, the MHA layer has a number of heads that receive different linearly projected versions of the queries, keys, and values and produce outputs in parallel that are then used to generate the final result.

[0048] In some embodiments, the queries, keys, and values input into the MHA layer 141 may be computed from vector embeddings generated by the positional encoding layer 115. The queries, keys, and values input into the MHA layer 151 may be computed from vector embeddings generated by the positional encoding layer 125. A query, key, or value may be avector the represents a token in a sequence. In some embodiments, a query matrix Q E BWx / imay bgconnpUtec| by multiply an embedding matrix X E BWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WqE Bdx / l, where d is the dimension of a vector embedding, N is the number of vector embeddings in the embedding matrix, and h is the number of attention heads. Each row in the query matrix may be a query. A key matrix K E BWx / 1may be computed by multiple an embedding matrix X E BWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WkE Bdxh. Each row in the key matrix may be a key. A value matrix V E BWx / 1may be computed by multiple an embedding matrix X E BWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WvE Bdx / l. Each row in the value matrix may be a value.

[0049] In some embodiments, the MHA layer 151 may implement masked multi-head selfattention. The MHA layer 151 may prevent positions from attending to subsequent positions. For instance, each token in the sequence may not be influenced by future tokens. This masking can ensure that the predictions of a particular position can depend on known outputs at positions before it and not depend on unknown outputs at positions after it.

[0050] In some embodiments, the MHA layer 153 may implement a cross-attention mechanism, such as encoder-decoder cross-attention. The MHA layer 153 may use outputs from the previous layer (i.e., the add & norm layer 152) as queries and use outputs from the encoder block 110 as keys and values. The cross-attention can align the encoder's input with the decoder's, empowering the decoder block 120 to identify and emphasize the most relevant parts of the encoder's input.

[0051] An add & norm layer in the transformer model 100, such as the add & norm layer 142, 144, 152, 154, and 156, has an addition operation followed by a layer normalization operation. The addition operation may be an addition of the output of the preceding layer and the input of the preceding layer. The preceding layer is a layer that is arranged right before the add & norm layer. For example, the preceding layer of the add & norm layer 142 is the MHA layer 141. As another example, the preceding layer of the add & norm layer 154 is the encoder-decoder attention layer 153.

[0052] Then the layer normalization operation is applied on the result of the addition operation, which may be denoted as LayerNorm(x + sublay er (x)'), where LayerNormdenotes layer normalization, x is the input of the preceding layer, and sublayer(x) denotes the output of the preceding layer. In some embodiments, the layer normalization operation may include a sequence of computations. In an example, the layer normalization operation may include a mean computation, which may be denoted as !Jxy, where Axyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and .xydenotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The layer normalization operation may convert ixyto a 3D tensor .xyz, e.g., by replicating every data element overz output points.

[0053] The layer normalization operation may also include an elementwise subtraction, which may be denoted as Dxyz= Axyz— nxyz. The layer normalization operation may further include a variance computation denoted asD2xyzand a division computation denoted as Mxy=1Mxymay be a 2D tensor. The layernormalization operation may also convert Mxyto a 3D tensor Mxyz, e.g., by replicating every data element over z output points. Further, the layer normalization operation may have an element multiplication denoted as A'xyz=xThe layer normalization operation may further compute yz= A"xyzX yz. LNxyzmay be the output of the layernormalization operation.

[0054] A feed forward layer (e.g., the feed forward layer 143 and the feed forward layer 155) may be a position-wise fully-connected feed forward network. In an example, the feed forward layer may include two linear layers with an activation function in between. An example of the activation function is Rectified Linear Unit (ReLU).

[0055] FIG. 2 illustrates computations in an attention layer without KV caching, in accordance with various embodiments. The attention layer may be an MHA layer, such as a multi-head self-attention layer. In some embodiments, the attention layer is in a decoder ofa transformer. The computations in the attention layer may include multiplication of a query matrix 210 and a key matrix 220, which results in an attention weight matrix 230. In some embodiments, the attention layer may be a masked attention layer. One or more elements in the attention weight matrix 230 may be masked. For instance, the elements highlighted with a dotted pattern in FIG. 2 may be masked. The computations in the attention layer also includes multiplication of the attention weight matrix 230 and a value matrix 240, which results in an output matrix 250 encoding new tokens. In other embodiments, the computations in the attention layer may include other computations, such as computations with a scaling function, SoftMax function, and so on. For the purpose of simplicity and illustration, these computations are not shown in FIG. 2.

[0056] Each of the query matrix 210, key matrix 220, and value matrix 240 may include a vector for each of the tokens in the input sequence. For the purpose of illustration and simplicity, the input sequence has four tokens: tokens 1-4. In the embodiments of FIG. 2, as the decoder does not implement KV caching, computations on all the key tokens in the key matrix 220 and all the value tokens in the value matrix 240 need to be conducted. Some of the computations have already been conducted in the previous inference stage, e.g., computations on the key tokens 1-3 and computations on the value tokens 1-3. The duplication of these computations can be a waste of computational resources, such as power, time, and so on.

[0057] FIG. 3 illustrates an example convolution, in accordance with various embodiments. The convolution may be a neural network operation in a convolutional layer of a DNN. The DNN may be a convolutional neural network. The convolution can be executed on an activation tensor 310 and filters 320 (individually referred to as "filter 320"). The filters may constitute a weight tensor of the convolution. The result of the convolution is an output tensor 330. In some embodiments, the convolution is performed by a DNN accelerator. The convolution may be executed by a processing engine, such as the processing engine in FIG. 5.

[0058] The activation tensor 310 may be computed in a previous layer of the DNN. In some embodiments (e.g., embodiments where the convolutional layer is the first layer of the DNN), the activation tensor 310 may be an image. In the embodiments of FIG. 3, the activation tensor 310 includes activations (also referred to as "input activations," "elements," or "input elements") arranged in a 3D matrix. The activation tensor 310 mayalso be referred to as an input tensor of the convolution. An input element is a data point in the activation tensor 310. The activation tensor 310 has a spatial size HinX WinX Cin, where Hinis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel (IC)), Winis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of activations in a row in the 3D matrix of each IC), and Cinis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of ICs). For the purpose of simplicity and illustration, the activation tensor 310 has a spatial size of 7x7x3, i.e., the activation tensor 310 includes three ICs and each IC has a 7x73D matrix. Each input element in the activation tensor 310 may be represented by a (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the activation tensor 310 may be different.

[0059] Each filter 320 includes weights arranged in a 3D matrix. The values of the weights may be determined through training the DNN. A filter 320 has a spatial size Hf x W X , is the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in a column in each kernel), Wf is the width of the filter (i.e., the length along the X axis, which indicates the number of weights in a row in each kernel), and C is the depth of the filter (i.e., the length along the Z axis, which indicates the number of channels). In some embodiments, equals Cin. For purpose of simplicity and illustration, each filter 320 in FIG. 3 has a spatial size of 3x3x3, i.e., the filter 320 includes 3 convolutional kernels with a spatial size of 3x3. In other embodiments, the height, width, or depth of the filter 320 may be different. The spatial size of the convolutional kernels is smaller than the spatial size of the 3D matrix of each IC in the activation tensor 310.

[0060] An activation or weight may take one or more bytes in a memory. The number of bytes for an activation or weight may depend on the data format. For example, when the activation or weight has an INT8 format, the activation takes one byte. When the activation or weight has a FP16 format, the activation or weight takes two bytes. Other data formats may be used for activations or weights.

[0061] In the convolution, each filter 320 slides across the activation tensor 310 and generates a 3D matrix for an output channel in the output tensor 330. In the embodiments of FIG. 3, the 3D matrix has a spatial size of 5x5. The output tensor 330 includes activations (also referred to as "output activations," "elements," or "output element") arranged in a 3Dmatrix. An output activation is a data point in the output tensor 330. The output tensor 330 has a spatial size HoutX WoutX Cout, where Houtis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in a column in the 3D matrix of each output channel), Woutis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 3D matrix of each output channel), and Coutis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels). Coutmay equal the number of filters 320 in the convolution. Houtand Woutmay depend on the heights and weights of the activation tensor 310 and each filter 320. In an example where the kernel size is lxl, Houtand Woutmay equal to Hinand Win, respectively.

[0062] As a part of the convolution, multiply-accumulate (MAC) operations can be performed on a 3x3x3 subtensor 315 (which is highlighted with a dotted pattern in FIG. 3) in the activation tensor 310 and each filter 320. The result of the MAC operations on the subtensor 315 and one filter 320 is an output activation. In some embodiments (e.g., embodiments where the convolution is an integral convolution), an output activation may include eight bits, e.g., one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), an output activation may include more than one byte. For instance, an output element may include two bytes.

[0063] After the MAC operations on the subtensor 315 and all the filters 320 are finished, a vector 335 is produced. The vector 335 is highlighted with a dotted pattern in FIG. 3. The vector 335 includes a sequence of output activations, which are arranged along the Z axis. The output activations in the vector 335 have the same (x, y) coordinate, but the output activations correspond to different output channels and have different Z coordinates. The output channel dimension along the Z axis may also be referred to as the K dimension or depth dimension. The dimension of the vector 335 along the Z axis may equal the total number of output channels in the output tensor 330. After the vector 335 is produced, further MAC operations are performed to produce additional vectors till the output tensor 330 is produced. In the embodiments of FIG. 3, the output tensor 330 is computed in a Z- major format. When the output tensor 330 is computed in the ZXY format, the vector that is adjacent to the vector 335 along the X axis may be computed right after the vector 335. When the output tensor 330 is computed in the ZYX format, the vector that is adjacent to the vector 335 along the Y axis may be computed right after the vector 335. The outputtensor 330 may be permuted, e.g., by the drain module 1790, and stored in a memory (e.g., the memory 550) in an X-major format or Y-major format.

[0064] In some embodiments, the MAC operations on a 3x3x3 subtensor (e.g., the subtensor 315) and a filter 320 may be performed by a plurality of MAC units. One or more MAC units may receive an input operand (e.g., an activation operand 317 shown in FIG. 3) and a weight operand (e.g., the weight operand 327 shown in FIG. 3). The activation operand 317 includes a sequence of activations having the same (x, y) coordinate but different z coordinates. The activation operand 317 includes an activation from each of the ICs in the activation tensor 310. The weight operand 327 includes a sequence of weights having the same (x, y) coordinate but different z coordinates. The weight operand 327 includes a weight from each of the channels in the filter 320. Activations in the activation operand 317 and weights in the weight operand 327 may be sequentially fed into a MAC unit. The MAC unit may receive an activation and a weight ("an activation-weight pair") at a time and multiple the activation and the weight. The position of the activation in the activation operand 317 may match the position of the weight in the weight operand 327. The activation and weight may correspond to the same channel.

[0065] Activations or weights may be floating-point numbers. Floating-point numbers may have various data formats, such as FP32, FP16, BF16, and so on. A floating-point number may be a positive or negative number with a decimal point. A floating-point number may be represented by a sequence of bits that includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing an exponent of the floating-point number, and bits representing a mantissa of the floating-point number. The mantissa is the part of a floating-point number that represents the significant digits of that number. The mantissa is multiplied by the base raised to the exponent to give the actual value of the floating-point number.

[0066] In some embodiments, the output activations in the output tensor 330 may be further processed based on one or more activation functions before they are written into the memory or inputted into the next layer of the DNN. The processing based on the one or more activation functions may be at least part of the post processing of the convolution. In some embodiments, the post processing may include one or more other computations, such as offset computation, bias computation, and so on. The results of the post processing may be stored in a local memory of the DNN accelerator 502 and be used as input to the nextY1DNN layer. In some embodiments, the input activations in the activation tensor 310 may be results of post processing of the previous DNN layer.

[0067] In some embodiments, convolution operations may be performed by a DNN accelerator, such as the DNN accelerator 502 in FIG. 5. A neural network operation in a transformer model, such as the transformer model 100, may be mapped to a convolution and performed by the DNN accelerator. In some embodiment, an input tensor of a MatMul operation (or other types of operations in the transformer model 100) may be treated (e.g., stored, loaded, or processed) as an activation tensor, and another input tensor of the MatMul operation may be treated as a weight tensor of the convolution. The DNN accelerator may perform at least part of a convolution on the activation tensor and weight tensor to compute the output of the MatMul operation. For example, a query vector, key vector, or value vector of an attention layer, which may have a shape[batch size, SL, dmodei], may be treated as an activation tensor or weight tensor of a convolution. One of the three dimensions may be treated as the IC dimension. In an example, SL is treated as the H dimension, dmodetis treated as the W dimension, and batch size is treated as the IC dimension. In other example, SL or dmodeimay be treated as the IC dimension.

[0068] An execution of a DNN includes execution of layers within the DNN. The DNN execution may be for DNN training or DNN inference. DNN inference may be a deployment of a trained DNN to perform an Al task, e.g., language generation and processing, speech recognition, robotics control, image and video processing, and so on. The workload of executing the DNN may be partitioned into a plurality of tasks, each task may be for executing at least part of a DNN layer. A DNN layer, e.g., a layer that has a large number of internal parameters or has a large amount of computations, may be executed through multiple tasks.

[0069] FIG. 4 illustrates partition of a workload of executing a DNN layer, in accordance with various embodiments. The partition of the workload is illustrated by partition of an output tensor 400 of the DNN layer. In an example where the DNN layer is an attention layer, the output tensor 400 may be the attention weight matrix 230 or output matrix 250. In an example where the DNN layer is a convolutional layer, the output tensor may be the output tensor 330 in FIG. 3. The output tensor 400 has a spatial size of H X W X K, where H denotes the height, W denotes the width, and K denotes the depth.

[0070] In the embodiments of FIG. 4, the output tensor 400 is split over all the three dimensions. As shown in FIG. 4, the output tensor 400 is split into subtensors 410, 420, 430, 440, 450, 460, 470, and 480, each of which is a portion of the output tensor 400. The workload of executing the DNN layer is split into eight smaller workloads for computing the eight subtensors, respectively. In other embodiments, the output tensor 400 may be split in different ways. For instance, the output tensor 400 may be split across one or two of the dimensions.

[0071] In an example in which the output tensor 400 is split over the K dimension, the output tensor 400 may be split into two subtensor: one including the subtensors 410, 420, 430, and 440; the other one including the subtensors 450, 460, 470, and 480. In an example in which the output tensor 400 is split over the H dimension, the output tensor 400 may be split into two subtensor: one including the subtensors 410, 420, 450, and 460; the other one including the subtensors 430, 440, 470, and 480. In an example in which the output tensor 400 is split over the W dimension, the output tensor 400 may be split into two subtensor: one including the subtensors 410, 430, 450, and 470; the other one including the subtensors 420, 440, 460, and 480.

[0072] In an example in which the output tensor 400 is split over the H and K dimension, the output tensor 400 may be split into four subtensor: one including the subtensors 410 and 420; another one including the subtensors 430 and 440, another one including the subtensors 450 and 460, and another one including the subtensors 470 and 480. In an example in which the output tensor 400 is split over the W and K dimension, the output tensor 400 may be split into four subtensor: one including the subtensors 410 and 430; another one including the subtensors 420 and 440, another one including the subtensors 450 and 470, and another one including the subtensors 460 and 480. In an example in which the output tensor 400 is split over the H and W dimension, the output tensor 400 may be split into four subtensor: one including the subtensors 410 and 450; another one including the subtensors 420 and 460, another one including the subtensors 430 and 470, and another one including the subtensors 440 and 480.

[0073] In some embodiments, the smaller workloads generated by splitting the workload of executing the DNN layer may be allocated to multiple tiles in a DNN accelerator (e.g., the DNN accelerator 502 in FIG. 5). That means the subtensors generated by splitting the output tensor 400 may be computed separately by multiple tiles. After a subtensor is generated, itmay be moved to a different tile for executing the next DNN layer. The workload for executing the next DNN layer may be split in a different manner from the way in which the output tensor 400 is split. For example, the output tensor 400 is split in the H, W, and K dimensions as shown in FIG. 4, versus the output tensor of the next DNN layer may be split in the H and K dimensions but not split in the W dimension. The output tensor 400 may be used as an input tensor of the next DNN layer.

[0074] FIG. 5 is a block diagram of a DNN system 500, in accordance with various embodiments. The whole DNN system 500 or a part of the DNN system 500 may be implemented in one or more computing devices, such as the computing device 1400 in FIG. 14. The DNN system 500 can generate and execute DNNs, such as transformer-based models (e.g., the transformer models described above), convolution-based models, and so on. As shown in FIG. 5, the DNN system 500 includes a host 501, a DNN accelerator 502, and a memory 503. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 500. For instance, the DNN system 500 may include multiple hosts or memories. Further, functionality attributed to a component of the DNN system 500 may be accomplished by a different component included in the DNN system 500 or a different system.

[0075] In some embodiments, some of the three components of the DNN system 500 may be implemented on separate chips. For example, the memory 503 may be on a separate chip from the DNN accelerator 502. In this example, the memory 502 is an off-chip memory from the perspective of the DNN accelerator 502. In some embodiments, the host 501 and the DNN accelerator 502 may be different types of processors. In an example, the host 501 may include one or more central processing units (CPUs) or graphics processing units (GPUs). The DNN accelerator 502 may be a neural processing unit, Al accelerator, or Al processor.

[0076] The host 501 may facilitate generation and deployment of DNNs. In some embodiments, the host 501 may generate and train DNNs. For instance, the host 501 can define the layered architecture of a DNN. The host 501 can also determine the internal parameters of the DNN through a DNN training process. The host 501 may also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the sparsity level of one or more deep learning tensors for the DNN.

[0077] The host 501 may deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, the host 501 may distribute trained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform tasks (e.g., image classification, motion planning, etc.) for which the DNNs were trained. In other embodiments, the host 501 may facilitate deployment of the DNNs using the DNN accelerator 502. For instance, the host 501 may receive data from a device or system coupled with the DNN system 500 and input the received data (or data generated by the host 501, e.g., based on the received data) into a DNN. The host 501 may generate instructions (e.g., configuration files) that control the operation of the DNN accelerator 502 during the DNN execution. The host 501 may receive an output of the DNN from the DNN accelerator 502. The host 501 may transmit the output of the DNN (or a result of processing the output of the DNN by the host 501) to the device or system. In some embodiments, the host 501 may control execution processes of trained, compressed, or validated DNNs. The host 501 may function as a compiler for DNNs executed by the DNN accelerator 502. The host 501 may perform compilation of DNNs and generate compilation descriptors, based on which the DNNs may be executed.

[0078] The host 501 may generate executable DNN models. The host 501 may also facilitate execution of DNN layers by the DNN accelerator 502. For instance, the host 501 may be the host for the execution of neural network operations by the DNN accelerator 502, e.g., the host for DNN inference. In some embodiments, the host 501 may receive an inference request, which may be a request to have a DNN model to make a prediction based on input data. The host 501 may also partition workloads for executing DNN layers into smaller workloads and determine allocation of the smaller workloads to tiles in the DNN accelerator 502. The host 501 may also determine and generate configuration registers for inter-tile communication for DNN layers executed by multiple tiles.

[0079] In some embodiments, the host 501 may compile DNNs to generate executable DNNs. The compilation may be performed offline, e.g., before the DNN execution. The host 501 may also schedule computational tasks in DNN execution during runtime. In an example, the host 501 may include a compiler and a runtime scheduler. The compiler may perform compilation of DNNs. In some embodiments, the compiler compiles information of DNNs to executable instructions that can be executed, e.g., by the DNN accelerator 502, to carry out neural network operations in DNNs. In some embodiments, the compiler maygenerate a graph representing a DNN. The graph may include nodes and edges. A node may represent a specific neural network operation in the DNN. An edge may connect two nodes and represent a connection between the two corresponding neural network operations. In an example, an edge may encode a tensor that flows from one of the neural network operations to the other neural network operation. The tensor may be an output tensor of the first neural network operation and an input tensor of the second neural network operation. The edge may encode one or more attributes of the tensor, such as size, shape, storage format, and so on. The compiler may use the graph to generate executable DNNs. For instance, the compiler may generate computer program instructions (e.g., compilation descriptors) for executing DNNs.

[0080] In some embodiments, the compiler may partition the workload of executing a DNN into a list of tasks. The compiler may generate one or more tasks for a single DNN layer. The compiler may determine that a layer is too large to be executed by a single tile, e.g., the compiler may determine that the number of internal parameters of the layer is beyond a threshold or the amount of computations in the layer is beyond a threshold. The compiler may partition the task of executing the layer into multiple tasks. In some embodiments, to partition the task of executing a layer, the compiler may partition the output tensor of the layer into subtensors. Each subtensor represents a separate workload or separate task. For instance, the compiler may determine the shape or size of each subtensor based on the shape or size of the output tensor and the number of tiles available for executing the DNN layer. The compiler may split the output tensor in the H dimension, W dimension, K dimension, or some combination thereof. The compiler may also build producer-consumer dependencies between tasks. For instance, the compiler may label a task as a producer task and another task as a consumer task as data computed from the first task is needed for performing the second task. The consumer task cannot be started until the producer task is complete. The producer-consumer dependencies may be implemented in the DNN accelerator 502 through hardware barriers in the DNN accelerator 502, which will be discussed further below.

[0081] In some embodiments, the compiler may generate virtual containers for tasks of executing DNNs. The virtual container of a task may indicate a virtual execution environment that includes graphical representation of resources needed for performing the task. The resources may include computational resources (e.g., PE, DSP, etc.), storageresources (e.g., on-chip memory, off-chip memory, etc.), control resources (e.g., barrier, task FIFO, DMA resource, etc.), and so on. In some embodiments, the compiler may follow a tiled architecture to generate the virtual containers. For instance, the compiler may place one or more virtual tiles into each virtual container. A virtual tile may be a graphical representation of resources within a tiled architecture. A virtual tile may be a graphical representation of hardware resources associated with a hardware tile (e.g., a tile in the DNN accelerator 502). Hardware resources associated with a hardware tile may include computational resources, storage resources, and control resources. The hardware resources may include resources inside the hardware tile plus resource outside the hardware tile but are allocated to the hardware tile. A virtual container may include multiple virtual tiles, indicating that the performance of the task requires multiple hardware tiles. A virtual container may have graphical representations of resources inside tiles as well as resources outside tiles.

[0082] In some embodiments, the compiler may also generate configuration registers for the tasks. The compiler may generate a set of configuration parameters for each task. The set of configuration parameters may constitute a job map register. Examples of configuration parameters in a job map register may include configuration parameter identifying the task, configuration parameter indicating the total number of tiles for performing the task, configuration parameter mapping virtual tile(s) to hardware tile(s), other types of configuration parameters, or some combination thereof. In some embodiments, the compiler may also generate configuration parameters that facilitates data read or data write, such as a configuration parameter that indicates the number of data elements to be processed, configuration parameter that indicates the memory address where an input data element may be fetched, configuration parameter that indicates memory address where an output data element may be stored, configuration parameter that indicates memory address where another configuration parameters may be stored, and so on. Configuration parameters or descriptors may be stored in registers associated with the corresponding components of the DNN accelerator 502. Such registers may be referred to as configuration registers or configuration space registers.

[0083] In some embodiments, the compiler may also generate configuration parameters that specify operation modes of components in the DNN accelerator 502, such as one or more configuration parameters specifying whether a sparse cell 515 operates in dense orsparse modes. In dense modes, the sparse cell 515 may execute a DNN layer without any sparsity acceleration. For instance, the sparse cell 515 may not skip computations of zero data elements in dense modes. In sparse modes, the sparse cell 515 may accelerate the execution of a DNN layer based on sparsity by skipping computations of zero-valued data elements. The compiler may determine whether to accelerate the layer based on weight sparsity, activation sparsity, or both. The compiler may select the sparse mode for a layer from a group of sparse modes that includes, for example, combined sparse mode in which the layer is accelerated based on both weight sparsity and activation sparsity, activation sparse mode in which the layer is accelerated based on activation sparsity but not based on weight sparsity, weight sparse mode in which the layer is accelerated based on weight sparsity but not based on activation sparsity, and a dense mode in which the layer is not accelerated based on sparsity.

[0084] In some embodiments, the compiler may perform optimization of DNNs. For instance, the compiler may translate high-level neural network descriptions into an efficient list of tasks. The compiler may gatherviable scheduling options and use a cost model to choose the most appropriate tensor split or tile strategies. In some embodiments, the compiler may use spatial tiling when the compiler decides to split the DNN layer across multiple tiles. For instance, the compiler may split the output tensor of the DNN layer over the H dimension, W dimension, K dimension, or some combination therefore to obtain subtensors within the output tensor. The compiler may assign the task of computing the subtensors to different tiles. In some embodiments, the compiler may use a temporal tiling technique, e.g., when the whole workload cannot fit in on-chip memory and therefore a DNN layer needs to be broken up into multiple workloads that will fit. In some embodiments, the compiler may use a cross engine tiling technique to minimize a layer execution time when multiple tiles are needed to complete a layer. Examples of this are DMA fetching the parameter or activations for the next tile while the PEs and / or DSPs are performing convolution on the previous tile. By utilizing those or other techniques, the compiler may ensure that the resulting executable DNN can efficiently use the hardware resources within the DNN accelerator 502 to achieve a faster and more efficient execution of the model.

[0085] The runtime scheduler schedules tasks of executing DNNs during runtime. In some embodiments, the runtime scheduler may facilitate distribution hardware resources toworkloads of compiled DNNs. For instance, for each task, the runtime scheduler may also identify one or more tiles that can perform a task based on one or more characteristics of the tile(s) and the task. The compiler may ensure that the tile has sufficient memory space and computational capacity (e.g., sufficient PEs or PPEs) to perform computations in the task. The runtime scheduler may dynamical allocate hardware resources for compiled DNN execution. In some embodiments, the runtime scheduler may allocate hardware resources in the DNN accelerator 502 to tasks of executing a DNN based on a schedule. By scheduling tasks effectively, the runtime scheduler can maximize parallelism and balance the workload across tiles. The runtime scheduler may also manage tasks prioritization to ensure critical models are executed promptly while effectively managing resources contention. The runtime scheduler may also facilitate task synchronization by coordinating the execution of tasks across multiple agents. Task synchronization can help achieve high throughput, low latency, and efficient utilization of hardware resources while ensuring correctness and reliability in the execution.

[0086] In some embodiments, the runtime scheduler may allocate hardware resources to DNN execution tasks by mapping virtual containers generated by the compiler to hardware resources in the DNN accelerator 502. The mapping is virtual to physical mapping, which is also referred to as virtual to physical translation. The runtime scheudler 505 can also determine the schedules of tasks. In some embodiments, the runtime scheduler determine the time when a task can be started based on one or more producer-consumer dependencies of the task. The runtime scheduler may determine that a task can be started after the one or more producer-consumer dependencies are all satisfied. The determination may be done through physical to virtual mapping. An exmaple of physical to virtual mapping may be the mapping of a status change of a hardware resource to a status change of one or more virtual containers. Virtual to physical and physical to virtual mapping may be facilitated by translation points implemented in the DNN accelerator 502, which are described below in more details. The runtime scheduler may perform the allocation of hardware resources through configuration registers, such as the configuration registers described above. Even though it is mentioned above that configuration registers may be generated by the compiler, configuration parameters may be generaed by the runtime scheduler in lieu or in addition of the compiler in some embodiments.

[0087] The DNN accelerator 502 performs tasks of executing DNNs. For instance, the DNN accelerator 502 can execute a DNN by running neural network operations in the DNN. The process of carrying out a neural network operation is also referred to as a process of executing the neural network operation or a process of performing the neural network operation. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. As shown in FIG. 5, the DNN accelerator 502 includes a data processing unit 510, a DSP array 520, a control module 530, a DMA module 540, and a memory 550. In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 502. For example, the DNN accelerator 502 may include a microcontroller. Further, functionality attributed to a component of the DNN accelerator 502 may be accomplished by a different component included in the DNN accelerator 502 or by a different module or system (e.g., the host 501). A component of the DNN accelerator 502 may be implemented in hardware, software, firmware, or some combination thereof.

[0088] The data processing unit 510 includes sparse cells 515. The sparse cells 515 may be arranged in an array including rows and columns. A sparse cell 515 may perform deep learning operations in DNNs. For instance, a sparse cell 515 may execute a DNN layer by running one or more deep learning operations in the DNN layer. A sparse cell 515 may execute a layer, or a portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple sparse cells 515 in parallel.

[0089] The sparse cell 515 may be capable of running various types of deep learning operations, such as convolution, layer normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Deep learning operations performed by the sparse cells 515 may include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, the sparse cells 515 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed.

[0090] A sparse cell 515 may include an array of PEs. An example PE may be or may include one or more multiply-accumulate (MAC) units that can perform MAC operations. In some embodiments (e.g., embodiments where the sparse cell 515 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include oneor more activations in the input tensor of the convolution. Different activations may be in different ICs. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN. The weights in the weight operand may be in different ICs.

[0091] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0092] In some embodiments, the sparse cells 515 may be capable of depthwise convolution, standard convolution, or both. In a depthwise convolution, an MAC unit may perform an MAC operation that includes a sequence of multiplications for an input operand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activation and weight in the same cycle may correspond to the same channel. The sequence of multiplication produces a product operand that includes a sequence of products. The MAC operation may also include accumulations in which multiple product operands are accumulated to produce an output operand of the MAC unit. The sparse cells 515 may output multiple output operands at a time, each of which is generated by a different MAC unit. In a standard convolution, MAC operations may include accumulations across the channels. For instance, as opposed to generating an output operand, a MAC unit may accumulate products across different channels to generate a single output point.

[0093] In some embodiments, the sparse cells 515 may perform MAC operations in quantized deep learning operations, such as MAC operations in a quantized convolution. Insome embodiments, an MAC unit in the sparse cells 515 may receive quantized activation and quantized weights and compute a quantized MAC result. The quantized MAC result may be a quantized value in an integer format and may be the output of the MAC unit. In some embodiments, the MAC unit may also include a quantization multiplier that can multiply a quantization scale with the quantized MAC result, and the output of the MAC unit may be a real value in a floating-point format. The MAC unit may include no quantization subtractors as zero-point offsetting is not needed for the MAC operations in quantized deep learning operations.

[0094] In some embodiments, the sparse cells 515 may include sparsity acceleration logic for facilitating sparsity acceleration. For instance, each processing cell in the sparse cells 515 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, a sparsity module accelerates computations in the sparse cells 515 based on sparsity in activations, sparsity in weights, or both. The sparsity module may include a storage unit that stores a sparsity tensor, which may be loaded to the storage unit. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0095] An activation sparsity tensor may be the sparsity tensor of an activation tensor and has the same number of elements as the activation tensor. An element in the activation sparsity tensor may indicate whether the corresponding element in the activation tensor is zero or not. For instance, a zero-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is zero. A one-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is nonzero. A weight sparsity tensor may be the sparsity tensor of a weight tensor and has the same number of elements as the weight tensor. An element in the weight sparsity tensor may indicate whether the corresponding element in the weight tensor is zero or not. For instance, a zero-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is zero. A one-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is nonzero. The sparsity module may generate a combined sparsity tensor using an activation sparsity tensor and a weight sparsity tensor. For instance, the sparsity module may multiply an element of the activation sparsity tensor with a corresponding element of the weight sparsity tensor tocompute an element of the combined sparsity tensor. The positions of the three elements in their corresponding sparsity tensors may match. In some embodiments, each element in a sparsity tensor may be a bit, and the sparsity tensor may be referred to as a sparsity bitmap.

[0096] The sparsity module may use the sparsity tensor to identify activations and weights to be used in MAC operations by the MAC units. In an embodiment where the sparse cells 515 operates in the combined sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a combined sparsity tensor. In an embodiment where the sparse cells 515 operates in the activation sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of an activation sparsity tensor. In an embodiment where the sparse cells 515 operate in the weight sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a weight sparsity tensor. The sparsity module may be bypassed in the dense mode as no sparsity acceleration would be conducted.

[0097] The sparse cells 515 may constitute a processing engine. Even though not shown in FIG. 5, the data processing unit may include one or more post-processing engines. A postprocessing engine processes outputs of a processing engine. The post-processing engine may include one or more post-processing elements (PPEs). In some embodiments, the PPEs in the post-processing engine may be arranged in an array that has rows and columns. In some embodiments, the post-processing engine computes activation functions. The postprocessing engine may receive outputs of the processing engine as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine may perform other types of post processing on outputs of the processing engine. For instance, the post-processing engine may apply a bias on an output of the processing engine. In some embodiments, the post-processing engine may be bypassed for certain neural network operations.

[0098] The DSP array 520 includes DSPs 525 arranged in an array. In some embodiments, a DSP 525 may have an architecture optimized for the operational needs of digital signal processing. A DSP 525may be implemented on a microprocessor chip, which may be separate from a chip implementing the data processing unit 510. In some embodiments, a DSP 525may be a Streaming Hybrid Architecture Vector Engine (SHAVE) DSP. In some embodiments, a DSP 525can be pipelined along with the inference pipeline and the DMAmodule 540, thereby enabling parallel computations to improve overall performance. In some embodiments, a DSP 525 may perform some computations in a neural network operation, while other computations in the neural network operation may be performed by the processing engine or post-processing engine.

[0099] In some embodiments, a DSP 525 may operate in accordance with a clock signal. For instance, the timing when the DSP 525 can execute instructions may be synchronized with the clock signal. The clock signal may be provided by a DSP clock. The DSP clock may be controlled by a firmware and a self-clock gate. For instance, the firmware and self-clock gate may enable the DSP clock to generate the clock signal when the DSP needs to execution instructions but disable the DSP clock to drop the clock signal when the DSP does not need to execute instructions. Certain aspects of self-clock gating are provided below in conjunction with FIGS. 8-11.

[0100] The control module 530 controls schedules of tasks performed by the DNN accelerator 502. As shown in FIG. 5, the control module 530 includes a task FIFO 533 and a barrier 535. The task FIFO 533 may be a storage unit (e.g., a buffer) that stores tasks. The task FIFO 533 may store a sequence of tasks in the workload of executing a DNN. A task may be pushed out of the task FIFO 533 and sent to the data processing unit 510 or DSP array 520, whichever is allocated to perform the task. The next task may become the first task in the task FIFO 533. In some embodiments, the tasks are being pushed out in the order in which they are added to the task FIFO 533. In other embodiments, the tasks may be pushed out in a different order, e.g., an order determined by the runtime scheduler. The task FIFO 533 may be used to dispatch tasks to data processing unit 510 and DSP array 520. In some embodiments, the task FIFO 533 may store information of each task, such as a context ID for each task. The context ID may identify the context to be processed in the task. The context may be a subtensor of the input tensor of a DNN layer. In other embodiments, the task FIFO 533 may store a task ID that identifies the task.

[0101] The barrier 535 blocks tasks from being sent to tiles or from being performed. In some embodiments, the control module 530 includes multiple barriers. The barrier 535 may be associated with tasks in the task FIFO 533. In an example, the barrier 535 has one or more consumer tasks and one or more producer tasks. The barrier 535 may be lifted after all the producer tasks associated with the barrier 535 are complete. Before that, all the consumer tasks associated with the barrier 535 are blocked by the barrier 535. The barrier535 can facilitate task synchronization by ensuring producer-consumer dependencies are met to maintain data coherency and avoid racing conditions.

[0102] The DMA module 540 facilitates data transfer between the memory 503 and the DNN accelerator 502 For example, the DMA module 540 can read data from the memory 503 and write data into the memory 550. As another example, the DMA module 540 can read data from the memory 550 and write data into the memory 503. The DMA module 540 provides a DMA feature that allows the DNN accelerator 502 to initiate data transfer between the memory 503 and the memory 550 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA module 540 may read tensors from the memory 503, modify the tensors in a way that is optimized for the DNN accelerator 502 before it writes the tensors into the memory 550.

[0103] The memory 550 is local to the DNN accelerator 502. In the embodiments of FIG. 5, the memory 550 is inside the DNN accelerator 502. The memory 550 may be an on-chip memory. In other embodiments, the memory 550 may be outside the DNN accelerator 502. Data in the memory 550 may be transferred to or from the memory 503, e.g., through the DMA module 540. For instance, input tensors of DNN layers may be copied from the memory 503 to the memory 550. In some embodiments, data in the memory 550 may be transferred to or from the local memory of another local memory of the DNN accelerator 502. The memory 550 may store data received, used, or generated by the data processing unit 510 or DSP array 520. Examples of the data may include input activations, weights, output activations, sparsity bitmaps, and so on.

[0104] In some embodiments, the memory 550 may store tensors to be processed by the data processing unit 510 or DSP array 520. The tensors may be input tensors of deep learning operations. The memory 550 may also store tensors generated by the data processing unit 510 or DSP array 520. The tensors may be output tensors of deep learning operations. In some embodiments, the memory 550 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which zero-valued elements (if any) are not removed. A dense tensor may be converted to a sparse tensor by removing one or more zero-valued elements in the dense tensor. A sparse tensor may also be referred to as a compressed tensor or packed tensor. The process of converting a dense tensor to a sparse tensor may be referred to as sparsity encoding.Sparsity encoding may also generate a sparsity tensor. Each element in the sparsity tensor may correspond to a different element in the dense tensor and indicate whether the element in the dense tensor is zero or not. The sparsity tensor may indicate positions of elements of the sparse tensor in the dense tensor. The sparsity tensor may be a sparsity bitmap, each element of which is a bit. A sparse tensor may be converted to a dense tensor through a densifying process, in which one or more zeros may be added to the sparse tensor based on the sparsity tensor.

[0105] In some embodiments, the memory 550 includes one or more SRAMs. The memory 550 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the memory 550 may include memory banks. The number of data banks in the memory 550 may be 16, 64, 128, 556, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the memory 550 in a single read cycle. In other embodiments, 16 bits can be transferred from the memory 550 in multiple read cycles, such as two cycles.

[0106] The memory 503 stores data received, generated, or used by the DNN system 500 for DNN executions. The memory 503 may be a system memory. In some embodiments, the memory 503 includes a dynamic random-access memory (DRAM). The memory 503 may store virtual contains and configuration registers generated by the host 501. When the DNN accelerator 502 executes operations in transformer models, at least part of the memory 503 may be used to implement KV caches, such as self-attention KV caches and cross-attention KV caches, in the transformer models. The KV caches may be updated during inference of the transformation model. Layout of data in the KV caches may be determined to optimize the efficiency of the DNN accelerator 502.

[0107] In some embodiments, the memory 503 may store data to be used by the DNN accelerator 502 for DNN execution. The memory 503 may store weights, such as weights of convolutional layers, which are determined by training DNNs. The memory 503 may furtherstore inputs to DNN layers or outputs of DNN layers, such as data generated by the data processing unit 510 or DSP array 520 from performing tasks in workloads of executing DNNs. Example deep learning operations include convolutions (also referred to as "convolutional operations"), layer normalization operations, SoftMax operations, matrix multiplication operations, pooling operations, elementwise operations, activation functions, other types of deep learning operations, or some combination thereof. The memory 503 may be a main memory of the DNN system 500. In some embodiments, the memory 503 includes one or more DRAMs.

[0108] FIG. 6 illustrates an example DNN execution sequence 600, in accordance with various embodiments. The DNN execution sequence 600 includes a configuration kernel 610, management kernels 620 (individually referred to as "management kernel 620"), and compute kernels 630 (individually referred to as "compute kernels"). Each kernel 610 may represent a stage in the DNN execution. For instance, the configuration kernel 610 may be a stage in which initial DSP configuration is performed. A management kernel 620 may be a stage in which a task assigned to the DSP is prepared for execution. The task may be fetched from a task FIFO and wait for one or more producer-consumer dependencies are satisfied. A compute kernel 630 may be a stage in which all the producer-consumer dependencies of the corresponding task are satisfied and the DSP transitions to a computation phase to perform the task (e.g., to execute a DNN layer or part of the layer). After the task is complete, the DSP transitions to another management kernel 620 for the next task to be performed by the DSP. The other management kernel 620 may be followed by another compute kernel 630. This may continue till all the tasks assigned to the DSP are complete.

[0109] In some embodiments, during the execution of a DNN, the configuration kernel 610 may be called once. After that, the DSP may alternate between management kernels 620 and compute kernels 630. The amount of time that the DSP spends in management kernels 620 and compute kernels 630 can vary greatly, depending on the executable DNN and efficiency of executing layer on DSPs versus PEs.

[0110] FIG. 7 illustrates clock gating timing during DNN executions, in accordance with various embodiments. For the purpose of illustrations, FIG. 7 shows two cycles of DNN execution. An example of the DNN execution may be the DNN execution illustrated in FIG. 6. FIG. 7 shows a signal 710, which may be a free running signal from a free running input clock. The signal 710 may be on during the whole process of the DNN execution. FIG. 7 alsoshows a signal 720 and a signal 730. The signal 720 may be a DSP clock enable signal, which may indicate the timing when a DSP clock is enabled or disabled. The signal 720 may be controlled and provided by a firmware. The signal 730 may be a DSP clock signal provided by a DSP clock. In some embodiments, the signal 730 is on when the signal 720 is on. The signal 730 may synchronize the operation of the DSP.

[0111] FIG. 7 further shows a signal 740 may be a DSP APB (advanced peripheral bus) signal. The signal 740 may indicate the timing to read or write a configuration space register associated with the DSP. The signal 740 corresponds to a kernel schedule 750 of the DSP. The kernel schedule includes segments corresponding to configuration kernels 715 (individually referred to as "configuration kernel 715"), management kernels 725 (individually referred to as "management kernel 725"), and compute kernels 735 (individually referred to as "compute kernel 735"). The kernel schedule 750 represents the two cycles of DNN execution, each cycle starting with a configuration kernel 715, which is followed by management kernels 725 and compute kernels 735. An example of the configuration kernels 715 may be the configuration kernel 610 in FIG. 6. Example of the management kernels 725 may include the management kernels 620 in FIG. 6. Examples of the compute kernels 735 may include the compute kernels 630 in FIG. 6. For each cycle of DNN execution, a DSP program counter may start at the beginning of the cycle of DNN execution (i.e., point A in FIG. 7), which may be the start of the corresponding configuration kernel 715. The DSP program counter may stop at the end of the last kernel (i.e., point B in FIG. 7), which may be a management kernel 725 or compute kernel 735. In the embodiment of FIG. 7, the DSP clock is on during all the kernels.

[0112] FIG. 8 illustrates self-clock gating timing during DNN executions, in accordance with various embodiments. For the purpose of illustrations, FIG. 8 shows two cycles of DNN execution. An example of the DNN execution may be the DNN execution illustrated in FIG. 6. FIG. 8 shows a signal 810, which may be a free running signal from a free running input clock. The signal 810 may be on during the whole process of the DNN execution. FIG. 8 also shows a signal 820 and a signal 830. The signal 820 may be a DSP clock enable signal, which may indicate the timing when a DSP clock is enabled or disabled. The signal 820 may be controlled and provided by a firmware. The signal 830 may be a DSP clock signal provided by a DSP clock. In some embodiments, the signal 830 is on when the signal 820 is on. The signal 830 may synchronize the operation of the DSP.

[0113] FIG. 8 further shows a signal 840 may be a DSP APB (advanced peripheral bus) signal. The signal 840 may indicate the timing to read or write a configuration space register associated with the DSP. The signal 840 corresponds to a kernel schedule 850 of the DSP. The kernel schedule includes segments corresponding to configuration kernels 815 (individually referred to as "configuration kernel 815"), management kernels 825 (individually referred to as "management kernel 825"), and compute kernels 835 (individually referred to as "compute kernel 835"). The kernel schedule 850 represents the two cycles of DNN execution, each cycle starting with a configuration kernel 815, which is followed by management kernels 825 and compute kernels 835. An example of the configuration kernels 815 may be the configuration kernel 610 in FIG. 6. Example of the management kernels 825 may include the management kernels 620 in FIG. 6. Examples of the compute kernels 835 may include the compute kernels 630 in FIG. 6. For each cycle of DNN execution, a DSP program counter may start at the beginning of the cycle of DNN execution (i.e., point A in FIG. 8), which may be the start of the corresponding configuration kernel 815. The DSP program counter may stop at the end of the last kernel (i.e., point B in FIG. 8), which may be a management kernel 825 or compute kernel 835.

[0114] Different from the embodiment of FIG. 7, the DSP clock is not on during all the kernels in the embodiments of FIG. 8. As indicated by the signals 820 and 830, the DSP clock is on during the configuration kernels 815 and compute kernels 835, but the DSP clock is off during the management kernels 725. The signal 820 may be controlled by both the firmware and a self-clock gate associated with the DSP clock. Similarly, the signal 830 may be controlled by both the DSP clock and the self-clock gate. The self-clock gate may be active during the management kernels 825 to gate the DSP clock from providing the signal 830. That way, the DSP may be deactivated or get into an idle stage during the management kernels 825. This can reduce the power consumption during the management kernels 825 and improve the efficiency and performance of the DNN accelerator. The self-clock gating mechanism may be implemented in a self-clock gating circuitry in the DNN accelerator.

[0115] FIG. 9 illustrates an example circuitry for DSP self-clock gating, in accordance with various embodiments. The circuitry may be a self-clock gating circuitry inside a DNN accelerator, such as the DNN accelerator 502. The circuitry includes a DSP 910, self-clock gate 920, gate controller 930, event detectors 940, event detector 945, event detector 950, event detector 955, and a register interface pipeline 960. The DSP 910 may be an exampleof the DSPs 525 in FIG. 5. The DSP 910 includes a CSR l / F (configuration space register interface) 915 and a clock 917. The CSR l / F 915 is coupled with the event detector 950. The clock 917 is coupled with the self-clock gate 920. The self-clock gate 920 is also coupled with the clock gate controller 930. The clock gate controller 930 is further coupled with the event detectors 940, event detector 945, event detector 950, event detector 955, and register interface pipeline 960. In other embodiments, the circuitry may include fewer, more, or different components.

[0116] The DSP 910 may include global port input (GPI) pins for receiving signals from other components of the circuitry and GPO pins for sending out signals to other components of the circuitry. As shown in FIG. 9, the DSP 910 may send a gate enable signal to the clock gate controller 930 through a GPO pin. In some embodiments, the DSP 910 may write the gate enable signal to the GPO pin and the gate enable signal may be sent to the clock gate controller 930 from the GPO pin. Additionally, the DSP 910 may send signals for programming barrier status wake to the event detector 940 through a GPO pin. Such signals may specify a trigger type or trigger mask so that the event detectors 940 may provide a barrier wake signal to the clock gate controller 930 when event detectors 940 detects an event having the trigger type or trigger mask. An example of such an event may be that a barrier blocking a computational task assigned to the DSP 910 has been lifted. Similarly, the DSP 910 may send signals for programming FIFO stats wake to the event detector 945 through a GPO pin. Such signals may specify a trigger type or trigger mask so that the event detectors 940 may provide a FIFO wake signal to the clock gate controller 930 when event detectors 940 detects an event having the trigger type or trigger mask. An example of such an event may be that a FIFO storing a computational task assigned to the DSP 910 has been emptied, e.g., the computational task has been dispatched from the FIFO to the DSP 910.

[0117] In some embodiments, the event detectors 940 and 945 are configured, e.g., by the DSP 910, to detect maskable wake events. The DSP 910 may be sensitive to different conditions at various stages of DNN execution as the DSP 910 may perform different computational task or computational tasks with different producer-consumer dependencies in different stages of DNN execution. In an example, the DSP 910 may be waiting for different barriers to be lifted or different FIFOs to be emptied for different tasks. Different from the 640 / / and 645 / / , the event detector event detector 950 and event detector 955 may be configured to detect non-maskable wake events. In some embodiments, the eventdetector 950 may detect external halt and provides a halt awake signal to the clock gate controller 930 after an external halt is detected. The external halt may be managed by firmware. The event detector 955 is coupled with the register interface pipeline 960. The event detector 955 may send a register active signal, which may indicate that there is active access to a configuration space register associated with the DSP 910, to the clock gate controller 930 when the event detector 955 detects that the access to the configuration space register is active.

[0118] The clock gate controller 930 may also receive signals indicating that the self-clock gate 920 can be enabled. For instance, the clock gate controller 930 receives a DSP clock enable signal form a firmware. The clock gate controller 930 also receives a gate enable signal from a GPO pin of the DSP 910. After receiving these signals, the clock gate controller 930 may determine whether it is safe to enable the self-clock gate 920 based on the signal from the event detector 955 and a memory idle signal from the DSP 910. The memory idle signal may indicate that there is no pending access to memory, including cache. After the clock gate controller 930 determines that the self-clock gate 920 can be enabled and that it is safe to enable the self-clock gate 920, the clock gate controller 930 may activate the selfclock gate 920. For instance, the clock gate controller 930 may command the self-clock gate 920 to gate the clock 917. The clock 917 may then stop the DSP clock signal.

[0119] The clock gate controller 930 may keep checking whether the clock 917 needs to be awakened. For instance, the clock gate controller 930 may allow receipt of wake signals from the event detectors 940, event detector 945, and event detector 950. After the clock gate controller 930 receives a wake signal, the clock gate controller 930 may send a clock awaken signal to the DSP 910 through a GPI pin of the DSP 910. The clock 917 may resume the clock signal so that the DSP 910 can start executing instructions. The DSP 910 may clear the gate enable signal at the corresponding GPO pin.

[0120] FIG. 10 illustrates a flow graph 1000 of an example self-clock gating handshake of a DSP, in accordance with various embodiments. The DSP may be inside a DNN accelerator that can be used for DNN executions. The DSP may be an example of the DSPs 525 in FIG. 5. The flow graph 1000 includes several steps in the self-clock gating handshake. In Step 1010, the DSP (or DSP kernel) reaches a point to wait for external dependency. For instance, the DSP has finished a computational task in a DNN execution and is waiting to perform the next computational task. In Step 1020, the DSP sets wake triggers and enables self-clock gatingusing a GPO pin. For instance, the DSP may set wake triggers by programming one or more event detectors. The DSP may set the type(s) and mask(s) of trigger events. The DSP may enable self-clock gating by providing a gate enable signal through its GPO pin. The gate enable signal may be sent to a control logic through the GPO pin.

[0121] In Step 1030, the control logic determines that it is safe to stop the DSP clock. In Step 1040, a control logic stops the DSP clock. For instance, a gate controller activates a self-clock gate, which gates the DSP clock from providing any clock signal. In Step 1050, a wake event is detected. The wake event may be a trigger event that can trigger awaken of the DSP clock. In Step 1060, the control logic sets a GPI pin of the DSP with a clock awaken signal and starts the DSP clock. In Step 1070, the DSP exits loop, awaiting for the GPI signal to be set. In Step 1080, the DSP clears the gate enable signal GPO and checks for awake reason. In embodiments where the DSP determines that the awake reason is valid, the DSP may resume execution of instruction in Step 1090. In embodiments where the DSP determines that the awake reason is invalid or the DSP has finished the performance of a computational task associated with the awake reason, Step 1020 and subsequent steps may be performed.

[0122] FIG. 11 illustrates timing of various signals involved in self-clock gating of a DSP, in accordance with various embodiments. The DSP may be inside a DNN accelerator that can be used for DNN executions. The DSP may be an example of the DSPs 525 in FIG. 5. FIG. 11 shows the timing of eight signals 1110, 1120, 1130, 1140, 1150, 1160, 1170, and 1180. The signal 1110 may be a free running clock signal. The signal 1110 may be always on despite the status of the DSP. For instance, the signal 1110 may be on as long as the DNN accelerator is on. The signal 1120 may be a DSP clock enable signal. In some embodiments, when the signal 1120 is on, the DSP is enabled; when the signal 1120 is off, the DSP is disabled. The signal 1130 may be a DSP clock signal. The signal 1130 may be generated by a DSP clock and may synchronize actions of the DSP. In some embodiments, when the signal 1130 is on, the DSP may be active and can execute instructions; when the signal 1130 is off, the DSP may become inactive and stop executing instructions. The signal 1140 may be a DSP executing signal, which indicates the timing when the DSP executes instructions. The signal 1150 may be a gate enable signal. In some embodiments, the signal 1150 is provided by the DSP. The signal 1160 may be a DSP access signal that indicates whether there is any access request pending for the DSP. The signal 1170 may be a gate awaken signal, which may be provided by a gate controller. The signal 1180 may be a wake signal, which may be providedby one or more event detectors. For the purpose of illustration, FIG. 11 shows points a-q that represent various critical times within a self-clock gating handshake.

[0123] For instance, point a may represent the start of the signal 1120. The signal 1120 may be provided by a firmware and may start when the firmware enables the DSP clocks. Point b is after point a. Point b may represent the stat of the signal 1130. Starting point b, the DSP can start executing instructions. Point c is after point b. Point c represents the start of instruction execution by the DSP. The DSP may execute instructions for accessing memory, accessing register, performing computation, and so on. Point c may represent the start of the DSP's operation. Power consumption may be started at point b or point c.

[0124] Point d may represent the start of the signal 1140. In some embodiments, the may reach an external dependency wait point. The DSP may program a wake condition and enable self-clock gating by starting the signal 1150. The signal 1160 may be on and off, indicating the DSP access traffic, such as pending memory traffic. The DSP may toggle the signal 1160. As shown in FIG. 11, the signal 1160 is dropped starting point e, in response to which the gate controller may enable the self-clock gate, which gates the DSP clock, so the signal 1130 drops at point f. In some embodiments, the gate controller may wait for the signal 1160 to drop to activate or enable the self-clock gate. The drop of the signal 1160 may indicate that there is no pending access request to be handled by the DSP so that it is safe to enable the self-clock gate and deactivate the DSP. In some embodiments, point e and point f may be the same time point. As the signal 1130 drops, the signal 1140 also drops as the DSP cannot execute instructions anymore. Power consumption is therefore eliminated.

[0125] Point g represents the start of the signal 1180. The signal 1180 may be provided to the gate controller by an event detector and may indicate that a trigger event has occurred. After receiving the signal 1180, the gate controller may generate the signal 1170, which starts at point h. As shown in FIG. 11, point h is after point g. The signal 1170 may be sent to the DSP, and the DSP clock may be awakened and resume the signal 1130 at point i. The signal 1140 also resumes, indicating that the DSP start executing instructions again. After that, the DSP clears the signal 1150, and the signal 1150 drops at point j. The gate controller clears the signal 1170, and the signal 1170 drops at point k. The time between point a and point k may be a cycle of self-clock gating handshake. This cycle may be repeated one or more times till the firmware stops the DSP clock. Point p indicates the end of the signal 1120, i.e., the time when the DSP clock is disabled by the firmware. Point q is after point pand represents the end of the signal 1130, i.e., the time when the DSP clock stops providing the signal 1130.

[0126] FIG. 12 illustrates an example sparse cell 1200, in accordance with various embodiments. The sparse cell 1200 may be an example of the sparce cells 515 in FIG. 5. As shown in FIG. 12, the sparse cell 1200 includes 16 MAC units 1210 (individually referred to as "MAC unit 1210"), which constitutes a MAC array having four rows and four columns. The MAC array has a spatial shape of 4x4, meaning the height of the MAC array is four and the width of the MAC array is also four. The sparse cell 1200 also includes 16 weight register files 1220 (individually referred to as "weight register file 1220"), 16 activation register files 1230 (individually referred to as "activation register file 1230"), four row buffers 1240 (individually referred to as "row buffer 1240"), and sparsity modules 1266 (individually referred to as "sparsity module 1266"). In other embodiments, the sparse cell 1200 may include fewer, more, or different components. For example, the sparse cell 1200 may include a different number of MAC units 1210, weight register files 1220, activation register files 1230, row buffers 1240, or sparsity modules 1266. As another example, the sparse cell 1200 may include column buffers in lieu of or in addition to the row buffers 1240. Also, the shape (e.g., the height or width) of the MAC array may be different.

[0127] The MAC units 1210 are configured to perform MAC operations. Each MAC unit 1210 may include one or more multipliers and one or more adders. A multiplier may multiply an activation with a weight at a time to compute a product. In some embodiments (e.g., embodiments where the MAC unit 1210 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. An adder may accumulate products computed by the multipliers. Even though not shown in FIG. 12, the sparse cell may include an adder tree including a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units 1210. The number of adders in the first tier may be half of the number of the MAC units 1210, and each adder may accumulate the outputs of two MAC units 1210. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the sparse cell 1200.

[0128] The weight register files 1220 store weights to be processed in MAC operations. In the embodiments of FIG. 12, four weight register files 1220 are grouped into a storage set that stores data to be used by a column of MAC units 1210. There are four storage sets corresponding to the four columns of MAC units 1210. In some embodiments, a weight register file 1220 may correspond to a MAC unit 1210 and store data to be processed by the MAC unit. In some embodiments, the four weight register files 1220 for a single column of MAC units 1210 constitute a data storage unit of the column.

[0129] The activation register files 1230 stores activations to be processed in MAC operations. In the embodiments of FIG. 12, four activation register files 1230 are grouped into a storage set that stores data to be used by a row of MAC units 1210. There are four storage sets corresponding to the four rows of MAC units 1210. In some embodiments, an activation register file 1230 may correspond to a MAC unit 1210 and store data to be processed by the MAC unit. In some embodiments, the four activation register files 1230 for a single row of MAC units 1210 constitute a data storage unit of the row. The row buffers 1240 store outputs of the MAC units 1210. Each row buffer 1240 may drain outputs of a single row of MAC units 1210.

[0130] The sparsity module 1266 facilitates dynamic sparsity-based acceleration or workload distribution in the sparse cell 1200. In the embodiments of FIG. 12, each sparsity module 1266 includes a sparsity tensor storage unit 1265 and a control logic 1267. The sparsity tensor storage unit 1265 stores combined sparsity tensors. A combined sparsity tensor stored in the sparsity tensor storage unit 1265 may correspond to an activation tensor and a weight tensor. A nonzero element in the combined sparsity tensor may correspond to a nonzero activation-weight pair that includes a nonzero activation and a nonzero weight. The position of the nonzero activation in the activation tensor may match the position of the nonzero weight in the weight tensor. The product of the nonzero activation and nonzero weight would be nonzero.

[0131] The control logic 1267 may control transmission of activations and weights stored from the weight register files 1220 and the activation register files 1230 to the MAC units 1210 based on sparsity tensors. For instance, the control logic 1267 may select a subset of the weights stored in the weight register files 1220 and select a subset of activations stored in the activation register files 1230 based on a sparsity tensor. The control logic 1267 may transmit the selected weights and activations to the MAC units 1210 for performing MACoperations. The other weights stored in the weight register files 1220 or the other activations stored in the activation register files 1230 are skipped from computation. In the embodiments of FIG. 12, each sparsity module 1266 controls sparsity acceleration or workload distribution in a respective MAC unit 1210. As the sparsity acceleration or workload distribution is either based on both weight sparsity and activation sparsity, 16 sparsity modules 1266 are used for acceleration computations in the 16 MAC units 1210.

[0132] As shown in FIG. 12, the sparse cell 1200 is associated with multiplexers (MUXs) 1203, 1204, 1205, and 1206. In other embodiments, the sparse cell 1200 may be associated with a different number of MUXs or other devices. The MUX 1203 facilitates loading weights, e.g., from the memory 550, into the weight register files 1220. The MUX 1204 facilitates loading activations, e.g., from the memory 550, into the activation register files 1230. The MUX 1205 facilitates loading sparsity tensors into the sparsity tensor storage unit 1265. The MUX 1206 may be a drain MUX that can facilitate draining outputs of the MAC units 1210, e.g., to the memory 550.

[0133] FIG. 13 is a flowchart of a method 1300 for DSP self-clock gating, in accordance with various embodiments. The method 1300 may be performed by the clock gate controller 930 in FIG. 9. Although the method 1300 is described with reference to the flowchart illustrated in FIG. 13, many other methods for executing neural network layer may alternatively be used. For example, the order of execution of the steps in FIG. 13 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0134] The clock gate controller 930 receives 1310 a gate enable signal from a DSP. An operation of the DSP for executing the neural network is synchronized by a clock signal generated by a clock.

[0135] The clock gate controller 930, in response to receiving the gate enable signal, commands 1320 a clock gate to stop the clock from providing the clock signal. In some embodiments, the clock gate controller 930, in response to receiving the gate enable signal, determines whether there is any external request to be finished by the DSP. In response to determining that there is no external request to be finished by the DSP, the clock gate controller 930 commands the clock gate to stop the clock from generating the clock signal.

[0136] In some embodiments, the clock gate controller 930 determines whether there is any external request to be finished by the DSP by determining whether an interface of a configuration space register associated with the DSP is active. In some embodiments, theclock gate controller 930 determines whether there is any external request to be finished by the DSP based on a signal from the DSP, the signal indicating whether there is any pending memory access.

[0137] The clock gate controller 930 receives 1330 a signal indicating that a trigger event has occurred. In some embodiments, the signal indicates that a hardware barrier, which is configured to prevent a computational task in an execution of a neural network from being performed, has been lifted. In some embodiments, the signal indicates that a FIFO-out unit has dispatched a computational task in an execution of a neural network to the DSP. In some embodiments, the clock gate controller 930 receives the signal from an event detector. The event detector is programmed by the DSP to send out the signal after the trigger event occurs. In some embodiments, the DSP programs the event detector with a type of the trigger event.

[0138] The clock gate controller 930, in response to receiving the signal, sends 1340 a clock awaken signal to the DSP. The clock is to resume the clock signal in response to the clock awaken signal. In some embodiments, the gate enable signal is received from an output port of the DSP. The gate enable signal is cleared at the output port in response to the clock awaken signal.

[0139] FIG. 14 is a block diagram of an example computing device 1400, in accordance with various embodiments. In some embodiments, the computing device 1400 can be used as at least part of the DNN system 500 in FIG. 5. A number of components are illustrated in FIG. 14 as included in the computing device 1400, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 1400 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1400 may not include one or more of the components illustrated in FIG. 14, but the computing device 1400 may include interface circuitry for coupling to the one or more components. For example, the computing device 1400 may not include a display device 1406, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1406 may be coupled. In another set of examples, the computing device 1400 may not include an audio input device 1418 or an audio output device 1408 but may include audio input or output device interface circuitry (e.g.,connectors and supporting circuitry) to which an audio input device 1418 or audio output device 1408 may be coupled.

[0140] The computing device 1400 may include a processing device 1402 (e.g., one or more processing devices). The processing device 1402 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 1400 may include a memory 1404, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 1404 may include memory that shares a die with the processing device 1402. In some embodiments, the memory 1404 includes one or more non-transitory computer- readable media storing instructions executable to perform operations for DSP self-clock gating (e.g., the method 1300 described in conjunction with FIG. 13) or some operations performed by one or more components of the DNN system 500. The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1402.

[0141] In some embodiments, the computing device 1400 may include a communication chip 1412 (e.g., one or more communication chips). For example, the communication chip 1412 may be configured for managing wireless communications for the transfer of data to and from the computing device 1400. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0142] The communication chip 1412 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for productsthat pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1412 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 1412 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1412 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 1412 may operate in accordance with other wireless protocols in other embodiments. The computing device 1400 may include an antenna 1422 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0143] In some embodiments, the communication chip 1412 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 1412 may include multiple communication chips. For instance, a first communication chip 1412 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1412 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 1412 may be dedicated to wireless communications, and a second communication chip 1412 may be dedicated to wired communications.

[0144] The computing device 1400 may include battery / power circuitry 1414. The battery / power circuitry 1414 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1400 to an energy source separate from the computing device 1400 (e.g., AC line power).

[0145] The computing device 1400 may include a display device 1406 (or corresponding interface circuitry, as discussed above). The display device 1406 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreendisplay, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0146] The computing device 1400 may include an audio output device 1408 (or corresponding interface circuitry, as discussed above). The audio output device 1408 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0147] The computing device 1400 may include an audio input device 1418 (or corresponding interface circuitry, as discussed above). The audio input device 1418 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0148] The computing device 1400 may include a GPS device 1416 (or corresponding interface circuitry, as discussed above). The GPS device 1416 may be in communication with a satellite-based system and may receive a location of the computing device 1400, as known in the art.

[0149] The computing device 1400 may include another output device 1410 (or corresponding interface circuitry, as discussed above). Examples of the other output device 1410 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0150] The computing device 1400 may include another input device 1420 (or corresponding interface circuitry, as discussed above). Examples of the other input device 1420 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0151] The computing device 1400 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In someembodiments, the computing device 1400 may be any other electronic device that processes data.

[0152] The following paragraphs provide various examples of the embodiments disclosed herein.

[0153] Example 1 provides a method for executing a neural network, including receiving a gate enable signal from a DSP, in which an operation of the DSP for executing the neural network is synchronized by a clock signal generated by a clock; in response to receiving the gate enable signal, commanding a clock gate to stop the clock from providing the clock signal; receiving a signal indicating that a trigger event has occurred; and in response to receiving the signal, sending a clock awaken signal to the DSP, the clock to resume the clock signal in response to the clock awaken signal.

[0154] Example 2 provides the method of example 1, in which commanding the clock gate to stop the clock from generating the clock signal includes in response to receiving the gate enable signal, determining whetherthere is any external request to be finished by the DSP; and in response to determining that there is no external request to be finished by the DSP, commanding the clock gate to stop the clock from generating the clock signal.

[0155] Example 3 provides the method of example 2, in which determining whether there is any external request to be finished by the DSP includes determining whether an interface of a configuration space register associated with the DSP is active.

[0156] Example 4 provides the method of example 2 or 3, in which determining whether there is any external request to be finished by the DSP includes determining whether there is any external request to be finished by the DSP based on a signal from the DSP, the signal indicating whether there is any pending memory access.

[0157] Example 5 provides the method of any one of examples 1-4, in which the signal indicates that a hardware barrier, which is configured to prevent a computational task in an execution of a neural network from being performed, has been lifted.

[0158] Example 6 provides the method of any one of examples 1-5, in which the signal indicates that a FIFO-out unit has dispatched a computational task for executing the neural network to the DSP.

[0159] Example 7 provides the method of any one of examples 1-6, in which receiving the signal includes receiving the signal from an event detector, the event detector programmed by the DSP to send out the signal after the trigger event occurs.

[0160] Example 8 provides the method of example 7, in which the DSP programs the event detector with a type of the trigger event.

[0161] Example 9 provides the method of any one of examples 1-8, in which the gate enable signal is received from an output port of the DSP, and the gate enable signal is cleared at the output port in response to the clock awaken signal.

[0162] Example 10 provides the method of any one of examples 1-9 in which the clock is enabled by a firmware before the gate enable signal is received from the DSP.

[0163] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations for executing a neural network, the operations including receiving a gate enable signal from a DSP, in which an operation of the DSP for executing the neural network is synchronized by a clock signal generated by a clock; in response to receiving the gate enable signal, commanding a clock gate to stop the clock from providing the clock signal; receiving a signal indicating that a trigger event has occurred; and in response to receiving the signal, sending a clock awaken signal to the DSP, the clock to resume the clock signal in response to the clock awaken signal.

[0164] Example 12 provides the one or more non-transitory computer-readable media of example 11, in which commanding the clock gate to stop the clock from generating the clock signal includes in response to receiving the gate enable signal, determining whether there is any external request to be finished by the DSP; and in response to determining that there is no external request to be finished by the DSP, commanding the clock gate to stop the clock from generating the clock signal.

[0165] Example 13 provides the one or more non-transitory computer-readable media of example 12, in which determining whether there is any external request to be finished by the DSP includes determining whether an interface of a configuration space register associated with the DSP is active.

[0166] Example 14 provides the one or more non-transitory computer-readable media of example 12 or 13, in which determining whetherthere is any external request to be finished by the DSP includes determining whether there is any external request to be finished by the DSP based on a signal from the DSP, the signal indicating whether there is any pending memory access.

[0167] Example 15 provides the one or more non-transitory computer-readable media of any one of examples 11-14, in which the signal indicates that a hardware barrier, which isconfigured to prevent a computational task in an execution of a neural network from being performed, has been lifted.

[0168] Example 16 provides the one or more non-transitory computer-readable media of any one of examples 11-15, in which the signal indicates that a FIFO-out unit has dispatched a computational task for executing the neural network to the DSP.

[0169] Example 17 provides the one or more non-transitory computer-readable media of any one of examples 11-16, in which receiving the signal includes receiving the signal from an event detector, the event detector programmed by the DSP to send out the signal after the trigger event occurs.

[0170] Example 18 provides the one or more non-transitory computer-readable media of example 17, in which the DSP programs the event detector with a type of the trigger event.

[0171] Example 19 provides the one or more non-transitory computer-readable media of any one of examples 11-18, in which the gate enable signal is received from an output port of the DSP, and the gate enable signal is cleared at the output port in response to the clock awaken signal.

[0172] Example 20 provides the one or more non-transitory computer-readable media of example 19, in which the clock is enabled by a firmware before the gate enable signal is received from the DSP.

[0173] Example 21 provides an apparatus, including a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations including receiving a gate enable signal from a DSP, an operation of the DSP synchronized by a clock signal generated by a clock, in response to receiving the gate enable signal, commanding a clock gate to stop the clock from providing the clock signal, receiving a signal indicating that a trigger event has occurred, and in response to receiving the signal, sending a clock awaken signal to the DSP, the clock to resume the clock signal in response to the clock awaken signal.

[0174] Example 22 provides the apparatus of example 21, in which commanding the clock gate to stop the clock from generating the clock signal includes in response to receiving the gate enable signal, determining whether there is any external request to be finished by the DSP; and in response to determining that there is no external request to be finished by the DSP, commanding the clock gate to stop the clock from generating the clock signal.

[0175] Example 23 provides the apparatus of example 21 or 22, in which the signal indicates that a hardware barrier, which is configured to prevent a computational task in an execution of a neural network from being performed, has been lifted.

[0176] Example 24 provides the apparatus of any one of examples 21-23, in which the signal indicates that a FIFO-out unit has dispatched a computational task in an execution of a neural network to the DSP.

[0177] Example 25 provides the apparatus of any one of examples 21-24, in which receiving the signal includes receiving the signal from an event detector, the event detector programmed by the DSP to send out the signal after the trigger event occurs.

[0178] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims1. A method for executing a neural network, comprising: receiving a gate enable signal from a digital signal processor, wherein an operation of the digital signal processor for executing the neural network is synchronized by a clock signal generated by a clock; in response to receiving the gate enable signal, commanding a clock gate to stop the clock from providing the clock signal; receiving a signal indicating that a trigger event has occurred; and in response to receiving the signal, sending a clock awaken signal to the digital signal processor, the clock to resume the clock signal in response to the clock awaken signal.

2. The method of claim 1, wherein commanding the clock gate to stop the clock from generating the clock signal comprises: in response to receiving the gate enable signal, determining whether there is any external request to be finished by the digital signal processor; and in response to determining that there is no external request to be finished by the digital signal processor, commanding the clock gate to stop the clock from generating the clock signal.

3. The method of claim 2, wherein determining whether there is any external request to be finished by the digital signal processor comprises: determining whether an interface of a configuration space register associated with the digital signal processor is active.

4. The method of claim 2 or 3, wherein determining whether there is any external request to be finished by the digital signal processor comprises: determining whether there is any external request to be finished by the digital signal processor based on a signal from the digital signal processor, the signal indicating whether there is any pending memory access.

5. The method of any one of claims 1-4, wherein the signal indicates that a hardware barrier, which is configured to prevent a computational task in an execution of a neural network from being performed, has been lifted.

6. The method of any one of claims 1-5, wherein the signal indicates that a first-in-first-out unit has dispatched a computational task for executing the neural network to the digital signal processor.

7. The method of any one of claims 1-6, wherein receiving the signal comprises: receiving the signal from an event detector, the event detector programmed by the digital signal processor to send out the signal after the trigger event occurs.

8. The method of claim 7, wherein the digital signal processor programs the event detector with a type of the trigger event.

9. The method of any one of claims 1-8, wherein the gate enable signal is received from an output port of the digital signal processor, and the gate enable signal is cleared at the output port in response to the clock awaken signal.

10. The method of any one of claims 1-9 wherein the clock is enabled by a firmware before the gate enable signal is received from the digital signal processor.

11. One or more non-transitory computer-readable media storing instructions executable to perform operations for executing a neural network, the operations comprising: receiving a gate enable signal from a digital signal processor, wherein an operation of the digital signal processor for executing the neural network is synchronized by a clock signal generated by a clock; in response to receiving the gate enable signal, commanding a clock gate to stop the clock from providing the clock signal; receiving a signal indicating that a trigger event has occurred; and in response to receiving the signal, sending a clock awaken signal to the digital signal processor, the clock to resume the clock signal in response to the clock awaken signal.

12. The one or more non-transitory computer-readable media of claim 11, wherein commanding the clock gate to stop the clock from generating the clock signal comprises: in response to receiving the gate enable signal, determining whether there is any external request to be finished by the digital signal processor; and in response to determining that there is no external request to be finished by the digital signal processor, commanding the clock gate to stop the clock from generating the clock signal.

13. The one or more non-transitory computer-readable media of claim 12, wherein determining whether there is any external request to be finished by the digital signal processor comprises: determining whether an interface of a configuration space register associated with the digital signal processor is active.

14. The one or more non-transitory computer-readable media of claim 12 or 13, wherein determining whether there is any external request to be finished by the digital signal processor comprises: determining whether there is any external request to be finished by the digital signal processor based on a signal from the digital signal processor, the signal indicating whether there is any pending memory access.

15. The one or more non-transitory computer-readable media of any one of claims 11-14, wherein the signal indicates that a hardware barrier, which is configured to prevent a computational task in an execution of a neural network from being performed, has been lifted.

16. The one or more non-transitory computer-readable media of any one of claims 11-15, wherein the signal indicates that a first-in-first-out unit has dispatched a computational task for executing the neural network to the digital signal processor.

17. The one or more non-transitory computer-readable media of any one of claims 11-16, wherein receiving the signal comprises: receiving the signal from an event detector, the event detector programmed by the digital signal processor to send out the signal after the trigger event occurs.

18. The one or more non-transitory computer-readable media of claim 17, wherein the digital signal processor programs the event detector with a type of the trigger event.

19. The one or more non-transitory computer-readable media of any one of claims 11-18, wherein the gate enable signal is received from an output port of the digital signal processor, and the gate enable signal is cleared at the output port in response to the clock awaken signal.

20. The one or more non-transitory computer-readable media of claim 19, wherein the clock is enabled by a firmware before the gate enable signal is received from the digital signal processor.

21. An apparatus for executing a neural network, the apparatus comprising: a clock configured to generate a clock signal; a digital signal processor, wherein an operation of the digital signal processor for executing the neural network is synchronized by the clock signal; a clock gate; and a gate controller configured to: receive a gate enable signal from the digital signal processor, in response to receiving the gate enable signal, command the clock gate to stop the clock from providing the clock signal, receive a signal indicating that a trigger event has occurred, and in response to receiving the signal, send a clock awaken signal to the digital signal processor, wherein the clock to resume the clock signal in response to the clock awaken signal.

22. The apparatus of claim 21, wherein the gate controller is configured to command the clock gate to stop the clock from generating the clock signal by: in response to receiving the gate enable signal, determining whether there is any external request to be finished by the digital signal processor; and in response to determining that there is no external request to be finished by the digital signal processor, commanding the clock gate to stop the clock from generating the clock signal.

23. The apparatus of claim 21 or 22, further comprising a hardware barrier configured to prevent a computational task in an execution of a neural network from being performed, wherein the signal indicates that the hardware barrier has been lifted.

24. The apparatus of any one of claims 21-23, further comprising a first-in-first-out unit configured to dispatch one or more computational tasks for executing the neural network to the digital signal processor, wherein the signal indicates that the first-in-first-out unit has dispatched a computational task to the digital signal processor.

25. The apparatus of any one of claims 21-24, further comprising an event detector programmed by the digital signal processor to send out the signal after the trigger event occurs, wherein the gate controller is configured to receive the signal by receiving the signal from the event detector.

Images