Execution of transformer-based model on neural network accelerator

Abstract

Dynamic input sequences of a decoder in a transformer model may be converted to a static input sequence through padding. The dimension of the padded input sequence may remain the same cross multiple inference phases, but the tokens in the padded input sequence may change. The padded input sequence includes one or more input tokens and one or more padding tokens. An attention mask may be generated and indicate positions of input tokens in the padded input sequence. Positions IDs may be determined and indicate positions of tokens in the token sequence predicted by the transformer model. A KV cache may be padded so that a dimension of the KV cache may match the dimension of the padded input sequence. The KV cache may store self-attention key tensors or self-attention value tensors. The padded input sequence, attention mask, position ID, or KV cache may be updated after each inference phase.

Description

EXECUTION OF TRANSFORMER-BASED MODEL ON NEURAL NETWORK ACCELERATORCross-Reference to Related Application

[0001] This application claims the benefit of and hereby incorporates by reference, for all purposes, the entirety of the contents of International Application No. PCT / CN2024 / 078732, filed February 27, 2024, and entitled "EXECUTION OF LARGE LANGUAGE MODELS WITH DEEP NEURAL NETWORK ACCELERATORS."Technical Field

[0002] This disclosure relates generally to neural network (also referred to as "deep neural network" or "DNN"), and more specifically, execution of transformer-based models on DNN accelerators.Background

[0003] DNNs are used extensively for a variety of artificial intelligence applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write. Therefore, techniques to improve efficiency of DNNs are needed.Brief Description of the Drawings

[0004] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0005] FIG. 1 illustrates an example transformer model, in accordance with various embodiments.

[0006] FIG. 2 illustrates an example embedding layer, in accordance with various embodiments.

[0007] FIG. 3 illustrates positional encoding, in accordance with various embodiments.

[0008] FIGS. 4A and 4B illustrate an example multi-head attention (MHA) layer, in accordance with various embodiments.

[0009] FIG. 5 illustrates an example linear classifier, in accordance with various embodiments.

[0010] FIG. 6 illustrates a first inference phase of a transformer model, in accordance with various embodiments.

[0011] FIG. 7 illustrates subsequent inference phases of the transformer model, in accordance with various embodiments.

[0012] FIG. 8 illustrates computations in a self-attention layer without key-value (KV) caching, in accordance with various embodiments.

[0013] FIG. 9 illustrates computations in a self-attention layer with KV caching, in accordance with various embodiments.

[0014] FIG. 10 illustrates a first inference phase of a transformer model with static shaped inputs, in accordance with various embodiments.

[0015] FIG. 11 illustrates cache updating in a transformer model with static shaped inputs, in accordance with various embodiments.

[0016] FIG. 12 illustrates a subsequent inference phase in a transformer model with static shaped inputs, in accordance with various embodiments.

[0017] FIG. 13 illustrates attention mask updating in a transformer model with static shaped inputs, in accordance with various embodiments.

[0018] FIG. 14 illustrates cache padding, in accordance with various embodiments.

[0019] FIGS. 15A and 15B illustrate graph modification for a transformer model with static shaped inputs, in accordance with various embodiments.

[0020] FIG. 16 illustrates an example convolution, in accordance with various embodiments.

[0021] FIG. 17 is a block diagram of a DNN system, in accordance with various embodiments.

[0022] FIG. 18 is a block diagram of a DNN module, in accordance with various embodiments.

[0023] FIG. 19 illustrates an example sparse cell, in accordance with various embodiments.

[0024] FIG. 20 illustrates an example sparse cell array, in accordance with various embodiments.

[0025] FIG. 21 illustrates an example processing element (PE), in accordance with various embodiments.

[0026] FIG. 22 is a flowchart of a method for executing a transformer model, in accordance with various embodiments.

[0027] FIG. 23 is a block diagram of an example computing device, in accordance with various embodiments.Detailed DescriptionOverview

[0028] The last decade has witnessed a rapid rise in artificial intelligence (Al) based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more deep learning operations (also referred to as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on.

[0029] Input or output data of deep learning operations may be arranged in data structures called tensors. A tensor is a data structure having multiple elements across one or more dimensions. Examples of tensors include vector (which is one-dimensional (ID) tensor), matrix (which is two-dimensional (2D) tensor), three-dimensional (3D) tensors, fourdimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along the axis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. Taking a convolutional layer for example, the input tensors include an activation tensor (also referred to as "input feature map (IFM)" or "input activation tensor") including one or more activations (also referred to as "input elements") and a weight tensor. The weight tensor may be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a group of filters (a 4D weight tensor). A convolution may be performed on the input activation tensor and weight tensor to compute an output activation tensor in the convolutional layer.

[0030] Tensors in DNNs can be saved in X-major (e.g., XYZ or XZY format), Y-major formats (e.g., YXZ or YZX format), or Z-major formats (e.g., ZXY or ZYX format). The format of a tensor may define the order in which the data points in the tensor are stored, written, or read. The first character may represent the dimension in which data points are contiguous in memory. The second character may represent the dimension in which data points can be accessed after the contiguous data points are accessed in memory. The third character may represent the dimension in which data points are accessed after the data points in the dimension represented by the second character are exhausted. Taking the ZXY format for example, the access order first starts in the Z dimension, then moves to the X dimension, and finally moves to the Y dimension. Data points in the tensor are contiguous in memory in the Z dimension, meaning data points having the same (x, y) coordinates are contiguous in memory. Using tensor permutation, the tensor may be read from memory in a different format.

[0031] The significant improvements in DNN model size and accuracy coupled with the rapid increase in computing power of execution platforms have led to the adoption of DNN applications even within resource constrained mobile and edge devices that have limited energy availability. DNN models may be executed, e.g., for training or inference, by DNN accelerators. A DNN accelerator may be or include one or more data processing units. A data processing unit may also be referred to as a compute block or compute tile. A data processing unit may include PEs that can carry out neural network operations.

[0032] Large Language Models (LLMs), including transformer-based neural networks like GPT (Generative Pre-trained Transformer) and BERT (Bidirectional Encoder Representations from Transformers) are used for Natural Language Processing (NLP) in various applications, from virtual assistants, document summarization, to coding assistants, language translation and speech recognition apps. Despite the remarkable capabilities of LLMs, deploying them and executing LLM inference on end user devices with DNN accelerators is not trivial due to their high computational intensity, expensive memory requirements, dynamic nature of input and output shapes, large sequence lengths, in addition to the inherent autoregressive nature of LLM generation stage.

[0033] The demand for computational resources during LLM inference depends on numerous factors, including LLM model architecture, number of parameters, and application. The term "large" in LLM signifies the scale of these models, encompassing boththe extensive number of model parameters (weights), often reaching scales of billions of parameters, and the vast corpus of training data. In addition, the number of floating-point operations (FLOPs) is primarily determined by the input sequence length and the model size. The compute complexity of the MHA module in encoder and decoder scales quadratically with the input sequence length. Also, substantial portions of text may result in potentially thousands of words / tokens (e.g., in content summarization) leading to high sequence length, which leads to high compute demand. This challenge results in high inference latency and high energy consumption.

[0034] LLMs usually exhibit a substantial memory footprint, stemming from the need to store extensive pre-trained model parameters (in billions) and maintain transient states during autoregressive generation phase, particularly during decoder with KV cache inference (cached inference) phase. These voluminous parameters not only surpass the capacity of standard device memory but also pose bottlenecks to the generation throughput. This challenge also impacts the energy consumption of the accelerator.

[0035] Dynamic shaped tensors in LLMs pose significant challenges for DNN accelerators due to their inherent complexity and variability. In contrast to computer vision applications where images conform to a fixed tensor shape, language expressions in words and sentences inherently exhibit dynamic characteristics. The length of a sentence fluctuates based on context, making it impractical to assume a constant length or predict the length for every sentence. This dynamic nature for LLMs poses a challenge for graph-based compilers commonly used in DNN accelerators. These accelerators typically rely on hardware architectures designed for specific tensor dimensions, making it difficult to adapt to the constantly changing input sizes of LLMs. Furthermore, dynamic shapes introduce irregular memory access patterns and unpredictable computational workloads, exacerbating the challenge for accelerators to efficiently utilize resources and achieve optimal performance.

[0036] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing execution of transformer models with static shaped KV caches and inputs. Static shaped KV caches and inputs may be implemented through padding. Keys and values, including self-attention keys and selfattention values, in transformer models may be cached and reused for cached inference oftransformer models. Attention masks and position IDs may be generated and used to facilitate cached inference with static shaped KV caches and inputs.

[0037] An example transformer model includes an encoder, a decoder, and a head. An inference request may be made to request the transformer model to make a prediction. During inference, the encoder may receive input tokens and generate an encoder output from the input tokens. A token may be a representation (e.g., numerical representation) of a signal (e.g., a word, image, audio, video, etc.) provided to the transformer model. The decoder may receive dynamic inputs and generate decoder outputs from the dynamic inputs and the encoder output during a sequence of inference phases. The dynamic inputs may be dynamic-shaped inputs. For instance, the shape of input tensor may be dynamic and vary. An inference phase may be a cycle of executing operations in the decoder and head to compute a single output tensor. The output tensor may be a token. An inference phase may also be referred to as an inference cycle. All the inference phases may constitute a whole inference process. The head may generate output tokens from the decoder outputs during the inference phases. For instance, the head may output a new token in each inference phase. A sequence of tokens may be obtained after all the inference phases are complete. The sequence of tokens ("output sequence") may be the prediction of the transformer model.

[0038] The decoder inputs may be dynamic as the token predicted in an inference phase is added to the decoder input for the next inference phase. The shape of the decoder input sequence is therefore dynamic across the inference phases. In various embodiments of the present disclosure, dynamic shaped decoder input may be converted to static shaped decoder input through padding. For the first inference phase, the initial input sequence may be padded by adding one or more padding tokens to the input sequence, result in a padded input sequence. Padding tokens may be skipped for computations in the decoder. After a new token is predicted, a padding token may be replaced with the new token to produce the input for the next inference phase. The padded input sequence can therefore have a static dimension across the inference phases. An attention mask may be generated for the padded input sequence. The attention mask may be a tensor with elements indicating positions of the input tokens or padding tokens in the padded input sequence. Positions IDs of input tokens may also be generated. Position IDs are positional parameters of tokens that indicate positions of the tokens in the output sequence of the transformer model.

[0039] One or more KV caches may be padded so that a dimension of the KV cache may match the corresponding dimension of the padded input sequence. The KV cache may store attention tensors, such as self-attention key tensors, self-attention value tensors, etc. The attention mask, position ID, and KV cache may be updated after each inference phase. With KV caching, the inference phase may be carried out by using the token predicted in the last inference phase, as opposed to the entire padded input sequence. As the KV cache has a static shape across multiple or even all the inference phase, the transformer model may be efficiently executed by a DNN accelerator.

[0040] In addition to padding, this disclosure provides a number of other approaches for efficient execution of transformer models on DNN accelerators, such as memory-efficient KV cache creation, graph modifications to meet accuracy constraints, and runtime input data and KV cache modification schemes. The disclosure also presents techniques for optimal data layout allocation, precision determination, efficient data handling, and the use of multiple decoder models. These comprehensive approaches can significantly enhance the performance and efficiency of transformer models. These techniques are designed to be generalizable, offering versatility across various transformer model architectures. These techniques can enable efficient execution of transformer models on client devices by leveraging the low power inference capability of the DNN accelerator.

[0041] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0042] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0043] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.However, the order of description should not be construed as to imply that these operationsare necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0044] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0045] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0046] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0047] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0048] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expresslylisted or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0049] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.Example Transformer Models

[0050] FIG. 1 illustrates an example transformer model 100, in accordance with various embodiments. The transformer model 100 may transform input sequences into output sequences. In some embodiments, the transformer model 100 is a neural network that can learn context and meaning by tracking relationships in sequential data, such as sequential words in a sentence, sequential audio signals, sequential images, and so on. In an example, the transformer model 100 may be an LLM. The transformer model 100 includes an encoder block 110, a decoder block 120, and a head block 130. In other embodiment, different or additional components may be included in the transformer model 100. Further, functionality attributed to a component of the transformer model 100 may be accomplished by a different component included in the transformer model 100 or a different model or module.

[0051] The encoder block 110 receives input sequences and generates matrix representations of the input sequences. In the embodiments of FIG. 1, the encoder block 110 receives inputs 101 and generates encoder outputs 102. In some embodiments, the inputs 101 may include one or more input tokens, such as words, phrases, sentences, images, audio signals, other types of input tokens, or some combination thereof. In an example, the inputs 101 may include a prompt received from a user of the transformer model 100. The prompt may include a question or request made by the user. A word in the prompt may be an input token. The encoder outputs 102 may include one or more vectors that are contextualized representations of the input 101. Each vector in the encoder outputs 102 may represent a token in the input 101 with contextual understanding.

[0052] The encoder block 110 includes an embedding layer 113, a positional encoding layer 115, and a plurality of layers 140 (individually referred to as "layer 140"). In other embodiments, the encoder block 110 may have different, fewer, or more components. Also, the arrangement of the components in the encoder block 110 may be different from thearrangement shown in FIG. 1. For the purpose of illustration, the encoder block 110 has N layers in FIG. 1, where N is an integer. Each layer 140 may include one or more neural network operations. The layers 140 may transform a sequence of embeddings into a representation that encapsulates the learned information from the input 101. Different layers 140 may have different internal parameters, e.g., different weights, bias, or other types of internal parameters. In some embodiments, the layers 140 have identical components. The components in a layer 140 may be layers and may also be referred to as sub-layers of the layer 140. As shown in FIG. 1, a layer 140 includes four sub-layers: an MHA layer 141, an add & norm layer 142, a feed forward layer 143, and another add & norm layer 144.

[0053] The decoder block 120 iteratively generates outputs 103 using encoded representations generated by the encoder block 110. The decoder block 120 includes an embedding layer 123, a positional encoding layer 125, and a plurality of layers 150 (individually referred to as "layer 150"). For the purpose of illustration, the decoder block 120 has N layers in FIG. 1, where N is an integer. In the embodiments of FIG. 2, the number of layers 150 in the decoder block 120 is the same as the number of layers 140 in the encoder block 110. In other embodiments, the number of layers 150 in the decoder block 120 may be different from the number of layers 140 in the encoder block 110. Each layer 150 may include one or more neural network operations. Different layers 150 may have different internal parameters. In some embodiments, the layers 150 may have identical components. The components in a layer 150 may be layers and may also be referred to as sub-layers of the layer 150. As shown in FIG. 1, a layer 150 includes six sub-layers: an MHA layer 151, an add & norm layer 152, an encoder-decoder attention layer 153, another add & norm layer 154, a feed forward layer 155, and another add & norm layer 156.

[0054] In some embodiments, a sequence of inference phases is performed in the decoder block 120 using encoder outputs, e.g., the encoder outputs 102. A matrix may be predicted through each inference phase. The outputs 103 may include a plurality of matrices. Each matrix may be further processed in the head block 130 to predict a token. The plurality of matrices may be used to predict a sequence of tokens. For the first inference phase, the decoder block 120 may receive one or more start tokens as input tokens and compute a first matrix from the input tokens and the output of the encoder block 110. The first matrix may be used by the head block 130 to predict a first token. The predicted token may be used as anew input token, in addition to the start token(s), in the second inference phase. Similarly, a second token may be predicted through the second inference phase and may be used in the third inference phase. This iteration may continue till all the inference phases are complete.

[0055] The head block 130 receives the output of the decoder block 120 and processes it in a linear layer 133 and a SoftMax layer 135. A linear operation may be performed on the output of the decoder block 120 in the linear layer 133. The linear operation may include a multiplication of the output of the decoder block 120 with a weight matrix. The output of the linear layer 133 may be a vector. In some embodiments, the head block 130 may function as a classifier. The number of data elements in the vector computed in the linear layer 133 may depend on the number of classes involved. In an example where there are M classes, where M is an integer, the vector computed in the linear layer 133 may have M data elements representing the prediction for the M classes, respectively.

[0056] The output of the linear layer 133 may be input into the SoftMax layer 135. A SoftMax function may be applied on the output of the linear layer 133 to compute probability scores. A probability score may have a value in the range from 0 to 1. In some embodiments, a probability value is computed for each data element in the vector computed in the linear layer 133. The highest one of the probability scores may be the key. The corresponding index of the key may point to the token that the transformer model 100 predicts as the next in the sequence. The final output of the transformer model 100 may be the sequence of predicted tokens. In some embodiments, the head block 130 may be a language modeling head.

[0057] An embedding layer (e.g., the embedding layer 113 or the embedding layer 123) converts an input of the embedding layer (e.g., the inputs 101 or the outputs 103) into one or more embeddings. An embedding may be a vector, which is also referred to as an embedding vector or a vector embedding. The vector embedding may include a sequence of data elements. In some embodiments, the embedding layer 113 may generate a plurality of embeddings, each of which may be converted from a different input token in the inputs 101. The embeddings may capture the semantic meaning of the tokens in the input 101. The embeddings may be numerical representations that capture the relationships or meanings of words, phrases, or other data types. In an example where the input 101 is a prompt including a sequence of words, the embedding layer 113 may generate an embedding from each word in the input 101. The embedding layer 123 in the decoder block 120 maygenerate a plurality of embeddings from tokens received by the decoder block 120 in a similar manner as the embedding layer 113. Certain aspects of embedding layers are described below in conjunction with FIG. 2.

[0058] A positional encoding layer (e.g., the positional encoding layer 115 or the positional encoding layer 125) performs positional encoding on embeddings generated in the corresponding embedding layer. In some embodiments, the positional encoding layer may apply one or more positional encoding vectors (e.g., a positional encoding vector 104 or positional encoding vector 105) on vector embeddings from the corresponding embedding layer to generate new vector embeddings that represents the embeddings with positional context. The positional encoding vector may encode information about the position of the embedding in a sequence of embeddings. In some embodiments, the positional encoding layer performs an addition operation on a positional encoding vector and a vector embedding. The addition operation may be elementwise addition. The positional encoding layer may output an embedding matrix that includes the vector embeddings computed in the positional encoding layer. Certain aspects of positional encoding layers are described below in conjunction with FIG. 3.

[0059] An MHA layer (e.g., the MHA layer 141, the MHA layer 151, or the MHA layer 153) may implement a multi-head attention mechanism, which may be a multi-head selfattention mechanism or a multi-head cross-attention mechanism. In some embodiments, the MHA layer 141 or the MHA layer 151 may implement a self-attention mechanism. For self-attention, the queries, keys, and values may come from the same place. For instance, for the MHA layer 141, the queries, keys, and values may all come from the positional encoding layer 115. For the MHA layer 151, the queries, keys, and values may all come from the positional encoding layer 125. The self-attention mechanism may enable the transformer model 100 to relate each token with other tokens. The MHA layer may compute attention scores from embeddings generated in the corresponding positional encoding layer. In some embodiments, the MHA layer may receive one or more queries, one or more keys, and one or more values. In some embodiments, the MHA layer has a number of heads that receive different linearly projected versions of the queries, keys, and values and produce outputs in parallel that are then used to generate the final result.

[0060] In some embodiments, the queries, keys, and values input into the MHA layer 141 may be computed from vector embeddings generated by the positional encoding layer 115.The queries, keys, and values input into the MHA layer 151 may be computed from vector embeddings generated by the positional encoding layer 125. A query, key, or value may be a vector the represents a token in a sequence. In some embodiments, a query matrix Q G IR'Vx / 1may be computed by multiply an embedding matrix X G JR'Vxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WqG Rdxft, where d is the dimension of a vector embedding, N is the number of vector embeddings in the embedding matrix, and h is the number of attention heads. Each row in the query matrix may be a query. A key matrix K G IRWx / lmay be computed by multiple an embedding matrix X GWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WkG JRdx / l. Each row in the key matrix may be a key. A value matrix V G JRWx / lmay be computed by multiple an embedding matrix X G JRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WvG Rdx / l. Each row in the value matrix may be a value.

[0061] In some embodiments, the MHA layer 151 may implement masked multi-head selfattention. The MHA layer 151 may prevent positions from attending to subsequent positions. For instance, each token in the sequence may not be influenced by future tokens. This masking can ensure that the predictions of a particular position can depend on known outputs at positions before it and not depend on unknown outputs at positions after it.

[0062] In some embodiments, the MHA layer 153 may implement a cross-attention mechanism, such as encoder-decoder cross-attention. The MHA layer 153 may use outputs from the previous layer (i.e., the add & norm layer 152) as queries and use outputs from the encoder block 110 as keys and values. The cross-attention can align the encoder's input with the decoder's, empowering the decoder block 120 to identify and emphasize the most relevant parts of the encoder's input. Certain aspects of MHA layers are described below in conjunction with FIGS. 4A and 4B.

[0063] An add & norm layer in the transformer model 100, such as the add & norm layer 142, 144, 152, 154, and 156, has an addition operation followed by a layer normalization operation. The addition operation may be an addition of the output of the preceding layer and the input of the preceding layer. The preceding layer is a layer that is arranged right before the add & norm layer. For example, the preceding layer of the add & norm layer 142is the MHA layer 141. As another example, the preceding layer of the add & norm layer 154 is the encoder-decoder attention layer 153.

[0064] Then the layer normalization operation is applied on the result of the addition operation, which may be denoted as LayerNorm(x + sublayer^)), where LayerNorm denotes layer normalization, x is the input of the preceding layer, and sublayer(x) denotes the output of the preceding layer. In some embodiments, the layer normalization operation may include a sequence of computations. In an example, the layer normalization operation may include a mean computation, which may be denoted as , whereAxyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and ixydenotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The layer normalization operation may convert yxyto a 3D tensor [J.xyz, e.g., by replicating every data element overz output points.

[0065] The layer normalization operation may also include an elementwise subtraction, which may be denoted as Dxyz= Axyz— fixyz. The layer normalization operation may further include a variance computation denoted as <J2xy=and a division computation denoted as Mxy—1Mxymay be a 2D tensor. The layernormalization operation may also convert Mxyto a 3D tensor Mxyz, e.g., by replicating every data element over z output points. Further, the layer normalization operation may have an element multiplication denoted as A'xyzXThe layer normalization operation may further compute yz= A"xyzX yz. LNxyzmay be the output of the layernormalization operation.

[0066] A feed forward layer (e.g., the feed forward layer 143 and the feed forward layer 155) may be a position-wise fully-connected feed forward network. In an example, the feedforward layer may include two linear layers with an activation function in between. An example of the activation function is Rectified Linear Unit (ReLU).

[0067] FIG. 2 illustrates an embedding operation in an embedding layer 200, in accordance with various embodiments. The embedding layer 200 may be an example of the embedding layer 113 or the embedding layer 123 in FIG. 1. As shown in FIG. 2, the embedding layer 200 receives an input sequence 201, which includes three words 202, 203, and 204. Each word may be a token. The embedding layer 200 generates a vector embedding 205 from the word 202. The embedding layer 200 also generates a vector embedding 206 from the word 203. The embedding layer 200 further generates a vector embedding 207 from the word 204. In the embodiments of FIG. 2, the vector embeddings 205, 206, and 207 have the same dimension, i.e., they each have five data elements. In other embodiments, the vector embedding 205, 206, or 207 may have a different dimension. Also, the input to the embedding layer 200 may be data of a type other than words, such as audio signals, images, and so on.

[0068] In some embodiments where the embedding layer 200 is in an encoder (e.g., the encoder block 110), the input sequence 201 may be an input received by the encoder, such as a prompt made by a user. The input sequence 201 may remain the same during inference of the encoder. In some embodiments where the embedding layer 200 is in a decoder (e.g., the decoder block 120), the input sequence 201 may change and the dimension of the input sequence 201 may be dynamic during inference of the decoder. In an example, the decoder inference may include a sequence of phases. Each inference phase may be conducted for predicting a token. For the first inference phase, the input sequence 201 may include one or more start tokens. For each subsequent inference phase (e.g., the second inference phase, the third inference phase, etc.), the input sequence 201 may include tokens predicted in the previous inference phases. The dimension of the input sequence may be increased by one after each inference phase.

[0069] FIG. 3 illustrates a positional encoding operation in a positional encoding layer, in accordance with various embodiments. The positional encoding layer may be an example of the positional encoding layer 115 or the positional encoding layer 125 in FIG. 1. The positional encoding operation includes an addition of a vector embedding 310 and a positional encoding vector 320. The vector embedding 310 may be generated by an embedding layer. The positional encoding vector 320 may encode information of theposition of the token represented by the vector embedding 310 in a sequence of tokens.The positional encoding operation computes a vector embedding 330, which represents the token with positional context. In some embodiments, the positional encoding operation may be an elementwise addition operation. A data element in the vector embedding 330 may equal the sum of a data element in the vector embedding 310 and a data element in the positional encoding vector 320. In the embodiments of FIG. 3, the vector embedding 310, positional encoding vector 320, and vector embedding 330 have the same dimension, i.e., they each have five data elements. In other embodiments, the vector embedding 310, positional encoding vector 320, or vector embedding 330 may have a different dimension.

[0070] FIGS. 4A and 4B illustrate an example MHA layer 400, in accordance with various embodiments. The MHA layer 400 may be an example of the MHA layer 141 or the MHA layer 151 in FIG. 1. As shown in FIG. 4A, the MHA layer 400 includes linear layers 410, 420, and 430, a MatMul layer 440, a scale layer 450, a SoftMax layer 460, another MatMul layer 470, a concatenation layer 480, and another linear layer 490. In other embodiments, the MHA layer 400 may include fewer, more, or different layers.

[0071] The MHA layer 400 receive three input matrices: a query matrix 401, a key matrix 402, and a value matrix 403, which are inputs of the linear layers 410, 420, and 430, respectively. The linear layers 410, 420, and 430 are in a linear block 415 of the MHA layer 400. In some embodiments, the MHA layer 400 includes a plurality of linear blocks that includes the linear block 415. For the purpose of illustration, the MHA layer 400 includes h linear blocks in FIG. 4A, where h is an integer. Each of the linear blocks may have the same layers as the linear block 415. Each linear block may compute three parameter matrices from the query matrix 401, key matrix 402, and value matrix 403, respectively. For instance, the linear layer 410 may perform a multiplication of the query matrix 401 with a weight matrix to compute a parameter matrix 404 shown in FIG. 4B. The linear layer 420 may perform a multiplication of the key matrix 402 with a weight matrix to compute a parameter matrix 405 shown in FIG. 4B. The linear layer 430 may perform a multiplication of the value matrix 403 with a weight matrix to compute a parameter matrix 406 shown in FIG. 4B.

[0072] The parameter matrix 404 may be denoted as QW , where Q is the query matrix 401 and W G j^modezxdq jsthe weight matrix. The parameter matrix 405 may be denoted as KW , where K is the key matrix 401 and W G jsthe weight matrix. Theparameter matrix 406 may be denoted where V is the value matrix 401 and W G ]^.modelxdkweight matrix, i may indicate the index of the head. dqis the dimension of a query vector. dkis the dimension of a key vector. dvis the dimension of a value vector. In some embodiments, dq= dk= dv= dmodei / h.

[0073] The MatMul layer 440, scale layer 450, mask layer 455, SoftMax layer 460, and MatMul layer 470 are in an attention block 425 of the MHA layer. The attention block 425 may implement a scaled dot-product attention mechanism. In some embodiments, the MHA layer 400 includes a plurality of attention blocks that includes the attention block 425. For the purpose of illustration, the MHA layer 400 includes h attention blocks in FIG. 4. Each of the attention blocks may have the same layers as the attention block 425. The linear block 415 and attention block 425 may constitute a head of the MHA layer 400. As the MHA layer 400 has h linear blocks and h attention blocks, the MHA layer 400 has h heads. A head may be denoted

[0074] A matrix multiplication operation may be performed on the parameter matrices 404 and 405 in the MatMul layer 440, which computes a score matrix 406. In some embodiments, the score matrix 406 may establish the degree of emphasis each token should place on other tokens. The 406 may include a plurality of scores. Each token may be assigned a score in relation to other tokens within the same time step. A higher score may indicate a higherfocus or emphasis. The score matrix 406 may be scaled in the scale layer 450. In some embodiments, the score matrix 406 is scaled down in the scale layer 450 by dividing the scores in the score matrix by the square root of the dimension of the query vector and the key vector, which may be denoted as d^. The output of the scale layer 450 may be a scaled matrix 408, which includes adjusted scores. The mask layer 455 may be optional in some embodiments. The mask layer 455 may add an attention mask (which may be an input to the attention block 425) to the output of the scale layer 450 to mask out some elements in the output of the scale layer 450. The positions of the masked out elements may be defined by the attention mask. A SoftMax function may be applied on the scaled matrix 408 in the SoftMax layer 460 to compute an attention weight matrix 409. The attention weight matrix 409 includes attention weights. The attention weights may be probability values ranging from 0 to 1. The SoftMax function may emphasize high scoresY1while diminishing low scores, which can enhance the model's ability to determine which tokens should get more attention.

[0075] In the MatMul layer 470, a matrix multiplication operation is performed on the attention weight matrix 409 computed in the SoftMax layer 460 and the parameter matrix 407 computed from value matrix 403 in the linear layer 430. The result of the matrix multiplication operation is a single-head output matrix 411, which is an output of the attention block 425.

[0076] As the MHA layer 400 has h attention blocks, there will be h single-head output matrices. The single-head output matrices are concatenated in the concatenation layer 480 to form a concatenated matrix. A linear operation (also referred to as "linear transformation") is performed on the concatenated matrix using a weight matrix in the linear layer 490. In some embodiments, the multi-head attention may be denoted as MultlHeacl(Q, K , V) = Concat (head1, head2, ... , headh)W°, where Concat denotes concatenation, and W° £hdvxdmodei js the wejght matrix in the linear layer 490.

[0077] FIG. 5 illustrates an example linear classifier 500, in accordance with various embodiments. The linear classifier 500 may be used in transformer models. In some embodiments, the linear classifier 500 may generate tokens based on outputs of decoders. The linear classifier 500 may be an example of the linear block 130 in FIG. 1. As shown in FIG. 5, the linear classifier 500 includes a linear layer 510 and a SoftMax layer 520. In other embodiments, the linear classifier 500 may include fewer, more, or different components.

[0078] The linear layer 510 is provided with a matrix 501. The matrix 501 may be an output of a decoder, e.g., the decoder block 120. A linear transformation may be performed on the matrix 501 and a weight matrix in the linear layer 510. The weight matrix may include weights, which are internal parameters of the linear layer 510. The linear layer outputs a vector 502. In some embodiments, the dimension of the vector 502 (e.g., the total number of elements in the vector 502) may be equal to the total number of classes associated with the Al task being performed by the transformer model. The vector 502 is provided to the SoftMax layer 520. The SoftMax layer 520 generates a vector 503 from the vector 502. In some embodiments, the dimension of the vector 503 may equal the dimension of the vector 502. Each element in the vector 503 may correspond to a predicted token and may indicate a probability score of the predicted token. The probability score may indicate the probabilitythat the prediction is correct. A predicted token 504 having the highest probability score may be selected and output from the linear classifier 500.

[0079] The output of the linear classifier 500 may be the output of the transformer model. The execution of the linear classifier 500 may be performed multiple times during inference of the transformer model. For instance, the transformer model may have multiple inference phases, and the linear classifier 500 may be executed at least once in each inference phase. The dimensions of the vectors and matrices shown in FIGS. 2-5 are example dimensions used for purpose of illustration and simplicity. Any of the vectors and matrices used or computed by operations illustrated in FIGS. 2-5 may have different dimensions.

[0080] FIG. 6 illustrates a first inference phase of a transformer model 600, in accordance with various embodiments. The transformer model 600 includes an encoder 610, a decoder 620, and a head 630. An example of the transformer model 600 may be the transformer model 100 in FIG. 1. In the embodiments of FIG. 6, the encoder 610 receives an input tensor 601. The input tensor 601 may be a feature map extracted from one or more images, text documents, audio files, videos, other types of data, or some combination thereof. The encoder 610 generates an output tensor 602 from the input tensor 601. The shape of the output tensor 602 may be denoted as [batch size, SLencoder, dmodei], where SLencodermay be the dimension along the X axis (i.e., the width of the output tensor 602), and dmodetmay be the dimension along the Y axis (i.e., the height of the output tensor 602). The encoder 610 may include a plurality of layers arranged in a sequence, such as the layers inside the encoder block 110 in IFG. 1. The output tensor 602 is provided to the decoder 620.

[0081] The decoder 620 receives the output tensor 602 and an input sequence 603. The input sequence 603 may be a sequence of tokens. A token may be a numerical representation of an input signal, such as word, image, audio signal, video signal, etc. The dimension of the input sequence 603, which may be denoted as SLinput, may be the total number of tokens in the input sequence 603. For the purpose of illustration and simplicity, SLinput's4. In other embodiments, the input sequence 603 may have a different shape. For instance, the input sequence 603 may be a 2D tensor. The dimension of the 2D tensor along the X axis may be SLinput, while the dimension of the 2D tensor along the Y axis may be a batch size indicating the number of batches in the input sequence 603.

[0082] The decoder 620 computes an output tensor 604, a self-attention key tensor 605, a self-attention value tensor 606, a cross-attention key tensor 607, and a cross-attention value tensor 608. In some embodiments, the shape of the output tensor 604 may be denoted as [batch size,SLinput, dmodei], The shape of the self-attention key tensor 605 or the shape of the self-attention value tensor 606 may be denoted as N X[batch size, h,SLinput, dhead], where N is the number of identical layers in the decoder (e.g., the number of layers 150 in the decoder block 120), h is the total number of heads in a MHA layer, and dheadis the dimension of a query vector, key vector, or value vector. In some embodiments, dmodei= h x dhead. The shape of the cross-attention key tensor 607 or the shape of the cross-attention value tensor 608 may be denoted as N x [batch size, h,SLencoder, head]-

[0083] The output tensor 604 may be provided to the head 630 and the head 630 outputs a predicted token 609. The shape of the token 609 may be denoted as [batch size, 1] . For the purpose of illustration and simplicity, batch size is 1 in FIG. 6. In other embodiments, batch size may be a larger number. The predicted token 609 may be stored in a buffer. In some embodiments, the predicted token 609 may be used to update the input sequence 603. For instance, the predicted token 609 may be added to the right of the input sequence 603. The updated input sequence may be used as the input sequence in the second inference phase. In the second inference phase, the decoder 620 may receive the updated input sequence and the output tensor 602 for predicting another token. The output tensor602 may remain the same during inference of the decoder 620. Certain aspects of subsequent inference phases are described below in conjunction with FIG. 7.

[0084] In some embodiments, the self-attention key tensor 605 and the self-attention value tensor 606 may be provided to a self-attention layer in the decoder 620, an example of such a self-attention layer is the MHA layer 151. The self-attention key tensor 605 may be stored in a self-attention key cache. The self-attention key cache may have the same shape as the self-attention key tensor 605. The self-attention value tensor 606 may be stored in a selfattention value cache. The self-attention value cache may have the same shape as the selfattention value tensor 606.

[0085] In some embodiments, the decoder 620 computes the self-attention key tensor 605 and the self-attention value tensor 606 from the input sequence 603. The input sequence603 may be dynamic during inference of the decoder 620. For instance, a new token may beadded to the input sequence 603 after each inference phase, as described above. As the input sequence 603 changes, the self-attention key tensor 605 and the self-attention value tensor 606 would also change. For instance, the dimension of the self-attention key tensor 605 or the self-attention value tensor 606 along the X axis may increase as SLnputincreases. The self-attention key cache and the self-attention value cache may change during all the inference phases of the decoder 620 to accommodate the changes in the selfattention key tensor 605 and the self-attention value tensor 606.

[0086] In some embodiments, the cross-attention key tensor 607 and the cross-attention value tensor 606 may be provided to a cross-attention layer in the decoder 620, an example of such a cross-attention layer is the MHA layer 153. The cross-attention key tensor 607 may be stored in a cross-attention key cache. The cross-attention key cache may have the same shape as the cross-attention key tensor 607. The cross-attention value tensor 608 may be stored in a cross-attention value cache. The cross-attention value cache may have the same shape as the cross-attention value tensor 608. In some embodiments, the decoder 620 computes the cross-attention key tensor 607 and the cross-attention value tensor 606 from the output tensor 602 generated in the encoder 610. As the output tensor 602 does not change during inference of the decoder 620, the cross-attention key tensor 607 and the cross-attention value tensor 606 may remain the same during all the inference phases of the decoder 620. The cross-attention key cache and the cross-attention value cache may remain the same during all the inference phases of the decoder 620.

[0087] FIG. 7 illustrates subsequent inference phases of the transformer model, in accordance with various embodiments. In the second inference phase, the decoder 620 may reuse the self-attention key tensor 605, self-attention value tensor 606, cross-attention key tensor 607, and cross-attention value tensor 608. The decoder 620 also receives the predicted token 609. The decoder 620 may compute self-attention key vectors from the predicted token 609 and concatenate the self-attention key vectors with the self-attention key tensor 605 to generate a new self-attention key tensor 615. For instance, a selfattention key vector for each head may be added to the right of a self-attention key matrix in the self-attention key tensor 605, and the self-attention key vector and the self-attention key matrix may correspond to the same head. The elements highlighted with a dot pattern in the self-attention key tensor 615 are the self-attention key vectors generated from the predicted token 609.

[0088] Similarly, the decoder 620 may compute self-attention value vectors from the predicted token 609 and concatenate the self-attention value vectors with the self-attention value tensor 606 to generate a new self-attention value tensor 616. For instance, a selfattention value vector for each head may be added to the right of a self-attention value matrix in the self-attention value tensor 606, and the self-attention value vector and the self-attention value matrix may correspond to the same head. The elements highlighted with a dot pattern in the self-attention value tensor 616 are the self-attention value vectors generated from the predicted token 609.

[0089] The decoder 620 also generates an output tensor 614. The decoder 620 may generate the output tensor 614 using the new self-attention key tensor 615 and new selfattention value tensor 616. The output tensor 614 is used by the head 630 to generate another predicted token 619. The predicted token 619 is the output of the transformer model 600 in the second inference phase.

[0090] One or more other subsequent inference phases may be conducted. In each subsequent inference phase, the decoder 620 receives a token predicted in the previous inference phase, a self-attention key tensor generated in the previous inference phase, a self-attention value tensor generated in the previous inference phase, the cross-attention key tensor 607, and the cross-attention value tensor 608. The decoder 620 may, in the subsequent inference phase, generate a larger self-attention key tensor and a larger selfattention value tensor, in addition to an output tensor which can be used by the head 630 to predict a new token.

[0091] In embodiments where the total number of inference phases is N, the input sequence 603 is updated to an input sequence 613 afterfV — 1 inference phases. In the last inference phase (i.e., the Nth inference phase), the decoder 620 may receive the predicted token generated in the (N — th inference phase, the self-attention key tensor generated in the (TV — 1) t / i inference phase, the self-attention value tensor generated in the (TV — 1) t / i inference phase, the cross-attention key tensor 607, and the cross-attention value tensor 608. The decoder 620 may generate a self-attention key tensor 625 and a selfattention value tensor 626 using the predicted token generated in the (N — l)th inference phase, the self-attention key tensor generated in the ( / V — l)t / i inference phase, and the self-attention value tensor generated in the ( / — l)t / i inference phase. The dimensions of the self-attention key tensor 625 or self-attention value tensor 626 along the X axis isSLinput + N. The decoder 620 also generates an output tensor 624, which is used by the head 630 to generate the last predicted token 629. The N tokens predicted by the transformer model in the N inference phases may constitute an output tensor 639, which may be the final output of the transformer model.

[0092] FIG. 8 illustrates computations in a self-attention layer without KV caching, in accordance with various embodiments. The self-attention layer may be a multi-head selfattention layer. In some embodiments, the self-attention layer is in a decoder of a transformer. The computations in the self-attention layer may include multiplication of a query matrix 810 and a key matrix 820, which results in an attention weight matrix 830. In some embodiments, the self-attention layer may be a masked self-attention layer. One or more elements in the attention weight matrix 830 may be masked. For instance, the elements highlighted with a dotted pattern in FIG. 8 may be masked. The computations in the self-attention layer also includes multiplication of the attention weight matrix 830 and a value matrix 840, which results in an output matrix 850 encoding new tokens. In other embodiments, the computations in the self-attention layer may include other computations, such as computations with a scaling function, SoftMax function, and so on. For the purpose of simplicity and illustration, these computations are not shown in FIG. 8.

[0093] Each of the query matrix 810, key matrix 820, and value matrix 840 may include a vector for each of the tokens in the input sequence. For the purpose of illustration and simplicity, the input sequence has four tokens: tokens 1-4. In the embodiments of FIG. 8, as the decoder does not implement KV caching, computations on all the key tokens in the key matrix 820 and all the value tokens in the value matrix 840 need to be conducted. Some of the computations have already been conducted in the previous inference phase, e.g., computations on the key tokens 1-3 and computations on the value tokens 1-3. The duplication of these computations can be a waste of computational resources, such as power, time, and so on.

[0094] FIG. 9 illustrates computations in a self-attention layer with KV caching, in accordance with various embodiments. Forthe purpose of illustration and simplicity, the self-attention in FIG. 9 may have the same query matrix, key matrix, and value matrix as the self-attention in FIG. 8. Different from the embodiments of FIG. 8, the decoder implements KV caching in the embodiment of FIG. 9. With the KV caching, the keys and values used in the previous inference phase(s) as well as data computed from the keys and values in theprevious inference phase(s) are cached and can be reused in the current inference phase. The KV caching can reduce the amount of computations in the self-attention layer. Data that can be retrieved from cache is highlighted with a dotted pattern in FIG. 9. The amount of multiplication is reduced. Therefore, computational resources can be saved. The performance and efficiency of the transformer model can be improved. In some embodiments, the computations in FIG. 9 are computations in the fourth inference phase of a decoder, which is carried out after the generation of three tokens in three inference phases that were previously carried out.Example Transformer Models with Static Shaped Inputs and KV Caches

[0095] FIG. 10 illustrates a decoder 1000 implementing static shaped inputs, in accordance with various embodiments. The decoder 1000 may be part of a transformer model. For the purpose of illustration and simplicity, the other components of the transformer model (e.g., encoder, head, etc.) are not shown. Also, FIG. 10 shows a first inference phase of a transformer model with. The decoder 1000 includes a decoder block 1010, a padding block 1020, a masking block 1030, and a position block 1040. In other embodiments, the decoder 1000 may include fewer, more, or different components.

[0096] The decoder 1000 receives an output tensor 1005 from the encoder. The output tensor 1005 is an encoder output tensor. The shape of the output tensor 1005 may be denoted as [batch size, SLencoder, dmodel], where SLencodermay be the dimension along the X axis (i.e., the width of the output tensor 1005), and dmodeimay be the dimension along the Y axis (i.e., the height of the output tensor 1005). The decoder block 1010 also receives an input sequence 603. The input sequence 603 may be a sequence of tokens. A token may be a numerical representation of an input signal, such as word, image, audio signal, video signal, etc. The dimension of the input sequence 603, which may be denoted as SLinput, may be the total number of tokens in the input sequence 603. For the purpose of illustration and simplicity, SLinputis 4. In other embodiments, the input sequence 603 may have a different shape. For instance, the input sequence 603 may be a 2D tensor. The dimension of the 2D tensor along the X axis may be SLinput, while the dimension of the 2D tensor along the Y axis may be a batch size indicating the number of batches in the input sequence 603.

[0097] The padding block 1020 pads the input sequence 1001 and generates a padded input sequence 1002. In some embodiments, the padding block 1020 adds padding tokens into the input sequence 1001. The padding tokens are highlighted by a dotted pattern in FIG. 10. All the padding tokens are added to the right of the input sequence 1001. Due to the addition of the padding tokens, the dimension of the padded input sequence 1002 is greater than the dimension of the input sequence 1001. In some embodiments, the dimension of the padded input sequence 1002 may be denoted as SLmax. SLmaxmay indicate a maximum decoder sequence length, which is no smaller than the largest input sequence length in the decoder 1000. The largest input sequence length in the decoder 1000 may be SLinput + , where N is the total number of inference phases or the total number of tokens predicted by the transformer model in the whole inference process. The number of padding tokens in the padded input sequence 1002 may be SLmax— SLinput. In some embodiments, padding tokens may not be involved in computations in the decoder block 1010. For instance, no queries, keys, or values may be generated for padding tokens.

[0098] The masking block 1030 generates an attention mask 1003. The attention mask 1003 indicates positions of input tokens in the padded input sequence 1002 and positions of the padding tokens in the padded input sequence 1002. In some embodiments, the attention mask 1003 has the same length as the padded input sequence 1002. Each element in the attention mask 1003 may correspond to a different token in the padded input sequence 1002 and indicates whether the token is an input token or a padding token. The attention mask elements corresponding to input tokens may have a first value, while the attention mask elements corresponding to padding tokens may have a second value. In an example, the first value may be 1, and the second value may be 0. In another example, the first value may be 1, and the second value may be a minimum possible value of a data format, e.g., the minimum possible value of FP32 (i.e., -3.4e+38).

[0099] The position block 1030 generates a positional encoding vector 1004. The positional encoding vector 1004 may indicate the position of each input token in the padded input sequence 1002. In some embodiments, the positional encoding vector 1004 has the same length as the padded input sequence 1002. Each element in the positional encoding vector 1004 may correspond to a different token in the padded input sequence 1002. The elements corresponding to input tokens indicate the positions of the input tokens in the padded input sequence 1002. In an example, the first element of the positional encoding vector 1004 maybe 0, the second element may be 1, the third element may be 2, and the fourth element may be 3, indicating the positions of the four input tokens, respectively. The other elements of the positional encoding vector 1004 may all be 0.

[0100] The decoder block 1010 receives the padded input sequence 1002 and the output tensor 1005. The decoder block 1010 computes an output tensor 1006, a self-attention key tensor 1007, a self-attention value tensor 1008, a cross-attention key tensor 1011, and a cross-attention value tensor 1012. In some embodiments, the shape of the output tensor 1006 may be denoted as [batch size,SLmax, dmodel]. The output tensor 1006 may include a valid subtensor with valid data and a padding subtensor with padding data. The shape of the valid subtensor may be [batch size,SLinput, dmodel]. The shape of the padding subtensor may be [batch size, SLmax— SLinput, dmodel]. The padding subtensor may be on the right of the valid subtensor. In some embodiments, the valid data may be used in the head for predicting a token, while the padding data may be skipped in the head.

[0101] The shape of the self-attention key tensor 1007 or the shape of the self-attention value tensor 1008 may be denoted as N X [batch size, h,SLinput, dhead], where N is the number of decoder blocks 1010 in the decoder (e.g., the number of layers 150 in the decoder block 120), h is the total number of heads in a MHA layer, and dheadis the dimension of a query vector, key vector, or value vector. In some embodiments, dmodei= / ix-head- The self-attention key tensor 1007 may be stored in a self-attention key cache, from which the self-attention key tensor 1007 may be retrieved and reused in subsequent inference phases. Similarly, the self-attention value tensor 1008 may be stored in a selfattention value cache, from which the self-attention value tensor 1008 may be retrieved and reused in subsequent inference phases.

[0102] The cache storing the self-attention key tensor 1007 or the self-attention value tensor 1008 may have a shape denoted as N X [batch size, h,SLmax, dhead]. In addition to the self-attention key tensor 1007 or the self-attention value tensor 1008, the cache may store padding elements. In FIG. 10, padding elements are highlighted by a dotted pattern. The padding elements in the cache may constitute a padding tensor with a shape denoted as N X [batch size, h,SLmax— SLinput, dhead]. The padding subtensor may be on the right of the self-attention key tensor 1007 or the self-attention value tensor 1008.

[0103] In some embodiments, the self-attention key tensor 1007 and the self-attention value tensor 1008 may be provided to a self-attention layer in the decoder block 1010, an example of such a self-attention layer is the MHA layer 151. In some embodiments, the selfattention key tensor 1007 and the self-attention value tensor 1008 may be used in the selfattention layer, while the padding data in the cache may be skipped in the self-attention layer.

[0104] In some embodiments, the cache may be padded based on the padded input sequence 1002. As described above, the input sequence 1001 can be dynamic, but the padded input sequence 1002 can be static for all the inference phases as new tokens predicted in previous inference phases can be added to the padded input sequence 1002 by replacing padding tokens in the padded input sequence 1002 without increasing the length of the padded input sequence 1002. Thus, the padded input sequence 1002 can be static shaped inputs to the decoder block 1010. In some embodiments, the padded input sequence 1002 is used in the first inference phase. The usage of the padded input sequence 1002 can be avoided in subsequent inference phases, which is an advantage of using KV caching. By switching from the dynamic input sequence to the static padded input sequence, the transformer model may be converted from a dynamic model to a static model. With such static shaped inputs, the caches can also be static. As the decoder block 1010 generates new keys or values in subsequent inference phases, the decoder block 1010 may replace padding data in the self-attention key tensor 1007 or the self-attention value tensor 1008 with the new keys or values. The dimensions of the caches may remain the same in the whole inference process. The cache can be static for all the inference phases.

[0105] The shape of the cross-attention key tensor 1011 or the shape of the cross-attention value tensor 1012 may be denoted as N x [batch size, h, SLencoder, dhead], which is not impacted by the padding of the input sequence 1001. In some embodiments, the crossattention key tensor 1011 and the cross-attention value tensor 1008 may be provided to a cross-attention layer in the decoder block 1010, an example of such a cross-attention layer is the MHA layer 153. The cross-attention key tensor 1011 may be stored in a crossattention key cache. The cross-attention key cache may have the same shape as the crossattention key tensor 1011. The cross-attention value tensor 1012 may be stored in a crossattention value cache. The cross-attention value cache may have the same shape as the cross-attention value tensor 1012. In some embodiments, the decoder block 1010 computesthe cross-attention key tensor 1011 and the cross-attention value tensor 1008 from the output tensor 1005. As the output tensor 1005 does not change during inference of the decoder block 1010, the cross-attention key tensor 1011 and the cross-attention value tensor 1008 may remain the same during all the inference phases of the decoder block 1010. The cross-attention key cache and the cross-attention value cache may remain the same during all the inference phases of the decoder block 1010.

[0106] FIG. 11 illustrates cache updating in a transformer model with static shaped inputs, in accordance with various embodiments. The transformer model may be an example of the transformer model in FIG. 10. The transformer model in FIG. 11 includes a cache update block 1100 for updating caches (e.g., self-attention key cache and self-attention value cache) in the decoder. FIG. 11 shows a cache 1110. The transformer model may have one or more other caches. For the purpose of illustration, FIG. 11 shows cache updating in the second inference phase. The cache update block 1100 may update caches in every subsequent inference phase.

[0107] As shown in FIG. 11, the cache 1110 stores an attention tensor 1113 and a padding tensor 1115. The attention tensor 1113 may include self-attention keys or self-attention values. The padding tensor 1115 may include padding elements. The padding elements are highlighted by a dotted pattern in FIG. 11. The cache 1110 may be a self-attention key cache or a self-attention value cache. The attention tensor 1113 may be a self-attention key tensor or a self-attention value tensor. In an example, the attention tensor 1113 may be the selfattention key tensor 1007 or the self-attention value tensor 1008. The attention tensor 1113 may be generated by the decoder in the previous inference phase, e.g., the first inference phase.

[0108] In the current inference phase, e.g., the second inference phase, the decoder generates a new attention tensor 1120. The decoder may compute the attention tensor 1120 from the token predicted by the transformer model in the previous inference phase. In some embodiments, the shape of the attention tensor 1120 may be denoted as N X [batch size, h, 1, dhead], where 1 is the dimension along the X axis, dheadis the dimension along the Y axis, and h is the dimension along the Z axis. The reason why the dimension along the X axis is 1 may be that the attention tensor 1120 is generated from a single token.

[0109] The cache update block 1100 may update the cache 1110 by writing the attention tensor 1120 to the cache 1110. The cache update block 1100 may place the attention tensor 1120 at the right of the attention tensor 1113. In some embodiments, the cache update block 1100 concatenates the attention tensor 1113 and the attention tensor 1120 and forms a new attention tensor 1117. As shown in FIG. 11, the attention tensor 1120 is placed at the right of the attention tensor 1113 for each head. For instance, the vector of a particular head in the attention tensor 1120 is placed at the right of the vector of the particular head in the attention tensor 1113. The cache update block 1100 does not change the dimension of the cache 1110 along the X axis. The cache update block 1100 may replace a portion of the padding tensor 1115 with the attention tensor 1120. The padding tensor 1115 is modified to a padding tensor 1119, which has a shorter length along the X axis than the padding tensor 1115.

[0110] In some embodiments, the cache update block 1100 may update the cache 1110 based on information from an inference counter 1130. In some embodiments, the inference counter 1130 counts the number of inference phases that have been completed. The inference counter 1130 may maintain a parameter countid, which indicates the number of inference phases that have been completed. The inference counter 1130 may update the value of the parameter every time an inference phase is complete. A position ID for the token corresponding to the attention tensor 1120 may be determined, e.g., by a positional encoding block in the transformer model. The position ID may indicate the position of the token in the token sequence predicted by the transformer model. In an example, the position ID may be denoted as SLinput+ countid— 1. The cache update block 1100 may write the attention tensor 1120 to the cache 1110 or concatenate the attention tensor 1120 with the attention tensor 1113 based on the position ID of the token. In an example where the position ID is 5, the cache update block 1100 writes the attention tensor 1120 to the cache 1110 at index = 5.

[0111] Compared to other KV cache updating methods like left shift and write, which implicitly involves copying data and works with 'left padding' all input tensors (input ID, mask, pos ID), the cache updating method illustrated in FIG. 11 can be much faster as it performs fewer operations, have less overhead and involves less data movement than the first method. In some embodiments, for every inference, the memory savings can be proportional to SLmax. Since the cached inference can run for any number of inferences,based on the input data, there can be quadratic savings in terms of memory footprint(oc given that the decoder generates output 1120 of size 1 instead of SLmaxin the x dimension.

[0112] FIG. 12 illustrates a subsequent inference phase in a transformer model with static shaped inputs, in accordance with various embodiments. The transformer model in FIG. 12 may be the same transformer model as the one in FIG. 11. In some embodiments (e.g., embodiments where SLinputis 4), the subsequent inference phase shown in FIG. 12 may be the fourth inference phase, and the transformer model may have output three tokens.

[0113] A decoder block 1200 in the transformer model receives a token 1201, a position ID1202 of the token, an attention mask 1203, a self-attention key tensor 1204, a self-attention value tensor 1205, a cross-attention key tensor 1206, and a cross-attention value tensor 1207. The token 1201 may be predicted by the transformer model in the previous inference phase. The position ID 1202 may indicate the position of the token in the token sequence generated by the transformer model through all the inference phases. The attention mask1203 may indicate the positions of tokens in a padded input sequence. The self-attention key tensor 1204 may be stored in part of a key cache, and the rest of the key cache may store padding elements, which are shown as boxes with a dotted pattern in FIG. 12. The self-attention value tensor 1205 may be stored in part of a value cache, and the rest of the value cache may store padding elements, which are shown as boxes with a dotted pattern in FIG. 12. The cross-attention key tensor 1206 and cross-attention value tensor 1207 may be generated by the decoder block 1200 in the first inference phase.

[0114] In the current inference phase, the decoder block 1200 generates a self-attention key tensor 1208 and a self-attention value tensor 1209 from the token 1201. The selfattention key tensor 1208 or self-attention value tensor 1209 may have the same shape as the attention tensor 1120 in FIG. 11. The decoder block 1200 also generate an output tensor 1211 by using the token 1201, the self-attention key tensor 1208, and a self-attention value tensor 1209 plus the self-attention key tensor 1204, self-attention value tensor 1205, crossattention key tensor 1206, and cross-attention value tensor 1207. In the embodiments of FIG. 12, the output tensor 1211 is a vector having a length of dmodel. The output tensor 1211 may be provided to a head 1210 of the transformer model. The head 1210 generates a token 1212 from the output tensor 1211.

[0115] During or after the current inference phase, the self-attention key cache may be updated to store the self-attention key tensor 1208. Also, the self-attention value cache may be updated to store the self-attention value tensor 1209. The padded input sequence may be updated to include the token 1212. The attention mask 1203 may be updated to indicate the position of the token 1212 in the padded input sequence. An inference counter (not shown in FIG. 12) may increase the value of the parameter countldby one. A position ID of the token 1212 may be determined, which may be SLinput+ countld— 1. countidmay refer to the number of inferences completed including first inference phase and the cache inference phase. The next inference phase may be then conducted.

[0116] In some embodiments, KV cache may be copied from a host memory space (e.g., a system memory) to an accelerator memory space (e.g., a local memory) during inference, which can introduce significant overhead, especially for large volumes of data. This is because data transfer between the host and the accelerator often involves moving data across diverse types of memories (e.g., from dynamic random-access memory (DRAM) to static random-access memory (SRAM)), which can be time-consuming and energy intensive. In some embodiments, the host and the accelerator can share the same physical memory, which can be leveraged to reduce the latency and improve the throughput of the decoder with KV cache inference. For instance, during cached inference, instead of passing all inputs to the inference request associated with decoder with KV cache model, an API may be used to obtain a system-memory pointer to the content of each of the inputs. After the first inference is completed, the cross-attention KV cache would not change as described above. Leveraging this fact, the memory pointers of the decoder with KV cache inputs corresponding to cross-attention KV cache can be assigned to the memory pointers of the decoder first inference cross-attention KV cache. Cross-attention cache can be large, and this ensures that there is no extra copy across host and accelerator and across two inference requests. Similarly, after every cached inference, the cache update block, masking block, position ID block can directly work on the data present at the memory pointer, which is obtained using the API. This can achieve faster inference latency by reducing extra overhead.

[0117] FIG. 13 illustrates attention mask updating in a transformer model with static shaped inputs, in accordance with various embodiments. FIG. 13 shows a masking block 1300 in the transformer model. The masking block 1300 may be an example of the masking block 1030in FIG. 10. The masking block 1300 may generate an attention mask before the first inference phase and update the attention mask throughout the inference of the transformer model. As shown in FIG. 13, the masking block 1300 receives an input sequence 1301 including input tokens. The input sequence 1301 is converted to a padded input sequence (not shown in FIG. 13) by padding the input sequence 1301. The masking block 1300 generates an attention mask 1302 indicating positions of input tokens in the padded input sequence. Each element in the attention mask 1302 corresponds to a token in the padded input sequence and indicates whether the token is an input token or padding token. For the purpose of illustration, elements of the attention masks 1302 for input tokens are represented by blank boxes in FIG. 13, while elements of the attention masks 1302 for padding tokens are represented by dotted boxes in FIG. 13. In some embodiments, the elements represented by the blank boxes have a value of 1, and the elements represented by the dotted boxes have a value of 0. The attention mask 1302 may be generated for the first inference phase before the first inference phase starts.

[0118] After the first inference phase is done and before the second inference phase, the masking block 1300 updates the attention mask 1302 and generates an attention mask 1303 by appending an element of 1 to the end of the attention mask 1302. The new element may correspond to the token generated in the second inference phase. The length of the attention mask 1303 is greater than the length of the attention mask 1302 by 1. In an example where the length of the attention mask 1302 is SLmax, which is the length of the padded input sequence, the length of the attention mask 1303 is SLmax+ 1. The increase of the attention mask length may be necessary in some embodiments such that the masked MHA layers attends to the new input IDs (size: 1) during rest of the inferences. For one new token input, at tth inference, these layers may compute Q+1and the new K and V tensors are concatenated to the right of self-attention KV cache input K , V ,- i G {1 ... . N — 1}. Putting 1 at the end of mask can allow the attention to be mathematically correct.

[0119] After the attention mask 1303 is generated, other attention mask updating by the masking block 1300 may be based on position IDs and would not change the length. The attention mask length may be SLmax+ 1 for all the subsequent inference phases. After the second inference phase and before the third inference phase, the masking block 1300 changes the attention mask 1303 to an attention mask 1304 by changing the first element of0, which may correspond to the first padding token, to an element of 1, which corresponds to the token generated in the second inference phase. The masking block 1300 may update the attention masks 1304 using a position ID 1314 of the token generated in the second inference phase. The position ID 1314 may be SLinput+ 1, where SLinputis the length of the input sequence. In an example where the position ID 1314 is 5, the masking block 1300 put a value 1 at the fifth position of the attention mask 1304. The fifth position of the attention mask 1303 has a value 0 before the updating. This may continue for the other subsequent inference phases. For instance, after the third inference phase and before the fourth inference phase, the masking block 1300 updates the attention mask 1304 by changing the first element of 0 to an element of 1, which corresponds to the token generated in the third inference phase.

[0120] After the last inference phase (i.e., the th inference phase), the masking block 1300 changes the first element of 0 to an element of 1, which corresponds to the token generated in the last inference phase and generates an attention mask 1305. The masking block 1300 may update the attention masks 1304 using a position ID 1315 of the token generated in the last inference phase. The position ID 1315 may be SLinput+ N — 1. As described above, the length of the attention mask 1305 is still SLmax+ 1.

[0121] FIG. 14 illustrates cache padding, in accordance with various embodiments. The cache padding in FIG. 14 is performed by a cache update block 1410. The cache update block 1410 may be an example of the cache update block 1100. FIG. 14 shows a selfattention KV cache 1401 which stores keys or values in an attention tensor generated in the first inference phase. The cache update block 1100 may perform padding on the selfattention KV cache 1401 after the first inference phase. In some embodiments, the cache update block 1410 may perform right padding on the self-attention KV cache 1401 and converts the self-attention KV cache 1401 into a right-padded self-attention KV cache 1402. The cache update block 1410 may perform right padding with zeros ("0s"). The 0s are represented by dotted boxes in FIG. 14. The cache update block 1410 may use a maximum length 1403 to pad the self-attention KV cache 1401. The maximum length 1403 may be SLmax. In some embodiments, the number of inserted zeros is equal to SLmax— SLinput. SLinput refers to an input length 1404, which is the length of the input sequence of the first inference.

[0122] The cache update block 1410 is associated with a masking block 1420. The masking block 1420 may be an example of the masking block 1030 in FIG. 10. The masking block 1420 may produce an attention mask 1405 based on the maximum length 1403 and the input length 1404 with SLinputnumber of Is on the left, SLmax— SLinputnum of 0s in the middle, and one 1 at the end to account for the next decoder cached inference. This can bring the mask shape and value distribution before the second inference same as in FIG. 13. The cache padding can reduce operation count and consequently inference latency of decoder first inference compared to fully padded first inference.

[0123] FIGS. 15A and 15B illustrate graph modifications for transformer models with static shaped inputs, in accordance with various embodiments. FIG. 15A shows a graph that represents part of a transformer model with static shaped inputs. The graph includes nodes and edges. A node represents an operation in the transformer model. An edge is a link between two nodes and may indicate data flow between the two operations. As described above, the previously computed self-attention tensors K , V i G {1 ... . N — 1} in the transformer model do not changes across inferences. A memory-efficient attention operation may be implemented in the transformer model. As shown in FIG. 15A, the edge between concatenation ("concat") operation ("op") inside an attention layer and output node 'Result' is removed. A new edge is created between the output of transpose op and the 'Result' node. This illustration is shown for a previously computed self-attention key tensor (static shaped), and it may be repeated for value tensor for all decoder blocks. In some embodiments, concat op may be mapped to a Digital Signal Processor (DSP) which can directly write the output of this concat op to the memory (e.g., a DRAM). In some embodiments (including embodiments of FIG. 11 or FIG. 12), the decoder with KV cache produces self-attention KV cache with '1' in the sequence length dimension, instead of SLmax- The memory write ops can be reduced proportionally to SLmax. Thus, by minimizing the volume of data written to the memory, the time taken for data transfer is reduced, thereby decreasing the overall latency of the inference process. Moreover, writing data to the memory can be a power-intensive operation. This optimization can reduce the power consumption of the accelerator leading to better energy efficiency. The removed edge is represented by a dash line. The added edge is represented by a dot line.

[0124] FIG. 15B illustrates modification of a graph representing part of a transformer model. The modification is illustrated by a dashed box in FIG. 15B. The transform model may notuse position IDs because it may process input data in a way that does not require explicit knowledge of token positions. Other reasons may relate to model architecture, task-specific requirements, data characteristics, etc. In the embodiments of FIG. 15B, the graph is modified to implement position IDs in the transformer model. Position IDs may be used for updating cache, updating attention mask, or other operations. As shown in FIG. 15B, the graph is modified by adding 'position ID' as explicit input to the decoder. Also, the embedding layer forward pass may be modified to extract positional embeddings from the location indicated by input: 'position ID.' A new operation, Gather, is also added to the graph to facilitate the usage of the position IDs in the transformer model.Example Convolution

[0125] Transformer models with static shaped inputs may be executed at least partially by DNN accelerators. For instance, matrix multiplication operations in transformer models may be executed by DNN accelerators, e.g., the DNN accelerator 1702 in FIG. 17. DNN accelerators may have desirable efficiency and performance for executing convolutions. For instance, the DNN accelerators may accelerate convolutions based on sparsity in activations or weights. In some embodiments, matrix multiplication operations in transformer models may be mapped to convolution operations.

[0126] FIG. 16 illustrates an example convolution, in accordance with various embodiments. The convolution may be a deep learning operation in a convolutional layer of a DNN. The convolution can be executed on an activation tensor 1610 and filters 1620 (individually referred to as "filter 1620"). The filters may constitute a weight tensor of the convolution. The result of the convolution is an output tensor 1630. In some embodiments, the convolution is performed by a DNN accelerator.

[0127] The activation tensor 1610 may be computed in a previous layer of the DNN. In some embodiments (e.g., embodiments where the convolutional layer is the first layer of the DNN), the activation tensor 1610 may be an image. In the embodiments of FIG. 16, the activation tensor 1610 includes activations (also referred to as "input activations," "elements," or "input elements") arranged in a 3D matrix. The activation tensor 1610 may also be referred to as an input tensor of the convolution. An input element is a data point in the activation tensor 1610. The activation tensor 1610 has a spatial size HinX WinX Cin, where Hinis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel), Winis the widthof the 3D matrix (i.e., the length along the X axis, which indicates the number of activations in a row in the 16D matrix of each input channel), and Cinis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of input channels). For the purpose of simplicity and illustration, the activation tensor 1610 has a spatial size of 7x7x3, i.e., the activation tensor 1610 includes three input channels and each input channel has a 7x7 16D matrix. Each input element in the activation tensor 1610 may be represented by a (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the activation tensor 1610 may be different.

[0128] Each filter 1620 includes weights arranged in a 3D matrix. The values of the weights may be determined through training the DNN. A filter 1620 has a spatial size H^ x W^ x C , is the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in a column in each kernel), Wf is the width of the filter (i.e., the length along the X axis, which indicates the number of weights in a row in each kernel), and C is the depth of the filter (i.e., the length along the Z axis, which indicates the number of channels). In some embodiments, equals Cin. For purpose of simplicity and illustration, each filter 1620 in FIG. 16 has a spatial size of 16x3x3, i.e., the filter 1620 includes 16 convolutional kernels with a spatial size of 16x3. In other embodiments, the height, width, or depth of the filter 1620 may be different. The spatial size of the convolutional kernels is smaller than the spatial size of the 16D matrix of each input channel in the activation tensor 1610.

[0129] An activation or weight may take one or more bytes in a memory. The number of bytes for an activation or weight may depend on the data format. For example, when the activation or weight has an INT8 format, the activation takes one byte. When the activation or weight has a FP16 format, the activation or weight takes two bytes. Other data formats may be used for activations or weights.

[0130] In the convolution, each filter 1620 slides across the activation tensor 1610 and generates a 16D matrix for an output channel in the output tensor 1630. In the embodiments of FIG. 16, the 16D matrix has a spatial size of 5x5. The output tensor 1630 includes activations (also referred to as "output activations," "elements," or "output element") arranged in a 3D matrix. An output activation is a data point in the output tensor 1630. The output tensor 1630 has a spatial size Houtx Woutx Cout, where Houtis theheight of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in a column in the 16D matrix of each output channel), W7outis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 16D matrix of each output channel), and Coutis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels). Coutmay equal the number of filters 1620 in the convolution. Houtand Woutmay depend on the heights and weights of the activation tensor 1610 and each filter 1620. In an example where the kernel size is lxl, Houtand Woutmay equal to Hinand Win, respectively.

[0131] As a part of the convolution, multiply-accumulate (MAC) operations can be performed on a 16x3x3 subtensor 1615 (which is highlighted with a dotted pattern in FIG. 16) in the activation tensor 1610 and each filter 1620. The result of the MAC operations on the subtensor 1615 and one filter 1620 is an output activation. In some embodiments (e.g., embodiments where the convolution is an integral convolution), an output activation may include 8 bits, e.g., one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), an output activation may include more than one byte. For instance, an output element may include two bytes.

[0132] After the MAC operations on the subtensor 1615 and all the filters 1620 are finished, a vector 1635 is produced. The vector 1635 is highlighted with a dotted pattern in FIG. 16. The vector 1635 includes a sequence of output activations, which are arranged along the Z axis. The output activations in the vector 1635 have the same (x, y) coordinate, but the output activations correspond to different output channels and have different Z coordinates. The dimension of the vector 1635 along the Z axis may equal the total number of output channels in the output tensor 1630. After the vector 1635 is produced, further MAC operations are performed to produce additional vectors till the output tensor 1630 is produced. In the embodiments of FIG. 16, the output tensor 1630 is computed in a Z-major format. When the output tensor 1630 is computed in the ZXY format, the vector that is adjacent to the vector 1635 along the X axis may be computed right after the vector 1635. When the output tensor 1630 is computed in the ZYX format, the vector that is adjacent to the vector 1635 along the Y axis may be computed right after the vector 1635. The output tensor 1630 may be permuted, e.g., by the drain module 1790, and stored in a memory (e.g., the local memory 340) in an X-major format or Y-major format.

[0133] In some embodiments, the MAC operations on a 3x3x3 subtensor (e.g., the subtensor 1615) and a filter 1620 may be performed by a plurality of MAC units. One or more MAC units may receive an input operand (e.g., an activation operand 1617 shown in FIG. 16) and a weight operand (e.g., the weight operand 1627 shown in FIG. 16). The activation operand 1617 includes a sequence of activations having the same (x, y) coordinate but different z coordinates. The activation operand 1617 includes an activation from each of the input channels in the activation tensor 1610. The weight operand 1627 includes a sequence of weights having the same (x, y) coordinate but different z coordinates. The weight operand 1627 includes a weight from each of the channels in the filter 1620. Activations in the activation operand 1617 and weights in the weight operand 1627 may be sequentially fed into a MAC unit. The MAC unit may receive an activation and a weight ("an activation-weight pair") at a time and multiple the activation and the weight. The position of the activation in the activation operand 1617 may match the position of the weight in the weight operand 1627. The activation and weight may correspond to the same channel.

[0134] Activations or weights may be floating-point numbers. Floating-point numbers may have various data formats, such as FP32, FP16, BF16, and so on. A floating-point number may be a positive or negative number with a decimal point. A floating-point number may be represented by a sequence of bits that includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing an exponent of the floating-point number, and bits representing a mantissa of the floating-point number. The mantissa is the part of a floating-point number that represents the significant digits of that number. The mantissa is multiplied by the base raised to the exponent to give the actual value of the floating-point number.

[0135] In some embodiments, the output activations in the output tensor 1630 may be further processed based on one or more activation functions before they are written into the memory or inputted into the next layer of the DNN. The processing based on the one or more activation functions may be at least part of the post processing of the convolution. In some embodiments, the post processing may include one or more other computations, such as offset computation, bias computation, and so on. The results of the post processing may be stored in a local memory of the compute block and be used as input to the next DNNlayer. In some embodiments, the input activations in the activation tensor 1610 may be results of post processing of the previous DNN layer.Example DNN System

[0136] FIG. 17 is a block diagram of a DNN system 1700, in accordance with various embodiments. The whole DNN system 1700 or a part of the DNN system 1700 may be implemented in one or more computing devices, such as the computing device 2300 in FIG. 23. The DNN system 1700 can generate and execute DNNs, such as transformer-based models (e.g., the transformer models described above), convolution-based models, and so on. As shown in FIG. 17, the DNN system 1700 includes a DNN module 1701 and a DNN accelerator 1702. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 1700. For instance, the DNN system 1700 may include multiple DNN modules or multiple DNN accelerators. Further, functionality attributed to a component of the DNN system 1700 may be accomplished by a different component included in the DNN system 1700 or a different system. In some embodiments, the DNN module 1701 and DNN accelerator 1702 may include different types of processing units. In an example, the DNN module 1701 may be implemented by one or more central processing units (CPUs). The DNN accelerator 1702 may also be referred to as a neural processing unit, Al accelerator, or Al processor. The DNN module 1701 and DNN accelerator 1702 may be implemented in the same chip or separate chips.

[0137] The DNN module 1701 facilitates generation and deployment of DNNs. In some embodiments, the DNN module 1701 may generate and train DNNs. For instance, the DNN module 1701 can define the layered architecture of a DNN. The DNN module 1701 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 1701 may also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the sparsity level of one or more deep learning tensors for the DNN.

[0138] The DNN module 1701 may also compress DNNs, e.g., during or after training. In some embodiments, the DNN module 1701 may prune weights in one or more layers of a DNN by changing nonzero valued weight to zeros. The DNN module 1701 may prune weights based on a target weight sparsity ratio. A weight sparsity ratio may be the ratio of the number of zero-valued weights to the total number of weights. In an example where the DNN module 1701 prunes weight during DNN training, the DNN module 1701 may pruneweight of a layerto achieve a target sparsity ratio after one or more epochs. The DNN module 1701 may prevent the pruned weights from changing values during the rest of the training process. Alternatively, the DNN module 1701 may allow the pruned weights to change values so that a pruned, zero-valued weight may have a nonzero value after further training. The DNN module 1701 may prune weights of the layer again after one or more additional epochs.

[0139] The DNN module 1701 may deploy trained, compressed, or validated DNNs for use in deep learning applications. In some embodiments, the DNN module 1701 may distribute trained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform tasks (e.g., image classification, motion planning, etc.) for which the DNNs were trained. In other embodiments, the DNN module 1701 may facilitate deployment of the DNNs using the DNN accelerator 1702. For instance, the DNN module 1701 may receive data from a device or system coupled with the DNN system 1700 and input the received data (or data generated by the DNN module 1701, e.g., based on the received data) into a DNN. The DNN module 1701 may generate instructions (e.g., configuration files) that control the operation of the DNN accelerator 1702 during the DNN execution. The DNN module 1701 may receive an output of the DNN from the DNN accelerator 1702. The DNN module 1701 may transmit the output of the DNN (or a result of processing the output of the DNN by the DNN module 1701) to the device or system. In some embodiments, the DNN module 1701 may control execution processes of trained, compressed, or validated DNNs. The DNN module 1701 may function as a compiler for DNNs executed by the DNN accelerator 1702. The DNN module 1701 may perform compilation of DNNs and generate compilation descriptors, based on which the DNNs may be executed.

[0140] The DNN module 1701 may generate executable transformer models. The DNN module 1701 may also facilitate execution of transformer modules by the DNN accelerator 1702. For instance, the DNN module 1701 may be the host for the execution of neural network operations in transformer models by the DNN accelerator 1702, e.g., the host for transformer model inference. In some embodiments, the DNN module 1701 may receive an inference request, which may be a request to have a transformer model to make a prediction based on input data. The DNN module 1701 may facilitate cached inference of the transformer model, in which attention tensors (e.g., key tensors, value tensor, etc.) may be cached and reused in the inference of the transformer model. The inference for makingthe prediction may include a sequence of inference phases, which generates a sequence of predicted tokens. The sequence of predicted tokens may be the prediction of the transformer model. In some embodiments, the DNN module 1701 may facilitate generation and execution of transformer models with static shaped caches and inputs. Certain aspects of the DNN module 1701 are provided below in conjunction with FIG. 18.

[0141] The DNN accelerator 1702 executes DNNs provided by the DNN module 1701. For instance, the DNN accelerator 1702 can execute a DNN by running deep learning operations in the DNN. The process of carrying out a deep learning operation is also referred to as a process of executing the deep learning operation or a process of performing the deep learning operation. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. As shown in FIG. 17, the DNN accelerator 1702 includes a memory 1710, a DMA (direct memory access) engine 1720, and data processing units 1730 (individually referred to as "data processing unit 1730"). In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 1702. For example, the DNN accelerator 1702 may include more than one memory 1710 or DMA engine 1720. As another example, the DNN accelerator 1702 may include a single data processing unit 1730. Further, functionality attributed to a component of the DNN accelerator 1702 may be accomplished by a different component included in the DNN accelerator 1702 or by a different system. A component of the DNN accelerator 1702 may be implemented in hardware, software, firmware, or some combination thereof.

[0142] The memory 1710 stores data associated with deep learning operations performed by the DNN accelerator 1702. The memory 1710 may be a system memory. In some embodiments, the memory 1710 includes a DRAM. When the DNN accelerator 1702 executes operations in transformer models, at least part of the memory 1710 may be used to implement KV caches, such as self-attention KV caches and cross-attention KV caches, in the transformer models. The KV caches may be updated during inference of the transformation model. Layout of data in the KV caches may be determined to optimize the efficiency of the DNN accelerator 1701.

[0143] In some embodiments, the memory 1710 may store data to be used by the data processing units 1730 for DNN execution. The memory 1710 may store weights, such as weights of convolutional layers, which are determined by training DNNs. The memory 1710 may further store inputs to DNN layers or outputs of DNN layers, such as data generated bythe data processing units 1730 from performing deep learning operations in DNNs. Example deep learning operations include convolutions (also referred to as "convolutional operations"), layer normalization operations, SoftMax operations, matrix multiplication operations, pooling operations, elementwise operations, activation functions, other types of deep learning operations, or some combination thereof. The memory 1710 may be a main memory of the DNN accelerator 1702. In some embodiments, the memory 1710 includes one or more dynamic random-access memories (DRAMs).

[0144] The DMA engine 1720 facilitates data transfer between the memory 1710 and local memories of the data processing units 1730. For example, the DMA engine 1720 can read data from the memory 1710 and write data into a local memory of a data processing unit 1730. As another example, the DMA engine 1720 can read data from a local memory of a data processing unit 1730and write data into the memory 1710. The DMA engine 1720 provides a DMA feature that allows the data processing unit 1730 to initiate data transfer between the memory 1710 and the local memories of the data processing units 1730 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA engine 1720 may read tensors from the memory 1710, modify the tensors in a way that is optimized for the data processing unit 1730 before it writes the tensors into the local memories of the data processing units 1730.

[0145] The data processing units 1730 perform deep learning operations in DNNs. For instance, a data processing unit 1730 may execute a DNN layer by running one or more deep learning operations in the DNN layer. A data processing unit 1730 may execute a layer, or a portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple data processing units 1730 in parallel. For instance, multiple data processing units 1730 may each perform a portion of a workload for a deep learning operation. Data may be shared between the data processing units 1730. A data processing unit 1730 may also be referred to as a neural processing unit, a compute block, or a compute tile.

[0146] The data processing units 1730 may be capable of running various types of deep learning operations, such as convolution, layer normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Deep learning operations performed by the data processing units 1730 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In anexample, the data processing unit 1730 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the data processing unit 1730 or another data processing unit 1730.

[0147] In the embodiments of FIG. 17, each data processing unit 1730 includes a local memory 1740, a sparsity mode module 1750, a load module 1760, a processing engine 1770, a post-processing engine 1780, and a drain module 1790. Some or all the components of the data processing unit 1730 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the data processing unit 1730. Further, functionality attributed to a component of the data processing unit 1730 may be accomplished by a different component included in the data processing unit 1730, a different data processing unit 1730, another component of the DNN accelerator 1702, or a different system. A component of the data processing unit 1730 may be implemented in hardware, software, firmware, or some combination thereof.

[0148] The local memory 1740 is local to the corresponding data processing unit 1730. In the embodiments of FIG. 17, the local memory 1740 is inside the data processing unit 1730. In other embodiments, the local memory 1740 may be outside the data processing unit 1730. Data in the local memory 1740 may be transferred to or from the memory 1710, e.g., through the DMA engine 1720. For instance, KV caches may be copied from the memory 1710 to the local memory 1740. In some embodiments, data in the local memory 1740 may be transferred to or from the local memory of another data processing unit 1730. The local memory 1740 may store data received, used, or generated by the sparsity mode module 1750, the load module 1760, the processing engine 1770, the post-processing engine 1780, or the drain module 1790. Examples of the data may include input activations, weights, output activations, sparsity bitmaps, and so on.

[0149] In some embodiments, the local memory 1740 may store tensors to be processed by the processing engine 1770 or the post-processing engine 1780. The tensors may be input tensors of deep learning operations. The local memory 1740 may also store tensors generated by the processing engine 1770 or the post-processing engine 1780. The tensors may be output tensors of deep learning operations. The layout of data points of a tensor in the local memory 1740 may depend on the format in which the tensor is stored. In some embodiments, the local memory 1740 may store tensors in various formats, including Z-major format, X-major format, and Y-major format. For a tensor with Z-major format, the local memory 1740 may store data points having the same (x, y) coordinate contiguously. For instance, the data points having the same (x, y) coordinate may be stored at a sequence of memory addresses in the local memory 1740. For a tensor with the ZXY format or ZYX format, the local memory 1740 may store data points having the same (x, y) coordinate contiguously. For instance, the data points having the same (x, y) coordinate may be stored at a sequence of memory addresses in the local memory 1740. For a tensor with X-major format, the local memory 1740 may store data points having the same (y, z) coordinate contiguously. For a tensor with Y-major format, the local memory 1740 may store data points having the same (x, z) coordinate contiguously.

[0150] In some embodiments, the local memory 1740 may store dense tensors (e.g., dense activation tensors, dense weight tensors, etc.), sparse tensors (e.g., sparse activation tensors, sparse weight tensors, etc.), and so on. A dense tensor may be a tensor from which zero-valued elements (if any) are not removed. A dense tensor may be converted to a sparse tensor by removing one or more zero-valued elements in the dense tensor. A sparse tensor may also be referred to as a compressed tensor or packed tensor. The process of converting a dense tensor to a sparse tensor may be referred to as sparsity encoding. Sparsity encoding may also generate a sparsity tensor. Each element in the sparsity tensor may correspond to a different element in the dense tensor and indicate whether the element in the dense tensor is zero or not. The sparsity tensor may indicate positions of elements of the sparse tensor in the dense tensor. The sparsity tensor may be a sparsity bitmap, each element of which is a bit. A sparse tensor may be converted to a dense tensor through a densifying process, in which one or more zeros may be added to the sparse tensor based on the sparsity tensor.

[0151] In some embodiments, the local memory 1740 includes one or more SRAMs. The local memory 1740 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 1740 may include memory banks. The number of data banks in the local memory 1740 may be 16, 64, 128, 1756, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a singlebyte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 1740 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 1740 in multiple read cycles, such as two cycles.

[0152] The sparsity mode module 1750 determines sparsity modes in which the data processing unit 1730 operates to execute DNN layers. For instance, the sparsity mode module 1750 may determine whether to accelerate a layer based on weight sparsity, activation sparsity, or both. The sparsity mode module 1750 select the sparsity mode for a layer from a group of sparsity modes that includes, for example, combined sparsity mode in which the layer is accelerated based on both weight sparsity and activation sparsity, activation sparsity mode in which the layer is accelerated based on activation sparsity but not based on weight sparsity, weight sparsity mode in which the layer is accelerated based on weight sparsity but not based on activation sparsity, and a dense mode in which the layer is not accelerated based on sparsity. In some embodiments (e.g., embodiments where a layer is executed by multiple data processing units 1730), the sparsity mode module 1750 may determine the sparsity mode for all the data processing units 1730 that executes the layer. In some embodiments, the sparsity mode module 1750 may receive configuration parameters from the DNN module 1701. A configuration parameter may correspond to a layer and indicate whether to accelerate the layer based on weight sparsity. The sparsity mode module 1750 may determine the sparsity mode of the layer based on the configuration parameter.

[0153] The load module 1760 loads data from the local memory 1740 to the processing engine 1770 or to the post-processing engine 1780. The load module 1760 may read tensors from the local memory 1740. The tensors may include sparse activation tensors, sparse weight tensors, activation sparsity tensors, weight sparsity tensors, and so on. In some embodiments, the load module 1760 may load data based on the sparsity mode determined by the sparsity mode module 1750. The load module 1760 may select different data to transmit to the processing engine 1770 in different sparsity modes. For instance, the load module 1760 may transmit an activation sparsity tensor and a weight sparsity tensor of a layer to the processing engine 1770 in the combined sparsity mode, while transmit theactivation sparsity tensor but not the weight sparsity tensor to the processing engine 1770 in the activation sparsity mode and transmit the weight sparsity tensor but not the activation sparsity tensor to the processing engine 1770 in the weight sparsity mode. In the dense mode, the load module 1760 does not transmit either the activation sparsity tensor or the weight sparsity tensor to the processing engine 1770.

[0154] In some embodiments, the load module 1760 may process (e.g., densify) data stored in the local memory 1740 before providing the data to the processing engine 1770. In an example, the load module 1760, while operating in the weight sparsity mode, may densify sparse activation tensors to generate dense activation tensors based on corresponding activation sparsity tensors. For instance, the load module 1760 may add one or more zeros into a sparse activation tensor based on an activation sparsity tensor associated with the sparse activation tensor to generate the dense activation tensor. The dense activation tensor includes one or more elements than the sparse activation tensor. The additional element(s) are zero-valued. The load module 1760 may identify one or more elements in the activation sparsity tensor that correspond to the zero-valued element(s), determine the position of each of the zero-valued element(s) in the dense activation tensor, and insert the zero-valued element(s) into the sparse activation tensor based on the determined positions. After the densification, the load module 1760 may transmit the dense activation tensors to the processing engine 1770. The load module 1760 may also transmit corresponding sparse weight tensors and weight sparsity tensors to the processing engine 1770. Activation sparsity tensor of the dense activation tensors may not be loaded to the processing engine 1770.

[0155] In another example, the load module 1760, while operating in the activation sparsity mode, may densify sparse weight tensors to generate dense weight tensors based on corresponding weight sparsity tensors by inserting zeros into sparse weight tensors. The densification of sparse weight tensors may be similar to the densification of sparse activation tensors described above. After the densification, the load module 1760 may transmit the dense weight tensors to the processing engine 1770. The load module 1760 may also transmit corresponding sparse activation tensors and activation sparsity tensors to the processing engine 1770. Weight sparsity tensor of the dense weight tensors may not be loaded to the processing engine 1770. In yet another example, the load module 1760, while operating in the dense mode, may densify both sparse weight tensors and sparse activationtensors. The load module 1760 may generate the input tensor and weight tensor of the layer and transmit the tensors to the processing engine 1770 for executing the layer without sparsity acceleration.

[0156] The processing engine 1770 performs operations in DNNs. The processing engine 1770 may accelerate neural network operations based on sparsity in data. In some embodiments, the processing engine 1770 may operate in a dense mode in which sparsity acceleration is not performed. The processing engine 1770 may include one or more processing cells. In some embodiments, the processing cells may be arranged in one or more rows and one or more columns in the processing engine 1770. Each processing cell may include PEs that may be arranged in an array that includes rows and columns. All the PEs in the processing engine 1770 may constitute a bigger array that includes more rows and columns.

[0157] An example PE may be or may include one or more MAC units that can perform MAC operations. In some embodiments (e.g., embodiments where the data processing unit 1730 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN. The weights in the weight operand may be in different input channels.

[0158] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the load module 1760, into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes forfeeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0159] In some embodiments, the processing engine 1770 may be capable of depthwise convolution, standard convolution, or both. In a depthwise convolution, an MAC unit may perform an MAC operation that includes a sequence of multiplications for an input operand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activation and weight in the same cycle may correspond to the same channel. The sequence of multiplication produces a product operand that includes a sequence of products. The MAC operation may also include accumulations in which multiple product operands are accumulated to produce an output operand of the MAC unit. The processing engine 1770 may output multiple output operands at a time, each of which is generated by a different MAC unit. In a standard convolution, MAC operations may include accumulations across the channels. For instance, as opposed to generating an output operand, a MAC unit may accumulate products across different channels to generate a single output point.

[0160] In some embodiments, the processing engine 1770 may perform MAC operations in quantized deep learning operations, such as MAC operations in a quantized convolution. In some embodiments, an MAC unit in the processing engine 1770 may receive quantized activation and quantized weights and compute a quantized MAC result. The quantized MAC result may be a quantized value in an integer format and may be the output of the MAC unit. In some embodiments, the MAC unit may also include a quantization multiplier that can multiply a quantization scale with the quantized MAC result, and the output of the MAC unit may be a real value in a floating-point format. The MAC unit may include no quantization subtractors as zero-point offsetting is not needed for the MAC operations in quantized deep learning operations.

[0161] In some embodiments, the processing engine 1770 may include sparsity acceleration logic for facilitating sparsity acceleration. For instance, each processing cell in the processing engine 1770 may include one or more sparsity modules. In an example, each MAC column or each MAC row may have a corresponding sparsity module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, a sparsity module accelerates computations in the processing engine 1770 based on sparsity in activations,sparsity in weights, or both. The sparsity module may include a storage unit that stores a sparsity tensor, which may be loaded to the storage unit by the load module 1760. The sparsity tensor may be an activation sparsity tensor, a weight sparsity tensor, or a combined sparsity tensor.

[0162] An activation sparsity tensor may be the sparsity tensor of an activation tensor and has the same number of elements as the activation tensor. An element in the activation sparsity tensor may indicate whether the corresponding element in the activation tensor is zero or not. For instance, a zero-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is zero. A one-valued in the activation sparsity tensor may indicate that the corresponding element in the activation tensor is nonzero. A weight sparsity tensor may be the sparsity tensor of a weight tensor and has the same number of elements as the weight tensor. An element in the weight sparsity tensor may indicate whether the corresponding element in the weight tensor is zero or not. For instance, a zero-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is zero. A one-valued in the weight sparsity tensor may indicate that the corresponding element in the weight tensor is nonzero. The sparsity module may generate a combined sparsity tensor using an activation sparsity tensor and a weight sparsity tensor. For instance, the sparsity module may multiply an element of the activation sparsity tensor with a corresponding element of the weight sparsity tensor to compute an element of the combined sparsity tensor. The positions of the three elements in their corresponding sparsity tensors may match. In some embodiments, each element in a sparsity tensor may be a bit, and the sparsity tensor may be referred to as a sparsity bitmap.

[0163] The sparsity module may use the sparsity tensor to identify activations and weights to be used in MAC operations by the MAC units. In an embodiment where the processing engine 1770 operates in the combined sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a combined sparsity tensor. In an embodiment where the processing engine 1770 operates in the activation sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of an activation sparsity tensor. In an embodiment where the processing engine 1770 operates in the weight sparsity mode, the sparsity module may identify activations and weights that correspond to nonzero valued elements of a weightsparsity tensor. The sparsity module may be bypassed in the dense mode as no sparsity acceleration would be conducted.

[0164] The post-processing engine 1780 processes outputs of the processing engine 1770. The post-processing engine 1780 may include one or more post-processing elements. In some embodiments, the post-processing elements in the post-processing engine 1780 may be arranged in an array that has rows and columns. In some embodiments, the postprocessing engine 1780 computes activation functions. The post-processing engine 1780 may receive outputs of the processing engine 1770 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 1780 may perform other types of post processing on outputs of the processing engine 1770. For instance, the post-processing engine 1780 may apply a bias on an output of the processing engine 1770. In some embodiments, the post-processing engine 1780 may be bypassed for certain neural network operations.

[0165] The drain module 1790 drains data from the processing engine 1770 or from the post-processing engine 1780. The drain module may write the data to the local memory 1740. The drained data may be tensors, such as output tensors of neural network operations. In some embodiments, the drain module 1790 may drain data on a cell level. For each processing cell, the drain module 1790 may drain outputs of PEs in the processing cell based on a row index or column index of each PE. For instance, the drain module 1790 may use a sequence of cycles to drain data from a processing cell. The drain module 1790 may drain the output of some of the PE s in each cycle. The sequence of the cycles may be configured based on a configuration parameter indicating the operation mode of the load module 1760.

[0166] In some embodiments, the drain module 1790 includes sparsity encoding logic that can convert outputs of the processing engine 1770 from a dense format to a sparse format. For instance, the drain module 1790 may be implemented with one or more sparsity encoders. A sparsity encoder converts dense data to compressed data based on sparsity in the dense data. For instance, the sparsity encoder may remove zeros in an activation tensor computed by the processing engine 1770 to convert the activation tensor to a compressed activation tensor. The sparsity encoder may also generate sparsity tensors, including activation sparsity tensors.

[0167] In some embodiments, the data drained from the processing engine 1770 may be at least part of an output tensor (e.g., the output tensor 1630 in FIG. 16) of a deep learning operation. The sparsity encoder may generate a compressed version of the output tensor. The sparsity encoder may identify every zero-valued activation in the output tensor and remove these activations from the output tensor to generate a compressed activation tensor (aka "sparse activation tensor"). The sparsity encoder may also generate one or more sparsity tensors for the output tensor. A sparsity tensor may correspond to a portion of the output tensor (e.g., the vector 1635 in FIG. 16). The sparsity tensor may include sparsity elements (e.g., bits), each of which corresponds to a different activation in the vector and indicates whether the corresponding activation is zeroed or not.

[0168] The drain module 1790 may write the compressed activation tensor and the one or more sparsity tensors into the local memory 1740. The sparse activation tensor and the one or more sparsity tensors may be further loaded to the memory 1710, e.g., through the DMA engine 1720. Additionally or alternatively, the sparse activation tensor and the one or more sparsity tensors may be loaded by the load module 1760 to the processing engine 1770 for further computation, e.g., for performing a deep learning operation in the next layer.

[0169] FIG. 18 is a block diagram of a DNN module 1800, in accordance with various embodiments. The DNN module 1800 may be an embodiment of the DNN module 1701 in FIG. 17. As shown in FIG. 18, the DNN module 1800 includes an interface module 1810, a training module 1820, a compressing module 1830, a compiler 1840, a transformer execution module 1850, and a datastore 1860. In other embodiments, alternative configurations, different or additional components may be included in the DNN module 1800. Further, functionality attributed to a component of the DNN module 1800 may be accomplished by a different component included in the DNN module 1800 or a different module or system.

[0170] The interface module 1810 facilitates communications of the DNN module 1800 with other modules or systems. For example, the interface module 1810 establishes communications between the DNN module 1800 with an external database to receive data that can be used to train DNNs or input into DNNs to perform tasks. As another example, the interface module 1810 supports the DNN module 1800 to distribute DNNs to other systems, e.g., computing devices configured to apply DNNs to perform tasks. The interface module 1810 may receive inference requests from users of transformer models.

[0171] The training module 1820 trains DNNs by using a training dataset. The training module 1820 forms the training dataset. In an example where the training module 1820 trains an DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe ground-truth classifications of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a part of the training dataset may be used to initially train the DNN, and the rest of the training dataset may be held back as a validation subset used by the training module 1820 to validate performance of a trained DNN. The portion of the training dataset not including the tuning subset and the validation subset may be used to train the DNN.

[0172] The training module 1820 also determines hyperparameters for training the DNN. Hyperparameters are variables specifying the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., weights of filters). In some embodiments, hyperparameters include variables determining the architecture of the DNN, such as number of hidden layers, etc. Hyperparameters also include variables which determine how the DNN is trained, such as batch size, number of epochs, etc. A batch size defines the number of training samples to work through before updating the parameters of the DNN. The batch size is the same as or smallerthan the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backwards through the entire network. The number of epochs defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each training sample in the training dataset has had an opportunity to update the parameters inside the DNN. An epoch may include one or more batches. The number of epochs may be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0173] The training module 1820 defines the architecture of the DNN, e.g., based on some of the hyperparameters. The architecture of the DNN includes an input layer, an output layer, and a plurality of hidden layers. The input layer of an DNN may include tensors (e.g., a multidimensional array) specifying attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layers are layers between the input layer andoutput layer. The hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully-connected layers, normalization layers, SoftMax or logistic layers, and so on. The convolutional layers of the DNN abstract the input image to a feature map that is represented by a tensor specifying the feature map height, the feature map width, and the feature map channels (e.g., red, green, blue images include 3 channels). A pooling layer is used to reduce the spatial volume of input image after convolution. It is used between two convolution layers. A fully-connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

[0174] In the process of defining the architecture of the DNN, the training module 1820 also adds an activation function to a hidden layer or the output layer. An activation function of a layer transforms the weighted sum of the input of the layer to an output of the layer. The activation function may be, for example, a ReLU activation function, a tangent activation function, or other types of activation functions.

[0175] After the training module 1820 defines the architecture of the DNN, the training module 1820 inputs a training dataset into the DNN. The training dataset includes a plurality of training samples. An example of a training sample includes an object in an image and a ground-truth label of the object. The training module 1820 modifies the parameters inside the DNN ("internal parameters of the DNN") to minimize the error between labels of the training objects that are generated by the DNN and the ground-truth labels of the objects. The internal parameters include weights of filters in the convolutional layers of the DNN. In some embodiments, the training module 1820 uses a cost function to minimize the error.

[0176] The training module 1820 may train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm will work through the entire training dataset. One epoch means that each sample in the training dataset has had an opportunity to update internal parameters of the DNN. After the training module 1820 finishes the predetermined number of epochs, the training module 1820 may stop updating the parameters in the DNN. The DNN having the updated parameters is referred to as a trained DNN.

[0177] The training module 1820 may also verify accuracy of trained or compressed DNNs. In some embodiments, the training module 1820 inputs samples in a validation dataset into a trained DNN and uses the outputs of the DNN to determine the model accuracy. In someembodiments, a validation dataset may be formed of some or all the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples, other than those in the training sets. In some embodiments, the training module 1820 may determine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN. The training module 1820 may use the following metrics to determine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN), where precision may be how many the DNN correctly predicted (TP or true positives) out of the total it predicted (TP + FP or false positives), and recall may be how many the DNN correctly predicted (TP) out of the total number of objects that did have the property in question (TP + FN or false negatives). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single measure.

[0178] The training module 1820 may compare the accuracy score with a threshold score. In an example where the training module 1820 determines that the accuracy score of the DNN is less than the threshold score, the training module 1820 may re-train the DNN. In one embodiment, the training module 1820 may iteratively re-train the DNN until the occurrence of a stopping condition, such as the accuracy measurement indication that the DNN may be sufficiently accurate, or a number of training rounds having taken place.

[0179] The compressing module 1830 compresses DNNs. For instance, the compressing module 1830 may add pruning operations to DNN layers to reduce computational complexity or memory usage. A pruning operation may prune weight tensors of a DNN layer by changing one or more nonzero valued weights of the layer to zeros. The modification may be done before, during, or after training. Weights may be pruned during training, during inference, or a combination of both. The compressing module 1830 may determine a sparsity ratio for a DNN layer. The sparsity ratio may be a ratio of the number of zero-valued weight to the total number of weights in the layer. The compressing module 1830 may perform the pruning operation till the sparsity ratio of the DNN layer meets a target sparsity ration, such as 10%, 20%, 30%, 180%, 50%, and so on.

[0180] In some embodiments, the compressing module 1830 may select one or more layers in a DNN and modify each selected layer with a pruning operation. For instance, the compressing module 1830 may select computationally complex layers, such as layers with large filters. For a pruning operation of a layer or of a type of layer, the compressing module 1830 may determine a weight threshold that would not cause a loss of the accuracy of theDNN to exceed an accuracy loss constraint. A pruning operation may modify weights having absolute values above the weight threshold to zeros and leave the other weights unchanged. The weight pruning can reduce memory storage as zero-valued weights may not be stored. Also, the number of operations in the layer can be reduced as computations on zero-valued weights can be skipped without impacting the output of the layer. In some embodiments, the compressing module 1830 may also measure energy saving, final DNN accuracy, or layer-wise sparsity caused by pruning operations.

[0181] After compressing a DNN, the compressing module 1830 may fine tune the DNN, e.g., through a retraining process. The compressing module 1830 may fine tunes DNNs after weights are pruned. In some embodiments, the fine-tuning process is a retraining or further training process. For instance, after weights in a DNN are pruned, the compressing module 1830 may further train the DNN by inputting a training dataset into the DNN. The values of the unpruned weights in the DNN may be modified based on outputs of the DNN and ground-truth labels of the training samples in the training dataset. In some embodiments, the values of the pruned weights (i.e., zero) are not changed during the fine-tuning process. For instance, the compressing module 1830 may place a mask over a pruned weight block and the mask can prevent values in the pruned weight blocks from being changed during the fine-tuning process. In other embodiments, the values of all weights, including the pruned weights, may be changed during the fine-tuning process. After one or more cycles of retraining and weight changing by the compressing module 1830, the compressing module 1830 may perform a new pruning process, e.g., by selecting weight blocks and pruning the selected weight blocks. In some embodiments, the weight pruning process may be repeated multiple times before the fine-tuning process is done. In some embodiments, the number of epochs in the fine-tuning process may be different from the number of epochs in the training process in which the pre-pruning values of the weights are determined. For instance, the fine-tuning process may have less epochs than the training process. In an example, the number of epochs in the fine-tuning process may be relatively small, such as 2, 3, 18, 5, and so on.

[0182] The compiler 1840 compiles information of DNNs to executable instructions that can be executed, e.g., by the DNN accelerator 1702, to carry out neural network operations in DNNs. The compiler 1840 may generate executable DNNs, e.g., DNNs that can be at least partially executed by the DNN accelerator 1701. In some embodiments, the compiler 1805may generate a graph representing a DNN. The graph may include nodes and edges. A node may represent a specific neural network operation in the DNN. An edge may connect two nodes and represent a connection between the two corresponding neural network operations. In an example, an edge may encode a tensor that flows from one of the neural network operations to the other neural network operation. The tensor may be an output tensor of the first neural network operation and an input tensor of the second neural network operation. The edge may encode one or more attributes of the tensor, such as size, shape, storage format, and so on. The compiler 1840 may use the graph to generate instructions (e.g., compilation descriptors). The instructions would be executed by components of the DNN accelerator 1702 to execute the DNN. In some embodiments, the compiler 1840 may modify graphs of transformer models with static shaped inputs. For instance, the compiler 1840 may perform graph modifications illustrated in FIG. 15A and FIG. 15B. Functionality attributed to the compiler 1840 may be accomplished by the transformer execution module 1850, or vice versa.

[0183] The transformer execution module 1850 facilitates execution of transformer models, including transformer models with static shaped KV caches and static shaped inputs. The transformer execution module 1850 may map operations in transformer models to the DNN accelerator 1710, e.g., to the data processing units 1730 in the DNN accelerator 1710. The transformer execution module 1850 may modify transformer models based on attributes of hardware resources (e.g., memories, compute elements, etc.) of the DNN accelerator 1710. The transformer execution module 1850 may also determine parameters that will be used in inference of transformer models.

[0184] In some embodiments, the transformer execution module 1850 may determine optimal data layouts in KV caches in transformer models. Memory footprint can be a challenge to both inference latency and energy consumption of transformer model inference on DNN accelerators. During cached inference using decoder, self-attention KV cache and cross-attention KV cache may undergo a series of transpose and reshape operations (e.g., the transpose and reshape operations shown in the graphs in FIGS. 15A and FIG. 15B) before getting fed to Matrix Multiplication (MatMul) operation inside MHA layer. Transpose operations are associated with data movement. In cases where the KV cache size is large, these operations not only result in compute cycles, but also cause as lot of memory transaction between the system memory (e.g., the memory 1710) and the local memory(e.g., the local memory 1740). The enormous size can also lead to many spills writes from the local memory to the system memory when the output of transpose op does not fit entirely within the local memory, even with tiled operation. Consequently, this brings in addition spill reads from the system memory to the local memory. An optimal layout for KV caches (including self-attention KV cache and cross-attention KV cache) may be selected to reduce or even remove the number of the transpose operations in the transformer model.

[0185] The optimal layout of KV cache may depend on how the matrix multiplication operation is mapped to the DNN accelerator. In some embodiments, matrix multiplication operation is mapped to convolution operation, e.g., by unrolling the 3D matrix multiplication operation into h X 2D matrix multiplication operation. The query tensor Q (1x1x64) may be reshaped to 1x64x1x1 as weights and this operation can be free. During model compilation, the compiler may see that past key tensor has transpose associated with the matrix multiplication operation and inserts a transpose operation. In an example where NHWC layout is supported, the compiler may insert another transpose operation, before reshaping the K tensor as weight. N may denote batch size, H may be the height of the tensor, W may be the width of the tensor, and C may be the length of the tensor in the channel dimension. Since the transpose operations cancel each other, the key tensors are already in the optimal layout, e.g., NCHW, for both self-attention and cross-attention. The compiler may omit the two transpose operations. Certain aspects of convolution operation are described above in conjunction with FIG. 16.

[0186] In some embodiments, the transformer execution module 1850 may reshape tensors in transformer models to achieve optimal data layout in KV caches. There may be no transpose associated with the matrix multiplication operation ("MatMul op") and hence one transpose operation may be introduced by the transformer execution module 1850 or the compiler 1840 to map the value tensor as weight. This indicates that the optimal layout for the value tensor may be NCWH (1x64x128) so that this transpose operation can be removed altogether. This methodology may be applied to the entire decoder model. This methodology may also finalize the output layout of both decoder first inference model (that outputs both self-attention KV and cross-attention KV) as well as the decoder with KV cache model. To avoid any additional runtime overhead and extra data copies between the host (e.g., the DNN module 1701) and the DNN accelerator 1702, it may be essential to specify that the output self-attention key cache and cross-attention key cache may have the samelayout as the input, e.g., NCHW. In some embodiments, the self-attention value cache and cross-attention value cache have the same layout as the output, e.g., NCWH. These specified layouts may be saved and used for model compilation by the compiler before the compiled model can be deployed on the DNN accelerator. In light of the cache layout optimization, the cache update block (e.g., the cache update block 1100 in FIG. 11 or the cache update block 1410 in FIG. 14) mat be modified to pad and update the key cache and the value cache in the third and the fourth dimensions, respectively, as the sequence dimension is the third dimension for key and is the fourth dimension for value.

[0187] In some embodiments, the transformer execution module 1850 may determine one or more data precisions for transformer model inference. For instance, the transformer execution module 1850 may determine data precisions for encoder input, encoder output, decoder input, decoder output, attention masks, position IDs, keys, values, queries, and so on. The transformer execution module 1850 may select different data precisions for different types of data in the transformer model. For instance, the transformer execution module 1850 may select different data precisions for different inputs and outputs. Examples of data precisions may be FP16, BF16, FP32, INT32, LIINT16, and so on. The transformer execution module 1850 may also determine compilation time precision used during model compilation, e.g., by the compiler 1840. The transformer execution module 1850 may also determine runtime precision for data consumed or generated in the transformer model during runtime. Different data precisions can result in different efficiency. For instance, padding the KV cache in FIG. 14 using FP16 data (0 values) ensures lOx speedup compared to FP32 padding. This is because KV cache sizes are large and unequal precision introduces extra data movement and overhead.

[0188] In some embodiments, the transformer execution module 1850 selects INT32 as the precision for input tokens as input tokens usually range from 0 to 65536 and therefore, can be initialized using LIINT16 during runtime to provide small amount of memory savings when the host transfer the data to the accelerator where it is upconverted to INT32. In some embodiments, the transformer execution module 1850 ensures that input tokens are not in any FP datatype as the embedding layer which takes the input tokens and outputs corresponding input embeddings may be a look up table and indices of the lookup table may have an integer format. The input tokens may have the same integer format.

[0189] In some embodiments, the positions inside attention masks corresponding to padding tokens may have the minimum possible value of FP32 (-3.4e+38), as opposed to 0. The transformer execution module 1850 may replace the -3.4e+38 value with -inf. In some embodiment, when -inf is multiplied by 0, it leads to NAN, and this causes the application to crash. The transformer execution module 1850 may identify all locations in model source code where FP32.min datatype is used with attention mask replaces these locations with FP16.min value (-65504). This can stop any overflow once the transformer model is compiled and deployed on the DNN accelerator 1702.

[0190] In some embodiments, the transformer execution module 1850 may determine one or more maximum lengths for KV cache in a transformer model. The maximum length may be used by the transformer model to pad an input sequence to obtain static shaped inputs across multiple inference phases of the decoder. The maximum length may also be used by the transformer model to pad the KV cache. In some embodiments, the transformer execution module 1850 may create multiple static decoders with KV caches having different maximum lengths for a transformer model. The transformer execution module 1850 may select one of the static decoders for a transformer model based on the maximum length of the KV cache and parameters of the transformer model.

[0191] In an example where SLmax= 1024 for the decoder, the transformer execution module 1850 may create a chain of five static decoders for SLmax=[128, 256, 512, 768, 1024], These decoders may be used in different inference phases. For instance, the first decoder with SLmax= 128 may be used in the first 128 inference phases, then switching to the second decoder with SLmax= 256, further switching to the third decoder with SLmax= 512 then the fourth decoder with SLmax= 768. The last decoder with SLmax= 1024 last. The cached inference may switch to the next decoder in response to a determination that the KV cache of the current decoder is full, e.g., a determination that the position ID has reached SLmaxof the current decoder. This ensures that the additional overhead of computing of padded tokens and data transfer of padded KV cache is not at the worst-case scenario. The transformer execution module 1850 may allow decoder(i) input tensors to be filled by decoder(i-l) data asynchronously. This will hide the latency of data transfers across two models.

[0192] Weights may be shared across these decoders. Using shared weights can be either enabled by using weights as inputs to model, rather than having them embedded in thecompiled blob. In some embodiments, the transformer execution module 1850 may keep weights outside compiled blob, weight sharing can be easier.

[0193] The datastore 1860 stores data received, generated, used, or otherwise associated with the DNN module 1800. For example, the datastore 1860 stores the datasets used by the training module 1820 and training module 1820. The datastore 1860 may also store data generated by the training module 1820 and training module 1820, such as the hyperparameters for training DNNs, internal parameters of trained DNNs (e.g., weights, etc.), data for sparsity acceleration (e.g., sparsity bitmap, etc.), and so on. The datastore 1860 may store graphs, configuration parameters, compilation descriptors, or other data generated by the compiler 1840 or the transformer execution module 1850. The datastore 1860 may include one or more memories. In the embodiment of FIG. 18, the datastore 1860 is a component of the DNN module 1800. In other embodiments, the datastore 1860 may be external to the DNN module 1800 and communicate with the DNN module 1800 through a network. In some embodiments, the datastore 1860 may be implemented in the memory 1710.

[0194] FIG. 19 illustrates an example sparse cell 1900, in accordance with various embodiments. The sparse cell 1900 may be a processing cell in a processing engine, e.g., the processing engine 1770 in FIG. 17. The sparse cell 1900 includes 16 MAC units 1910 (individually referred to as "MAC unit 1910"), which constitutes a MAC array having four rows and four columns. The MAC array has a spatial shape of 19x4, meaning the height of the MAC array is four and the width of the MAC array is also 19. The sparse cell 1900 also includes 16 weight register files 1920 (individually referred to as "weight register file 1920"), 16 activation register files 1930 (individually referred to as "activation register file 1930"), four row buffers 1940 (individually referred to as "row buffer 1940"), and sparsity modules 1960 (individually referred to as "sparsity module 1960"). In other embodiments, the sparse cell 1900 may include fewer, more, or different components. For example, the sparse cell 1900 may include a different number of MAC units 1910, weight register files 1920, activation register files 1930, row buffers 1940, or sparsity modules 1960. As another example, the sparse cell 1900 may include column buffers in lieu of or in addition to the row buffers 1940. Also, the shape (e.g., the height or width) of the MAC array may be different.

[0195] The MAC units 1910 are configured to perform MAC operations. Each MAC unit 1910 may include one or more multipliers and one or more adders. A multiplier may multiply anactivation with a weight at a time to compute a product. In some embodiments (e.g., embodiments where the MAC unit 1910 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. An adder may accumulate products computed by the multipliers. Even though not shown in FIG. 19, the sparse cell may include an adder tree including a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units 1910. The number of adders in the first tier may be half of the number of the MAC units 1910, and each adder may accumulate the outputs of two MAC units 1910. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the sparse cell 1900.

[0196] The weight register files 1920 store weights to be processed in MAC operations. In the embodiments of FIG. 19, four weight register files 1920 are grouped into a storage set that stores data to be used by a column of MAC units 1910. There are four storage sets corresponding to the four columns of MAC units 1910. In some embodiments, a weight register file 1920 may correspond to a MAC unit 1910 and store data to be processed by the MAC unit. In some embodiments, all the 16 weight register files 1920 constitute a weight storage unit.

[0197] The activation register files 1930 stores activations to be processed in MAC operations. In the embodiments of FIG. 19, four activation register files 1930 are grouped into a storage set that stores data to be used by a row of MAC units 1910. There are four storage sets corresponding to the four rows of MAC units 1910. In some embodiments, an activation register file 1930 may correspond to a MAC unit 1910 and store data to be processed by the MAC unit. In some embodiments, all the 16 activation register files 1930 constitute an activation storage unit. The row buffers 1940 store outputs of the MAC units 1910. Each row buffer 1940 may drain outputs of a single row of MAC units 1910.

[0198] The sparsity module 1960 facilitates dynamic sparsity-based acceleration in the sparse cell 1900. In the embodiments of FIG. 19, each sparsity module 1960 includes a sparsity tensor storage unit 1965 and a control logic 1967. The sparsity tensor storage unit 1965 stores combined sparsity tensors. A combined sparsity tensor stored in the sparsitytensor storage unit 1965 may correspond to an activation tensor and a weight tensor. A nonzero element in the combined sparsity tensor may correspond to a nonzero activationweight pair that includes a nonzero activation and a nonzero weight. The position of the nonzero activation in the activation tensor may match the position of the nonzero weight in the weight tensor. The product of the nonzero activation and nonzero weight would be nonzero.

[0199] The control logic 1967 may control transmission of activations and weights stored from the weight register files 1920 and the activation register files 1930 to the MAC units 1910 based on sparsity tensors. For instance, the control logic 1967 may select a subset of the weights stored in the weight register files 1920 and select a subset of activations stored in the activation register files 1930 based on a combined sparsity tensor. The selected weights and activations constitute nonzero activation-weight pairs. The control logic 1967 may transmit the selected weights and activations to the MAC units 1910 for performing MAC operations. The other weights stored in the weight register files 1920 and the other activations stored in the activation register files 1930 are skipped from computation. In the embodiments of FIG. 19, each sparsity module 1960 controls sparsity acceleration in a respective MAC unit 1910. As the sparsity acceleration is either based on both weight sparsity and activation sparsity, 16 sparsity modules 1960 are used for acceleration computations in the 16 MAC units 1910.

[0200] As shown in FIG. 19, the sparse cell 1900 is associated with multiplexers (MUXs) 1903, 1904, 1905, and 1906. In other embodiments, the sparse cell 1900 may be associated with a different number of MUXs or other devices. The MUX 1903 facilitates loading weights, e.g., from the local memory 340, into the weight register files 1920. The MUX 1904 facilitates loading activations, e.g., from the local memory 340, into the activation register files 1930. The MUX 1905 facilitates loading sparsity tensors into the sparsity tensor storage unit 1965. The MUX 1906 may be a drain MUX that can facilitate draining outputs of the MAC units 1910, e.g., to the local memory 340.

[0201] In some embodiments, the sparse cell 1900 may also execute matrix multiplications converted from Fourier transform operations. For an example Fourier transform operation, the MAC units 1910 may perform MAC operations in the two sequences of matrix multiplications converted from the Fourier transform operation. The weight register files 1920 may be used to store data points in transformation tensor of the Fourier transformoperation. The activation register file 1930 may be used to store data points in the input tensor of the Fourier transform operation. The row buffers 1940 may store data points in the output tensor of the Fourier transform operation.

[0202] FIG. 20 illustrates a sparse cell array 2000, in accordance with various embodiments. The sparse cell array 2000 may be an example of the processing engine 1770 in FIG. 17. In FIG. 20, the sparse cell array 2000 includes sparse cells 2010 (individually referred to as "sparse cell 2010") arranged in four columns and four rows, an activation memory 2020, and a weight memory 2030. The sparse cell array 2000 may also be referred to as a data processing unit. In other embodiments, the sparse cell array 2000 may include fewer, more, or different components. For instance, the sparse cell array 2000 may include a different number of columns, rows, or sparse cells 2010.

[0203] Each sparse cell 2010 may perform sparsity accelerated MAC operations. The sparse cells 2010 may facilitate dynamic sparsity mode. For instance, the sparsity modes of a sparse cell 2010 may be dynamically changed between a combined sparsity mode, an activation sparsity mode, a weight sparsity mode, and a dense mode. An embodiment of a sparse cell 2010 may be the sparse cell 1900 in FIG. 19. The activation memory 2020 stores activations, such as activations in input tensors of deep learning operations. Activations may be loaded from the activation memory 2020 to sparse cells 2010. The weight memory 2030 stores weights, such as weights in filters of deep learning operations. Weights may be loaded from the weight memory 2030 to sparse cells 2010. The activation memory 2020 or weight memory 2030 may be a buffer. In other embodiments, the sparse cell array 2000 may include a dense data memory and a sparse data memory in lieu of the activation memory 2020 and weight memory 2030. The dense data memory may store dense tensors, e.g., dense tensors generated by the load module 1760. The sparse data memory may store sparse tensors.

[0204] The sparse cell array 2000 may also execute matrix multiplications in Fourier transform operations. The activation memory 2020 may be used to store input tensors of the Fourier transform operations. The weight memory 2030 may be used to store transformation matrices of the Fourier transform operations.

[0205] FIG. 21 illustrates an example PE 2100, in accordance with various embodiments. The PE 2100 may be a unit component of a processing cell, e.g., a processing cell in the processing engine 1770. In the embodiments of FIG. 21, the PE 2100 includes an MAC unit2105, an activation register file 2110, a weight register file 2120, an output register file 2150, and a sparsity accelerator 2160. The MAC unit 2105 includes a multiplier 2130 and an adder 2140. In other embodiments, the PE 2100 may include fewer, more, or different components.

[0206] The activation register file 2110 stores an activation operand, which may be a context. The activation register file 2110 may be an example of the activation register files 1930 in FIG. 19. The weight register file 2120 stores a weight operand. The weight register file 2120 may be an example of the weight register files 1920 in FIG. 9. The activation operand and weight operand may be loaded from a memory (e.g., the memory 340) into the activation register file 2110 and the weight register file 2120, respectively. The sparsity accelerator 2160 receives a sparsity bitmap 2115 that corresponds to the sparse tensor in the weight register file 2120. The sparsity bitmap 2115 may be a combined sparsity bitmap when the MAC unit 2105 operates in a combined sparsity mode. The sparsity bitmap 2115 may be an activation sparsity bitmap when the MAC unit 2105 operates in an activation sparsity mode. The sparsity bitmap 2115 may be a weight sparsity bitmap when the MAC unit 2105 operates in a weight sparsity mode. The sparsity bitmap 2115 may have the same size (e.g., the same number of elements) as or a larger size than the activation operand or the weight operand.

[0207] Using the sparsity bitmap 2115, the sparsity accelerator 2160 selects four activations from the activation register file 2110 and selects four weights from the weight register file 2120. The sparsity accelerator 2160 transmits the selected activations and weights to the multiplier 2130. These selected data elements correspond to the nonzero valued elements of the sparsity bitmap 2115. The four selected activations and the four selected weights may constitute four activation-weight pairs. The multiplier 2130 may compute a product based on each activation-weight pair and therefore, compute four products in total. The four products may be provided to the adder 2140. Even though FIG. 21 shows a single multiplier 2130, the MAC unit 2105 may include multiple multipliers that can perform multiple multiplication operations at the same time.

[0208] The adder 2140 accumulates the four products and computes a unit-level internal partial sum. The four unselected elements of the dense tensor are not processed to save power and time, which would not impact the value of the unit-level internal partial sum. For instance, when the dense tensor is a dense activation tensor, the weights corresponding tothe unselected activations are zeros so the products of the unselected activations and the weights would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. Similarly, when the dense tensor is a dense weight tensor, the activations corresponding to the unselected weights are zeros so the products of the unselected weights and the activations would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. In other embodiments, the MAC unit 2105 may operate in a dense mode in which the sparsity bitmap 2115 is not used and the sparsity accelerator 2160 is inactive. The MAC unit 2105 may process all the activations in the activation operand and all the weights in the weight operand.

[0209] The unit-level internal partial sum may be stored in the output register file 2150. In some embodiments, the unit-level internal partial sum may be used multiple times. For instance, the activation operand may represent N data blocks in the input tensor of the convolution, where N is an integer greater than 1. Instead of processing all the N data blocks to compute N unit-level internal partial sums, the unit-level internal partial sum is computed once and used N times in the convolutional layers as N unit-level internal partial sums.

[0210] In some embodiments, the PE 2100 receives one or more PE-level internal partial sums from one or more other PEs. The adder 2140 or an accumulator (not shown in FIG. 21) can accumulate the one or more PE-level internal partial sums with the PE-level internal partial sum of the PE 2100 and store the result of the accumulation (i.e., a multi-PE internal partial sum) in the output register file 2150. The one or more other PEs may be in the same column as the PE 2100 in a sparse cell. The multi-unit internal partial sum may be a columnlevel internal partial sum. In some embodiments, the PE-level internal partial sum of the PE 2100 or the multi-unit internal partial sum may be sent to one or more other PEs for further accumulation.Example Method for Transformer Model Execution

[0211] FIG. 22 is a flowchart of a method for executing a transformer model, in accordance with various embodiments. The method 2200 may be performed by the DNN system 1700 in FIG. 17. Although the method 2200 is described with reference to the flowchart illustrated in FIG. 22, many other methods for executing transformer models may alternatively beused. For example, the order of execution of the steps in FIG. 22 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0212] The DNN system 1700 generates 2210 a padded input sequence by adding one or more padding tokens to an input sequence. The input sequence comprises one or more input tokens. In some embodiments, the one or more padding tokens are added to a right edge of the input sequence. In some embodiments, the input sequence includes one or more start tokens. In some embodiments, the input sequence includes one or more tokens predicted by the transformer model. In some embodiments, the DNN system 1700 generates a positional encoding vector for the padded input sequence, the positional encoding vector indicating a position of each token in the padded input sequence. The decoder computes the attention tensor based on the positional encoding vector.

[0213] In some embodiments, the DNN system 1700 generates an attention mask that indicates one or more positions of the one or more padding tokens in the padded input sequence and one or more positions of the one or more input tokens in the padded input sequence. The decoder computes the attention tensor based on the attention mask. In some embodiments, a dimension of the attention mask is the same as a dimension of the padded input sequence. In some embodiments, the attention mask includes a sequence of elements, each of which corresponds to a token in the padded input sequence and indicates whether the token is an input token or a padding token. In some embodiments, the attention mask comprises a first group of elements and a second group of elements. The second group of elements is afterthe first group of elements. Each element in the first group has a first value and indicates a position of an input token in the padded input sequence. Each element in the second group has a second value and indicates a position of a padding token in the padded input sequence. In some embodiments, the attention mask further comprises an additional element that is afterthe second group of elements, and the additional element has the first value.

[0214] The DNN system 1700 provides 2220 the padded input sequence to a decoder of the transformer model. The decoder computes an attention tensor from the one or more input tokens. A dimension of the attention tensor is equal to a dimension of the input sequence. In some embodiments, the attention tensor comprises keys or values. The DNN system 1700 provides the attention tensor to a self-attention layer of the decoder. In some embodiments, the attention tensor comprises one or more vectors. A vector is computed inthe decoder from a single input token. In some embodiments, the DNN system 1700 provides the attention tensor to a MHA layer of the decoder. The MHA layer comprises multiple heads. A head comprises one or more neural network operations performed using a part of the attention tensor. A number of heads in the MHA layer is equal to another dimension of the cache. In some embodiments, the decoder is selected from decoders having caches with different dimensions.

[0215] The DNN system 1700 stores 2230 the attention tensor in a cache. A dimension of the cache is equal to a dimension of the padded input sequence. In some embodiments, the DNN system 1700 stores the attention tensor in the cache when the dimension of the cache is equal to the dimension of the input sequence. After storing the attention in the cache, the DNN system 1700 changes the dimensions of the cache to the dimension of the padded input sequence.

[0216] In some embodiments, the decoder computes the attention tensor in a first inference phase. The decoder computes a new attention tensor in a second inference phase that is after the first inference phase. The DNN system 1700 stores the new attention tensor in the cache. In some embodiments, the DNN system 1700 updates the padded input sequence by replacing a padding token with a predicted token. The predicted token is generated by the transformer model in the first inference phase. The decoder computes the new attention tensor in the second inference phase based on the last generated token, e.g., the predicted token. In some embodiments, a dimension of the new attention tensor is one. In some embodiments, the DNN system 1700 stores the new attention tensor by obtaining a concatenated tensor by concatenating the new attention tensor to a right edge of the attention tensor and storing the concatenated tensor in the cache.Example Computing Device

[0217] FIG. 23 is a block diagram of an example computing device 2300, in accordance with various embodiments. In some embodiments, the computing device 2300 can be used as at least part of the DNN system 1700. A number of components are illustrated in FIG. 23 as included in the computing device 2300, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 2300 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computingdevice 2300 may not include one or more of the components illustrated in FIG. 23, but the computing device 2300 may include interface circuitry for coupling to the one or more components. For example, the computing device 2300 may not include a display device 2306, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 2306 may be coupled. In another set of examples, the computing device 2300 may not include an audio input device 2318 or an audio output device 2308 but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 2318 or audio output device 2308 may be coupled.

[0218] The computing device 2300 may include a processing device 2302 (e.g., one or more processing devices). The processing device 2302 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 2300 may include a memory 2304, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 2304 may include memory that shares a die with the processing device 2302. In some embodiments, the memory 2304 includes one or more non-transitory computer- readable media storing instructions executable to perform operations for executing transformer models (e.g., the method 2200 described in conjunction with FIG. 22) or some operations performed by one or more components of the DNN system 1700. The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 2302.

[0219] In some embodiments, the computing device 2300 may include a communication chip 2312 (e.g., one or more communication chips). For example, the communication chip 2312 may be configured for managing wireless communications for the transfer of data to and from the computing device 2300. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0220] The communication chip 2312 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 2312 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 2312 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 2312 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 2312 may operate in accordance with other wireless protocols in other embodiments. The computing device 2300 may include an antenna 2322 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0221] In some embodiments, the communication chip 2312 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 2312 may include multiple communication chips. For instance, a first communication chip 2312 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 2312 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 2312 may be dedicated to wirelesscommunications, and a second communication chip 2312 may be dedicated to wired communications.

[0222] The computing device 2300 may include battery / power circuitry 2314. The battery / power circuitry 2314 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 2300 to an energy source separate from the computing device 2300 (e.g., AC line power).

[0223] The computing device 2300 may include a display device 2306 (or corresponding interface circuitry, as discussed above). The display device 2306 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0224] The computing device 2300 may include an audio output device 2308 (or corresponding interface circuitry, as discussed above). The audio output device 2308 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0225] The computing device 2300 may include an audio input device 2318 (or corresponding interface circuitry, as discussed above). The audio input device 2318 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0226] The computing device 2300 may include a GPS device 2316 (or corresponding interface circuitry, as discussed above). The GPS device 2316 may be in communication with a satellite-based system and may receive a location of the computing device 2300, as known in the art.

[0227] The computing device 2300 may include another output device 2310 (or corresponding interface circuitry, as discussed above). Examples of the other output device 2310 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0228] The computing device 2300 may include another input device 2320 (or corresponding interface circuitry, as discussed above). Examples of the other input device 2320 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, aQuick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0229] The computing device 2300 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 2300 may be any other electronic device that processes data.Select Examples

[0230] The following paragraphs provide various examples of the embodiments disclosed herein.

[0231] Example 1 provides a method of executing a transformer model, the method including generating a padded input sequence by adding one or more padding tokens to an input sequence, the input sequence including one or more input tokens; providing the padded input sequence to a decoder of the transformer model, the decoder computing an attention tensor from the one or more input tokens, a dimension of the attention tensor equal to a dimension of the input sequence; and storing the attention tensor in a cache, a dimension of the cache equal to a dimension of the padded input sequence.

[0232] Example 2 provides the method of example 1, in which the attention tensor includes keys or values, and the method further includes providing the attention tensor to a selfattention layer of the decoder.

[0233] Example 3 provides the method of example 1 or 2, in which the attention tensor includes one or more vectors, and a vector is computed in the decoder from a single input token.

[0234] Example 4 provides the method of any one of examples 1-3, further including generating an attention mask that indicates one or more positions of the one or more padding tokens in the padded input sequence and one or more positions of the one or more input tokens in the padded input sequence, in which the decoder computes the attention tensor based on the attention mask.

[0235] Example 5 provides the method of example 4, in which the attention mask includes a first group of elements and a second group of elements, the second group of elements is after the first group of elements, each element in the first group has a first value and indicates a position of an input token in the padded input sequence, and each element in the second group has a second value and indicates a position of a padding token in the padded input sequence.

[0236] Example 6 provides the method of example 5, in which the attention mask further includes an additional element that is after the second group of elements, and the additional element has the first value.

[0237] Example 7 provides the method of any one of examples 1-6, in which the decoder computes the attention tensor in a first inference phase, the decoder computes a new attention tensor in a second inference phase that is after the first inference phase, and the method further includes storing the new attention tensor in the cache.

[0238] Example 8 provides the method of example 7, further including updating the padded input sequence by replacing a padding token with a predicted token, the predicted token generated by the transformer model in the first inference phase, in which the decoder computes the new attention tensor in the second inference phase based on the predicted token.

[0239] Example 9 provides the method of example 7 or 8, in which a dimension of the new attention tensor is one.

[0240] Example 10 provides the method of any one of examples 7-9, in which storing the new attention tensor includes obtaining a concatenated tensor by concatenating the new attention tensor to a right edge of the attention tensor; and storing the concatenated tensor in the cache.

[0241] Example 11 provides the method of any one of examples 1-10, in which the one or more padding tokens are added to a right edge of the input sequence.

[0242] Example 12 provides the method of any one of examples 1-11, in which storing the attention tensor in the cache includes storing the attention tensor in the cache when the dimension of the cache is equal to the dimension of the input sequence; and after storing the attention in the cache, changing the dimensions of the cache to the dimension of the padded input sequence.

[0243] Example 13 provides the method of any one of examples 1-12, further including providing the attention tensor to an MHA layer of the decoder, in which the MHA layer includes multiple heads, a head includes one or more neural network operations performed using a part of the attention tensor, and a number of heads in the MHA layer is equal to another dimension of the cache.

[0244] Example 14 provides the method of any one of examples 1-13, further including generating a positional encoding vector for the padded input sequence, the positional encoding vector indicating a position of each token in the padded input sequence, in which the decoder computes the attention tensor based on the positional encoding vector.

[0245] Example 15 provides the method of any one of examples 1-14, in which the decoder is selected from decoders having caches with different dimensions.

[0246] Example 16 provides one or more non-transitory computer-readable media storing instructions executable to perform operations for executing a transformer model, the operations including generating a padded input sequence by adding one or more padding tokens to an input sequence, the input sequence including one or more input tokens; providing the padded input sequence to a decoder of the transformer model, the decoder computing an attention tensor from the one or more input tokens, a dimension of the attention tensor equal to a dimension of the input sequence; and storing the attention tensor in a cache, a dimension of the cache equal to a dimension of the padded input sequence.

[0247] Example 17 provides the one or more non-transitory computer-readable media of example 16, in which the attention tensor includes keys or values, and the operations further include providing the attention tensor to a self-attention layer of the decoder.

[0248] Example 18 provides the one or more non-transitory computer-readable media of example 16 or 17, in which the operations further include generating a positional encoding vector for the padded input sequence, the positional encoding vector indicating a position of each token in the padded input sequence, in which the decoder computes the attention tensor based on the positional encoding vector.

[0249] Example 19 provides the one or more non-transitory computer-readable media of any one of examples 16-18, in which the operations further include generating an attention mask that indicates one or more positions of the one or more padding tokens in the padded input sequence and one or more positions of the one or more input tokens in the paddedinput sequence, in which the decoder computes the attention tensor based on the attention mask.

[0250] Example 20 provides the one or more non-transitory computer-readable media of any one of examples 16-19, in which the decoder computes the attention tensor in a first inference phase, the decoder computes a new attention tensor in a second inference phase that is after the first inference phase, and the operations further include storing the new attention tensor in the cache.

[0251] Example 21 provides the one or more non-transitory computer-readable media of example 20, in which the operations further include updating the padded input sequence by replacing a padding token with a predicted token, the predicted token generated by the transformer model in the first inference phase, in which the decoder computes the new attention tensor in the second inference phase based on the predicted token.

[0252] Example 22 provides the one or more non-transitory computer-readable media of any one of examples 16-21, in which storing the attention tensor in the cache includes storing the attention tensor in the cache when the dimension of the cache is equal to the dimension of the input sequence; and after storing the attention in the cache, changing the dimensions of the cache to the dimension of the padded input sequence.

[0253] Example 23 provides the one or more non-transitory computer-readable media of any one of examples 16-22, in which the operations further include providing the attention tensor to a MHA layer of the decoder, in which the MHA layer includes multiple heads, a head includes one or more neural network operations performed using a part of the attention tensor, and a number of heads in the MHA layer is equal to another dimension of the cache.

[0254] Example 24 provides an apparatus, including a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations for executing a tensor multiplication operation in a neural network, the operations including generating a padded input sequence by adding one or more padding tokens to an input sequence, the input sequence including one or more input tokens, providing the padded input sequence to a decoder of a transformer model, the decoder computing an attention tensor from the one or more input tokens, a dimension of the attention tensor equal to a dimension of the input sequence, and storing the attentiontensor in a cache, a dimension of the cache equal to a dimension of the padded input sequence.

[0255] Example 25 provides the apparatus of example 24, in which the decoder computes the attention tensor in a first inference phase and computes a new attention tensor in a second inference phase that is after the first inference phase, and the operations further include storing the new attention tensor in the cache; and updating the padded input sequence by replacing a padding token with a predicted token, the predicted token generated by the transformer model in the first inference phase, in which the decoder computes the new attention tensor in the second inference phase based on the predicted token.

[0256] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims1. A method of executing a transformer model, comprising: generating a padded input sequence by adding one or more padding tokens to an input sequence, the input sequence comprising one or more input tokens; providing the padded input sequence to a decoder of the transformer model, the decoder computing an attention tensorfrom the one or more input tokens, a dimension of the attention tensor equal to a dimension of the input sequence; and storing the attention tensor in a cache, a dimension of the cache equal to a dimension of the padded input sequence.

2. The method of claim 1, wherein the attention tensor comprises keys or values, and the method further comprises providing the attention tensor to a self-attention layer of the decoder.

3. The method of claim 1 or 2, wherein the attention tensor comprises one or more vectors, and a vector is computed in the decoder from a single input token.

4. The method of any one of claims 1-3, further comprising: generating an attention mask that indicates one or more positions of the one or more padding tokens in the padded input sequence and one or more positions of the one or more input tokens in the padded input sequence, wherein the decoder computes the attention tensor based on the attention mask.

5. The method of claim 4, wherein the attention mask comprises a first group of elements and a second group of elements, the second group of elements is after the first group of elements, each element in the first group has a first value and indicates a position of an input token in the padded input sequence, and each element in the second group has a second value and indicates a position of a padding token in the padded input sequence.

6. The method of claim 5, wherein the attention mask further comprises an additional element that is after the second group of elements, and the additional element has the first value.

7. The method of any one of claims 1-6, wherein the decoder computes the attention tensor in a first inference phase, the decoder computes a new attention tensor in a second inference phase that is after the first inference phase, and the method further comprises storing the new attention tensor in the cache.

8. The method of claim 7, further comprising: updating the padded input sequence by replacing a padding token with a predicted token, the predicted token generated by the transformer model in the first inference phase, wherein the decoder computes the new attention tensor in the second inference phase based on the predicted token.

9. The method of claim 7 or 8, wherein a dimension of the new attention tensor is one.

10. The method of any one of claims 7-9, wherein storing the new attention tensor comprises: obtaining a concatenated tensor by concatenating the new attention tensor to a right edge of the attention tensor; and storing the concatenated tensor in the cache.

11. The method of any one of claims 1-10, wherein the one or more padding tokens are added to a right edge of the input sequence.

12. The method of any one of claims 1-11, wherein storing the attention tensor in the cache comprises: storing the attention tensor in the cache when the dimension of the cache is equal to the dimension of the input sequence; and after storing the attention in the cache, changing the dimensions of the cache to the dimension of the padded input sequence.

13. The method of any one of claims 1-12, further comprising: providing the attention tensor to a multi-head attention layer of the decoder,wherein the multi-head attention layer comprises multiple heads, a head comprises one or more neural network operations performed using a part of the attention tensor, and a number of heads in the multi-head attention layer is equal to another dimension of the cache.

14. The method of any one of claims 1-13, further comprising: generating a positional encoding vector for the padded input sequence, the positional encoding vector indicating a position of each token in the padded input sequence, wherein the decoder computes the attention tensor based on the positional encoding vector.

15. The method of any one of claims 1-14, wherein the decoder is selected from decoders having caches with different dimensions.

16. One or more non-transitory computer-readable media storing instructions executable to perform operations for executing a transformer model, the operations comprising: generating a padded input sequence by adding one or more padding tokens to an input sequence, the input sequence comprising one or more input tokens; providing the padded input sequence to a decoder of the transformer model, the decoder computing an attention tensorfrom the one or more input tokens, a dimension of the attention tensor equal to a dimension of the input sequence; and storing the attention tensor in a cache, a dimension of the cache equal to a dimension of the padded input sequence.

17. The one or more non-transitory computer-readable media of claim 16, wherein the attention tensor comprises keys or values, and the operations further comprise providing the attention tensor to a self-attention layer of the decoder.

18. The one or more non-transitory computer-readable media of claim 16 or 17, wherein the operations further comprise: generating a positional encoding vector for the padded input sequence, the positional encoding vector indicating a position of each token in the padded input sequence,wherein the decoder computes the attention tensor based on the positional encoding vector.

19. The one or more non-transitory computer-readable media of any one of claims 16-18, wherein the operations further comprise: generating an attention mask that indicates one or more positions of the one or more padding tokens in the padded input sequence and one or more positions of the one or more input tokens in the padded input sequence, wherein the decoder computes the attention tensor based on the attention mask.

20. The one or more non-transitory computer-readable media of any one of claims 16-19, wherein the decoder computes the attention tensor in a first inference phase, the decoder computes a new attention tensor in a second inference phase that is after the first inference phase, and the operations further comprise storing the new attention tensor in the cache.

21. The one or more non-transitory computer-readable media of claim 20, wherein the operations further comprise: updating the padded input sequence by replacing a padding token with a predicted token, the predicted token generated by the transformer model in the first inference phase, wherein the decoder computes the new attention tensor in the second inference phase based on the predicted token.

22. The one or more non-transitory computer-readable media of any one of claims 16-21, wherein storing the attention tensor in the cache comprises: storing the attention tensor in the cache when the dimension of the cache is equal to the dimension of the input sequence; and after storing the attention in the cache, changing the dimensions of the cache to the dimension of the padded input sequence.

23. The one or more non-transitory computer-readable media of any one of claims 16-22, wherein the operations further comprise: providing the attention tensor to a multi-head attention layer of the decoder,wherein the multi-head attention layer comprises multiple heads, a head comprises one or more neural network operations performed using a part of the attention tensor, and a number of heads in the multi-head attention layer is equal to another dimension of the cache.

24. An apparatus, comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations for executing a tensor multiplication operation in a neural network, the operations comprising: generating a padded input sequence by adding one or more padding tokens to an input sequence, the input sequence comprising one or more input tokens, providing the padded input sequence to a decoder of a transformer model, the decoder computing an attention tensor from the one or more input tokens, a dimension of the attention tensor equal to a dimension of the input sequence, and storing the attention tensor in a cache, a dimension of the cache equal to a dimension of the padded input sequence.

25. The apparatus of claim 24, wherein the decoder computes the attention tensor in a first inference phase and computes a new attention tensor in a second inference phase that is after the first inference phase, and the operations further comprise: storing the new attention tensor in the cache; and updating the padded input sequence by replacing a padding token with a predicted token, the predicted token generated by the transformer model in the first inference phase, wherein the decoder computes the new attention tensor in the second inference phase based on the predicted token.

Images