Attention-based neural network

Abstract

A method of implementing a compressed attention-based neural network and hardware logic for implementing a compressed attention-based neural network are described herein. The compressed attention-based neural network includes a compressed attention layer arranged to implement an attention function. An embedded tensor is received at the compressed attention layer. The compressed attention layer rearranges and segments the elements of the embedded tensor to form one or more embedded sub-matrices. The compressed attention layer applies a set of one or more key weight sub-matrices to the respective one or more embedded sub-matrices, and splices the results of applying the one or more key weight sub-matrices to the respective one or more embedded sub-matrices to determine a key matrix. The compressed attention layer applies a set of one or more query weight sub-matrices to the corresponding one or more embedded sub-matrices, and splices results of applying the one or more query weight sub-matrices to the corresponding one or more embedded sub-matrices to determine a query matrix.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to UK patent applications 2403969.5 and 2403991.9, both filed on 20 March 2024, which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to attention-based neural networks. In particular, methods for compressing attention-based neural networks and processing systems for compressing attention-based neural networks are described herein. Furthermore, methods for implementing compressed attention-based neural networks and hardware logic for implementing compressed attention-based neural networks are described herein. Background Art

[0004] A neural network (NN) is a form of artificial network consisting of multiple interconnected layers that can be used in machine learning applications. Each layer of a neural network can be one of several different types. The type of operation performed on the layer's input data depends on the layer type. An attention layer is one type of layer that can be implemented in a neural network. A neural network that includes one or more attention layers is referred to as an attention-based neural network.

[0005] "Attention" refers to a technique or structural configuration that allows a neural network to focus on a certain portion (or portions) of its input. Attention can be used to characterize the relationship between different parts of different data. Applications of attention include, but are not limited to, natural language processing (NLP) and computer vision. For example, in NLP, attention mechanisms can enable a neural network model to focus on certain words in a sentence. In computer vision, for example, attention can enable a neural network to focus on certain parts of a scene.

[0006] Attention mechanisms can be divided into two categories:

[0007] - "Self-attention" manages and quantifies the interdependencies between the input elements themselves.

[0008] - "Cross-Attention" (also known as general attention) manages and quantifies the mutual dependencies between two sets of input elements.

[0009] These different types of attention are used differently by different neural network architectures. For example, in NLP, self-attention itself can be used to understand the context of a sentence. It is used in this way in Google's bidirectional encoder representations from transformers (BERT) technology.

[0010] In applications such as machine translation, self-attention and cross-attention can be applied together to allow the network to focus on different parts of an input sentence in the input language and build relationships between parts of the input sentence and a target sentence in the target language.

[0011] Transformer networks are currently a major example of attention-based networks. The transformer architecture was introduced in Vaswani et al. (“Attention is all you need”, in Advances in Neural Information Processing Systems 30 (NIPS) 2017, https: / / arxiv.org / abs / 1706.03762). The transformer model architecture is proposed as an alternative to using recursion for sequence modeling. The initial architecture is based on an encoder stack and a decoder stack, each of which includes multiple layers. However, more generally, transformer networks can be built around various configurations of encoder stacks and / or decoder stacks, for example:

[0012] Single encoder stack, multiple decoder stacks — see Vaswani et al.

[0013] - Single encoder stack, single decoder stack - for example, Object Detection Transformer (DETR) network;

[0014] - Single encoder stacks - examples include BERT, visual transformers, and word language models;

[0015] - Multiple decoder stacks - for example, the Generative Pre-trained Transformer (GPT) 1 / 2 / 3 family of models.

[0016] Transformer networks have been shown to provide powerful attention-based architectures with state-of-the-art accuracy across a wide range of modalities and tasks. For 2D images, these include image classification, object detection, action recognition, segmentation, super-resolution, enhancement, and colorization; for video, these include activity recognition and video forecasting (a type of time series forecasting); for 3D representations (such as meshes or point clouds), these include classification and segmentation; for text, these include language modeling and generation, next sentence prediction, classification, and question answering; and for audio, these include speech recognition and speech synthesis. There are also multimodal applications where the input and output come from different modalities. Examples of this include visual question answering, reasoning, and image captioning.

[0017] The attention function can be described as mapping a query and a set of key-value pairs to an output. The output can be calculated as a weighted sum of the values, where the weight assigned to each value is calculated by a compatibility function between the query and the corresponding key. The keys, queries, and values ​​can be represented by a key matrix (K), a query matrix (Q), and a value matrix (V). The attention layer in an attention-based neural network is arranged to implement the attention function based on the key matrix, the query matrix, and the value matrix.

[0018] When storing a neural network, such as an attention-based neural network, data representing all operations to be performed by the layers of the neural network needs to be stored, such as the weights of the matrices to be applied to the layers and the data defining the connections between different layers within the neural network. Typically, a large amount of data is required to represent the neural network. When implementing the neural network on hardware logic, such as a neural network accelerator (NNA) or a graphics processing unit (GPU), the data defining the neural network is typically stored in "off-chip" memory. The hardware logic can implement the layers of the neural network by reading the data defining the layers (e.g., data defining the weights of the matrices to be used in the layers) at runtime. Reading this data from the off-chip memory can require a significant amount of memory bandwidth. It is desirable to reduce the amount of data that needs to be read to implement the neural network. Furthermore, when implementing the neural network, it is desirable to reduce the number of operations, such as multiply-accumulate (MAC) operations, required to reduce latency and / or power consumption of the neural network implementation. Summary of the Invention

[0019] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0020] A method of implementing a compressed attention-based neural network on hardware logic is provided, wherein the compressed attention-based neural network includes a compressed attention layer arranged to implement an attention function, the method comprising, at the compressed attention layer:

[0021] Receives an embedding tensor;

[0022] Rearranging and partitioning elements of the embedding tensor to form one or more embedding sub-matrices;

[0023] applying a set of one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a bond matrix;

[0024] applying a set of one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a query matrix;

[0025] applying a set of one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a value matrix; and

[0026] The attention function is implemented using the determined key matrix, the determined query matrix, and the determined value matrix.

[0027] The rearranging and splitting the elements of the embedding tensor may include reordering the elements of the embedding tensor.

[0028] The rearrangement and partitioning of the elements of the embedding tensor can match the rearrangement and partitioning of the rows and columns of the query weight matrix, the key weight matrix and the value weight matrix, thereby being used to determine the set of the one or more key weight sub-matrices, the set of the one or more query weight sub-matrices and the set of the one or more value weight sub-matrices.

[0029] The embedding tensor may represent: (i) an input sequence, (ii) an output from an encoder layer in the compressed attention-based neural network, or (iii) an output from a decoder layer in the compressed attention-based neural network.

[0030] The method may further comprise:

[0031] Receive an input sequence;

[0032] determining an embedding tensor such that it represents the input sequence; and

[0033] The embedding tensor is provided to the compressed attention layer.

[0034] The rearranging and splitting of the elements of the embedding tensor to form one or more embedding sub-matrices may be performed by: (i) one or more aggregation layers of the compressed attention layer, or (ii) an aggregation layer and a splitting layer of the compressed attention layer.

[0035] The method may further include applying an output matrix to the result of implementing the attention function before providing the output of the compressed attention layer, and wherein the ordering of the rows and columns of the output matrix is ​​complementary to the rearrangement of the elements of the embedding tensor.

[0036] Implementing the attention function may include using a scaled dot product attention computation.

[0037] The attention function Attention(Q,K,V) can be given by the following formula:

[0038]

[0039] Where Q is the query matrix, K T is the transpose of the key matrix, V is the value matrix, and d k are the dimensions of the query matrix and the key matrix.

[0040] The compressed attention layer can be configured to implement multi-head attention.

[0041] The compressed attention layer can be configured to implement multi-head attention by:

[0042] Splitting the key matrix, the query matrix, and the value matrix into a plurality of headers;

[0043] determining a head attention result for each of the heads; and

[0044] The determined head attention results of each of the heads are concatenated to determine an attention result of the compressed attention layer.

[0045] The compressed attention layer may be:

[0046] a self-attention layer within an encoder of the compressed attention-based neural network;

[0047] a self-attention layer within a decoder of the compressed attention-based neural network; or

[0048] An encoder-decoder attention layer within the decoder of the compressed attention-based neural network.

[0049] The encoder and / or decoder may include a feed-forward layer and / or a normalization layer.

[0050] The compressed attention layer may be an encoder-decoder attention layer within a decoder of the compressed attention-based neural network. The key matrix and the value matrix may be determined using a first embedding tensor representing an output from an encoder layer in the compressed attention-based neural network, and the query matrix may be determined using a second embedding tensor representing an output from a previous layer in the decoder.

[0051] The compressed attention-based neural network may include a transformer network.

[0052] The compressed attention-based neural network can be a large language model.

[0053] The compressed attention-based neural network may be implemented to perform one of: natural language processing, language translation, computer vision processing, image processing, text processing, and speech processing.

[0054] Hardware logic is provided, the hardware logic being configured to implement a compressed attention-based neural network, wherein the compressed attention-based neural network comprises a compressed attention layer arranged to implement an attention function, wherein the compressed attention layer is configured to:

[0055] Receives an embedding tensor;

[0056] Rearranging and partitioning elements of the embedding tensor to form one or more embedding sub-matrices;

[0057] applying a set of one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a bond matrix;

[0058] applying a set of one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a query matrix;

[0059] applying a set of one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a value matrix; and

[0060] The attention function is implemented using the determined key matrix, the determined query matrix, and the determined value matrix.

[0061] The compressed attention layer may include one or more clustering layers configured to rearrange and partition the elements of the embedding tensor to form the one or more embedding sub-matrices.

[0062] The compressed attention layer may include:

[0063] a clustering layer configured to rearrange the elements of the embedding tensor; and

[0064] A splitting layer is configured to split the output from the aggregation layer to form the one or more embedding sub-matrices.

[0065] The compressed attention layer may include:

[0066] a bond concatenation layer configured to concatenate results of applying the one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices to determine the bond matrix;

[0067] a query concatenation layer configured to concatenate results of applying the one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices to determine the query matrix; and

[0068] A value concatenation layer is configured to concatenate results of applying the one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices to determine the value matrix.

[0069] The compressed attention layer may include a processing block configured to apply an output matrix to a result of implementing the attention function before providing an output of the compressed attention layer, wherein the ordering of the rows and columns of the output matrix is ​​complementary to the rearrangement of the elements of the embedding tensor.

[0070] The compressed attention layer can be configured to implement multi-head attention.

[0071] The compressed attention layer may be:

[0072] a self-attention layer within an encoder of the compressed attention-based neural network;

[0073] a self-attention layer within a decoder of the compressed attention-based neural network; or

[0074] An encoder-decoder attention layer within the decoder of the compressed attention-based neural network.

[0075] The encoder and / or decoder may include a feed-forward layer and / or a normalization layer.

[0076] The compressed attention-based neural network may include an encoder stack and a decoder stack.

[0077] Hardware logic may include a neural network accelerator or a graphics processing unit.

[0078] A compressed attention-based neural network is provided, the compressed attention-based neural network comprising a compressed attention layer arranged to implement an attention function, wherein the compressed attention layer is configured to:

[0079] Receives an embedding tensor;

[0080] Rearranging and partitioning elements of the embedding tensor to form one or more embedding sub-matrices;

[0081] applying a set of one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a bond matrix;

[0082] applying a set of one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a query matrix;

[0083] applying a set of one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a value matrix; and

[0084] The attention function is implemented using the determined key matrix, the determined query matrix, and the determined value matrix.

[0085] A computer-readable storage medium having encoded thereon a compressed attention-based neural network is provided.

[0086] A computer-implemented method for compressing an attention-based neural network may be provided, the method comprising:

[0087] receiving an attention-based neural network comprising an attention layer, the attention layer arranged to implement an attention function based on a key matrix, a query matrix, and a value matrix, wherein the attention layer is configured to determine the key matrix using a key weight matrix, determine the query matrix using a query weight matrix, and determine the value matrix using a value weight matrix;

[0088] determining a combination matrix having the same size and shape as the key weight matrix, the query weight matrix, and the value weight matrix, wherein for each element of the combination matrix: (i) if the corresponding element of the key weight matrix, the query weight matrix, and the value weight matrix is ​​zero, then the element of the combination matrix is ​​determined to be zero, and (ii) if at least one of the corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix is ​​non-zero, then the element of the combination matrix is ​​determined to be non-zero;

[0089] determining a rearrangement of rows and columns of the combined matrix that aggregates a plurality of non-zero values ​​of the combined matrix into a set of one or more combined sub-matrices having a greater average density of non-zero values ​​than the combined matrix;

[0090] rearranging the rows and columns of the key weight matrix, the query weight matrix, and the value weight matrix to match the determined rearrangement of the rows and the columns of the combination matrix to determine a set of one or more key weight sub-matrices, a set of one or more query weight sub-matrices, and a set of one or more value weight sub-matrices corresponding to the set of one or more combination sub-matrices; and

[0091] The output comprises a compressed attention-based neural network comprising a compressed attention layer, wherein the compressed attention layer is arranged to implement the attention function by performing a compression operation according to: (i) the set of the one or more key weight sub-matrices, (ii) the set of the one or more query weight sub-matrices, and (iii) the set of the one or more value weight sub-matrices.

[0092] A processing system for compressing an attention-based neural network may be provided, the processing system comprising at least one processor configured to:

[0093] receiving an attention-based neural network comprising an attention layer, the attention layer arranged to implement an attention function based on a key matrix, a query matrix, and a value matrix, wherein the attention layer is configured to determine the key matrix using a key weight matrix, determine the query matrix using a query weight matrix, and determine the value matrix using a value weight matrix;

[0094] determining a combination matrix having the same size and shape as the key weight matrix, the query weight matrix, and the value weight matrix, wherein for each element of the combination matrix: (i) if the corresponding element of the key weight matrix, the query weight matrix, and the value weight matrix is ​​zero, then the element of the combination matrix is ​​determined to be zero, and (ii) if at least one of the corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix is ​​non-zero, then the element of the combination matrix is ​​determined to be non-zero;

[0095] determining a rearrangement of rows and columns of the combined matrix that aggregates a plurality of non-zero values ​​of the combined matrix into a set of one or more combined sub-matrices having a greater average density of non-zero values ​​than the combined matrix;

[0096] rearranging the rows and columns of the key weight matrix, the query weight matrix, and the value weight matrix to match the determined rearrangement of the rows and the columns of the combination matrix to determine a set of one or more key weight sub-matrices, a set of one or more query weight sub-matrices, and a set of one or more value weight sub-matrices corresponding to the set of one or more combination sub-matrices; and

[0097] The output comprises a compressed attention-based neural network comprising a compressed attention layer, wherein the compressed attention layer is arranged to implement the attention function by performing a compression operation according to: (i) the set of the one or more key weight sub-matrices, (ii) the set of the one or more query weight sub-matrices, and (iii) the set of the one or more value weight sub-matrices.

[0098] The processing system and / or hardware logic may be embodied in hardware on an integrated circuit. A method of manufacturing the processing system and / or hardware logic at an integrated circuit manufacturing system may be provided. An integrated circuit definition data set may be provided that, when processed in the integrated circuit manufacturing system, configures the system to manufacture the processing system and / or hardware logic. A non-transitory computer-readable storage medium may be provided that has a computer-readable description of the processing system stored thereon that, when processed in the integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the processing system and / or hardware logic.

[0099] An integrated circuit manufacturing system may be provided, comprising: a non-transitory computer-readable storage medium having stored thereon a computer-readable description of a processing system and / or hardware logic; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the processing system and / or hardware logic; and an integrated circuit production system configured to manufacture the processing system and / or hardware logic based on the circuit layout description.

[0100] A computer program code for performing any of the methods described herein may be provided. A non-transitory computer readable storage medium may be provided having computer readable instructions stored thereon, which, when executed in a computer system, cause the computer system to perform any of the methods described herein.

[0101] As will be apparent to the skilled person, the features described above may be combined as appropriate and with any aspect of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0103] Figure 1A shows an implementation of one layer of an encoder stack for a transformer network according to one example;

[0104] Figure 1B shows an implementation of one layer of a decoder stack for a transformer network according to one example;

[0105] Figure 1C An attention layer configured to implement a scaled dot-product attention computation within an attention-based neural network is shown;

[0106] Figure 2A Demonstrate example attention-based neural networks;

[0107] Figure 2B exhibit Figure 2A More details on the attention-based neural network presented in

[15] ;

[0108] Figure 3 An attention layer configured to implement multi-head attention is shown;

[0109] Figure 4 Presenting example systems for implementing neural networks, such as attention-based neural networks;

[0110] Figure 5 Demonstrate a processing system for compressing attention-based neural networks;

[0111] Figure 6 showing an example matrix, the matrix including a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values;

[0112] Figure 7 presenting an example rearranged matrix including a plurality of sub-matrices;

[0113] Figure 8 Showcase example hypergraph models;

[0114] Figure 9 A first example is shown in which a compression operation can be performed using multiple sub-matrices;

[0115] Figure 10 A second example is shown in which a compression operation can be performed using multiple sub-matrices;

[0116] Figure 11A shows a graph with dimension C out ×C in ×H w ×W w The coefficient set C out ×C in flat.

[0117] Figure 11B It shows how a compressed convolution operation can be performed using a subset of the coefficient set of a convolutional layer.

[0118] Figure 12 Flowchart of the method for compressing attention-based neural networks;

[0119] Figure 13A query weight matrix, a key weight matrix, and a value weight matrix are shown, each including a plurality of elements that are zero and a plurality of elements that are non-zero;

[0120] Figure 14A presenting a hypergraph model showing how a value weight matrix can be partitioned;

[0121] Figure 14B Display basis Figure 14A The rearranged value weight matrix of the hypergraph model segmentation shown in ;

[0122] Figure 15A Shown is a Figure 13 A combination matrix of zero and non-zero elements determined by the query weight matrix, key weight matrix, and value weight matrix shown in ;

[0123] Figure 15B Presenting a hypergraph model showing how to partition a combination matrix;

[0124] Figure 15C Display basis Figure 15B The rearranged combination matrix of the hypergraph model partitioning shown in ;

[0125] Figure 15D Display has been based on Figure 15B The rearranged query weight matrix, the rearranged key weight matrix and the rearranged value weight matrix of the hypergraph model shown in are rearranged;

[0126] Figure 16 A flowchart of a method for implementing a compressed attention-based neural network;

[0127] Figure 17 Demonstrating an example of a compressed attention layer for a compressed attention-based neural network;

[0128] Figure 18 Another example of a compressed attention layer of a compressed attention-based neural network is shown;

[0129] Figure 19 shows a computer system in which the processing system is implemented; and

[0130] Figure 20 An integrated circuit fabrication system for producing integrated circuits embodying processing systems and / or hardware logic is shown. DETAILED DESCRIPTION

[0131] The following description is presented by way of example to enable any person skilled in the art to make and use the invention.The present invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0132] Embodiments will now be described by way of example only. In the following description, a transformer network will be used as an example of an attention-based neural network. This is a convenient and useful example, as such networks have been found to produce good performance in many tasks. However, it should be understood that the scope of this disclosure is not limited to transformer networks. This disclosure is equally applicable to other attention-based neural networks.

[0133] To understand the converter network examples in the following description, it is assumed that the reader has been briefed on the above-cited document by Vaswani et al.

[0134] Attention-based neural networks, such as transformer networks, learn how to best route information between segments of an input sequence (called tokens). For example, the input sequence could be a series of words, such as "the fox jumped over the lazy dog," with each word considered a token in the input sequence. The attention-based neural network can learn to associate the word "fox" with the word "jumped," and the word "lazy" with the word "dog." Attention can be applied to any type of data that can be formatted as a sequence. For example, it can also be used with image data, where an image is typically represented as a series of patches that are then used as tokens in the sequence.

[0135] The transformer network may include an encoder stack including one or more encoders. The transformer network may also (or alternatively) include a decoder stack including one or more decoders. Figure 1A An implementation of one layer 100 of an encoder stack in a transformer network is shown. There may be many such layers (or "encoders") in the encoder stack, as indicated by the "Nx" notation in the diagram. The number of encoders (N) in the encoder stack is at least one, but in many examples will be greater than one. The general structure of the encoder will be known from the literature by Vaswani et al. The input to the encoder stack is based on an input sequence of "tokens" or "elements." Before being passed to the first layer of the encoder, the input 101 is "embedded" using a linear projection 102. For example, an embedding algorithm is used to convert each input token (e.g., each input word in a sentence) into a vector (i.e., a numerical structure). At adder 104, a positional encoding 103 may be added to the result of the embedding. The positional encoding 103 injects information about the position of the token in the sequence. The output from adder 104 is an embedding tensor representing the input sequence. The embedding tensor may be a vector.

[0136] As in the work by Vaswani et al., this example uses periodic functions of different frequencies for positional encoding. It should be understood that embedding and positional encoding are only performed at the input of the first layer of the encoder stack. Subsequent layers in the stack ( Figure 1A (not shown) will take its input directly from the output of the previous layer.

[0137] Layer 100 of the encoder stack includes: a multi-headed attention block 110 (which is an example of an attention layer); an addition and normalization block 120; a feedforward block 130; and another addition and normalization block 140. The multi-headed attention block 110 performs self-attention on the input. In this example, this is accomplished by a plurality of scaled dot product attention (SDPA) blocks 112-1 to 112-n, where n is the number of heads. However, it will be appreciated that other forms of attention computation may be used in place of SDPA. The outputs of the multiple heads are concatenated and linearly projected 114. The output of the multi-headed attention block 110 is provided as input to an addition and normalization block 120. This block 120 includes an adder 122 configured to add the output of the multi-headed attention block 110 to the input to the block. Layer normalization 124 is applied to the result of the addition 122.

[0138] The output of the addition and normalization block 120 is provided as input to the feedforward block 130. This block includes one or more feedforward neural network layers. In this example, there are two feedforward layers 132 and 134. "Blocks" (e.g., a multi-head attention block and a feedforward block) can be referred to as "layers" (e.g., a multi-head attention layer and a feedforward layer).

[0139] The output of the feedforward block 130 is provided as input to another addition and normalization block 140. This block has a similar structure to the addition and normalization block 120. An adder 142 is configured to add the output of the feedforward block 132 to the input of the block. Layer normalization 144 is applied to the output of the addition 142. The output of the other addition and normalization block 140 forms the output of this layer of the encoder stack. If the current layer is not the last layer of the encoder stack, then the output of the addition and normalization block 140 is input to the next layer of the encoder stack (i.e., input to the next encoder). If the current layer is the last layer of the encoder stack, then the output of the addition and normalization block 140 forms the output of the encoder stack. Depending on the specific application of the transformer network, this output may be provided to one or more prediction heads, or to a decoder stack. That is, the encoder stack may be used alone or in conjunction with a decoder stack (such as that described below with reference to Figure 1B Decoder stack described above).

[0140] Figure 1BAn implementation of one layer (ie, one 'decoder') 150 of a decoder stack of a transformer network is shown. Depending on the specific application of the transformer network, the decoder stack may be used alone or in conjunction with an encoder stack (eg, as described above with reference to FIG. Figure 1A Like the encoder stack, the decoder stack typically includes N layers, where N is greater than or equal to 1. Figure 1B The layer 150 of the decoder stack shown in is shown as the first layer of the decoder stack. Before the input 151 is passed to the layer 150, the input is embedded at block 152 and a position encoding 153 can be added at block 154. In this respect, blocks 152 and 154 are similar to blocks 102 and 104. However, the specific embedding and position encoding are usually different between the encoder and the decoder. The input is also usually different. For example, in a transformer network trained for language translation, the token can be a word. During the training phase, the encoder stack takes a sequence of words (such as a sentence) in the source language as input. The decoder stack takes a sequence of words (such as a sentence) in the target language as input. For each language, the embedding (which can be learned) and the position encoding may be different. The output from the adder 154 is an embedding tensor representing the input sequence. The embedding tensor can be a vector.

[0141] As with the encoder layer 100, the general structure of the decoder layer 150 will be known from Vaswani et al. The layer 150 of the decoder stack includes: a first multi-head attention block 111 (which is an example of an attention layer); a first addition and normalization block 121; a second multi-head attention block 161 (which is an example of an attention layer); a second addition and normalization block 171; a feedforward block 131; and another addition and normalization block 141. The multi-head attention block 111 has the same structure as the multi-head attention block 110 of the encoder layer. It uses multiple SDPA blocks ( Figure 1B (not shown) performs self-attention on the input. As mentioned above, a "block" can be referred to as a "layer".

[0142] The output of the multi-head attention block 111 is provided as input to a first addition and normalization block 121. This first addition and normalization block is similar in structure and function to the addition and normalization block 120 of the encoder layer. When the decoder stack is used together with the encoder stack, the output of the addition and normalization block 121 is provided as input to a second multi-head attention block 161. This block 161 also receives the output of the encoder stack as input. The second multi-head attention block 161 performs cross attention (rather than self-attention) between the decoder input and the encoder input. The output of the second multi-head attention block 161 is provided as input to a second addition and normalization block 171. Similar to the addition and normalization blocks 120 and 121, this block adds the input and output of the second multi-head attention block 161 together and performs layer normalization on the result of the addition. The output of the second addition and normalization block 171 is provided as input to the feedforward block 131. When the decoder stack is used alone (ie, no encoder stack is present), the feedforward block 131 receives its input directly from the first addition and normalization block 121 .

[0143] The feedforward block 131 is similar in structure and function to the feedforward block 130 of the encoder layer. It includes one or more feedforward neural network layers, for example, two feedforward layers. The output of the feedforward block 131 is provided as input to another addition and normalization block 141, which is similar in structure and function to the addition and normalization block 140 at the end of the encoder layer. The output of the other addition and normalization block 141 forms the output of this layer of the decoder stack. If the current layer is not the last layer of the decoder stack, the output of the addition and normalization block 141 is input to the next layer of the decoder stack (i.e., input to the next decoder). If the current layer is the last layer of the decoder stack, the output of the addition and normalization block 141 forms the output of the decoder stack. This output can be provided to one or more prediction heads.

[0144] In some applications of transformer networks, the decoder stack is iterated, either for a fixed or variable number of iterations. For example, using the machine translation example above, the decoder stack may be executed for several iterations until the decoder outputs an "end" token, which indicates that the target language has reached the end of a sentence and terminates the iteration. In each iteration, one more word of the corresponding sentence in the target language is predicted.

[0145] Figure 1CAn attention layer 180 is shown that is configured to implement a scaled dot product attention (SDPA) calculation within an attention-based neural network. The attention layer 180 is configured to implement a single SDPA calculation, but it should be understood that if the attention layer implements multi-head attention, then a corresponding SDPA calculation may be performed for each head. As described above, the input sequence 181 is embedded at block 182, and a positional encoding 183 may be added at block 184 to determine an embedding tensor representing the input sequence. In this respect, blocks 182 and 184 are similar to blocks 102 and 104 and blocks 152 and 154 described above. As described above, the embedding tensor may be a vector and may be referred to as an "embedded vector" or "embedding." The embedding tensor is passed to the attention layer 180. The attention layer 180 determines a query matrix (Q), a key matrix (K), and a value matrix (V) based on the embedding tensor. In particular, the attention layer adds the query weight matrix (W) to the key matrix (K) at block 186. Q ) is applied to the embedding tensor to determine the query matrix; the attention layer at block 188 converts the key weight matrix (W K ) is applied to the embedding tensor to determine the key matrix; and the attention layer at block 190 converts the value weight matrix (W V ) is applied to the embedding tensor to determine the value matrix. In the case of self-attention layers (e.g., self-attention layers 110 and 111), the query matrix, key matrix, and value matrix (Q, K, and V) are all determined from the same embedding tensor, as Figure 1C In the case of an encoder-decoder attention layer implemented in the decoder (e.g., crisscross attention layer 161), the key matrix and value matrix (K and V) are determined from the embedding tensor output by the encoder, while the query matrix (Q) is determined from the embedding tensor output by the previous layer in the decoder. The key weight matrix (W K ), query weight matrix (W Q ) and value weight matrix (W V )'s value.

[0146] The query matrix (Q) and the key matrix (K) are input to the matrix multiplication block 192. Generally, if there are multiple batches of input data, the matrix multiplication block 192 can be configured to perform batch matrix multiplication. The matrix multiplication can be performed by finding the transpose of the key matrix and subtracting the query matrix (Q) from the transpose of the key matrix (K). T) multiplication. The output of (batch) matrix multiplication 192 provides one input to addition operation 194. The other input to addition 194 is the attention mask defined for the input sequence. For example, if the input sequence is padded, the attention mask may contain zero values ​​in positions corresponding to the original input sequence; and in positions corresponding to the padded input sequence, the attention mask may contain large negative values. The intention is that the output of addition 194 is dominated by these large negative values ​​in positions corresponding to the padded positions. In some examples, padding is not applied to the input sequence, the attention mask is not applied, and the attention layer does not include adder 194.

[0147] The output of the addition 194 is input to the Softmax function 196. This computes the Softmax over the features for which attention is being computed. The Softmax function involves applying an exponential function to the input. As will be appreciated by those skilled in the art, the softmax function takes as input a vector z of K real numbers and normalizes it to a probability distribution consisting of K probabilities proportional to the exponent of the input numbers. For example, the output of the softmax function, σ(z), is i Can be determined as where i=1,…,K and

[0148] The output of the Softmax function 196 forms one input to a (batch) matrix multiplication 198. The other input to the (batch) matrix multiplication 198 is the value matrix (V). Blocks 192, 194, 196, and 198 together implement the attention function Attention(Q, K, V), which is given by:

[0149]

[0150] Where Q is the query matrix, K T is the transpose of the key matrix, V is the value matrix, and d k is the dimension of the query matrix and the key matrix. In particular, the query matrix and the key matrix are n×d K matrix, where n is the number of tokens in the input sequence, and d k is the dimension (i.e., number of features) of each of those tokens.

[0151] As can be understood from the above description, there are three main inputs to the attention mechanism: query (Q), key (K) and value (V). The query and key have dimensions d K , and the value has dimension d V In the examples described in this article, d K =d V. A query is submitted for each token in the input sequence, and these queries are matched to a series of keys that describe the values ​​we want to know about. The similarity of a given query to the keys determines the amount of information to retrieve from each value for that particular query. The output of the first matrix multiplication, where we get the similarity of each query to each key, is called the attention matrix. The attention matrix depicts how much attention each token in the sequence pays to each key. These values ​​are then passed through a softmax function to scale the values ​​to a probability distribution that sums to 1, and also to sharpen the distribution so that higher values ​​are higher and lower values ​​are lower. The output of the softmax function can then be multiplied by the values ​​to determine the result of the attention function. According to Equation 1 and Figure 1C The attention layer shown in Figure 1 determines the value of the attention function in the manner described above as an example of implementing "scaled dot product attention." In practice, the attention function is computed simultaneously for a set of queries that are packed together into a query matrix (Q). The keys and values ​​are also packed together into matrices (K and V).

[0152] Figure 2A An example attention-based neural network 200 is shown. Figure 2B More details of the attention-based neural network 200 are shown. The attention-based neural network 200 is a transformer network and includes an encoder stack 202 and a decoder stack 204. The transformer network 200 includes a first embedding block 206, which is configured to determine an embedding tensor representing the input sequence, as described above. The embedding block 206 may also take into account positional encoding ( Figure 2A and Figure 2B The embedding block 206 is configured to provide an embedding tensor representing the input sequence (which may be referred to as an “input embedding tensor”) to the encoder stack 202. The encoder stack 202 includes m encoders 208, where m ≥ 1. Each of the encoders 208 may correspond to Figure 1A The encoder 100 shown in FIG, and includes a self-attention layer, a feed-forward layer, and two addition and normalization layers. The encoder stack 202 is configured such that the output from each encoder 208 is provided as input to the next encoder 208 in the encoder stack 202 until the final encoder 208 is reached. m . From the final encoder 208 m The output of is provided as input to the decoder stack 204. The transformer network 200 includes a second embedding block 210 configured to determine an embedding tensor representing the target sequence, as described above. The embedding block 210 may also take into account positional encoding ( Figure 2A and Figure 2B). The embedding block 210 is configured to provide an embedding tensor representing the target sequence (which may be referred to as a “target embedding tensor”) to the decoder stack 204. The decoder stack 204 includes n decoders 212, where n ≥ 1. The number of encoders (m) in the encoder stack may or may not be equal to the number of decoders (n) in the decoder stack. Each of the decoders 212 may correspond to Figure 1B The decoder 150 shown in FIG, includes a self-attention layer, a cross-attention layer, a feed-forward layer, and three addition and normalization layers. The decoder stack 204 is configured such that the output from each decoder 212 is provided as input to the next decoder 212 in the decoder stack 204 until the final decoder 212 is reached. n .like Figure 2A As shown in FIG. 1 , the final encoder 208 from the encoder stack 202 m The output of is provided as input to each of the decoders 212 in the decoder stack 204. The transformer 200 includes a linear layer 214 and a softmax layer 216. The linear layer 214 is configured to be coupled to the final decoder 212 in the decoder stack 204. n The softmax layer 216 is configured to apply a softmax function to the output of the linear layer 214 to determine the output of the transformer 200. The output of the transformer 200 may be, for example, one or more translated words representing a translation of the original input word.

[0153] like Figure 2B As shown in FIG, the encoder 2081 includes a self-attention layer 2221 and a feed-forward layer 2241. m Including self-attention layer 222 m and feedforward layer 224 m , and the decoder 2121 includes a self-attention layer 2321, an encoder-decoder attention layer 2331 and a feedforward layer 2341. The self-attention layer 2221 in the encoder 2081 includes: block 226 Q1 , which is configured to apply the query weight matrix to the input embedding tensor to determine the query matrix Q used in the self-attention layer 2221; block 226 K1 , which is configured to apply the key weight matrix to the input embedding tensor to determine the key matrix K for use in the self-attention layer 2221; and block 226 V1 , which is configured to apply the value weight matrix to the input embedding tensor to determine the value matrix V for use in the self-attention layer 2221. The self-attention layer 2221 is configured to implement the attention function using the Q, K, and V matrices as described above.

[0154] Similarly, encoder 208 m Self-attention layer 222 inm Includes: Block 226 Qm , which is configured to apply the query weight matrix to the embedding tensor received from the previous encoder to determine the weight matrix for the self-attention layer 222 m Query matrix Q in block 226 Km , which is configured to apply the key weight matrix to the embedding tensor received from the previous encoder to determine the weight matrix for the self-attention layer 222 m The key matrix K in block 226; and Vm , which is configured to apply a value weight matrix to the embedding tensor received from the previous encoder to determine the weights for the self-attention layer 222 m The value matrix V in the self-attention layer 222 m is configured to implement the attention function using the Q, K, and V matrices as described above.

[0155] Similarly, the self-attention layer 2321 in the decoder 2121 includes: block 236 Q1 , which is configured to apply the query weight matrix to the target embedding tensor to determine the query matrix Q used in the self-attention layer 2321; block 236 K1 , which is configured to apply the key weight matrix to the target embedding tensor to determine the key matrix K for use in the self-attention layer 2321; and block 236 V1 , which is configured to apply the value weight matrix to the target embedding tensor to determine the value matrix V for use in the self-attention layer 2321. The self-attention layer 2321 is configured to implement the attention function using the Q, K, and V matrices as described above.

[0156] The encoder-decoder attention layer determines its key matrix and value matrix in a different way than the self-attention layer. In particular, the encoder-decoder attention layer 2331 in the decoder 2121 includes: block 238 Q1 , which is configured to apply the query weight matrix to the embedding tensor received from the self-attention layer 2321 to determine the query matrix Q used in the encoder-decoder attention layer 2331; block 238 K1 , which is configured to apply the key weight matrix to the final encoder 208 from the encoder stack 202 m The received embedding tensor is used to determine the key matrix K used in the encoder-decoder attention layer 2331; and block 238 V1 , which is configured to apply the value weight matrix to the final encoder 208 from the encoder stack 202 m The received embedding tensor is used to determine the value matrix V used in the encoder-decoder attention layer 2331. The encoder-decoder attention layer 2331 is configured to implement the attention function using the Q, K, and V matrices as described above.

[0157] As described above, the attention layer in an attention-based neural network can be configured to implement multi-head attention. Multi-head attention allows the neural network to control the mixing of information between segments (i.e., "tokens") of the input sequence, which may lead to the generation of richer representations, which in turn allows for improved performance of machine learning tasks. In particular, multi-head attention allows the attention layer to learn multiple ways to "index" the value matrix V. In contrast, in vanilla attention (without multiple heads), the network only learns one way to route information.

[0158] Figure 3 An attention layer configured to implement multi-head attention is shown. In particular, Figure 3 The attention layer shown in FIG has a split block 302, an attention block 304, and a concatenation block 306. The split block 302 can be arranged to split the query matrix, the key matrix, and the value matrix into multiple heads. For example, the split block 302 can linearly project the query, key, and value h times using different learned linear projections. Figure 3 In the example shown in , each of the query matrix (Q), key matrix (K), and value matrix (V) is an n×d matrix, and the split block 302 splits each of the Q, K, and V matrices into h heads, where each head is an n×d / h matrix. As described above, the attention block 304 is configured to implement an attention function for each head to determine a head attention result for each head. The head attention result of each of the h heads is represented as an n×d / h matrix. The splicing block 306 is configured to splice the determined head attention results of the heads to determine the attention result of the attention layer, which is Figure 3 d matrix 308. In this way, the Q, K, and V matrices are simply decomposed by features into different heads, each responsible for learning a different representation. Each head goes through the attention mechanism (the heads are all processed in exactly the same way, with slightly smaller dimensions of the matrices). And at the end, each head is stitched together to form the output n×d matrix 308. It should be noted that the "split" into multiple heads may only be a logical split. That is, the query, key, and value matrices may not be physically split into separate smaller matrices, one matrix for each attention head. Instead, a single data matrix can be used for each of the query, key, and value matrices, with each matrix for each attention head having a logically separate part. Each attention head can simply operate on its 'own' logical part of the matrix.

[0159] A neural network can be defined by a software model. For example, the software model can define a series of layers of the neural network (e.g., the number of layers, the order of the layers, and the connectivity between the layers), and define each layer in the series according to the operations it is configured to perform and the set of coefficients it will use. In general, a neural network can be implemented in hardware, software, or any combination thereof.

[0160] Figure 4 An example system 400 for implementing a neural network (e.g., an attention-based neural network) in hardware is shown. System 400 includes hardware components (e.g., hardware logic, such as a processing element implemented in hardware) and software components (e.g., firmware, and programs and tasks for execution at the hardware logic). System 400 includes hardware logic 402 for implementing a neural network. The hardware logic may, for example, include a neural network accelerator (NNA) or a graphics processing unit (GPU). In other words, system 400 may include a hardware neural network accelerator (NNA). That is, system 400 may include a neural network accelerator (NNA) implemented in hardware. Hardware logic 402 may be configured to perform operations to be performed by a layer of a neural network. Hardware logic 402 includes a plurality of configurable resources that enable implementation of different types of neural network layers, such as an attention layer, a fully connected layer, and a convolutional layer. In other words, implementing a neural network may include configuring the hardware logic included in system 400 to process data input to the system according to the layers of the neural network.

[0161] In more detail, system 400 includes an input 401 for receiving input data. The input data received at input 401 includes input activation data. The input data received at input 401 also includes a set of coefficients for each layer of the neural network. The set of coefficients may also be referred to as weights.

[0162] Input data received at input 401 may be written to memory 404 included in system 400. Memory 404 may be accessible by hardware logic 402. Memory 404 may be system memory accessible to hardware logic 402 via a data bus. Hardware logic 402 may be implemented on a chip (e.g., a semiconductor die and / or an integrated circuit package), and memory 404 may not be physically located on the same chip (e.g., a semiconductor die and / or an integrated circuit package) as hardware logic 402. Therefore, memory 404 may be referred to as "off-chip memory" and / or "external memory." Memory 404 may be coupled to input buffer 406 at hardware logic 402 to provide input activation data to hardware logic 402. Memory 404 may be coupled to coefficient buffer 430 at hardware logic 402 to provide coefficient sets to hardware logic 402.

[0163] Input buffer 406 may be arranged to store input activation data required by hardware logic 402. Coefficient buffer 430 may be arranged to store coefficient sets required by hardware logic 402. Input buffer 406 may contain some or all of the input activation data associated with one or more operations performed at hardware logic 402 in a given cycle, as will be described herein. Coefficient buffer 430 may contain some or all of the coefficient sets associated with one or more operations processed at hardware logic 402 in a given cycle, as will be described herein. Figure 4 The various buffers of the hardware logic 402 shown in FIG. 4 may be implemented in any suitable manner, for example, as any number of data memories local to the hardware logic 402 (e.g., on the same semiconductor die and / or disposed within the same integrated circuit package), or accessible to the hardware logic 402 via a data bus or other interconnect.

[0164] exist Figure 4In the embodiment of the present invention, system 400 also includes memory 428 for storing at least some of the coefficient sets and / or input activation data required for the neural network implemented at hardware logic 402. Hardware logic 402 (e.g., an NNA or GPU) may be implemented on a chip (e.g., a semiconductor die and / or an integrated circuit package), and memory 428 may be physically located on the same chip (e.g., a semiconductor die and / or an integrated circuit package) as hardware logic 402. Therefore, memory 428 may be referred to as "on-chip memory" and / or "local memory." Data stored in on-chip memory 428 can be accessed by hardware logic 402 without consuming memory bandwidth to off-chip memory 404. In other words, on-chip memory 428 may not have sufficient storage space to simultaneously store all input data required to implement the neural network. Therefore, a subset of the input data may be periodically written from off-chip memory 404 to on-chip memory 428. Coefficient buffer 430 may be configured to receive coefficient sets from on-chip memory 428 to reduce bandwidth between off-chip memory 404 and coefficient buffer 430. Input buffer 406 can be configured to receive input activation data from on-chip memory 428 in order to reduce the bandwidth between off-chip memory 404 and input buffer 406. A neural network accelerator (NNA) is a hardware accelerator designed to accelerate the processing of a neural network. As known to those skilled in the art, a hardware accelerator is hardware designed to perform a specific set of one or more functions more efficiently than a general-purpose processing unit, such as a central processing unit (CPU). Therefore, compared to a general-purpose CPU that can be configured to perform any number of functions, an accelerator can only perform a limited set of one or more functions. An NNA includes one or more hardware accelerators designed to accelerate one or more neural network operations. Therefore, a graphics processing unit (GPU) having one or more hardware accelerators designed to accelerate one or more neural network operations can be understood as an NNA.

[0165] exist Figure 4 , hardware logic 402 includes an input buffer 406, a coefficient buffer 430, a plurality of multiplication engines 408, a plurality of accumulators 410, an accumulation buffer 412, and an output buffer 416. Each multiplication engine 408, along with its corresponding accumulator 410 and its share of the resources of the accumulation buffer 412, represents a processing element 414 of the hardware logic 402. Figure 4 Three processing elements 414 are shown in FIG. 4 , but in general any number of processing elements may be present.

[0166] Each processing element 414 may receive a set of input activation values ​​from input buffer 406 and a set of coefficients from coefficient buffer 430. By operating on the set of input activation values ​​and the set of coefficients, the processing element may be used to perform operations of a layer of a neural network. The processing elements 414 of the hardware logic 402 may be independent processing subsystems of the hardware logic 402 that may operate in parallel. Each processing element 414 includes a multiplication engine 408 that is configured to perform multiplications between the set of coefficients and the set of input activation values. In an example, the multiplication engine 408 may be configured to perform a fully connected operation (e.g., when implementing a fully connected layer) or a convolution operation (e.g., when implementing a convolutional layer) between the set of coefficients and the set of input activation values. The multiplication engine 408 may perform these operations by each multiplication engine 408 comprising a plurality of multipliers, each multiplier configured to multiply a coefficient and a corresponding input activation value to produce a multiplied output value. The multipliers may be followed, for example, by a tree of adders arranged to compute the sum of the multiplication outputs in a manner dictated by the operations performed by the layer.In some examples, these multiply-accumulate calculations may be pipelined.

[0167] As described herein, neural networks are often described as including many layers. A large number of multiply-accumulate calculations must typically be performed at hardware logic 402 to perform the operations performed by each layer of the neural network. This is because the input activation data and coefficient sets for each layer are typically very large. Because multiplication engine 408 may need to perform more than one pass to produce a complete operation output (e.g., because multiplication engine 408 may only receive and process a portion of the coefficient set and input activation values), hardware logic 402 may include multiple accumulators 410. Each accumulator 410 receives the output of multiplication engine 408 and adds it to the previous output of multiplication engine 408 associated with the same operation. Depending on the implementation of hardware logic 402, multiplication engine 408 may not process the same operation in consecutive cycles, so an accumulation buffer 412 may be provided to store the partially accumulated output of a given operation. The appropriate partial result may be provided to accumulator 410 by accumulation buffer 412 each cycle.

[0168] The accumulation buffer 412 may be coupled to the output buffer 416 to allow the output buffer 416 to receive output activation data of intermediate layers of the neural network operated at the hardware logic 402, as well as output data of a final layer (e.g., a layer that performs the final operation of the network implemented at the hardware logic 402). The output buffer 416 may be coupled to the on-chip memory 428 and / or the off-chip memory 404, and output data stored in the output buffer 416 (e.g., output activation data input as input activation data to a subsequent layer, or final output data output by the neural network) may be written to the on-chip memory and / or the off-chip memory.

[0169] In general, hardware logic (e.g., neural network accelerator (NNA)) 402 may also include any other suitable processing logic. For example, in some instances, hardware logic 402 may include reduction logic (e.g., for implementing a max pooling or average pooling operation), activation logic (e.g., for applying an activation function such as a sigmoid function or a step function), or any other suitable processing logic. For simplicity, Figure 4 Such a unit is not shown in FIG.

[0170] As described herein, the coefficient sets used by the layers of a typical neural network typically include a large number of coefficients. Hardware logic 402, such as a neural network accelerator, can implement the layers of the neural network by reading the input activation values ​​and coefficient sets of the layers at runtime, such as directly from off-chip memory 404 or via on-chip memory 428, as described herein with reference to Figure 4 As described. A large amount of memory bandwidth may be required to read a large number of coefficient sets from memory. The memory bandwidth required to read in the coefficient sets may be referred to as "weight bandwidth". The memory bandwidth required to read in the activation value sets may be referred to as "activation bandwidth". In addition, the time it takes to read a large coefficient set from memory may also affect the time it takes for the neural network to perform the operations performed by the layer. This may affect the latency of the neural network. Large coefficient sets may also place large computational demands on the processing elements of the neural network accelerator that implements the neural network, such as the processing elements 414 of the hardware logic 402 described herein. That is, layers with large coefficient sets may require those processing elements to perform a large number of multiply and accumulate (MAC) operations.

[0171] Typically, the inventors have observed that the coefficient sets of a layer of a typical neural network have a majority of coefficients equal to zero (e.g., "zero coefficients" or "0"). This is particularly true in trained neural networks, as the training process can often drive most of the coefficients to zero. In the attention layer of an attention-based neural network, these coefficients can include elements of the key weight matrix, elements of the query weight matrix, and elements of the value weight matrix. Performing element-by-element multiplication between an input activation value and a zero coefficient will inevitably produce a zero output value, regardless of the value of the input activation value.

[0172] Therefore, it is undesirable to incur the weight bandwidth, latency, and computational requirements penalties caused by using large coefficient sets for layers of a neural network for most element-wise multiplications performed using coefficients of the coefficient set, which inevitably produce zero output values. It is also undesirable to incur the activation bandwidth "cost" of reading activation values ​​from memory for element-wise multiplications performed only using the activation values ​​and zero coefficients, which inevitably produce zero output values.

[0173] This document describes methods of compressing attention-based neural networks and processing systems for compressing attention-based neural networks to address one or more of the problems described in the preceding paragraphs.

[0174] Figure 5 A processing system 500 is shown for compressing an attention-based neural network according to the principles described herein. The processing system 500 includes at least one processor 504 and a memory 506.

[0175] At least one processor 504 can be implemented in hardware, software, or any combination thereof. The at least one processor 504 can be a microprocessor, a controller, or any other suitable type of processor for processing computer-executable instructions. The at least one processor 504 can be configured to perform the method for compressing an attention-based neural network according to the principles described herein.

[0176] The memory 506 may be accessible by at least one processor 504. The memory 506 may be a system memory accessible to the at least one processor 504 via a data bus. The at least one processor 504 may be implemented on a chip (e.g., a semiconductor die and / or an integrated circuit package), and the memory 506 may not be physically located on the same chip (e.g., a semiconductor die and / or an integrated circuit package) as the at least one processor 504. Therefore, the memory 506 may be referred to as "off-chip memory" and / or "external memory". Alternatively, the at least one processor 504 may be implemented on a chip (e.g., a semiconductor die and / or an integrated circuit package), and the memory 506 may be physically located on the same chip (e.g., a semiconductor die and / or an integrated circuit package) as the at least one processor 504. Therefore, the memory 506 may be referred to as "on-chip memory" and / or "local memory". Still alternatively, Figure 5 The memory 506 shown in may represent any combination of “on-chip” and “off-chip” memory, such as one or more “on-chip” memories and one or more “off-chip” memories arranged in a memory hierarchy.

[0177] Memory 506 may store computer-executable instructions for performing a method for compressing an attention-based neural network according to the principles described herein. Memory 506 may store an attention-based neural network received at processing system 500 for compression (e.g., a software model defining the neural network, as described herein). Memory 506 may store compressed neural network output as a result of at least one processor executing the computer-executable instructions stored by memory 506. Memory 506 may store the compressed attention-based neural network for subsequent implementation (e.g., at a system for implementing a neural network).

[0178] Processing system 500 may be used to configure system 400 for implementing an attention-based neural network. Figure 5 The system 400 shown in FIG. 4 may have the same Figure 4 The system 400 may include hardware logic 402 for implementing an attention-based neural network, wherein the hardware logic 402 may be implemented as a neural network accelerator (NNA) or a graphics processing unit (GPU). The hardware logic 402 of the system 400 may be configured to implement a compressed attention-based neural network according to the principles described herein.

[0179] Reference Figures 6 to 11B Techniques for compressing a layer of a neural network are described. A matrix representing a set of coefficients of a selected layer of the neural network is determined. The matrix includes a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values. The matrix representing the set of coefficients of the selected layer of the received neural network may not have subgraph partitioning.

[0180] Figure 6 An example matrix 600 is shown, which includes a plurality of elements representing non-zero values ​​and a plurality of elements representing zero values. Figure 6 , elements of the matrix 600 representing non-zero values ​​are marked with an "X," while elements of the matrix 600 representing zero values ​​are not marked. For example, elements located in row 1, column 7 and row 1, column 10 of the matrix 600 represent non-zero values, while other elements located in row 1 of the matrix 600 represent zero values. The example matrix 600 does not have subgraph partitioning.

[0181] In a first example, the selected layer of the neural network is a fully connected layer arranged to perform a fully connected operation, or any other type of layer arranged to perform a matrix multiplication. In a first example, the determined matrix 600 may include a set of coefficients for the layer. The plurality of elements representing non-zero values ​​may be a plurality of non-zero coefficients. A non-zero coefficient is any coefficient having a positive or negative value that is not equal to zero. The plurality of elements representing zero values ​​may be a plurality of zero coefficients. A zero coefficient is a coefficient having a value equal to zero. Figure 6 , in a first example, the matrix 600 includes a plurality of non-zero coefficients marked with "X" and a plurality of zero coefficients that are not marked.

[0182] In a first example, the selected layer of the received neural network can be arranged to perform a fully connected operation by performing a matrix multiplication using a matrix 600 comprising a set of coefficients for the layer and an input matrix comprising a set of input activation values ​​for the layer. For example, as described herein, in a fully connected layer, a matrix multiplication WX=Y can be performed, where: W is a matrix comprising a set of coefficients and having a dimension C. out ×C in (Right now, Figure 614×14) coefficient matrix (e.g., matrix 600); X is an input matrix including the input activation value set and having dimension M×N, where C in = M; and Y is a set of output values ​​and has dimension C out ×N output matrix. That is, the number of rows in the coefficient matrix can represent the number of output channels of the fully connected layer (“C out ”), and the number of columns of the coefficient matrix can represent the number of input channels of the fully connected layer (“C in ”). Alternatively, in a fully connected layer, a matrix multiplication XW=Y can be performed, where: X is an input matrix comprising a set of input activation values ​​and having dimensions M×N; W is a matrix comprising a set of coefficients and having dimensions C in ×C out (Right now, Figure 6 14×14) coefficient matrix (e.g., matrix 600), where C in = N; and Y is a set of output values ​​and has dimensions M×C out That is, the number of rows of the coefficient matrix can represent the number of input channels of the fully connected layer (“C in ”), and the number of columns of the coefficient matrix can represent the number of output channels of the fully connected layer (“C out ”).

[0183] In a second example, the selected layer of the received neural network is a convolutional layer. The convolutional layer includes a set of coefficients arranged in one or more filters, each of the one or more filters being arranged in one or more input channels, each input channel of each filter including a respective subset of the set of coefficients of the convolutional layer. In the second example, the selected convolutional layer of the received neural network can be arranged to perform a convolution operation by convolving a set of input activation values ​​of the convolutional layer with a set of coefficients of the convolutional layer. That is, a convolutional layer having spatial dimensions H a ×W a The Nth input channel of the input activation data can be connected with the spatial dimension H w ×W w Convolve the Nth input channel of each filter with the set of coefficients.

[0184] In a second example, determining a matrix includes, for each input channel of each filter, determining whether the input channel of the filter includes a non-zero coefficient. In response to determining that the input channel of the filter includes at least one non-zero coefficient, the input channel of the filter may be represented by elements representing non-zero values ​​in the matrix. In response to determining that the input channel of the filter includes only zero coefficients (i.e., excluding any non-zero coefficients within a subset of the coefficient set of the convolutional layer), the input channel of the filter may be represented by elements representing zero values ​​in the matrix. For completeness, binary "1" may be used to represent non-zero values ​​in the matrix, while binary "0" may be used to represent zero values ​​in the matrix, but this is not required. For example, binary "0" may be used to represent non-zero values ​​in the matrix, while binary "1" may be used to represent zero values ​​in the matrix. It should also be understood that any other arbitrarily selected value may be used to represent non-zero or zero values ​​in the matrix. As an illustrative example, one or more bits indicating even numbers may be used to represent non-zero values ​​in the matrix, while one or more bits indicating odd numbers may be used to represent zero values ​​in the matrix.

[0185] refer to Figure 6 In the second example, the matrix 600 includes a plurality of elements marked with an "X" representing non-zero values ​​and a plurality of unmarked elements representing zero values. In the second example, the coefficients have been determined for a convolutional layer having a set of coefficients arranged in 14 filters (e.g., output channels) (e.g., represented by the 14 rows of the matrix 600). Figure 6 , each of the 14 filters is arranged in 14 input channels (e.g., represented by the 14 columns of the matrix 600). For example, row 1 of the matrix 600 can represent the first filter of the convolutional layer, and each column in row 1 represents one of the 14 input channels of the first filter. Thus, the elements located in row 1, column 7 and row 1, column 10 of the matrix 600 represent input channels of the first filter that include at least one non-zero coefficient, while the other elements located in row 1 of the matrix 600 represent input channels of the first filter that include only zero coefficients.

[0186] In other words, in the second example, the matrix 600 may represent a matrix having dimension C. out ×C in ×H w ×W w The coefficient set C of the convolutional layer out ×C in plane. This is Figure 11A In order to have a dimension C out ×C in ×H w ×W w The four-dimensional coefficient set of the convolutional layer can be visualized, for example, by replacing the convolutional layer with dimension H w ×Ww The coefficients of each two-dimensional input channel of each filter are represented as corresponding one-dimensional coefficient sequences to make H w ×W w The dimension is "flattened". These one-dimensional coefficient sequences are Figure 11A , which spans a three-dimensional cuboid 1100 representing a set of coefficients of a convolutional layer and is labeled H w W w One dimension. C out ×C in Plane in Figure 11A 1100. Thus, each one-dimensional coefficient sequence H w W w can be called in C out ×C in In the plane there is (C in , C out ) coordinates. As described herein, matrix 600 is Figure 11A The C of the three-dimensional cuboid 1100 shown in out ×C in As described herein, in the second example, each element of matrix 600 represents an input channel of a filter of a convolutional layer. Figure 11A The elements marked with "X" in the matrix 600 representing non-zero values ​​represent the out ×C in The plane has the (C in , C out ) coordinates of the one-dimensional coefficient sequence H w W w At least one coefficient in is non-zero. Figure 11A The elements in the matrix 600 marked as zero are represented in C out ×C in The plane has the (C in , C out ) coordinates of the one-dimensional coefficient sequence H w W w There are no non-zero coefficients in .

[0187] In a second example, this document describes that a matrix may be determined such that each row of the matrix represents a corresponding output channel (e.g., a filter) of a convolutional layer, and each column of the matrix represents a corresponding input channel of the convolutional layer. It should be understood that, alternatively, in the second example, a matrix may be determined such that each row of the matrix represents a corresponding input channel of the convolutional layer, and each column of the matrix represents a corresponding output channel (e.g., a filter) of the convolutional layer. After defining the matrix to be filled in this manner, the elements of the matrix may be filled accordingly by evaluating whether the input channel of the filter represented by each element includes a non-zero coefficient.

[0188] The rows and / or columns of matrix 600 may be rearranged (e.g., reordered) so that multiple elements of the matrix representing non-zero values ​​are clustered into one or more sub-matrices. The one or more sub-matrices have a ratio of the number of elements representing non-zero values ​​to the total number of elements of the one or more sub-matrices that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements of the matrix. In other words, the "non-zero density" of the one or more sub-matrices as a whole is greater than the "non-zero density" of the matrix.

[0189] In some examples, each of the one or more sub-matrices can have a ratio of the number of elements representing non-zero values ​​to the total number of elements in the sub-matrix that is greater than the ratio of the number of elements representing non-zero values ​​to the total number of elements in the matrix. In other words, in these examples, the "non-zero density" of each of the one or more sub-matrices is greater than the "non-zero density" of the matrix, but this need not be the case.

[0190] In a first example, the one or more sub-matrices include a subset of the set of coefficients of the layer. In a second example, the one or more sub-matrices include elements representing a subset of the input channels of the filters of the set of coefficients of the convolutional layer.

[0191] Figure 7 An example rearranged matrix 710 is shown including a plurality of sub-matrices 702-1, 702-2, 702-3, 703-4. Figure 6 The rows and columns of the matrix 600 shown in FIG. Figure 7 The rearranged matrix 710 shown in FIG. That is, Figure 7 Shown in different order Figure 6 14 rows and 14 columns of the matrix 600 shown in . For example, Figure 6 Column 2 of the matrix 600 shown in FIG. 6 , having elements in rows 2, 7, and 13 representing non-zero values, has been shifted "left" and is now Figure 7 The first column of the rearranged matrix 710 shown in FIG. In another example, Figure 6Row 7 of the matrix 600 shown in FIG. 6 having elements in columns 2, 4, and 7 representing non-zero values ​​has been shifted "up" and is now Figure 7 The first row of the rearranged matrix 710 is shown in . By rearranging the rows and columns as described herein, multiple elements of the matrix 600 representing non-zero values ​​are clustered into sub-matrices 702-1, 702-2, 702-3, 703-4.

[0192] Matrix 600 includes 45 elements representing non-zero values ​​and a total of 196 (i.e., 14×14) elements. Therefore, the "non-zero density" of matrix 600 is 0.23 (i.e., 45 / 196). Sub-matrices 702-1, 702-2, 702-3, and 703-4 also include 45 elements representing non-zero values, but include a total of 103 (i.e., (3×4)+(4×5)+(3×5)+(4×14)) elements. Therefore, the "non-zero density" of multiple sub-matrices 702-1, 702-2, 702-3, and 703-4 is 0.44 (i.e., 45 / 103). Therefore, the "non-zero density" of multiple sub-matrices 702-1, 702-2, 702-3, and 703-4 as a whole is greater than the "non-zero density" of matrix 600.

[0193] Figure 7 The rearranged matrix 710 shown in FIG is in a "single-border block-diagonal matrix form". This means that the rearranged matrix includes one or more block arrays arranged on a diagonal line and a border array arranged along one side of the rearranged matrix. The one or more block arrays and the border array are non-overlapping. Figure 7 In FIG, the block arrays are labeled 702-1, 702-2, and 702-3 and are arranged on a diagonal line from upper left to lower right. Figure 7, the border array is labeled 702-4 and is arranged along the bottom of the rearranged matrix 710. In other examples, one or more block arrays may be arranged on a different diagonal (e.g., from top right to bottom left), or may not be arranged on a diagonal at all. That is, for example, the columns of the rearranged matrix 710 may be arranged differently so that the "horizontal" positions of two or more block arrays are swapped. For example, the columns of the rearranged matrix 710 may be arranged differently so that the "horizontal" positions of block arrays 702-1 and 702-2 are swapped, such that block array 702-2 is the "leftmost" block array. For example, more specifically, the columns of the rearranged matrix 710 may alternatively be ordered: [9, 13, 3, 1, 11, 2, 7, 10, 4, 6, 8, 14, 12, 5]. In other examples, the border arrays may be arranged along different sides of the rearranged matrix (e.g., along the top, left, or right side of the matrix), or there may be more than one border matrix, each of which is arranged along a different side of the rearranged matrix. The border arrays may be referred to as horizontal arrays or vertical arrays. The horizontal array may be arranged horizontally on the rearranged matrix. The horizontal array may span the entire width of the rearranged matrix. The vertical array may be arranged vertically on the rearranged matrix. The vertical array may span the entire height of the rearranged matrix. In these other examples, the rearranged matrix may be referred to as a "border block matrix form." A matrix in the form of a single border block-diagonal matrix may also be referred to as being in border block matrix form. As will be understood by those skilled in the art, the border array need not be arranged along one side of the rearranged matrix. That is, for example, the border array may be arranged between two block arrays in the block array. For example, the rows of the rearranged matrix 710 may be arranged differently so that the "vertical" positions of one or more block arrays in the border array and the block array are swapped. For example, the rows of the rearranged matrix 710 can be arranged differently so that the "vertical" positions of the boundary array 702-4 and the block array 702-3 are swapped. For example, more specifically, the rows of the rearranged matrix 710 can be ordered alternatively: [7, 1, 2, 6, 8, 3, 14, 13, 12, 4, 10, 9, 11, 5]. In summary, the rows and / or columns of the matrix can be rearranged to form a rearranged matrix that includes: one or more block arrays arranged along the diagonals of the rearranged matrix, and / or one or more block arrays not arranged along the diagonals of the rearranged matrix; and one or more horizontal arrays arranged horizontally on the rearranged matrix, and / or one or more vertical arrays arranged vertically on the rearranged matrix. The multiple elements of the matrix that represent non-zero values ​​can be aggregated into one or more block arrays and one or more horizontal and / or vertical arrays of the rearranged matrix.In other words, the rows and / or columns of the matrix can be rearranged to form: a rearranged matrix in the form of a border block matrix; or a rearranged matrix as a block matrix, wherein the block matrix includes an array that can be arranged into (e.g., capable of being rearranged into) the form of a border block matrix.

[0194] As described herein, the matrix 600 does not have a subgraph partitioning. As will be appreciated by those skilled in the art, this means that it is not possible to rearrange the matrix 600 into a block-diagonal matrix form consisting of (e.g., exclusively including) a plurality of block arrays arranged on a diagonal line in which all non-zero values ​​of the matrix 600 are clustered.

[0195] The rows and / or columns of the matrix can be rearranged according to a hypergraph model. The hypergraph model can be used to convert the matrix into a "single-bounded block-diagonal matrix form." The hypergraph model can be formed based on the corresponding row and column positions of each of the multiple elements representing non-zero values ​​within the matrix.

[0196] In one example, the hypergraph model is a "rownet" hypergraph model. Figure 6 The matrix 600 shown in FIG can be rearranged using a row-net hypergraph model to form Figure 7 The rearranged matrix 710 shown in . Forming a row-net hypergraph model includes forming a vertex representing each column of the matrix, the vertex including an element representing a non-zero value; and forming a net representing each row of the matrix, the net including an element representing a non-zero value. For each of the multiple elements representing non-zero values ​​within the matrix, the vertex representing the column of the matrix including the element representing the non-zero value is connected to the net representing the row of the matrix including the element representing the non-zero value. Figure 8 To understand this example, the diagram shows Figure 6 The matrix 600 shown in FIG. 8 forms an example rownet hypergraph model 812 .

[0197] exist Figure 8 , the vertices representing the columns of the matrix 600 are shown as v i , where i is the index of the column. For example, column 10 of matrix 600 consists of vertex v 10 The net representing the rows of matrix 600 is shown as n j , where j is the index of the row. For example, row 1 of matrix 600 is represented by vertex n1. As described herein, elements located in row 1, column 10 of matrix 600 represent non-zero values. Thus, as an example, vertex v represents column 10 of matrix 600 that includes elements representing non-zero values. 10 is connected to net n1 representing row 1 of matrix 600 including elements representing non-zero values. Figure 8 The network hypergraph model shown in Figure 6The same vertex-net connection process is repeated for each of the non-zero valued elements in the matrix 600 shown in FIG. 6 , as shown by comparing Figure 6 and Figure 8 Will understand.

[0198] In other words, we can construct a row-net hypergraph model for the coefficient matrix W as follows. Let H = (V, N) be a hypergraph H with vertex set V and net set N. Each column W(:, i) consists of vertex v i ∈V, and each row W(j,:) is represented by the network n j ∈N indicates that if there is an element in the coefficient matrix W that represents a non-zero value W(i,j), then the network n j Connect vertices v i . By the network j The connected vertices can be represented as

[0199] It should be understood that when forming a hypergraph model (e.g., a row-net hypergraph model), vertices may not be formed for columns of a matrix that do not include any elements representing non-zero values ​​(not shown in the figure), and a net may not be formed for rows of a matrix that do not include any elements representing non-zero values ​​(not shown in the figure).

[0200] In another example, the hypergraph model is a "columnnet" hypergraph model. Forming the columnnet hypergraph model includes forming a net representing each column of a matrix, the net including an element representing a non-zero value; and forming a vertex representing each row of the matrix, the vertex including an element representing a non-zero value. For each of a plurality of elements within the matrix representing non-zero values, the net representing the column of the matrix including the element representing the non-zero value is connected to the vertex representing the row of the matrix including the element representing the non-zero value.

[0201] In other words, we can construct a column-net hypergraph model for the coefficient matrix W as follows. Let H = (V, N) be a hypergraph H with vertex set V and net set N. Each row W(j,:) consists of vertex v j ∈V, and each column W(:,i) is represented by the network n i ∈N indicates that if there is an element in the coefficient matrix W that represents a non-zero value W(i,j), then the network n i Connect vertices v j . By the network i The connected vertices can be represented as

[0202] It should be understood that when forming a hypergraph model (e.g., a column-net hypergraph model), a net may not be formed for the columns of a matrix that does not include any elements representing non-zero values ​​(not shown in the figure), and / or vertices may not be formed for the rows of a matrix that does not include any elements representing non-zero values ​​(not shown in the figure).

[0203] Once formed, the hypergraph model can be partitioned. Figure 8 A row-net hypergraph model is shown that has been partitioned into three parts 812-1, 812-2, and 812-3. It should be understood that the hypergraph model can be partitioned into any suitable number of parts. The rows and / or columns of the matrix can be rearranged according to the partitioned hypergraph model so that multiple elements of the matrix representing non-zero values ​​are grouped into one or more sub-matrices.

[0204] For example, in Figure 8 , section 812-1 corresponds to Figure 7 The elements representing non-zero values ​​located in the rows of the matrix represented by the nets are grouped into the block array 702-1, and the nets are connected only to the vertices of the columns of the matrix within the representation portion 812-1. For example, in Figure 8 In the example, nets n1, n2, and n7 are connected only to vertices within portion 812-1, namely, to vertices v7 and v 10 ; vertices v2, v4, and v 10 ; and vertices v2, v4, and v7. Thus, the rows and columns of matrix 600 may be rearranged so that the elements representing non-zero values ​​located in rows 1, 2, and 7 are clustered into block array 702-1, as shown in FIG. Figure 7 In a similar manner: Section 812-2 corresponds to Figure 7 and portion 812-3 corresponds to the block array 702-2 shown in FIG. Figure 7 The block array 702-3 shown in FIG.

[0205] Elements representing non-zero values ​​located in rows of the matrix represented by nets connected to vertices representing columns of the matrix in more than one section are gathered into a bounding array 702-4. Figure 8 In the example, net n4 is connected to vertices in parts 812-1 and 812-2, namely vertices v1, v7, v9, and v 10 and v 11 .like Figure 8 As shown in the 10 、n 12 and n 13 Also connected to vertices in more than one part. Therefore, the rows and columns of the matrix 600 are rearranged so that the elements representing non-zero values ​​located in rows 4, 10, 12, and 13 are gathered into a boundary array 702-4, as shown in FIG. Figure 7 The hypergraph model can be partitioned according to a load balancing constraint that aims to minimize the number of nets connected to vertices in more than one part, for example, aiming to minimize the number of rows of the boundary array.

[0206] As will be appreciated by those skilled in the art, a hypergraph model formed for a matrix with subgraph partitioning (not shown in the figure) will not include any nets (or vertices) that are connected to vertices (or nets) in more than one part. That is, there will be no nets (or vertices) that are "connected" to any part. This will enable the matrix to be converted into a block-diagonal matrix form consisting of (e.g., exclusively including) a plurality of block arrays arranged on the diagonal where all non-zero values ​​of the matrix are clustered.

[0207] It should be understood that any row or column of the matrix that does not contain any elements representing non-zero values ​​(e.g., any row or column that does not form a net or vertex, as the case may be, when forming a hypergraph model) can be rearranged (e.g., arbitrarily) to any row or column position within the rearranged matrix. Alternatively, another "empty" block array can be formed (not shown in the figure), and the elements of the rows and columns that do not contain any elements representing non-zero values ​​can be gathered into the other "empty" block array. The "empty" block array can be used in a manner equivalent to the "non-empty" block array during future calculations performed in the compression layer (as will be further described herein), or may not be used in future calculations performed in the compression layer (e.g., discarded from future calculations).

[0208] In other words, a K-way vertex partition of a hypergraph model H can be defined as a set of mutually incoherent and exhaustive subsets V of vertices m ∈V composed of Π(H)={V1,V2,…V K}, where for all V m ∈Π(H), if m≠n and but Make According to the partition Π(H), by the network n j The connected partial set can be defined as the connectivity set Λ(n j ), and by the network j The number of connected parts can be defined as the connectivity λ(n j )=|Λ(n j )|. If the network j Connected to multiple parts (ie, λ(n j )>1), the net is called cut (external), otherwise (i.e., λ(n j )=1) is called uncut (inside). j Connect only part of V m The vertex in (i.e. Λ(n j )=V m and pins(n j )=V m ), then the network is part of V m Intranet. Part V mThe internal network set is denoted as N m , and the external network set of partition Π(H) is denoted as N S Therefore, a K-way vertex partition Π(H) can also induce a (K+1)-way partition {N1,N2,…N K ; N S Here, N S is a net separator, removing the net separator yields K disconnected vertex parts V1, V2, ... V K and K disconnected network parts N1, N2, ... N K .

[0209] The hypergraph model H can be partitioned to achieve load balancing Minimize the number of cut nets where part V m The weight is The average part weight is And the maximum allowed imbalance ratio is ∈. Each vertex v i The weight can be determined as w(v i )=1. The purpose of the load balancing constraint may be to produce equally sized block arrays that can each fit in the coefficient buffer of a neural network accelerator (NNA), e.g. Figure 4 In the coefficient buffer 430 of the neural network accelerator (NNA) 402 shown in .

[0210] K-way partition Π(H)={V1,V2,…V K}={N1,N2,…N k ; N S} can cause a partial sorting of the rows and columns of the coefficient matrix W. In this sorting, in the example where the hypergraph model is formed as a row-net hypergraph model, m+1 The columns associated with the vertices in V m (where m = 1, 2, ... K-1) are then sorted. Similarly, with part V m+1 Intranet N m+1 The rows that represent the m (where m = 1, 2, ... K-1) internal network N m After the associated line. With the external network N S The associated rows are finally sorted into a bounding array. In other words, the vertex v i ∈V m This means arranging the column W(:,i) into the mth column slice, with the internal network n j ∈N m This means arranging the row W(j,:) into the mth row slice, and the external network n j ∈NS This means aligning the row W(j,:) to the boundary matrix.

[0211] In the examples described herein, where the hypergraph model is formed as a row-net hypergraph model, partitioning the hypergraph model as described herein will cause the matrix to be rearranged into Figure 7 , where the border array is arranged along the bottom of the rearranged matrix 710. In an alternative example described herein, where the hypergraph model is formed as a column-net hypergraph model, partitioning the hypergraph model as described herein will rearrange the matrix into a single bordered block-diagonal form, where the border array is arranged along the right side of the rearranged matrix (e.g., see Figure 10 ). The example rearranged matrix 1010 is shown in FIG.

[0212] It should be understood that the use of a hypergraph model in step S506 is not essential. There are other methods for rearranging the rows and / or columns of a matrix so as to cluster multiple elements of the matrix representing non-zero values ​​into one or more sub-matrices. For example, a hypergraph clustering algorithm or a graph partitioning algorithm can be used instead for this purpose.

[0213] Figure 9 It shows how compression operations can be performed using multiple sub-matrices. Figure 9 A "compressed" version of the matrix multiplication WX=Y is shown. Figure 9 A rearranged coefficient matrix 910 is shown that includes a plurality of sub-matrices labeled Bl 902-1, B2 902-2, B3 902-3, Rl 902-4, R2 902-5, and R3 902-6. Figure 9 The rearranged matrix 910 shown in FIG has Figure 7 The rearranged matrix 710 shown in FIG has the same characteristics as described herein. Figure 9 As shown, each block array of the boundary block matrix 910 can be labeled as a submatrix B, and the boundary array of the boundary block matrix 910 can be divided into multiple submatrices R. The boundary array of the boundary block matrix 910 can be divided according to the number of columns included in each block array. That is, submatrix R1 can have the same number of columns as submatrix B1; submatrix B2 can have the same number of columns as submatrix B2; submatrix R3 can have the same number of columns as submatrix B3. In the first example, the multiple submatrices B1, B2, B3, R1, R2, R3 of the compression layer include a subset of the coefficient set included in the coefficient matrix W of the selected layer (e.g., matrix 600).

[0214] Figure 9Also shown are a plurality of input sub-matrices labeled X1 904-1, X2 904-2, and X3 904-3. In a first example, the plurality of input sub-matrices X1, X2, and X3 of the compressed layer each include a corresponding subset of the set of input activation values ​​included in the input matrix X of the selected layer. In a first example, the rows of the input matrix X can be rearranged (e.g., reordered or permuted) so as to correspond to the rearranged columns of the rearranged coefficient matrix. Each of the plurality of input sub-matrices X1, X2, and X3 can include only the input activation values ​​of the input matrix X, which will be multiplied by the coefficients included in the plurality of sub-matrices B1, B2, B3, R1, R2, and R3 of the compressed layer, as further explained below. The number of rows included in the plurality of input sub-matrices X1, X2, and X3 can be equal to the number of columns included in the plurality of sub-matrices B1 and R1, B2 and R2, and B3 and R3, respectively. That is, the input submatrix X1 may have a number of rows equal to the number of columns of the submatrices B1 and R1; the input submatrix X2 may have a number of rows equal to the number of columns of the submatrices B2 and R2; and the input submatrix X3 may have a number of rows equal to the number of columns of the submatrices B3 and R3. The rows of the input matrix X may be rearranged (e.g., reordered or permuted) to correspond to the rearranged columns of the rearranged coefficient matrix such that: the input submatrix X1 includes input activation values ​​from the rows of X that correspond to (e.g., are multiplied by) the coefficient values ​​of the columns of the submatrices B1 and R1; the input submatrix X2 includes input activation values ​​from the rows of X that correspond to (e.g., are multiplied by) the coefficient values ​​of the columns of the submatrices B2 and R2; and the input submatrix X3 includes input activation values ​​from the rows of X that correspond to (e.g., are multiplied by) the coefficient values ​​of the columns of the submatrices B3 and R3. In examples where each column of the rearranged coefficient matrix contains at least one element representing a non-zero coefficient, the rows of the input matrix X can be rearranged (e.g., reordered or permuted) to form input sub-matrices X1, X2, X3 without discarding any input activation values ​​from the input matrix X. That is, the sum of the number of input activation values ​​in all the input sub-matrices X1, X2, X3 can be equal to the number of input activation values ​​in the input matrix X.

[0215] Figure 9Also shown are multiple output sub-matrices labeled Y1 906-1, Y2 906-2, Y3 906-3, and Y4 906-4. In a first example, each of the multiple output sub-matrices Y1, Y2, Y3, Y4 of the compressed layer includes a corresponding subset of the set of output values ​​included in the output matrix Y of the selected layer. In the first example, when a "compressed" version of the matrix multiplication WX=Y is to be performed, the multiple input sub-matrices X1, X2, X3 and the multiple sub-matrices B1, B2, B3, R1, R2, R3 can be used to form (e.g., a compressed fully connected operation can be performed) the multiple output sub-matrices Y1, Y2, Y3, Y4 using the following equations (1) and (2):

[0216] Y i =B i X i , where i=1,2…K (1)

[0217]

[0218] It should be understood that equations (1) and (2) are general formulas that can be used to perform compressed fully connected operations (e.g., a "compressed" version of the matrix multiplication WX=Y) using submatrices of any K-way partitioned single-boundary block-diagonal matrix rearranged according to the row-net hypergraph model. In this particular example, where K=3: the output submatrix Y1 can be formed by performing the matrix multiplication Y1=B1X1; the output submatrix Y2 can be formed by performing the matrix multiplication Y2=B2X2; the output submatrix Y3 can be formed by performing the matrix multiplication Y3=B3X3; and the output submatrix Y4 can be formed by performing the matrix multiplication Y4=R1X1+R2X2+R3X3.

[0219] As described in this paper, the row network hypergraph model can be used to form Figure 9 For completeness, Figure 10 Shown is a rearranged matrix 1010 that may be formed according to a column-net hypergraph model, as described herein.

[0220] Figure 10 Another example of how a compression operation can be performed using multiple sub-matrices is shown. Figure 10 Another "compressed" version of the matrix multiplication WX=Y is shown. Figure 10 A rearranged coefficient matrix 1010 is shown comprising a plurality of sub-matrices labeled B1 1002-1, B2 1002-2, B3 1002-3, C1 1002-4, C2 1002-5, and C3 1002-6. Figure 10As shown, each block array of the boundary block matrix 1010 can be labeled as a submatrix B, and the boundary array of the boundary block matrix 1010 can be divided into multiple submatrices C. The boundary array of the boundary block matrix 1010 can be divided according to the number of rows included in each block array. That is, submatrix C1 can have the same number of rows as submatrix B1; submatrix C2 can have the same number of rows as submatrix B2; submatrix C3 can have the same number of rows as submatrix B3. In the first example, the multiple submatrices B1, B2, B3, C1, C2, C3 of the compression layer include a subset of the coefficient set included in the coefficient matrix W of the selected layer (e.g., matrix 600).

[0221] Figure 10Also shown are multiple input sub-matrices labeled X1 1004-1, X2 1004-2, X3 1004-3, and X4 1004-4. In a first example, the rows of the input matrix X can be rearranged (e.g., reordered or permuted) to correspond to the rearranged columns of the rearranged coefficient matrix. Each of the multiple input sub-matrices X1, X2, X3, X4 of the compression layer includes a corresponding subset of the set of input activation values ​​included in the input matrix X of the selected layer. In a first example, each of the multiple input sub-matrices X1, X2, X3, X4 can include only the input activation values ​​of the input matrix X, which will be multiplied by the coefficients included in the multiple sub-matrices B1, B2, B3, C1, C2, C3 of the compression layer. The number of rows included in the multiple input sub-matrices X1, X2, X3, X4 can be equal to the number of columns included in the multiple sub-matrices B1 and B2, B3 and C1, and C2 and C3, respectively. That is: the number of rows of the input submatrix X1 can be equal to the number of columns of the submatrix B1; the number of rows of the input submatrix X2 can be equal to the number of columns of the submatrix B2; the number of rows of the input submatrix X3 can be equal to the number of columns of the submatrix B3; and the number of rows of the input submatrix X4 can be equal to the number of columns of the submatrices C1, C2 and C3. The rows of the input matrix X can be rearranged (e.g., reordered or permuted) to correspond to the rearranged columns of the rearranged coefficient matrix such that: the input sub-matrix X1 includes activation values ​​from the rows of X that correspond to (e.g., are to be multiplied by) the coefficient values ​​of the columns of the sub-matrix B1; the input sub-matrix X2 includes activation values ​​from the rows of X that correspond to (e.g., are to be multiplied by) the coefficient values ​​of the columns of the sub-matrix B2; the input sub-matrix X3 includes activation values ​​from the rows of X that correspond to (e.g., are to be multiplied by) the coefficient values ​​of the columns of the sub-matrix B3; and the input sub-matrix X4 includes activation values ​​from the rows of X that correspond to (e.g., are to be multiplied by) the coefficient values ​​of the columns of the sub-matrices C1, C2, and C3. In examples where each column of the rearranged coefficient matrix contains at least one element representing a non-zero coefficient, the rows of the input matrix X can be rearranged (e.g., reordered or permuted) to form the input sub-matrices X1, X2, X3, X4 without discarding any input activation values ​​from the input matrix X. That is, the sum of the number of input activation values ​​in all input sub-matrices X1, X2, X3, and X4 can be equal to the number of input activation values ​​in the input matrix X.

[0222] Figure 10Also shown are multiple output sub-matrices labeled Y1 1006-1, Y2 1006-2, and Y3 1006-3. In a first example, each of the multiple output sub-matrices Y1, Y2, Y3 of the compressed layer includes a corresponding subset of the set of output values ​​included in the output matrix Y of the selected layer. In the first example, when a "compressed" version of the matrix multiplication WX=Y is to be performed, the multiple input sub-matrices X1, X2, X3, X4 and the multiple sub-matrices B1, B2, B3, C1, C2, C3 can be used to form (e.g., a compressed fully connected operation can be performed) the multiple output sub-matrices Y1, Y2, Y3 using the following equation (3):

[0223] Y i =B i X i +C i X K+1 , where i=1,2…K (3)

[0224] It should be understood that equation (3) is a general formula that can be used to perform a compressed fully connected operation (e.g., a "compressed" version of the matrix multiplication WX=Y) using a submatrix of any K-way partitioned single-boundary block-diagonal matrix rearranged according to the column hyperedge hypergraph model. In this particular example, where K=3: the output submatrix Y1 can be formed by performing the matrix multiplication Y1=B1X1+C1X4; the output submatrix Y2 can be formed by performing the matrix multiplication Y2=B2X2+C2X4; and the output submatrix Y3 can be formed by performing the matrix multiplication Y3=B3X3+C3X4.

[0225] In a second example defined herein, the layer of the received neural network to be compressed is a convolutional layer, which is arranged to perform a convolution operation by convolving a set of input activation values ​​of the convolutional layer with a set of coefficients of the convolutional layer. In the second example, each of the one or more sub-matrices includes a plurality of elements representing a corresponding subset of input channels of a filter of the set of coefficients of the convolutional layer. For example, returning to reference Figure 6 and Figure 7 , Figure 6 The matrix 600 shown in comprises 196 elements representing 196 filter input channels (i.e., 14 input channels for 14 filters). Figure 7 The plurality of sub-matrices 702-1, 702-2, 702-3, and 702-4 shown in FIG include 103 elements representing 103 filter input channels (ie, a subset of the 14 input channels of the 14 filters).

[0226] For example, Figure 11B It shows how compression operations can be performed using multiple subsets of the set of coefficients of a convolutional layer. Figure 11B According to the reference Figure 11A The described principle uses a rearranged set of coefficients of a three-dimensional convolutional layer of a cuboid 1110. out ×C in Display on plane and Figure 7 The rearranged matrix 710 and Figure 9 The rearranged matrix 910 shown in FIG has equivalent characteristics. The C of the rearranged cuboid 1110 is arranged according to the row and column order of the rearranged matrix 910 (e.g., after rearrangement of the matrix 600 shown on the cuboid 1100). out ×C in In the plane there is (C in , C out ) coordinates of the one-dimensional coefficient sequence H w W w Each of them.

[0227] As described herein, rearranged coefficient matrix 910 includes a plurality of sub-matrices labeled Bl, B2, B3, Rl, R2, and R3. Figure 11B The multiple sub-matrices B1, B2, B3, R1, R2 and R3 shown in this article are the same as those in the reference Figure 9 The multiple sub-matrices B1, B2, B3, R1, R2 and R3 described have the same characteristics. Each of the multiple sub-matrices B1, B2, B3, R1, R2 and R3 includes multiple elements representing a corresponding subset of the input channels of the filter of the coefficient set of the convolutional layer.

[0228] As described in the paper, the input activation dataset of a convolutional layer can have dimension C in ×H a ×W a In the second example, the input activation data C of the convolutional layer in The dimensions may be rearranged (e.g., reordered or permuted) so as to correspond to the rearranged set of coefficients of the convolutional layer. in dimension. Figure 11B 1104 sets of rearranged input activation data for the convolutional layer are shown. Figure 11B As shown in FIG, the rearranged input activation data set can be divided into multiple subsets X1, X2, X3 of input activation data. The multiple subsets of input activation data X1, X2, X3 include input channels (C in ) may be equal to the input channels (C in ). That is, the subset of input activation data X1 has input channels (Cin ) may be equal to the number of input channels (C in ); The subset of input activation data X2 has input channels (C in ) can be equal to the number of input channels (C) in the subset of input channels of the filter of the coefficient set of the convolutional layer represented by the elements of the sub-matrices B2 and R2. in ) number; and the subset of input activation data X3 has input channels (C in ) may be equal to the number of input channels (C) in the subset of input channels of the filter of the coefficient set of the convolutional layer represented by the elements of the sub-matrices B3 and R3. in ). The number of input activation data of the convolutional layer is C in The dimensions may be rearranged (e.g., reordered or permuted) so as to correspond to the rearranged set of coefficients of the convolutional layer. in Dimension, such that: the subset of input activation data X1 includes the input channels (C in ), the activation values ​​corresponding to a subset of the input channels of the filter of the coefficient set of the convolutional layer represented by the elements of the sub-matrices B1 and R1 (e.g., to be convolved with it); the subset of input activation data X2 includes the input channels (C in ), the activation values ​​corresponding to a subset of the input channels of the filter of the coefficient set of the convolutional layer represented by the elements of the sub-matrices B2 and R2 (e.g., to be convolved with it); and the subset of input activation data X3 includes the input channels (C in ), which corresponds to a subset of the input channels of the filter of the coefficient set of the convolutional layer represented by the elements of the sub-matrices B3 and R3 (e.g., to be convolved with it). In the example where each column of the rearranged matrix contains at least one element representing a non-zero value, the C of the input activation data of the convolutional layer in The dimensions can be rearranged (e.g., reordered or permuted) to form subsets of the input activation data X1, X2, X3 without discarding any input activation values ​​from the input activation data. That is, the sum of the number of input activation values ​​in all subsets of the input activation data X1, X2, X3 can be equal to the number of values ​​in the input activation data.

[0229] Figure 11BAlso shown are multiple subsets of output activation data 1106 labeled Y1, Y2, Y3, and Y4. In a second example, each of the multiple subsets of output activation data Y1, Y2, Y3, Y4 for a compressed layer includes a corresponding subset of the set of output values ​​included by the output data of the selected layer. In the second example, the multiple subsets of output activation data Y1, Y2, Y3, Y4 can be formed (e.g., a compressed convolution operation can be performed) using the multiple subsets of input activation data X1, X2, X3 and the multiple subsets of input channels of the filter of the set of coefficients of the convolutional layer represented by the elements of the sub-matrices B1, B2, B3, R1, R2, R3 using equations (4) and (5) as follows:

[0230] where i=1,2…K(4)

[0231]

[0232] symbol represents the convolution operation. That is, Indicates that the input activates the data X i The subset of i The elements of represent the subset of the input channels of the filter of the coefficient set of the convolutional layer. It should be understood that equations (4) and (5) are general formulas that can be used to perform compressed convolution operations based on any K-way partitioned single boundary block-diagonal matrix rearranged according to the row hyperedge hypergraph model. In this specific example, where K=3: the convolution can be performed by To form a subset of the output activation data Y1; this can be done by performing a convolution To form a subset of the output activation data Y2; this can be done by performing a convolution to form a subset of the output activation data Y3; and can be achieved by performing convolution to form a subset of the output activation data Y4.

[0233] Based on the principles described herein, it should also be understood that, although not shown in the figures or described in detail for the sake of brevity, the following equation (6) is a general formula that can be used to perform compressed convolution operations based on any K-way partitioned single-boundary block-diagonal matrix submatrix rearranged according to a column hyperedge hypergraph model.

[0234] where i=1,2…K(6)

[0235] For example, in the specific example where K=3 (e.g., as Figure 10 , as described herein): the convolution can be performed by performing To form a subset of the output activation data Y1; this can be done by performing a convolution to form a subset of the output activation data Y2; and can be achieved by performing convolution to form a subset of the output activation data Y3.

[0236] Figure 12 Demonstrates methods for compressing attention-based neural networks. Figure 12 The method presented in

[15] for compressing attention-based neural networks is a computer-implemented method. Figure 5 The processing system 500 shown in FIG. 5 may be configured to perform Figure 12 The method presented in

[15] for compressing attention-based neural networks.

[0237] In step S1202, one or more processors 504 of the processing system 500 receive an attention-based neural network. The received attention-based neural network may include a transformer network. The received attention-based neural network may be as described above with reference to Figures 1A to 3 As described herein, and may be defined by a software model. For example, the software model may define a series of layers of the received attention-based neural network (e.g., the number of layers, the order of the layers, and the connectivity between the layers), and define each layer in the series according to the operations it is configured to perform and the set of coefficients to be used. The received attention-based neural network may be a trained neural network. That is, as will be appreciated by a skilled person, the received attention-based neural network may have been previously trained iteratively by processing training data in a forward pass; evaluating the accuracy of the output of the forward pass; and updating the set of coefficients for the layers in a backward pass. As described herein, the training process may typically drive to zero a majority of the coefficients of the set of coefficients used by the layers of the neural network. The attention-based neural network (e.g., the software model defining the neural network) may be trained in a manner similar to that described herein. Figure 5 Received at the processing system 500 shown in and stored in the memory 506.

[0238] The attention-based neural network includes an attention layer, such as a self-attention layer or an encoder-decoder attention layer as described above. As described above, the attention layer is arranged to implement an attention function based on a key matrix (K), a query matrix (Q) and a value matrix (V), wherein the attention layer is configured to use a key weight matrix (W K ) to determine the key matrix (K), using the query weight matrix (W Q ) to determine the query matrix (Q), and use the value weight matrix (W V ) to determine the value matrix (V). As described above, the key matrix, query matrix, and value matrix have the same size and shape. Each of the key matrix, query matrix, and value matrix can be an n×d matrix, such as Figure 3 As shown in .

[0239] The elements of the key weight matrix, query weight matrix, and value weight matrix are the coefficients of the attention layer that can be trained. Typically, many elements of the key weight matrix, query weight matrix, and value weight matrix are zero, that is, have zero values.

[0240] Figure 13 The query weight matrix (W Q )1302, key weight matrix (W K )1304 Sum value weight matrix (W V ) 1306, each of which includes a number of elements that are zero and a number of elements that are non-zero. In the description herein, an element is "zero" if it has a zero value, and an element is "non-zero" if it has a non-zero value. Figure 6 and Figure 7 The same symbols in Figure 13 Each non-zero element (i.e., each element having a non-zero value) in the query weight matrix 1302, the key weight matrix 1304, and the value weight matrix 1306 is marked with an "X," while zero elements (i.e., elements having a value of zero) are not marked. For example, the elements located in the 1st row, 2nd column and the 1st row, 5th column of the value weight matrix 1306 represent non-zero values, while the other elements located in the 1st row of the value weight matrix 1306 represent zero values. None of the query weight matrix 1302, the key weight matrix 1304, and the value weight matrix 1306 have subgraph separation.

[0241] Figure 14A A hypergraph model is presented that illustrates how the value weight matrix can be partitioned. Figure 14A In the example shown in , the hypergraph model may be a "row hyperedge" hypergraph model. According to the above description of the row hyperedge hypergraph model, a vertex representing each column of the value weight matrix is ​​formed, the vertex includes at least one element representing a non-zero value, and a hyperedge representing each row of the value weight matrix is ​​formed, the hyperedge includes an element representing a non-zero value. Figure 14A , the vertices of the hypergraph model are shown as circles, and the hyperedges of the hypergraph model are shown as triangles. For each of the non-zero valued elements within the value weight matrix, the vertex including the non-zero element representing the column of the value weight matrix is ​​connected to the hyperedge including the non-zero element representing the row of the value weight matrix. In another example, the hypergraph model may be a "column hyperedge" hypergraph model, as described above. In order to form a column hyperedge hypergraph model, a hyperedge including the non-zero valued element is formed to represent each column of the matrix, and a vertex including the non-zero valued element is formed to represent each row of the matrix.

[0242] Once formed, the hypergraph model can be partitioned. Figure 14AThe hypergraph model of row hyperedges is shown as being partitioned into three parts 1402-1, 1402-2, and 1402-3. It should be understood that the hypergraph model can be partitioned into any suitable number of parts. The rows and / or columns of the value weight matrix can be rearranged according to the partitioned hypergraph model so that multiple elements of the value weight matrix representing non-zero values ​​are aggregated into one or more sub-matrices.

[0243] For example, Figure 14B Display basis Figure 14A The rearranged value weight matrix 1406 of the hypergraph model partition shown in FIG. The rearranged value weight matrix 1406 includes the same information as the value weight matrix 1306, but the rows and columns of the matrix have been reordered and partitioned into multiple block arrays (1404-1, 1404-2, 1404-3) and a horizontal boundary array 1404-4. The block array and the boundary array are sub-matrices. Figure 14A and Figure 14B , portion 1402-1 of the hypergraph model corresponds to block array 1404-1, portion 1402-2 of the hypergraph model corresponds to block array 1404-2, and portion 1402-3 of the hypergraph model corresponds to block array 1404-3. Hyperedge n3 corresponding to row 3 of the value weight matrix corresponds to boundary array 1404-4. In other words, elements representing non-zero values ​​located in rows of the value weight matrix represented by hyperedges connected to vertices representing columns of the value weight matrix in more than one portion are aggregated into boundary array 1404-4.

[0244] To apply the value weight matrix to the embedding tensor to determine the value matrix, the embedding tensor can be reordered and partitioned to correspond to the reordering and partitioning of the value weight matrix into value weight sub-matrices. A set of value weight sub-matrices (1404-1, 1404-2, 1404-3, 1404-4) can be applied to the embedding sub-matrices representing the embedding tensor to determine the value matrix.

[0245] The query weight matrix can be independently rearranged and partitioned into a set of one or more query weight sub-matrices, and the key weight matrix can be independently rearranged and partitioned into a set of one or more key weight sub-matrices to be applied to the embedding tensor to determine the query matrix and the key matrix. Since the positions of non-zero elements will (usually) be different in the query weight matrix, the key weight matrix, and the value weight matrix, the optimal reordering and partitioning of the query, key, and value weight matrices will (usually) be different. If different reordering and partitioning are applied to the query, key, and value weight matrices to determine the query, key, and value matrices, then the embedding tensor will need to be reordered and partitioned differently to determine the query, key, and value matrices, and before the query, key, and value matrices are multiplied together, at least some of the rows and columns of the matrices will need to be reordered so that the correct elements of the matrices correspond to each other.

[0246] However, in the example described below, the same reordering and partitioning is applied to each of the query weight matrix, the key weight matrix, and the value weight matrix to determine the query weight submatrix, the key weight submatrix, and the value weight submatrix, respectively. This means that the embedding tensor only needs to be reordered and partitioned into the embedding submatrix once. The query weight submatrix, the key weight submatrix, and the value weight submatrix can all be applied to the same embedding submatrix to determine the query matrix, the key matrix, and the value matrix, respectively. In addition, since the rows and columns of the determined query matrix, the determined key matrix, and the determined value matrix all have the same reordering, the arrays can be combined (e.g., in matrix multiplication or convolution) without first reordering the rows or columns of the matrices. In addition, the compressed attention layer may include a processing block (e.g., a "linear layer") that processes the results of the attention function before providing the output of the compressed attention layer (e.g., by applying the output matrix to the results of the attention function). The rows and columns of the output matrix can be rearranged (e.g., complementary) according to the rearrangement of the rows and columns of each of the query weight matrix, the key weight matrix, and the value weight matrix. In this way, the output from the attention layer (which may be referred to as the 'output embedding tensor') is not changed by the rearrangement and partitioning of the elements of the embedding tensor. For example, the order of the output channels of the query weight matrix, key weight matrix, and value weight matrix may be applied to the order of the input channels of the processing block that determines the output of the compressed attention layer. This means that there is no need to add an additional aggregation layer to the compressed attention layer to undo the rearrangement and partitioning of the elements of the embedding tensor. It should be noted that if the permutation of the input channels of a processing block matches (i.e., is the same as) the permutation of the output channels of that block for determining the key matrix, query matrix, and value matrix, then the attention layer will produce the same output (e.g., the same ordering of the elements within the output embedding tensor) as if no permutation was applied in the attention layer.

[0247] In step S1204, one or more processors 504 of the processing system 500 determine a combination matrix C. The combination matrix has the same size and shape as each of the key weight matrix, the query weight matrix, and the value weight matrix. For each element of the combination matrix: (i) if the corresponding element of each of the key weight matrix, the query weight matrix, and the value weight matrix is ​​zero, then the element of the combination matrix is ​​determined to be zero, and (ii) if at least one of the corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix is ​​non-zero, then the element of the combination matrix is ​​determined to be non-zero. "Corresponding elements" are elements that are in the same position within the corresponding matrix. For example, step S1204 may involve determining the elements of the combination matrix by summing the corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix. It should be noted that all elements in the key weight matrix, the query weight matrix, and the value weight matrix can be represented in floating point format and can generally be positive, negative, or zero. Unless each of the three floating point values ​​is zero itself, the sum of the three corresponding floating point values ​​is unlikely to be exactly equal to zero. In fact, in the current context of summing the corresponding values ​​of the elements of the key weight matrix, the query weight matrix, and the value weight matrix (for which most of the elements typically have zero values), the following situation is unlikely: if the sum of the corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix is ​​equal to zero, then it is acceptable to consider that this indicates that all of those corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix are zero (and, in the event that such consideration is unacceptable, an alternative method can be adopted to implement step S1204, such as testing the elements in each matrix individually to see if the elements are zero). In addition, if the sum of the corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix is ​​not equal to zero, then this (definitely) indicates that at least one of those corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix is ​​non-zero. In this example, the element at position (i, j) in the combined matrix has a value C[i, j], which is given as follows:

[0248] If W K (i,j)≠0 or E Q (i,j)≠0 or W V (i, j) ≠ 0, then C(i, j) = x

[0249] If W K (i, j)=0 and W Q (i, j)=0 and W V (i, j) = 0, then C(i, j) = 0

[0250] Where x is any arbitrary non-zero value. In the example given above, the elements of the combination matrix are determined by summing the corresponding elements of the key weight matrix, the query weight matrix, and the value weight matrix, so that C(i, j) = W K (i,j)+W Q (i,j)+W V (i, j). In other examples, the elements of the combination matrix may be determined in a different manner in step S1204, rather than summing the corresponding values ​​of the key weight matrix, the query weight matrix, and the value weight matrix.

[0251] Figure 15A In step S1204, based on Figure 13 The combined matrix (C) 1501 is determined by the query weight matrix 1302, key weight matrix 1304, and value weight matrix 1306 shown in FIG. The combined matrix (C) 1501 includes multiple elements that are zero and multiple elements that are non-zero. As described above, if an element is "zero", then the element has a zero value, and if an element is "non-zero", then the element has a non-zero value. Figure 6 、 Figure 7 and Figure 13 In the case of the same symbol, Figure 15A Each non-zero element (i.e., each element having a non-zero value) in the combination matrix 1501 is marked with an "X," while zero elements (i.e., elements having a zero value) are not marked. For example, the elements located in the 1st row, 2nd column and the 1st row, 5th column of the combination matrix 1501 represent non-zero values, while the other elements located in the 1st row of the combination matrix 1501 represent zero values. The combination matrix 1501 does not have subgraph partitioning.

[0252] In step S1206, the one or more processors 504 of the processing system 500 rearrange the rows and columns of the combination matrix so as to cluster the multiple non-zero values ​​of the combination matrix into a set of one or more combination sub-matrices. Step S1206 may include performing a permutation of the rows and columns of the combination matrix and partitioning the permuted rows and columns of the combination matrix to determine a set of one or more combination sub-matrices. The set of one or more combination sub-matrices has an average density of non-zero values ​​greater than that of the combination matrix. In some examples, the rearrangement of the rows and columns of the matrix may not actually be performed. Instead, step S1206 may only involve determining a rearrangement of the rows and columns of the combination matrix that clusters the multiple non-zero values ​​of the combination matrix into a set of one or more combination sub-matrices (i.e., in some examples, the rearrangement may not actually be applied to the combination matrix).

[0253] Figure 15B A hypergraph model is presented that shows how to partition the combination matrix. Figure 15BIn the example shown in , the hypergraph model may be a "row hyperedge" hypergraph model. According to the above description of the row hyperedge hypergraph model, a vertex representing each column of the combination matrix is ​​formed, the vertex includes at least one element representing a non-zero value, and a hyperedge representing each row of the combination matrix is ​​formed, the hyperedge includes an element representing a non-zero value. Figure 15B , the vertices of the hypergraph model are shown as circles, and the hyperedges of the hypergraph model are shown as triangles. For each of the non-zero valued elements within the combination matrix, the vertex including the non-zero element representing the column of the combination matrix is ​​connected to the hyperedge including the non-zero element representing the row of the combination matrix. In another example, the hypergraph model may be a "column hyperedge" hypergraph model, as described above. To form a column hyperedge hypergraph model, a hyperedge including the non-zero valued element representing each column of the matrix is ​​formed, and a vertex including the non-zero valued element representing each row of the matrix is ​​formed.

[0254] As described above, once formed, the hypergraph model may be partitioned (in step S1206 ). Figure 15B A hypergraph model is shown that has been partitioned into two parts 1502-1 and 1502-2. It should be understood that the hypergraph model can be partitioned into any suitable number of parts. The rows and / or columns of the combination matrix can be rearranged according to the partitioned hypergraph model so as to aggregate multiple elements of the combination matrix representing non-zero values ​​into one or more sub-matrices. The hypergraph model is partitioned according to a load balancing constraint. In particular, a P-way partitioning of the combination matrix is ​​determined using a hypergraph that minimizes the number of cut hyperedges under the load balancing constraint.

[0255] For example, Figure 15C Display basis Figure 15B The rearranged combined matrix 1506 of the hypergraph model partition shown in FIG. The rearranged combined matrix 1506 includes the same information as the combined matrix 1501, but the rows and columns of the matrix have been reordered and partitioned into multiple block arrays (1504-1, 1504-2) and a horizontal boundary array 1504-3. The block array and the boundary array are sub-matrices. Figure 15B and Figure 15C , portion 1502-1 of the hypergraph model corresponds to block array 1504-1, and portion 1502-2 of the hypergraph model corresponds to block array 1504-2. Hyperedge n3 corresponding to row 3 of the combination matrix corresponds to boundary array 1504-3. In other words, elements representing non-zero values ​​located in rows of the combination matrix represented by hyperedges connected to vertices representing columns of the combination matrix in more than one portion are aggregated into boundary array 1504-3.

[0256] To apply the value weight matrix to the embedding tensor to determine the value matrix, the embedding tensor can be reordered and partitioned to correspond to the reordering and partitioning of the value weight matrix into value weight sub-matrices. A set of value weight sub-matrices (1404-1, 1404-2, 1404-3, 1404-4) can be applied to the embedding sub-matrices representing the embedding tensor to determine the value matrix.

[0257] In step S1208, the one or more processors 504 of the processing system 500 rearrange the rows and columns of the key weight matrix, the query weight matrix, and the value weight matrix to match the rearrangement of the rows and columns of the combination matrix (determined in step S1206) to determine a set of one or more key weight sub-matrices, a set of one or more query weight sub-matrices, and a set of one or more value weight sub-matrices corresponding to the set of one or more combination sub-matrices. In this manner, the reordering and partitioning are the same for each of the following: the key weight matrix, the query weight matrix, and the value weight matrix. Therefore, each of the set of key weight sub-matrices, the set of query sub-matrices, and the set of value weight sub-matrices has an element at a position that matches the position of the corresponding element in the set of combination sub-matrices.

[0258] Figure 15D Display has been based on Figure 15B The rearranged query weight matrix 1508, the rearranged key weight matrix 1512, and the rearranged value weight matrix 1516 of the hypergraph model rearranged and partitioned as shown in FIG. The rearranged query weight matrix 1508 includes Figure 13 , but the rows and columns of the matrix have been reordered and divided into multiple block arrays (1510-1, 1510-2) and a horizontal boundary array 1510-3. Similarly, the rearranged key weight matrix 1512 includes the same information as the query weight matrix 1302 shown in FIG. Figure 13 , but the rows and columns of the matrix have been reordered and divided into multiple block arrays (1514-1, 1514-2) and a horizontal boundary array 1514-3. Similarly, the rearranged value weight matrix 1516 includes the same information as the key weight matrix 1304 shown in FIG. Figure 13 , but the rows and columns of the matrix have been reordered and partitioned into multiple block arrays (1518-1, 1518-2) and a horizontal boundary array 1518-3. It can be seen that the ordering and partitioning of the rows and columns of the rearranged query weight matrix 1508, the rearranged key weight matrix 1512, and the rearranged value weight matrix 1516 are the same as Figure 15C The ordering and partitioning of the rows and columns of the rearranged combination matrix 1506 shown in is the same.

[0259] The non-zero density of the plurality of query weight sub-matrices 1510-1, 1510-2, and 1510-3 is generally greater than the non-zero density of the query weight matrix 1302. The non-zero density of the plurality of key weight sub-matrices 1514-1, 1514-2, and 1514-3 is generally greater than the non-zero density of the key weight matrix 1304. The non-zero density of the plurality of value weight sub-matrices 1518-1, 1518-2, and 1518-3 is generally greater than the non-zero density of the value weight matrix 1306.

[0260] Figure 15D Each of the rearranged matrices (1508, 1512, and 1516) shown in FIG is in a "single-border block-diagonal matrix form." This means that the rearranged matrix includes one or more block arrays arranged on a diagonal line and a border array arranged along one side of the rearranged matrix. The one or more block arrays and the border array are non-overlapping. Figure 15D In , the block arrays are arranged on the diagonal line from the upper left to the lower right. Figure 15DIn the example, the block arrays are arranged along the bottom of the rearranged matrix. In other examples, the same functional benefits can be achieved, although the block arrays can be arranged on different diagonals (e.g., top left to bottom left), or may not be arranged on a diagonal at all. That is, for example, the columns of the rearranged matrix can be arranged in different ways so that the "horizontal" positions of two or more block arrays are swapped. In other examples, the block arrays can be arranged along different sides of the rearranged matrix (e.g., along the top, left, or right side of the matrix), or there can be more than one block array, each arranged along a different side of the rearranged matrix. The block arrays can be referred to as horizontal arrays or vertical arrays. The horizontal array can be arranged horizontally on the rearranged matrix. The horizontal array can span the entire width of the rearranged matrix. The vertical array can be arranged vertically on the rearranged matrix. The vertical array can span the entire height of the rearranged matrix. In these other examples, the rearranged matrix can be referred to as "block matrix form." A matrix in a single block matrix-diagonal matrix form can also be referred to as in block matrix form. As will be understood by those skilled in the art, the border array need not be arranged along one side of the rearranged matrix. That is, for example, the border array can be arranged between two block arrays in the block array. In summary, the rows and / or columns of the matrix can be rearranged to form a rearranged matrix, the rearranged matrix comprising: one or more block arrays arranged along the diagonals of the rearranged matrix, and / or one or more block arrays not arranged along the diagonals of the rearranged matrix; and one or more horizontal arrays arranged horizontally on the rearranged matrix, and / or one or more vertical arrays arranged vertically on the rearranged matrix. The multiple elements representing non-zero values ​​included in the matrix can be gathered into the block array and the horizontal and / or vertical arrays of the rearranged matrix. In other words, the rows and / or columns of the matrix can be rearranged to form: a rearranged matrix in the form of a border block matrix; or a rearranged matrix as a block matrix, the block matrix comprising an array that can be arranged into (e.g., can be rearranged into) the form of a border block matrix.

[0261] It can be seen that the rearranged value weight matrix 1516 has a different ordering of rows and columns and Figure 14B1406 . The partitioning of the rearranged value weight matrix 1406 provides sub-matrices 1404-1, 1404-2, 1404-3, and 1404-4 that collectively have a greater 'non-zero density' than would be provided by partitioning the rearranged value weight matrix 1516 into sub-matrices 1518-1, 1518-2, and 1518-3. This is because the rearrangement and partitioning applied to determine the rearranged value weight matrix 1406 only considers the values ​​of the elements in the value weight matrix 1306. In contrast, the arrangement and partitioning applied to determine the rearranged value weight matrix 1516 considers the values ​​of the elements in the query weight matrix 1302, the key weight matrix 1304, and the value weight matrix 1306 (by using the combination matrix 1501). The inventors of this case have recognized that although the non-zero density of the sub-matrices of the rearranged matrices 1508, 1512 and 1516 is not as high as when each of the query weight matrix, key weight matrix and value weight matrix is ​​rearranged and partitioned separately, there are some significant benefits to using the same rearrangement and partitioning for each of the query weight matrix, key weight matrix and value weight matrix in the attention layer of the attention-based neural network. In particular, as will be seen below, the number of aggregation layers that need to be implemented in the attention layer will be reduced. This is because the same rearrangement and partitioning can be applied to the embedding tensor to determine a set of one or more embedding sub-matrices, and the query weight sub-matrix, key weight sub-matrix and value weight sub-matrix can be applied to the set of one or more embedding sub-matrices to determine the query matrix, key matrix and value matrix. In addition, because the same rearrangement of rows and columns is applied to determine the query matrix, key matrix and value matrix, there is no need to apply another rearrangement of rows and / or columns of the query matrix, key matrix and value matrix before the matrix can be combined with other matrices, such as in a matrix multiplication or convolution operation.

[0262] As described in the following examples, the compressed attention layer includes a processing block (e.g., a convolution block) that is configured to apply an output matrix to the result of implementing the attention function before providing the output of the compressed attention layer. In step S1209, the one or more processors 504 of the processing system 500 rearrange the rows and columns of the output matrix according to the determined rearrangement of the rows and columns of the combination matrix (e.g., complement each other). In this way, the compressed attention layer is configured to provide an output (which is an embedding tensor) that has elements in the correct order, such as in an order that matches the order of the elements of the embedding tensor received at the compressed attention layer. For example, the order of the elements of the output embedding tensor may not change due to the compression of the attention layer. It should be noted that in some examples, the order of the elements of the output embedding tensor may not match the order of the elements of the embedding tensor received at the compressed attention layer. Furthermore, in some examples, step S1209 may not be performed.

[0263] In step S1210, the one or more processors 504 of the processing system 500 output a compressed attention-based neural network comprising a compressed attention layer, wherein the compressed attention layer is arranged to implement an attention function by performing a compression operation based on: (i) a set of one or more key weight sub-matrices, (ii) a set of one or more query weight sub-matrices, and (iii) a set of one or more value weight sub-matrices. The compressed attention layer is arranged to perform the same type of operation (i.e., an attention operation) as the attention layer of the (uncompressed) neural network. That is, the compressed layer is arranged to perform the operation based on the query weight sub-matrix, the key weight sub-matrix, and the value weight sub-matrix formed in step S1208, for example, rather than being arranged to perform the operation based on the complete query weight matrix, the complete key weight matrix, and the complete value weight matrix. In other words, only a subset of the set of coefficients represented by the elements of the one or more sub-matrices formed in step S1208 can be used by the compressed attention layer. The other coefficients in the set of coefficients represented by the elements of the query, key, and value matrices that are not included in the one or more sub-matrices formed in step S1208 are not used by the compressed attention layer. The other coefficients are only zero coefficients, so that the element-by-element multiplication between the input activation value and any of those other coefficients will inevitably produce a zero output value. Therefore, the compressed attention layer does not lose "information" by not using the other coefficients. In other words, the method of compressing an attention-based neural network described herein is a lossless method of compressing an attention-based neural network.

[0264] Step S1210 may include storing the compressed attention-based neural network for subsequent implementation. Figure 5, at least one processor 504 may write the compressed neural network to the memory 506 for subsequent implementation. Step S1210 may include outputting a computer-readable description of the compressed attention-based neural network, which, when used in a system for implementing a neural network (e.g., Figure 4 When implemented at the system 400 shown in FIG. 1 , the computer-readable description causes the compressed attention-based neural network to be executed. Step S1210 may include configuring the system for implementing the attention-based neural network to implement the compressed attention-based neural network.

[0265] Step S1210 may include configuring hardware logic to implement the compressed attention-based neural network. The hardware logic may include a neural network accelerator or a graphics processing unit. For example, referring to Figure 5 , the at least one processor 504 may be configured to configure the hardware logic 402 included in the system 400 for implementing a neural network to implement a compressed attention-based neural network.

[0266] According to this article reference Figure 12 The described method of compressing a received attention-based neural network is advantageous because only a subset of the set of coefficients of the attention layer of the received attention-based neural network (i.e., only a subset of the elements of the query weight matrix, key weight matrix, and value weight matrix) is used by the compressed attention layer of the compressed attention-based neural network. That is, only the subset of the set of coefficients represented by the elements of the submatrix formed in step S1208 is used by the compressed attention layer. The other coefficients (i.e., the elements of the query weight matrix, key weight matrix, and value weight matrix that have the value zero and are not contained in the query weight submatrix, key weight submatrix, and value weight submatrix) do not need to be stored in memory, such as off-chip memory 404. This reduces the memory footprint of the compressed attention-based neural network relative to the received attention neural network. Furthermore, this means that fewer coefficients need to be read into hardware logic (e.g., a neural network accelerator) at runtime in order to implement the compressed attention layer relative to the (uncompressed) attention layer. This reduces the weight bandwidth required to implement the compressed attention layer relative to the (uncompressed) attention layer. For the same reason, the time it takes to read in the set of coefficients for the compressed attention layer from memory is also reduced relative to the (uncompressed) attention layer. This may reduce the latency of implementing the compressed attention-based neural network relative to the (uncompressed) attention-based neural network. Likewise, the smaller set of coefficients used by the compressed attention layer, relative to the larger set of coefficients used by the (uncompressed) attention layer, also reduces the computational demands on the processing elements of the hardware logic implementing the compressed attention-based neural network by reducing the number of multiplication and accumulation operations performed by these processing elements in order to perform the compression operations.

[0267] Figure 16 A method for implementing a compressed attention-based neural network is presented. The compressed attention-based neural network is the same as that described above. Figure 12 The compressed attention-based neural network includes a compressed attention layer arranged to implement an attention function. Figure 16 The method of implementing a compressed attention-based neural network shown in is a computer-implemented method. Processing system 400, such as Figure 4 and Figure 5 The hardware logic 402 shown in FIG. 4 may be configured to perform Figure 16

[0014] The method of implementing a compressed attention-based neural network shown in

[0015] As described above, the attention-based neural network can be a transformer network.

[0268] In step S1602, the processing system 400, such as the hardware logic 402, receives an embedding tensor at a compressed attention layer of a compressed attention-based neural network. Figures 1A to 3 As described, the embedding tensor may represent an input sequence, an output from a previous layer of an encoder in a compressed attention-based neural network, or an output from a previous layer of a decoder in a compressed attention-based neural network. For example, the compressed attention-based neural network may be configured to: receive an input sequence (e.g., Figure 1A 、 Figure 1B and 1C 101, 151 or 181 shown in ); determining an embedding tensor representing the input sequence; and providing the embedding tensor to a compressed attention layer.

[0269] Figure 17 A first example of a compressed attention layer of a compressed attention-based neural network is presented. The compressed attention-based neural network is configured to provide the same functionality as the (uncompressed) attention-based neural network. In particular, the compressed attention layer of the compressed attention-based neural network is configured to provide the same functionality (i.e., the same output) as the (uncompressed) attention layer of the (uncompressed) attention-based neural network. As described above, the (uncompressed) attention layer is configured to: receive one or more embedding tensors (in Figure 17denoted as "embedding" in

[0045] ); determining a key matrix by applying a key weight matrix to one of the one or more received embedding tensors; determining a query matrix by applying a query weight matrix to one of the one or more received embedding tensors; and determining a value matrix by applying a value weight matrix to one of the one or more received embedding tensors. In contrast, a compressed attention layer of a compressed attention-based neural network is configured to: receive one or more embedding tensors; rearrange and partition elements of the embedding tensors into a set of one or more embedding sub-matrices; determine a key matrix by applying a set of one or more key weight sub-matrices to the one or more embedding sub-matrices; determine a query matrix by applying a set of one or more query weight sub-matrices to the one or more embedding sub-matrices; and determine a value matrix by applying a set of one or more value weight sub-matrices to the one or more embedding sub-matrices.

[0270] In particular, in step S1604, the processing system 400, such as the hardware logic 402, rearranges and partitions the elements of the embedding tensor to form one or more embedding sub-matrices. During compression of the compressed attention layer, the one or more embedding sub-matrices are determined by rearranging and partitioning the elements of the embedding tensor to match the rearrangement and partitioning of the rows and columns of the query matrix, the key matrix, and the value matrix, as described above, and step S1604 is performed by: (i) one or more aggregation layers (such as Figure 17 ), or (ii) the aggregation and splitting layers of the compressed attention layers (as described below) Figure 18 ). Figure 17 In the example shown in , the compressed attention layer includes one or more aggregation layers configured to rearrange and partition the elements of the embedding tensor to form one or more embedding sub-matrices. In particular, Figure 17 A compressed attention layer is shown that includes two clustering layers 17021 and 17022, each of which is configured to receive an embedding tensor and output a corresponding one of the embedding sub-matrices. In this example, each of the query weight matrix, the key weight matrix, and the value weight matrix is ​​split into three sub-matrices (i.e., two block arrays and one boundary array), for example, as Figure 15D As shown in the example of . The embedding tensor is split into two embedding sub-matrices. Typically, the embedding tensor is split into a number of embedding sub-matrices, which is one less than the number of sub-matrices into which the query weight matrix, key weight matrix, and value weight matrix are each split.

[0271] Figure 17 The compressed attention layer shown in FIG also includes nine convolution blocks 1704. Three of the convolution blocks (1704 Q1 、1704 Q2 and 1704Q3 ) is configured to determine the query matrix Q. The three convolution blocks (1704 K1 、1704 K2 and 1704 K3 ) is configured to determine the key matrix K. The three convolution blocks (1704 V1 、1704 V2 and 1704 V3 ) is configured to determine a value matrix V. The convolution block is configured to implement matrix multiplication as a convolution operation. In other examples, matrix multiplication can be implemented using a matrix multiplication ("matmul") block instead of a convolution block. The inventors of this case have discovered that, in some systems, using a convolution block to implement a matrix multiplication operation is more efficient than using a matrix multiplication block to implement a matrix multiplication operation. It should be noted that a matrix multiplication operation can be considered a specific example of a convolution operation.

[0272] The first clustering layer 17021 determines a first embedding sub-matrix from the embedding tensor. The first clustering layer 17021 provides the first embedding sub-matrix to the three convolution blocks 1704 Q1 、1704 K1 and 1704 V1 ,like Figure 17 The second clustering layer 17022 determines a second embedding sub-matrix from the embedding tensor. The second clustering layer 17022 provides the second embedding sub-matrix to the three convolution blocks 1704. Q2 、1704 K2 and 1704 V2 ,like Figure 17 The embedding tensor is provided to three convolution blocks 1704. Q3 、1704 K3 and 1704 V3 ,like Figure 17 , shown by the dashed line in FIG.

[0273] Steps S1606, S1608 and S1610 may be performed in any order, sequentially or in parallel. In step S1606, convolution block 1704 K1 、1704 K2 and 1704 K3 The set of key weight sub-matrices is applied to one or more embedding sub-matrices. In particular, convolution block 1704 K1 The first key weight sub-matrix 1514-1 is applied to the first embedding sub-matrix to determine the first partial key result, convolution block 1704 K2 The second key weight sub-matrix 1514-2 is applied to the second embedding sub-matrix to determine a second partial key result, and the convolution block 1704 K3The third key weight sub-matrix 1514-3 is applied to the (full) embedding tensor to determine the third partial key result. The first partial key result, the second partial key result, and the third partial key result are provided to the key concatenation layer 1706 K . Key splicing layer 1706 K The first partial key result, the second partial key result, and the third partial key result are concatenated to determine the key matrix K.

[0274] In step S1608, the convolution block 1704 Q1 、1704 Q2 and 1704 Q3 The set of query weight sub-matrices is applied to one or more embedding sub-matrices. In particular, convolution block 1704 Q1 The first query weight sub-matrix 1510-1 is applied to the first embedding sub-matrix to determine a first partial query result, convolution block 1704 Q2 The second query weight sub-matrix 1510-2 is applied to the second embedding sub-matrix to determine a second partial query result, and the convolution block 1704 Q3 The third query weight sub-matrix 1510-3 is applied to the (full) embedding tensor to determine the third partial query result. The first partial query result, the second partial query result, and the third partial query result are provided to the query concatenation layer 1706 Q Query splicing layer 1706 Q The query matrix Q is determined by concatenating the first part of the query results, the second part of the query results, and the third part of the query results.

[0275] In step S1610, the convolution block 1704 V1 、1704 V2 and 1704 V3 The set of value weight sub-matrices is applied to one or more embedding sub-matrices. In particular, convolution block 1704 V1 The first value weight sub-matrix 1518-1 is applied to the first embedding sub-matrix to determine a first partial value result, convolution block 1704 V2 The second value weight sub-matrix 1518-2 is applied to the second embedding sub-matrix to determine a second partial value result, and the convolution block 1704 V3 The third value weight sub-matrix 1518-3 is applied to the (full) embedding tensor to determine a third partial value result. The first partial value result, the second partial value result, and the third partial value result are provided to the value concatenation layer 1706 V . Value splicing layer 1706 V The first partial value result, the second partial value result, and the third partial value result are concatenated to determine the value matrix V.

[0276] When the query matrix Q, key matrix K and value matrix V have been determined, the method of implementing attention can be as described above with respect to Figures 1A to 3 Specifically, in step S1612, the processing system 400, e.g., the hardware logic 402, implements the attention function using the determined key matrix, the determined query matrix, and the determined value matrix. As described above, step S1612 may include using a scaled dot product attention calculation, e.g., as described above with respect to Figure 1C For example, as described above, the attention function Attention(Q, K, V) can be given by:

[0277]

[0278] Where Q is the query matrix, K T is the transpose of the key matrix, V is the value matrix, and d k is the dimension of query matrix and key matrix. To implement this attention function, Figure 17 The compressed attention layer shown in FIG includes: a transpose block 1710 configured to determine the transpose of the key matrix; a first multiplication block 1708 configured to determine the transpose K of the query matrix Q and the key matrix T The dot product (i.e., multiplication) between ; attention and softmax block 1712, which is configured to divide the output of the first multiplication block 1708 by An attention mask is applied, and a softmax function is applied; and a second multiplication block 1714 configured to multiply the output of the attention and softmax block 1712 with a value matrix V.

[0279] Figure 17 The compressed attention layer shown in also includes a processing block 1716 configured to apply an output matrix to the result of implementing the attention function before providing the output of the compressed attention layer. Figure 17In the example shown in , the processing block is a convolution block, but in other examples, the processing block can be a different type of processing block, such as a matrix multiplication block. In step S1613, the final convolution block 1716 applies the output matrix (e.g., as a convolution) to the output of the second multiplication block 1714 to reorder the channels and determine the output of the compressed attention layer (i.e., the "attention result"). The ordering of the rows and columns of the output matrix is ​​complementary to the rearrangement of the elements of the embedding tensor performed in step S1604. In other words, the rows and columns of the output matrix can be ordered (e.g., complementary) according to the ordering of the rows and columns of each of the query weight matrix, the key weight matrix, and the value weight matrix. In some examples, the reordering of the channels applied by the convolution block 1716 is opposite to the reordering of the channels applied by the aggregation layers 17021 and 17022 to determine the output from the compressed attention layer. For example, the input channel order of the convolution block 1716 can be complementary to the rearrangement of the elements of the embedding tensor (performed in step S1604). In this way, the ordering of the elements of the output embedding tensor (i.e., the output of the compressed attention layer) can match the ordering of the elements of the embedding tensor received at the compressed attention layer. For example, the order of the output channels of the query weight matrix, key weight matrix, and value weight matrix can be applied to the input channel order of processing block 1716 to determine the output of the compressed attention layer; and the compressed attention layer produces the same output as the original (uncompressed) attention layer would produce (i.e., the same output embedding tensor with the same element ordering). It should be noted that there is no need to add additional aggregation layers to the compressed attention layer to 'undo' the rearrangement and splitting of the elements of the embedding tensor. It should also be noted that if the permutation of the input channels of a processing block matches the permutation of the output channels of that block (i.e., is the same) for determining the key matrix, query matrix, and value matrix, then the compressed attention layer will produce the same output as if no permutation was applied at the compressed attention layer.

[0280] In other instances, where the compressed attention layer is followed by another layer (referred to as the “next layer”) in which the input embedding tensor is to be rearranged and partitioned, the ordering of the rows and columns of the output matrix applied by convolution block 1716 can cause the ordering of the elements in the output embedding tensor to match the desired ordering of the elements in the embedding tensor received by the next layer, so that the next layer does not need to include a clustering layer for rearranging the elements of the embedding tensor it receives (although it may still have a splitting layer that partitions the embedding tensor it receives into embedding sub-matrices).

[0281] It should also be noted that in some examples, step S1613 may not be performed.

[0282] The compressed attention layer can be configured to implement multi-head attention, as described above. In particular, the compressed attention layer can be configured to implement multi-head attention by splitting the key matrix, the query matrix, and the value matrix into multiple heads; determining a head attention result for each of the heads; and concatenating the determined head attention results for each of the heads to determine an attention result of the compressed attention layer.

[0283] In step S1614, the processing system 400, such as the hardware logic 402, processes the output of the compressed attention layer in the compressed attention-based neural network.

[0284] As described above, the compressed attention-based neural network can be a transformer network. As described above, the compressed attention-based neural network can include an encoder stack and a decoder stack. For example, the compressed attention-based neural network can be a large language model (LLM). To give some examples, the compressed attention-based neural network can be implemented to perform one of the following: natural language processing, language translation, computer vision processing, image processing, text processing, and speech processing.

[0285] For example, the compressed attention layer may be a self-attention layer within the encoder of the compressed attention-based neural network. As another example, the compressed attention layer may be a self-attention layer within the decoder of the compressed attention-based neural network. As another example, the compressed attention layer may be an encoder-decoder attention layer within the decoder of the compressed attention-based neural network. Figures 1A to 3 As described, the encoder and / or decoder may include a feed-forward layer and / or a normalization layer.

[0286] As mentioned above Figure 2B As described, when the compressed attention layer is an encoder-decoder attention layer within a decoder of a compressed attention-based neural network, a key matrix and a value matrix are determined using a first embedding tensor representing an output from an encoder layer in the compressed attention-based neural network, and a query matrix is ​​determined using a second embedding tensor representing an output from a previous layer in the decoder.

[0287] Figure 18 Showing a second example of a compressed attention layer for an attention-based neural network. Figure 18 The compressed attention layer shown in Figure 17 The compressed attention layer is the same as shown in Figure 17 In the example shown in FIG, two aggregation layers 17021 and 17022 are Figure 18 In this way, Figure 18The compressed attention layer shown in includes: an aggregation layer 1802 configured to rearrange elements of an embedding tensor; and a splitting layer 1803 configured to split the output from the aggregation layer 1802 to form one or more embedding sub-matrices.

[0288] In this example, in step S1604, the split layer 1803 determines the first embedding sub-matrix and provides it to the three convolution blocks 1704 Q1 、1704 K1 and 1704 V1 ,like Figure 18 The split layer 1803 also determines the second embedding sub-matrix and provides it to the three convolution blocks 1704. Q2 、1704 K2 and 1704 V2 ,like Figure 18 The output from the aggregation layer 1802 is provided to three convolution blocks 1704. Q3 、1704 K3 and 1704 V3 ,like Figure 18 It should be noted that in Figure 17 In the example shown in FIG, the channel order of the received embedding tensor is used to determine the convolution layer corresponding to the boundary block (i.e., corresponding to the convolution block 1704 Q3 、1704 K3 and 1704 V3 Therefore, in Figure 17 In the example shown in , no aggregation layer is used before the boundary block convolution. After obtaining the single boundary form, the columns of the boundary block can be rearranged in any order we want, separate from the diagonal blocks. Multiple parts of the boundary block in the single boundary form can also be combined into a single entity. In contrast, in Figure 18 In the example shown in , the channels of the received embedding tensor are reordered using a single clustering layer 1802. In this example, the column order of the boundary blocks is the same as the column order of the single boundary form of the matrix.

[0289] The remainder of the method (ie, steps S1606 to S1614 ) is performed as described above.

[0290] It can be seen that in Figure 17 and Figure 18In the example shown in , in step 1604, two clustering layers (17021 and 17022) or one clustering layer and one splitting layer (1802 and 1803) are implemented to rearrange and partition the elements of the embedding tensor to form one or more embedding sub-matrices, and these same embedding sub-matrices are used to determine each of the query matrix, key matrix, and value matrix. This is possible because the query weight matrix, key weight matrix, and value weight matrix are all rearranged and partitioned in the same way. If the query weight matrix, key weight matrix, and value weight matrix are not rearranged and partitioned in the same way, then the embedding tensor will need to be rearranged and partitioned in a different way to determine the query, key, and value matrices, and therefore, a greater number of clustering layers and / or splitting layers will be required in the compressed attention layer. Furthermore, since the key matrix, query matrix, and value matrix are all reordered in the same way, they can be applied directly to each other (e.g., in multiplication blocks 1708 and 1716) without the need to implement any clustering layer or splitting layer after the concatenation layer 1706. Thus, the method described herein where the query weight matrix, key weight matrix, and value weight matrix are all rearranged and partitioned in the same manner allows for a reduction in the number of aggregation layers and / or splitting layers that need to be implemented in the compressed attention layer. This yields technical benefits in attention-based neural networks, such as reduced latency and power consumption for implementing attention-based neural networks. The identical ordering of the input and output channels of the projection matrix ensures correct attention scores, and the number of aggregation layers is minimized by using the same reordering for each of K, Q, and V.

[0291] Figure 19 A computer system is shown in which the neural network system described herein may be implemented. The computer system includes a CPU 1902, a GPU 1904, a memory 1906 (e.g., equivalent to Figure 4 ), a neural network accelerator (NNA) 1908, and other devices 1914, such as a display 1916, a speaker 1918, and a camera 1922 (e.g., equivalent to Figure 4 Input 401 shown in ). Figure 4 1908. The hardware logic 402 shown in FIG. 1902 may be implemented using a GPU 1904 or a NNA 1908. In some examples, a camera 1922 may input activation data directly into a neural network accelerator (NNA) 1908. In some examples, the NNA may be a GPU with one or more hardware accelerators for accelerating one or more neural network operations. It should be understood that one or more of the depicted components may be omitted from the system. The components of the computer system may communicate with each other via a communication bus 1920.

[0292] The systems described herein are shown in the figures as comprising a plurality of functional blocks. This is for illustrative purposes only and is not intended to define a strict division between the different logical elements of such entities. Each functional block may be provided in any suitable manner. It should be understood that intermediate values ​​described herein as being formed by at least one processor need not be physically generated by the at least one processor at any point and may merely represent logical values ​​that conveniently describe the processing performed by the at least one processor between its inputs and outputs.

[0293] The processing system described herein can be embodied in hardware on an integrated circuit. The processing system described herein can be configured to perform any method described herein. In general, any of the functions, methods, techniques or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuitry) or any combination thereof. The terms "module," "function," "component," "element," "unit," "block," and "logic" can be used herein to generally refer to software, firmware, hardware, or any combination thereof. In the case of software implementation, a module, function, component, element, unit, block, or logic represents program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein can be performed by one or more processors that execute code, which causes the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, and other technologies to store instructions or other data and can be accessed by a machine.

[0294] As used herein, the terms computer program code and computer-readable instructions refer to any type of executable code for a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, byte code, code that defines an integrated circuit (e.g., a hardware description language or netlist), and code expressed in programming language code such as C, Java, or OpenCL. Executable code can be, for example, any type of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes a processor of a computer system supporting the executable code to perform the tasks specified by the code.

[0295] A processor, computer, or computer system may be any kind of device, machine, or specialized circuit, or a collection or portion thereof, that has processing capabilities such that it can execute instructions. A processor may be or include any kind of general-purpose or specialized processor, such as a CPU, GPU, NNA, system on a chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0296] The present invention is also intended to cover software (e.g., HDL (Hardware Description Language) software) that defines the configuration of hardware as described herein, said software being used to design integrated circuits or to configure programmable chips to perform desired functions. That is, a computer-readable storage medium may be provided on which is encoded computer-readable program code in the form of an integrated circuit definition data set, said computer-readable program code, when processed (i.e., executed) in an integrated circuit manufacturing system, configures said system to manufacture a processing system configured to perform any of the methods described herein, or to manufacture a processing system including any of the apparatus described herein. The integrated circuit definition data set may be, for example, an integrated circuit description.

[0297] Thus, a method of manufacturing a processing system as described herein at an integrated circuit manufacturing system may be provided. Furthermore, an integrated circuit definition data set may be provided which, when processed in an integrated circuit manufacturing system, enables the method of manufacturing a processing system to be performed.

[0298] The integrated circuit definition data set may be in the form of computer code, for example, as a netlist, code for configuring a programmable chip, as a hardware description language that defines hardware suitable for fabrication at any level in the integrated circuit, including as register transfer level (RTL) code, as a high-level circuit representation (e.g., Verilog or VHDL), and as a low-level circuit representation (e.g., OASIS (RTM) and GDSII). The higher-level representation (e.g., RTL) that logically defines hardware suitable for fabrication in the integrated circuit may be processed at a computer system configured to generate a manufacturing definition of the integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining those elements to generate a manufacturing definition of the integrated circuit so defined by the representation. As is typically the case when software is executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate a manufacturing definition of the integrated circuit in order to execute the code that defines the integrated circuit in order to generate the manufacturing definition of the integrated circuit.

[0299] Now refer to Figure 20An example of processing an integrated circuit definition data set at an integrated circuit manufacturing system to configure the system into a manufacturing processing system is described.

[0300] Figure 20 An example of an integrated circuit (IC) manufacturing system 2002 is shown, which is configured to manufacture a processing system as described in any of the examples herein. Specifically, the IC manufacturing system 2002 includes a layout processing system 2004 and an integrated circuit generation system 2006. The IC manufacturing system 2002 is configured to receive an IC definition data set (e.g., defining a processing system as described in any of the examples herein), process the IC definition data set, and generate an IC (e.g., embodying a processing system as described in any of the examples herein) based on the IC definition data set. Processing the IC definition data set configures the IC manufacturing system 2002 to manufacture integrated circuits embodying a processing system and / or hardware logic as described in any of the examples herein.

[0301] The layout processing system 2004 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout based on an IC definition data set are known in the art and may, for example, involve synthesizing RTL code to determine a gate-level representation of the circuit to be generated, such as for logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined from the gate-level representation of the circuit. This can be done automatically or with user input to optimize the circuit layout. When the layout processing system 2004 has determined the circuit layout, it can output the circuit layout definition to the IC generation system 2006. The circuit layout definition can be, for example, a circuit layout description.

[0302] As is known in the art, IC generation system 2006 generates an IC based on a circuit layout definition. For example, IC generation system 2006 may implement a semiconductor device manufacturing process to generate the IC, which may involve a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask that can be used in the photolithography process to generate the IC based on the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 2006 may be in the form of computer-readable code that IC generation system 2006 can use to generate a suitable mask for generating the IC.

[0303] The various processes performed by IC manufacturing system 2002 may all be performed at a single location, such as by one party. Alternatively, IC manufacturing system 2002 may be a distributed system, such that some processes may be performed at different locations and by different parties. For example, some of the following stages may be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of the circuit to be produced; (ii) generating a circuit layout based on the gate-level representation; (iii) generating a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0304] In other examples, processing of an integrated circuit definition data set at an integrated circuit manufacturing system may configure the system to manufacture a processing system without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set may define a configuration of a reconfigurable processor, such as an FPGA, and processing the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0305] In some embodiments, the integrated circuit manufacturing definition data set, when processed in the integrated circuit manufacturing system, can cause the integrated circuit manufacturing system to generate the device as described herein. Figure 20 Configuring an integrated circuit fabrication system in the manner described may enable the fabrication of devices as described herein.

[0306] In some examples, the integrated circuit definition data set may include software that runs on, or in combination with, hardware defined at the data set. Figure 20 In the example shown in , the IC production system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined in the integrated circuit definition dataset when manufacturing the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0307] Compared to known embodiments, the concepts set forth in this application can improve performance in devices, equipment, modules and / or systems (and in the methods implemented herein). Performance improvements can include one or more of improved computing performance, reduced latency, increased throughput and / or reduced power consumption. During the manufacture of such devices, equipment, modules and systems (e.g., in integrated circuits), a trade-off can be made between performance improvements and physical implementations to improve manufacturing methods. For example, a trade-off can be made between performance improvements and layout area to match the performance of known embodiments, but using less silicon. For example, this can be accomplished by reusing functional blocks in a serial manner or sharing functional blocks between elements of a device, device, module and / or system. In contrast, the concepts set forth in this application that result in improvements to the physical implementations of devices, equipment, modules and systems (e.g., reduced silicon area) can be weighed against performance improvements. This can be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0308] The applicant hereby independently discloses each individual feature described herein, as well as any combination of two or more such features, to the extent that such feature or combination can be implemented in view of the common general knowledge of a person skilled in the art based on the present specification as a whole, regardless of whether such feature or combination of features solves any problem disclosed herein. In view of the foregoing description, it will be clear to a person skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A method of implementing a compressed attention-based neural network on hardware logic, wherein the compressed attention-based neural network comprises a compressed attention layer arranged to implement an attention function, the method comprising, at the compressed attention layer: Receives an embedding tensor; Rearranging and partitioning elements of the embedding tensor to form one or more embedding sub-matrices; applying a set of one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a bond matrix; applying a set of one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a query matrix; applying a set of one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a value matrix; as well as The attention function is implemented using the determined key matrix, the determined query matrix, and the determined value matrix.

2. The method of claim 1 , wherein the rearranging and splitting the elements of the embedding tensor comprises reordering the elements of the embedding tensor.

3. A method according to claim 1 or 2, wherein the rearrangement and partitioning of the elements of the embedding tensor matches the rearrangement and partitioning of the rows and columns of the query weight matrix, the key weight matrix and the value weight matrix, thereby being used to determine the set of the one or more key weight sub-matrices, the set of the one or more query weight sub-matrices and the set of the one or more value weight sub-matrices.

4. The method of any preceding claim, wherein the embedding tensor represents: (i) an input sequence, (ii) an output from an encoder layer in the compressed attention-based neural network, or (iii) an output from a decoder layer in the compressed attention-based neural network.

5. The method according to claim 4, wherein the method further comprises: Receive an input sequence; Determining the embedding tensor such that it represents the input sequence; as well as The embedding tensor is provided to the compressed attention layer.

6. A method according to any preceding claim, wherein the rearranging and splitting of the elements of the embedding tensor to form one or more embedding sub-matrices is performed by: (i) one or more aggregation layers of the compressed attention layer, or (ii) aggregation layers and splitting layers of the compressed attention layer.

7. A method according to any preceding claim, wherein the method further comprises applying an output matrix to the result of implementing the attention function before providing the output of the compressed attention layer, and wherein the ordering of the rows and columns of the output matrix is ​​complementary to the rearrangement of the elements of the embedding tensor.

8. A method according to any preceding claim, wherein said implementing said attention function comprises scaling a dot-product attention computation.

9. The method according to claim 8, wherein the attention function Attention(Q, K, V) is given by the following formula: Where Q is the query matrix, K T is the transpose of the key matrix, V is the value matrix, and d k is the dimension of the query matrix and key matrix.

10. The method of any preceding claim, wherein the compressed attention layer is configured to implement multi-head attention.

11. The method of claim 10, wherein the compressed attention layer is configured to implement multi-head attention by: Splitting the key matrix, the query matrix, and the value matrix into a plurality of headers; determining a head attention result for each of the heads; as well as The determined head attention results of each of the heads are concatenated to determine an attention result of the compressed attention layer.

12. The method of any preceding claim, wherein the compressed attention layer is: a self-attention layer within an encoder of the compressed attention-based neural network; a self-attention layer within a decoder of the compressed attention-based neural network; or An encoder-decoder attention layer within the decoder of the compressed attention-based neural network.

13. The method of any preceding claim, wherein the compressed attention layer is an encoder-decoder attention layer located within a decoder of the compressed attention-based neural network, wherein the key matrix and the value matrix are determined using a first embedding tensor representing an output from an encoder layer in the compressed attention-based neural network, and wherein the query matrix is ​​determined using a second embedding tensor representing an output from a previous layer in the decoder.

14. The method of any preceding claim, wherein the compressed attention-based neural network comprises a transformer network.

15. The method of any preceding claim, wherein the compressed attention-based neural network is a large language model.

16. The method of any preceding claim, wherein the compressed attention-based neural network is implemented to perform one of: natural language processing, language translation, computer vision processing, image processing, text processing, and speech processing.

17. Hardware logic configured to implement a compressed attention-based neural network, wherein the compressed attention-based neural network comprises a compressed attention layer arranged to implement an attention function, wherein the compressed attention layer is configured to: Receives an embedding tensor; Rearranging and partitioning elements of the embedding tensor to form one or more embedding sub-matrices; applying a set of one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a bond matrix; applying a set of one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a query matrix; applying a set of one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a value matrix; as well as The attention function is implemented using the determined key matrix, the determined query matrix, and the determined value matrix.

18. The hardware logic of claim 17, wherein the compressed attention layer comprises: a bond concatenation layer configured to concatenate results of applying the one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices to determine the bond matrix; a query concatenation layer configured to concatenate results of applying the one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices to determine the query matrix; as well as A value concatenation layer is configured to concatenate results of applying the one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices to determine the value matrix.

19. A compressed attention-based neural network comprising a compressed attention layer arranged to implement an attention function, wherein the compressed attention layer is configured to: Receives an embedding tensor; Rearranging and partitioning elements of the embedding tensor to form one or more embedding sub-matrices; applying a set of one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating the results of applying the one or more bond weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a bond matrix; applying a set of one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more query weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a query matrix; applying a set of one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices and concatenating results of applying the one or more value weight sub-matrices to the corresponding one or more embedding sub-matrices to determine a value matrix; as well as The attention function is implemented using the determined key matrix, the determined query matrix, and the determined value matrix.

20. A computer-readable storage medium having encoded thereon the compressed attention-based neural network of claim 19.

Images