Tensor processing

Optimizing the processing of Transformer model through grouping matrix multiplication and fill-free algorithm, the calculation and storage overhead problems caused by variable-length input sequences are solved, and efficient processing and acceleration of the BERT Transformer model is achieved.

CN118917370APending Publication Date: 2024-11-08DOUYIN VISION CO LTD +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410968236.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-07-18
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing natural language processing (NLP) pre-trained model based on Transformer model consumes a large amount of parameter space and computing resources during training and inference, resulting in increased costs and difficulty in effectively processing variable-length input sequences, resulting in useless markup calculations and Storage overhead.

Method used

By dividing the pending input tensor into multiple matrices, and leveraging grouping matrix multiplication (GEMM) and fill-free algorithms, avoiding useless computing, optimizing multi-head attention mechanism (MHA) and memory binding operations, using kernel fusion and grouping GEMM reduces global memory access and improves computing efficiency.

Benefits of technology

It significantly reduces the computation and storage overhead, improves the processing efficiency of the BERT Transformer model, and accelerates the computing speed of the single-layer BERT Transformer model, especially in the case of variable-length inputs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917370A_ABST
    Figure CN118917370A_ABST
Patent Text Reader

Abstract

A method for tensor processing is presented. A first tensor to which attention is to be applied is obtained, the first tensor representing an input of a batch having a variable sequence length; the first tensor is divided into a plurality of matrices based on the number of inputs of the batch and the number of heads of attention, where a matrix of the plurality of matrices has a dimension corresponding to a sequence length of the inputs of the batch and a dimension corresponding to a size of the heads of attention; the attention generates a second tensor by being applied to the plurality of matrices based on a grouped matrix multiplication, respectively, where a result matrix of the grouped matrix multiplication includes a plurality of blocks, each block being computed by a set of threads having a shared memory head by a set of threads, thus avoiding computing of useless flags.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to tensor processing, and more particularly, to methods, apparatus, and computer program products for tensor processing. Background Art

[0002] Currently, Transformer models have been widely used in various deep learning (DL) applications, such as language modeling, neural machine translation, recommendation systems, etc. The past decade has witnessed the rapid development of natural language processing (NLP) pre-training models based on Transformer models, which has greatly accelerated the progress of NLP. Among all Transformer-based pre-training models, Bidirectional Encoder Representations from Transformer (BERT) outperformed a large number of NLP tasks when it was created. However, BERT-like models consume increasingly larger parameter space and correspondingly more computing resources. This increases the cost of training and inference of BERT-like models. In this case, how to accelerate these models has become a hot topic. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for tensor processing is provided. In the method, a first tensor to which attention is to be applied is obtained, the first tensor representing a batch of inputs with a variable sequence length; based on the number of inputs of the batch and the number of heads of attention, the first tensor is divided into multiple matrices, wherein the matrices in the multiple matrices have dimensions corresponding to the sequence length of the inputs of the batch and dimensions corresponding to the size of the head of attention; attention is applied to multiple matrices respectively based on grouped matrix multiplication to generate a second tensor, wherein the result matrix of the grouped matrix multiplication includes multiple blocks, each block is calculated head by head by a group of threads with a shared memory. Therefore, the calculation of useless marks is avoided.

[0004] In a second aspect of the present disclosure, an electronic device is provided, comprising: a computer processor coupled to a computer readable memory unit, the memory unit comprising instructions, which implement the method according to the first aspect of the present disclosure when executed by the computer processor.

[0005] In a third aspect of the present disclosure, a computer program product is provided, which includes a computer-readable storage medium having program instructions embodied therein, and the program instructions can be executed by an electronic device to enable the electronic device to perform the method according to the first aspect of the present disclosure.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of some embodiments of the present disclosure in the accompanying drawings, in which like reference numerals generally refer to like parts in some embodiments of the present disclosure.

[0008] Figure 1 A schematic diagram showing an example architecture of a Transformer;

[0009] Figure 2A A schematic diagram showing the basic architecture of a Transformer encoder according to some embodiments of the present disclosure;

[0010] Figure 2B A schematic diagram of a Transformer encoder architecture with kernel fusion optimization according to some embodiments of the present disclosure is shown;

[0011] Figure 2C A schematic diagram of a Transformer encoder architecture with kernel fusion and no padding optimization according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of a no-fill algorithm according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram showing a grouped general matrix multiplication (GEMM) demonstration according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic diagram of a group GEMM-based fused multi-head attention (MHA) according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A schematic diagram of Warp prefetching for grouped GEMM according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A schematic diagram showing a fused softmax reduction at the end of a grouped GEMM according to some embodiments of the present disclosure;

[0017] Figure 8 An example flow chart showing a method for tensor processing according to some embodiments of the present disclosure; and

[0018] Fig. 9 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0019] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these implementations are only for illustrative purposes, and contribute to those skilled in the art to understand and implement the present disclosure, without implying any limitation to the scope of the present disclosure. The disclosure described herein can be implemented in various ways different from the modes described below.

[0020] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0021] References in this disclosure to "one implementation," "implementation," "example implementation," etc. indicate that the implementation being described may include a particular feature, structure, or characteristic, but not every implementation necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example implementation, it is submitted that it is within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other implementations, whether or not explicitly described.

[0022] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the example implementation, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0023] The terms used herein are used only for the purpose of describing a particular implementation and are not intended to limit the example implementations. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "comprises", "including", "having", "having", "including", and / or "comprising" when used herein specify the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0024] The principles of the present disclosure will now be described with reference to some embodiments. It should be understood that these implementations are only for illustrative purposes and are helpful for those skilled in the art to understand and implement the present disclosure without implying any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various ways different from the ways described below. In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present disclosure belongs.

[0025] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant rules.

[0026] It is understandable that before using the technical solution disclosed in the embodiments of the present invention, the user should be notified of the type, scope of use and usage scenarios of the personal information involved in the present invention in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0027] For example, in response to receiving an activity request from a user, a prompt message is sent to the user to clearly inform the user that the requested operation will require obtaining and using the user's personal information. Therefore, the user can independently choose whether to provide personal information to the electronic device, application, server or storage medium or other software or hardware that performs the operation of the technical solution of the present application according to the prompt message.

[0028] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending the prompt information to the user may include, for example, a pop-up window, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not limit the implementation of the present disclosure. Other methods that meet relevant laws and regulations are also applicable to the implementation of the present disclosure.

[0030] As used herein, the term "kernel" refers to machine executable code that performs one or more operations when executed. For example, a central processing unit (CPU) can launch a kernel to a graphics processing unit (GPU).

[0031] As used herein, if two or more operations are fused, it means that the operations can be performed by the same kernel without accessing global memory more than once. A kernel for performing fused operations may be referred to as a fused kernel.

[0032] As used herein, the term "epilogue" may refer to the ending portion of a kernel, such as a fusion kernel. The epilogue may be used for post-processing. The term "prologue" may refer to the beginning portion of a kernel, such as a fusion kernel. The prologue may be used for pre-processing.

[0033] In the present disclosure, hidden_dim represents the size of the implicit dimension, head_num represents the number of heads of attention (also known as head number) or attention mechanism, and head_size represents the size of each head implicit dimension, also known as head size. In addition, hidden_dim is equal to head_num×head_size. The term "batch size" or batch_size represents the number of inputs in the input of a batch. The term "sequence length" or seq_len represents the length of the input sequence. For example, if the input is a sentence with 5 words, the sequence length is 5.

[0034] As used herein, the term "thread block" represented by CTA refers to a group of threads (e.g., a group of threads of a GPU) with shared memory. Therefore, threads in a thread block can access data or instructions in shared memory without accessing global memory. The term "thread warp" or "warp" can refer to a subset of threads in a thread block. For example, a warp can include 32 threads. Threads in a warp can perform inter-thread communication within the thread, and therefore can efficiently communicate with other threads during execution.

[0035] Figure 1 A schematic diagram of the encoder-decoder model architecture of the Transformer is shown. The Transformer includes a stack of multiple encoding layers and decoder layers. The input tensor is processed in the input embedding layer 110. After performing positional encoding at layer 120, it is sent to the encoder, where block 130 is repeated N times. Block 130 includes a multi-head attention layer 131, followed by a feed-forward network (FFN) layer 133. Layer normalization operations such as add bias and normalization (add&norm) layers 132 and add&norm layers 133 are applied after both the MHA layer 131 and the FFN layer 133.

[0036] exist Figure 1, the right-shifted output tensor is processed in the output embedding layer 140. After performing positional encoding at layer 150, it is sent to the decoder, where block 160 is repeated N times. Block 160 includes two sets of consecutive MHA layers (such as masked multi-head attention layer 161 and multi-head attention layer 163) and one FFN layer 165, and each operation is normalized at add&norm layer 162, add&norm layer 164, and add&norm layer 166. Then, the probability is output via linear layer 170 and softmax layer 180.

[0037] although Figure 1 Both an encoder and a decoder for a Transformer are shown in FIG, but a Transformer-based model may include either the encoder or the decoder, or may include part of the encoder or the decoder.

[0038] Self-attention is a key module of the Transformer architecture. Conceptually, self-attention computes the importance of each position of the input sequence, where information from other positions is taken into account. Self-attention receives three input tensors: query (Q), key (K), and value (V). Self-attention can be divided into multiple heads. The Q and K tensors are first multiplied (also called 1 st GEMM) to compute the dot product of the query for all keys. This dot product is then represented by the implicit dimension d k Scaled and passed through a softmax operation to compute the weights corresponding to the value tensor. Each head of the output tensor is then passed through the tensor V (also called 2 nd GEMM) are connected by multiplication. Self-attention can be expressed as the following mathematical formula: Among them, the multi-head attention formula is: Multihead(Q,K,V)=Concat(head i ,…,head h ), where Head i =Attention(Q i ,K i ,V i ).

[0039] In the present invention, some of the following embodiments are described with respect to an encoder or a portion thereof, which can be extended to other models including a decoder. In addition, although some embodiments are described with respect to MHA, it should be understood that the tensor processing solution of the present disclosure is applicable to a single head attention. Operation fusion and no padding

[0040] To better understand the tensor processing solution of the present disclosure, some examples based on algorithm and kernel level optimization are now described. Figure 2A A schematic diagram showing the basic architecture of the Transformer encoder. Figure 2A and the subsequent Figure 2B and 2C In , the standard BERT Transformer encoder is shown for illustration purposes without any limitation. The tensor processing scheme of the present application can be applied to any suitable structure with attention, especially self-attention.

[0041] The input tensor is first processed through the pipeline by multiplying it by the built-in attribute matrix for Q, K, and V position encoding. This operation can be implemented using three separate GEMM operations or in batch mode. Since the attribute matrices corresponding to Q, K, and V have the same shape (which is represented as hidden_dim×hidden_dim), they are packed into continuous memory space and a single batched GEMM kernel is launched, which calculates Q, K, and V to reduce kernel launch overhead at runtime. The bias matrices for Q, K, and V are then added to the encoded tensor after passing through the self-attention module.

[0042] In some embodiments of the present disclosure, attention is part of a Transformer-based model. The following uses the multi-head attention mechanism as an example to illustrate BERT Transformer.

[0043] exist Figure 2A In Figure 2, the BERT Transformer encoder includes an MHA module, a projection module, a forward feed network, and a layer normalization module. The MHA module 206 includes two batched GEMM layers and a softmax layer. The encoder pipeline can be represented as a series of mathematical operations, including several GEMMs (shown as rectangular blocks) and other memory-bound operations (shown as parallelogram blocks).

[0044] Specifically, at block 202, the input tensors (Q, K, V) are computed in GEMM #0. At block 204, biases (Q, K, V) are added. At block 208, a batched GEMM is performed for the Q and K tensors. At block 210, a softmax operation is used to compute weights corresponding to the value tensor. At block 212, a batched GEMM is performed for the QxK and V tensors. At block 214, the output tensors are transposed, and at block 216, GEMM #1 is performed.

[0045] In addition, at box 220, bias is added to the tensor. At box 222, layer normalization is performed. Then, GEMM #2 is performed at box 226, and bias addition and activation are performed at box 228. At box 230, GEMM #3 is performed. Subsequently, bias is added at box 234, and layer normalization is performed at box 236.

[0046] exist Figure 2A In , the MHA module 206 is the most time-consuming part of the Transformer, for example, almost half or more than half of the total execution time. In order to exploit hardware efficiency, some solutions adopt a batching strategy in which multiple batches are executed simultaneously. Since batch execution requires the same shape of tasks in different batches, fixed-length inputs are assumed when designing the software. However, this assumption cannot always hold true because Transformer models often face the problem of variable-length inputs. In order to directly deploy models with variable-length inputs to conventional frameworks that only support fixed-length models, a direct solution is to pad all sequences to the maximum sequence length with zeros. However, this immediately leads to redundant computation of wasted padding tokens. These padded zeros also introduce significant memory overhead, which prevents large Transformer models from being effectively deployed.

[0047] To lift the restriction on fixed sequence lengths, some solutions provide explicit support for models with variable sequence lengths. Sequences with similar lengths are grouped before launching the batch kernel to minimize padding overhead. However, this active grouping approach still introduces non-removable padding overhead when the grouped and padded sequences have similar but different lengths.

[0048] In contrast to the training process, which is computed offline, the inference phase of the serving system must be processed online with low latency. Efficient solutions for NLP models require sophisticated kernel-level optimizations and explicit end-to-end designs to avoid wasted computation on zero tokens when processing variable-length inputs. However, existing solutions do not meet these expectations.

[0049] Modules containing memory-bound operations, such as feed-forward networks (with layer normalization) and bias addition followed by element-wise activation, can be optimized. In some embodiments, these memory-bound operations can be improved by fusing different kernels and reusing data in registers to reduce global memory accesses.

[0050] Figure 2B A schematic diagram of a Transformer encoder architecture with kernel fusion optimization according to some embodiments of the present disclosure is shown. In this example, memory bound operations are optimized by kernel fusion. For example, layer normalization and activation are fused with their respective continuous kernels.

[0051] exist Figure 2B In, with Figure 2A Compared to the example embodiment of , box 218 is replaced by box 238, and box 232 is replaced by box 244. That is, these tensors are subjected to bias addition and layer normalization of two-layer fusion.

[0052] BERT Transformer consists of a series of GEMM and memory-bound operations. The result tensor needs to be added by the input tensor and normalized after the projection and feed-forward networks of BERT Transformer. Instead of launching two separate kernels, these operations can be fused into a single kernel and reuse data at the register level. In addition to kernel fusion, the computational throughput of layer normalization is increased by assigning more workload to each thread.

[0053] Less fusion Figure 2A to Figure 2B , box 224 is replaced by box 240. That is, GEMM is fused with bias addition and activation. In this way, a fused kernel is provided to reduce global memory access. In contrast, an unfused implementation would call GEMM, store the output to global memory, and then load the result matrix from global memory for further element-by-element operations. By fusing the operations into a single kernel, the result matrix of the GEMM is kept in registers, and thus those fused element-by-element operations are performed by reusing data at the register level. Once the element-by-element transformation (e.g., adding bias and activation) is completed, the result is stored to global memory.

[0054] At blocks 238 and 244, the fused bias addition and normalization are performed. After MHA, the result tensor (which has a representation as valid_word cnt ×hidden_dim) is added to the input tensor (biases) and layer normalization is performed.

[0055] In a standard BERT configuration, for example, the head number and head size can be fixed to 12 and 64. The original implementation introduces two rounds of memory access to load and store tensors. In some embodiments, a fused kernel can be provided that only needs to access global memory in one round to complete both layer normalization and bias addition. Kernel fusion for this sub-kernel can improve the performance of the Transformer encoder.

[0056] At block 240, a fused GEMM with bias addition and activation is performed. After projection via matrix multiplication, the result tensor is added to the input tensor and element-wise activation is performed. Instead of storing the GEMM output to global memory and loading it again to add bias and activation, the fused implementation reuses the GEMM result matrix at the register level by implementing a custom and fused end.

[0057] Because the real-time serving process receives sentences with various words as input tensors, the sequence length of the input sentences can vary between batches even within a batch. For such input tensors consisting of sentences with variable lengths, the conventional solution is to pad them to the maximum sequence length with useless tokens, which results in significant computational and memory overhead.

[0058] To address this issue, in some embodiments, a no-padding algorithm may be used to pack the input tensor and store positioning information for other Transformer operations to index the original sequence.

[0059] Reference now Figure 3 , which shows a schematic diagram of a no-filling algorithm according to some embodiments of the present disclosure. Figure 3 The example embodiment of presents an example of a no-padding algorithm. Take the original input tensor 320 of 3 sentences (batch size 3) as an example. The longest sentence contains 5 word tokens, while the other two have 2 and 4 words. The height of the sample input tensor is 3, which is equal to the implicit dimension.

[0060] If a conventional method is used to pad all sentences to a maximum sequence length by padding zeros, a padded input tensor 310 will be generated. The elements 1 or 0 of the mask matrix 330 correspond to valid tokens or padding tokens of the padded input tensor 330 with a variable sequence length, respectively. By calculating the prefix sum of the mask matrix, the padded tokens are skipped, and the position indexes of all valid tokens are provided in the position offset information 350. An efficient kernel can be implemented to calculate the prefix sum and the position offset information 350.

[0061] As shown in thread map 340, each warp computes the prefix sum of the tokens of the entire sentence, so there are a total of batch_size warps allocated in each thread block for prefix sum calculations. Once the prefix sum is calculated, the input tensor 320 can be packed into a continuous memory area so that a packed input tensor 370 is generated and the total number of words used in future calculations is reduced from seq_len×batch_size to the actual effective word count of the packed tensor.

[0062] References Figure 3 A no-padding algorithm is described. It should be noted that the batch sizes and sequence lengths shown are examples and not any limitation on the scope of protection.

[0063] By utilizing the no-padding algorithm of the present disclosure, the processing of the Transformer can be enhanced without introducing useless calculations on the padding elements. Figure 2C A schematic diagram of the Transformer encoder architecture with kernel fusion and no padding optimization is shown. Figure 2CThe example in

[15] presents a detailed modification of BERT by introducing a no-padding algorithm.

[0064] Before performing the positional encoding, the prefix sum of the mask matrix may be calculated to pack the input tensor so that the calculation of useless marks is avoided in the first GEMM at block 248. Specifically, at block 246, the prefix sum is calculated and no padding is performed. At block 248, the packed input tensor (Q, K, V) is calculated in one GEMM.

[0065] exist Figure 2C In the example of , since batch GEMM in MHA requires the same problem shape between different inputs, the tensor is unpacked before entering the attention module. Once MHA is completed, the output tensor is packed again so that all remaining operations can benefit from the padding-free algorithm. It should be noted that padding and unpadding operations are fused with memory-bound footprints, such as adding bias and transposing, to minimize the overhead caused by this feature.

[0066] like Figure 2C As shown, at block 250, fused reconstruction padding and bias addition is performed to unpack the tensor to which MHA is to be applied. As described above, self-attention is applied to the unpacked tensor at the MHA module 206.

[0067] At block 252, the output tensor of the MHA module 206 is fused without padding and transposed. At block 254, a GEMM is performed on the packed tensor without padding. Furthermore, at block 256, a fused GEMM with bias addition and layer normalization is performed on the packed tensor without padding. At block 258, a fused GEMM with bias addition and activation is performed on the packed tensor without padding. At block 260, a GEMM is performed on the packed tensor without padding. At block 262, a fused GEMM with bias addition and layer normalization is performed on the packed tensor without padding.

[0068] The no-padding algorithm according to some embodiments of the present disclosure is designed to ensure semantic preservation. An array of valid labeled mapping relationships between the original tensor and the encapsulated tensor is maintained. The Transformer operates on the encapsulated tensor, and intermediate operations (e.g., MHA, layer normalization, and activation) refer to the position array to ensure correctness. At the end of each layer, the output tensor is reconstructed based on the position array so that the entire pipeline is semantically preserved. Table 1

[0069] Table 1 shows the number of calculations required for variable length input, where averagesequencelength=α×maximum represents the implicit dimension, m represents batch_size×max_seq_len, k represents the implicit dimension head_num×head_size, and bs represents the batch size.

[0070] Table 1 counts floating point computations for a single-layer BERT Transformer. Computations for memory-bound operations are not included as they are negligible compared to the listed modules. Enabling the no-padding algorithm eliminates redundant computations for all compute-intensive modules except MHA due to the limitations of batched GEMMs. Enabling the no-padding algorithm further speeds up BERT Transformer by 24.7% when the average sequence length is equal to 60% of the maximum. Optimizing multi-head attention

[0071] exist Figure 2C In the example of , the no-padding algorithm effectively reduces the wasted computation for variable-length inputs. However, the GEMM operation in MHA still cannot benefit from the no-padding algorithm. In other words, the GEMM operation in MHA has redundant computations, which will become more and more significant when the sequence length increases, as shown in Table 1. The complexity of MHA is quadratic in the sequence length, while the complexity of all other GEMMs is linear in the sequence length. In view of this, it is necessary to provide a high-performance MHA while maintaining the benefits of the no-padding algorithm to avoid redundant computations on useless tokens.

[0072] To this end, MHA can be optimized to avoid useless calculations. Some example embodiments of the optimization of MHA are described below. In such an embodiment, the tensor MHA to be applied is also referred to as an input tensor. The input tensor may represent the input of a batch with a variable sequence length. Another tensor output after MHA is also referred to as an output tensor or a target tensor. The input tensor and the output tensor may have dimensions of batch_size×seq_len×hidden_dim, respectively, which are equal to the size of batch_size×seq_len×head_num×head_size. The input tensor may be transposed, and the transposed input tensor may have dimensions of batch_size×head_num×seq_len×head_size. Therefore, based on the batch size of the input tensor and the head number of the MHA, the input tensor may be divided into multiple matrices, specifically matrices with batch_size×head_num. The matrix in the multiple matrices has a dimension of seq_len×head_size. As a result, the computation in MHA (expressed as P=Q×K) can be divided into the problem of batch_size×head_num GEMMs. Similarly, the computation in MHA can also be divided into the problem of batch_size×head_num GEMMs.

[0073] In some embodiments, if the maximum sequence length of the input tensor is below a threshold, attention can be applied to the input tensor by using a single kernel. GEMM for Q×K, GEMM for P×V, and softmax operations can be fused into this kernel, which can be referred to as an MHA computation kernel. Thus, application attention to a matrix among multiple matrices can be performed using a shared memory associated with a thread allocated for another matrix.

[0074] For example, unpadded fused MHA can be used for short input sequences. For short input sequences, the intermediate matrix P is kept in shared memory and registers throughout the MHA computation kernel to completely eliminate the quadratic memory overhead. Figure 3 The positioning information obtained in the above method is used to access the Q, K, and V tensors to avoid redundant calculation of zero padding for the MHA module.

[0075] For example, a 3D grid map can be launched for a fused MHA for short sequence launches: {head_num, seq_len / spkit_seq_len, batch_size}. Here, split_seq_len is a user-defined parameter that determines the size of the sequence block that precedes the thread block. The thread warp count of a thread block is represented by: max_seq_len: split_seq_len / 16×(max_seq_len / 16). Each thread block will have Q(max_seq_len×head_size), K(head_size×max_seq_len), and V The blocks of Q, K, V are loaded into shared memory and the MHA of the blocks of the result tensor is calculated. Three shared memory buffers are allocated to hold the Q, K, V sub-matrices. Due to the algorithmic nature of MHA, K and V blocks can be reused in the same shared memory buffer. The intermediate matrices of MHA can be kept and reused in another pre-allocated shared memory buffer.

[0076] The workflow of the fused MHA for short sequences is direct and efficient. For example, each thread can first load its own Q and K blocks into shared memory and calculate GEMM for P=Q×K. Element-by-element addition bias and scaling operations can be fused with the loading process to implicitly store memory latency. GEMM is calculated using any suitable interface of a processing device (e.g., a GPU) to take advantage of the tensor cores of the processing device. The intermediate matrix P can be kept in shared memory during reduction. Because this fused MHA is designed for short sequences, each thread can load the entire sequence P from shared memory into a register file for both reduction and element-by-element exponential transformation in the softmax operation. Once the softmax operation is completed, the block P is loaded into shared memory to calculate the second GEMM=P×V, and the resulting tensor O is then stored in global memory.

[0077] Considering the limited resources of register files and shared memory, fused MHA for short sentences may no longer be feasible for long sequences exceeding a length threshold. Therefore, in some embodiments of the present disclosure, grouped matrix multiplication (e.g., grouped GEMM) can be used to apply MHA to input tensors with long sequences. The result matrix of the grouped matrix multiplication can include multiple blocks, each block being calculated by a group of threads with shared memory.

[0078] To better understand the present disclosure, grouped GEMM is now briefly introduced as an example of grouped matrix multiplication. Unlike batched GEMM, where all GEMM subproblems are required to have the same shape, grouped GEMM allows for arbitrary shapes of subproblems. This is achieved by a built-in scheduler that iterates over all GEMM subproblems in a cyclic manner.

[0079] Figure 4 An example operation of a grouped GEMM using an example with 3 subproblems is shown. Figure 4 Three result matrices 410, 420, and 430 of the GEMM are shown, each of which is divided according to the tile size of the thread block. Assume that 3 thread blocks (including CTA#0, CTA#1, and CTA#2) are started, and each CTA calculates a fixed-size CTA tile at each step until all GEMM subproblems are covered. Logically, the GPU calculates in waves. In the first wave, all three CTAs calculate 3 tiles (in Figure 4 ). Then in the second CTA wave, CTA#0 moves to the lower right block of GEMM 1, while CTA#1 and CTA#2 move to the subproblem of GEMM 1. In the final CTA wave, CTA#0 and CTA#1 continue to compute tasks in GEMM1 and GEMM 2, while CTA#2 remains idle because there are no more free blocks in the computation graph.

[0080] Since the grouped GEMM lifting restricts the shape of the subproblems, it can directly benefit from MHA problems with variable length inputs. The total number of MHA problems is equal to batch_size×head_num. MHA problems between different batches can have different sequence lengths, while the sequence lengths within the same batch can be the same. The grouped GEMM scheduler iterates over all attention units in a loop. In each attention unit, the GEMM P is first calculated. i =Q i ×K i , and can be used for P i Perform softmax operation. Second GEMMO i =P i ×V i The final attention result is provided. Here i indicates the problem of the i-th group MHA with variable shape.

[0081] When using grouped GEMM to implement attention, the number of matrix multiplication problems batch_size×head_num can be large. Since from a thread perspective, each thread needs to traverse and read the size information of all problems, reading the size information of the problem causes significant overhead.

[0082] In some embodiments, some problem size information can be shared. For example, size information can be shared between matrices in multiple matrices corresponding to the input of a batch of inputs (to be calculated by the grouped GEMM). For the same input, the sequence lengths of different heads are the same, and the problem size is also the same. By sharing, parameter storage is reduced from batch_size×head_num to batch_size.

[0083] On the other hand, grouped GEMM frequently checks with the built-in scheduler on the current task allocation, which causes runtime overhead. To address this issue, the grouped GEMM scheduler can be optimized.

[0084] To this end, in some embodiments of the present disclosure, parameters of a matrix multiplication problem, or in other words information about the size of a matrix to be calculated, may be read by one thread of a warp and shared with another thread of the warp through inter-thread communication.

[0085] Figure 6 An optimization of the original packet GEMM scheduler is shown. Figure 6 In the example shown, the thread warp includes 32 threads from T0, T1 to T30 and T31, and there are 63 problems. In iteration #0, each thread from T0, T1 to T30 and T31 can read the size information of the problems of 63 problems, and can share the size information with the other 31 threads. Therefore, the size information of the first 32 problems can be read. In iteration #1, each thread from T0, T1... to T30 and T31 can read the size information of the problems of multiple problems (it should be noted that thread T31 can read the dummy size information) and can share the size information with the other 31 threads. It can be seen that instead of requiring one thread to calculate the current task metadata, there are 32 threads in the thread warp to calculate the block index for one access. Therefore, the scheduler access overhead can be reduced by 32.

[0086] To further improve performance, the softmax operation after Q x K can be fused with the matrix multiplication, which saves memory access operations for the intermediate matrices compared to a separate kernel for the softmax operation.

[0087] Because the softmax operation needs to reduce the entire row of data, but due to the size limit of the shared memory, the thread block cannot process the entire row of data, and the communication between thread blocks is inefficient, the entire softmax operation cannot be completed within the end for calculating Q x K. In some embodiments, the softmax operation can be divided into three steps. The first step can be integrated into the end of Q x K, the third step can be integrated into the prologue of P x V, and a lightweight kernel can be added in the middle to implement the second step.

[0088] Reference now Figure 5 In a first step 511, a partial reduction may be performed. For example, the intermediate matrix Pi may be divided into M×N intermediate blocks, such as intermediate blocks 502, 504, 506, and 508. In this example, M=4 and N=4. For an intermediate block in the plurality of intermediate blocks, a partial reduction result of a softmax operation along the rows of the intermediate block may be determined. The partial reduction result may include a maximum value along the rows of the intermediate block. In the example, max j =max(x0,…,x n ), where j represents the block along the row direction of Pi, and x t (t=0, ... n) represents the value of the element with column index t in the row. The partial reduction result may include the sum of the exponential differences between the values ​​of the rows along the middle block and the maximum value. For example, For example, it can be calculated at the end for Q x K.

[0089] In a second step 512, a full reduction may be performed, for example, by using a separate kernel. Specifically, a full reduction result of the softmax operation along the rows of the intermediate matrix may be determined based on the corresponding partial reduction results determined for the plurality of intermediate partitions. For example, max = max(max0, ..., max N ),as well as

[0090] At the third step 513, the fused element-wise operation can be performed in a single kernel. Specifically, the intermediate matrix Pi is transformed by the kernel based on the full reduction result. For example Then, a group matrix multiplication can be performed on the transformed intermediate matrix Pi and the values ​​of the matrix. Figure 5 As shown, the transformed Pi is multiplexed with the value matrix Vi to obtain the matrix Oi as the result of MHA.

[0091] Taking grouped GEMM as an example, the memory footprint of the softmax operation can be fused into two grouped GEMMs of MHA. Figure 7 An example of tail fusion for softmax reduction is shown. The result matrix 710 of the grouped GEMM has dimensions M×N. The CTA computes M C ×N C Submatrix 720. For example, M C and N C Both can be set to 128 to maximize the performance of grouped GEMM. In this case, according to the thread mapping assignment, each CTA has 128 threads, and the thread mapping is arranged as 8×16, where each thread holds a 128-bit register block in each step. After intra-thread reduction, M R×N C The (8x128) sub-matrix 730 is reduced to an 8x16 matrix 740, where the result of one reduction is held by one thread.

[0092] An intra-thread reduction is performed to further reduce the 8×16 matrix 740 from the column dimension to a vector 750, which can be implemented via a warp reordering for efficiency. A similar reduction (within-thread reductions performed later) can be performed to calculate the maximum and sum in the trailing portion. Once both max and sum are reduced, they are stored to global memory.

[0093] In these embodiments, MHA is optimized by fusing the softmax operation into the GEMM without computing useless padding tokens under variable length inputs. For short sequences, intermediate matrices are kept in registers and shared memory. For long sequences, a group-based GEMM-based fused MHA is adopted, and the softmax operation is fused into a customized GEMM tail to implicitly save memory latency. In both implementations, the input matrix is ​​accessed based on position information obtained from the no-padding algorithm so that no redundant calculations are introduced.

[0094] Since cross-threadblock communication is impractical under current GPU programming models, the specification within a threadblock only provides partial specifications. Therefore, it is necessary to launch separate lightweight kernels, such as Figure 5 As shown in the figure, full reduction is performed. In partial reduction, the target tensor of each attention unit is sq_len×seq_len, while full reduction only reduces seq_len×seq_len / 128. Therefore, the workload of full reduction is negligible to that of partial reduction. In fact, the full reduction kernel only accounts for about 2% of the total execution time in fused MHA.

[0095] Once the fully reduced max and sum vectors have been obtained, we are ready to perform an element-wise transformation of the output matrix of the first GEMM For implicitness and memory latency, these element-wise operations are fused into the main loop of the second GEMM.

[0096] The baseline MHA is a computation chain consisting of a batch of GEMMs, a softmax, and another batch of GEMMs. The time and memory complexity of all these operations is quadratic in the sequence length. Since the no-padding algorithm directly reduces the effective sequence length, MHA with variable-length input also obtains direct improvements.

[0097] The fused MHA according to an embodiment of the present disclosure, which is explicitly designed to handle both short and long sequences, combines a no-padding algorithm to alleviate the memory overhead of the intermediate matrices in the MHA caused by padding of variable-length inputs. For example, a highly optimized MHA can speed up a single-layer BERT Transformer by 19% compared to the previous step. As a result, the fully optimized version exceeds Figure 2A The baseline implementation in is 60%, and the rest of the operations in the forward BERT Transformer are near-optimal GEMM operations.

[0098] In view of this, the present invention proposes a tensor processing scheme, in particular, a scheme for applying attention. The solution is optimized for variable length sequences and therefore has high performance. The optimized solution not only brings algorithm-level innovations that free the attention mechanism from padding overhead, but also combines architecture-aware optimizations to accelerate the functional modules of the attention mechanism. The optimized fused MHA and other step-by-step optimizations together provide significant acceleration over the current state-of-the-art models based on the attention mechanism. In addition, although example embodiments are described with respect to BERT, this is only an example. The scheme is applicable to any model with an attention mechanism. In addition, although the above example embodiments are described with respect to NLP, the solution can be applied to any data processing, such as images or digital data.

[0099] The above paragraphs have described the details of tensor processing. According to some embodiments of the present disclosure, a method for tensor processing is provided. For more details about the method, please refer to Figure 8 . Figure 8 An example flow chart of a method 800 for tensor processing according to some embodiments of the present disclosure is shown.

[0100] At box 810, a first tensor to which attention is to be applied is obtained, the first tensor representing the input of a batch with a variable sequence length. At box 820, the first tensor is divided into a plurality of matrices based on the number of inputs of the batch and the number of heads of attention. A matrix in the plurality of matrices has a dimension corresponding to the sequence length of the input of the batch and a dimension corresponding to the size of the head of attention. At box 830, a second tensor is generated by applying attention to the plurality of matrices respectively based on a grouped matrix multiplication (e.g., a grouped GEMM), wherein the result matrix of the grouped matrix multiplication includes a plurality of blocks, each block being calculated by a group of threads having a shared memory.

[0101] In some embodiments of the present disclosure, applying attention to multiple matrices includes: for a matrix among the multiple matrices, performing grouped matrix multiplication on a query of the matrix (denoted as Q) and a key of the matrix (denoted as K) to generate an intermediate matrix, wherein the intermediate matrix includes multiple intermediate blocks, each intermediate block is calculated by a group of threads having a shared memory; based on the multiple intermediate blocks, determining a full reduction result of a softmax operation; transforming the intermediate matrix based on the full reduction result; and performing grouped matrix multiplication on the transformed intermediate matrix and a value of the matrix (denoted as V).

[0102] In some embodiments of the present disclosure, determining a complete reduction result of a softmax operation includes: determining, for an intermediate block of a plurality of intermediate blocks, a partial reduction result of a softmax operation along the rows of the intermediate block; and determining, based on the respective partial reduction results determined for the plurality of intermediate blocks, a complete reduction result of a softmax operation along the rows of the intermediate block.

[0103] In some embodiments of the present disclosure, the partial reduction result includes at least one of: a maximum value along a row of the intermediate block, or a sum of exponentials of differences between a value along a row of the intermediate block and a maximum value.

[0104] In some embodiments of the present disclosure, size information is shared between matrices in a plurality of matrices corresponding to inputs of a batch of inputs.

[0105] In some embodiments of the present disclosure, a group of threads includes multiple groups of threads, and threads in a group of threads in the multiple groups of threads are configured to have inter-thread communication within the group of threads, and size information about the matrix is ​​read by a thread in the group of threads and shared with another thread in the group of threads through inter-thread communication.

[0106] In some embodiments of the present disclosure, generating the second tensor by applying attention to a plurality of matrices, respectively, based on grouped matrix multiplication is in response to: an upper limit sequence length of a sequence length of the batch input exceeds a length threshold.

[0107] In some embodiments of the present disclosure, method 800 also includes: obtaining a third tensor to which attention is to be applied, the third tensor representing another batch of inputs having a variable sequence length; dividing the third tensor into a plurality of additional matrices based on the number of inputs of the other batch and the number of attention heads, wherein the additional matrices in the plurality of additional matrices have dimensions corresponding to the sequence length of the inputs of the other batch and dimensions corresponding to the size of the attention heads; and in response to an upper limit sequence length of the sequence length of the inputs of the other batch being lower than a length threshold, generating a fourth tensor by applying attention to the plurality of additional matrices, wherein the attention to the additional matrices is applied using a shared memory associated with the thread allocated for the additional matrices.

[0108] In some embodiments of the present disclosure, attention is part of a Transformer-based model.

[0109] According to some embodiments of the present disclosure, a device for tensor processing is provided. The device includes: an acquisition module, configured to acquire a first tensor to which attention is to be applied, the first tensor representing a batch of inputs with a variable sequence length; a division module, configured to divide the first tensor into multiple matrices based on the number of inputs of the batch and the number of heads of attention, wherein the matrices in the multiple matrices have dimensions corresponding to the sequence length of the inputs of the batch and dimensions corresponding to the size of the heads of attention; a generation module, configured to generate a second tensor by applying the attention to the multiple matrices respectively based on grouped matrix multiplication, wherein the result matrix of the grouped matrix multiplication includes multiple blocks, each block being calculated by a group of threads with a shared memory. Furthermore, the device may also include other units that implement other steps in the above method.

[0110] According to some embodiments of the present disclosure, an electronic device for implementing the above method is provided. The electronic device includes: a computer processor coupled to a computer-readable memory unit, the memory unit includes instructions, and the instructions implement a method for tensor processing when executed by the computer processor, including: obtaining a first tensor to which attention is to be applied, the first tensor representing a batch of inputs with a variable sequence length; based on the number of inputs of the batch and the number of heads of attention, dividing the first tensor into multiple matrices, wherein the matrices in the multiple matrices have dimensions corresponding to the sequence length of the inputs of the batch and dimensions corresponding to the size of the head of attention; generating a second tensor by applying attention to multiple matrices respectively based on grouped matrix multiplication, wherein the result matrix of the grouped matrix multiplication includes multiple blocks, each block being counted by a group of threads with a shared memory.

[0111] In some embodiments of the present disclosure, applying the attention to the multiple matrices includes: for the matrix among the multiple matrices, performing the grouped matrix multiplication on the query of the matrix and the key of the matrix to generate an intermediate matrix, wherein the intermediate matrix includes multiple intermediate blocks, each intermediate block is calculated by the group of threads having the shared memory; based on the multiple intermediate blocks, determining a complete reduction result of a softmax operation; transforming the intermediate matrix based on the complete reduction result; and performing the grouped matrix multiplication on the transformed intermediate matrix and the value of the matrix.

[0112] In some embodiments of the present disclosure, determining a complete reduction result of a softmax operation includes: determining, for an intermediate block of the multiple intermediate blocks, a partial reduction result of the softmax operation along the rows of the intermediate block; and determining, based on the respective partial reduction results determined for the multiple intermediate blocks, a complete reduction result of the softmax operation along the rows of the intermediate block.

[0113] In some embodiments of the present disclosure, the partial reduction result includes at least one of: a maximum value along a row of the intermediate block, or a sum of exponential differences between a value of the row along the intermediate block and the maximum value.

[0114] In some embodiments of the present disclosure, size information is shared between matrices in the plurality of matrices corresponding to inputs of the batch of inputs.

[0115] In some embodiments of the present disclosure, the group of threads includes multiple groups of threads, and the threads in a group of threads in the multiple groups of threads are configured to have inter-thread communication within the group of threads, and the size information about the matrix is ​​read by a thread in the group of threads and shared with another thread in the group of threads through the inter-thread communication.

[0116] In some embodiments of the present disclosure, generating the second tensor by applying the attention to the multiple matrices respectively based on the grouped matrix multiplication is in response to: an upper limit sequence length of a sequence length of the input of the batch exceeds a length threshold.

[0117] In some embodiments of the present disclosure, the method also includes: obtaining a third tensor to which attention is to be applied, the third tensor representing another batch of inputs having a variable sequence length; dividing the third tensor into a plurality of additional matrices based on the number of inputs of the other batch and the number of heads of the attention, wherein the additional matrices in the plurality of additional matrices have dimensions corresponding to the sequence length of the inputs of the other batch and dimensions corresponding to the size of the heads of the attention; and in response to an upper limit sequence length of the sequence length of the inputs of the other batch being lower than the length threshold, generating a fourth tensor by applying the attention to the plurality of additional matrices, wherein the attention to the additional matrices is applied using a shared memory associated with a thread allocated for the additional matrices.

[0118] In some embodiments of the present disclosure, the attention is part of a Transformer-based model.

[0119] According to some embodiments of the present disclosure, a computer program product is provided. The computer program product includes a computer-readable storage medium having program instructions embodied therein. The program instructions can be executed by an electronic device to enable the electronic device to perform the method 800 .

[0120] Fig. 9 900 in which various embodiments of the present disclosure may be implemented. Fig. 9 The computing device 900 shown is for illustration purposes only and does not in any way imply any limitation on the functionality and scope of the present disclosure. The computing device 900 may be used to implement the above method 800 in some embodiments of the present disclosure. Fig. 9 As shown, the computing device 900 may be a general computing device. The computing device 900 may include at least one or more processors or processing units 910, a memory 920, a storage unit 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960.

[0121] Processing unit 910 may be a physical or virtual processor and may implement various processes based on programs stored in memory 920. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 900. Processing unit 910 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0122] The computing device 900 typically includes various computer storage media. Such media can be any media accessible to the computing device 900, including but not limited to volatile and non-volatile media, or removable and inseparable media. The memory 920 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 930 can be any removable or inseparable medium, and can include a machine-readable medium, such as a memory, a flash drive, a disk, or another other medium that can be used to store information and / or data and can be accessed in the computing device 900.

[0123] The computing device 900 may also include additional removable / inseparable volatile / non-volatile storage media. Fig. 9, but a magnetic disk drive for reading from and / or writing to a removable and non-volatile magnetic disk and an optical disk drive for reading from and / or writing to a removable non-volatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more data medium interfaces.

[0124] The communication unit 940 communicates with another computing device via a communication medium. In addition, the functionality of the components in the computing device 900 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 900 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0125] Input device 950 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 960 may be one or more of various output devices, such as a display, speaker, printer, etc. With the aid of communication unit 940, computing device 900 may also communicate with one or more external devices (not shown) (such as storage devices and display devices), one or more of which enable a user to interact with computing device 900 or any device (such as a network card, modem, etc.) that enables computing device 900 to communicate with one or more other computing devices (if necessary). Such communication may be performed via an input / output (I / O) interface (not shown).

[0126] In some implementations, some or all components of the computing device 900 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (such as the Internet) using a suitable protocol. For example, a cloud computing provider provides an application via a wide area network that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a server at a remote location. The computing resources in a cloud computing environment may be merged or distributed at a location in a remote data center. Cloud computing infrastructure may provide services through a shared data center, although they appear as a single access point for users. Therefore, cloud computing architecture may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided from a conventional server or installed directly or otherwise installed on a client device.

[0127] The functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0128] The program code for executing the method of the subject matter described herein can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code enables the functions / operations specified in the flow chart and / or block diagram to be implemented when executed by the processor or controller. The program code can be executed in whole or in part on the machine, in part on the machine, in part on a remote machine, or in whole on a remote machine or server.

[0129] In the context of the present disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] In addition, although operations are shown in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or performing all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these details should not be interpreted as limitations on the scope of the subject matter described herein, but should be interpreted as descriptions of features that may be specific to a particular implementation. Certain features described in the context of a separate implementation may also be implemented in combination in a single implementation. On the contrary, the various features described in a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination.

[0131] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0132] It can be understood from the foregoing that the specific implementation of the presently disclosed technology has been described herein for illustrative purposes, but various modifications may be made without departing from the scope of the present disclosure. Therefore, the technology of the present disclosure is not limited to the appended claims.

[0133] The implementation of the subject matter and functional operations described in the present disclosure can be implemented in various systems, digital electronic circuits or computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The implementation of the subject matter described in this specification can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium for data processing devices to execute or control the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that implements a machine-readable propagation signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" includes all devices, equipment and machines for processing data, such as a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0134] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0135] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to receive data from or transfer data to or transfer data to one or more mass storage devices, which are used to store data, such as magnetic, magneto-optical disks or optical disks. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0136] The specification and drawings are to be regarded as exemplary only, in which the exemplary arrangements are examples.As used herein, the use of "or" is intended to include "and / or" unless the context clearly dictates otherwise.

[0137] Although the present disclosure contains many details, these details should not be interpreted as limitations on the scope of any disclosure or claimed content, but should be interpreted as descriptions of features that may be specific to a particular implementation of a particular disclosure. Certain features described in the present disclosure in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed, in some cases, one or more features from the claimed combination may be cut out from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0138] Similarly, although operations are shown in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In addition, the separation of various system components in the embodiments described in this disclosure should not be understood as requiring such separation in all embodiments. Only some implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this disclosure.

Claims

1. A method for tensor processing, comprising: Obtain a first tensor to which attention is applied, the first tensor representing an input of a batch having a variable sequence length; Based on the number of inputs of the batch and the number of attention heads, dividing the first tensor into a plurality of matrices, wherein a matrix in the plurality of matrices has a dimension corresponding to the sequence length of the inputs of the batch and a dimension corresponding to a size of the attention head; A second tensor is generated by applying the attention to the plurality of matrices, respectively, based on grouped matrix multiplication, wherein a result matrix of the grouped matrix multiplication includes a plurality of blocks, each block being calculated by a group of threads having a shared memory.

2. The method of claim 1 , wherein applying the attention to the plurality of matrices comprises: For the matrix in the plurality of matrices, performing the grouped matrix multiplication on the query of the matrix and the key of the matrix, to generate an intermediate matrix, wherein the intermediate matrix includes a plurality of intermediate blocks, each intermediate block being calculated by the group of threads having the shared memory; Determining a complete reduction result of a softmax operation based on the plurality of intermediate blocks; transforming the intermediate matrix based on the complete reduction result; The grouped matrix multiplication is performed on the transformed intermediate matrix and the values ​​of the matrix.

3. The method of claim 2, wherein determining a fully reduced result of a softmax operation comprises: For an intermediate block of the plurality of intermediate blocks, determining a partial reduction result of the softmax operation along a row of the intermediate block; as well as Based on the respective partial reduction results determined for the plurality of intermediate blocks, a full reduction result of the softmax operation along the rows of the intermediate blocks is determined.

4. The method according to claim 3, wherein the partial reduction result comprises at least one of: the maximum value along the rows of the middle tile, or The sum of the exponentials of the differences between the values ​​of the row along the intermediate partition and the maximum value. 5 . The method of claim 1 , wherein size information is shared between matrices in the plurality of matrices corresponding to inputs of the batch of inputs.

6. The method of claim 1 , wherein the group of threads comprises a plurality of groups of threads, and threads in a group of threads in the plurality of groups of threads are configured to have inter-thread communication within the group of threads, and The size information about the matrix is ​​read by a thread in the set of threads and shared with another thread in the set of threads through the inter-thread communication.

7. The method of claim 1 , wherein generating the second tensor by applying the attention to the plurality of matrices, respectively, based on the grouped matrix multiplication is in response to: An upper limit sequence length of the sequence length of the batch input exceeds a length threshold.

8. The method according to claim 7, further comprising: Get a third tensor to which attention will be applied, the third tensor representing another batch of inputs with variable sequence length; Based on the number of the further batch of inputs and the number of the attention heads, dividing the third tensor into a plurality of further matrices, wherein further matrices of the plurality of further matrices have a dimension corresponding to the sequence length of the further batch of inputs and a dimension corresponding to a size of the attention heads; as well as In response to an upper limit sequence length of sequence lengths of the another batch of inputs being lower than the length threshold, generating a fourth tensor by applying the attention to the plurality of additional matrices, wherein the attention to the additional matrices is applied using a shared memory associated with a thread allocated for the additional matrices.

9. The method of claim 1, wherein the attention is part of a Transformer-based model.

10. An electronic device comprising a computer processor coupled to a computer readable memory unit, the memory unit comprising instructions which, when executed by the computer processor, implement a method for tensor processing comprising: Obtain a first tensor to which attention is applied, the first tensor representing an input of a batch having a variable sequence length; Based on the number of inputs of the batch and the number of attention heads, dividing the first tensor into a plurality of matrices, wherein a matrix in the plurality of matrices has a dimension corresponding to the sequence length of the inputs of the batch and a dimension corresponding to a size of the attention head; A second tensor is generated by applying the attention to the plurality of matrices, respectively, based on grouped matrix multiplication, wherein a result matrix of the grouped matrix multiplication includes a plurality of blocks, each block being calculated by a group of threads having a shared memory.

11. The electronic device of claim 10, wherein applying the attention to the plurality of matrices comprises: For the matrix in the plurality of matrices, performing the grouped matrix multiplication on the query of the matrix and the key of the matrix, to generate an intermediate matrix, wherein the intermediate matrix includes a plurality of intermediate blocks, each intermediate block being calculated by the group of threads having the shared memory; Determining a complete reduction result of a softmax operation based on the plurality of intermediate blocks; transforming the intermediate matrix based on the complete reduction result; The grouped matrix multiplication is performed on the transformed intermediate matrix and the values ​​of the matrix.

12. The electronic device of claim 11, wherein determining a fully reduced result of the softmax operation comprises: For an intermediate block of the plurality of intermediate blocks, determining a partial reduction result of the softmax operation along a row of the intermediate block; as well as Based on the respective partial reduction results determined for the plurality of intermediate blocks, a full reduction result of the softmax operation along the rows of the intermediate blocks is determined.

13. The electronic device according to claim 12, wherein the partial reduction result comprises at least one of: the maximum value along the rows of the middle tile, or The sum of the exponentials of the differences between the values ​​of the row along the intermediate partition and the maximum value. 14 . The electronic device of claim 10 , wherein size information is shared between matrices in the plurality of matrices corresponding to inputs of the batch of inputs.

15. The electronic device of claim 10, wherein the group of threads comprises a plurality of groups of threads, and threads in a group of threads in the plurality of groups of threads are configured to have inter-thread communication within the group of threads, and The size information about the matrix is ​​read by a thread in the set of threads and shared with another thread in the set of threads through the inter-thread communication.

16. The electronic device of claim 10, wherein generating the second tensor by applying the attention to the plurality of matrices, respectively, based on the grouped matrix multiplication is in response to: An upper limit sequence length of the sequence length of the batch input exceeds a length threshold.

17. The electronic device according to claim 16, further comprising: Get a third tensor to which attention will be applied, the third tensor representing another batch of inputs with variable sequence length; Based on the number of the further batch of inputs and the number of the attention heads, dividing the third tensor into a plurality of further matrices, wherein further matrices of the plurality of further matrices have a dimension corresponding to the sequence length of the further batch of inputs and a dimension corresponding to a size of the attention heads; as well as In response to an upper limit sequence length of sequence lengths of the another batch of inputs being lower than the length threshold, generating a fourth tensor by applying the attention to the plurality of additional matrices, wherein the attention to the additional matrices is applied using a shared memory associated with a thread allocated for the additional matrices.

18. The electronic device of claim 10, wherein the attention is part of a Transformer-based model.

19. A computer program product, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic device to cause the electronic device to perform a method for tensor processing, the method comprising: Obtain a first tensor to which attention is applied, the first tensor representing an input of a batch having a variable sequence length; Based on the number of inputs of the batch and the number of attention heads, dividing the first tensor into a plurality of matrices, wherein a matrix in the plurality of matrices has a dimension corresponding to the sequence length of the inputs of the batch and a dimension corresponding to a size of the attention head; A second tensor is generated by applying the attention to the plurality of matrices, respectively, based on grouped matrix multiplication, wherein a result matrix of the grouped matrix multiplication includes a plurality of blocks, each block being calculated by a group of threads having a shared memory.

20. The computer program product of claim 19, wherein applying the attention to the plurality of matrices comprises: For the matrix in the plurality of matrices, performing the grouped matrix multiplication on the query of the matrix and the key of the matrix, to generate an intermediate matrix, wherein the intermediate matrix includes a plurality of intermediate blocks, each intermediate block being calculated by the group of threads having the shared memory; Determining a complete reduction result of a softmax operation based on the plurality of intermediate blocks; transforming the intermediate matrix based on the complete reduction result; The grouped matrix multiplication is performed on the transformed intermediate matrix and the values ​​of the matrix.

Citation Information

Patent Citations

  • Graphics processor data processing method, text processing method and devices, and equipment

    CN111475775A

  • Systems and methods for providing vector-wise sparsity in a neural network

    US20210065005A1

  • Object-Centric Learning with Slot Attention

    US20210383199A1

  • Method and apparatus for length-aware local tiling in a sparse attention module in a transformer

    US20230153381A1