Attention calculation method, device, equipment, storage medium and program product

Abstract

The application relates to an attention calculation method, device, equipment, storage medium and program product. The method comprises the following steps: dividing a query matrix, a key matrix and a value matrix determined based on input features into blocks to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; combining the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices; each group of sub-matrices comprises one query sub-matrix, one key sub-matrix and one value sub-matrix; in the process of attention calculation for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel manner, and the attention output of the two query sub-modules is obtained. The method can improve the attention calculation performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to an attention calculation method and device, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] In large model inference, the time consumption of the attention operator accounts for a large proportion in the Transfomer, and therefore, the optimization of attention performance is always the top priority.

[0003] In the related art, when calculating attention, the query matrix (Q), the key matrix (K) are first subjected to MMA calculation, then subjected to softmax, and finally the obtained result S and the value matrix (V) are subjected to MMA calculation. The three calculations have data dependency and are therefore executed in series. The MMA calculation of QK is completed, the softmax calculation is started, and finally the MMA calculation of SV is executed. The tcore (tensor calculation core) is used for the MMA calculation of QK and SV, and the vecore (vector calculation core) is used for the softmax calculation. Therefore, in the calculation of QK, the vecore is in an idle state, and similarly, in the processing of softmax, the tcore is in an idle state. Therefore, the serial execution method cannot achieve full-load operation of the two calculation cores, and the Attention operator cannot achieve optimal performance. SUMMARY

[0004] Therefore, it is necessary to provide an attention calculation method, device, computer device, computer readable storage medium and computer program product capable of improving the performance of attention calculation in view of the above technical problems.

[0005] In a first aspect, the present application provides an attention calculation method, comprising:

[0006] The query matrix, the key matrix and the value matrix determined based on the input feature are divided into blocks to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; the plurality of key sub-matrices and the plurality of value sub-matrices correspond one by one;

[0007] The plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices are combined to obtain a plurality of groups of sub-matrices; each group of sub-matrices includes one query sub-matrix, one key sub-matrix and one value sub-matrix, and at least one of the query sub-matrix and the key sub-matrix is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other;

[0008] In the attention calculation process for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, and the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel calculation mode to obtain the attention output of the two query sub-modules.

[0009] In one of the embodiments, the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel calculation mode to obtain the attention output of the two query sub-modules, including:

[0010] The two query sub-modules are determined as a first query sub-module and a second query sub-module.

[0011] After matrix multiplication of the first query sub-module and the key sub-matrix is performed to obtain a calculation result, the calculation result is subjected to softmax processing, and at the same time, matrix multiplication of the second query sub-module and the key sub-matrix is performed.

[0012] After the calculation result is subjected to softmax processing to obtain an attention weight matrix, matrix multiplication of the attention weight and the value sub-matrix is performed to obtain a first attention output, and at the same time, the calculation result of the matrix multiplication of the second query sub-module and the key sub-matrix is subjected to softmax processing, and based on the obtained attention weight, matrix multiplication of the value sub-matrix is performed to obtain a second attention output.

[0013] In one of the embodiments, the method further includes:

[0014] In the attention calculation process of any query sub-module, the matrix multiplication of the query sub-module and the key sub-matrix in the same group is performed by a tensor calculation core to obtain a calculation result, and the calculation result is sent to a vector calculation core.

[0015] The calculation result is subjected to softmax processing by the vector calculation core to obtain an attention weight matrix, and the attention weight matrix is sent to the tensor calculation core.

[0016] The tensor calculation core performs matrix multiplication of the attention weight matrix and the value sub-matrix corresponding to the key sub-matrix to obtain an attention output.

[0017] In one of the embodiments, the combination of the plurality of query sub-matrices, the plurality of key sub-matrices, and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices includes:

[0018] For each query sub-matrix, each key sub-matrix is traversed to obtain a plurality of initial combinations including one query sub-matrix and one key sub-matrix.

[0019] adding the value sub-matrix corresponding to the key sub-matrix of the initial combination to each initial combination to obtain the plurality of groups of sub-matrices.

[0020] In one of the embodiments, in the process of attention calculation for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, including:

[0021] The query sub-matrix in the group of sub-matrices is divided into two query sub-modules along the sequence dimension of the query sub-matrix.

[0022] In one of the embodiments, the method further includes:

[0023] After the attention calculation of each group of sub-matrices, a cross-block compensation operation is performed.

[0024] In a second aspect, the present application further provides an attention calculation device, including:

[0025] a matrix blocking unit, configured to block a query matrix, a key matrix and a value matrix determined based on input features to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; the plurality of key sub-matrices and the plurality of value sub-matrices correspond to each other one by one;

[0026] a matrix combination unit, configured to combine the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices; each group of sub-matrices includes one query sub-matrix, one key sub-matrix and one value sub-matrix, and at least one of the query sub-matrix and the key sub-matrix is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other;

[0027] an attention calculation unit, configured to divide the query sub-matrix in each group of sub-matrices into two query sub-modules in the process of attention calculation for each group of sub-matrices, and process the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices in an alternating parallel manner to obtain the attention output of the two query sub-modules.

[0028] In a third aspect, the present application further provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0029] blocking a query matrix, a key matrix and a value matrix determined based on input features to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; the plurality of key sub-matrices and the plurality of value sub-matrices correspond to each other one by one;

[0030] combining the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices; each group of sub-matrices comprises one query sub-matrix, one key sub-matrix and one value sub-matrix, and at least one of the query sub-matrices and the key sub-matrices is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other;

[0031] In the attention calculation process for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, and the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel calculation manner to obtain the attention output of the two query sub-modules.

[0032] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the following steps:

[0033] The query matrix, the key matrix and the value matrix determined based on the input features are blocked to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; the plurality of key sub-matrices and the plurality of value sub-matrices correspond to each other one by one;

[0034] combining the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices; each group of sub-matrices comprises one query sub-matrix, one key sub-matrix and one value sub-matrix, and at least one of the query sub-matrices and the key sub-matrices is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other;

[0035] In the attention calculation process for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, and the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel calculation manner to obtain the attention output of the two query sub-modules.

[0036] In a fifth aspect, the present application further provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the following steps:

[0037] The query matrix, the key matrix and the value matrix determined based on the input features are blocked to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; the plurality of key sub-matrices and the plurality of value sub-matrices correspond to each other one by one;

[0038] The plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices are combined to obtain a plurality of groups of sub-matrices; each group of sub-matrices comprises a query sub-matrix, a key sub-matrix and a value sub-matrix, and at least one of the query sub-matrix and the key sub-matrix is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other;

[0039] In the attention calculation process for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, and the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel calculation manner to obtain the attention output of the two query sub-modules.

[0040] The above attention calculation method, device, computer equipment, computer readable storage medium and computer program product divide the query matrix, the key matrix and the value matrix determined based on the input features to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices are combined to obtain a plurality of groups of sub-matrices; in the attention calculation process for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, and the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel calculation manner to obtain the attention output of the two query sub-modules. The method first divides the query matrix, the key matrix and the value matrix to form a plurality of groups of sub-matrices, and further divides the query sub-matrix when performing attention calculation on each group of sub-matrices, and executes the attention calculation process of the two query sub-modules in an alternating parallel manner, so that the computing cores can be fully utilized and the calculation performance of the attention operator can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creating any inventive labor.

[0042] Figure 1 It is a schematic diagram of the attention calculation process in the prior art;

[0043] Figure 2 It is a schematic diagram of the flow of the attention calculation method in an embodiment;

[0044] Figure 3 It is a schematic diagram of the flow of the alternating parallel attention calculation process in an embodiment;

[0045] Figure 4 Fig. 2 is a schematic diagram of an attention computation flow for another embodiment;

[0046] Figure 5 Fig. 3 is a structural block diagram of an attention computation device for an embodiment;

[0047] Figure 6 Fig. 4 is an internal structural diagram of a computer device for an embodiment. DETAILED DESCRIPTION

[0048] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0049] It should be noted that the terms "first", "second", and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application refers to two or more.

[0050] It can be understood that in the related art, there are several ways to optimize the performance of attention, for example, FlashAttention reduces the exchange of data between GMB (Global Memory Bandwidth, referring to the GPU global memory bandwidth) and HBM (High Bandwidth Memory) by caching the intermediate results of QK and softmax calculation on GMB, thereby greatly improving the performance; for example, preload Q or KV value on-chip cache to reduce bandwidth pressure; for example, some performance optimization methods for causal mask (causal mask) in the form of upper triangular can significantly improve the performance of attention operator, thereby improving the performance of the entire transfomer.

[0051] However, as Figure 1As shown, the calculation of the attention operator in these methods is first the MMA calculation of QK, then the softmax processing, and finally the MMA calculation of SV. These three calculations have data dependencies, so they are currently executed in series. The softmax calculation cannot be started until the QK matrix calculation is completed, and the SV MMA calculation cannot be started until the softmax calculation is completed. Therefore, this serial method cannot achieve full-load operation of the two calculation cores, and the attention operator cannot achieve optimal performance.

[0052] Based on the above problems, the present application provides an attention calculation method which can improve the performance of the attention operator.

[0053] The method provided by the present application is suitable for attention calculation in the fields of speech processing, image processing, text processing, video processing, etc. By embedding and encoding the content to be subjected to attention calculation in any field, input features are obtained, and then the attention calculation method provided by the present application can be further executed. In addition, the attention operator for attention calculation in the present application can be various types of attention operators such as MQA (Multi-Query Attention), GQA (Grouped Query Attention), MHA (Multi-Head Attention), MLA (Multi-Head Linear Attention), etc.

[0054] In one embodiment, as shown in Figure 2 An attention calculation method is provided, which is applied to an artificial intelligence processor. In the present embodiment, the artificial intelligence processor is any one of a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), an NPU (Neural network Processing Unit), a DPU (Deep learning Processing Unit), an APU (Accelerated Processing Unit), and a GPGPU (General-Purpose Graphics Processing Unit). In the present embodiment, the artificial intelligence processor is not specifically limited, and the following description will be given by taking the GPU as an example.

[0055] In the present embodiment, the method comprises the following steps:

[0056] Step S210, the query matrix, the key matrix and the value matrix determined based on the input feature are blocked to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices.

[0057] The plurality of key sub-matrices and the plurality of value sub-matrices correspond one by one.

[0058] The input feature refers to original feature data to be executed for attention calculation, which can be understood as a feature matrix obtained after embedding / encoding processing of an input sequence, with a dimension of sequence length x feature dimension, and is original data for generating the query matrix, the key matrix and the value matrix. The input sequence can be a modalized data such as text, image, voice, etc., and the input feature is obtained by embedding / encoding the modalized data.

[0059] Exemplarily, the input feature is first linearly mapped to obtain three types of feature matrices: a query matrix (Query, Q), a key matrix (Key, K) and a value matrix (Value, V). The three types of feature matrices are further blocked to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices.

[0060] It should be noted that the number of query matrix blocks can be the same as or different from the number of key matrix blocks, i.e., the number of query sub-matrices can be the same as or different from the number of key sub-matrices, but the number of key sub-matrices is the same as the number of value sub-matrices and corresponds one by one in content.

[0061] For example, the query matrix is divided into four sub-matrices, the key matrix is divided into three sub-matrices, and the value matrix is divided into three sub-matrices corresponding to the key matrix.

[0062] Step S220, the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices are combined to obtain a plurality of groups of sub-matrices.

[0063] Each group of sub-matrices includes one query sub-matrix, one key sub-matrix and one value sub-matrix, and at least one of the query sub-matrix and the key sub-matrix is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other.

[0064] Exemplarily, each query sub-matrix, each key sub-matrix and each value sub-matrix are combined, and one query sub-matrix, one key sub-matrix and one value sub-matrix are selected each time to form a group of sub-matrices, thereby forming a plurality of groups of sub-matrices.

[0065] It should be noted that the key sub-matrix and the value sub-matrix correspond to each other, and can be understood as having a binding relationship, that is, when the key sub-matrix is selected, the corresponding value sub-matrix is also determined. Therefore, the combination of the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices is essentially the combination of the plurality of query sub-matrices and the plurality of key sub-matrices, and after obtaining each combination, the corresponding value sub-matrix of each key sub-matrix is added to the combination to obtain a plurality of groups of sub-matrices.

[0066] For example, there are four query sub-matrices Q1, Q2, Q3 and Q4, three key sub-matrices K1, K2 and K3, and three value sub-matrices V1, V2 and V3, which are combined to obtain 12 groups of sub-matrices. Specifically, for Q1, three groups of sub-matrices {Q1 K1 V1}, {Q1 K2 V2} and {Q1 K3 V3} can be obtained, and similarly, for Q2, Q3 and Q4, three groups of sub-matrices can also be obtained, thereby obtaining 12 groups of sub-matrices.

[0067] In step S230, the query sub-matrix in each group of sub-matrices is divided into two query sub-modules, and the attention calculation process of the two query sub-modules, the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel manner to obtain the attention output of the two query sub-modules.

[0068] Generally, when attention calculation is performed for each group of sub-matrices, the query sub-matrix and the key sub-matrix in the group of sub-matrices are first multiplied, then the calculation result is processed by softmax, and finally the processing result of softmax is multiplied with the value sub-matrix to obtain the attention output. This serial processing method cannot achieve the optimal performance of attention calculation. Therefore, in this embodiment, when attention calculation is performed for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is further divided into two query sub-modules, and the two query sub-modules, the key sub-matrix and the value sub-matrix are calculated in an alternating parallel manner to obtain the attention output corresponding to the two query sub-modules.

[0069] For example, after the query matrix, the key matrix and the value matrix are divided into four query sub-matrices (Q1, Q2, Q3, Q4), three key sub-matrices (K1, K2, K3) and three value sub-matrices (V1, V2, V3), respectively, when attention calculation is performed on each group of sub-matrices, for example, {Q1 K1 V1}, the query sub-matrix Q1 is further divided into two query sub-modules: Q11 and Q12, and attention calculation is performed on the two query sub-modules and K1 and V1 respectively to obtain two attention outputs, and the fused result can be used as the attention output of the group of sub-matrices. Specifically, attention calculation is performed on Q11 and K1 and V1, and attention calculation is performed on Q12 and K1 and V1, and the two are calculated alternately and in parallel to obtain two attention outputs.

[0070] In the above attention calculation method, the query matrix, the key matrix and the value matrix determined based on the input features are divided into multiple query sub-matrices, multiple key sub-matrices and multiple value sub-matrices; the multiple query sub-matrices, the multiple key sub-matrices and the multiple value sub-matrices are combined to obtain multiple groups of sub-matrices; in the process of attention calculation on each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, and the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating and parallel manner to obtain the attention outputs of the two query sub-modules. This method first divides the query matrix, the key matrix and the value matrix, and forms multiple groups of sub-matrices, and when attention calculation is performed on each group of sub-matrices, the query sub-matrix is further divided again, and the attention calculation process is performed on the two query sub-modules in an alternating and parallel manner, so that the computing cores can be fully utilized and the computing performance of the attention operator can be improved.

[0071] In an exemplary embodiment, as shown in Figure 3 Step S230 processes the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices in an alternating and parallel manner to obtain the attention outputs of the two query sub-modules, including:

[0072] Step S310 determines that the two query sub-modules are a first query sub-module and a second query sub-module;

[0073] Step S320 performs matrix multiplication on the first query sub-module and the key sub-matrix to obtain a calculation result, and then performs softmax processing on the calculation result, while performing matrix multiplication on the second query sub-module and the key sub-matrix;

[0074] Step S330, after the softmax processing of the calculation result, the attention weight matrix is obtained, the matrix multiplication is performed on the attention weight and the value sub-matrix to obtain the first attention output, and the softmax processing is performed on the calculation result of the matrix multiplication of the second query sub-module and the key sub-matrix. The matrix multiplication is performed on the obtained attention weight and the value sub-matrix to obtain the second attention output.

[0075] Specifically, the attention calculation of the query sub-module, the key sub-matrix and the value sub-matrix includes multiple stages and is a serial calculation. If two query sub-modules are a first query sub-module and a second query sub-module, the parallel alternation of the attention processes of the two query sub-modules and the key sub-matrix and the value sub-matrix is calculated, which can be understood as that while one stage of calculation of the first query sub-module is performed, another stage of calculation of the second query sub-module is performed.

[0076] Reference Figure 4 , a set of sub-matrices processed is {Q1 K1 V1}, the query sub-matrix Q1 is further divided into two query sub-modules: Q11 and Q12, the calculation of the attention process of the first query sub-module is indicated by the white modules, that is, {Q11 K1 V1}; the calculation of the attention process of the second query sub-module is indicated by the gray modules, that is, {Q12 K1 V1}. First, the matrix multiplication (MMA) is performed on the first query sub-module Q11 and the key sub-matrix K1 to obtain a calculation result, and then the softmax processing is performed on the calculation result while the matrix multiplication is performed on the second query sub-module Q12 and the key sub-matrix K1. After the softmax processing of the calculation result, the attention weight matrix S1 is obtained, the matrix multiplication is performed on the attention weight matrix and the value sub-matrix V1, and the softmax processing is performed on the calculation result of the matrix multiplication of the second query sub-module Q12 and the key sub-matrix K1. Then, the matrix multiplication is performed on the attention weight matrix obtained by the softmax processing and the value sub-matrix V1 to obtain the second attention output of the second query sub-module Q12. That is, after the MMA calculation of Q11 and K1, the softmax processing is started, and the MMA calculation of Q12 and K1 is started at the same time; then the MMA calculation of the processing result (denoted as S1) of the softmax and V1 is started, and the softmax processing of the MMA calculation result of Q12 and K1 is started, and then the MMA calculation of S2 and V1 is performed to obtain the second attention output.

[0077] In this embodiment, the calculation of the attention processes of the first query sub-module and the second query sub-module is alternately and in parallel, the calculation core of the attention mechanism can be fully utilized, the calculation processes are mutually covered, and thus the attention calculation performance can be improved.

[0078] In an exemplary embodiment, the method further includes: in the attention calculation process of any query submodule, through the tensor calculation core, performing matrix multiplication of the query submodule with the key matrix of the same group to obtain the calculation result, and sending the calculation result to the vector calculation core; through the vector calculation core, performing softmax processing on the calculation result to obtain the attention weight matrix, and sending the attention weight matrix to the tensor calculation core; the tensor calculation core performing matrix multiplication of the attention weight matrix with the value matrix corresponding to the key matrix to obtain the attention output.

[0079] Still with Figure 4 For example, when calculating the attention process, matrix multiplication of QK and SV is performed through the tensor computation core, and softmax processing is performed through the vector computation core. Taking the attention calculation of the first query submodule as an example, the tensor computation core performs matrix multiplication of query submodule Q11 with the key matrix K1 in the same group to obtain the calculation result, which is then sent to the vector computation core. The vector computation core performs softmax processing on the calculation result to obtain the attention weight matrix S1, which is then sent to the tensor computation core. The tensor computation core performs matrix multiplication of the attention weight matrix S1 with the value matrix V1 corresponding to the key matrix K1 to obtain the first attention output.

[0080] It's understandable that after the tensor computation core sends the result of the matrix multiplication of Q11 and K1 to the vector computation core, the tensor computation core is idle. Therefore, while the vector computation core is performing softmax processing on the first query submodule, the matrix multiplication of the second query submodule Q12 and K1 can be processed in parallel. Similarly, after the vector computation core performs softmax processing on the first query submodule and sends the resulting attention weight matrix to the tensor computation core, the vector computation core is again idle. While the tensor computation core is performing matrix multiplication of S1 and V1, the second query submodule can be simultaneously subjected to softmax processing. After processing, the result is sent back to the tensor computation core for matrix multiplication of S2 and V2.

[0081] In this embodiment, compared with the traditional serial computing method, this solution enables the tensor computing core and the vector computing core to process in parallel, reducing pipeline bubbles and giving full play to the advantages of GPU multi-core parallel computing.

[0082] In an example embodiment, the step S220 combines the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices, including: for each query sub-matrix, traversing each key sub-matrix to obtain a plurality of initial combinations comprising one query sub-matrix and one key sub-matrix; and adding the value sub-matrix corresponding to the key sub-matrix in each initial combination to obtain the plurality of groups of sub-matrices.

[0083] For example, the query matrix, the key matrix and the value matrix are divided into four query sub-matrices (Q1, Q2, Q3, Q4), three key sub-matrices (K1, K2, K3) and three value sub-matrices (V1, V2, V3), respectively. For each query sub-matrix, each key sub-matrix is traversed to obtain a plurality of initial combinations comprising one query sub-matrix and one key sub-matrix. Since the key sub-matrix and the value sub-matrix correspond to each other, the value sub-matrix corresponding to the key sub-matrix in each initial combination can be directly added to obtain a plurality of groups of sub-matrices. For example, for Q2, each key sub-matrix K1, K2, K3 is traversed to obtain three initial combinations: Q2K1, Q2K2, Q2K3. K1 corresponds to V1, K2 corresponds to V2, and K3 corresponds to V3. Therefore, by adding the corresponding value sub-matrix in the three initial combinations, three groups of sub-matrices Q2K1V1, Q2K2V2, Q2K3V3 can be obtained.

[0084] In this embodiment, after the query matrix, the key matrix and the value matrix are initially divided, they are combined into a plurality of groups of sub-matrices to facilitate attention calculation on each group of sub-matrices.

[0085] In an example embodiment, in the process of attention calculation on each group of sub-matrices in step S230, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, including: dividing the query sub-matrix in the group of sub-matrices into two query sub-modules along the sequence dimension of the query sub-matrix.

[0086] The sequence dimension, i.e. the number dimension of the query sub-matrix, corresponds to the token number of the input sequence. For example, the dimension of Q is Nq x dk, Nq is the sequence dimension, corresponding to the number of queries; dk is the feature dimension, corresponding to the feature dimension of a single query.

[0087] Exemplarily, assuming that the dimension of QK is [128, 256], the QK is divided into two sub-modules [64, 256] during the division. It can be understood that because the feature dimension is an inherent attribute of the Query matrix, the splitting will destroy the calculation logic of the Q-K inner product, and the sequence dimension is a collection of independent Queries, and each sub-block can still calculate the attention weight with the K sub-matrix independently after splitting. Therefore, the embodiment divides the query sub-matrix along the sequence dimension, so that the data between the two query sub-modules is independent of each other, and there is no dependency relationship. Therefore, the first query sub-module and the second query sub-module can be independently executed in series MMA (QxK), softmax, MMA (SxV), update, and the two can be alternately processed in parallel.

[0088] In one embodiment, the method further comprises: performing a cross-block compensation operation after the attention calculation of each group of sub-matrices.

[0089] Exemplarily, assuming that the query matrix Q is divided into Q1, Q2, Q3, Q4, the key matrix K is divided into K1, K2, K3, and the value sub-matrix is divided into V1, V2, V3, 12 groups of sub-matrices can be obtained by combination. Each query sub-matrix, for example, Q1, performs attention calculation with all K, V sub-matrices. For example, Q1 performs attention calculation with K1, V1 (denoted as the first group of sub-matrices) to obtain the attention output Attention{ Q1K1V1}. Since the softmax processing in the attention calculation process is to calculate the maximum value of the row, only the maximum value of the first group of sub-matrices is calculated initially. Then, the second group of sub-matrices {Q1 K2 V2} is calculated to obtain the attention output Attention{ Q1K2V2}, and based on the maximum value calculated by the softmax in the first group of sub-matrices and the maximum value calculated by the softmax in the second group of sub-matrices, the attention output of the first group of sub-matrices { Q1K1V1} is corrected. In this way, the attention outputs of the previously executed groups of sub-matrices are corrected to realize cross-block compensation.

[0090] It can be understood that after the attention calculation of each group of sub-matrices, the cross-block compensation operation is performed, which can calibrate the local attention output of each group of sub-matrices and ensure the accuracy of the output result, as shown in the update shown in Figure 4 .

[0091] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0092] Based on the same inventive concept, the embodiments of the present application also provide an attention calculation device for implementing the above-mentioned attention calculation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more attention calculation device embodiments provided below can refer to the limitations of the attention calculation method in the above text, which will not be repeated here.

[0093] In an exemplary embodiment, as shown in Figure 5 An attention calculation device is provided, comprising:

[0094] The matrix blocking unit 510 is configured to block the query matrix, the key matrix and the value matrix determined based on the input features to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; the plurality of key sub-matrices and the plurality of value sub-matrices correspond one-to-one;

[0095] The matrix combination unit 520 is configured to combine the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices; each group of sub-matrices includes a query sub-matrix, a key sub-matrix and a value sub-matrix, and at least one of the query sub-matrix and the key sub-matrix is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other;

[0096] The attention calculation unit 530 is configured to, in the process of attention calculation for each group of sub-matrices, divide the query sub-matrix in the group of sub-matrices into two query sub-modules, and process the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices in an alternating parallel manner to obtain the attention output of the two query sub-modules.

[0097] In one of the embodiments, the attention calculation unit 530 is configured to determine two query sub-modules as a first query sub-module and a second query sub-module; perform matrix multiplication on the first query sub-module and the key sub-matrix to obtain a calculation result, and perform softmax processing on the calculation result, while performing matrix multiplication on the second query sub-module and the key sub-matrix; perform softmax processing on the calculation result to obtain an attention weight matrix, and perform matrix multiplication on the attention weight and the value sub-matrix to obtain a first attention output, while performing softmax processing on the calculation result of the matrix multiplication of the second query sub-module and the key sub-matrix, and performing matrix multiplication on the obtained attention weight and the value sub-matrix to obtain a second attention output.

[0098] In one of the embodiments, the attention calculation unit 530 is further configured to, in the attention calculation process of any query sub-module, perform matrix multiplication on the query sub-module and the key sub-matrix of the same group by the tensor calculation core to obtain a calculation result, and send the calculation result to the vector calculation core; perform softmax processing on the calculation result by the vector calculation core to obtain an attention weight matrix, and send the attention weight matrix to the tensor calculation core; and perform matrix multiplication on the attention weight matrix and the value sub-matrix corresponding to the key sub-matrix by the tensor calculation core to obtain an attention output.

[0099] In one of the embodiments, the matrix combination unit 520 is configured to, for each query sub-matrix, combine each key sub-matrix to obtain a plurality of initial combinations each including one query sub-matrix and one key sub-matrix; and add the value sub-matrix corresponding to the key sub-matrix of each initial combination to the initial combination to obtain a plurality of groups of sub-matrices.

[0100] In one of the embodiments, the attention calculation unit 530 is further configured to divide the query sub-matrices in the group of sub-matrices into two query sub-modules along the sequence dimension of the query sub-matrices.

[0101] In one of the embodiments, the apparatus further includes a compensation unit configured to perform a cross-block compensation operation after the attention calculation of each group of sub-matrices.

[0102] The above-mentioned modules in the attention calculation apparatus can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to the above-mentioned modules.

[0103] In one of the embodiments, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data during attention calculation. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize an attention calculation method.

[0104] Those skilled in the art can understand that, Figure 6 The skilled in the art can understand that,

[0105] In one embodiment, a computer device is also provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps in the above method embodiments.

[0106] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to realize the steps in the above method embodiments.

[0107] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to realize the steps in the above method embodiments.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0109] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0110] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An attention computation method characterized by comprising: The method comprises: determining a query matrix, a key matrix and a value matrix based on input features, and obtaining a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices by blocking the query matrix, the key matrix and the value matrix; the plurality of key sub-matrices and the plurality of value sub-matrices correspond to each other one by one; combining the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices; each group of sub-matrices comprises one query sub-matrix, one key sub-matrix and one value sub-matrix, and at least one of the query sub-matrix and the key sub-matrix is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other; in the process of attention calculation for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, and the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel manner to obtain the attention output of the two query sub-modules; in the attention calculation process of any query sub-module, a tensor calculation core is used to perform matrix multiplication of the query sub-module and the key sub-matrix in the same group to obtain a calculation result, and the calculation result is sent to a vector calculation core; the vector calculation core is used to perform softmax processing on the calculation result to obtain an attention weight matrix, and the attention weight matrix is sent to the tensor calculation core; the tensor calculation core is used to perform matrix multiplication of the attention weight matrix and the value sub-matrix corresponding to the key sub-matrix to obtain an attention output.

2. The method of claim 1, wherein, the attention calculation process of the two query sub-modules and the key sub-matrix and the value sub-matrix in the group of sub-matrices is processed in an alternating parallel manner to obtain the attention output of the two query sub-modules, comprising: determining that the two query sub-modules are a first query sub-module and a second query sub-module; after performing matrix multiplication of the first query sub-module and the key sub-matrix to obtain a calculation result, performing softmax processing on the calculation result, and simultaneously performing matrix multiplication of the second query sub-module and the key sub-matrix; after obtaining the attention weight matrix by performing softmax processing on the calculation result, performing matrix multiplication of the attention weight and the value sub-matrix to obtain a first attention output, and simultaneously performing softmax processing on the calculation result of the matrix multiplication of the second query sub-module and the key sub-matrix, and performing matrix multiplication of the obtained attention weight and the value sub-matrix to obtain a second attention output.

3. The method of claim 1, wherein, the combining of the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices, comprising: for each query sub-matrix, combining each key sub-matrix to obtain a plurality of initial combinations comprising one query sub-matrix and one key sub-matrix; in each initial combination, adding the value sub-matrix corresponding to the key sub-matrix of the initial combination to obtain the plurality of groups of sub-matrices.

4. The method of claim 1, wherein, in the process of attention calculation for each group of sub-matrices, the query sub-matrix in the group of sub-matrices is divided into two query sub-modules, comprising: Along the sequence dimension of the query sub-matrices, the query sub-matrices in the group of sub-matrices are divided into two query sub-modules.

5. The method of claim 1, wherein, The method further comprises: After attention calculation of each group of sub-matrices, a cross-block compensation operation is performed.

6. An attention computing apparatus characterized by comprising: The device comprises: A matrix blocking unit is configured to block a query matrix, a key matrix and a value matrix determined based on input features to obtain a plurality of query sub-matrices, a plurality of key sub-matrices and a plurality of value sub-matrices; the plurality of key sub-matrices and the plurality of value sub-matrices correspond to each other one by one; A matrix combination unit is configured to combine the plurality of query sub-matrices, the plurality of key sub-matrices and the plurality of value sub-matrices to obtain a plurality of groups of sub-matrices; each group of sub-matrices comprises one query sub-matrix, one key sub-matrix and one value sub-matrix, and at least one of the query sub-matrix and the key sub-matrix is different between the groups of sub-matrices, and the key sub-matrix and the value sub-matrix in each group of sub-matrices correspond to each other; An attention calculation unit is configured to, in an attention calculation process for each group of sub-matrices, divide the query sub-matrices in the group of sub-matrices into two query sub-modules, and process the attention calculation processes of the two query sub-modules and the key sub-matrices and the value sub-matrices in the group of sub-matrices in an alternating parallel manner to obtain attention outputs of the two query sub-modules. The attention calculation unit is further configured to, in the attention calculation process of any query sub-module, perform matrix multiplication of the query sub-module and the key sub-matrix in the same group by a tensor calculation core to obtain a calculation result, send the calculation result to a vector calculation core, perform softmax processing on the calculation result by the vector calculation core to obtain an attention weight matrix, and send the attention weight matrix to the tensor calculation core; and the tensor calculation core performs matrix multiplication of the attention weight matrix and the value sub-matrix corresponding to the key sub-matrix to obtain an attention output.

7. The apparatus of claim 6, wherein, The matrix combination unit is further configured to, for each query sub-matrix, combine each key sub-matrix to obtain a plurality of initial combinations comprising one query sub-matrix and one key sub-matrix; and add the value sub-matrix corresponding to the key sub-matrix in each initial combination to obtain the plurality of groups of sub-matrices.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images