Data processing method and device and computing device cluster

Abstract

The data processing method comprises the steps that a query vector matrix, a key vector matrix and a value vector matrix are obtained, and the query vector matrix, the key vector matrix and the value vector matrix are all obtained through calculation based on an input sequence; reading an attention mask from a memory of the image processor, wherein the attention mask is uploaded by a user or is generated by an element arrangement rule specified by the user; based on an attention mask, calculating the query vector matrix, the key vector matrix and the value vector matrix to obtain a reasoning result related to the input sequence, the attention mask being used for reconstructing a second sub-matrix, and the second sub-matrix being used for obtaining the reasoning result; wherein each row in the second sub-matrix is a row selected from the attention mask; or, each row in the second sub-matrix is obtained by constructing sequentially adjacent elements in one row of the attention mask, the row number of the attention mask is smaller than the row number of the query vector matrix, and the column number of the attention mask is smaller than the row number or the column number of the key vector matrix. According to the method, the video memory occupation of the attention mask can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), and in particular to a data processing method and device and a computing device cluster. BACKGROUND

[0002] The Transformer model is a neural network model based on a self-attention mechanism (Self-attention), which has been widely used in natural language processing tasks such as machine translation, text generation, and text classification, and has achieved remarkable results. Self-attention is a key component of the Transformer model, which is a mechanism for establishing global dependencies in the input sequence to better capture long-range dependencies in the sequence.

[0003] In Self-attention, each token in the input sequence generates three vectors: a query vector (query, Q for short), a key vector (key, K for short), and a value vector (value, V for short). The Q associated with each token can form a Q matrix, the K associated with each token can form a K matrix, and the V associated with each token can form a V matrix. In the calculation process, a new matrix can be obtained by performing matrix cross multiplication operations on the Q matrix and the K matrix. This new matrix represents the correlation between each token and other tokens, i.e., the attention weight. Then, by performing matrix cross multiplication operations on the new matrix and the V matrix, the inference result can be obtained.

[0004] From the calculation process of Self-attention, it can be seen that Self-attention requires two matrix cross multiplication operations on Q, K, and V. In order to limit the current token to establish dependencies only before the time node, an attention mask (attention_mask) can be introduced to perform matrix point multiplication operations on the output matrix obtained by QxK, i.e., to perform mask processing. Currently, the memory capacity required to store the attention mask is the square of the length of the input sequence, which occupies a large amount of memory. Especially in the context of a super-long input sequence, the memory occupation of the attention_mask will cause a great burden. SUMMARY

[0005] The present application provides a data processing method, device, computing device cluster, computer storage medium, and computer product, which can reduce the memory occupation of the attention_mask.

[0006] In a first aspect, the present application provides a data processing method, comprising: obtaining a Q matrix, a K matrix and a V matrix, wherein the Q matrix, the K matrix and the V matrix are calculated based on an input sequence; reading an attention_mask from a memory of a graphics processing unit (GPU), wherein the attention_mask is generated based on an element arrangement rule uploaded or specified by a user; and performing calculation on the Q matrix, the K matrix and the V matrix based on the attention_mask to obtain an inference result related to the input sequence, wherein the attention_mask is used to reconstruct a second sub-matrix, the second sub-matrix is used to obtain the inference result, each row in the second sub-matrix is selected from the attention_mask, or each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention_mask, and the number of rows of the attention_mask is less than the number of rows of the Q matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix.

[0007] In this way, the second sub-matrix can be reconstructed by the attention_mask in the calculation process to perform mask processing by the second sub-matrix. The number of columns of the attention_mask is the same as the number of columns of the second sub-matrix, or the number of rows of the attention_mask is less than the number of rows of the Q matrix and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix, so the size of the attention_mask is smaller than the size of the matrix P' obtained based on the Q matrix and the K matrix, while the size of the previously used attention_mask (such as the attention_mask used in the prior art) is the same as the size of the matrix P', so the size of the attention_mask used in the data processing method provided in the embodiments of the present application is also smaller than the previously used attention_mask, and the memory usage of the attention_mask can be reduced by the data processing method provided in the embodiments of the present application. Figure 1

[0008] In a possible implementation, the second sub-matrix can be used to perform matrix point multiplication with a first sub-matrix to obtain an intermediate result, the intermediate result is used to perform calculation with a sub-matrix contained in the V matrix (such as matrix cross multiplication) to obtain the inference result, the first sub-matrix is obtained based on matrix block processing on a matrix P', and the matrix P' is obtained by matrix cross multiplication based on the Q matrix and the K matrix. In this way, the first sub-matrix can be masked by the second sub-matrix, and the inference result is calculated based on the intermediate result of the mask processing.

[0009] ​In a possible implementation, each row in the second sub-matrix is a row selected from the attention_mask, the attention_mask has n columns and (n+1) rows, and n is the number of columns of the second sub-matrix.

[0010] In a possible implementation, each row in the second sub-matrix is a row selected from the attention_mask, the number of elements 1 in the s-th row of the attention_mask is s, 0≤s≤n, and the elements 1 are arranged in sequence from left to right in each row of the new attention_mask that contains elements 1.

[0011] In a possible implementation, each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention_mask, the number of rows of the attention_mask is less than the number of rows of the Q matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix; wherein the number of rows of the attention_mask is (a*S), the number of columns of the attention_mask is (b*S), S is the maximum of the number of rows and the number of columns of the second sub-matrix, a and b are integers, and both are greater than or equal to 2.

[0012] In a possible implementation, each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention_mask, the number of rows of the attention_mask is less than the number of rows of the Q matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix. When the number of rows of the attention_mask is less than or equal to the number of columns of the attention_mask, the number of elements 1 in the s-th row of the attention_mask is (s+1), 0≤s, and the elements 1 are sequentially and adjacently distributed from left to right in each row of the attention_mask. When the number of rows of the attention_mask is greater than the number of columns of the attention_mask, the number of elements 1 in the s-th row of the attention_mask is (s+1), 0≤s

[0013] In a possible implementation, the element arrangement rule can include an upper triangle, a lower triangle, an upper triangle with diagonal lines moving up or down, or a lower triangle with diagonal lines moving up or down. The upper triangle with diagonal lines moving up refers to a boundary line between 0 and 1 in the matrix being located on the upper side of a center diagonal line of the matrix, where the center diagonal line and the boundary line are parallel. The upper triangle with diagonal lines moving down refers to a boundary line between 0 and 1 in the matrix being located on the lower side of a center diagonal line of the matrix, where the center diagonal line and the boundary line are parallel. For example, the upper triangle can be understood as 1 in the matrix being located above the boundary line between 0 and 1 in the matrix, and 0 being located below the boundary line between 0 and 1 in the matrix.

[0014] In a possible implementation, the calculation of the Q matrix, the K matrix, and the V matrix based on the attention mask includes: performing matrix cross multiplication on the Q matrix and the K matrix to obtain a matrix P'; performing matrix multiplication on the matrix P' and the attention mask, and processing the matrix multiplication result by using a softmax function to obtain a matrix P; and performing matrix cross multiplication on the matrix P and the V matrix to obtain an inference result.

[0015] In a possible implementation, each row in the second submatrix is a row selected from the attention mask. At this time, the matrix multiplication on the matrix P' and the attention mask includes: based on a position of a first submatrix currently participating in calculation in the matrix P' in the matrix P', calculating a first row index of a starting row of a target submatrix in the attention mask, the target submatrix being a second submatrix used for matrix multiplication with the first submatrix currently participating in calculation, and the first row index being used to indicate the position of the starting row of the target submatrix in the attention mask; based on the first row index, calculating row indexes of remaining rows of the target submatrix in the attention mask; based on the row indexes of each row of the target submatrix in the attention mask, selecting rows identified by the row indexes from the attention mask, and combining the selected rows in a row direction of the target submatrix according to an arrangement order of rows in the target submatrix to obtain the target submatrix; and performing matrix multiplication on the first submatrix currently participating in calculation in the matrix P' and the target submatrix. In this way, by querying the row indexes of each row of the target submatrix currently participating in calculation and used for matrix multiplication in the attention mask, the target submatrix currently participating in calculation can be reconstructed by using the attention mask, and participate in calculation.

[0016] In a possible implementation, before the target sub-matrix is obtained by combining the rows selected from the attention mask according to the arrangement order of the rows in the target sub-matrix based on the row index of each row of the target sub-matrix in the attention mask, the method further includes: in a case where the row index of any row of the target sub-matrix in the attention mask is less than 0, setting the row index of the any row in the attention mask to 0; in a case where the row index of any row of the target sub-matrix in the attention mask is greater than the maximum value of the row indexes in the attention mask, setting the row index of the any row in the attention mask to the maximum value of the row indexes in the attention mask. In this way, the row index that does not meet the requirement can be converted into the row index that meets the requirement, so as to facilitate restoring the target sub-matrix currently participating in the calculation by using the attention mask.

[0017] In a possible implementation, each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention mask, and the number of rows of the attention mask is less than the number of rows of the Q matrix, and the number of columns of the attention mask is less than the number of rows or columns of the K matrix. At this time, the matrix point multiplication of the matrix P' and the attention mask includes: based on the position of the first sub-matrix currently participating in the calculation in the matrix P', calculating a first row index of a first target row and a first column index of a first target column in the attention mask, where the first target row includes a starting row of the target sub-matrix, the first target column includes a starting column of the target sub-matrix, and the target sub-matrix is a sub-matrix used for the matrix point multiplication with the first sub-matrix currently participating in the calculation; selecting a matrix with the same shape as the first sub-matrix from the attention mask based on the first row index and the first column index, to obtain the target sub-matrix; and performing the matrix point multiplication of the first sub-matrix currently participating in the calculation and the target sub-matrix.

[0018] In a possible implementation, the matrix point multiplication of the first sub-matrix currently participating in the calculation in the matrix P' and the target sub-matrix includes: in a case where all elements of the target sub-matrix are 0, directly setting the calculation result to 0; and in a case where all elements of the target sub-matrix are 1, directly taking the first sub-matrix currently participating in the calculation in the matrix P' as the calculation result. In this way, some invalid calculations can be reduced, so as to improve the calculation efficiency.

[0019] In a possible implementation, before the calculation of the Q matrix, the K matrix and the V matrix based on the attention mask, the method further includes: receiving a user's designation of invalid data in an input matrix and / or an output matrix, the input matrix being a left matrix and / or a right matrix participating in the matrix operation, and the output matrix being a result of the matrix operation on the left matrix and the right matrix. At this time, the calculation of the Q matrix, the K matrix and the V matrix based on the attention mask includes: in the case of matrix operation on any two of the Q matrix, the K matrix, the V matrix and the attention mask, skipping the calculation related to the invalid data. In this way, some invalid calculations can be reduced, thereby improving the calculation efficiency.

[0020] In a possible implementation, when the K matrix is a matrix before transposition, that is, the K matrix needs to be transposed to perform matrix multiplication with the Q matrix, the number of columns of the attention mask is less than the number of rows of the K matrix, and when the K matrix is a matrix after transposition, that is, the K matrix does not need to be transposed to perform matrix multiplication with the Q matrix, the number of columns of the attention mask is less than the number of columns of the K matrix. In this way, the size of the attention mask can be less than the size of the matrix P' obtained according to the Q matrix and the K matrix, and the size of the previously used attention mask (for example, the attention mask used in the prior art) is the same as the size of the matrix P', so that the use of the attention mask provided in the present application can reduce the memory occupation. Figure 1

[0021] In a possible implementation, the rows in the second sub-matrix are selected from consecutive rows of the attention mask, and the columns in the second sub-matrix are selected from consecutive columns of the attention mask. In this way, the first element of each row in the second sub-matrix is located in the same column of the attention mask. For example, if the shape of the second sub-matrix is 3x3, if the elements in the first row of the second sub-matrix are selected from the second row of the attention mask, the elements in the second row of the second sub-matrix are selected from the third row of the attention mask, and the elements in the third row of the second sub-matrix are selected from the fourth row of the attention mask. If the first element of the first row of the second sub-matrix is located in the first column of the attention mask, the first elements of the second row and the third row of the second sub-matrix are located in the first column of the attention mask. In this way, the second sub-matrix can be constructed from the attention mask.

[0022] ​In a second aspect, the present application provides a data processing apparatus, comprising: a communication module and a processing module. The communication module is configured to obtain a Q matrix, a K matrix and a V matrix, wherein the Q matrix, the K matrix and the V matrix are calculated based on an input sequence. The processing module is configured to read an attention_mask from a memory of a graphics processing unit (GPU), the attention_mask being generated according to an element arrangement rule specified by a user or uploaded by the user, and perform calculation on the Q matrix, the K matrix and the V matrix based on the attention_mask to obtain an inference result related to the input sequence, the attention_mask being used to reconstruct a second sub-matrix, and the second sub-matrix being used to obtain the inference result. Each row in the second sub-matrix is selected from the attention_mask, or each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention_mask, and the number of rows of the attention_mask is less than the number of rows of the Q matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix.

[0023] In a possible implementation, the second sub-matrix can be used to perform matrix point multiplication with the first sub-matrix to obtain an intermediate result, and the intermediate result is used to perform calculation with a sub-matrix included in the V matrix to obtain the inference result. The first sub-matrix is obtained based on matrix block processing of a matrix P', and the matrix P' is obtained by performing matrix cross multiplication on the Q matrix and the K matrix.

[0024] In a possible implementation, each row in the second sub-matrix is selected from the attention_mask, the number of columns of the attention_mask is n, the number of rows of the attention_mask is (n+1), and n is the number of columns of the second sub-matrix.

[0025] In a possible implementation, each row in the second sub-matrix is selected from the attention_mask, the number of elements 1 in the s-th row of the attention_mask is s, 0≤s≤n, and the elements 1 are sequentially adjacent from left to right in each row of the attention_mask containing the element 1.

[0026] In a possible implementation, each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention mask, and the number of rows of the attention mask is less than the number of rows of the Q matrix, and the number of columns of the attention mask is less than the number of rows or columns of the K matrix; wherein the number of rows of the attention mask is (a*S), the number of columns of the attention mask is (b*S), S is the maximum of the number of rows and the number of columns of the second sub-matrix, a and b are both integers and are both greater than or equal to 2.

[0027] In a possible implementation, each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention mask, and the number of rows of the attention mask is less than the number of rows of the Q matrix, and the number of columns of the attention mask is less than the number of rows or columns of the K matrix. When the number of rows of the attention mask is less than or equal to the number of columns of the attention mask, the number of elements 1 in the s-th row of the attention mask is (s+1), 0≤s, and the elements 1 are sequentially adjacent from left to right in each row of the attention mask. When the number of rows of the attention mask is greater than the number of columns of the attention mask, the number of elements 1 in the s-th row of the attention mask is (s+1), 0≤s

[0028] In a possible implementation, the element arrangement rule includes upper triangle, lower triangle, upper triangle with diagonal line moving up or down, or lower triangle with diagonal line moving up or down.

[0029] In a possible implementation, when the processing module performs calculation on the Q matrix, the K matrix and the V matrix based on the attention mask, the processing module is specifically configured to: perform matrix cross multiplication on the Q matrix and the K matrix to obtain a matrix P'; perform matrix dot multiplication on the matrix P' and a new attention mask, and perform processing on the matrix dot multiplication result by using a softmax function to obtain a matrix P; and perform matrix cross multiplication on the matrix P and the V matrix to obtain an inference result.

[0030] In a possible implementation, each row in the second sub-matrix is a row selected from the attention_mask, and the number of rows of the attention_mask is less than the number of rows of the Q matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix. At this time, when the processing module performs the matrix point multiplication on the matrix P' and the attention_mask, the processing module is specifically configured to: based on a position of the first sub-matrix currently participating in the calculation in the matrix P', calculate a first row index of a starting row of a target sub-matrix in the attention_mask, the target sub-matrix being a second sub-matrix used for the matrix point multiplication with the first sub-matrix currently participating in the calculation, the first row index being used to indicate the position of the starting row of the target sub-matrix in the attention_mask; based on the first row index, calculate row indexes of the remaining rows of the target sub-matrix in the attention_mask; based on the row indexes of the rows of the target sub-matrix in the attention_mask, select rows identified by the row indexes from the attention_mask, and combine the rows selected from the attention_mask in the row direction of the target sub-matrix according to the arrangement order of the rows in the target sub-matrix, to obtain the target sub-matrix; and perform the matrix point multiplication on the first sub-matrix currently participating in the calculation in the matrix P' and the target sub-matrix.

[0031] In a possible implementation, before the processing module selects the rows identified by the row indexes from the attention_mask based on the row indexes of the rows of the target sub-matrix in the attention_mask, and combines the rows selected from the attention_mask in the row direction of the target sub-matrix according to the arrangement order of the rows in the target sub-matrix, to obtain the target sub-matrix, the processing module is further configured to: in a case where a row index of any row of the target sub-matrix in the attention_mask is less than 0, set the row index of the any row in the attention_mask to 0; and in a case where the row index of the any row of the target sub-matrix in the attention_mask is greater than a maximum value of the row indexes in the attention_mask, set the row index of the any row in the attention_mask to the maximum value of the row indexes in the attention_mask.

[0032] In a possible implementation, each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention mask. At this time, when performing the matrix point multiplication on the matrix P' and the attention mask, the processing module is specifically configured to: based on a position of the first sub-matrix currently participating in the calculation in the matrix P', calculate a first row index of a first target row and a first column index of a first target column in the attention mask, where the first target row includes a starting row of the target sub-matrix, the first target column includes a starting column of the target sub-matrix, and the target sub-matrix is a sub-matrix used for the matrix point multiplication with the first sub-matrix currently participating in the calculation; select a matrix with the same shape as the first sub-matrix from the attention mask based on the first row index and the first column index, to obtain the target sub-matrix; and perform the matrix point multiplication on the first sub-matrix currently participating in the calculation and the target sub-matrix.

[0033] In a possible implementation, when performing the matrix point multiplication on the first sub-matrix currently participating in the calculation and the target sub-matrix in the matrix P', the processing module is specifically configured to: in a case where all elements of the target sub-matrix are 0, directly set the calculation result as 0; and in a case where all elements of the target sub-matrix are 1, directly take the first sub-matrix as the calculation result.

[0034] In a possible implementation, before the processing module performs the calculation on the Q matrix, the K matrix and the V matrix based on the attention mask, the communication module is further configured to: receive a user's specification of invalid data in an input matrix and / or an output matrix, the input matrix being a left matrix and / or a right matrix participating in the matrix operation, and the output matrix being a result of the matrix operation on the left matrix and the right matrix. At this time, when performing the calculation on the Q matrix, the K matrix and the V matrix based on the attention mask, the processing module is specifically configured to: in a case where any two of the Q matrix, the K matrix, the V matrix and the attention mask are subjected to the matrix operation, skip the calculation related to the invalid data.

[0035] In a possible implementation, when the K matrix is a matrix before transposition, that is, the K matrix needs to be transposed to perform the matrix multiplication with the Q matrix, the number of columns of the attention mask is less than the number of rows of the K matrix, and when the K matrix is a matrix after transposition, that is, the K matrix does not need to be transposed to perform the matrix multiplication with the Q matrix, the number of columns of the attention mask is less than the number of columns of the K matrix.

[0036] In a possible implementation, the rows in the second sub-matrix are selected from consecutive rows of the attention_mask, and the columns in the second sub-matrix are selected from consecutive columns of the attention_mask. Wherein, the first element of each row in the second sub-matrix is located in the same column of the attention_mask.

[0037] In a third aspect, the present application provides a data processing apparatus, comprising at least one processor and an interface; the at least one processor acquires program instructions through the interface; the at least one processor executes the program instructions to implement the method described in the first aspect or any possible implementation manner of the first aspect. Exemplarily, the data processing apparatus can be but is not limited to a chip.

[0038] In a fourth aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device executes instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0039] In a fifth aspect, the present application provides a computer readable storage medium, comprising computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation manner of the first aspect. Exemplarily, the computing device cluster can comprise one or more computing devices.

[0040] In a sixth aspect, the present application provides a computer program product comprising instructions, when the instructions are run by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation manner of the first aspect. Exemplarily, the computing device cluster can comprise one or more computing devices.

[0041] It can be understood that the beneficial effects of the above-mentioned target aspects to the sixth aspect can refer to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a schematic diagram of a self-attention calculation process provided by an embodiment of the present application;

[0043] Figure 2 is a schematic diagram of mask processing of the calculation results of the Q matrix and the K matrix by the attention_mask provided by an embodiment of the present application;

[0044] Figure 3is a process diagram provided by an embodiment of the application for obtaining a new attention mask from an original attention mask;

[0045] Figure 4 is a step diagram provided by an embodiment of the application for reconstructing each base block used for mask processing from the new attention mask;

[0046] Figure 5 is a process diagram provided by an embodiment of the application for reconstructing a base block used for mask processing from the new attention mask;

[0047] Figure 6 is a process diagram provided by an embodiment of the application for calculating a Q matrix, a K matrix and a V matrix;

[0048] Figure 7 is a diagram provided by an embodiment of the application for defining data to be filtered in an input matrix and an output matrix through an interface;

[0049] Figure 8 is a diagram provided by an embodiment of the application for comparing even distribution and uneven distribution of calculation of a plurality of base blocks to each calculation unit;

[0050] Figure 9 is a step diagram provided by an embodiment of the application for reconstructing each base block used for mask processing from the new attention mask;

[0051] Figure 10 is a process diagram provided by an embodiment of the application for reconstructing a base block used for mask processing from the new attention mask;

[0052] Figure 11 is a flow diagram of a data processing method provided by an embodiment of the application;

[0053] Figure 12 is a structural diagram of a data processing apparatus provided by an embodiment of the application;

[0054] Figure 13 is a structural diagram of a computing device provided by an embodiment of the application;

[0055] Figure 14 is a structural diagram of a computing device cluster provided by an embodiment of the application;

[0056] Figure 15 is a structural diagram of another computing device cluster provided by an embodiment of the application;

[0057] Figure 16 is a structural schematic diagram of another data processing apparatus provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The term "and / or" used in this document is a description of an association relationship between associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The symbol " / " in this document means an or relationship of associated objects, for example, A / B means A or B.

[0059] The terms "first" and "second" and the like in the description and claims of this document are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, and are not used to describe a specific order of the response messages.

[0060] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration, in no way limiting. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design schemes. Rather, the words "exemplary" or "for example" are used to present the relevant concept in a specific manner.

[0061] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0062] First, some technical terms involved in the present application are introduced.

[0063] (1) Matrix cross multiplication and matrix point multiplication

[0064] Matrix cross multiplication, also known as "matrix multiplication", is an operation between two matrices. The number of columns of the left matrix must be equal to the number of rows of the right matrix. The result of matrix cross multiplication is obtained by multiplying each row of the left matrix with each column of the right matrix and then adding the results. For example:

[0065] Matrix point multiplication, also known as "Hadamard product", is also an operation between two matrices. The shapes of the two matrices must be the same, i.e., the number of rows and the number of columns of the two matrices are the same. The result of matrix point multiplication is obtained by multiplying the elements at corresponding positions in the two matrices. For example:

[0066]

[0067] (2) Matrix block

[0068] Matrix tiling is an operation that cuts a large matrix into smaller blocks to facilitate parallel computing or improve cache utilization. Generally, a larger matrix may not be loaded into memory at once in a computer, so dividing it into small blocks can better utilize computing resources and memory.

[0069] (3) Base block

[0070] The base block refers to the smallest matrix unit obtained by cutting a matrix. Assuming that the matrix is 2560x2560, the matrix is divided into multiple 256x256 small matrices after cutting, and each 256x256 small matrix is a base block. In this embodiment, the base block can be any rectangle, for example, it can be a square or a rectangle. Exemplarily, the base block can also be referred to as a "sub-matrix".

[0071] (4) Element arrangement rule of matrix

[0072] The element arrangement rule of the matrix refers to the way in which the elements in the matrix are arranged according to a specific rule or pattern. Exemplarily, for the attention_mask, the element arrangement rule can be, but is not limited to, how to arrange 0 and 1 in the attention_mask to determine which positions are visible and which positions are invisible. Exemplarily, the element arrangement rule can include upper triangle or lower triangle. The upper triangle refers to that in the matrix, the non-zero elements only appear in the positions above the main diagonal line, and the positions below the main diagonal line are all zero; the lower triangle refers to that in the matrix, the non-zero elements only appear in the positions below the main diagonal line, and the positions above the main diagonal line are all zero.

[0073] The technical solutions provided in the present application will be described below.

[0074] Exemplarily, Figure 1 A schematic diagram of a calculation process of self-attention provided by an embodiment of the present application is shown. As shown in the figure, Figure 1 the calculation process can include: inputting a sequence X and an attention_mask at S11. The type of the elements in the attention_mask is a Boolean type, that is, the elements are 0 or 1. At S12, the Q matrix, the K matrix and the V matrix are calculated. Wherein, Q=X×W Q , K=X×W K , V=X×W V . W Q , W K and W Vare learned weight matrices used for linear transformation. For each token in the input sequence X, Q, K and V can be obtained through linear transformation. In S13, QxK matrix multiplication is performed and divided by a scaling factor to obtain matrix P'. Wherein, d k is the dimension of Q and K. In S14, matrix point multiplication of matrix P' and attention_mask is performed, and the matrix point multiplication result is processed through a softmax function to obtain matrix P, i.e. P = softmax(P' * attention_mask). In S15, matrix cross multiplication of P and V is performed to obtain the output output. It should be understood that the attention_mask in the embodiment can be a multi-dimensional matrix, such as a two-dimensional matrix, a three-dimensional matrix, etc. When the attention_mask is a two-dimensional matrix, its size can be represented as (m x n), m is the number of rows of the attention_mask, and n is the number of columns of the attention_mask. When the attention_mask is a three-dimensional matrix, its size can be represented as (B x m x n), B is the depth or layer of the attention_mask, m is the number of rows of the attention_mask, and n is the number of columns of the attention_mask.

[0075] In Figure 1 , the type of elements in the attention_mask can also be other types, such as float type, etc. When the elements in the attention_mask are of other types, the attention_mask can be multiplied by -10000 (of course, other numerical values can also be replaced, which are not limited here); Then, the calculation result obtained is added to P'; Finally, the normalization operation is performed through the softmax function to obtain the matrix P. At this time, P = softmax(P' + attention_mask*(-10000)).

[0076] Generally, in order to reduce the number of memory accesses and reduce the amount of calculation to improve the calculation efficiency, in Figure 1 the calculation process of self-attention, the matrixes used for matrix cross multiplication and matrix point multiplication operations can be matrix blocked, that is, the matrixes are cut into small sub-matrixes. Then, the basic blocks obtained by matrix blocking are used for matrix cross multiplication or matrix point multiplication operations. For example, as Figure 2 shown, when the type of elements in the attention_mask is Boolean type, and it is a lower triangular matrix, matrix P' and attention_mask can be cut into multiple 2x2 sub-matrixes when calculating matrix P. In Figure 2In the process of cutting the attention_mask, three types of sub-matrices can be obtained, namely: and Therefore, in Figure 2 In the calculation process, for the `attention_mask`, as long as these three types of submatrices can be reconstructed, the calculation of matrix P′ and `attention_mask` can be completed. The new matrix capable of reconstructing these three types of submatrices must contain matrices [0 0], [1 0], and [1 1]. The smallest matrix containing matrices [0 0], [1 0], and [1 1] can be...

[0077] In the matrix In this context, a submatrix can be obtained by stacking the first row twice in the row direction (i.e., by performing vertical stacking or row stacking). The submatrix can be obtained by stacking the third row twice in the row direction. By stacking the second and third rows in the row direction

[0078] This will allow us to obtain the submatrix. Therefore, through the matrix Can be rebuilt Figure 2 The attention_mask shown. Therefore, the matrix Can be replaced Figure 2 The attent ion_mask shown is significantly smaller than the matrix. Figure 2 The attent ion_mask is shown. Therefore, through the matrix... To replace Figure 2 The attent ion_mask shown can significantly reduce video memory usage.

[0079] Based on the above concept, this embodiment proposes a method to reduce the memory usage of the attention ion_mask. This method mainly involves replacing the original attention ion_mask (i.e., the attention ion_mask used in previous calculations) with a new attention ion_mask, such as... Figure 1The new attention mask can be obtained based on the basis block after cutting the original attention mask. At least one row or multiple rows in the new attention mask are stacked along the direction of the original row (i.e. vertical stacking or row stacking), and each basis block after cutting the original attention mask can be obtained. The maximum number of stacking is the number of rows of the basis block. In addition, if the size of the basis block is mxn, the shape of the new attention mask is (n+1)x n, that is, the number of columns of the new attention mask is the same as the number of columns of the basis block, and the number of rows of the new attention mask is equal to the number of columns of the basis block plus 1. For example, referring to FIG. 1, assuming that the original attention mask is shown in (A) of FIG. 1, if the size of the basis block is 3x3 when the matrix is blocked, the cutting result can be shown in (B) of FIG. 1. After classifying the cutting result shown in (B) of FIG. 1, three types of basis blocks shown in (C) of FIG. 1 can be obtained. Therefore, when the original attention mask is used for subsequent calculation, only the three types of basis blocks shown in (C) of FIG. 1 are needed. Further, the three types of basis blocks shown in (C) of FIG. 1 can be used to construct the new attention mask shown in (D) of FIG. 1.

[0080] The new attention mask can be obtained based on the basis block after cutting the original attention mask. At least one row or multiple rows in the new attention mask are stacked along the direction of the original row (i.e. vertical stacking or row stacking), and each basis block after cutting the original attention mask can be obtained. The maximum number of stacking is the number of rows of the basis block. In addition, if the size of the basis block is mxn, the shape of the new attention mask is (n+1)x n, that is, the number of columns of the new attention mask is the same as the number of columns of the basis block, and the number of rows of the new attention mask is equal to the number of columns of the basis block plus 1. For example, referring to FIG. 1, assuming that the original attention mask is shown in (A) of FIG. 1, if the size of the basis block is 3x3 when the matrix is blocked, the cutting result can be shown in (B) of FIG. 1. After classifying the cutting result shown in (B) of FIG. 1, three types of basis blocks shown in (C) of FIG. 1 can be obtained. Therefore, when the original attention mask is used for subsequent calculation, only the three types of basis blocks shown in (C) of FIG. 1 are needed. Further, the three types of basis blocks shown in (C) of FIG. 1 can be used to construct the new attention mask shown in (D) of FIG. 1. Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 ​​​​​​​The new attention_mask shown in (D). Wherein, Figure 3 The first row of the new attention_mask shown in (D) is stacked 3 times along the direction of the original row, which gives us the result. Figure 3 The base block of type (C) shown in the middle. Figure 3 The last row of the new attent ion_mask shown in (D) is stacked 3 times along the direction of the original row to obtain the result. Figure 3 The base block of type shown on the right side of (C). [Will] Figure 3 The second, third, and fourth rows of the new attention_mask shown in (D) are stacked once along the direction of the original rows to obtain the result. Figure 4 The basic block of type (C) shown on the left. See also... Figure 4 Since the size of the base block is 3x3, the size of the new attention_mask is 4x3.

[0081] As the above analysis shows, once the element arrangement pattern of the original `attention_mask` is known (e.g., upper triangle, lower triangle, upper triangle shifted diagonally or downwards, or lower triangle shifted diagonally or downwards, etc.), and the size of the basic blocks, a new `attention_mask` that can replace the original `attention_mask` can be calculated. The element arrangement pattern of the original `attention_mask` can be user-specified or system default (this method can also be understood as user-specified). Additionally, the new `attention_mask` can also be uploaded by the user. After obtaining the new `attention_mask`, the most important step is how to reconstruct the various basic blocks used for masking processing using the new `attention_mask`, that is, reconstruct the basic blocks used for matrix dot products with the basic blocks in matrix P′. The process of reconstructing the various basic blocks used for masking processing using the new `attention_mask` is described below.

[0082] For example, Figure 5 This illustration shows a process provided by an embodiment of the present application, in which the various basic blocks used for masking are reconstructed from a new `attention_mask` during the self-attention calculation. For example... Figure 5 As shown, the process may include the following steps:

[0083] S401. Determine the top-left corner coordinates (x, y) of the first submatrix currently involved in the calculation in matrix P′, where matrix P′ is obtained by performing a matrix cross product of matrix Q and matrix K.

[0084] In this embodiment, when performing calculations using matrix P′, a preset algorithm can be used to calculate the coordinates of each sub-matrix (i.e., basic block) obtained by cutting matrix P′, so as to obtain the coordinates of the top-left corner of the first participating sub-matrix. Here, the x-direction of the coordinates is the row direction of matrix P′, the y-direction is the column direction of matrix P′, and the origin is the top-left corner of matrix P′. For example, as... Figure 5 As shown in (A), matrix P′ is a 6x6 matrix. Dividing it into four 3x3 submatrices yields four submatrices of size 3x3. The top-left corner of matrix P′ is the origin, the row direction is the x-direction, and the column direction is the y-direction. The top-left corner coordinates of each submatrix are (0,0), (3,0), (0,3), and (3,3), respectively.

[0085] S402. Based on the top-left corner coordinates (x, y) of the first submatrix currently involved in the calculation, calculate the index of the first row of the target submatrix in the new attention_mask. The target submatrix is ​​the submatrix used to perform a matrix dot product with the first submatrix currently involved in the calculation.

[0086] In this embodiment, the top-left corner coordinates (x, y) of the first submatrix currently being calculated can be input into a pre-defined calculation formula to calculate the index of the first row of the target submatrix in the new `attention_mask`. The first row index indicates the position of the target submatrix's starting row in the new `attention_mask`. For example, if the row index of the new `attention_mask` is [0, 1, 2, 3, 4], then when the first row index is 2, it indicates that the starting row of the target submatrix is ​​the third row in the new `attention_mask`. The row index of the new `attention_mask` can be an identifier of the row in the new `attention_mask`. The target submatrix is ​​the second submatrix used to perform a matrix dot product with the first submatrix currently being calculated. For example, the pre-defined calculation formula can be: x - y + 1. (Continue reading...) Figure 5 ,like Figure 5 As shown in (B), by performing the calculation using the above formula, we can obtain the index of the first row of the starting row of each target submatrix in the new attention_mask.

[0087] It should be understood that the starting value of the row index of the new attention mask can be determined according to the situation, for example, the starting value can be 0 or 1, which is not limited here. The calculation formula of the above row index can be selected according to the actual situation, as long as the starting row of the target sub-matrix in the row index of the new attention mask can be calculated, which is not limited here. In addition, the top-left corner coordinate of the first sub-matrix can also be replaced by other coordinates, such as the bottom-right corner coordinate, etc., which can be determined according to the actual situation, which is not limited here. Wherein, S401 and S402 can be understood as calculating the starting row of the target sub-matrix in the row index of the new attention mask based on the position of the first sub-matrix currently participating in the calculation in the matrix P'.

[0088] S403, the first row index of the starting row of the target sub-matrix in the new attention mask is accumulated step by step, the step length is 1, and the number of accumulations is the number of rows m of the first sub-matrix minus 1, to calculate the row index of the remaining rows of the target sub-matrix in the new attention mask.

[0089] In this embodiment, after calculating the first row index of the starting row of the target sub-matrix in the new attention mask, the row index can be accumulated step by step. The step length of the accumulation is 1, and the number of accumulations is the number of rows m of the base block minus 1, to calculate the row index of the remaining rows of the target sub-matrix in the new attention mask. Wherein, the result of each accumulation is the row index of a row of the target sub-matrix in the new attention mask. For example, when the row index of the starting row of the target sub-matrix in the new attention mask is 2, and the number of accumulations is 2, then the row index of each row of the target sub-matrix in the new attention mask is: 2, 3 and 4. For example, continue to refer to Figure 5 , by accumulating the row index of the first index calculated in (B) of Figure 5 , the row index of each row of each target sub-matrix in the new attention mask shown in (C) of Figure 5 can be obtained. For example, this step can be understood as: based on the first row index of the starting row of the target sub-matrix in the new attention mask, calculating the row index of the remaining rows of the target sub-matrix in the new attention mask.

[0090] S404, judging whether the i-th row index contained in the row index of each row of the target sub-matrix calculated in the new attention mask is less than 0, the initial value of i is 1.

[0091] In this embodiment, after calculating the row indices of each row of the target submatrix in the new attention_mask, the row indices may be negative or greater than the maximum value of the row indices in the new attention_mask. Therefore, the calculated row indices of each row of the target submatrix in the new attention_mask can be corrected. Specifically, when the i-th row index in the calculated row indices is less than 0, step S405 can be executed; otherwise, step S406 is executed.

[0092] S405. Set the index of the i-th row to 0 and execute S408.

[0093] In this embodiment, when the index of the i-th row is less than 0, the index of the i-th row can be set to 0. For example, see below. Figure 5 ,exist Figure 5 In (C), the row indices of each row of the target submatrix that needs to be operated on with the submatrix with the top-left corner coordinate (0,3) in the new attention_mask are -2, -1, and 0. Since -2 and -1 are both less than 0, they can be set to 0, thus obtaining... Figure 5 The row indices of each row of the target submatrix shown in (D) that needs to be operated on with the submatrix with the top left corner coordinate (0,3) are 0, 0, 0 in the new attention_mask.

[0094] S406. Determine whether the index of the i-th row is greater than the maximum value of the row index of the new attention_mask.

[0095] In this embodiment, if the index of the i-th row is greater than the maximum value of the row index of the new attention_mask, S407 can be executed; otherwise, S408 is executed.

[0096] S407. Set the index of the i-th row to the maximum value of the row index of the new attention_mask, and execute S408.

[0097] In this embodiment, when the index of the i-th row is greater than the maximum value of the row index of the new attention_mask, the index of the i-th row can be set to the maximum value of the row index of the new attention_mask. For example, see [link to documentation]. Figure 5 ,exist Figure 5 In (C), the row indices of the target submatrix that needs to be operated on with the submatrix at the top left corner (3,0) in the new attribution_mask are 4, 5, and 6. Since these three row indices are all greater than 1, 2, and 3, 4, 5, and 6, respectively, the row indices of the target submatrix are 4, 5, and 6. Figure 5 The maximum value of the row index of the attent ion_mask shown in (A) is 3, so both can be set to 3, thus obtaining...Figure 6 The row indexes of the target sub-matrix in the new attention_mask, i.e. 3, 3, 3, which is shown in (D) and needs to be operated with the sub-matrix with the top-left corner coordinate (3, 0).

[0098] S408, i = i + 1, and determine whether i is less than or equal to the total number of the row indexes of the target sub-matrix in the new attention_mask.

[0099] In this embodiment, after the determination of the i-th row index is completed, i can be set to i + 1, and it is determined whether the new i is less than the total number of the row indexes of the target sub-matrix in the new attention_mask. When it is less, it indicates that there are still row indexes to be determined, and at this time, S404 can be returned to be executed; otherwise, S409 is executed.

[0100] S409, based on the row indexes of the target sub-matrix in the new attention_mask, the target sub-matrix is reconstructed from the new attention_mask, and the matrix point multiplication is performed on the first sub-matrix currently participating in the calculation in the matrix P' and the target sub-matrix.

[0101] In this embodiment, after the row indexes of the target sub-matrix in the new attention_mask are determined, the rows identified by the row indexes can be selected from the new attention_mask. Then, according to the arrangement order of the rows in the target sub-matrix, the selected rows are combined in the row direction of the target sub-matrix, for example, the selected rows are stacked in the row direction, and the target sub-matrix can be reconstructed. Then, the matrix point multiplication can be performed on the first sub-matrix currently participating in the calculation in the matrix P' and the target sub-matrix.

[0102] When the matrix point multiplication is performed on the first sub-matrix currently participating in the calculation in the matrix P' and the target sub-matrix reconstructed from the new attention_mask, according to the foregoing introduction of the matrix point multiplication, if the elements in the target sub-matrix reconstructed from the new attention_mask are all 1, the result of the matrix point multiplication of the two is the same as the first sub-matrix currently participating in the calculation in the matrix P'; if the elements in the target sub-matrix reconstructed from the new attention_mask are all 0, the result of the matrix point multiplication of the two is all 0. For example, continuing to refer to Figure 6 As Figure 6As shown in (E), the matrix multiplication of the first submatrix with top-left corner coordinates (0,3) and the target submatrix 52 reconstructed from the attention_mask results in both being 0; the matrix multiplication of the first submatrix with top-left corner coordinates (3,0) and the target submatrix 53 reconstructed from the attention_mask results in the first submatrix with top-left corner coordinates (3,0). Therefore, when a submatrix reconstructed from the attention_mask is participating in the calculation and all its elements are 0, the calculation can be skipped, and the result can be set to 0. Similarly, when a submatrix reconstructed from the attention_mask is participating in the calculation and all its elements are 1, the calculation can be skipped, and the currently participating submatrix in matrix P′ can be used as the result. In other words, when the elements of a submatrix reconstructed from the attention_mask are all 0 or all 1, that submatrix is ​​not included in the calculation. This reduces invalid calculations and improves computational efficiency. For example, please refer to... Figure 6 (A), by Figure 6 As shown in (B), the elements in the base blocks reconstructed from the new attack ion mask for matrix P′ (i.e., submatrices 61, 62, and 63) are all 1. Therefore, these can be skipped during calculation, and the base blocks 61, 62, and 63 can be directly used as the base blocks in matrix P. Similarly, the elements in the base blocks reconstructed from the new attack ion mask for matrix P′ (64, 65, and 66) are all 0. Therefore, these can be skipped during calculation, and the elements in matrix P corresponding to the base blocks 64, 65, and 66 can be set to 0. For the three base blocks 67, 68, and 69 in matrix P′ (excluding base blocks 61, 62, 63, 64, 65, and 66), matrix multiplication can be performed with the corresponding base blocks reconstructed from the new attack ion mask. Thus, in... Figure 6 The matrix P′ and the new attention_mask now only require 3 matrix multiplications, compared to 9 previously. This demonstrates a significant improvement in computational efficiency.

[0103] In addition, in the process of matrix point multiplication of the matrix P' and the new attention_mask to obtain the matrix P, if all the elements in a basic block (i.e., a sub-matrix) are 0, then the result of the matrix point multiplication of the basic block is also all 0, so there is no need to calculate, and the calculation result can be directly set to 0. Therefore, in the calculation of the matrix P', the calculation of the basic block at the same position as the basic block can be directly skipped, and all the elements in the output matrix are filled with 0. For example, referring to the original attention_mask shown in (B) of FIG. 6, the basic blocks 64, 65 and 66 in the matrix P' do not need to participate in the calculation, so in the calculation of the matrix P' in (A) of FIG. 6, the calculation of the basic blocks 64, 65 and 66 in the matrix P' can be skipped, and all the elements in the output matrix are directly filled with 0. In this way, a part of the calculation amount can be reduced, and the calculation efficiency can be improved. Similarly, referring to (A) of FIG. 6, in the calculation of the matrix O, when all the elements in the currently calculated basic block in the matrix P are 0, the calculation can also be skipped, and all the elements in the output matrix are filled with 0, so as to reduce the calculation amount and improve the calculation efficiency. Figure 6 Figure 6 Figure 7 Figure 7

[0104] In order to facilitate the system to quickly and accurately determine the basic block that does not need to participate in the calculation, the user can customize the filtering of the basic block in the input matrix and the output matrix in the calculation process through a pre-defined interface. The subsequent calculation related to these basic blocks that need to be filtered can be skipped. For example, the user can define the upper triangular basic block or the lower triangular basic block of the input matrix, or define the upper triangular basic block or the lower triangular basic block of the output matrix. Among them, Figure 7 The basic blocks 64, 65 and 66 in the matrix P' shown in (A) of FIG. 6 can be understood as the upper triangular basic block of the matrix P', and the basic blocks 61, 62 and 63 in the matrix P' can be understood as the lower triangular basic block of the matrix P'. For example, the pre-defined interface can be: Matmul(A, B, C, lowerTriangleA, lowerTriangleB, lowerTriangleC). Among them, A is the input left matrix; B is the input right matrix; C is the output matrix; lowerTriangleA is a Boolean type, indicating whether to filter the upper triangular of the input left matrix; lowerTriangleB is a Boolean type, indicating whether to filter the upper triangular of the input right matrix; lowerTriangleC is a Boolean type, indicating whether to filter the upper triangular of the output matrix. For example, as shown in (B) of FIG. 6, in the calculation of the matrix P', the calculation of the basic blocks 64, 65 and 66 in the matrix P' can be skipped, and all the elements in the output matrix are filled with 0. Figure 7 Figure 7 ​​​​​In (A) of FIG. 1, the input matrix is non-triangular and the output matrix is triangular, at this time, the calculation related to the data in the gray part of the output matrix can be skipped and the result is automatically filled as 0. In (B) of FIG. 1, both the input matrix and the output matrix are triangular, at this time, the calculation related to the data in the gray part of the input matrix and the output matrix can be skipped and the result is automatically filled as 0. In (C) of FIG. 1, the left input matrix is triangular and the right input matrix and the output matrix are non-triangular, at this time, the calculation related to the data in the gray part of the left input matrix is skipped. In (D) of FIG. 1, the right input matrix is triangular and the left input matrix and the output matrix are non-triangular, at this time, the calculation related to the data in the gray part of the right input matrix is skipped. Based on this, the user can specify the invalid data in the input matrix and / or the output matrix. Wherein, the input matrix is the left matrix and / or the right matrix participating in the matrix operation, and the output matrix is the result of the matrix operation on the left matrix and the right matrix. And in the case of matrix operation on any two of the Q matrix, the K matrix, the V matrix and the new attention mask, the calculation related to the invalid data is skipped. Thus, the calculation efficiency is improved. Figure 8 Figure 8 Figure 8

[0105] In the process of self-attention calculation, when the calculation is performed by performing matrix blocking on the input matrix, parallel calculation of multiple basic blocks will be involved, at this time, if the system contains multiple calculation units, the calculation of multiple basic blocks can be evenly distributed to each calculation unit to ensure that the calculation amount of each calculation unit is uniform and improve the calculation efficiency. For example, as shown in (A) of FIG. 2, there are 6 basic blocks participating in the calculation in a certain input matrix, and there are 3 calculation units in the system. In (A) of FIG. 2, each calculation unit processes the calculation of two basic blocks, and in (B) of FIG. 2, calculation unit 1 processes the calculation of 1 basic block, calculation unit 2 processes the calculation of 2 basic blocks, and calculation unit 3 processes the calculation of 3 basic blocks. In (A) of FIG. 2, if the time for each calculation unit to process 1 basic block is t, then the total time required for the basic block processing mode shown in (A) of FIG. 2 is 2t, and the total time required for the basic block processing mode shown in (B) of FIG. 2 is 3t. It can be seen that evenly distributing the calculation of multiple basic blocks to each calculation unit can shorten the calculation time and improve the calculation efficiency. Figure 8 Figure 8 Figure 8 Figure 9 Figure 9 Figure 4

[0106] ​​​​​​​​​In the embodiment, the new attention_mask can be in other forms in addition to the aforementioned forms. For example, the new attention_mask has a row number of (a*s) and a column number of (b*S), a and b are both integers and are both greater than or equal to 2, and S is the maximum of the row number and the column number of the first sub-matrix in the aforementioned matrix P' that is currently involved in the calculation. For example, when the shape of the first sub-matrix is m*n, the shape of the new attention_mask is (a*max(m, n))x(b*max(m, n)). At this time, each row in the second sub-matrix that needs to be multiplied by the first sub-matrix can be constructed by sequentially adjacent elements in a row of the new attention_mask. The row number of the new attention_mask is less than the row number of the Q matrix. When the K matrix is the matrix before transposition, that is, the K matrix needs to be transposed to be multiplied by the Q matrix, the column number of the new attention_mask is less than the row number of the K matrix, and when the K matrix is the matrix after transposition, that is, the K matrix can be multiplied by the Q matrix without transposition, the column number of the new attention_mask is less than the column number of the K matrix. In this way, the size of the new attention_mask is smaller than the size of the original attention_mask, so that the use of the new attention_mask can reduce the memory occupation. For example, when the row number of the attention_mask is less than or equal to the column number of the attention_mask, the number of elements 1 in the s-th row of the attention_mask is (s+1), 0≤s, and the elements 1 in each row of the attention_mask are sequentially adjacent from left to right. When the row number of the attention_mask is greater than the column number of the attention_mask, the number of elements 1 in the s-th row of the attention_mask is (s+1), 0≤s

[0107] For example, Figure 10 Another process of reconstructing each basic block used for mask processing by a new attention_mask in the process of self-attention calculation is shown in the embodiment of the application. As shown in FIG. 6, the process of reconstructing each basic block used for mask processing by a new attention_mask in the process of self-attention calculation includes the following steps. Figure 10As shown, the process can include the following steps:

[0108] S901, determine the top-left corner coordinates (x, y) of the first sub-matrix currently participating in the calculation in the matrix P', wherein the matrix P' is obtained by matrix cross-multiplication according to the Q matrix and the K matrix. For details, see the description in S401 above, which will not be repeated here. Figure 10

[0109] S902, based on the top-left corner coordinates (x, y) of the first sub-matrix, calculate the first row index of the first target row and the first column index of the first target column in the new attention_mask, wherein the first target row contains the starting row of the target sub-matrix, and the first target column contains the starting column of the target sub-matrix. The target sub-matrix is a sub-matrix used for matrix point multiplication with the first sub-matrix currently participating in the calculation.

[0110] In this embodiment, the top-left corner coordinates (x, y) of the first sub-matrix currently participating in the calculation can be input into a pre-set calculation formula to calculate the first row index of the first target row and the first column index of the first target column in the new attention_mask. Wherein the first target row contains the starting row of the target sub-matrix, and the first target column contains the starting column of the target sub-matrix. For example, if the row index of the new attention_mask is [0, 1, 2, 3, 4] and the column index is [0, 1, 2, 3, 4], when the first row index is 2, it means that the starting row of the target sub-matrix is in the third row of the new attention_mask, and when the first column index is 1, it means that the starting column of the target sub-matrix is in the second column of the new attention_mask. The row index of the new attention_mask can be the identifier of the row of the new attention_mask, and the column index of the new attention_mask can be the identifier of the column of the new attention_mask. In addition, the target sub-matrix is a sub-matrix used for matrix point multiplication with the first sub-matrix currently participating in the calculation. For example, if the shape of the first sub-matrix is m x n, when x-y<0, the pre-set calculation formula can be: first row index = 0 (i.e. the starting row of the new attention_mask), first column index = max(-(x-y), m); when x-y≥0, the pre-set calculation formula can be: first row index = min((x-y), n), first column index = 0 (i.e. the starting column of the new attention_mask).

[0111] S903, select a matrix with the same shape as the first sub-matrix from the new attention_mask based on the first row index and the first column index to obtain the target sub-matrix.

[0112] ​In this embodiment, after obtaining the first row index and the first column index, a matrix with the same shape as the first sub-matrix can be selected from the new attention_mask based on these two indices to obtain the target sub-matrix. For example, as... Figure 10 As shown, when the new attention_mask is Figure 10 As shown in (A), if the shape of the first submatrix is ​​3x4, and the calculated row index = 1 and column index = 0, then the following can be filtered out: Figure 10 The target submatrix is ​​shown in region 1004 of (D). For example, when calculating using the top-left corner coordinates of the first submatrix, if the shape of the first submatrix is ​​m×n, in obtaining the target submatrix, one can start from the element indicated by the first row index and the first column index, and continuously take n numbers from left to right in the row indicated by the first row index to obtain the first row of the target submatrix. Then, the first row index is accumulated with a step size of 1, and the number of accumulations is the number of rows m of the first submatrix minus 1. Each time a row index is obtained through accumulation, one can start from the element indicated by that row index and the first column index, and continuously take n numbers from left to right in the row indicated by that row index to obtain a row of the target submatrix. For example, see [link to documentation]. Figure 10 ,exist Figure 10 In (A), we can start from the first element of the first row of the new attention_mask and take four consecutive numbers from left to right to obtain the result. Figure 10 The row of elements shown in region 1001 of (B) is the first row of the target submatrix. Then, starting from the first element of the second row of the new attention_mask, four consecutive numbers are taken from left to right to obtain the result. Figure 10 The row of elements shown in region 1002 of (C) is the second row of the target submatrix. Ideally, starting from the first element of the third row of the new attention_mask, take four consecutive numbers from left to right to obtain the desired result. Figure 9 The row of elements shown in region 1003 of (D) is the third row of the target submatrix. Figure 11 By stacking the rows identified by regions 1001, 1002, and 1003 in the row direction, we can obtain... Figure 11 The target submatrix shown in (D) is 1004. Of course, the rows and columns to be selected are selected from the new attention_mask, and then the overlapping part of the selected rows and columns is used as the target submatrix. The specific selection can be determined according to the actual situation, and is not limited here.

[0113] S904. Perform matrix dot product on the first submatrix currently being calculated and the target submatrix.

[0114] In this embodiment, after the target sub-matrix is determined, the first sub-matrix currently participating in the calculation in the matrix P' and the target sub-matrix can be multiplied.

[0115] It should be understood that, Figure 4 The left upper corner coordinate of the first sub-matrix is taken as an example to introduce how to reconstruct the target sub-matrix. Of course, the left upper corner coordinate of the first sub-matrix can be replaced by the right upper corner coordinate, the right lower corner coordinate, and the like. After the replacement, the selection mode of the rows and columns involved in the step can be adaptively changed, and the changed scheme is still within the protection scope of the present application.

[0116] Next, based on the above content, a data processing method provided by an embodiment of the present application is introduced.

[0117] Exemplarily, Figure 9 A flowchart of a data processing method provided by an embodiment of the present application is shown. It can be understood that the method can be executed by any device, equipment, platform, device cluster having computing and processing capabilities. Exemplarily, the method can be executed by a data processing device, wherein the device can be realized by software and / or hardware, can be configured in a computing device, and typically can be configured in a server. As shown in the figure, Figure 4 The data processing method can include the following steps:

[0118] S1101, obtain a Q matrix, a K matrix and a V matrix, wherein the Q matrix, the K matrix and the V matrix are calculated based on an input sequence. In this embodiment, the Q matrix, the K matrix and the V matrix can be externally transmitted or stored in the GPU, which is not limited here.

[0119] S1102, read an attention_mask from the memory of the GPU, wherein the attention_mask is generated by an element arrangement rule uploaded or specified by a user. Exemplarily, the attention_mask can be the aforementioned new attention_mask.

[0120] In this embodiment, the attention_mask can be stored in the GPU. When the attention_mask is needed for calculation, the attention_mask can be read from the GPU. The attention_mask can be uploaded by the user or generated by an element arrangement rule (such as upper triangle, lower triangle, upper triangle with diagonal line moving up or down, or lower triangle with diagonal line moving up or down, etc.) specified by the user.

[0121] S1103, based on the attention_mask, performing calculation on the Q matrix, the K matrix and the V matrix to obtain an inference result related to the input sequence, wherein in the calculation process, the attention_mask is used to reconstruct a second sub-matrix, the second sub-matrix is used to perform matrix point multiplication with a first sub-matrix to obtain an intermediate result, the intermediate result is used to perform calculation with the V matrix to obtain the inference result, the first sub-matrix is obtained based on matrix block processing on a matrix P', and the matrix P' is obtained by performing matrix cross multiplication on the Q matrix and the K matrix. Illustratively, the second sub-matrix can be understood as being used to obtain the inference result. In the second sub-matrix, each row is selected from the attention_mask; or in the second sub-matrix, each row is constructed by sequentially adjacent elements in a row of the attention_mask, and the number of rows of the attention_mask is less than the number of rows of the Q matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix.

[0122] In this embodiment, after reading the attention_mask, the Q matrix, the K matrix and the V matrix can be calculated based on the attention_mask to obtain an inference result related to the input sequence. Among them, the Q matrix and the K matrix can be first cross-multiplied to obtain a matrix P'. Then, the matrix P' and the attention_mask are multiplied by the matrix, and the matrix point multiplication result is processed by the softmax function to obtain a matrix P. Finally, the matrix P and the V matrix are cross-multiplied to obtain the inference result. When the matrix P' and the attention_mask are multiplied by the matrix, the matrix P' can be divided into blocks, and then each first sub-matrix obtained by the matrix block processing is calculated. When the first sub-matrix obtained by the matrix block processing is calculated, the second sub-matrix which needs to be multiplied by each first sub-matrix is reconstructed from the attention_mask, and then the two sub-matrices are multiplied by the matrix. For the reconstruction process, please refer to the description in the foregoing Figure 5 or Figure 9 .

[0123] In addition, from Figure 1As described in the text, during the calculation process, the attention_mask provided in this embodiment can be used to reconstruct the second sub-matrix, and each row in the second sub-matrix is ​​a row selected from the attention_mask. This second sub-matrix can be used to perform a matrix dot product with the first sub-matrix to obtain an intermediate result. This intermediate result can be used to perform calculations with the V matrix (e.g., performing a matrix cross product with a sub-matrix contained in the V matrix) to obtain the inference result. The first sub-matrix is ​​obtained by matrix partitioning of the matrix P′. For example, each first sub-matrix obtained by matrix partitioning the matrix P′ can have a second sub-matrix with which it is multiplied by a matrix dot product. Furthermore, in some implementations, the shape of the attention_mask can be (n+1)×n, that is, (n+1) rows and n columns, where n is the number of columns in the second sub-matrix reconstructed from the attention_mask. Furthermore, the number of elements 1 in the s-th row of the `attent ion_mask` can be s, where 0 ≤ s ≤ n, and in each row of the `attent ion_mask` containing elements 1, the elements 1 are distributed sequentially from left to right. For example, as shown... Figure 11 As shown in (A), the 0th row of the attent ion_mask is all 0, the 1st row has one element 1 on the far left, the 2nd row has two elements 1 arranged adjacently from left to right, and the 3rd row has three elements 1 arranged adjacently from left to right.

[0124] And by Figure 12 As described in the text, during the calculation process, the `attention_mask` provided in this embodiment can be used to reconstruct the second submatrix. Each row of this second submatrix is ​​constructed by sequentially connecting adjacent elements in a row of the `attention_mask`, and the number of rows in the `attention_mask` is less than the number of rows in the Q matrix, and the number of columns in the `attention_mask` is less than the number of rows or columns in the K matrix. Specifically, when the K matrix is ​​the matrix before its transpose, the number of columns in the `attention_mask` is less than the number of rows in the K matrix, and when the K matrix is ​​the matrix after its transpose, the number of columns in the `attention_mask` is less than the number of columns in the K matrix.

[0125] Thus, during the computation process, the second sub-matrix can be reconstructed using the attention_mask, which can then be used for masking. Since the number of columns in the attention_mask used is the same as the number of columns in the second sub-matrix, or the number of rows in the attention_mask is less than the number of rows in the Q matrix and the number of columns is less than the number of rows or columns in the K matrix, the size of the attention_mask is smaller than the size of the matrix P′ obtained from the Q and K matrices. Previously used attention_masks (such as...) Figure 12 The size of the attention_mask used in the data processing method provided in this application is the same as the size of the matrix P′. Therefore, the size of the attention_mask used in the data processing method provided in this application is also smaller than the attention_mask used previously. Thus, the memory usage of the attention_mask can be reduced by the data processing method provided in this application.

[0126] In some embodiments, prior to S1103, the user may specify invalid data in the input matrix and / or output matrix. The input matrix is ​​the left and / or right matrix involved in the matrix operation, and the output matrix is ​​the result of the matrix operation performed on the left and right matrices. Then, in S1103, the Q matrix, K matrix, V matrix, and... Figure 12 When performing matrix operations on any two elements of the `attent_mask` used in the algorithm, calculations related to invalid data are skipped. This allows some invalid calculations to be skipped, thereby improving computational efficiency.

[0127] It's understandable that the number of rows in an `attent ion_mask` can be interpreted as the total number of rows in the `attent ion_mask`, and the number of columns in an `attent ion_mask` can be interpreted as the total number of columns in the `attent ion_mask`. Similarly, the number of rows in a matrix can be interpreted as the total number of rows in the matrix, and the number of columns in a matrix can be interpreted as the total number of columns in the matrix.

[0128] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments described above can be combined according to actual conditions, and the combined solutions are still within the protection scope of this application.

[0129] Based on the methods in the above embodiments, this application also provides a data processing apparatus.

[0130] For example, Figure 12A structural schematic diagram of a data processing apparatus is shown. As shown in Figure 13 The data processing apparatus 1200 includes a communication module 1201 and a processing module 1202. The communication module 1201 is configured to obtain a Q matrix, a K matrix, and a V matrix, wherein the Q matrix, the K matrix, and the V matrix are calculated based on an input sequence. The processing module 1202 is configured to read an attention_mask from a memory of a GPU, the attention_mask being generated according to an element arrangement rule specified by a user or uploaded by the user, and perform calculation on the Q matrix, the K matrix, and the V matrix based on the attention_mask to obtain an inference result related to the input sequence, wherein the attention_mask is used to reconstruct a second submatrix, the second submatrix is used to obtain the inference result, each row in the second submatrix is selected from the attention_mask, or each row in the second submatrix is constructed by sequentially adjacent elements in a row of the attention_mask, the number of rows of the attention_mask is less than the number of rows of the Q matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix.

[0131] In some embodiments, the second submatrix is used to perform matrix point multiplication with a first submatrix, the first submatrix is obtained based on matrix block decomposition on a matrix P', and the matrix P' is obtained by performing matrix cross multiplication on the Q matrix and the K matrix.

[0132] In some embodiments, each row in the second submatrix is selected from the attention_mask, the number of columns of the attention_mask is n, the number of rows of the attention_mask is (n+1), and n is the number of columns of the second submatrix.

[0133] In some embodiments, each row in the second submatrix is selected from the attention_mask, the number of elements 1 in the s-th row of the attention_mask is s, 0≤s≤n, and the elements 1 are sequentially adjacent from left to right in each row of the attention_mask containing the elements 1.

[0134] In some embodiments, each row in the second submatrix is constructed by sequentially adjacent elements in a row of the attention_mask, the number of rows of the attention_mask is less than the number of rows of the Q matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix. At this time, the number of rows of the attention_mask is (a*S), the number of columns of the attention_mask is (b*S), S is the maximum of the number of rows and the number of columns of the second submatrix, a and b are both integers and are both greater than or equal to 2.

[0135] In some embodiments, each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention_mask, and the number of rows of the attention_mask is less than the number of rows of the QW matrix, and the number of columns of the attention_mask is less than the number of rows or columns of the K matrix. When the number of rows of the attention_mask is less than or equal to the number of columns of the attention_mask, the number of elements 1 in the s-th row of the attention_mask is (s+1), 0≤s, and the elements 1 in each row of the attention_mask are sequentially adjacent from left to right. When the number of rows of the attention_mask is greater than the number of columns of the attention_mask, the number of elements 1 in the s-th row of the attention_mask is (s+1), 0≤s

[0136] In some embodiments, the element arrangement rule includes upper triangle, lower triangle, upper triangle with diagonal line moving up or down, or lower triangle with diagonal line moving up or down.

[0137] In some embodiments, when the processing module 1202 performs calculation on the Q matrix, the K matrix and the V matrix based on the attention_mask, the processing module 1202 is specifically configured to: perform matrix cross multiplication on the Q matrix and the K matrix to obtain a matrix P'; perform matrix point multiplication on the matrix P' and the attention_mask, and perform processing on the matrix point multiplication result by using a softmax function to obtain a matrix P; and perform matrix cross multiplication on the matrix P and the V matrix to obtain an inference result.

[0138] In some embodiments, each row in the second sub-matrix is a row selected from the attention_mask. At this time, when the processing module 1202 performs the matrix point multiplication on the matrix P' and the attention_mask, it is specifically configured to: based on the position of the first sub-matrix currently participating in the calculation in the matrix P' in the matrix P', calculate a first row index of a starting row of a target sub-matrix in the attention_mask, the target sub-matrix being a second sub-matrix used for the matrix point multiplication with the first sub-matrix currently participating in the calculation, the first row index being used to indicate the position of the starting row of the target sub-matrix in the attention_mask; based on the first row index, calculate row indexes of the remaining rows of the target sub-matrix in the attention_mask; based on the row indexes of each row of the target sub-matrix in the attention_mask, select the rows identified by the row indexes from the attention_mask, and combine the rows selected from the attention_mask in the row direction of the target sub-matrix according to the arrangement order of the rows in the target sub-matrix to obtain the target sub-matrix; and perform the matrix point multiplication on the first sub-matrix currently participating in the calculation in the matrix P' and the target sub-matrix.

[0139] In some embodiments, before the processing module 1202 selects the rows identified by the row indexes from the attention_mask based on the row indexes of each row of the target sub-matrix in the attention_mask, and combines the rows selected from the attention_mask in the row direction of the target sub-matrix according to the arrangement order of the rows in the target sub-matrix to obtain the target sub-matrix, it is further configured to: in the case that the row index of any row of the target sub-matrix in the attention_mask is less than 0, set the row index of the any row in the attention_mask to 0; and in the case that the row index of any row of the target sub-matrix in the attention_mask is greater than the maximum value of the row indexes of the rows in the attention_mask, set the row index of the any row in the attention_mask to the maximum value of the row indexes of the rows in the attention_mask.

[0140] In some embodiments, each row in the second sub-matrix is constructed by sequentially adjacent elements in a row of the attention mask, and the number of rows of the attention mask is less than the number of rows of the Q matrix, and the number of columns of the attention mask is less than the number of rows of the K matrix. At this time, when the processing module 1202 performs matrix multiplication on the matrix P' and the attention mask, it is specifically used for: based on the position of the first sub-matrix currently participating in the calculation in the matrix P', calculating a first row index of a first target row and a first column index of a first target column in the attention mask, wherein the first target row includes the starting row of the target sub-matrix, the first target column includes the starting column of the target sub-matrix, and the target sub-matrix is a sub-matrix used for matrix multiplication with the first sub-matrix currently participating in the calculation; selecting a matrix with the same shape as the first sub-matrix from the attention mask based on the first row index and the first column index to obtain the target sub-matrix; and performing matrix multiplication on the first sub-matrix currently participating in the calculation and the target sub-matrix.

[0141] In some embodiments, when the processing module 1202 performs matrix multiplication on the first sub-matrix currently participating in the calculation and the target sub-matrix in the matrix P', it is specifically used for: in the case that all elements of the target sub-matrix are 0, directly setting the calculation result to 0; and in the case that all elements of the target sub-matrix are 1, directly taking the first sub-matrix currently participating in the calculation in the matrix P' as the calculation result.

[0142] In some embodiments, before the processing module 1202 performs calculation on the Q matrix, the K matrix and the V matrix based on the attention mask, the communication module 1201 is further used for: receiving the user's designation of invalid data in the input matrix and / or the output matrix, the input matrix being a left matrix and / or a right matrix participating in matrix operation, and the output matrix being a result of matrix operation on the left matrix and the right matrix. At this time, when the processing module 1202 performs calculation on the Q matrix, the K matrix and the V matrix based on the attention mask, it is specifically used for: in the case that any two of the Q matrix, the K matrix, the V matrix and the attention mask are subjected to matrix operation, skipping the calculation related to the invalid data.

[0143] In some embodiments, when the K matrix is a matrix before transposition, i.e., the K matrix needs to be transposed to perform matrix multiplication with the Q matrix, the number of columns of the attention mask is less than the number of rows of the K matrix, and when the K matrix is a matrix after transposition, i.e., the K matrix does not need to be transposed to perform matrix multiplication with the Q matrix, the number of columns of the attention mask is less than the number of columns of the K matrix.

[0144] In some embodiments, the rows in the second sub-matrix are selected from consecutive rows of the attention_mask, and the columns in the second sub-matrix are selected from consecutive columns of the attention_mask. Wherein, the first element of each row in the second sub-matrix is in the same column of the attention_mask.

[0145] In some embodiments, Figure 13 The communication module 1201 and the processing module 1202 shown in FIG. 12 can be implemented by software or by hardware. For example, the implementation of the communication module 1201 is described below. Similarly, the implementation of the processing module 1202 can refer to the implementation of the communication module 1201.

[0146] As an example of a software functional unit, the communication module 1201 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the communication module 1201 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs. Each AZ includes one data center or multiple data centers in close geographical proximity. Generally, one region can include multiple AZs.

[0147] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Generally, one VPC is set up in one region, and communication between two VPCs in the same region, or between VPCs in different regions, requires a communication gateway in each VPC to realize the interconnection between VPCs.

[0148] As an example of a hardware functional unit, the communication module 1201 may include at least one computing device, such as a server. Alternatively, the communication module 1201 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0149] The multiple computing devices included in the communication module 1201 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the communication module 1201 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the communication module 1201 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0150] It should be noted that, in other embodiments, the communication module 1201 can be used to execute any step in the data processing method described in the above embodiments, and the processing module 1202 can also be used to execute any step in the data processing method described in the above embodiments. Furthermore, the communication module 1201 can also be combined with the processing module 1202 to be responsible for executing any step in the data processing method described in the above embodiments. In addition, the steps implemented by the communication module 1201 and the processing module 1202 can also be specified as needed, and different steps in the data processing method described in the above embodiments can be implemented by the communication module 1201 and the processing module 1202 respectively. Figure 12 The data processing device 1200 shown has all the functions of the data processing device.

[0151] This application also provides a computing device 1300. For example... Figure 12 As shown, the computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate with each other via the bus 1302. The computing device 1300 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1300.

[0152] Bus 1302 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, or the like. For ease of representation, Figure 14 Only one line is used to represent the bus in the figure, but this does not mean that there is only one bus or only one type of bus. Bus 1304 can include paths for communicating information among the various components of computing device 1300 (e.g., memory 1306, processor 1304, communication interface 1308).

[0153] Processor 1304 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other processing component(s). Processor 1304 can include single or multiple programmable processing units.

[0154] Memory 1306 can include a volatile memory (e.g., random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM)), a flash memory, a mechanical hard drive (HDD), or a solid-state drive (SSD), among others. Memory 1306 can include a memory buffer, a cache, or the like.

[0155] The executable program code stored in memory 1306 is executed by processor 1304 to respectively implement the functions of communication module 1201 and processing module 1202 shown in the foregoing Figure 12 embodiments, thereby implementing the data processing method described in the foregoing embodiments. That is, memory 1306 stores instructions for executing the data processing method described in the foregoing embodiments.

[0156] Alternatively, the executable program code stored in memory 1306 is executed by processor 1304 to respectively implement the functions of data processing apparatus 1200 shown in the foregoing Figure 15 embodiments, thereby implementing the data processing method described in the foregoing embodiments. That is, memory 1306 stores instructions for executing the data processing method described in the foregoing embodiments.

[0157] The communication interface 1308 enables communication among the computing device 1300 and other devices or communication networks using, for example but not limited to, a transceiver module such as a network interface card, a Bluetooth® transceiver, a Bluetooth® Low Energy transceiver, a Near Field Communication transceiver, or the like.

[0158] The embodiments of the present disclosure also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0159] As shown in Figure 15 , the computing device cluster includes at least one computing device 1300. The memory 1306 in one or more computing devices 1300 in the computing device cluster can store the same instructions for performing the data processing method described in the above embodiments.

[0160] In some possible implementations, the memory 1306 in one or more computing devices 1300 in the computing device cluster can also respectively store partial instructions for performing the data processing method described in the above embodiments. In other words, the combination of one or more computing devices 1300 can collectively execute the instructions for performing the data processing method described in the above embodiments.

[0161] It should be noted that the memory 1306 in different computing devices 1300 in the computing device cluster can store different instructions, respectively used to perform the partial functions of the data processing apparatus 1200 described in the above embodiments. That is, the instructions stored in the memory 1306 in different computing devices 1300 can implement the functions of one or more modules in the communication module 1201 and the processing module 1202. Figure 15

[0162] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. Figure 14 A possible implementation is shown. As shown in Figure 15 , two computing devices 1300A and 1300B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation, the memory 1306 in the computing device 1300A stores instructions for performing the functions of the communication module 1201. Meanwhile, the memory 1306 in the computing device 1300B stores instructions for performing the functions of the processing module 1202.

[0163] It should be understood that Figure 16 ​The functions of the computing device 1300A shown in FIG. 13A can also be completed by a plurality of computing devices 1300. Similarly, the functions of the computing device 1300B can also be completed by a plurality of computing devices 1300.

[0164] The present embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manner of the computing device cluster as described above. Figure 16 and Figure 16 The difference is that the same instructions for executing the method in the above embodiment can be stored in the memory 1306 of one or more computing devices 1300 in the computing device cluster.

[0165] In some possible implementation manners, part of the instructions for executing the aforementioned data processing method can also be respectively stored in the memory 1306 of one or more computing devices 1300 in the computing device cluster. In other words, the combination of one or more computing devices 1300 can collectively execute the instructions for executing the aforementioned data processing method.

[0166] Based on the method in the above embodiment, the present embodiment also provides a data processing apparatus. Please refer to ​ , ​ for a structural schematic diagram of another data processing apparatus provided by the present embodiment. As shown in ​ , the data processing apparatus 1600 includes one or more processors 1601 and interface circuits 1602. Optionally, the data processing apparatus 1600 can also include a bus 1603. Wherein:

[0167] The processor 1601 can be an integrated circuit chip with processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit or the instruction in the form of software in the processor 1601. The processor 1601 described above can be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and process disclosed in the present embodiment can be implemented or executed. The general purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0168] The interface circuit 1602 can be used for sending or receiving data, instructions or information. The processor 1601 can process the data, instructions or other information received by the interface circuit 1602, and can send the processed information out through the interface circuit 1602.

[0169] Optionally, the data processing device 1600 further includes a memory, which can include a read-only memory and a random access memory, and provides the processor with operation instructions and data. A part of the memory can also include a non-volatile random access memory (NVRAM).

[0170] Optionally, the memory stores executable software modules or data structures, and the processor can perform corresponding operations by calling operation instructions stored in the memory (which can be stored in an operating system).

[0171] Optionally, the interface circuit 1602 can be used to output the execution result of the processor 1601.

[0172] It should be noted that the functions of the processor 1601 and the interface circuit 1602 respectively can be realized by hardware design, software design, or a combination of hardware and software, which is not limited here.

[0173] It should be understood that each step of the above method embodiments can be completed by a logic circuit in the form of hardware in the processor or instructions in the form of software.

[0174] Based on the method in the above embodiments, the embodiments of the present application provide a computer-readable storage medium including computer program instructions, when the computer program instructions are executed by a computing device cluster including at least one computing device, the computing device cluster executes the method in the above embodiments. Illustratively, the computer-readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center including one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc.

[0175] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product including instructions, when the instructions are run by a computing device, the computing device cluster including at least one computing device executes the method in the above embodiments.

[0176] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0177] The method steps in the embodiments of the present application can be realized by hardware or by the processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0178] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0179] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0180] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the query vector matrix Q, the key vector matrix K, and the value vector matrix V, wherein the Q matrix, the K matrix, and the V matrix are all calculated based on the input sequence; Read the attention mask (attention_mask) from the memory of the graphics processing unit (GPU). The attention_mask is generated by the user-uploaded element arrangement rule or by the user-specified element arrangement rule. Based on the attention_mask, calculations are performed on the Q matrix, the K matrix, and the V matrix to obtain an inference result related to the input sequence. The attention_mask is used to reconstruct a second sub-matrix, which is used to obtain the inference result. Each row in the second sub-matrix is ​​a row selected from the attention_mask; or, each row in the second sub-matrix is ​​constructed by sequentially adjacent elements in a row of the attention_mask, and the number of rows in the attention_mask is less than the number of rows in the Q matrix, and the number of columns in the attention_mask is less than the number of rows or columns in the K matrix.

2. The method according to claim 1, characterized in that, The second submatrix is ​​used to perform a matrix dot product with the first submatrix to obtain an intermediate result. The intermediate result is used to perform calculations with the V matrix to obtain the inference result. The first submatrix is ​​obtained by matrix partitioning of matrix P′, and matrix P′ is obtained by performing a matrix cross product of the Q matrix and the K matrix.

3. The method according to claim 1 or 2, characterized in that, Each row in the second submatrix is ​​a row selected from the attention_mask, which has n columns and (n+1) rows, where n is the number of columns in the second submatrix.

4. The method according to any one of claims 1-3, characterized in that, Each row in the second submatrix is ​​a row selected from the attention_mask. The s-th row of the attention_mask contains s elements of type 1, where 0 ≤ s ≤ n. In each row of the attention_mask containing element 1, element 1 is distributed sequentially from left to right.

5. The method according to claim 1 or 2, characterized in that, Each row in the second sub-matrix is ​​constructed by sequentially connecting adjacent elements in a row of the attention_mask, and the number of rows in the attention_mask is less than the number of rows in the Q matrix, and the number of columns in the attention_mask is less than the number of rows or columns in the K matrix; Wherein, the number of rows of the attention_mask is (a*S), the number of columns of the attention_mask is (b*S), S is the largest of the number of rows and columns of the second sub-matrix, and a and b are both integers, and both are greater than or equal to 2.

6. The method according to any one of claims 1-5, characterized in that, The arrangement rules of the elements include upper triangles, lower triangles, upper triangles that shift diagonally upwards or downwards, or lower triangles that shift diagonally upwards or downwards.

7. The method according to any one of claims 1-6, characterized in that, The calculation of the Q matrix, K matrix, and V matrix based on the attention_mask includes: Perform a matrix cross product on the Q matrix and the K matrix to obtain the matrix P′; Perform a matrix dot product on matrix P′ and the attention_mask, and process the matrix dot product result using the softmax function to obtain matrix P; Perform a matrix cross product on the matrix P and the matrix V to obtain the inference result.

8. The method according to claim 7, characterized in that, Each row in the second submatrix is ​​a row selected from the attention_mask; The matrix dot product of matrix P′ and attention_mask includes: Based on the position of the first submatrix currently participating in the calculation in matrix P′, the index of the first row of the target submatrix in the attention_mask is calculated. The target submatrix is ​​a second submatrix used to perform a matrix dot product with the first submatrix currently participating in the calculation. The first row index is used to indicate the position of the starting row of the target submatrix in the attention_mask. Based on the first row index, calculate the row index of each remaining row of the target sub-matrix in the attention_mask; Based on the row index of each row of the target submatrix in the attention_mask, the rows identified by each row index are selected from the attention_mask, and the rows selected from the attention_mask are combined in the row direction of the target submatrix according to the row arrangement order in the target submatrix to obtain the target submatrix. Perform a matrix dot product on the first submatrix currently involved in the calculation and the target submatrix.

9. The method according to claim 8, characterized in that, Before obtaining the target submatrix, based on the row index of each row in the attention_mask, selecting the row identified by each row index from the attention_mask, and combining the rows selected from the attention_mask in the row direction of the target submatrix according to the row arrangement order in the target submatrix, the process further includes: If the row index of any row in the target submatrix is ​​less than 0 in the attention_mask, the row index of that row in the attention_mask is set to 0. If the row index of any row in the target submatrix is ​​greater than the maximum value of the row index in the attention_mask, then the row index of that row in the attention_mask is set to the maximum value of the row index in the attention_mask.

10. The method according to claim 7, characterized in that, Each row in the second sub-matrix is ​​constructed by sequentially connecting adjacent elements in a row of the attention_mask, and the number of rows in the attention_mask is less than the number of rows in the Q matrix, and the number of columns in the attention_mask is less than the number of rows or columns in the K matrix; The matrix dot product of matrix P′ and attention_mask includes: Based on the position of the first submatrix currently participating in the calculation in matrix P′, the first row index of the first target row and the first column index of the first target column in the attention_mask are calculated, wherein the first target row contains the starting row of the target submatrix, the first target column contains the starting column of the target submatrix, and the target submatrix is ​​a submatrix used to perform a matrix dot product with the first submatrix; Based on the first row index and the first column index, a matrix with the same shape as the first sub-matrix is ​​selected from the attention_mask to obtain the target sub-matrix; Perform a matrix dot product on the first submatrix and the target submatrix.

11. The method according to any one of claims 8-10, characterized in that, The matrix dot product of the first submatrix currently being calculated and the target submatrix includes: If all elements of the target submatrix are 0, the calculation result is directly set to 0; If all elements of the target submatrix are 1, the first submatrix currently involved in the calculation is directly used as the calculation result.

12. The method according to any one of claims 1-11, characterized in that, Before calculating the Q matrix, K matrix, and V matrix based on the attention_mask, the following steps are also included: The system receives a user's specification of invalid data in the input matrix and / or output matrix, wherein the input matrix is ​​the left matrix and / or right matrix participating in matrix operations, and the output matrix is ​​the result of matrix operations performed on the left matrix and right matrix. The calculation of the Q matrix, K matrix, and V matrix based on the attention_mask includes: When performing matrix operations on any two of the Q matrix, K matrix, V matrix, and attention_mask, calculations related to the invalid data are skipped.

13. The method according to any one of claims 1-12, characterized in that, When the K matrix is ​​the matrix before transpose, the number of columns of the attention_mask is less than the number of rows of the K matrix; When the K matrix is ​​the transposed matrix, the number of columns of the attention_mask is less than the number of columns of the K matrix.

14. The method according to any one of claims 1-13, characterized in that, The rows in the second submatrix are selected from consecutive rows of the attention_mask, and the columns in the second submatrix are selected from consecutive columns of the attention_mask. The first element of each row in the second submatrix is ​​located in the same column of the attention_mask.

15. A data processing apparatus, characterized in that, include: The communication module is used to obtain the query vector Q matrix, the key vector K matrix, and the value vector V matrix, wherein the Q matrix, the K matrix, and the V matrix are all calculated based on the input sequence; The processing module is used to read a new attention mask (attention_mask) from the memory of the graphics processing unit (GPU), wherein the attention_mask is generated by the user-uploaded element arrangement rule or by the user-specified element arrangement rule. The processing module is further configured to perform calculations on the Q matrix, the K matrix, and the V matrix based on the attention_mask to obtain an inference result related to the input sequence. The attention_mask is used to reconstruct a second sub-matrix, which is used to obtain the inference result. Each row in the second sub-matrix is ​​a row selected from the attention_mask; or, each row in the second sub-matrix is ​​constructed by sequentially adjacent elements in a row of the attention_mask, and the number of rows in the attention_mask is less than the number of rows in the Q matrix, and the number of columns in the attention_mask is less than the number of rows or columns in the K matrix.

16. A data processing apparatus, characterized in that, Includes at least one processor and interface; The at least one processor obtains program instructions through the interface; The at least one processor is configured to execute the program line instructions to implement the method as described in any one of claims 1-14.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-14.

18. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-14, wherein the cluster of computing devices includes at least one computing device.

19. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-14, wherein the computing device cluster includes at least one computing device.

Images