Data processing method and device

Abstract

The invention provides a data processing method and device, and relates to the field of computers, the method is applied to a chip, the chip comprises a calculation core, and the method comprises the steps that the calculation core determines a first result O'obtained by calculating a first matrix, a first sub-matrix K'and a second sub-matrix V 'according to a self-attention mechanism; wherein the first matrix is a query matrix Q or a sub-matrix in the matrix Q, the first sub-matrix K'is a sub-matrix in a keyword matrix K, and the second sub-matrix V 'is a sub-matrix in a value matrix V. The calculation core determines a second result O ''obtained by calculating the first matrix, the third sub-matrix K'' and the fourth sub-matrix V ''according to a self-attention mechanism. Wherein the third sub-matrix K ''is a sub-matrix except the first sub-matrix K'in the matrix K, and the fourth sub-matrix V'' is a sub-matrix except the second sub-matrix V 'in the matrix V. The calculation core determines a third result at least according to the first result O'and the second result O '', and the third result is a matrix O or a sub-matrix of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to a self-attention mechanism.

Description

[0001] This application claims the priority of the Chinese patent application filed with the China Patent Office on August 7, 2023, with application number 202310988066.5 and application name “Data Processing Method and Device”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present invention relates to the field of computers and in particular to a data processing method and device. Background Art

[0003] In the context of big data and big computing, artificial intelligence technology represented by machine learning has developed rapidly and has become the core foundation of key technologies such as computer vision, intelligent speech, natural language processing, biometric recognition, and recommendation systems. It has been widely used in financial risk control, medical diagnosis, smart cities and other fields, and has gradually become one of the main forces driving the information revolution and social development.

[0004] In machine learning models, the self-attention mechanism is often used to process sequence data, so that the model can focus on the relationship between the input sequence elements, thereby capturing the dependency between the sequence elements.

[0005] Therefore, how to improve the computational efficiency when calculating the query matrix, key matrix and value matrix corresponding to the sequence data according to the self-attention mechanism is a problem that needs to be solved at present. Summary of the invention

[0006] The present application provides a data processing method and device for improving computational efficiency in the calculation process of a self-attention mechanism.

[0007] In a first aspect, a data processing method is provided, the method is applied to a chip, the chip includes a computing core, the method includes: the computing core determines a first result O' of calculating a first matrix, a first submatrix K' and a second submatrix V' according to a self-attention mechanism. The first matrix is ​​a query matrix Q or a submatrix in the matrix Q, the first submatrix K' is a submatrix in the key matrix K, and the second submatrix V' is a submatrix in the value matrix V. y1, N and d are positive integers, x≤N, y1<N. The calculation kernel determines the second result O" of calculating the first matrix, the third submatrix K" and the fourth submatrix V" according to the self-attention mechanism, wherein the third submatrix K" is a submatrix other than the first submatrix K' in the matrix K, and the fourth submatrix V" is a submatrix other than the second submatrix V' in the matrix V. y2 is a positive integer less than N-y1. The calculation kernel determines the third result based on at least the first result O' and the second result O", wherein the third result is the matrix O or a submatrix of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism, O∈R N×d .

[0008] In one implementation, the first matrix is ​​a query matrix Q, and the third result is a matrix O obtained by calculating the matrix Q, the matrix K, and the matrix V according to the self-attention mechanism.

[0009] In one implementation, the first matrix is ​​a submatrix Q′ of the matrix Q, Q′∈R x×d , x is a positive integer less than N, and the third result is the submatrix O″′ of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism, O″′∈R x×d The method further includes: obtaining a matrix O based at least on the third result.

[0010] In the above method, the keyword matrix K is divided into sub-matrices by row (wherein the sub-matrices include the first sub-matrix K' and the third sub-matrix K'), and the value matrix V is divided into sub-matrices by row (wherein the sub-matrix includes the second sub-matrix V' and the fourth sub-matrix V'), and during the operation, the calculation kernel determines the first result O' (that is, the result of calculating the first matrix, the first sub-matrix K' and the second sub-matrix V' according to the self-attention mechanism) and the second result O" (that is, the result of calculating the first matrix, the third sub-matrix K' and the fourth sub-matrix V' according to the self-attention mechanism), and then determines the third result (that is, the calculation results corresponding to the x elements when the matrix Q, the matrix K and the matrix V are calculated according to the self-attention mechanism) at least based on the first result O' and the second result O'. In this way, compared to operating the matrix Q, matrix K and matrix V as a whole, since the above method of the present application divides the matrix K and matrix V into sub-matrices with smaller data volumes, the amount of intermediate data generated during the operation is smaller, which can reduce the amount of intermediate data. Since the SRAM capacity in the computing core is limited, intermediate data that exceeds the SRAM capacity will be stored in HBM. Therefore, the intermediate data with a smaller data volume allows the chip to store the generated intermediate data as much as possible in the SRAM with a faster reading speed during the operation, thereby reducing the chip's access to the HBM.

[0011] In one implementation, the method further includes: the computing core determines the values ​​of x, y1 and / or y2 respectively according to the SRAM capacity of the computing core.

[0012] Through the above implementation method, the data amount of the divided first sub-matrix K', the second sub-matrix V', the third sub-matrix K", and the fourth sub-matrix V" can be adapted to the SRAM capacity of the computing core, so that when the computing core calculates the first sub-matrix K', the second sub-matrix V', the third sub-matrix K", and the fourth sub-matrix V" according to the self-attention mechanism, the SRAM can be used to store data as much as possible to avoid storing the data in the HBM.

[0013] In one implementation, the computing core determines a second result O'' of calculating the first matrix, the third submatrix K'', and the fourth submatrix V'' according to the self-attention mechanism, including: the computing core determines a first intermediate result S according to the first matrix and the third submatrix K''; wherein the first intermediate result S is a transposed matrix K'' of the first matrix and the third submatrix K'' TThe result of matrix multiplication. The computing core determines the second intermediate result P according to the first intermediate result S; wherein the second intermediate result P is the result of normalizing the first intermediate result S. The computing core determines the second result O" according to the second intermediate result P and the fourth sub-matrix V", wherein the second result O" is the result of matrix multiplication of the second intermediate result P and the fourth sub-matrix V".

[0014] Through the above implementation, a second result O" of calculating the first matrix, the third sub-matrix K" and the fourth sub-matrix V" according to the self-attention mechanism can be calculated.

[0015] In one implementation, the first intermediate result S and the second intermediate result P are stored in a static random access memory SRAM within the computing core.

[0016] In the above implementation, the first intermediate result S and the second intermediate result P are stored in a static random access memory SRAM in the computing core, so that the computing core can quickly read the first intermediate result S and the second intermediate result P to perform subsequent steps.

[0017] In one implementation, the method further includes:

[0018] Before determining the second intermediate result P, reading the S from the SRAM;

[0019] Before determining the second result O", the P is read from the SRAM.

[0020] In a second aspect, a chip is provided, the chip comprising a computing core, the computing core comprising: a self-attention computing unit, used to determine a first result O' of computing a first matrix, a first submatrix K', and a second submatrix V' according to a self-attention mechanism; wherein the first matrix is ​​a submatrix in a query matrix Q or the first matrix is ​​a query matrix Q, the first submatrix K' is a submatrix in a keyword matrix K, and the second submatrix V' is a submatrix in a value matrix V; x, y1, N and d are positive integers, x≤N, y1<N. The self-attention calculation unit is also used to determine the second result O" of calculating the first matrix, the third submatrix K" and the fourth submatrix V" according to the self-attention mechanism; wherein the third submatrix K" is a submatrix other than the first submatrix K' in the matrix K, and the fourth submatrix V" is a submatrix other than the second submatrix V' in the matrix V; wherein, y2 is a positive integer less than N-y1, A fusion unit is used to determine a third result according to the first result O' and the second result O", wherein the third result is a matrix O or a sub-matrix of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism,

[0021]

[0022] In one implementation, the first matrix is ​​a query matrix Q, and the third result is a matrix O obtained by calculating the matrix Q, the matrix K, and the matrix V according to the self-attention mechanism.

[0023] In one implementation, the first matrix is ​​a submatrix Q′ of the matrix Q, Q′∈R x×d , x is a positive integer less than N, and the third result is the submatrix O″′ of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism, O″′∈R x×d The method further includes: obtaining a matrix O based at least on the third result.

[0024] In one implementation, the computing core further includes: a partitioning unit, configured to determine the values ​​of x, y1 and / or y2 according to the SRAM capacity of the computing core.

[0025] In one implementation, the self-attention calculation unit is further used to determine a second result O'' calculated according to the self-attention mechanism for the first matrix, the third sub-matrix K'' and the fourth sub-matrix V'', including: the self-attention calculation unit is further used to determine a first intermediate result S according to the first matrix and the third sub-matrix K''; wherein the first intermediate result S is a transposed matrix K'' of the first matrix and the third sub-matrix K'' T The result of matrix multiplication. The self-attention calculation unit is also used to determine the second intermediate result P according to the first intermediate result S; wherein the second intermediate result P is the result of normalizing the first intermediate result S. The self-attention calculation unit is also used to determine the second result O" according to the second intermediate result P and the fourth sub-matrix V", wherein the second result O" is the result of matrix multiplication of the second intermediate result P and the fourth sub-matrix V".

[0026] In one implementation, the first intermediate result S and the second intermediate result P are stored in a static random access memory SRAM within the computing core.

[0027] In a third aspect, a chip is provided, comprising a computing core, the computing core comprising a memory and a processor, the memory being used to store computer instructions, and the processor being used to call and execute computer instructions from the memory to implement the method provided in the first aspect or any implementation of the first aspect.

[0028] In a fourth aspect, a data processing system is provided, comprising a chip and a control device. The chip comprises a plurality of computing cores. The control device is used to obtain a computing task, wherein the computing task is used to indicate that the task data is calculated according to the self-attention mechanism; the task data comprises: the query matrix, the key matrix and the value matrix corresponding to the h head dimensions of the multi-head self-attention corresponding to the training data of the batch dimensions of B batches, respectively. The control device is also used to divide the computing task into a plurality of subtasks; wherein the query matrix, the key matrix and the value matrix corresponding to the same head dimension of the training data of the same batch dimension are divided into the task data of the same subtask; each subtask is used to indicate that the task data in the subtask is calculated according to the self-attention mechanism. The control device is also used to send a plurality of subtasks to different computing cores in the chip, respectively. After receiving the subtask from the control device, each computing core in the chip executes the method provided in the first aspect or any implementation of the first aspect.

[0029] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a chip, the chip executes the method provided in the first aspect or any implementation of the first aspect.

[0030] In a sixth aspect, a computer program product is provided, the computer program product comprising instructions, and when the instructions are executed on a chip, the chip executes the method provided in the first aspect or any implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 One of the operational flow diagrams of a self-attention mechanism provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of the structure of a transformer class model provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of access rates and capacities of different memories provided in an embodiment of the present application;

[0034] Figure 4 A second schematic diagram of the operation flow of a self-attention mechanism provided in an embodiment of the present application;

[0035] Figure 5A third schematic diagram of the operation flow of a self-attention mechanism provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of dividing a matrix Q, a matrix K, and a matrix V into sub-matrices provided in an embodiment of the present application;

[0037] Figure 7 A fourth schematic diagram of the operation flow of a self-attention mechanism provided in an embodiment of the present application;

[0038] Figure 8 A fifth schematic diagram of the operation flow of a self-attention mechanism provided in an embodiment of the present application;

[0039] Fig. 9 A sixth schematic diagram of the operation flow of a self-attention mechanism provided in an embodiment of the present application;

[0040] Fig.10 A seventh schematic diagram of the operation flow of a self-attention mechanism provided in an embodiment of the present application;

[0041] Fig.11 A schematic diagram of multi-operation stream parallel processing provided in an embodiment of the present application;

[0042] Fig.12 FIG8 is a schematic diagram of a computational flow of a self-attention mechanism provided in an embodiment of the present application;

[0043] Fig.13 A ninth schematic diagram of a computational flow of a self-attention mechanism provided in an embodiment of the present application;

[0044] Fig.14 A schematic diagram of dividing task data provided in an embodiment of the present application;

[0045] Fig.15 A flowchart of a data processing method provided in an embodiment of the present application;

[0046] Fig.16 One of the structural schematic diagrams of a chip provided in an embodiment of the present application;

[0047] Fig.17 A second schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0048] Fig.18A schematic diagram of the structure of a data processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0050] In order to facilitate understanding of this embodiment, the relevant technologies involved in the technical solution provided by this embodiment are first introduced:

[0051] 1. Self-attention mechanism

[0052] The self-attention mechanism is a commonly used mechanism in machine learning models. In the self-attention mechanism, sequence data can be processed so that the model can focus on the relationship between sequence elements in the sequence data, thereby capturing the dependency between sequence elements.

[0053] Specifically, in the self-attention mechanism, for a sequence X of length N (i.e., the sequence X includes N elements), Figure 1 As shown, the query matrix Q, key matrix K and value matrix V corresponding to the sequence X can be obtained through the corresponding dense layer. Among them, the matrix Q, matrix K and matrix V are matrices with N rows and d columns, that is, N is the number of elements in the sequence, and d is the dimension of the embedding vector.

[0054] Then, in three steps: transform the matrix Q and the transposed matrix K T Perform matrix multiplication to obtain matrix S, use softmax function to normalize matrix S to obtain matrix P, perform matrix multiplication on matrix P and matrix V to obtain matrix O (where, ), thereby obtaining the calculation result of the self-attention mechanism (i.e., matrix O).

[0055] For example, the calculation result of the self-attention mechanism (matrix O) can be obtained using the following formula (1):

[0056]

[0057] Among them, QK T That is, the transposed matrix K of matrix Q and matrix K T The matrix S obtained by matrix multiplication is That is, the matrix P obtained by normalizing the matrix S, That is, the matrix O obtained by matrix multiplication of matrix P and matrix V.

[0058] In addition, in formula (1), in order to decouple the calculation result from the embedding vector dimension d, we use It is understandable that in actual applications, it is also possible not to use QK T Normalize. In other words, in practical applications, the following formula (2) can also be used to obtain the calculation result of the self-attention mechanism (i.e., matrix O):

[0059] O = softmax(QK T )V Formula (2)

[0060] In addition, in the multi-head self-attention mechanism, the result of the self-attention mechanism can be calculated for each head by referring to the content of the above formula (1) or formula (2), and then the result of the multi-head self-attention mechanism is obtained according to the result of the self-attention mechanism corresponding to each head.

[0061] For example, in the multi-head self-attention mechanism, the query tensor Q, key tensor K, and value tensor V corresponding to the sequence X satisfy: Among them, h is the number of heads, N is the number of elements in the sequence, and d is the dimension of the embedding vector. Then, using formula (1) or formula (2), we can get the result of the self-attention mechanism of each head (called tensor O), and then get the result of the multi-head self-attention mechanism.

[0062] In addition, when the training data includes multiple batches of sequences, the results of the self-attention mechanism of each batch can be calculated by referring to the above formula (1) or formula (2).

[0063] For example, when the training data includes B batches of sequences, in the multi-head self-attention mechanism, the query tensor Q, key tensor K, and value tensor V corresponding to the sequences in the training data satisfy: Among them, B is the sequence batch, h is the number of heads, N is the number of elements in each batch of sequences, and d is the embedding vector dimension. Then, using formula (1) or formula (2), we can get the result of the self-attention mechanism of each head (called tensor O), and then get the result of the multi-head self-attention mechanism.

[0064] The self-attention mechanism is often used in transformer-like models. For example, Figure 2 It is a schematic diagram of the structure of a transformer-like model. The model may include N encoders with the same structure and N decoders with the same structure. An encoder includes two sublayers: multi-head self-attention and feed forward. A decoder includes three sublayers: multi-head self-attention, encoder-decoder attention, and feed forward. Specifically, in the multi-head self-attention sublayer in the encoder and decoder, the calculation can be performed according to the above formula (1) or formula (2).

[0065] 2. Smart Chip

[0066] In the post-Moore era, although the density of chip transistors continues to increase, it is difficult to further improve the power density and performance density, which means that it is impossible to increase computing power through process improvements. Therefore, an important branch of chip development is the domain specific architecture, also known as smart chips. For example, smart chips include various graphics processing units (GPUs), tensor processors (neural processing units, NPUs), and neural processing units (neural processing units, NPUs).

[0067] Compared with general-purpose chips such as central processing units (CPUs), smart chips have the characteristics of strong specialization, simple design, the ability to customize computing units according to the specific characteristics of applications, simplify control logic, and the ability to design storage structures and data paths that are compatible with the computing characteristics of specific fields. Although smart chips sacrifice versatility and flexibility, they have achieved higher performance and energy efficiency, and have been widely used in high-performance computing, artificial intelligence, cryptography and other fields.

[0068] The storage architecture of the smart chip can be divided into multiple levels. The closer the storage is to the computing core of the smart chip, the smaller its capacity and the faster its read and write speed; conversely, the farther the storage is from the computing core of the smart chip, the larger its capacity and the slower its read and write speed. For example, Figure 3 As shown, the static random-access memory (SRAM) closest to the computing core may have a capacity of MB level (20 MB is taken as an example in the figure) and an access rate of 10 TB / s level (19 TB / s is taken as an example in the figure); the high bandwidth memory (HBM) next to the SRAM and close to the computing core may have a capacity of GB level (40 GB is taken as an example in the figure) and an access rate of 1 TB / s level (1.5 TB / s is taken as an example in the figure); the dynamic random access memory (DRAM) next to the SRAM and HBM and close to the computing core may have a capacity of TB level (>1 TB is taken as an example in the figure) and an access rate of 10 GB / s level (12.8 GB / s is taken as an example in the figure).

[0069] The following is an example to introduce the calculation process of the self-attention mechanism in related technologies. ) is used as an example for calculation, such as Figure 4 As shown, including the following S101-S103 contents:

[0070] S101, the computing core in the chip reads the matrix Q and the matrix K from the HBM, and converts the matrix Q and the matrix K into the transposed matrix K T Perform matrix multiplication to obtain matrix S.

[0071] Specifically, in the calculation kernel, the i-th row in the matrix Q (i.e., Q * ) and the j-th row in the matrix K (i.e., K + ) to perform dot product operations to obtain the elements S in the matrix S ij . The dot product of the first row of matrix Q and each row of matrix K constitutes the first row of matrix S (i.e. S1), the dot product of the second row of matrix Q and each row of matrix K constitutes the second row of matrix S (i.e. S2), and so on. The matrix S is N rows and N columns, i.e.

[0072] Among them, since the SRAM capacity in the computing core is limited, after calculating the matrix S, the computing core will write the matrix S into the HBM to perform the next operation.

[0073] S102 . The computing core reads the matrix S from the HBM and normalizes the matrix S.

[0074] Specifically, in the calculation core, the i-th row (i.e., S * Then, the following formula (3) is used to obtain the i-th row of the matrix P in the normalized result (i.e., P * ):

[0075]

[0076] Among them, the matrix P has N rows and N columns, that is,

[0077] After calculating the matrix P, the computing core writes the matrix P into the HBM for the next operation.

[0078] S103, the computing core reads the matrix P and the matrix V from the HBM, and performs matrix multiplication on the matrix P and the matrix V to obtain the result of the self-attention mechanism (ie, the matrix O).

[0079] Specifically, in the calculation kernel, the i-th row in the matrix P (i.e., P * ) and the j-th row in the matrix V (i.e. V * ) performs dot product operations to obtain the elements O in the matrix O ij . The dot product of the first row of matrix P and each row of matrix V constitutes the first row of matrix O (i.e., O1), the dot product of the second row of matrix P and each row of matrix V constitutes the second row of matrix S (i.e., S2), and so on. The matrix O is N rows and d columns, i.e.

[0080] It can be seen that in the above Figure 4 In the related technologies shown, on the one hand, a large amount of data will be generated during the calculation process and needs to be stored. For example, after the N×N matrix S is calculated in S101, the matrix S needs to be stored in order to perform subsequent operations; for another example, after the N×N matrix P is calculated in S102, the matrix P needs to be stored in order to perform subsequent operations. In particular, as the sequence length of the training data continues to increase (that is, the value of N continues to increase), the amount of data generated during the calculation process will increase quadratically. On the other hand, due to the limited capacity of SRAM, the above-mentioned related technologies often need to store the data generated during the calculation process in HBM. For example, in the above S101, the matrix S is stored in HBM, and in S102, the matrix P is stored in HBM.

[0081] As a result, a large number of HBM accesses are required during the calculation process. For example, in the process of S101, the calculation core needs to read the matrix Q and matrix K from the HBM and store the matrix S in the HBM. That is, the access complexity of the calculation core to the HBM in S101 is O(Nd+N 2 ); In the process of S102, the computing core needs to read the matrix S from the HBM and store the matrix P in the HBM, that is, the access complexity of the computing core to the HBM in S102 is O(N 2 ); In the process of S103, the computing core needs to read the matrix P from the HBM and store the matrix O in the HBM, that is, the access complexity of the computing core to the HBM in S103 is O(Nd+N 2 ). In view of the above problem, the embodiments of the present application consider that when calculating the query matrix Q, key matrix K and value matrix V corresponding to the sequence X according to the self-attention mechanism, the matrix Q, the matrix K and the matrix V can be divided into sub-matrices with smaller data volumes.

[0082] Embodiment 1:

[0083] Because the on-chip SRAM memory is limited, the embodiment of the present application needs to segment the input data and further segment the computing tasks on the AI ​​Core into more fine-grained subtasks for scheduling. Figure 5 As shown, the embodiment of the present application puts AI Core on Divide into more fine-grained The hyperparameters m and n here need to satisfy (m×d+2×n×d)×2≤M, where M is the capacity of SRAM (I unit BK) and the input data type is assumed to be FLOAT16).

[0084] like Figure 5 As shown, for fine-grained subtask data The embodiment of the present application reads them into the on-chip SRAM memory of the smart chip and sequentially completes the calculation of the following three steps:

[0085] Step 1:

[0086] Step 2:

[0087] Step 3: O i =P ij V j

[0088] Step 1: Execute Q i and The matrix multiplication of the result Sij The physical meaning of is the weight of the correlation between Query and Key. Step 2 ij Perform Softmax (Softmax is a common mathematical method for weight normalization) to calculate the weight normalization result P ij Step 3: Execute P ij and V j The matrix multiplication of the weight and the weighted sum of Value obtains the result of the attention mechanism O i . Due to S ij It is the total intermediate result S = QK T In order to ensure the mathematical equivalence of the calculation results with those before fusion, the calculation results of the above three steps need to be weighted accumulated in each cycle to obtain the final calculation result O.

[0089] Subtask scheduling: Fusion computing requires the intermediate calculation results to be processed in each cycle. Weighted accumulation means that the old O i Read from HBM into the chip, and then the new O i With the old O i After weighted accumulation, the result is written back to HBM. Therefore, the additional cost introduced by fusion computing is the frequent HMB IO access to the calculation result O. In order to reduce the frequent reading and writing of the calculation result O, the embodiment of the present application arranges the fine-grained subtasks in two layers of loops: 1) outer loop. In the outer loop, the embodiment of the present application reads in 2) Inner loop. In the inner loop, the present embodiment reads in Based on this arrangement, the computation of the subtask will be O i After the weighted accumulation gets the correct result, calculate O i+1 Here the weight parameters α and β are dynamic variables, and their update methods are as follows:

[0090] m ij =rowmax(S ij )

[0091] l ij =rowsum(P ij )

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] It is worth noting that Figure 5 As shown, the embodiment of the present application will be O * The intermediate results are not written into HBM, but cached in the on-chip SRAM until the correct calculation result is obtained after an outer loop is completed. * Write back to HBM. In addition, the embodiment of the present application uses the on-chip L2 cache to pre-read input data, thereby hiding the overhead of reading data from HBM and reducing the calculation time in the core.

[0099] Embodiment 2:

[0100] For example Figure 6 As shown, on the one hand, each m rows in the N×d matrix Q is divided into a submatrix, and then the matrix Q can be divided into Sub-matrices (such as Figure 6 middle, The sub-matrices can include On the other hand, every n rows in the N×d matrix K are divided into a submatrix, and the matrix K can be divided into Sub-matrices (such as Figure 6 middle, The sub-matrices can include On the other hand, each n-row in the N×d matrix V is divided into a submatrix, and the matrix V can be divided into Sub-matrices (such as Figure 6 middle, The sub-matrices can include etc.), where the size of each submatrix is ​​n×d.

[0101] After dividing the matrix Q, the matrix K, and the matrix V into sub-matrices, these sub-matrices can be used to obtain the calculation results (i.e., the matrix O) when the matrix Q, the matrix K, and the matrix V are calculated according to the self-attention mechanism. The following takes the process of calculating the calculation results corresponding to the first m sequence elements in the matrix O (i.e., the process of calculating the first m rows in the matrix O) as an example to introduce the operation process of the calculation kernel in the embodiment of the present application. Specifically, Figure 7 As shown, the operation process of the computing kernel may include:

[0102] S201, the calculation kernel is based on the self-attention mechanism and Perform calculations and obtain the results

[0103] in, is the submatrix corresponding to the first m sequence elements in the matrix Q, is the submatrix corresponding to the first n sequence elements in matrix K, is the submatrix corresponding to the first n sequence elements in matrix V, in addition,

[0104] Specifically, Figure 8 As shown, S201 may include the following contents of S2011-S2013:

[0105] S2011, Computational Core Computation and The matrix multiplication result of

[0106] in, Right now is an m×n matrix.

[0107] For example, Figure 8 As shown, the computing core can first read from the HBM and Then use the Matmul function to calculate and The matrix multiplication result of Among them, due to The amount of data is small (m×n), so we can Stored in SRAM for subsequent processing.

[0108] In actual application, the content of S2021 can also be implemented by using the matrix multiplication dedicated hardware unit (such as Cube unit) in the computing core.

[0109] S2012, Calculation and Verification Normalize it and get

[0110] in, Right now is a matrix with m rows and n columns.

[0111] For example, Figure 8 As shown, the calculation kernel can use the softmax function to Normalize each row in to get Among them, due to The amount of data is small (m×n), so we can Stored in SRAM for subsequent processing.

[0112] In actual application, the content of S2022 can also be implemented by using the vector computing hardware unit (such as Vector unit) in the computing core.

[0113] S2013, Computational Core Computation and The matrix multiplication result of

[0114] For example, Figure 8 As shown, the calculation kernel can be calculated using the Matmul function and The matrix multiplication result of Among them, due to The amount of data is small (m×d), so Stored in SRAM for subsequent processing.

[0115] In actual application, the content of S2023 can be implemented by using a matrix multiplication dedicated hardware unit (such as a Cube unit) in the computing core.

[0116] S202, the calculation kernel is based on the self-attention mechanism and Perform calculations and obtain the results

[0117] in, is the submatrix corresponding to the first m sequence elements in the matrix Q, is the submatrix corresponding to the sequence elements from n+1th to 2nth in matrix K, is the submatrix corresponding to the sequence elements from n+1th to 2nth in the matrix V, in addition,

[0118] Specifically, Fig. 9 As shown, S202 may include the following contents of S2021-S2023:

[0119] S2021, Computational Core Computing and The matrix multiplication result of

[0120] in, Right now is an m×n matrix.

[0121] The specific implementation process of S2021 can refer to the calculation of S2011 above. In addition, Similarly, we can Stored in SRAM for subsequent processing.

[0122] S2022, Calculation and Verification Normalize it and get

[0123] in, Right now is a matrix with m rows and n columns.

[0124] The specific implementation process of S2022 can refer to the calculation of S2012 above. In addition, Stored in SRAM for subsequent processing.

[0125] S2023, Computational Core Computation and The matrix multiplication result of

[0126] The specific implementation process of S2023 can refer to the calculation of S2013 above. In addition, Stored in SRAM for subsequent processing.

[0127] S203, calculation core basis and Determine the self-attention mechanism and The result of the calculation when the calculation is performed.

[0128] It is understandable that For the general and The matrix obtained by splicing is For the general and The concatenated matrix.

[0129] For example, the parameters can be calculated according to the following formula group (4): and and then That is, according to the self-attention mechanism and The calculation results when performing the calculation:

[0130]

[0131]

[0132]

[0133]

[0134] M new =max(M old , M))

[0135]

[0136]

[0137]

[0138] Among them, ⊙ represents the row-by-row multiplication operation, and diag represents the conversion to a diagonal matrix.

[0139] S204, the calculation kernel is based on the self-attention mechanism and Perform calculations and obtain the results

[0140] in, is the submatrix corresponding to the first m sequence elements in the matrix Q, is the submatrix corresponding to the 2n+1th to 3nth sequence elements in matrix K, is the submatrix corresponding to the 2n+1th to 3nth sequence elements in the matrix V, in addition,

[0141] Specifically, the process is similar to that of S201 and S202. Fig.10 As shown, S204 may specifically include:

[0142] S2041, Calculation Core Calculation and The matrix multiplication result of

[0143] in, Right now is an m×n matrix.

[0144] S2042, Calculation and Verification Normalize it and get

[0145] in, Right now is a matrix with m rows and n columns.

[0146] S2043, Calculation Core Calculation and The matrix multiplication result of

[0147] For the specific implementation process of the above S2041-S2043, please refer to the contents of S2011-S2013 or S2021-S2023 above, and the repeated contents will not be repeated here.

[0148] S205, calculation basis and Determine the self-attention mechanism and The result of the calculation when the calculation is performed.

[0149] It is understandable that For the general and The matrix obtained by splicing is For the general and The concatenated matrix.

[0150] For example, the parameters can be calculated according to the following formula group (5): and and then That is, according to the self-attention mechanism and The calculation results when performing the calculation:

[0151]

[0152]

[0153] M new =max(M old , M))

[0154]

[0155]

[0156]

[0157] Among them, M old The value of is M in formula group (4) new , L old The value of is L in formula group (4) new .

[0158] Next, we can refer to the contents of S202-S205 above to continue traversing the remaining sub-matrices in the matrix K and the matrix V, and then obtain the self-attention mechanism for and The calculation results when the calculation is performed Where y is a multiple of n,

[0159] Among them, it is taken into consideration that: since the matrix K and the matrix V are divided into sub-matrices for operation in the embodiment of the present application, when the matrix multiplication operation is performed using the matrix multiplication dedicated hardware unit and the vector operation is performed using the vector calculation hardware unit, multiple operation flows can be executed in parallel. For ease of description, in the embodiment of the present application, the operation process corresponding to a sub-matrix of the matrix Q, a sub-matrix of the matrix K, and a sub-matrix of the matrix V is referred to as an "operation flow". For example, the above S201, S202, S204 and S206 can be used as an operation flow respectively.

[0160] For example Fig.11 As shown, after executing S2011 using the dedicated hardware unit for matrix multiplication, it is possible to execute S2021 using the dedicated hardware unit for matrix multiplication without waiting for other steps of S201 to be executed; after executing S2012 using the dedicated hardware unit for vector calculation, S2022 can be executed using the dedicated hardware unit for vector calculation; after executing S2013 using the dedicated hardware unit for matrix multiplication, S2041 can be executed using the dedicated hardware unit for matrix multiplication; after executing S2022 using the dedicated hardware unit for vector calculation, S2042 can be executed using the dedicated hardware unit for vector calculation, and so on. In this way, the operation efficiency can be improved.

[0161] After that, the matrix K and the last submatrix of the matrix V can be processed according to the following process S206-S207. and Calculations are performed to obtain the calculation results corresponding to the first m sequence elements in the matrix O (i.e., the result of calculating the matrix Q, the matrix K, and the matrix V according to the self-attention mechanism).

[0162] S206, the calculation kernel is based on the self-attention mechanism and Perform calculations and obtain the results

[0163] in, is the submatrix corresponding to the first m sequence elements in the matrix Q, is the submatrix corresponding to the Ny-th to N-th sequence elements in the matrix K, is the submatrix corresponding to the Ny-th to N-th sequence elements in the matrix V, in addition,

[0164] Specifically, the process is similar to that of S201, S202 and S204. Fig.12 As shown, S206 may include:

[0165] S2061, Calculation Core Calculation and The matrix multiplication result of

[0166] in, Right now It is an m×(Ny) matrix.

[0167] S2062, Calculation and Verification Normalize it and get

[0168] in, Right now It is an m×(Ny) matrix.

[0169] S2063, Calculation Core Calculation and The matrix multiplication result of

[0170] For the specific implementation process of the above S2061-S2063, reference can be made to the contents of S2011-S2013, S2021-S2023 or S2041-S2043 above, and the repeated contents will not be repeated here.

[0171] S207, calculation core basis and Determine the self-attention mechanism and The result of the calculation when the calculation is performed.

[0172] It is understandable that That is the matrix K, That is, matrix V. Therefore, the calculation result of S207 is the calculation result corresponding to the first m sequence elements in matrix O (that is, the result of calculating matrix Q, matrix K and matrix V according to the self-attention mechanism).

[0173] For example, the parameters can be calculated according to formula group (4) above: and And the parameters are calculated according to formula group (5) and The process of calculating the parameters and and then That is, according to the self-attention mechanism and The calculation result during the calculation is the calculation result corresponding to the first m sequence elements in the matrix O (that is, the result of calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism).

[0174] Similarly, referring to the above process of S201-S207, the calculation results corresponding to other sequence elements in the matrix O can be calculated, thereby obtaining the matrix O.

[0175] It can be seen that through the above process of the embodiment of the present application, the matrix Q, the matrix K and the matrix V can be divided into multiple sub-matrices in an even manner, and these sub-matrices can be used to calculate the calculation results (i.e., matrix O) when the matrix Q, the matrix K and the matrix V are calculated according to the self-attention mechanism.

[0176] In this way, on the one hand, the storage space occupied by the intermediate data can be reduced, thereby reducing the amount of access to the HBM. Figure 8 After the operation process shown in the figure is completed, you only need to Store for use in subsequent steps; Fig. 9 After the operation process shown in the figure is completed, you only need to and M for updating the dynamic parameters α and β new and L new The intermediate data is stored in SRAM for use in subsequent steps, and so on. When these intermediate data are stored in SRAM, the amount of access to HBM can be further reduced.

[0177] On the other hand, in the above process of the embodiment of the present application, since the matrix K and the matrix V are divided into sub-matrices for operation, when the matrix multiplication operation is performed using the matrix multiplication dedicated hardware unit and the vector operation is performed using the vector calculation hardware unit, the operation processes corresponding to multiple sub-matrices can be executed in parallel. For details, please refer to the above Fig.11 This can further improve the computational efficiency.

[0178] In addition, in one implementation, the matrix Q, the matrix K, and the matrix V can be divided into sub-matrices according to the capacity of the SRAM corresponding to the computing core, so that the SRAM can be used to store data during the operation process of each operation flow, avoiding storing the data in the HBM.

[0179] Specifically, Fig.13 As shown, before executing S201-S207, it also includes:

[0180] S208. The computing core divides the matrix Q, the matrix K and the matrix V into sub-matrices according to the capacity of the SRAM corresponding to the computing core.

[0181] For example, consider a flow of operations with Fig. 9 Taking the operation flow corresponding to S201 as an example, on the one hand, the computing core needs to read the sub-matrices of matrix Q, matrix K and matrix V, and the data volume is (m×d+(n×d)×2)×bt. Among them, bt represents the number of bytes of each data. On the other hand, the two matrices generated by S2021 and S2022 need to be stored during the operation, and the data volume is ((m×n)×2)×bt. On the other hand, the matrix generated by S2023 also needs to be stored, and the data volume is (m×d)×bt. On the other hand, two dynamic parameters M need to be stored. old and L old , the data volume is 2m×bt.

[0182] Among them, the sub-matrices of matrix K and matrix V can reuse storage space, and the two matrices generated by S2021 and S2022 can reuse storage space. Therefore, the matrix Q, matrix K and matrix V can be divided into sub-matrices according to the conditions of the following formula (6):

[0183] (m×d+n×d+m×n+m×d+2m)×bt≤M Formula (6)

[0184] Wherein, M is the capacity of the SRAM corresponding to the computing core.

[0185] In one implementation, considering that in actual applications, on the one hand, in a computing task, it is usually necessary to calculate multiple batches of training data according to the multi-head self-attention mechanism, that is, the task data in a computing task may include query matrices, key matrices and value matrices of h head dimensions corresponding to multiple batches of training data; on the other hand, the chip performing the operation may usually have multiple computing cores.

[0186] Therefore, the computing task can be divided into multiple subtasks, and each subtask can be sent to different computing cores. Among them, the query matrix, key matrix, and value matrix corresponding to the same head dimension of the training data of the same batch dimension in the task data are divided into the task data of the same subtask.

[0187] For example Fig.14As shown, assume that the task data of the computing task includes B batches of training data (such as b_1, b_2, ..., b_B in the figure), and each training data includes a query matrix, a key matrix, and a value matrix of H head dimensions (such as h_1, h_2, ..., h_H in the figure include three matrices Q, K, and V respectively). In addition, the chip includes C computing cores (such as c_1, c_2, ..., c_C in the figure).

[0188] The query matrix, key matrix, and value matrix corresponding to the same head dimension of the training data of the same batch dimension are divided into the task data of the same subtask. The task data of each subtask can include Group query matrix, key matrix and value matrix.

[0189] In this way, when each subtask is sent to different computing cores for operation, on the one hand, multiple computing cores can be used for parallel processing, thereby improving the operation efficiency. On the other hand, each computing core can process the task data according to the above process S201-S208, thereby achieving the effect achieved by the above S201-S208.

[0190] In a possible design, when a computing task is divided into multiple subtasks, the division may be performed in such a manner that task data corresponding to the same batch dimension in the task data is divided into the same subtask.

[0191] Specifically, considering that in a memory (such as DRAM), task data is usually arranged according to the batch dimension, the task data corresponding to the same batch dimension can be divided into the same subtask. Fig.14 In the example, the query matrix, key matrix, and value matrix of each head dimension corresponding to b_1 are divided into the same subtask; the query matrix, key matrix, and value matrix of each head dimension corresponding to b_2 are divided into the same subtask, and so on. In this way, since the task data corresponding to the same batch dimension is continuous in the memory, the computing core can quickly read the data required for the operation from the memory, thereby improving the computing efficiency.

[0192] In addition, it should be noted that, in the above embodiments, for ease of explanation, the matrix Q, the matrix K, and the matrix V are divided into multiple sub-matrices by equal division. In the specific implementation process, the equal division method may not be adopted. For example, when the above N is not divisible by m, the multiple sub-matrices corresponding to the matrix Q may include sub-matrices with a specification of m×d, and may also include other regular sub-matrices (i.e., sub-matrices with a number of rows other than m); for another example, when the above N is not divisible by n, the multiple sub-matrices corresponding to the matrix K or the matrix V may include sub-matrices with a specification of n×d, and may also include other regular sub-matrices (i.e., sub-matrices with a number of rows other than n).

[0193] In combination with the above content, a data processing method provided by an embodiment of the present application is introduced below. This method can be used for a chip to calculate the query matrix Q, key matrix K and value matrix V corresponding to the sequence X according to the self-attention mechanism. Specifically, Fig.15 As shown, the method may include:

[0194] S301. A computing core in a chip determines a first result O' obtained by computing a first matrix, a first submatrix K', and a second submatrix V' according to a self-attention mechanism.

[0195] Specifically, the chip may include one or more computing cores. Here, only the processing process of one computing core is introduced as an example. It is understandable that in actual application, other computing cores in the chip can also refer to the content of this method to process data.

[0196] Wherein, the first matrix is ​​a submatrix in the query matrix Q or the first matrix is ​​the query matrix Q, the first submatrix K' is a submatrix in the key matrix K, and the second submatrix V' is a submatrix in the value matrix V; x, y1, N and d are positive integers, x≤N, y1<N.

[0197] For example, when this method is applied to Figure 7 In the process shown, the first result O' can be Figure 7 In the process shown, the The first matrix can be Figure 7 In the process shown At this time, x=m. The first submatrix K' can be Figure 7 In the process shown The second sub-matrix V' can be Figure 7 In the process shown

[0198] The specific implementation process of S301 can refer to the above Figure 7 The contents of S201-S205 in the process shown.

[0199] S302. The computing core determines a second result O" of computing the first matrix, the third submatrix K" and the fourth submatrix V" according to the self-attention mechanism.

[0200] The third submatrix K" is a submatrix other than the first submatrix K' in the matrix K, and the fourth submatrix V" is a submatrix other than the second submatrix V' in the matrix V; wherein, y2 is a positive integer less than N-y1,

[0201] For example, the second result O" can be Figure 7 In the process shown, the The third sub-matrix K' can be Figure 7 In the process shown The fourth sub-matrix V' can be Figure 7 In the process shown

[0202] The specific implementation process of S302 can refer to the above Figure 7 The content of S206 in the process shown.

[0203] In one implementation, the first matrix is ​​a query matrix Q, and the third result is a matrix O obtained by calculating the matrix Q, the matrix K, and the matrix V according to the self-attention mechanism.

[0204] In one implementation, the first matrix is ​​a submatrix Q′ of the matrix Q, Q′∈R x×d , x<N, the third result is the submatrix O″′ of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism, O″′∈R x×d The method further includes: obtaining a matrix O based at least on the third result.

[0205] In one implementation, the above S302 may specifically include:

[0206] S3021. The computing core determines a first intermediate result S according to the first matrix and the third sub-matrix K″.

[0207] The first intermediate result S is the transposed matrix K' of the first matrix and the third submatrix K' T The result of performing a matrix multiplication.

[0208] For example, the specific implementation process of S3021 can refer to the implementation process of S2061 above. The first intermediate result S can be obtained by S2061.

[0209] S3022. The computing core determines a second intermediate result P according to the first intermediate result S.

[0210] The second intermediate result P is the result of normalizing the first intermediate result S.

[0211] For example, the specific implementation process of S3022 can refer to the implementation process of S2062 above. The second intermediate result P can be obtained by S2062.

[0212] S3023. The computing core determines a second result O" according to the second intermediate result P and the fourth sub-matrix V".

[0213] The second result O" is the result of matrix multiplication of the second intermediate result P and the fourth sub-matrix V".

[0214] For example, the specific implementation process of S3023 may refer to the implementation process of S2063 mentioned above.

[0215] In one implementation, the first intermediate result S and the second intermediate result P are stored in a static random access memory SRAM within the computing core.

[0216] For example, in Fig.12 middle and and stored in SRAM.

[0217] S303. The computing core determines a third result based at least on the first result O′ and the second result O″.

[0218] The third result is the matrix O or a submatrix of the matrix O obtained by calculating the matrix Q, the matrix K, and the matrix V according to the self-attention mechanism.

[0219] In one implementation, the first matrix is ​​a query matrix Q, and the third result is a matrix O obtained by calculating the matrix Q, the matrix K, and the matrix V according to the self-attention mechanism.

[0220] In one implementation, the first matrix is ​​a submatrix Q′ of the matrix Q, Q′∈R x×d, x is a positive integer less than N, and the third result is the submatrix O″′ of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism, O″′∈R x×d The method further includes: obtaining a matrix O based at least on the third result.

[0221] For example, the third result can be Figure 7 The process shown in FIG. 1 is obtained by S207. in, That is, according to the self-attention mechanism and The calculation result during the calculation is the calculation result corresponding to the first m sequence elements in the matrix O (that is, the result of calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism).

[0222] The specific implementation process of S303 can refer to the above Figure 7 The content of S207 in the process shown.

[0223] In one implementation, the method may further include:

[0224] S304: The computing core determines the values ​​of x, y1 and / or y2 according to the SRAM capacity of the computing core.

[0225] The specific implementation process of S304 may refer to the corresponding content of S208 above, and will not be elaborated here.

[0226] It is understandable that, in order to implement the above-mentioned embodiments, the chip includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0227] like Fig.16 The figure is a schematic diagram of the structure of a chip provided by the present application. The chip 40 can be used to implement the above Figure 7-Figure 15 The functions of each step performed by the described chip.

[0228] Specifically, Fig.16 As shown, the chip 40 may include one or more computing cores. The figure takes computing cores 41, 42, ..., and 4n as examples for illustration. Among them, each computing core in the chip 40 can be used to implement the above Figure 7-Figure 15 The functions of each step performed by the described chip.

[0229] The following description is made by taking the functional modules included in the computing core 41 as an example. It can be understood that when multiple computing cores in the chip 40 are used to perform computing tasks of the self-attention mechanism, the functional modules included in each computing core can refer to the functional modules included in the computing core 41 below.

[0230] The computing core 41 may include: a self-attention computing unit 411 and a fusion unit 412 .

[0231] The self-attention calculation unit 411 is used to determine a first result O' of calculating the first matrix, the first sub-matrix K' and the second sub-matrix V' according to a self-attention mechanism.

[0232] Wherein, the first matrix is ​​a submatrix in the query matrix Q or the first matrix is ​​the query matrix Q, the first submatrix K' is a submatrix in the key matrix K, and the second submatrix V' is a submatrix in the value matrix V; x, y1, N and d are positive integers, x≤N, y1<N.

[0233] The self-attention calculation unit 411 is also used to determine a second result O" of calculating the first matrix, the third sub-matrix K" and the fourth sub-matrix V" according to the self-attention mechanism.

[0234] The third submatrix K" is a submatrix other than the first submatrix K' in the matrix K, and the fourth submatrix V" is a submatrix other than the second submatrix V' in the matrix V; wherein, y2 is a positive integer less than N-y1,

[0235] The fusion unit 412 is configured to determine a third result according to the first result O' and the second result O".

[0236] The third result is the matrix O or a submatrix of the matrix O obtained by calculating the matrix Q, the matrix K, and the matrix V according to the self-attention mechanism.

[0237] In one implementation, the first matrix is ​​a query matrix Q, and the third result is a matrix O obtained by calculating the matrix Q, the matrix K, and the matrix V according to the self-attention mechanism.

[0238] In one implementation, the first matrix is ​​a submatrix Q′ of the matrix Q, Q′∈R x×d, x<N, the third result is the submatrix O″′ of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to the self-attention mechanism, O″′∈R x×d The method further includes: obtaining a matrix O based at least on the third result.

[0239] In one implementation, the computing core further includes: a partitioning unit 413, configured to determine the values ​​of x, y1 and / or y2 according to the SRAM capacity of the computing core.

[0240] In one implementation, the self-attention calculation unit 412 is further used to determine a second result O'' of calculating the first matrix, the third sub-matrix K'' and the fourth sub-matrix V'' according to the self-attention mechanism, including:

[0241] The self-attention calculation unit 412 is further used to determine a first intermediate result S according to the first matrix and the third sub-matrix K''; wherein the first intermediate result S is a transposed matrix K'' of the first matrix and the third sub-matrix K'' T The result of performing a matrix multiplication.

[0242] The self-attention calculation unit 412 is also used to determine a second intermediate result P based on the first intermediate result S; wherein the second intermediate result P is a result of normalizing the first intermediate result S.

[0243] The self-attention calculation unit 412 is also used to determine the second result O" based on the second intermediate result P and the fourth sub-matrix V", and the second result O" is the result of matrix multiplication of the second intermediate result P and the fourth sub-matrix V".

[0244] In one implementation, the first intermediate result S and the second intermediate result P are stored in a static random access memory SRAM within the computing core.

[0245] Fig.17 This is a schematic diagram of the structure of another chip provided in this embodiment. The chip 50 can be used to implement the above Figure 7-Figure 15 The functions of each step performed by the described chip.

[0246] Specifically, Fig.17 As shown, the chip 50 may include one or more computing cores. The figure takes computing core 51, computing core 52, ..., computing core 5n as an example for illustration. Among them, each computing core in the chip 50 can be used to implement the above Figure 7-Figure 15 The functions of each step performed by the described chip.

[0247] The following description is made by taking the functional modules included in the computing core 51 as an example. It can be understood that when multiple computing cores in the chip 50 are used to perform computing tasks of the self-attention mechanism, the functional modules included in each computing core can refer to the functional modules included in the computing core 51 below.

[0248] The computing core 51 may include: a processor 511 , a memory 512 , a communication interface 513 , and some or all of the communication lines 514 .

[0249] The processor 511 is used to execute the above Figure 7-Figure 15 The functions of each step performed by the described chip.

[0250] Specifically, the processor 511 may include a general-purpose central processing unit (CPU), a microprocessor, a field programmable gate array (FPGA), a digital signal processor (DSP) or an application-specific integrated circuit (ASIC), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0251] In addition, the memory 512 may be SRAM, HBM, or the like.

[0252] The memory 512 stores computer instructions. The processor 511 can execute all or part of the steps in the method provided in this embodiment by executing the computer instructions stored in the memory 512. Fig.17 As shown, the computer instructions stored in the memory 512 may include software modules for implementing the functions of the self-attention calculation unit 411, the fusion unit 412, and the division unit 413. The processor 511 may execute the method provided in this embodiment by executing the computer instructions stored in the memory 512.

[0253] Optionally, the computer-executable instructions in this embodiment may also be referred to as application program codes, which is not specifically limited in this embodiment.

[0254] In addition, the communication interface 513 is used to connect the various components in the computing core 51.

[0255] In addition, the present application also provides a data processing system. Fig.18As shown, the data processing system 60 includes a control device 61 and a chip 62. The chip 62 may include multiple computing cores. The figure takes computing cores 621, 622, ..., and 62n as examples for explanation.

[0256] The control device 61 can be used to Fig.14 In the process shown, the computing task is divided into multiple subtasks, and the multiple subtasks are sent to multiple computing cores in the chip 62. So that each computing core in the chip 62 can realize the above Figure 7-Figure 15 The functions of each step performed by the described chip. In actual application, the functions of the control device 61 can be implemented by the driver of the chip 62, or when the chip 62 is an intelligent chip other than the host CPU, the functions of the control device 61 can be implemented by the host CPU. In addition, the control device 61 can also be integrated in the chip 62, in which case the functions of the control device 61 can be implemented by the software / hardware module in the chip 62. The specific form of the control device 61 may not be limited in the embodiments of the present application.

[0257] Specifically, the control device 61 is used to obtain a computing task. The computing task is used to indicate that the task data is calculated according to the self-attention mechanism. The task data includes: the query matrix, key matrix and value matrix corresponding to the h head dimensions in the multi-head self-attention corresponding to the training data of B batch dimensions.

[0258] The control device 61 is also used to divide the computing task into multiple subtasks. The query matrix, key matrix and value matrix corresponding to the same head dimension of the training data of the same batch dimension are divided into the task data of the same subtask; each subtask is used to indicate that the task data in the subtask is calculated according to the self-attention mechanism.

[0259] The control device 61 is also used to send multiple subtasks to different computing cores in the chip 60 respectively.

[0260] After receiving the subtask from the control device 61, each computing core in the chip 62 executes the above Figure 7-Figure 15 The steps performed by the described chip.

[0261] In addition, the embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are run on a chip, the chip can be operated according to the above-mentioned embodiment of the present application. Figure 7-Figure 15 All or part of the steps performed by the described chip.

[0262] The present application also provides a computer program product including instructions. When the computer program product is run on a chip, the chip can be executed according to the above-mentioned embodiment of the present application. Figure 7-Figure 15 All or part of the steps performed by the described chip.

[0263] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a resource change device. Of course, the processor and the storage medium can also be present in the resource change device as discrete components.

[0264] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in this embodiment is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a communication device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as an SSD.

[0265] In this embodiment, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different implementations are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment based on their internal logical relationships.

[0266] In this embodiment, "at least one" means one or more, "more than one" means two or more, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, for elements (element) in the singular form "a", "an" and "the", unless the context clearly stipulates otherwise, it does not mean "one or only one", but means "one or more than one". For example, "a device" means one or more such devices. Furthermore, at least one (at least one of) ......." means one or any combination of subsequent associated objects, for example, "at least one of A, B and C" includes A, B, C, AB, AC, BC, or ABC. In the text description of this embodiment, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula of this embodiment, the character " / " indicates that the previous and next associated objects are in a "division" relationship.

Claims

1. A data processing method, characterized in that: The method is applied to a chip, wherein the chip includes a computing core, and the method includes: The calculation core calculates the first matrix, the first submatrix K' and the second submatrix V' according to the self-attention mechanism to obtain a first result O'; wherein the first matrix is ​​a submatrix Q' in the query matrix Q or the first matrix is ​​the query matrix Q, the first submatrix K' is a submatrix in the key matrix K, and the second submatrix V' is a submatrix in the value matrix V; x, y1, N and d are positive integers, x<N, y1<N; The computing core calculates the first matrix, the third submatrix K'' and the fourth submatrix V'' according to the self-attention mechanism to obtain a second result O''; wherein the third submatrix K'' is a submatrix of the matrix K other than the first submatrix K'', and the fourth submatrix V'' is a submatrix of the matrix V other than the second submatrix V''; wherein, y2 is a positive integer less than N-y1, The computing core determines a third result based at least on the first result O' and the second result O", wherein the third result is a matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to a self-attention mechanism, or the third result is a submatrix of the matrix O, 2. The method according to claim 1, characterized in that The method further includes: the computing core determining the values ​​of the x, the y1 and / or the y2 according to the SRAM capacity of the computing core.

3. The method according to claim 1 or 2, characterized in that: The calculation core determines a second result O'' of calculating the first matrix, the third sub-matrix K'' and the fourth sub-matrix V'' according to the self-attention mechanism, including: The computing core determines a first intermediate result S according to the first matrix and the third sub-matrix K''; wherein the first intermediate result S is a transposed matrix K'' of the first matrix and the third sub-matrix K''. T The result of performing matrix multiplication; The computing core determines a second intermediate result P according to the first intermediate result S; wherein the second intermediate result P is a result of normalizing the first intermediate result S; The calculation core determines the second result O" according to the second intermediate result P and the fourth sub-matrix V", and the second result O" is the result of matrix multiplication of the second intermediate result P and the fourth sub-matrix V".

4. The method according to claim 3, characterized in that The first intermediate result S and the second intermediate result P are stored in a static random access memory SRAM within the computing core.

5. The method according to claim 4, characterized in that Also includes: Before determining the second intermediate result P, reading the S from the SRAM; Before determining the second result O", the P is read from the SRAM.

6. A chip, characterized in that: The chip includes a computing core, and the computing core includes: A self-attention calculation unit, used to determine a first result O' of calculating a first matrix, a first submatrix K', and a second submatrix V' according to a self-attention mechanism; wherein the first matrix is ​​a submatrix in a query matrix Q or the first matrix is ​​the query matrix Q, the first submatrix K' is a submatrix in a key matrix K, and the second submatrix V' is a submatrix in a value matrix V; x, y1, N and d are positive integers, x≤N, y1<N; The self-attention calculation unit is further used to determine a second result O" of calculating the first matrix, the third submatrix K" and the fourth submatrix V" according to the self-attention mechanism; wherein the third submatrix K" is a submatrix other than the first submatrix K' in the matrix K, and the fourth submatrix V" is a submatrix other than the second submatrix V' in the matrix V; wherein, y2 is a positive integer less than N-y1, a fusion unit, configured to determine a third result at least according to the first result O' and the second result O", wherein the third result is a matrix O or a submatrix of the matrix O obtained by calculating the matrix Q, the matrix K and the matrix V according to a self-attention mechanism, 7. The chip according to claim 6, characterized in that: The computing core also includes: A partitioning unit is used to determine the value of x, y1 and / or y2 according to the SRAM capacity of the computing core.

8. The chip according to claim 6 or 7, characterized in that: The self-attention calculation unit is further used to determine a second result O'' of calculating the first matrix, the third sub-matrix K'' and the fourth sub-matrix V'' according to the self-attention mechanism, including: The self-attention calculation unit is further used to determine a first intermediate result S according to the first matrix and the third sub-matrix K''; wherein the first intermediate result S is a transposed matrix K'' of the first matrix and the third sub-matrix K''. T The result of performing matrix multiplication; The self-attention calculation unit is further used to determine a second intermediate result P according to the first intermediate result S; wherein the second intermediate result P is a result of normalizing the first intermediate result S; The self-attention calculation unit is also used to determine the second result O" based on the second intermediate result P and the fourth sub-matrix V", and the second result O" is the result of matrix multiplication of the second intermediate result P and the fourth sub-matrix V".

9. The chip according to claim 8, characterized in that: The first intermediate result S and the second intermediate result P are stored in a static random access memory SRAM within the computing core.

10. A chip, characterized in that: The computing core comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to implement the method as claimed in any one of claims 1 to 4.

11. A data processing system, characterized in that: It includes a chip and a control device; the chip includes multiple computing cores; The control device is used to obtain a computing task, wherein the computing task is used to instruct to calculate the task data according to the self-attention mechanism; the task data includes: query matrices, key matrices and value matrices corresponding to h head dimensions in the multi-head self-attention corresponding to the training data of B batch dimensions respectively; The control device is further used to divide the computing task into a plurality of subtasks; wherein the query matrix, key matrix and value matrix corresponding to the same head dimension of the training data of the same batch dimension are divided into the task data of the same subtask; each of the subtasks is used to indicate that the task data in the subtask is calculated according to the self-attention mechanism; The control device is further used to send the multiple subtasks to different computing cores in the chip respectively; After receiving the subtask from the control device, each computing core in the chip executes the method according to any one of claims 1 to 4.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program. When the computer program is executed by a chip, the chip executes the method according to any one of claims 1 to 4.

13. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are executed on a chip, the chip executes the method according to any one of claims 1 to 4.

Images