Multi-head attention mechanism operator optimization method and system for domestic heterogeneous processor

Abstract

The invention discloses an optimization method for a multi-head attention mechanism operator oriented to a domestic heterogeneous processor, which improves the efficiency of the multi-head attention mechanism operator from four aspects, and comprises the following steps of: firstly, combining the general computing power of a CPU (Central Processing Unit) and the advantages of a DSP (Digital Signal Processor) in the aspect of parallel computing; a high-performance multi-head attention mechanism operator operation method on a CPU + DSP heterogeneous processor is constructed; secondly, a previous softmax optimization algorithm is used for fusing the two batch processing matrix multiplication and a softmax operator, so that the memory access overhead of DSP equipment is reduced; then, a vector operation instruction of a DSP calculation core is used in the calculation process of a multi-head attention operator, and SIMD calculation characteristics of DSP equipment are fully utilized; and finally, during data transmission and calculation, a double-flow water mechanism is adopted, the data transmission time is covered, and the multi-head attention mechanism operator calculation efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the fields of computer technology, deep learning, and heterogeneous acceleration technology, and more specifically, to an optimization method and system for a multi-head attention mechanism operator for domestic heterogeneous processors. Background Art

[0002] In recent years, large models based on Transformer have developed rapidly, and model parameters have continued to increase, resulting in more stringent performance requirements for model training and reasoning. The multi-head attention mechanism operator is one of the core components of the Transformer model. It captures different features in the input sequence by calculating multiple attention heads in parallel. When processing sequence data, the multi-head attention mechanism operator has significant advantages. It can dynamically focus on information at different positions in the input sequence, thereby capturing long-distance dependencies. In addition, the execution time of the multi-head attention mechanism operator often accounts for more than 70% of the total model execution time. Therefore, making full use of hardware computing resources and optimizing and accelerating the multi-head attention mechanism operator is the key to improving model training and reasoning performance.

[0003] A heterogeneous processor is a computing device that integrates multiple different types of processing cores, designed to optimize the performance of specific tasks by working together. At present, the optimization method for the multi-head attention mechanism operator operation for heterogeneous processors has been widely used to improve the training and reasoning performance of the Transformer model. This optimization method is for the CPU (Central Processing Unit, referred to as CPU) + GPU (Graphics Processing Unit, referred to as GPU) heterogeneous processor system. Its optimization strategy at the implementation level is mainly in four aspects: parallelization (multi-head parallelism, matrix multiplication parallelism), memory optimization (memory access mode optimization, data reuse), algorithm optimization (Softmax optimization, quantization), and hardware acceleration (Tensor Cores, dedicated processors).

[0004] However, existing optimization methods for multi-head attention operators for heterogeneous processors have some non-negligible defects:

[0005] First, this optimization method is only applicable to CPU+GPU heterogeneous processors, and is completely unsuitable for CPU+DSP heterogeneous processors.

[0006] Second, since the optimization method does not use operator fusion technology between the batch matrix multiplication and the Softmax process, there is a lot of expensive memory access overhead between different operators. Summary of the invention

[0007] In view of the above defects or improvement needs of the prior art, the present invention provides an optimization method and system for a multi-head attention mechanism operator for domestic heterogeneous processors, which aims to solve the technical problem that the existing optimization method for the multi-head attention mechanism operator for heterogeneous processors is only applicable to CPU+GPU heterogeneous processors, but is completely inapplicable to CPU+DSP heterogeneous processors, and the technical problem that a large amount of expensive memory access overhead exists between different operators due to the lack of operator fusion technology between the batch matrix multiplication and the Softmax process of the optimization method.

[0008] To achieve the above object, according to one aspect of the present invention, a method for optimizing a multi-head attention mechanism operator for domestic heterogeneous processors is provided, comprising the following steps:

[0009] (1) The CPU obtains the calculation parameters of the multi-head attention mechanism operator, allocates DSP storage space according to the calculation parameters of the multi-head attention mechanism operator, initializes the DSP execution thread to obtain the thread number of the DSP execution thread, inputs the calculation parameters of the multi-head attention mechanism operator into the DSP thread execution function according to the thread number of the DSP execution thread and runs it to start all the computing cores in the DSP.

[0010] (2) Each computing core in the DSP obtains the total number of computing tasks of all computing cores in the DSP according to the computing parameters of the multi-head attention mechanism operator, and obtains its own computing tasks to be executed based on the obtained total number of computing tasks and the total number of computing cores in the DSP.

[0011] (3) Each computing core in the DSP sets counter j = 0;

[0012] (4) Each computing core in the DSP determines whether j is less than the total number of computing tasks cal_num of all computing cores in the DSP obtained in step (2). If so, it proceeds to step (5); otherwise, it proceeds to step (16);

[0013] (5) Each computing core in the DSP obtains the query matrix pointer my_queries, key-value matrix pointer my_keys, value matrix pointer my_values, and output matrix pointer my_o corresponding to the computing task to be executed by the computing core obtained in step (2) according to the computing parameters of the multi-head attention mechanism operator; obtains the key-value matrix according to the key-value matrix pointer, and transposes the key-value matrix to obtain the transposed matrix of the key-value matrix, respectively obtains the query matrix and the value matrix (the size of which is SEQ_LEN rows and HEAD_SIZE columns) according to the query matrix pointer and the value matrix pointer, sets the block parameter BLOCK_SEQ_LEN according to the type of the input matrix pointer, and blocks the query matrix and the value matrix according to the block parameter to obtain multiple first sub-matrices and multiple second sub-matrices respectively, and obtains the total number of blocks corresponding to the computing task to be executed by the computing core according to the block parameter.

[0014] (6) Each computing core in the DSP sets counter i=0;

[0015] (7) Each computing core in the DSP determines whether i is less than the total number of blocks sr. If yes, it proceeds to step (9); otherwise, it proceeds to step (16);

[0016] (8) Each computing core in the DSP divides the transposed matrix of the key value matrix into a third submatrix of multiple HEAD_SIZE rows and BLOCK_SEQ_LEN columns according to the block parameters obtained in step (5), obtains a fourth submatrix of size BLOCK_SEQ_LEN rows and HEAD_SIZE columns according to the pointer of the output matrix, and initializes all elements in the fourth submatrix to 0.

[0017] (9) Each computing core in the DSP sets counter k=0;

[0018] (10) Each computing core in the DSP creates and initializes four arrays of length BLOCK_SEQ_LEN, Gmax and Gmax_temp to -inf, and GL and GL_temp to 0.

[0019] (11) Each computing core in the DSP determines whether k is less than or equal to i. If so, it proceeds to step (12); otherwise, it proceeds to step (16);

[0020] (12) Each computing core in the DSP performs a matrix multiplication operation on the i-th first submatrix obtained in step (8) and the k-th third submatrix obtained in step (10), and the result obtained is the fifth submatrix:

[0021] (13) Each computing core in the DSP determines whether k is equal to i. If so, a mask matrix is ​​generated, which is a matrix with a size of BLOCK_SEQ_LEN rows and BLOCK_SEQ_LEN columns, with all upper triangular values ​​being -inf and lower triangular values ​​being 0, and a matrix addition operation is performed on the fifth submatrix obtained in step (12) and the mask matrix to obtain the fifth submatrix after the mask operation, and then the process goes to step (14); otherwise, the process goes to step (14);

[0022] (14) Each computing core in the DSP performs softmax calculation on the fifth submatrix after the mask operation obtained in step (13) to obtain a sixth submatrix, updates the fourth submatrix using the intermediate result in the softmax calculation process to obtain an updated fourth submatrix, calculates the matrix multiplication result between the sixth submatrix and the second submatrix as a slice O_new of the fourth submatrix, and adds the obtained slice O_new to the updated fourth submatrix to obtain the final updated fourth submatrix.

[0023] (15) Each computing core in the DSP sets a counter k=k+1 and returns to step (11);

[0024] (16) Each computing core in the DSP sets counter i=i+1 and returns to step (7);

[0025] (17) Each computing core in the DSP sets a counter j=j+n_threads and returns to step (3);

[0026] (18) Each computing core in the DSP returns the fourth submatrix obtained in step (14) to the CPU as the calculation result of the DSP multi-head attention mechanism operator;

[0027] (19) The CPU passes the calculation results of the multi-head attention mechanism operator from each computing core in the DSP into the calculation parameters of the multi-head attention mechanism operator.

[0028] Preferably, the calculation parameters of the multi-head attention mechanism operator in step (1) include: the function name of the multi-head attention mechanism operator, the number of input data batches batch_size, the number of heads n_heads, the length of the input sequence SEQ_LEN, the size of the head HEAD_SIZE, the pointer of the input matrix (which includes the query matrix pointer queries, the key-value matrix pointer keys, the value matrix pointer values), and the pointer o of the output matrix.

[0029] Preferably, the process of allocating DSP storage space according to the calculation parameters of the multi-head attention mechanism operator in step (1) is specifically as follows: first, according to the number of input data batches, the number of heads, the length of the input sequence, and the size of the head in the calculation parameters of the multi-head attention mechanism operator, the size of the input matrix and the output matrix is ​​obtained. The calculation formula is:

[0030] size=batch_size*SEQ_LEN*n_heads*HEAD_SIZE

[0031] Then, according to the calculated sizes of the input matrix and output matrix, the hthread_mall oc interface is called to allocate the storage space of the DSP input matrix and output matrix. Then, the input matrix and output matrix are transferred to the storage space of the DSP input matrix and output matrix respectively. Finally, the pointers of the input matrix and output matrix are pointed to the storage space of the input matrix and output matrix respectively.

[0032] In step (1), the CPU initializes the DSP execution thread to obtain the thread number of the DSP execution thread. Specifically, the process includes obtaining the DSP cluster number and the number of DSP cores started, n_threads, initializing the DSP execution thread according to the obtained DSP cluster number and the number of DSP cores started, and calling the hthread_create_group function to obtain the thread number of the DSP execution thread.

[0033] In step (1), the multi-head attention mechanism operator calculation parameters are input into the DSP thread execution function according to the thread number of the DSP execution thread and run to start each computing core in the DSP. Specifically, this process is to create an empty parameter array, store the obtained calculation parameters of the multi-head attention mechanism operator and the pointers of the input matrix and the output matrix into the parameter array, and then call the hthread_group_exec function to input the obtained DSP execution thread number and parameter array into the DSP thread execution function in turn to start each computing core in the DSP.

[0034] Preferably, in step (2), each computing core in the DSP is based on the number of batches of input data. batch _size and number of headers n_ h ea d s To determine the total number of computing tasks cal_num for all computing cores in DSP, specifically:

[0035] cal_num = batch_size × n_heads

[0037] Preferably, in step (5), each computing core in the DSP obtains the query matrix pointer my_queries, the key matrix pointer my_keys, the value matrix pointer my_values, and the output matrix pointer my_o corresponding to the current computing task to be executed according to the computing parameters of the multi-head attention mechanism operator. This process adopts the following calculation formula:

[0038] my_queries = queries + offset

[0039] my_keys = keys + offset

[0040] my_values ​​= values ​​+ offset

[0041] my_o=o+offset

[0042] Among them are:

[0043] offset=batch_idx×SEQ_LEN×n_heads×HEAD_SIZE+head_idx×HEAD_SIZE

[0044] And there are:

[0045] batch_idx=j / n_heads

[0046] head_idx=j%n_heads

[0047] In step (5), each computing core in the DSP obtains the query matrix and the value matrix respectively according to the query matrix pointer and the value matrix, sets the block parameter BLOCK_SEQ_LEN according to the type of the pointer of the input matrix, and blocks the query matrix and the value matrix according to the block parameter to obtain multiple first sub-matrices and second sub-matrices, and obtains the total number of blocks corresponding to the computing task to be executed by the computing core according to the block parameter. This process is specifically as follows: for each computing core in the DSP, the pointer type of the input matrix is ​​single precision or half precision, and its block parameter is set to 96 or 192 respectively, the query matrix is ​​divided into multiple first sub-matrices of BLOCK_SEQ_LEN rows and HEAD_SIZE columns, and the total number of blocks sr corresponding to the computing task to be executed by the computing core in the DSP is obtained: sr = SEQ_LEN / BLOCK_SEQ_LEN.

[0048] Preferably, step (12) comprises the following sub-steps:

[0049] (12-1) Each computing core in the DSP performs block processing on the first submatrix obtained in step (8) to obtain CntQ (where CntQ = BLOCK_SEQ_LEN\Q_SM_BLOCK_RO W) small blocks with Q_SM_BLOCK_ROW rows and HEAD_SZIE columns, where Q_SM_BLOCK_ROW> the instruction delay of the DSP multiply-add instruction and is close to the power of two of the instruction delay of the DSP multiply-add instruction;

[0050] (12-2) Each computing core in the DSP sets a counter q=0;

[0051] (12-3) Each computing core in the DSP starts the DMA transfer function twice to transfer the CntQ small blocks obtained in step (12-1) to the SM in sequence to obtain the first sub-matrix small block in the SM space.

[0052] (12-4) Each computing core in the DSP determines whether q is less than CntQ. If so, it calculates the matrix multiplication between the first submatrix block obtained in step (12-3) and the third submatrix to obtain the fifth submatrix of Q_SM_BLOCK_ROW rows and BLOCK_SEQ_LEN columns, and then proceeds to step (12-5), otherwise the process ends.

[0053] (12-5) Each computing core in the DSP sets a counter q=q+1 and returns to step (12-4).

[0054] Preferably, step (14) specifically includes the following sub-steps:

[0055] (14-1) Each computing core in the DSP obtains the maximum value max_now of each row in the fifth submatrix obtained in step (13), and the index sum L_NOW of all rows in the fifth submatrix, and obtains the sixth submatrix according to the maximum value max_now and the index sum L_NOW, and uses the sixth submatrix to update the fourth submatrix to obtain an updated fourth submatrix;

[0056] (14-2) Each computing core in the DSP performs a matrix multiplication operation on the sixth submatrix and the second submatrix obtained in step (8), and the operation result is used as a slice O_new of the fourth submatrix, and the obtained slice O_new is added to the updated fourth submatrix to obtain the final updated fourth submatrix.

[0057] Preferably, step (14-1) specifically includes the following sub-steps:

[0058] (14-1-1) Each computing core in the DSP sets the counter oi=0;

[0059] (14-1-2) Each computing core in the DSP determines whether oi is less than the block parameter BLOCK_SEQ_LEN. If so, it goes to step (14-1-3), otherwise the process ends;

[0060] (14-1-3) Each computing core in the DSP calculates the maximum value max_temp of the oi-th row in the fifth submatrix, and assigns the larger value of max_temp and GMAX[oi] to GMAX_temp[oi], where GMAX[0] = 0;

[0061] (14-1-4) Each computing core in the DSP performs exponential operations on all elements of the oi-th row of the fifth submatrix and then sums them up, and assigns the sum result to GL_temp[oi].

[0062] (14-1-5) Each computing core in the DSP updates the oi-th row in the fourth submatrix to G_L[oi] = G_L[oi]*exp(G_MAX[oi]-G__temp[oi]) / (GL_temp[oi]*GL[oi])+GL_temp[oi] to obtain the updated fourth submatrix, performs exponential operation on each element in the oi-th row of the fourth submatrix, divides all the exponential operation results by GMAX_temp[oi], and uses the results as the oi-th row of the sixth submatrix, and assigns GMAX_temp[oi] to GMAX[oi].

[0063] (14-1-6) Each computing core in the DSP sets the counter oi=oi+1 and returns to step (14-1-2).

[0064] Preferably, step (14-2) specifically comprises the following steps:

[0065] (14-2-1) Each computing core in the DSP performs block processing on the second sub-matrix according to the block parameter BLOCK_SEQ_LEN to obtain a second sub-matrix block of nv BLOCK_SEQ_LEN rows and BLOCK_SEQ_LEN columns, where nv = HEAD_SIZE / BLOCK_SEQ_LEN.

[0066] (14-2-2) Each computing core in the DSP sets counter ii=0;

[0067] (14-2-3) Each computing core in the DSP determines whether ii is less than nv. If so, it obtains the ii-th second submatrix block obtained in step (14-2-1) and then proceeds to step (14-2-4). Otherwise, the process ends.

[0068] (14-2-4) Each computing core in the DSP performs block processing on the sixth submatrix obtained in step (14-1) according to the block parameter Q_SM_BLOCK_ROW to obtain a sixth submatrix block of np Q_SM_BLOCK_ROW rows and BLOCK_SEQ_LEN columns, where np = BLOCK_SEQ_LEN / Q_SM_BLOCK_ROW.

[0069] (14-2-5) Each computing core in DSP sets counter jj=0;

[0070] (14-2-6) Each computing core in the DSP determines whether jj is less than np. If so, it performs a matrix multiplication operation on the jjth sixth submatrix block obtained in step (14-2-4) and the second submatrix block obtained in step (14-2-3). The operation result is used as the sliced ​​block, and then goes to step (14-2-7). Otherwise, it goes to step (14-2-8);

[0071] (14-2-7) Each computing core in the DSP sets counter jj=jj+1 and returns to step (14-2-5);

[0072] (14-2-8) Each computing core in the DSP sets counter ii=ii+1 and returns to step (14-2-2);

[0073] (14-2-9) Each computing core in the DSP splices the small slice blocks obtained in step (14-2-6) to obtain a slice O_new, and adds the slice to the fourth submatrix updated in step (14-1) to obtain the final updated fourth submatrix.

[0074] According to another aspect of the present invention, there is provided an optimization system for a multi-head attention mechanism operator for domestic heterogeneous processors, comprising:

[0075] The first module is set in the CPU and is used to obtain the calculation parameters of the multi-head attention mechanism operator, allocate DSP storage space according to the calculation parameters of the multi-head attention mechanism operator, initialize the DSP execution thread to obtain the thread number of the DSP execution thread, and input the calculation parameters of the multi-head attention mechanism operator into the DSP thread execution function according to the thread number of the DSP execution thread and run it to start all computing cores in the DSP.

[0076] The second module is set in each computing core in the DSP, and is used to obtain the total number of computing tasks of all computing cores in the DSP according to the computing parameters of the multi-head attention mechanism operator, and obtain its own computing tasks to be executed based on the obtained total number of computing tasks and the total number of computing cores in the DSP.

[0077] The third module is arranged in each computing core in the DSP and is used to set the counter j=0;

[0078] The fourth module is set in each computing core in the DSP, and is used to determine whether j is less than the total computing task number cal_num of all computing cores in the DSP obtained by the second module. If yes, it is transferred to the fifth module, otherwise it is transferred to the sixteenth module;

[0079] The fifth module is set in each computing core in the DSP, and is used to obtain the query matrix pointer my_queries, key value matrix pointer my_keys, value matrix pointer my_values, and output matrix pointer my_o corresponding to the computing task to be executed by the computing core obtained in the second module according to the computing parameters of the multi-head attention mechanism operator; obtain the key value matrix according to the key value matrix pointer, and transpose the key value matrix to obtain the transposed matrix of the key value matrix, respectively obtain the query matrix and the value matrix (the size of which is SEQ_LEN rows and HEAD_SI ZE columns) according to the query matrix pointer and the value matrix pointer, set the block parameter BLOCK_SEQ_LEN according to the type of the input matrix pointer, and block the query matrix and the value matrix according to the block parameter to obtain multiple first sub-matrices and multiple second sub-matrices respectively, and obtain the total number of blocks corresponding to the computing task to be executed by the computing core according to the block parameter.

[0080] A sixth module, which is arranged in each computing core in the DSP, is used to set a counter i=0;

[0081] The seventh module is set in each computing core in the DSP, and is used to determine whether i is less than the total number of blocks sr. If yes, it goes to the ninth module, otherwise it goes to the sixteenth module;

[0082] The eighth module is set in each computing core in the DSP, and is used to divide the transposed matrix of the key value matrix into a third sub-matrix with multiple HEAD_SIZE rows and BLOCK_SEQ_LEN columns according to the block parameters obtained in the fifth module, obtain a fourth sub-matrix with a size of BLOCK_SEQ_LEN rows and HEAD_SIZE columns according to the pointer of the output matrix, and initialize all elements in the fourth sub-matrix to 0.

[0083] A ninth module, which is arranged in each computing core in the DSP, and is used to set a counter k=0;

[0084] The tenth module is set in each computing core in the DSP, and is used to create and initialize four arrays Gmax and Gmax_temp with a length of BLOCK_SEQ_LEN to -inf, and GL and GL_temp to 0.

[0085] The eleventh module is set in each computing core in the DSP, and is used to determine whether k is less than or equal to i. If so, it is transferred to the twelfth module, otherwise it is transferred to the sixteenth module;

[0086] The twelfth module is set in each computing core in the DSP, and is used to perform a matrix multiplication operation on the i-th first sub-matrix obtained by the eighth module and the k-th third sub-matrix obtained by the tenth module, and the result obtained is the fifth sub-matrix:

[0087] The thirteenth module is set in each computing core in the DSP, and is used to determine whether k is equal to i. If so, a mask matrix is ​​generated, which is a matrix with a size of BLOCK_SEQ_LEN rows and BLOCK_SEQ_LEN columns, with all upper triangular values ​​being -inf and lower triangular values ​​being 0, and a matrix addition operation is performed on the fifth submatrix obtained in step (12) and the mask matrix to obtain the fifth submatrix after the mask operation, and then the process is transferred to the fourteenth module; otherwise, the process is transferred to the fourteenth module;

[0088] The fourteenth module is arranged in each computing core in the DSP, and is used to perform softmax calculation on the fifth submatrix after the mask operation obtained by the thirteenth module to obtain the sixth submatrix, update the fourth submatrix using the intermediate result in the softmax calculation process to obtain the updated fourth submatrix, calculate the matrix multiplication result between the sixth submatrix and the second submatrix as the slice O_new of the fourth submatrix, and add the obtained slice O_new to the updated fourth submatrix to obtain the final updated fourth submatrix.

[0089] A fifteenth module is provided in each computing core in the DSP, and is used to set a counter k=k+1, and return to the eleventh module;

[0090] A sixteenth module, which is provided in each computing core in the DSP, is used to set a counter i=i+1 and return to the seventh module;

[0091] A seventeenth module is provided in each computing core in the DSP, and is used to set a counter j=j+n_threads, and return to the third module;

[0092] An eighteenth module is provided in each computing core in the DSP, and is used to return the fourth sub-matrix obtained in the fourteenth module to the CPU as the calculation result of the DSP multi-head attention mechanism operator;

[0093] The nineteenth module is set in the CPU and is used to transfer the calculation results of the multi-head attention mechanism operator from each computing core in the DSP into the calculation parameters of the multi-head attention mechanism operator.

[0094] In general, the technical solution conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0095] (1) The present invention adopts steps (1) to (19), and in the heterogeneous computing process for CPU+DSP, combines the characteristics of CPU and DSP, fully utilizes the general computing power of CPU and the advantages of DSP in parallel computing, thereby constructing a high-performance multi-head attention mechanism operator operation method on CPU+DSP heterogeneous processors, solving the problem of lack of accelerated multi-head attention mechanism operation for CPU+DSP heterogeneous processors in existing methods;

[0096] (2) The present invention uses steps (5) to (16), which uses an online softmax optimization algorithm to perform operator fusion on batch matrix multiplication and softmax operation in the multi-head attention mechanism, thereby realizing data reuse and reducing the hardware access to DDR memory (Double Data Rate Memory, DDR for short), thereby reducing the memory access overhead between different operators and improving the computational efficiency of the multi-head attention mechanism operators;

[0097] (3) Since the present invention adopts step (12) and step (14), the vector operation instructions of the DSP computing core are used in the calculation process, and the SIMD (single instruction multiple data) computing characteristics of the DSP device are fully utilized;

[0098] (4) Since the present invention adopts step (12), its dual-pipeline computing mechanism can construct two pipelines during calculation, so that the data transmission time is hidden by the calculation time, thereby improving the utilization rate of device resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 It is a flow chart of the optimization method of the multi-head attention mechanism operator for domestic heterogeneous processors of the present invention;

[0100] Figure 2 is a schematic diagram of the multi-core parallel partitioning process of the present invention;

[0101] Figure 3 It is a schematic diagram of block control of the query matrix Q, the transpose KT of the key matrix, and the QKT matrix multiplication result matrix S of the present invention;

[0102] Figure 4 It is a schematic diagram of matrix block control of the softmax operation result matrix P, value matrix V, and output matrix O of the S matrix of the present invention;

[0103] Figure 5 It is a schematic diagram of the double water flow mechanism of the present invention. DETAILED DESCRIPTION

[0104] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0105] The heterogeneous processor platform involved in the present invention is the MT3000 heterogeneous multi-core microprocessor, which is a mass-produced chip of Tianhe's new generation supercomputer. It is completely independently developed and designed by my country and has powerful computing performance. MT3000 consists of a 16-core ARMv8 CPU and 4 DSP acceleration clusters. Each DSP cluster consists of 6 DSP supernodes. Each DSP supernode consists of 4 DSP cores. The cluster shares 6MB of on-chip storage space Global Shared Memory (GSM) and 32GB of off-chip DDR space.

[0106] The DSP core includes an instruction fetch unit (IFU), a scalar processing unit (SPU), a 64KB scalar memory (SM), a vector processing unit (VPU), and a 768KB array memory (AM). There are 16 vector processing units (VPEs) in each VPU, and each VPE can execute 3 floating-point multiplication and addition instructions at a time. Each storage space transfers data through direct memory access (DMA). The SM cannot transfer data to the AM through DMA, and needs to be transferred through GSM, or the data in the AM is read into the vector register (VR) and then read into the scalar space through the scalar vector register (SVR).

[0107] The basic idea of ​​the present invention is to improve the efficiency of the multi-head attention mechanism operator from four aspects. First, combining the general computing power of the CPU and the advantages of the DSP in parallel computing, a high-performance multi-head attention mechanism operator operation method on the CPU+DSP heterogeneous processor is constructed; secondly, the prior softmax optimization algorithm is used to fuse the two batch matrix multiplications and the softmax operator, thereby reducing the memory access overhead of the DSP device; then, the vector operation instructions of the DSP computing core are used in the calculation process of the multi-head attention operator, making full use of the SIMD computing characteristics of the DSP device; finally, during data transmission and calculation, a dual-pipeline mechanism is adopted to mask the data transmission time and improve the calculation efficiency of the multi-head attention mechanism operator.

[0108] like Figure 1 As shown, the present invention provides an optimization method for a multi-head attention mechanism operator for domestic heterogeneous processors, comprising the following steps:

[0109] (1) The CPU obtains the calculation parameters of the multi-head attention mechanism operator, allocates DSP storage space according to the calculation parameters of the multi-head attention mechanism operator, initializes the DSP execution thread to obtain the thread number of the DSP execution thread, inputs the calculation parameters of the multi-head attention mechanism operator into the DSP thread execution function according to the thread number of the DSP execution thread and runs it to start all the computing cores in the DSP.

[0110] Specifically, the calculation parameters of the multi-head attention mechanism operator in this step include: the function name of the multi-head attention mechanism operator, the number of input data batches batch_size, the number of heads n_heads, the length of the input sequence SEQ_LEN, the size of the head HEAD_SIZE, the pointer of the input matrix (which includes the query matrix pointer queries, the key-value matrix pointer keys, the value matrix pointer values), and the pointer o of the output matrix.

[0111] In this step, the process of allocating DSP storage space according to the calculation parameters of the multi-head attention mechanism operator is as follows: first, according to the number of input data batches, the number of heads, the length of the input sequence, and the size of the head in the calculation parameters of the multi-head attention mechanism operator, the size of the input matrix and the output matrix is ​​obtained. The calculation formula is:

[0112] size=batch_size*SEQ_LEN*n_heads*HEAD_SIZE

[0113] Then, according to the calculated sizes of the input matrix and output matrix, the hthread_mall oc interface is called to allocate the storage space of the DSP input matrix and output matrix. Then, the input matrix and output matrix are transferred to the storage space of the DSP input matrix and output matrix respectively. Finally, the pointers of the input matrix and output matrix are pointed to the storage space of the input matrix and output matrix respectively.

[0114] In this step, the CPU initializes the DSP execution thread to obtain the thread number of the DSP execution thread. Specifically, the process includes obtaining the DSP cluster number and the number of started DSP cores n_threads, initializing the DSP execution thread according to the obtained DSP cluster number and the number of started DSP cores, and calling the hthread_create_group function to obtain the thread number of the DSP execution thread.

[0115] Specifically, the number of DSP cores started in this step is exactly the same as the number of heads in the calculation parameters of the multi-head attention mechanism operator, so as to avoid wasting hardware resources due to too many DSP cores started.

[0116] In this step, the multi-head attention mechanism operator calculation parameters are input into the DSP thread execution function according to the thread number of the DSP execution thread and run to start each computing core in the DSP. Specifically, this process is to create an empty parameter array, store the obtained calculation parameters of the multi-head attention mechanism operator and the pointers of the input matrix and the output matrix into the parameter array, and then call the hthread_grou p_exec function to input the obtained DSP execution thread number and parameter array into the DSP thread execution function in turn to start each computing core in the DSP.

[0117] (2) Each computing core in the DSP obtains the total number of computing tasks of all computing cores in the DSP according to the computing parameters of the multi-head attention mechanism operator, and obtains its own computing tasks to be executed according to the obtained total number of computing tasks and the total number of computing cores in the DSP (such as Figure 2 shown).

[0118] Specifically, each computing core in the DSP is based on the number of batches of input data. b atc h _size and the number of heads n_ h ea d s To determine the total number of computing tasks cal_num for all computing cores in the DSP:

[0119] cal_num = batch_size × n_heads

[0120] Each computing core in the DSP obtains computing tasks based on the total number of computing tasks and its computing core number. If the number of input data batches = 2, the number of heads = 4, and the total number of computing cores in the DSP = 3, the total number of computing tasks is 8. The first computing core in the DSP obtains computing tasks 0, 3, and 6 to be executed, the second computing core in the DSP obtains computing tasks 1, 4, and 7 to be executed, and the third computing core in the DSP obtains computing tasks 2 and 5 to be executed.

[0121] (3) Each computing core in the DSP sets counter j = 0;

[0122] (4) Each computing core in the DSP determines whether j is less than the total number of computing tasks cal_num of all computing cores in the DSP obtained in step (2). If so, it proceeds to step (5); otherwise, it proceeds to step (16);

[0123] (5) Each computing core in the DSP obtains the query matrix pointer my_queries, key value matrix pointer my_keys, value matrix pointer my_values, and output matrix pointer my_o corresponding to the computing task to be performed by the computing core obtained in step (2) according to the computing parameters of the multi-head attention mechanism operator; obtains the key value matrix according to the key value matrix pointer, and transposes the key value matrix to obtain the transposed matrix of the key value matrix; obtains the query matrix and the value matrix (both of which have a size of SEQ_LEN rows and HEAD_SIZE columns) according to the query matrix pointer and the value matrix pointer, respectively; sets the block parameter BLOCK_SEQ_LEN according to the type of the input matrix pointer, and performs an operation on the query matrix (see Figure 3 ) and value matrix (see Figure 4 ) are divided into blocks respectively to obtain multiple first sub-matrices and multiple second sub-matrices, and the total number of blocks corresponding to the computing tasks to be executed by the computing core is obtained according to the block parameters.

[0124] In this step, each computing core in the DSP obtains the query matrix pointer my_queries, key value matrix pointer my_keys, value matrix pointer my_values, and output matrix pointer my_o corresponding to the current computing task to be executed according to the calculation parameters of the multi-head attention mechanism operator. This process uses the following calculation formula:

[0125] my_queries = queries + offset

[0126] my_keys = keys + offset

[0127] my_values ​​= values ​​+ offset

[0128] my_o=o+offset

[0129] Among them are:

[0130] offset=batch_idx×SEQ_LEN×n_heads×HEAD_SIZE+head_idx×HEAD_SIZE

[0131] And there are:

[0132] batch_idx=j / n_heads

[0133] head_idx=j%n_heads

[0134] In this step, each computing core in the DSP obtains the query matrix and the value matrix according to the query matrix pointer and the value matrix respectively, sets the block parameter BLOCK_SEQ_LEN according to the type of the input matrix pointer, and divides the query matrix and the value matrix into blocks according to the block parameter (see Figure 3 ) to obtain multiple first sub-matrices and second sub-matrices, and obtain the total number of blocks corresponding to the computing task to be executed by the computing core according to the block parameter. This process is specifically as follows: the pointer type of each computing core in the DSP for the input matrix is ​​single precision or half precision, and its block parameter is set to 96 or 192 respectively, and the query matrix is ​​divided into multiple first sub-matrices of BLOCK_SEQ_LEN rows and HEAD_SIZE columns, and the total number of blocks sr corresponding to the computing task to be executed by the computing core in the DSP is obtained: sr = SEQ_LEN / BLOCK_SEQ_LEN.

[0135] (6) Each computing core in the DSP sets counter i=0;

[0136] (7) Each computing core in the DSP determines whether i is less than the total number of blocks sr. If yes, it proceeds to step (9); otherwise, it proceeds to step (16);

[0137] (8) Each computing core in the DSP divides the transposed matrix of the key value matrix into a third submatrix of multiple HEAD_SIZE rows and BLOCK_SEQ_LEN columns according to the block parameters obtained in step (5), obtains a fourth submatrix of size BLOCK_SEQ_LEN rows and HEAD_SIZE columns according to the pointer of the output matrix, and initializes all elements in the fourth submatrix to 0.

[0138] (9) Each computing core in the DSP sets counter k=0;

[0139] (10) Each computing core in the DSP creates and initializes four arrays of length BLOCK_SEQ_LEN, Gmax and Gmax_temp to -inf, and GL and GL_temp to 0.

[0140] (11) Each computing core in the DSP determines whether k is less than or equal to i. If so, it proceeds to step (12); otherwise, it proceeds to step (16);

[0141] (12) Each computing core in the DSP performs a matrix multiplication operation on the i-th first submatrix obtained in step (8) and the k-th third submatrix obtained in step (10), and the result obtained is the fifth submatrix:

[0142] This step includes the following sub-steps:

[0143] (12-1) Each computing core in the DSP performs block processing on the first submatrix obtained in step (8) to obtain CntQ (where CntQ = BLOCK_SEQ_LEN\Q_SM_BLOCK_ROW) blocks of Q_SM_BLOCK_ROW rows and HEAD_SZIE columns (see Figure 3 ), where Q_SM_BLOCK_ROW>the instruction latency of the DSP multiply-add instruction, and is close to a power of two of the instruction latency of the DSP multiply-add instruction;

[0144] For example, if the instruction latency of the DSP multiply-add instruction is 6, the value range of Q_SM_BLOCK_ROW is a power of two greater than 6 (8, 16, 32, . . . ), preferably 8.

[0145] (12-2) Each computing core in the DSP sets a counter q=0;

[0146] (12-3) Each computing core in the DSP starts the DMA transfer function twice to transfer the CntQ small blocks obtained in step (12-1) to the SM in sequence to obtain the first sub-matrix small block in the SM space.

[0147] Specifically, during the calculation process of step (12-4), two independent input and output data spaces are created in the SM. While calculating the data of the q time step, the data required for the calculation of the q+1 time step is transmitted. While calculating the data of the q+1 time step, the data required for the calculation of the q+2 time step is transmitted, and so on. This dual-pipeline mechanism achieves the purpose of covering up the calculation time and data transmission time (see Figure 5 ).

[0148] The advantage of this sub-step (12-3) is that two pipelines are constructed during the DSP computing core calculation, so that the data transmission time is hidden by the computing time, thereby improving the utilization rate of device resources.

[0149] (12-4) Each computing core in the DSP determines whether q is less than CntQ. If so, it calculates the matrix multiplication between the first submatrix block obtained in step (12-3) and the third submatrix to obtain the fifth submatrix of Q_SM_BLOCK_ROW rows and BLOCK_SEQ_LEN columns, and then proceeds to step (12-5), otherwise the process ends.

[0150] (12-5) Each computing core in the DSP sets a counter q=q+1 and returns to step (12-4);

[0151] (13) Each computing core in the DSP determines whether k is equal to i. If so, a mask matrix is ​​generated, which is a matrix with a size of BLOCK_SEQ_LEN rows and BLOCK_SEQ_LEN columns, with all upper triangular elements being -inf and all lower triangular elements being 0. A matrix addition operation is performed on the fifth submatrix obtained in step (12) and the mask matrix to obtain the fifth submatrix after the mask operation, and then the process goes to step (14). Otherwise, the process goes to step (14);

[0152] (14) Each computing core in the DSP performs softmax calculation on the fifth submatrix after the mask operation obtained in step (13) to obtain a sixth submatrix, updates the fourth submatrix using the intermediate result in the softmax calculation process to obtain an updated fourth submatrix, calculates the matrix multiplication result between the sixth submatrix and the second submatrix as a slice O_new of the fourth submatrix, and adds the obtained slice O_new to the updated fourth submatrix, thereby obtaining the final updated fourth submatrix (see Figure 4 ).

[0153] This step (14) specifically includes the following sub-steps:

[0154] (14-1) Each computing core in the DSP obtains the maximum value max_now of each row in the fifth submatrix obtained in step (13), and the index sum L_NOW of all rows in the fifth submatrix, and obtains the sixth submatrix according to the maximum value max_now and the index sum L_NOW, and uses the sixth submatrix to update the fourth submatrix to obtain an updated fourth submatrix;

[0155] This step (14-1) specifically includes the following sub-steps:

[0156] (14-1-1) Each computing core in the DSP sets the counter oi=0;

[0157] (14-1-2) Each computing core in the DSP determines whether oi is less than the block parameter BLOCK_SEQ_LEN. If so, it goes to step (14-1-3), otherwise the process ends;

[0158] (14-1-3) Each computing core in the DSP calculates the maximum value max_temp of the oi-th row in the fifth submatrix, and assigns the larger value of max_temp and GMAX[oi] to GMAX_temp[oi], where GMAX[0] = 0;

[0159] (14-1-4) Each computing core in the DSP performs exponential operations on all elements of the oi-th row of the fifth submatrix and then sums them up, and assigns the sum result to GL_temp[oi].

[0160] (14-1-5) Each computing core in the DSP updates the oi-th row in the fourth submatrix to G_L[oi] = G_L[oi]*exp(G_MAX[oi]-G__temp[oi]) / (GL_temp[oi]*GL[oi])+GL_temp[oi] to obtain the updated fourth submatrix, performs exponential operation on each element in the oi-th row of the fourth submatrix, divides all the exponential operation results by GMAX_temp[oi], and uses the results as the oi-th row of the sixth submatrix, and assigns GMAX_temp[oi] to GMAX[oi].

[0161] (14-1-6) Each computing core in the DSP sets the counter oi=oi+1 and returns to step (14-1-2);

[0162] (14-2) Each computing core in the DSP performs a matrix multiplication operation on the sixth submatrix and the second submatrix obtained in step (8), and the operation result is used as the slice O_new of the fourth submatrix (see Figure 4 ). And add the obtained slice O_new to the updated fourth sub-matrix to obtain the final updated fourth sub-matrix.

[0163] This step (14-2) specifically includes the following steps:

[0164] (14-2-1) Each computing core in the DSP performs block processing on the second sub-matrix according to the block parameter BLOCK_SEQ_LEN to obtain a second sub-matrix block of nv BLOCK_SEQ_LEN rows and BLOCK_SEQ_LEN columns, where nv = HEAD_SIZE / BLOCK_SEQ_LEN.

[0165] (14-2-2) Each computing core in the DSP sets counter ii=0;

[0166] (14-2-3) Each computing core in the DSP determines whether ii is less than nv. If so, it obtains the ii-th second submatrix block obtained in step (14-2-1) and then proceeds to step (14-2-4). Otherwise, the process ends.

[0167] (14-2-4) Each computing core in the DSP performs block processing on the sixth submatrix obtained in step (14-1) according to the block parameter Q_SM_BLOCK_ROW to obtain a sixth submatrix block of np Q_SM_BLOCK_ROW rows and BLOCK_SEQ_LEN columns, where np = BLOCK_SEQ_LEN / Q_SM_BLOCK_ROW.

[0168] (14-2-5) Each computing core in DSP sets counter jj=0;

[0169] (14-2-6) Each computing core in the DSP determines whether jj is less than np. If so, it performs a matrix multiplication operation on the jjth sixth submatrix block obtained in step (14-2-4) and the second submatrix block obtained in step (14-2-3). The operation result is used as the sliced ​​block, and then goes to step (14-2-7). Otherwise, it goes to step (14-2-8);

[0170] (14-2-7) Each computing core in the DSP sets counter jj=jj+1 and returns to step (14-2-5);

[0171] (14-2-8) Each computing core in the DSP sets counter ii=ii+1 and returns to step (14-2-2);

[0172] (14-2-9) Each computing core in the DSP splices the sliced ​​blocks obtained in step (14-2-6) to obtain a slice O_new, and adds the slice to the fourth submatrix updated in step (14-1) to obtain the final updated fourth submatrix;

[0173] The advantage of the above steps (12) to (14) is that the vector operation instructions of the DSP computing core are used in the calculation process, making full use of the SIMD (Single Instruction Multiple Data) computing characteristics of the DSP device.

[0174] (15) Each computing core in the DSP sets a counter k=k+1 and returns to step (11);

[0175] (16) Each computing core in the DSP sets counter i=i+1 and returns to step (7);

[0176] The advantage of the above steps (5) to (16) is that an online softmax optimization algorithm is used to perform operator fusion on batch matrix multiplication and softmax operation in the multi-head attention mechanism, thereby realizing data reuse and reducing the hardware access to DDR memory, thereby reducing the memory access overhead between different operators and improving the computational efficiency of the multi-head attention mechanism operators.

[0177] (17) Each computing core in the DSP sets a counter j=j+n_threads and returns to step (3);

[0178] (18) Each computing core in the DSP returns the fourth submatrix obtained in step (14) to the CPU as the calculation result of the DSP multi-head attention mechanism operator;

[0179] (19) The CPU passes the calculation results of the multi-head attention mechanism operator from each computing core in the DSP into the calculation parameters of the multi-head attention mechanism operator.

[0180] The advantage of the above steps (1) to (19) is that they combine the characteristics of the CPU and the DSP, making full use of the general computing power of the CPU and the advantages of the DSP in parallel computing.

[0181] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An optimization method for multi-head attention mechanism operators for domestic heterogeneous processors, characterized in that: The following steps are involved: (1) The CPU obtains the calculation parameters of the multi-head attention mechanism operator, allocates DSP storage space according to the calculation parameters of the multi-head attention mechanism operator, initializes the DSP execution thread to obtain the thread number of the DSP execution thread, inputs the calculation parameters of the multi-head attention mechanism operator into the DSP thread execution function according to the thread number of the DSP execution thread and runs it to start all the computing cores in the DSP. (2) Each computing core in the DSP obtains the total number of computing tasks of all computing cores in the DSP according to the computing parameters of the multi-head attention mechanism operator, and obtains its own computing tasks to be executed based on the obtained total number of computing tasks and the total number of computing cores in the DSP. (3) Each computing core in the DSP sets counter j = 0; (4) Each computing core in the DSP determines whether j is less than the total number of computing tasks cal_num of all computing cores in the DSP obtained in step (2). If so, it proceeds to step (5); otherwise, it proceeds to step (16); (5) Each computing core in the DSP obtains the query matrix pointer my_queries, key-value matrix pointer my_keys, value matrix pointer my_values, and output matrix pointer my_o corresponding to the computing task to be executed by the computing core obtained in step (2) according to the computing parameters of the multi-head attention mechanism operator; obtains the key-value matrix according to the key-value matrix pointer, and transposes the key-value matrix to obtain the transposed matrix of the key-value matrix, respectively obtains the query matrix and the value matrix (the size of which is SEQ_LEN rows and HEAD_SIZE columns) according to the query matrix pointer and the value matrix pointer, sets the block parameter BLOCK_SEQ_LEN according to the type of the input matrix pointer, and blocks the query matrix and the value matrix according to the block parameter to obtain multiple first sub-matrices and multiple second sub-matrices respectively, and obtains the total number of blocks corresponding to the computing task to be executed by the computing core according to the block parameter. (6) Each computing core in the DSP sets counter i=0; (7) Each computing core in the DSP determines whether i is less than the total number of blocks sr. If yes, it proceeds to step (9); otherwise, it proceeds to step (16); (8) Each computing core in the DSP divides the transposed matrix of the key value matrix into a third submatrix of multiple HEAD_SIZE rows and BLOCK_SEQ_LEN columns according to the block parameters obtained in step (5), obtains a fourth submatrix of size BLOCK_SEQ_LEN rows and HEAD_SIZE columns according to the pointer of the output matrix, and initializes all elements in the fourth submatrix to 0. (9) Each computing core in the DSP sets counter k=0; (10) Each computing core in the DSP creates and initializes four arrays of length BLOCK_SEQ_LEN, Gmax and Gmax_temp to -inf, and GL and GL_temp to 0. (11) Each computing core in the DSP determines whether k is less than or equal to i. If so, it proceeds to step (12); otherwise, it proceeds to step (16); (12) Each computing core in the DSP performs a matrix multiplication operation on the i-th first submatrix obtained in step (8) and the k-th third submatrix obtained in step (10), and the result obtained is the fifth submatrix: (13) Each computing core in the DSP determines whether k is equal to i. If so, a mask matrix is ​​generated, which is a matrix with a size of BLOCK_SEQ_LEN rows and BLOCK_SEQ_LEN columns, with all upper triangular values ​​being -inf and lower triangular values ​​being 0, and a matrix addition operation is performed on the fifth submatrix obtained in step (12) and the mask matrix to obtain the fifth submatrix after the mask operation, and then the process goes to step (14); otherwise, the process goes to step (14); (14) Each computing core in the DSP performs softmax calculation on the fifth submatrix after the mask operation obtained in step (13) to obtain a sixth submatrix, updates the fourth submatrix using the intermediate result in the softmax calculation process to obtain an updated fourth submatrix, calculates the matrix multiplication result between the sixth submatrix and the second submatrix as a slice O_new of the fourth submatrix, and adds the obtained slice O_new to the updated fourth submatrix to obtain the final updated fourth submatrix. (15) Each computing core in the DSP sets a counter k=k+1 and returns to step (11); (16) Each computing core in the DSP sets counter i=i+1 and returns to step (7); (17) Each computing core in the DSP sets a counter j=j+n_threads and returns to step (3); (18) Each computing core in the DSP returns the fourth submatrix obtained in step (14) to the CPU as the calculation result of the DSP multi-head attention mechanism operator; (19) The CPU passes the calculation results of the multi-head attention mechanism operator from each computing core in the DSP into the calculation parameters of the multi-head attention mechanism operator.

2. The optimization method for multi-head attention mechanism operators for domestic heterogeneous processors according to claim 1 is characterized in that: The calculation parameters of the multi-head attention mechanism operator in step (1) include: the function name of the multi-head attention mechanism operator, the number of input data batches batch_size, the number of heads n_heads, the length of the input sequence SEQ_LEN, the size of the head HEAD_SIZE, the pointer of the input matrix (which includes the query matrix pointer queries, the key matrix pointer keys, the value matrix pointer value s), and the pointer o of the output matrix.

3. The optimization method for the multi-head attention mechanism operator for domestic heterogeneous processors according to claim 1 or 2 is characterized in that: The specific process of allocating DSP storage space according to the calculation parameters of the multi-head attention mechanism operator in step (1) is as follows: first, according to the number of input data batches, the number of heads, the length of the input sequence, and the size of the head in the calculation parameters of the multi-head attention mechanism operator, the size of the input matrix and the output matrix is ​​obtained. The calculation formula is: size=batch_size*SEQ_LEN*n_heads*HEAD_SIZE Then, according to the calculated sizes of the input matrix and output matrix, the hthread_mall oc interface is called to allocate the storage space of the DSP input matrix and output matrix. Then, the input matrix and output matrix are transferred to the storage space of the DSP input matrix and output matrix respectively. Finally, the pointers of the input matrix and output matrix are pointed to the storage space of the input matrix and output matrix respectively. In step (1), the CPU initializes the DSP execution thread to obtain the thread number of the DSP execution thread. Specifically, the process includes obtaining the DSP cluster number and the number of DSP cores started, n_threads, initializing the DSP execution thread according to the obtained DSP cluster number and the number of DSP cores started, and calling the hthread_create_group function to obtain the thread number of the DSP execution thread. In step (1), the multi-head attention mechanism operator calculation parameters are input into the DSP thread execution function according to the thread number of the DSP execution thread and run to start each computing core in the DSP. Specifically, this process is to create an empty parameter array, store the obtained calculation parameters of the multi-head attention mechanism operator and the pointers of the input matrix and the output matrix into the parameter array, and then call the hthread_group_exec function to input the obtained DSP execution thread number and parameter array into the DSP thread execution function in turn to start each computing core in the DSP.

4. The optimization method for multi-head attention mechanism operators for domestic heterogeneous processors according to any one of claims 1 to 3, characterized in that: In step (2), each computing core in the DSP calculates the number of batches of input data. b atc h _size and the number of heads n_ h ea d s To determine the total number of computing tasks cal_num for all computing cores in the DSP, specifically: cal_num=batch_size×n_heads.

5. The optimization method for multi-head attention mechanism operators for domestic heterogeneous processors according to claim 4 is characterized in that: In step (5), each computing core in the DSP obtains the query matrix pointer my_queries, key value matrix pointer my_keys, value matrix pointer my_values, and output matrix pointer my_o corresponding to the current computing task to be executed according to the computing parameters of the multi-head attention mechanism operator. This process adopts the following calculation formula: my_queries = queries + offset my_keys = keys + offset my_values ​​= values ​​+ offset my_o=o+offset Among them are: offset=batch_idx×SEQ_LEN×n_heads×HEAD_SIZE+head_idx×HEAD_SIZE And there are: batch_idx=j / n_heads head_idx=j%n_heads In step (5), each computing core in the DSP obtains the query matrix and the value matrix respectively according to the query matrix pointer and the value matrix, sets the block parameter BLOCK_SEQ_LEN according to the type of the pointer of the input matrix, and blocks the query matrix and the value matrix according to the block parameter to obtain multiple first sub-matrices and second sub-matrices, and obtains the total number of blocks corresponding to the computing task to be executed by the computing core according to the block parameter. This process is specifically as follows: for each computing core in the DSP, the pointer type of the input matrix is ​​single precision or half precision, and its block parameter is set to 96 or 192 respectively, the query matrix is ​​divided into multiple first sub-matrices of BLOCK_SEQ_LEN rows and HEAD_SIZE columns, and the total number of blocks sr corresponding to the computing task to be executed by the computing core in the DSP is obtained: sr = SEQ_LEN / BLOCK_SEQ_LEN.

6. The optimization method for multi-head attention mechanism operators for domestic heterogeneous processors according to claim 5 is characterized in that: Step (12) includes the following sub-steps: (12-1) Each computing core in the DSP performs block processing on the first submatrix obtained in step (8) to obtain CntQ (where CntQ = BLOCK_SEQ_LEN\Q_SM_BLOCK_RO W) small blocks with Q_SM_BLOCK_ROW rows and HEAD_SZIE columns, where Q_SM_BLOCK_ROW> the instruction delay of the DSP multiply-add instruction and is close to the power of two of the instruction delay of the DSP multiply-add instruction; (12-2) Each computing core in the DSP sets a counter q=0; (12-3) Each computing core in the DSP starts the DMA transfer function twice to transfer the CntQ small blocks obtained in step (12-1) to the SM in sequence to obtain the first sub-matrix small block in the SM space. (12-4) Each computing core in the DSP determines whether q is less than CntQ. If so, it calculates the matrix multiplication between the first submatrix block obtained in step (12-3) and the third submatrix to obtain the fifth submatrix of Q_SM_BLOCK_ROW rows and BLOCK_SEQ_LEN columns, and then proceeds to step (12-5), otherwise the process ends. (12-5) Each computing core in the DSP sets a counter q=q+1 and returns to step (12-4).

7. The optimization method for multi-head attention mechanism operators for domestic heterogeneous processors according to claim 6 is characterized in that: Step (14) specifically includes the following sub-steps: (14-1) Each computing core in the DSP obtains the maximum value max_now of each row in the fifth submatrix obtained in step (13), and the index sum L_NOW of all rows in the fifth submatrix, and obtains the sixth submatrix according to the maximum value max_now and the index sum L_NOW, and uses the sixth submatrix to update the fourth submatrix to obtain an updated fourth submatrix; (14-2) Each computing core in the DSP performs a matrix multiplication operation on the sixth submatrix and the second submatrix obtained in step (8), and the operation result is used as a slice O_new of the fourth submatrix, and the obtained slice O_new is added to the updated fourth submatrix to obtain the final updated fourth submatrix.

8. The optimization method for multi-head attention mechanism operators for domestic heterogeneous processors according to claim 7 is characterized in that: Step (14-1) specifically includes the following sub-steps: (14-1-1) Each computing core in the DSP sets the counter oi=0; (14-1-2) Each computing core in the DSP determines whether oi is less than the block parameter BLOCK_SEQ_LEN. If so, it goes to step (14-1-3), otherwise the process ends; (14-1-3) Each computing core in the DSP calculates the maximum value max_temp of the oi-th row in the fifth submatrix, and assigns the larger value of max_temp and GMAX[oi] to GMAX_temp[oi], where GMAX[0] = 0; (14-1-4) Each computing core in the DSP performs exponential operations on all elements of the oi-th row of the fifth submatrix and then sums them up, and assigns the sum result to GL_temp[oi]. (14-1-5) Each computing core in the DSP updates the oi-th row in the fourth submatrix to G_L[oi] = G_L[oi]*exp(G_MAX[oi]-G__temp[oi]) / (GL_temp[oi]*GL[oi])+GL_temp[oi] to obtain the updated fourth submatrix, performs exponential operation on each element in the oi-th row of the fourth submatrix, divides all the exponential operation results by GMAX_temp[oi], and uses the results as the oi-th row of the sixth submatrix, and assigns GMAX_temp[oi] to GMAX[oi]. (14-1-6) Each computing core in the DSP sets the counter oi=oi+1 and returns to step (14-1-2).

9. The optimization method for multi-head attention mechanism operators for domestic heterogeneous processors according to claim 8, characterized in that: Step (14-2) specifically includes the following steps: (14-2-1) Each computing core in the DSP performs block processing on the second sub-matrix according to the block parameter BLOCK_SEQ_LEN to obtain a second sub-matrix block of nv BLOCK_SEQ_LEN rows and BLOCK_SEQ_LEN columns, where nv = HEAD_SIZE / BLOCK_SEQ_LEN. (14-2-2) Each computing core in the DSP sets counter ii=0; (14-2-3) Each computing core in the DSP determines whether ii is less than nv. If so, it obtains the ii-th second submatrix block obtained in step (14-2-1) and then proceeds to step (14-2-4). Otherwise, the process ends. (14-2-4) Each computing core in the DSP performs block processing on the sixth submatrix obtained in step (14-1) according to the block parameter Q_SM_BLOCK_ROW to obtain a sixth submatrix block of np Q_SM_BLOCK_ROW rows and BLOCK_SEQ_LEN columns, where np = BLOCK_SEQ_LEN / Q_SM_BLOCK_ROW. (14-2-5) Each computing core in DSP sets counter jj=0; (14-2-6) Each computing core in the DSP determines whether jj is less than np. If so, it performs a matrix multiplication operation on the jjth sixth submatrix block obtained in step (14-2-4) and the second submatrix block obtained in step (14-2-3). The operation result is used as the sliced ​​block, and then goes to step (14-2-7). Otherwise, it goes to step (14-2-8); (14-2-7) Each computing core in the DSP sets counter jj=jj+1 and returns to step (14-2-5); (14-2-8) Each computing core in the DSP sets counter ii=ii+1 and returns to step (14-2-2); (14-2-9) Each computing core in the DSP splices the small slice blocks obtained in step (14-2-6) to obtain a slice O_new, and adds the slice to the fourth submatrix updated in step (14-1) to obtain the final updated fourth submatrix.

10. An optimization system for multi-head attention mechanism operators for domestic heterogeneous processors, characterized in that: include: The first module is set in the CPU and is used to obtain the calculation parameters of the multi-head attention mechanism operator, allocate DSP storage space according to the calculation parameters of the multi-head attention mechanism operator, initialize the DSP execution thread to obtain the thread number of the DSP execution thread, and input the calculation parameters of the multi-head attention mechanism operator into the DSP thread execution function according to the thread number of the DSP execution thread and run it to start all computing cores in the DSP. The second module is set in each computing core in the DSP, and is used to obtain the total number of computing tasks of all computing cores in the DSP according to the computing parameters of the multi-head attention mechanism operator, and obtain its own computing tasks to be executed based on the obtained total number of computing tasks and the total number of computing cores in the DSP. The third module is arranged in each computing core in the DSP and is used to set the counter j=0; The fourth module is set in each computing core in the DSP, and is used to determine whether j is less than the total computing task number cal_num of all computing cores in the DSP obtained by the second module. If yes, it is transferred to the fifth module, otherwise it is transferred to the sixteenth module; The fifth module is set in each computing core in the DSP, and is used to obtain the query matrix pointer my_queries, key value matrix pointer my_keys, value matrix pointer my_values, and output matrix pointer my_o corresponding to the computing task to be executed by the computing core obtained in the second module according to the computing parameters of the multi-head attention mechanism operator; obtain the key value matrix according to the key value matrix pointer, and transpose the key value matrix to obtain the transposed matrix of the key value matrix, respectively obtain the query matrix and the value matrix (the size of which is SEQ_LEN rows and HEAD_SI ZE columns) according to the query matrix pointer and the value matrix pointer, set the block parameter BLOCK_SEQ_LEN according to the type of the input matrix pointer, and block the query matrix and the value matrix according to the block parameter to obtain multiple first sub-matrices and multiple second sub-matrices respectively, and obtain the total number of blocks corresponding to the computing task to be executed by the computing core according to the block parameter. A sixth module, which is arranged in each computing core in the DSP, is used to set a counter i=0; The seventh module is set in each computing core in the DSP, and is used to determine whether i is less than the total number of blocks sr. If yes, it goes to the ninth module, otherwise it goes to the sixteenth module; The eighth module is set in each computing core in the DSP, and is used to divide the transposed matrix of the key value matrix into a third sub-matrix with multiple HEAD_SIZE rows and BLOCK_SEQ_LEN columns according to the block parameters obtained in the fifth module, obtain a fourth sub-matrix with a size of BLOCK_SEQ_LEN rows and HEAD_SIZE columns according to the pointer of the output matrix, and initialize all elements in the fourth sub-matrix to 0. A ninth module, which is arranged in each computing core in the DSP, and is used to set a counter k=0; The tenth module is set in each computing core in the DSP, and is used to create and initialize four arrays Gmax and Gmax_temp with a length of BLOCK_SEQ_LEN to -inf, and GL and GL_temp to 0. The eleventh module is set in each computing core in the DSP, and is used to determine whether k is less than or equal to i. If so, it is transferred to the twelfth module, otherwise it is transferred to the sixteenth module; The twelfth module is set in each computing core in the DSP, and is used to perform a matrix multiplication operation on the i-th first sub-matrix obtained by the eighth module and the k-th third sub-matrix obtained by the tenth module, and the result obtained is the fifth sub-matrix: The thirteenth module is set in each computing core in the DSP, and is used to determine whether k is equal to i. If so, a mask matrix is ​​generated, which is a matrix with a size of BLOCK_SEQ_LEN rows and BLOCK_SEQ_LEN columns, with all upper triangular values ​​being -inf and lower triangular values ​​being 0, and a matrix addition operation is performed on the fifth submatrix obtained in step (12) and the mask matrix to obtain the fifth submatrix after the mask operation, and then the process is transferred to the fourteenth module; otherwise, the process is transferred to the fourteenth module; The fourteenth module is arranged in each computing core in the DSP, and is used to perform softmax calculation on the fifth submatrix after the mask operation obtained by the thirteenth module to obtain the sixth submatrix, update the fourth submatrix using the intermediate result in the softmax calculation process to obtain the updated fourth submatrix, calculate the matrix multiplication result between the sixth submatrix and the second submatrix as the slice O_new of the fourth submatrix, and add the obtained slice O_new to the updated fourth submatrix to obtain the final updated fourth submatrix. A fifteenth module is provided in each computing core in the DSP, and is used to set a counter k=k+1, and return to the eleventh module; A sixteenth module, which is provided in each computing core in the DSP, is used to set a counter i=i+1 and return to the seventh module; A seventeenth module is provided in each computing core in the DSP, and is used to set a counter j=j+n_threads, and return to the third module; An eighteenth module is provided in each computing core in the DSP, and is used to return the fourth sub-matrix obtained in the fourteenth module to the CPU as the calculation result of the DSP multi-head attention mechanism operator; The nineteenth module is set in the CPU and is used to transfer the calculation results of the multi-head attention mechanism operator from each computing core in the DSP into the calculation parameters of the multi-head attention mechanism operator.

Images