Acceleration method for batch matrix multiplication running on computing platform

Abstract

The present application provides an acceleration method for batch matrix multiplication running in a computing platform. The method includes: dividing a batch result matrix into a maximum number of task blocks, wherein the batch result matrix includes multiple result matrices, and any result matrix is ​​a matrix obtained by multiplying two matrices; determining a balanced partitioning strategy based on the maximum number of task blocks, wherein the balanced partitioning strategy is a strategy for evenly distributing multiple task blocks to multiple computing cores of a neural network processing unit NPU in the computing platform; calling the computing unit in the computing core according to the balanced partitioning strategy, performing matrix multiplication operations on all task blocks, and obtaining the calculation results of the batch result matrix. This method solves the problem of low computational efficiency of the batch matrix multiplication calculation method in the prior art.

Description

Technical Field

[0001] The present application relates to the field of high-performance computing, and in particular, to an acceleration method for batch matrix multiplication running on a computing platform, an acceleration device for batch matrix multiplication, and a computer-readable storage medium. Background Art

[0002] As the number of parameters of various artificial intelligence models continues to increase, the required computing power also increases. Batch matrix multiplication can significantly improve computing efficiency through hardware features and algorithm support.

[0003] In the prior art, there is no implementation of batch matrix multiplication with good performance on the computing platform. Generally, a single matrix multiplication is called cyclically to complete the batch matrix multiplication operation. In this way, the idle rate of the computing cores in the NPU is high, the parallelism of the calculation is not high, the computing power provided by the hardware facilities is not well utilized, and the computational efficiency of the batch matrix multiplication calculation method is not high. Summary of the invention

[0004] The main purpose of the present application is to provide an acceleration method for batch matrix multiplication running in a computing platform, an acceleration device for batch matrix multiplication and a computer-readable storage medium, so as to at least solve the problem of low computational efficiency of the batch matrix multiplication calculation method in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, there is provided an acceleration method for batch matrix multiplication running in a computing platform, comprising: dividing a batch result matrix into a maximum number of task blocks, wherein the batch result matrix includes multiple result matrices, and any of the result matrices is a matrix obtained by multiplying two matrices; determining a balanced partitioning strategy according to the maximum number of task blocks, wherein the balanced partitioning strategy is a strategy for evenly distributing the multiple task blocks to the multiple computing cores of the neural network processing unit NPU in the computing platform; calling the computing unit in the computing core according to the balanced partitioning strategy, performing matrix multiplication operations on all the task blocks, and obtaining the calculation results of the batch result matrix.

[0006] Optionally, the batch result matrix is ​​divided into a maximum number of task blocks, including: determining the size of a single task block based on the memory space of a unified buffer UB, where the unified buffer UB is a unified buffer in the NPU of the computing platform; determining the maximum total number of task blocks of the batch result matrix based on the largest MNK in the batch result matrix and the size of the single task block, where the MNK is the dimension of rows, columns, and inner products in the corresponding matrix multiplication.

[0007] Optionally, determining the size of a single task block according to the memory space of the unified buffer includes: assuming that the size of the single task block is m×m, and the calculation formula of m is: , where K is the memory space size of the unified buffer.

[0008] Optionally, determining the maximum total number of task blocks of the batch result matrix according to the largest MNK in the batch result matrix and the size of the single task block includes: performing matrix multiplication on matrix A and matrix B, where matrix A and matrix B are two result matrices in the batch result matrix; passing the M value, K value of matrix A and the K value, N value of matrix B to the program in the form of an array, assuming that the total number of the batch result matrix is ​​batchCount, the batch is batchID, and the length and width of the matrix of the batch are MbatchID, KbatchID and NbatchID, wherein, ; Traverse the three arrays of M, N, and K to obtain the largest M, N, and K values ​​in the three arrays, Mmax, Kmax, and Nmax; according to Mmax, Kmax, Nmax, and the size of the single task block, divide the single result matrix into multiple task blocks, and the maximum number of task blocks for a single result matrix ; According to the maximum number of task blocks of a single result matrix, the maximum total number of task blocks of the batch result matrix is ​​determined to be .

[0009] Optionally, the method further includes: dividing each of the result matrices according to the maximum number of task blocks of a single result matrix; wherein dividing each of the result matrices according to the maximum number of task blocks of a single result matrix includes: defining the serial number of the target task block as ID1, the batch in which the target task block is located as batchID1, and the serial number of the target task block in the batch as tilesID1, wherein, , , , the calculation formula for batchID1 and tilesID1 is , .

[0010] Optionally, the balanced partitioning strategy includes: a first allocation step, allocating a maximum number of the plurality of task blocks according to the computing core sequence number coreID of the NPU starting from number 0, and sequentially allocating them to each computing core of the NPU, and the computing task amount of each computing core is , where task(i) represents the number of task blocks assigned to the computing core with id i, and c is the number of computing cores. is the serial number of the computing core; a second allocation step, when there are still unallocated task blocks after the last computing core is allocated, the unallocated task blocks are re-allocated starting from the computing core with coreID 0, and are allocated to each computing core of the NPU in sequence; the first allocation step and the second allocation step are repeated at least once until all the task blocks are allocated; an optimization step, optimizing the multiplication calculation method of the batch result matrix.

[0011] Optionally, the optimization step includes: setting a judgment function: ,in, , , where ID is the serial number of the task block, batchID is the batch to which the task block belongs, tilesID is the serial number of the task block in the batch, MbatchID, NbatchID and KbatchID are the MNK of the matrix of the batch to which the task block belongs, and x and y are the number of rows and columns of the task block in the result matrix respectively; before the computing core executes the task block calculation, the judgment function is used to determine whether the task block is a valid task block; if the task block is a valid task block, the computing core executes the matrix multiplication operation of the task block; if the task block is not a valid task block, the computing core skips the calculation of the task block and directly performs the calculation of the next task block.

[0012] Optionally, the computing unit in the computing core is called according to the balanced partitioning strategy to perform matrix multiplication operations on all the task blocks to obtain the calculation results of the batch result matrix, including: an operation step, for each batch of the result matrix, a matrix multiplication operation is performed on a row of task blocks and a column of task blocks in any two of the result matrices to obtain multiple calculation results; an accumulation step, accumulating multiple calculation results to obtain the final result matrix of the task blocks in the result matrix of the corresponding batch; a judgment step, judging whether the calculations of all the task blocks in the result matrix are completed, and when the calculations of all the task blocks in the result matrix are completed, determining that the calculation of the result matrix is ​​completed; repeating the operation step, the accumulation step and the judgment step at least once until all the result matrices are calculated to obtain the calculation results of the batch result matrix.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, there is provided an acceleration device for batch matrix multiplication, comprising: a segmentation unit, used to segment a batch result matrix into a maximum number of task blocks, wherein the batch result matrix comprises multiple result matrices, and any of the result matrices is a matrix obtained by multiplying two matrices; a determination unit, used to determine a balanced partitioning strategy according to the maximum number of task blocks, wherein the balanced partitioning strategy is a strategy of evenly distributing the multiple task blocks to the multiple computing cores of the neural network processing unit NPU in the computing platform; a calling unit, used to call the computing unit in the computing core according to the balanced partitioning strategy, perform matrix multiplication operations on all the task blocks, and obtain the calculation results of the batch result matrix.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the batch matrix multiplication acceleration methods.

[0015] Applying the technical solution of the present application, the batch result matrix is ​​divided into the maximum number of task blocks, wherein the batch result matrix includes multiple result matrices, and any result matrix is ​​a matrix obtained by multiplying two matrices; a balanced partitioning strategy is determined according to the maximum number of task blocks, wherein the balanced partitioning strategy is a strategy of evenly distributing multiple task blocks to multiple computing cores of the neural network processing unit NPU in the computing platform; the computing units in the computing core are called according to the balanced partitioning strategy to perform matrix multiplication operations on all task blocks to obtain the calculation results of the batch result matrix. In this solution, by proposing a preset balanced partitioning strategy based on the maximum number of task blocks, the computing units in the computing core are called to complete batch matrix multiplication, which accelerates the calculation of batch matrix multiplication, thereby solving the problem of low calculation efficiency of batch matrix multiplication calculation methods in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A schematic flow chart of an acceleration method for batch matrix multiplication running in a computing platform provided according to an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of task block division of a single result matrix of an acceleration method for batch matrix multiplication running in a computing platform provided in an embodiment of the present application is shown;

[0019] Figure 3The following is a diagram showing the task block allocation and logic task block diagram of an acceleration method for batch matrix multiplication running on a computing platform according to an embodiment of the present application. Figure 1 ;

[0020] Figure 4 The following is a diagram showing the task block allocation and logic task block diagram of an acceleration method for batch matrix multiplication running on a computing platform according to an embodiment of the present application. Figure 2 ;

[0021] Figure 5 An overall flow chart of an acceleration method for batch matrix multiplication running in a computing platform provided according to an embodiment of the present application is shown;

[0022] Figure 6 A structural block diagram of an acceleration device for batch matrix multiplication provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] As introduced in the background technology, the prior art generally adopts a loop calling a single matrix multiplication to complete the batch matrix multiplication operation. In this way, the idle rate of the computing cores in the NPU is relatively high, the parallelism of the calculation is not high, and the computing power provided by the hardware facilities is not well utilized. In order to solve the problem of low computational efficiency of the batch matrix multiplication calculation method, the embodiments of the present application provide a batch matrix multiplication acceleration method running in a computing platform, a batch matrix multiplication acceleration device and a computer-readable storage medium.

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] Figure 1 FIG. 1 is a flow chart of a method for accelerating batch matrix multiplication running on a computing platform according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0029] Step S101, dividing a batch result matrix into a maximum number of task blocks, wherein the batch result matrix includes a plurality of result matrices, and any of the result matrices is a matrix obtained by multiplying two matrices;

[0030] Specifically, the batch result matrix is ​​divided into the maximum number of task blocks. The batch result matrix includes multiple result matrices. Each result matrix is ​​obtained by multiplying two matrices. These result matrices are divided into as many small blocks as possible for parallel processing to improve computing efficiency. The purpose of dividing the task blocks is to allocate them to multiple computing nodes to speed up the computing speed. This can effectively utilize the advantages of parallel computing and improve the overall computing efficiency.

[0031] Step S102, determining a balanced partitioning strategy according to the maximum number of task blocks, wherein the balanced partitioning strategy is a strategy of evenly distributing the plurality of task blocks to the plurality of computing cores of the neural network processing unit NPU in the computing platform;

[0032] Specifically, the task blocks are evenly distributed to multiple computing cores of the neural network processing unit NPU in the computing platform, which ensures that each computing core can be fully utilized to avoid the situation where some computing cores are overloaded and the performance is degraded. Through the balanced partitioning strategy, it is possible to avoid the situation where some computing cores are overloaded while other computing cores are idle, so that each computing core can efficiently participate in the calculation of batch matrix multiplication, thereby maximizing the overall performance of the computing platform.

[0033] Step S103, calling the computing unit in the computing core according to the balanced partitioning strategy, performing matrix multiplication operations on all the task blocks, and obtaining the calculation results of the batch result matrix.

[0034] Specifically, a balanced partitioning strategy is used to allocate computing tasks and call computing units to perform matrix multiplication operations to obtain the calculation results of batch result matrices. This strategy can ensure that computing tasks are evenly distributed among various computing units, fully utilize computing resources, and improve computing efficiency. In this way, large-scale matrix multiplication operations can be effectively completed and accurate results can be obtained. The purpose of the balanced partitioning strategy is to ensure that each computing core in the NPU can evenly distribute tasks when performing batch matrix multiplication, avoiding the situation where some computing cores are overloaded while other computing cores are idle.

[0035] In the above-mentioned embodiment, the batch result matrix is ​​divided into the maximum number of task blocks, wherein the above-mentioned batch result matrix includes multiple result matrices, and any of the above-mentioned result matrices is a matrix obtained by multiplying two matrices; a balanced partitioning strategy is determined according to the above-mentioned maximum number of task blocks, wherein the above-mentioned balanced partitioning strategy is a strategy of evenly distributing the multiple above-mentioned task blocks to the multiple computing cores of the neural network processing unit NPU in the above-mentioned computing platform; according to the above-mentioned balanced partitioning strategy, the computing unit in the above-mentioned computing core is called to perform matrix multiplication operations of all the above-mentioned task blocks to obtain the calculation results of the above-mentioned batch result matrix. In this scheme, by proposing a preset balanced partitioning strategy based on the maximum number of task blocks, the computing unit in the computing core is called to complete the batch matrix multiplication, which accelerates the calculation of the batch matrix multiplication, thereby solving the problem of low calculation efficiency of the batch matrix multiplication calculation method in the prior art.

[0036] In one embodiment of the present application, a batch result matrix is ​​divided into a maximum number of task blocks, including: determining the size of a single task block according to the memory space of a unified buffer UB, where the unified buffer UB is a unified buffer in an NPU of the computing platform; determining the maximum total number of task blocks of the batch result matrix according to the largest MNK in the batch result matrix and the size of the single task block, where the MNK corresponds to the dimensions of rows, columns, and inner products in matrix multiplication.

[0037] Specifically, in the NPU of the computing platform, UB is an internal storage resource used to temporarily store data so that the computing core can perform efficient computing. The memory space of UB is limited, so the size of the task block needs to be determined according to the capacity of UB to ensure that the data of each task block can be fully loaded into UB, thereby improving computing efficiency. In order to determine the size of a single task block, it is necessary to ensure that the memory space of UB can accommodate the data required by the task block.

[0038] Specifically, MNK is used to describe the size of the matrix. M (Matrix Rows) represents the number of rows of the matrix, N (Matrix Columns) represents the number of columns of the matrix, and K (Matrix Inner Dimension) represents the internal dimension when two matrices are multiplied, that is, the number of columns of the first matrix and the number of rows of the second matrix. In batch matrix multiplication, MNK usually represents the matrix dimension in each batch. For example, for two matrices A and B, if the dimension of matrix A is M×K and the dimension of matrix B is K×N, then the dimension of the result of their multiplication, matrix C, is M×N. Batch matrix multiplication involves multiple such matrix multiplication operations. The matrix dimensions in each batch may be different, but they can all be described by MNK.

[0039] In a specific embodiment, the size of a single task block is determined according to the memory space of the unified buffer, including: assuming that the size of the single task block is m×m, and the calculation formula of m is: , where K is the memory space size of the unified buffer.

[0040] Specifically, in order to complete the calculation of a single task block, the UB memory space must be able to store two small matrices of size , where the data type of each element is single precision of 4 bytes. The occupied memory size is 2×m×m×4 bytes. In order to meet the hardware architecture requirements of the NPU of the computing platform, integers that are multiples of 32 bytes are rounded down. Thus, the calculation formula for m can be obtained.

[0041] In one embodiment of the present application, the maximum value of the total number of task blocks of the batch result matrix is ​​determined based on the largest MNK in the batch result matrix and the size of the single task block, including: performing matrix multiplication on matrix A and matrix B, where the matrix A and the matrix B are two of the result matrices in the batch result matrix; passing the M value, K value of the matrix A and the K value, N value of the matrix B to the program in the form of an array, assuming that the total number of the batch result matrices is batchCount, the batch is batchID, and the length and width of the matrix of the batch are MbatchID, KbatchID and NbatchID, wherein, ; Traverse the three arrays of M, N, and K to obtain the largest M, N, and K values ​​in the three arrays, which are Mmax, Kmax, and Nmax; according to Mmax, Kmax, Nmax, and the size of the single task block, divide the single result matrix into multiple task blocks, and the maximum number of task blocks for the single result matrix is ,like Figure 2 As shown, Figure 2Schematic diagram of task block division of a single result matrix. The dimension of matrix A is N×M, the dimension of matrix B is M×N, and the dimension of matrix C, the result matrix of the multiplication of matrix A and matrix B, is N×N. The maximum number of task blocks of a single result matrix C can be obtained; according to the maximum number of task blocks of a single result matrix, the maximum value of the total number of task blocks of the batch result matrix is ​​determined as .

[0042] Specifically, batch matrix multiplication is to calculate multiple matrices simultaneously to obtain multiple result matrices. For example, matrix A and matrix B are two input matrices in the batch result matrix. The dimension of matrix A is M×K, and the dimension of matrix B is K×N. For batch matrix multiplication, these values ​​can be different in matrices of different batches. In batch matrix multiplication, the dimension information M, K, and N of matrix A and matrix B are usually passed into the program in the form of arrays. These arrays represent the dimension information of each batch matrix. For example, for batchCount matrices, there will be M arrays, K arrays, and N arrays, and the length of each array is batchCount. Traverse the M arrays, K arrays, and N arrays to find the largest M value, N value, and K value, which are recorded as Mmax, Kmax, and Nmax respectively. These maximum values ​​will be used for subsequent task block division. According to Mmax, Kmax, and Nmax and the size of a single task block, the maximum number of task blocks for a single result matrix is ​​calculated.

[0043] In a specific embodiment, the method further includes: dividing each of the above result matrices according to the maximum number of task blocks of a single result matrix; wherein, dividing each of the above result matrices according to the maximum number of task blocks of a single result matrix includes: defining the serial number of the target task block as ID1, the batch in which the target task block is located as batchID1, and the serial number of the target task block in the batch as tilesID1, wherein, , , , the calculation formula for batchID1 and tilesID1 is , .

[0044] Specifically, assuming that there is a task block with a serial number of ID, to find out which batch the task block corresponding to this serial number is in and its specific position in the batch, it involves the organization and indexing method of task blocks in batch matrix multiplication. Each result matrix will be divided into several task blocks, and the serial numbers of these task blocks are continuously assigned according to batches. Each result matrix is ​​divided according to the maximum number of task blocks in a single result matrix. According to the serial number of a task block, the batch to which this task block belongs and the serial number of this task block in the batch can be directly determined. Assuming that the serial number of a task block is ID, the batch to which this task block belongs is batchID, and the serial number of this task block in the batch is tilesID, where, , , , we can get the relative position of the task block in the batch to which it belongs, because the number of task blocks in each batch is fixed. This division and indexing method avoids complex judgment and calculation of each task block at runtime, and can quickly locate its specific position in the batch calculation process directly through the task block serial number, which not only simplifies the indexing process, but also saves storage space and computing resources, thereby improving the computational efficiency of batch matrix multiplication on the NPU. In addition, since the task block is designed to be a size that can be efficiently processed by the NPU hardware, this strategy can also make full use of the parallel computing capabilities of the hardware, reduce the idle time of the computing core, and further improve the parallelism and overall performance of the calculation.

[0045] In one embodiment of the present application, the above-mentioned balanced division strategy includes: a first allocation step, allocating the maximum number of the above-mentioned task blocks according to the computing core sequence number coreID of the above-mentioned NPU starting from No. 0, and sequentially allocating them to each computing core of the above-mentioned NPU, and the computing task amount of each of the above-mentioned computing cores is , where task(i) represents the number of task blocks assigned to the computing core with id i, and c represents the number of computing cores. is the serial number of the computing core; in the second allocation step, when there are still unallocated task blocks after the last computing core is allocated, the unallocated task blocks are reallocated again starting from the computing core with coreID 0, and are allocated to each computing core of the NPU in sequence; the first allocation step and the second allocation step are repeated at least once until all the task blocks are allocated; and an optimization step is performed to optimize the multiplication calculation method of the batch result matrix.

[0046] Specifically, since the NPU of the computing platform reads data in column priority, in order to facilitate data reading, the numbering of task blocks is also performed in column priority. Figure 1 like Figure 3 As shown, the task block allocation and logical task block diagram Figure 2 like Figure 4 As shown, for the convenience of demonstration, assume that Mmax is M0, Nmax is N0, and divide according to the maximum number of task blocks of a single result matrix, such as Figure 3 As shown, the result matrix C0 numbered 0 is divided into a fixed number of task blocks, such as Figure 4 As shown, the result matrix C1 numbered 1 is also divided into a fixed number of task blocks. The task block numbers of the result matrix C0 and the result matrix C1 are continuous. Repeat the above steps until all the result matrices are divided. Among them, M0 and K0 are the dimensions of the matrix A in the first batch, K0 and N0 are the dimensions of the matrix B in the first batch, M1 and K1 are the dimensions of the matrix A in the second batch, and K1 and N1 are the dimensions of the matrix B in the second batch. Assume that the number of computing cores is c and the computing core serial number is , the computing task volume of each computing core is: , where task(i) indicates the number of task blocks assigned to the computing core with id i. The task blocks are assigned according to the order of the task blocks from small to large, and multiple task blocks are cyclically assigned to the computing cores of the NPU. The task blocks are assigned starting from number 0 according to the computing core order of the NPU, and are assigned to the computing cores of the NPU in sequence. If there are still unassigned task blocks when the last computing core is assigned, the remaining task blocks will be reassigned starting from computing core number 0, and will be assigned to the computing cores of the NPU in sequence. Repeat the above steps until all task blocks are assigned.

[0047] In one embodiment of the present application, the above optimization step includes: setting a judgment function: ,in, , , where ID is the serial number of the task block, batchID is the batch to which the task block belongs, tilesID is the serial number of the task block in the batch, MbatchID, NbatchID and KbatchID are the MNK of the matrix of the batch to which the task block belongs, and x and y are the number of rows and columns of the task block in the result matrix respectively; before the computing core executes the task block calculation, the judgment function is used to judge whether the task block is a valid task block; if the task block is a valid task block, the computing core executes the matrix multiplication operation of the task block; if the task block is not a valid task block, the computing core skips the calculation of the task block and directly performs the calculation of the next task block.

[0048] Specifically, in the above preset balanced partitioning strategy, some task blocks are logically divided, but there is actually no data available for calculation, such as Figure 4Task blocks 28 to 31 in the task block list are there to optimize the indexing process. By using the serial number of a task block, we can directly determine the batch to which the task block belongs and the serial number of the task block in the batch without performing additional judgments and calculations, thus saving storage space and computing resources. Therefore, when the computing core processes such logical task blocks, it sets a judgment function so that the computing core skips the calculation of the logical task block, thus reducing the amount of calculation on the cube computing unit and avoiding errors in the computing core. For a task block with a serial number of ID, we can calculate that the batch to which the task block belongs is batchID, and the serial number of the task block in the batch is tilesID, where , , . Through batchID, we can get the MNK of the matrix of the batch where this task block is located as MbatchID, NbatchID and KbatchID. Assuming that the number of rows and columns of the task block in the result matrix are x and y respectively, we can calculate x and y as: , , set the judgment function to: , based on the maximum number of task blocks, a balanced partitioning strategy is preset, and all the divided task blocks are first cyclically allocated to each computing core until all the task blocks are allocated. Each time the computing core calculates a task block, it first uses a judgment function to determine whether the task block needs to be calculated. If it is a valid task block, the task block is calculated; if it is a logically divided task block, it does not need to be calculated, and it is directly skipped to calculate the next task block. The present invention presets a balanced partitioning strategy based on the maximum number of task blocks, makes full use of the hardware architecture of the NPU, improves computing efficiency through parallelism, conceals data transfer time, and ensures load balancing of the computing core.

[0049] In one embodiment of the present application, the computing unit in the computing core is called according to the balanced partitioning strategy to perform matrix multiplication operations on all the task blocks to obtain the calculation results of the batch result matrix, including: an operation step, for each batch of the result matrix, a matrix multiplication operation is performed on a row of task blocks and a column of task blocks in any two of the result matrices to obtain multiple calculation results; an accumulation step, accumulating the multiple calculation results to obtain the final result matrix of the task blocks in the result matrix of the corresponding batch; a judgment step, judging whether the calculations of all the task blocks in the result matrix are completed, and determining that the calculation of the result matrix is ​​completed when the calculations of all the task blocks in the result matrix are completed; repeating the operation step, the accumulation step and the judgment step at least once until all the result matrices are calculated to obtain the calculation results of the batch result matrix.

[0050] Specifically, according to the balanced partitioning strategy, the computing unit is called to perform matrix multiplication of the task block, and the calculation of all task blocks is completed to finally obtain the batch result matrix. The matrix multiplication operation of the task block is similar to the general matrix multiplication. A task block in the result matrix CbatchID of a batch with batchID needs to perform matrix multiplication on a row of task blocks of the matrix AbatchID and a column of task blocks of the matrix BbatchID in sequence, and add the results together. In this way, the calculation of this task block in the result matrix CbatchID is completed. When all task blocks in the result matrix CbatchID are calculated, the result matrix CbatchID is calculated. When all result matrices CbatchID are calculated, the batch result matrix of batch matrix multiplication is obtained. ,in For a task block in the result matrix with batch ID , in matrix A Perform matrix multiplication of the task block with the i-th block from left to right in the row and the i-th block from top to bottom in the y-th column of the general matrix B to obtain the intermediate result , where x and y are the number of rows and columns of a task block in the result matrix C, respectively, and , , ; Then, all the intermediate results are added together by matrix addition to get the final result of the task block ; Find all task blocks , you can get the complete result matrix ; Find all , you can get the batch result matrix of batch matrix multiplication Therefore, find all task blocks , you can get the calculation result of batch matrix multiplication.

[0051] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the acceleration method of batch matrix multiplication running in the computing platform of the present application will be described in detail below in combination with specific embodiments.

[0052] This embodiment relates to a specific method for accelerating batch matrix multiplication running on a computing platform. The overall flow chart is as follows: Figure 5 As shown, the following steps are included:

[0053] Step S1: Determine the size of a single task block according to the memory space size of the unified buffer UB.

[0054] Step S2: Traverse the dimensions MNK of all matrices and find the largest MNK.

[0055] Step S3: Determine the maximum total number of task blocks in the batch result matrix according to the size of a single task block and the maximum MNK.

[0056] Step S4: Allocate the maximum number of task blocks according to the computing core serial number coreID starting from 0 and sequentially to each computing core.

[0057] Step S5: Determine whether all task blocks have been assigned.

[0058] Step S6: If the task blocks have not been allocated, jump to step S4.

[0059] Step S7: When the task block is allocated, determine whether the task block is a valid task block.

[0060] Step S8: When the task block is a valid task block, the computing unit in the computing core is called to perform the matrix multiplication operation of the task block.

[0061] Step S9: When the task block is not a valid task block, the computing core skips the calculation of the task block and directly performs the calculation of the next task block.

[0062] Step S10: The matrix multiplication operations of all task blocks are completed, and the calculation results of the batch result matrix are obtained.

[0063] Specifically, in the NPU of the computing platform, UB is an internal storage resource used to temporarily store data so that the computing core can perform efficient computing. The memory space of UB is limited, so the size of the task block needs to be determined according to the capacity of UB to ensure that the data of each task block can be fully loaded into UB, thereby improving computing efficiency. In order to determine the size of a single task block, it is necessary to ensure that the memory space of UB can accommodate the data required by the task block.

[0064] MNK is used to describe the size of a matrix. M (Matrix Rows) represents the number of rows of a matrix, N (Matrix Columns) represents the number of columns of a matrix, and K (Matrix Inner Dimension) represents the internal dimension when two matrices are multiplied, that is, the number of columns of the first matrix and the number of rows of the second matrix. In batch matrix multiplication, MNK usually represents the matrix dimension in each batch. For example, for two matrices A and B, if the dimension of matrix A is M×K and the dimension of matrix B is K×N, then the dimension of the result of their multiplication, matrix C, is M×N. Batch matrix multiplication involves multiple such matrix multiplication operations. The matrix dimensions in each batch may be different, but they can all be described by MNK.

[0065] In order to complete the calculation of a single task block, the UB memory space must be able to store two small matrices of size m, where the data type of each element is single precision of 4 bytes. The occupied memory size is 2×m×m×4 bytes. In order to meet the hardware architecture requirements of the NPU of the computing platform, integers that are multiples of 32 bytes are rounded down. Thus, the calculation formula for m can be obtained.

[0066] Batch matrix multiplication is to calculate multiple matrices simultaneously to obtain multiple result matrices. Matrix A and Matrix B are two input matrices in the batch result matrix. The dimension of matrix A is M×K, and the dimension of matrix B is K×N. For batch matrix multiplication, these values ​​can be different in matrices of different batches. In batch matrix multiplication, the dimension information M, K, and N of matrix A and matrix B are usually passed to the program in the form of arrays. These arrays represent the dimension information of each batch matrix. For example, for batchCount matrices, there will be M arrays, K arrays, and N arrays, and the length of each array is batchCount. Traverse the M arrays, K arrays, and N arrays to find the largest M value, N value, and K value, which are recorded as Mmax, Kmax, and Nmax respectively. These maximum values ​​will be used for subsequent task block division. According to Mmax, Kmax, and Nmax and the size of a single task block, the maximum number of task blocks for a single result matrix is ​​calculated.

[0067] Suppose there is a task block with the serial number ID. To find out which batch the task block corresponding to this serial number is in and its specific position in the batch, it involves the organization and indexing method of task blocks in batch matrix multiplication. Each result matrix will be divided into several task blocks, and the serial numbers of these task blocks are continuously assigned according to batches. Each result matrix is ​​divided according to the maximum number of task blocks in a single result matrix. According to the serial number of a task block, the batch to which this task block belongs and the serial number of this task block in the batch can be directly determined. Suppose the serial number of a task block is ID, the batch to which this task block belongs is batchID, and the serial number of this task block in the batch is tilesID, where, , , , we can get the relative position of the task block in the batch to which it belongs, because the number of task blocks in each batch is fixed. This division and indexing method avoids complex judgment and calculation of each task block at runtime, and can quickly locate its specific position in the batch calculation process directly through the task block serial number, which not only simplifies the indexing process, but also saves storage space and computing resources, thereby improving the computational efficiency of batch matrix multiplication on the NPU. In addition, since the task block is designed to be a size that can be efficiently processed by the NPU hardware, this strategy can also make full use of the parallel computing capabilities of the hardware, reduce the idle time of the computing core, and further improve the parallelism and overall performance of the calculation.

[0068] Since the NPU of the computing platform reads data in column priority, in order to facilitate data reading, the numbering of task blocks is also performed in column priority. Figure 1 like Figure 3 As shown, the task block allocation and logical task block diagram Figure 2 like Figure 4 As shown, for the convenience of demonstration, assume that Mmax is M0, Nmax is N0, and divide according to the maximum number of task blocks of a single result matrix, such as Figure 3 As shown, the result matrix C0 numbered 0 is divided into a fixed number of task blocks, such as Figure 4 As shown, the result matrix C1 numbered 1 is also divided into a fixed number of task blocks. The task block numbers of the result matrix C0 and the result matrix C1 are continuous. Repeat the above steps until all the result matrices are divided. Among them, M0 and K0 are the dimensions of the matrix A in the first batch, K0 and N0 are the dimensions of the matrix B in the first batch, M1 and K1 are the dimensions of the matrix A in the second batch, and K1 and N1 are the dimensions of the matrix B in the second batch. Assume that the number of computing cores is c and the computing core serial number is , the computing task volume of each computing core is: Among them, task(i) indicates the number of task blocks assigned to the computing core with id i. The task blocks are assigned according to the sequence number of the task blocks from small to large, and multiple task blocks are cyclically assigned to each computing core of the NPU. The task blocks are assigned starting from number 0 according to the sequence number of the computing core of the NPU, and are assigned to each computing core of the NPU in sequence; if there are still unassigned task blocks when the last computing core is assigned, the remaining task blocks will be re-assigned starting from computing core number 0, and will be assigned to each computing core of the NPU in sequence; repeat the above steps until all task blocks are assigned.

[0069] In the balanced partitioning strategy, some task blocks are logically divided, but there is actually no data to calculate, such as Figure 4Task blocks 28 to 31 in the task block list are there to optimize the indexing process. By using the serial number of a task block, we can directly determine the batch to which the task block belongs and the serial number of the task block in the batch without performing additional judgments and calculations, thus saving storage space and computing resources. Therefore, when the computing core processes such logical task blocks, it sets a judgment function so that the computing core skips the calculation of the logical task block, thus reducing the amount of calculation on the cube computing unit and avoiding errors in the computing core. For a task block with a serial number of ID, we can calculate that the batch to which the task block belongs is batchID, and the serial number of the task block in the batch is tilesID, where , , . Through batchID, we can get the MNK of the matrix of the batch where this task block is located as MbatchID, NbatchID and KbatchID. Assuming that the number of rows and columns of the task block in the result matrix are x and y respectively, we can calculate x and y as: , , set the judgment function to: , based on the maximum number of task blocks, a balanced partitioning strategy is preset, and all the divided task blocks are first cyclically allocated to each computing core until all the task blocks are allocated. Each time the computing core calculates a task block, it first uses a judgment function to determine whether the task block needs to be calculated. If it is a valid task block, the task block is calculated; if it is a logically divided task block, it does not need to be calculated, and it is directly skipped to calculate the next task block. The present invention presets a balanced partitioning strategy based on the maximum number of task blocks, makes full use of the hardware architecture of the NPU, improves computing efficiency through parallelism, conceals data transfer time, and ensures load balancing of the computing core.

[0070] According to the balanced division strategy, the calculation unit is called to perform matrix multiplication of the task block, and the calculation of all task blocks is completed to finally obtain the batch result matrix. The matrix multiplication operation of the task block is similar to the general matrix multiplication. A task block in the result matrix CbatchID of a batch with batchID needs to perform matrix multiplication on a row of task blocks of the matrix AbatchID and a column of task blocks of the matrix BbatchID in sequence, and add the results together. In this way, the calculation of this task block in the result matrix CbatchID is completed. When all task blocks in the result matrix CbatchID are calculated, the result matrix CbatchID is calculated. When all result matrices CbatchID are calculated, the batch result matrix of batch matrix multiplication is obtained. ,in For a task block in the result matrix with batch ID , in matrix A Perform matrix multiplication of the task block with the i-th block from left to right in the row and the i-th block from top to bottom in the y-th column of the general matrix B to obtain the intermediate result , where x and y are the number of rows and columns of a task block in the result matrix C, respectively, and , , ; Then, all the intermediate results are added together by matrix addition to get the final result of the task block ; Find all task blocks , you can get the complete result matrix ; Find all , you can get the batch result matrix of batch matrix multiplication Therefore, find all task blocks , you can get the calculation result of batch matrix multiplication.

[0071] The embodiment of the present application also provides an acceleration device for batch matrix multiplication. It should be noted that the acceleration device for batch matrix multiplication in the embodiment of the present application can be used to execute the acceleration method for batch matrix multiplication running in the computing platform provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been described will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0072] The following is an introduction to the batch matrix multiplication acceleration device provided in the embodiment of the present application.

[0073] Figure 6 Schematic diagram of an acceleration device for batch matrix multiplication according to an embodiment of the present application. Figure 6 As shown, the device comprises:

[0074] A splitting unit 10 is used to split the batch result matrix into a maximum number of task blocks, wherein the batch result matrix includes a plurality of result matrices, and any of the result matrices is a matrix obtained by multiplying two matrices;

[0075] Specifically, the splitting unit splits the batch result matrix into the maximum number of task blocks. The batch result matrix includes multiple result matrices, each of which is obtained by multiplying two matrices. These result matrices are split into as many small blocks as possible to facilitate parallel processing and improve computing efficiency. The purpose of splitting the task blocks is to allocate them to multiple computing nodes to speed up the computing speed, which can effectively utilize the advantages of parallel computing and improve the overall computing efficiency.

[0076] A determination unit 20, configured to determine a balanced partitioning strategy according to the maximum number of task blocks, wherein the balanced partitioning strategy is a strategy of evenly distributing the plurality of task blocks to a plurality of computing cores of a neural network processing unit NPU in a computing platform;

[0077] Specifically, the determination unit evenly distributes the task blocks to multiple computing cores of the neural network processing unit NPU in the computing platform, which ensures that each computing core can be fully utilized to avoid the situation where some computing cores are overloaded and cause performance degradation. Through the balanced partitioning strategy, it is possible to avoid the situation where some computing cores are overloaded while other computing cores are idle, so that each computing core can efficiently participate in the calculation of batch matrix multiplication, thereby maximizing the overall performance of the computing platform.

[0078] The calling unit 30 is used to call the computing unit in the computing core according to the balanced partitioning strategy, perform matrix multiplication operations on all the task blocks, and obtain the calculation results of the batch result matrix.

[0079] Specifically, the calling unit uses a balanced partitioning strategy to allocate computing tasks and calls the computing unit to perform matrix multiplication operations to obtain the calculation results of the batch result matrix. This strategy can ensure that the computing tasks are evenly distributed among the various computing units, fully utilize computing resources, and improve computing efficiency. In this way, large-scale matrix multiplication operations can be effectively completed and accurate results can be obtained. The purpose of the balanced partitioning strategy is to ensure that the various computing cores in the NPU can evenly distribute tasks when performing batch matrix multiplication, avoiding the situation where some computing cores are overloaded while other computing cores are idle.

[0080] In one embodiment of the present application, the above-mentioned segmentation unit includes:

[0081] A first determination module is used to determine the size of a single task block according to the memory space of a unified buffer UB, where the unified buffer UB is a unified buffer in the NPU of the computing platform;

[0082] The second determination module is used to determine the maximum total number of task blocks in the batch result matrix according to the largest MNK in the batch result matrix and the size of the single task block. The MNK corresponds to the dimensions of rows, columns and inner products in matrix multiplication.

[0083] In a specific embodiment, the first determination module includes: assuming that the size of the single task block is m×m, the calculation formula of m is: , where K is the memory space size of the unified buffer.

[0084] Specifically, in order to complete the calculation of a single task block, the UB memory space must be able to store two small matrices of size , where the data type of each element is single precision of 4 bytes. The occupied memory size is 2×m×m×4 bytes. In order to meet the hardware architecture requirements of the NPU of the computing platform, integers that are multiples of 32 bytes are rounded down. Thus, the calculation formula for m can be obtained.

[0085] In one embodiment of the present application, the second determining module includes:

[0086] An operator module, used for performing a matrix multiplication operation on a matrix A and a matrix B, wherein the matrix A and the matrix B are two of the above result matrices in the above batch result matrix;

[0087] The transmission submodule is used to pass the M value and K value of the above matrix A and the K value and N value of the above matrix B to the program in the form of an array. Let the total number of the above batch result matrices be batchCount, the batch be batchID, and the length and width of the matrix of the above batch be MbatchID, KbatchID and NbatchID, where, ;

[0088] The traversal submodule is used to traverse the three arrays of M, N, and K to obtain the largest M value, N value, and K value in the three arrays as Mmax, Kmax, and Nmax;

[0089] The sub-module is used to divide the single result matrix into multiple task blocks according to Mmax, Kmax, Nmax and the size of the single task block. The maximum number of task blocks for a single result matrix is ;

[0090] The determination submodule is used to determine the maximum total number of task blocks of the batch result matrix according to the maximum number of task blocks of the single result matrix. .

[0091] In a specific embodiment, the second determination module further includes: dividing each of the above result matrices according to the maximum number of task blocks of a single result matrix; wherein, dividing each of the above result matrices according to the maximum number of task blocks of a single result matrix includes: defining the serial number of the target task block as ID1, the batch in which the above target task block is located as batchID1, and the serial number of the above target task block in the above batch is tilesID1, wherein, , , , the calculation formula for batchID1 and tilesID1 is , .

[0092] In one embodiment of the present application, the above-mentioned balanced division strategy includes: a first allocation step, allocating the maximum number of the above-mentioned task blocks according to the computing core sequence number coreID of the above-mentioned NPU starting from No. 0, and sequentially allocating them to each computing core of the above-mentioned NPU, and the computing task amount of each of the above-mentioned computing cores is , where task(i) represents the number of task blocks assigned to the computing core with id i, and c represents the number of computing cores. is the serial number of the computing core; in the second allocation step, when there are still unallocated task blocks after the last computing core is allocated, the unallocated task blocks are reallocated again starting from the computing core with coreID 0, and are allocated to each computing core of the NPU in sequence; the first allocation step and the second allocation step are repeated at least once until all the task blocks are allocated; and an optimization step is performed to optimize the multiplication calculation method of the batch result matrix.

[0093] In one embodiment of the present application, the above optimization step includes: setting a judgment function: ,in, , , where ID is the serial number of the task block, batchID is the batch to which the task block belongs, tilesID is the serial number of the task block in the batch, MbatchID, NbatchID and KbatchID are the MNK of the matrix of the batch to which the task block belongs, and x and y are the number of rows and columns of the task block in the result matrix respectively; before the computing core executes the task block calculation, the judgment function is used to judge whether the task block is a valid task block; if the task block is a valid task block, the computing core executes the matrix multiplication operation of the task block; if the task block is not a valid task block, the computing core skips the calculation of the task block and directly performs the calculation of the next task block.

[0094] In one embodiment of the present application, the calling unit includes:

[0095] A calculation module is used for performing a matrix multiplication operation on a row of task blocks and a column of task blocks in any two of the result matrices for each batch of the result matrices to obtain multiple calculation results;

[0096] An accumulation module, used for accumulating a plurality of the above calculation results to obtain a final result matrix of the above task blocks in the above result matrix of the corresponding batch;

[0097] A judging module, used for judging whether the calculation of all the task blocks in the result matrix is ​​completed, and determining that the calculation of the result matrix is ​​completed when the calculation of all the task blocks in the result matrix is ​​completed;

[0098] The calculation module, the accumulation module and the judgment module are repeatedly executed at least once until all the result matrices are calculated to obtain the calculation results of the batch result matrices.

[0099] The batch matrix multiplication acceleration device includes a processor and a memory. The segmentation unit, determination unit, calling unit, etc. are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0100] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0101] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the acceleration method of batch matrix multiplication running in the computing platform.

[0102] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0108] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0109] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0111] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for accelerating batch matrix multiplication running in a computing platform, characterized in that: include: Divide a batch result matrix into a maximum number of task blocks, wherein the batch result matrix includes a plurality of result matrices, and any of the result matrices is a matrix obtained by multiplying two matrices; Determining a balanced partitioning strategy according to the maximum number of task blocks, wherein the balanced partitioning strategy is a strategy of evenly distributing the plurality of task blocks to the plurality of computing cores of the neural network processing unit NPU in the computing platform; The computing unit in the computing core is called according to the balanced partitioning strategy to perform matrix multiplication operations on all the task blocks to obtain the calculation results of the batch result matrix. The balanced partitioning strategy includes: In the first allocation step, the maximum number of the task blocks are allocated starting from 0 according to the computing core sequence number coreID of the NPU, and are allocated to each computing core of the NPU in sequence. The computing task amount of each computing core is ,in, express is the number of task blocks assigned to the computing core i, c is the number of computing cores, is the serial number of the computing core; The second allocation step, when there are still unallocated task blocks after the last computing core is allocated, the unallocated task blocks are allocated again starting from the computing core with coreID 0, and are allocated to each computing core of the NPU in sequence; Repeating the first assigning step and the second assigning step at least once until all the task blocks are assigned; The optimization step optimizes the multiplication calculation method of the batch result matrix.

2. The method according to claim 1, characterized in that Split the batch result matrix into the maximum number of task blocks, including: Determine the size of a single task block according to the memory space of a unified buffer UB, wherein the unified buffer UB is a unified buffer in the NPU of the computing platform; The maximum value of the total number of task blocks in the batch result matrix is ​​determined according to the largest MNK in the batch result matrix and the size of the single task block, where MNK corresponds to the dimensions of rows, columns, and inner products in matrix multiplication.

3. The method according to claim 2, characterized in that The size of a single-task block is determined based on the memory space of the unified buffer, including: Assume that the size of the single task block is m×m, and the calculation formula of m is: , where K is the memory space size of the unified buffer.

4. The method according to claim 2, characterized in that: Determining the maximum total number of task blocks in the batch result matrix according to the largest MNK in the batch result matrix and the size of the single task block includes: Performing a matrix multiplication operation on matrix A and matrix B, wherein matrix A and matrix B are two result matrices in the batch result matrix; Pass the M and K values ​​of the matrix A and the K and N values ​​of the matrix B to the program in the form of an array. Let the total number of the batch result matrices be batchCount, the batch be batchID, and the length and width of the matrix of the batch be M. batchID , K batchID and N batchID ,in, ; Traverse the three arrays of M, N, and K, and get the largest M value, N value, and K value among the three arrays. max , K max and N max ; According to M max , K max 、N max and the size of the single task block, dividing the single result matrix into multiple task blocks, the maximum number of task blocks for a single result matrix ; According to the maximum number of task blocks of a single result matrix, the maximum total number of task blocks of the batch result matrix is ​​determined as .

5. The method according to claim 4, characterized in that The method further comprises: dividing each of the result matrices according to the maximum number of task blocks of a single result matrix; The result matrices are divided according to the maximum number of task blocks of a single result matrix, including: defining the serial number of the target task block as ID1, the batch of the target task block as batchID1, and the serial number of the target task block in the batch as tilesID1, wherein, , , , the calculation formula for batchID1 and tilesID1 is , .

6. The method according to claim 1, characterized in that The optimization step comprises: Set the judgment function: ,in, , , where ID is the serial number of the task block, batchID is the batch to which the task block belongs, tilesID is the serial number of the task block in the batch, and M batchID 、N batchID is the MN of the matrix of the batch where the task block is located, and x and y are the number of rows and columns of the task block in the result matrix respectively; Before the computing core executes the task block calculation, using the judgment function to judge whether the task block is a valid task block; In the case where the task block is a valid task block, the computing core performs a matrix multiplication operation of the task block; In the case that the task block is not a valid task block, the computing core skips the calculation of the task block and directly performs the calculation of the next task block.

7. The method according to claim 1, characterized in that The computing unit in the computing core is called according to the balanced partitioning strategy to perform matrix multiplication operations on all the task blocks to obtain the calculation results of the batch result matrix, including: A calculation step, for each batch of the result matrices, performing a matrix multiplication operation on a row of task blocks and a column of task blocks in any two of the result matrices to obtain multiple calculation results; An accumulation step of accumulating the plurality of calculation results to obtain a final result matrix of the task blocks in the result matrix of the corresponding batch; A judging step, judging whether the calculation of all the task blocks in the result matrix is ​​completed, and determining that the calculation of the result matrix is ​​completed when the calculation of all the task blocks in the result matrix is ​​completed; The calculation step, the accumulation step and the judgment step are repeatedly performed at least once until all the result matrices are calculated to obtain the calculation results of the batch result matrices.

8. An acceleration device for batch matrix multiplication, characterized in that: include: A splitting unit, used for splitting a batch result matrix into a maximum number of task blocks, wherein the batch result matrix includes a plurality of result matrices, and any of the result matrices is a matrix obtained by multiplying two matrices; A determination unit, configured to determine a balanced partitioning strategy according to the maximum number of task blocks, wherein the balanced partitioning strategy is a strategy of evenly distributing a plurality of the task blocks to a plurality of computing cores of a neural network processing unit NPU in a computing platform; A calling unit is used to call the computing unit in the computing core according to the balanced partitioning strategy, perform matrix multiplication operations on all the task blocks, and obtain the calculation results of the batch result matrix. The balanced partitioning strategy includes: In the first allocation step, the maximum number of the task blocks are allocated starting from 0 according to the computing core sequence number coreID of the NPU, and are allocated to each computing core of the NPU in sequence. The computing task amount of each computing core is ,in, express is the number of task blocks assigned to the computing core i, c is the number of computing cores, is the serial number of the computing core; The second allocation step, when there are still unallocated task blocks after the last computing core is allocated, the unallocated task blocks are allocated again starting from the computing core with coreID 0, and are allocated to each computing core of the NPU in sequence; Repeating the first assigning step and the second assigning step at least once until all the task blocks are assigned; The optimization step optimizes the multiplication calculation method of the batch result matrix.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the batch matrix multiplication acceleration method according to any one of claims 1 to 7.

Images