A GPU batch matrix multiplication accelerator and its processing method

Abstract

The invention discloses a GPU batch matrix multiplication accelerator, comprising a main branch circuit composed of a workgroup manager, an instruction decoding processor, a high-speed storage unit, a shared memory unit, a computing unit, and a DMA, which is used to perform batch order optimization on matrix data to obtain matrix slices, and load the matrix slices into the computing unit; a side branch circuit composed of an instruction decoding processor, a high-speed storage unit, a shared memory unit, a computing unit, and a DMA, which directly performs kernel function calculation on the matrix data to obtain matrix calculation results when batch order optimization is not performed; the invention can effectively improve the utilization efficiency of the computing unit and achieve load balancing of the task amount of each computing unit, improve the computing density during operation, achieve higher instruction parallelism, thread parallelism, and memory access parallelism, thereby giving full play to the computing power of the hardware and achieving the purpose of computing acceleration.

Description

Technical Field

[0001] The present invention relates to the research field of high performance computing, and in particular to a GPU batch matrix multiplication accelerator and a processing method thereof. Background Art

[0002] Basic Linear Algebra Subprograms (BLAS) are interface standards for a series of basic linear algebra operations, which are widely used in various fields of scientific computing and industry. Some more advanced languages ​​and computing libraries are also implemented by calling BLAS interfaces (R language, Matlab, Numpy, Lapack, etc.). Nowadays, with the development of BLAS, various BLAS library implementations based on different platforms and hardware architectures have emerged, such as CuBLAS, RocBLAS, MKL, MAGMA, OpenBLAS, etc., which have played a vital role in the development of modern science and industry.

[0003] Classic BLAS applications often have good performance for large inputs (large matrix and vector operations); however, for scenarios where the problem scale is small and the number of problems is large, resources are not fully utilized due to the small size of each problem, which greatly affects performance and efficiency. However, with the continuous development of high-performance computing and machine learning, this type of problem often occurs and has become an emerging trend, which forces classic BLAS to be optimized and expanded in line with the development.

[0004] In order to address the above challenges, the concept and preliminary implementation of batch BLAS have been proposed in recent years. It solves the dilemma of poor performance of classic BLAS in this scenario by processing multiple sub-problems in parallel. Among these batch BLAS routines, batch matrix multiplication has received extensive attention and research as the most commonly used linear algebra operation, and the industry is in urgent need of high-performance batch matrix multiplication implementation to promote development. However, the existing batch BLAS implementations and functions are still relatively rudimentary. For batch matrix multiplication, the current GPU-based BLAS libraries (CuBLAS, RocBLAS) only support fixed-size problem inputs, but not variable-size matrix calculations, which greatly limits its application scenarios.

[0005] In order to improve the applicability and computational efficiency of batch matrix-related calculations, the industry's most advanced MAGMA computing library has proposed the vbatch method to support variable-size matrix calculations. Among them, the most popular is the xgemm_vbatched API. Compared with the current CUDA and ROCm platforms, when solving variable-size batch matrix multiplications, it is necessary to call the xgemm routine in a loop to solve each calculation instance in turn. It can simultaneously load all calculation instances to the GPU for parallel calculation, reducing the idle GPU resources and excessive repeated scheduling resource overhead caused by a single instance being too small, greatly improving the algorithm execution efficiency and computational parallelism, and thus achieving a very considerable acceleration.

[0006] However, when performing matrix calculations on variable-size matrices, the simultaneous calls of different kernel functions between different computing instances will cause irregularities within the hardware during calculations. In addition, the work groups generated by the different kernel function partitioning strategies between instances will also cause an imbalance in the workload of each computing unit when loading the computing unit due to the Round-robin scheduling algorithm, causing uneven distribution of GPU computing resources, which in turn causes great damage to computing parallelism and computing density.

[0007] In the prior art, there are the following problems: when the existing GPU-based batch matrix multiplication computing library solves large-batch matrix multiplication of variable sizes, the combined effect of the variable-size matrix input distribution and the workgroup scheduling algorithm will cause serious load imbalance and internal irregularities between computing units, affecting the execution efficiency of the computing pipeline, making it impossible for the GPU to fully utilize the parallel capabilities of the hardware, thereby reducing the final algorithm calculation rate. Summary of the invention

[0008] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide a GPU batch matrix multiplication accelerator and a processing method thereof. This accelerator is embedded in a large batch of matrix input calculation processes, and can analyze the internal distribution characteristics of the matrix input, and according to the input properties obtained by the analysis, after optimizing and adjusting the batch order, the adjusted matrix input is loaded into the kernel function for parallel calculation, and the final calculation result is obtained. In the batch order optimization, the batch order of the matrix will be combined with the hardware scheduling characteristics of the GPU, and readjusted through a specific algorithm, which can effectively improve the utilization efficiency of the computing unit and achieve load balancing of the task volume of each computing unit, improve the computing density during runtime, and achieve higher instruction parallelism, thread parallelism and memory access parallelism, so as to give full play to the computing power of the hardware and achieve the purpose of computing acceleration.

[0009] The first object of the present invention is to provide a GPU batch matrix multiplication accelerator;

[0010] The second object of the present invention is to provide a GPU batch matrix multiplication acceleration processing method.

[0011] The first object of the present invention is achieved by the following technical solutions:

[0012] A GPU batch matrix multiplication accelerator, comprising:

[0013] The instruction decoding processor reads and decodes the DMA transmission packet submitted to the instruction queue by the CPU, and transmits the decoded result to the DMA engine to load the matrix data from the CPU;

[0014] A computing unit, used to perform parallel calculations on the loaded matrix slices, and to parallelly calculate the slice calculation results in the matrix slices through kernel functions;

[0015] Shared memory unit, used to store the matrix part of the intermediate calculation process of matrix slices;

[0016] A high-speed storage unit for storing relevant data;

[0017] Workgroup manager, used for work management between units;

[0018] The main branch circuit is composed of a workgroup manager, an instruction decoding processor, a high-speed storage unit, a shared memory unit, a computing unit, and a DMA, and is used to batch-order optimize the matrix data, obtain matrix slices, and load the matrix slices into the computing unit;

[0019] The side branch circuit is composed of an instruction decoding processor, a high-speed storage unit, a shared memory unit, a computing unit, and a DMA. When batch order optimization is not performed, the kernel function is directly calculated on the matrix data to obtain the matrix calculation result.

[0020] Furthermore, the workgroup manager divides the grid according to the configuration information, splits out a plurality of workgroups, sends the data of the plurality of workgroups to the computing unit for computing, and manages the order of scheduling the workgroups to perform the computing.

[0021] Furthermore, the storage related data is specifically as follows: the related data includes input matrix size information, batch size information, matrix elements and pointers, batch sequence information, strategy buckets, grid configuration information, and output matrix.

[0022] Furthermore, when a batch matrix multiplication problem instance is input to the accelerator, the accelerator determines whether to enter the main branch circuit through the batch matrix information, counts the workload of each matrix multiplication problem instance in the main branch and roughly sorts them, takes the best partition size for the problem instance in combination with the size of the kernel function and the configuration of the computing unit, and finely sorts the size of the problem instance to optimize the final batch order, loads the optimized matrix data into the computing unit for calculation, and returns the final result to the instruction decoding processor; if the optimization branch is not entered, the side branch circuit is entered, and the matrix data is directly loaded into the computing unit to use the kernel function for calculation, and the result is returned.

[0023] The second object of the present invention is achieved by the following technical solutions:

[0024] A GPU batch matrix multiplication acceleration processing method comprises the following steps:

[0025] Input the matrix data into the accelerator, judge the matrix data, and choose whether to perform batch sequence optimization based on the judgment result;

[0026] If batch order optimization is required, statistical analysis is performed on the matrix size of the matrix data to obtain statistical analysis results;

[0027] Match and divide the matrix data according to the slice size of the kernel function to obtain the division result;

[0028] Classify according to the matrix data pointer to obtain the classification result;

[0029] Rearrange the batch order of data input according to statistical analysis results, division results, and classification results;

[0030] Recall and calculate according to the rearranged order combined with the kernel function to obtain the final calculation result;

[0031] If batch order optimization is not required, the kernel function is directly used to calculate and obtain the final calculation result.

[0032] Furthermore, the batch order optimization is specifically as follows: analyzing the current matrix data input; traversing the matrix size of each instance in the matrix data input, statistically recording the workload required for each instance to complete the calculation, and the slice size of the kernel function corresponding to each instance, marking the kernel function that meets the instance and converting its kernel function into the index of the current instance, and recording and classifying the index of each instance; after classifying all instance records, traversing all instance indexes again to obtain the final batch order, and rewriting the sorted instance pointers to obtain optimized matrix data, and loading the optimized matrix data into the GPU for parallel computing.

[0033] Furthermore, the classification of all instance records is specifically as follows: traverse all instances of the entire batch sequence matrix data input, analyze M, N, K corresponding to all instances, M is the number of rows of matrix A and matrix C in the instance matrix, N is the number of columns of matrix B and matrix C in the instance matrix, K is the number of columns of matrix A and the number of rows of matrix B in the instance matrix; use the K value as the workload required for the inner loop of the kernel function, and sort it as the primary key for rough sorting; again, classify each instance in combination with the M and N corresponding to the current instance to calculate the kernel function of the current instance, index the kernel function, and classify each instance according to the size of the corresponding different kernel functions.

[0034] Furthermore, the sorting is specifically performed as follows: using the K value of each instance as the primary key for coarse sorting, that is, sorting the order of all instances from small to large according to the workload of the kernel function, using an operation-based sorting algorithm for all instances, storing each marked index object in a policy bucket corresponding to the kernel function, and extracting all index tags stored in each policy bucket to obtain the final sorted instance order.

[0035] Furthermore, each instance is classified according to the size of the corresponding different kernel functions, specifically:

[0036] Calculate the optional partitioning strategies for the matrix C in the current instance. The strategies to be selected meet the following requirements:

[0037]

[0038] Among them, M i is the number of rows of the corresponding matrix A and matrix C of the current calculation instance, N i is the number of columns of the corresponding matrices B and C; T Mj is the number of rows of the submatrices of matrix A and matrix C of the current kernel function partitioning strategy, T Nj is the number of columns of the corresponding submatrices of matrix B and matrix C;

[0039] After obtaining multiple candidate strategies, select the partition strategy that meets the following requirements as the best partition strategy for the current computing instance. At this time, the best partition strategy has the highest parallelism, and the requirements are:

[0040] max{T Mj ×T Nj}

[0041] The index j of the currently selected best partitioning strategy is used as the label index of the instance classification, and is placed in the corresponding strategy bucket, and the current best partitioning strategy is used as the corresponding basis for the kernel function used by the instance.

[0042] Furthermore, the rewriting of the sorted instance pointers is specifically: traversing the sorted instance order, and re-adjusting the order of the memory pointer, dimension pointer, leading dimension pointer and scalar pointer of each instance and writing them back to the pointer array of the matrix data input, so that the API interface can be directly called during calculation.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. The present invention can be implemented and natively executed on mainstream scientific high-performance computing platforms such as CUDA and ROCm, which greatly increases the versatility and applicability of the software, covering various parallel computing cards and graphics processors that support general computing from NVIDIA and AMD.

[0045] 2. The present invention is an optimization component for kernel function call execution, which can effectively reduce the problem of imbalance in the total amount of resources inside the computing unit caused by inconsistent instance workload and hardware scheduling algorithm in the solution process of the existing batch variable-size matrix multiplication library, improve the load balancing of computing resources, and improve the density and parallelism of computing while reducing resource waste, thereby improving the efficiency and speed of computing.

[0046] 3. The variable-size batch matrix multiplication targeted by the present invention is widely used in high-performance computing scenarios, machine learning, and artificial intelligence scenarios, and is one of the optimization focuses of current high-performance heterogeneous systems. The invention is of great significance to the design and construction of related applications and the combination of advanced algorithms in these scenarios, and can accelerate the execution efficiency of applications and algorithms in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a structural block diagram of a GPU batch matrix multiplication accelerator described in the present invention;

[0048] Figure 2 It is a flow chart of a GPU batch matrix multiplication acceleration processing method described in the present invention;

[0049] Figure 3 It is a calculation flow chart of a GPU batch matrix multiplication accelerator described in the present invention;

[0050] Figure 4 It is an internal flow chart of a GPU batch matrix multiplication accelerator described in the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0052] Example 1

[0053] A GPU batch matrix multiplication accelerator, such as Figure 1 As shown, including:

[0054] The instruction decoding processor reads and decodes the DMA transmission packet submitted to the instruction queue by the CPU, and transmits the decoded result to the DMA engine to load the matrix data from the CPU;

[0055] A computing unit, used to perform parallel calculations on the loaded matrix slices, and to parallelly calculate the slice calculation results in the matrix slices through kernel functions;

[0056] Shared memory unit, used to store the matrix part of the intermediate calculation process of matrix slices;

[0057] A high-speed storage unit for storing relevant data;

[0058] Workgroup manager, used for work management between units;

[0059] The main branch circuit is composed of a workgroup manager, an instruction decoding processor, a high-speed storage unit, a shared memory unit, a computing unit, and a DMA, and is used to batch-order optimize the matrix data, obtain matrix slices, and load the matrix slices into the computing unit;

[0060] The side branch circuit is composed of an instruction decoding processor, a high-speed storage unit, a shared memory unit, a computing unit, and a DMA. When batch order optimization is not performed, the kernel function is directly calculated on the matrix data to obtain the matrix calculation result.

[0061] Furthermore, the workgroup manager divides the grid according to the configuration information, splits out a plurality of workgroups, sends the data of the plurality of workgroups to the computing unit for computing, and manages the order of scheduling the workgroups to perform the computing.

[0062] Furthermore, the storage related data is specifically as follows: the related data includes input matrix size information, batch size information, matrix elements and pointers, batch sequence information, strategy buckets, grid configuration information, and output matrix.

[0063] The implementation steps of the accelerator are as follows: Figure 2 As shown, the CPU and GPU are combined as follows Figure 3 The specific internal flow chart of the accelerator is shown in Figure 4 As shown:

[0064] Step 1: First, make a judgment based on the current matrix input. If the batch itself is already in an optimized order, or it is explicitly specified that no order optimization is required, then enter the bypass branch and directly enter the kernel function call to perform the calculation. Otherwise, enter the batch order optimization process.

[0065] Step 2: If you enter the batch optimization process, first analyze the current matrix input, traverse the matrix size of each instance in the input, and statistically record the workload required by the work group corresponding to each instance. The workload is determined by K, that is, the number of columns in matrix A and the number of rows in matrix B.

[0066] Step 3: Perform a rough sort of the first round of each instance with K as the primary key, and arrange all the instances in the batch from large to small according to the workload corresponding to the work group they generated. At this time, the kernel functions corresponding to all the instances in the batch are still not regularized and need to be further optimized.

[0067] Step 4: Create the same number of temporary strategy buckets as the kernel function partitioning strategy to sort each instance again.

[0068] Step 5: Calculate the optimal kernel function and the label index for classification for each instance, and put the index into the corresponding strategy bucket. The optimal kernel function and label index are obtained by the following steps:

[0069] ① Calculate the optional partitioning strategy of the matrix C in the current calculation instance. The strategy to be selected meets the following requirements:

[0070] ②

[0071] Among them, M i is the number of rows of the corresponding matrix A and matrix C of the current calculation instance, N i is the number of columns of the corresponding matrices B and C; T Mj is the number of rows of the submatrices of matrix A and matrix C of the current kernel function partitioning strategy, T Nj is the number of columns of the corresponding submatrices of matrix B and matrix C;

[0072] ② After obtaining multiple candidate strategies, select the partition strategy that meets the following requirements as the best partition strategy for the current computing instance. Since all feasible partition strategies for the current instance have been screened out in ①, the partition strategy with the largest number of processed elements is used as the best partition strategy to increase the number of matrix elements processed by a single work group. At the same time, a larger partition strategy can have a higher shared memory utilization rate, and the instruction density and computing density are the highest. That is, the strategy that meets the following conditions is the best partition strategy:

[0073] max{T Mj ×T Nj}

[0074] The index j of the currently selected strategy is used as the label index of the instance classification, and is placed in the corresponding strategy bucket, and the current strategy is used as the corresponding basis for the kernel function used by the instance.

[0075] Step 6: Traverse all strategy buckets and all marked indexes in the buckets to obtain the adjusted batch order, and write the memory pointer, dimension pointer, leading dimension pointer and scalar pointer of the instance corresponding to the index back to the corresponding pointer array of the original input to facilitate direct API calls during calculation.

[0076] Step 7: For the optimized batch order input, call the marked kernel function in parallel to calculate the final operation result. The result is stored according to the modified memory pointer to ensure data consistency and correctness.

[0077] Step 8: The kernel function integrates the calculation results and returns them to the memory, and the overall calculation is completed.

[0078] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A GPU batch matrix multiplication accelerator, characterized in that: include: The instruction decoding processor reads and decodes the DMA transfer packet submitted to the instruction queue by the CPU, and transmits the decoded result to the DMA transfer engine to load the matrix data from the CPU; A computing unit, used to perform parallel calculations on the loaded matrix slices, and to parallelly calculate the slice calculation results in the matrix slices through kernel functions; Shared memory unit, used to store the matrix part of the intermediate calculation process of matrix slices; A high-speed storage unit for storing relevant data; Workgroup manager, used for work management between units; The main branch circuit is composed of a workgroup manager, an instruction decoding processor, a high-speed storage unit, a shared memory unit, a computing unit, and a DMA, and is used to batch-order optimize the matrix data, obtain matrix slices, and load the matrix slices into the computing unit; The side branch circuit is composed of an instruction decoding processor, a high-speed storage unit, a shared memory unit, a computing unit, and a DMA. When batch order optimization is not performed, the kernel function is directly calculated on the matrix data to obtain the matrix calculation result; When a batch matrix multiplication problem instance is input to the accelerator, the accelerator determines whether to enter the main branch circuit through the batch matrix information, counts the workload of each matrix multiplication problem instance in the main branch and roughly sorts it, takes the best partition size for the problem instance in combination with the size of the kernel function and the configuration of the computing unit, and finely sorts the size of the problem instance to optimize the final batch order, loads the optimized matrix data into the computing unit for calculation, and returns the final result to the instruction decoding processor; If the optimization branch is not entered, the side branch circuit is entered, and the matrix data is directly loaded into the calculation unit to use the kernel function for calculation and return the result.

2. A GPU batch matrix multiplication accelerator according to claim 1, characterized in that: The workgroup manager divides the grid according to the configuration information, splits out a plurality of workgroups, sends the data of the plurality of workgroups to the computing unit for computing, and manages the order of scheduling the workgroups to perform the computing.

3. The GPU batch matrix multiplication accelerator according to claim 1, characterized in that: The storage related data specifically includes: the related data includes input matrix size information, batch size information, matrix elements and pointers, batch sequence information, strategy buckets, grid configuration information, and output matrix.

4. A GPU batch matrix multiplication acceleration processing method, applied to the GPU batch matrix multiplication accelerator according to any one of claims 1 to 3, characterized in that: The following steps are involved: Input the matrix data into the accelerator, judge the matrix data, and choose whether to perform batch sequence optimization based on the judgment result; If batch order optimization is required, statistical analysis is performed on the matrix size of the matrix data to obtain statistical analysis results; Match and divide the matrix data according to the slice size of the kernel function to obtain the division result; Classify according to the matrix data pointer to obtain the classification result; Rearrange the batch order of data input according to statistical analysis results, division results, and classification results; Recall and calculate according to the rearranged order combined with the kernel function to obtain the final calculation result; If batch order optimization is not required, the final calculation result is obtained through kernel function calculation.

5. A GPU batch matrix multiplication acceleration processing method according to claim 4, characterized in that: The batch order optimization is specifically as follows: analyzing the current matrix data input; traversing the matrix size of each instance in the matrix data input, statistically recording the workload required for each instance to complete the calculation, and the slice size of the kernel function corresponding to each instance, marking the kernel function that meets the instance and converting its kernel function into the index of the current instance, and recording and classifying the index of each instance; after classifying all instance records, traversing all instance indexes again to obtain the final batch order, and rewriting the sorted instance pointers to obtain optimized matrix data, and loading the optimized matrix data into the GPU for parallel computing.

6. A GPU batch matrix multiplication acceleration processing method according to claim 5, characterized in that: The classification of all instance records is specifically as follows: traverse all instances of the entire batch sequence matrix data input, analyze M, N, K corresponding to all instances, M is the number of rows of matrix A and matrix C in the instance matrix, N is the number of columns of matrix B and matrix C in the instance matrix, K is the number of columns of matrix A and the number of rows of matrix B in the instance matrix; use the K value as the required workload of the inner loop of the kernel function, and sort it as the primary key for rough sorting; again, classify each instance in combination with the M and N corresponding to the current instance to calculate the kernel function of the current instance, index the kernel function, and classify each instance according to the size of the corresponding different kernel functions.

7. A GPU batch matrix multiplication acceleration processing method according to claim 6, characterized in that: The sorting is specifically performed as follows: the K value of each instance is used as the primary key for rough sorting, that is, the order of all instances is sorted from small to large according to the workload of the kernel function, and an operation-based sorting algorithm is used for all instances. Each marked index object is stored in a policy bucket corresponding to the kernel function, and all index tags stored in each policy bucket are extracted to obtain the final sorted instance order.

8. The GPU batch matrix multiplication acceleration processing method according to claim 6, characterized in that: The examples are classified according to the sizes of the corresponding different kernel functions, specifically: Calculate the optional partitioning strategies for the matrix C in the current instance. The strategies to be selected meet the following requirements: Among them, M i is the number of rows of the corresponding matrix A and matrix C of the current calculation instance, N i is the number of columns of the corresponding matrix B and matrix C; T Mj is the number of rows of the matrix A and the submatrix of the matrix C of the current kernel function partitioning strategy, T Nj is the number of columns of the corresponding submatrices of matrix B and matrix C; After obtaining multiple candidate strategies, select the partition strategy that meets the following requirements as the best partition strategy for the current computing instance. At this time, the best partition strategy has the highest parallelism, and the requirements are: max{T Mj ×T Nj } The index j of the currently selected best partitioning strategy is used as the label index of the instance classification, and is placed in the corresponding strategy bucket, and the current best partitioning strategy is used as the corresponding basis for the kernel function used by the instance.

9. The GPU batch matrix multiplication acceleration processing method according to claim 5, characterized in that: The method of rewriting the sorted instance pointers is specifically: traversing the sorted instance order, and re-adjusting the order of the memory pointer, dimension pointer, leading dimension pointer and scalar pointer of each instance and writing them back to the pointer array of the matrix data input, so that the API interface can directly call them when performing calculations.

Images