A matrix multiplication slice configuration selection method and system

Abstract

The present invention relates to a matrix multiplication slice configuration selection method and system. First, all possible slice configuration schemes of thread blocks are listed according to hardware characteristics; then, the thread blocks in each slice configuration scheme are divided into multiple threads in the M direction and the N direction, and each slice configuration scheme is preliminarily screened according to data reuse performance; then, a slice configuration scheme that can fully utilize hardware computing resources is selected from the reserved schemes; then, a slice configuration scheme with a maximum computing memory access ratio is further selected; finally, a slice configuration scheme with a minimum number of kernel function launches is found as the optimal slice configuration scheme. Compared with the prior art, the present invention determines the slice scheme according to the hardware resource usage of the chip and the size of the input matrix, and has strong platform portability; and can dynamically select the slice scheme in real time, which is more friendly and convenient for upper-level users who call the matrix multiplication algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of high performance computing, and in particular relates to a matrix multiplication slice configuration selection method and system. Background Art

[0002] Matrix multiplication operations are widely used, such as playing an important role in deep learning, scientific computing, and image processing. The performance of matrix multiplication is affected by many factors such as the scale of calculation, the size of the calculation case, hardware computing resources, and data reuse, so there are many parameter configuration schemes.

[0003] Some existing matrix multiplication slicing solutions use empirical values ​​when designing the solution. However, due to changes in the software and hardware platforms (such as compiler upgrades, replacement of new generation chips), this method may fail or inaccurately slice. In addition, for variant matrix multiplications in convolutional reasoning and training, the empirical thresholds are significantly different due to the significantly different hardware resources such as shared memory and registers used, so such solutions are not easy to promote. Some matrix multiplication slicing solutions test the time of each slicing solution through performance testing, and then manually select the optimal configuration solution. However, in actual applications, the size of matrix multiplications will continue to change, and the method of manually obtaining the best configuration through performance testing seriously wastes time and manpower. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a matrix multiplication slice configuration selection method and system, which adopts the following technical solutions:

[0005] A matrix multiplication slice configuration selection method comprises the following steps:

[0006] Step 1: List all possible slicing configurations of thread blocks based on hardware characteristics;

[0007] Step 2: Divide the thread blocks in each slicing configuration scheme into multiple threads in the M direction and the N direction, and preliminarily screen each slicing configuration scheme based on data reuse performance;

[0008] Step 3: From the sharding configuration schemes that are initially screened and retained, select and retain the sharding configuration scheme that can fully utilize the hardware computing resources;

[0009] Step 4: Calculate the computational memory access ratio of each shard configuration scheme retained in step 3, select the shard configuration scheme with the largest computational memory access ratio and retain it;

[0010] Step 5: Calculate the number of kernel function transmissions of each sharding configuration scheme retained in step 4, and find the sharding configuration scheme with the minimum number of kernel function transmissions as the optimal sharding configuration scheme.

[0011] Furthermore, in step 2, if a general multiply-add instruction is used to perform matrix multiplication, the number of threads in the M direction and the N direction meets the following conditions:

[0012] 16≤(block_m_threads,block_n_threads)≤max(block_m_threads) / 16;

[0013] If the matrix multiplication hardware acceleration unit is used to perform matrix multiplication operations, the number of threads in the M direction and the N direction must meet the following conditions:

[0014]

[0015] Among them, block_m_threads and block_n_threads represent the number of threads in the M direction and the number of threads in the N direction respectively, and TCUmin shapesize represents the minimum number of threads in the M direction and the N direction supported by the matrix multiplication acceleration unit TCU.

[0016] Furthermore, in step 2, the method for preliminarily screening each shard configuration scheme according to data reuse performance is:

[0017] Exclude sharding configurations with fewer active threads than a set threshold based on register usage;

[0018] Alternatively, you can first list the data reuse solutions for the sharding configuration solution, and then perform performance tests using matrix multiplication test cases of various sizes to exclude data reuse solutions whose performance is lower than the set threshold in all tests.

[0019] Furthermore, the step 3 specifically includes the following steps:

[0020] S1. Obtain the minimum number of thread blocks min_theory_block_num required for each shard configuration scheme to fully occupy the hardware computing resources;

[0021] S2. For the matrix multiplication between the left matrix of m*k and the right matrix of k*n, input the size of the matrix multiplication (m, k, n) and calculate the number of thread blocks Thread_Blocks under each slicing configuration scheme.

[0022]

[0023] Among them, m_tiles and n_tiles represent the number of data multiplexing times in the M direction and the N direction respectively;

[0024] S3. If Thread_Blocks is not less than min_theory_block_num, the sharding configuration scheme is retained; if the Thread_Blocks of all sharding configuration schemes are less than min_theory_block_num, it means that the size of the input matrix multiplication is too small to fully utilize the computing resources. The data reuse times in all sharding configuration schemes are set to 1 and retained.

[0025] Furthermore, the calculation formula for calculating the memory access ratio TLP in step 4 is:

[0026]

[0027] Among them, the slice shape of the m*k left matrix is ​​(block_m_threads*m_tiles,block_k_threads), and the slice shape of the k*n right matrix is ​​(block_k_threads,block_n_threads*m_tiles), where block_k_threads represents the slice length in the K direction.

[0028] Furthermore, the calculation formula of the kernel function launch times Launch_kernel_times in step 5 is:

[0029]

[0030] Where block_per_cu represents the number of thread blocks that can exist in parallel on each computing unit, and

[0031]

[0032] max available register num of a thread indicates the maximum number of registers that can be used by each thread due to chip hardware limitations; num of registers actually used indicates the number of registers actually used by each thread; maxshared memory sizeof a SM indicates the maximum shared memory size that can be used by each stream processor cluster; shared memory sizeused by per thread block indicates the shared memory size actually used by each thread block.

[0033] A matrix multiplication slice configuration selection system comprises an input module, a compilation module and an implementation module of the matrix multiplication slice configuration selection method, wherein the input module is used to input the size of the matrix multiplication, and the compilation module is used to use the configuration parameters of each slice configuration scheme as the template parameters of the matrix multiplication kernel function and compile and generate the kernel functions of all slice configuration schemes; during operation, the size of the matrix multiplication is input, and the implementation module of the matrix multiplication slice configuration selection method selects the best slice configuration scheme and the corresponding kernel function to perform the matrix multiplication operation.

[0034] The beneficial effects of the present invention are as follows: the present invention determines the sharding scheme according to the hardware resource usage of the chip and the size of the input matrix, does not adopt empirical values, and the sharding is accurate and has strong platform portability; compared with the manual selection method through performance testing, the present invention can dynamically select the sharding scheme in real time, which is more friendly and convenient for upper-level users who call the matrix multiplication algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of conventional block matrix multiplication;

[0036] Figure 2 This is a schematic diagram of block matrix multiplication proposed by the present invention;

[0037] Figure 3 The figure is a flow chart of the matrix multiplication slice configuration selection method of the present invention. DETAILED DESCRIPTION

[0038] The present invention will now be described in further detail with reference to the accompanying drawings.

[0039] like Figure 1 As shown, in conventional block matrix multiplication, each thread block calculates a sub-block Pb of TILE_WIDTH×TILE_WIDTH, and each thread processes one element in the sub-block. Figure 2 As shown, each thread block calculates m×n TILE_WIDTH×TILE_WIDTH sub-blocks Pb, which can further reuse data and reduce the number of memory accesses, thereby improving the performance of matrix multiplication operations.

[0040] The main process of the matrix multiplication slice method proposed in the present invention is as follows: Figure 3 As shown:

[0041] First, all slicing configuration schemes of thread blocks are listed according to hardware characteristics. These configuration schemes have less register and shared memory usage, which can ensure that there are enough active warps and good computing parallelism on the stream processor cluster. Too few threads in a block will cause too much shared memory usage, and too many threads in a block will cause too much register usage, both of which will cause the number of active warps to decrease. Warp refers to a thread group, which is the basic execution unit in the stream processor cluster. The GPGPU chip of Tianshu Zhixin used in this embodiment contains 64 threads in one warp.

[0042] Then, according to the actual application and hardware characteristics, the thread blocks in the configuration scheme are divided into block_m_threads threads in the M direction and block_n_threads threads in the N direction.

[0043] If general multiplication and addition instructions are used for matrix multiplication operations, the minimum values ​​of block_m_threads and block_n_threads are the number of banks of the shared memory. The shared memory is divided into multiple memory modules of equal size called banks. For example, the number of banks of Tianshu Zhixin GPGPU chip is 16. This can effectively avoid bank conflicts when accessing the shared memory. The maximum values ​​of block_m_threads and block_n_threads are the maximum value of block_m_threads divided by 16, that is,

[0044] 16≤(block_m_threads,block_n_threads)≤max(block_m_threads) / 16.

[0045] If the matrix multiplication hardware acceleration unit (TCU) is used for matrix multiplication operations, the minimum values ​​of block_m_threads and block_n_threads are determined by the matrix multiplication shape supported by the chip, and the maximum values ​​of block_m_threads and block_n_threads are equal to the maximum block_m_threads divided by the minimum value of block_m_threads and block_n_threads, that is,

[0046]

[0047] Next, based on the performance test, list the reuse schemes that may have better performance. Data reuse can reduce global memory access and thus improve the computational access ratio, but too much data reuse will cause excessive register usage, resulting in a decrease in the number of active warps. Therefore, the data reuse schemes that will definitely have a very small number of active warps can be excluded by register usage. Alternatively, multiple groups of data reuse schemes can be listed, and then performance tests can be performed through matrix multiplication test cases of various sizes to exclude those data reuse schemes that have poor performance in all test cases.

[0048] In this embodiment, when TCU is used to perform matrix multiplication operations, the number of multiplexing times (m_tiles, n_tiles) in the M direction and the N direction is (1, 2), and when general multiplication and addition instructions are used to perform matrix multiplication operations, the number of multiplexing times (m_tiles, n_tiles) in the m direction and the n direction is (2, 4). The corresponding configuration schemes (block_m_threads, block_n_threads, m_tiles, n_tiles) are (4, 64, 1, 1), (4, 64, 1, 2), (32, 32, 2, 1), (32, 32, 2, 2), etc. There are 52 configuration schemes in this embodiment.

[0049] Next, calculate the minimum number of thread blocks min_theory_block_num required for each slicing configuration to fully occupy the hardware computing resources, and calculate the number of thread blocks Thread_Blocks in each slicing configuration according to the size of the input matrix multiplication (m, k, n). The matrix multiplication size between the m*k left matrix and the k*n right matrix can be expressed as (m, k, n).

[0050]

[0051] If Thread_Blocks is not less than min_theory_block_num, the sharding configuration scheme is retained for further screening; if the Thread_Blocks of all sharding configuration schemes are less than min_theory_block_num, it means that the size of the use case matrix multiplication is too small to fully utilize the computing resources. The number of data reuse times in all configuration schemes is set to 1 and then participate in further screening, that is, (block_m_threads, block_n_threads, 1, 1).

[0052] Then, the computational memory access ratio of the reserved sharding configuration schemes is calculated, and the configuration scheme with the maximum computational memory access ratio is found. The formula for calculating the memory access ratio TLP is:

[0053]

[0054] Finally, according to the hardware characteristics and the usage of shared memory and registers, the number of thread blocks that can exist simultaneously on each computing unit is calculated, block_per_cu.

[0055]

[0056] Find the sharding configuration with the minimum kernel launch times Launch_kernel_times,

[0057]

[0058] At this point, one or a few configurations will be retained, and one of them will be selected as the best configuration.

[0059] The present invention also proposes a matrix multiplication slice configuration selection system, which uses the configuration parameters of each configuration scheme as the template parameters of the matrix multiplication kernel function, compiles and generates kernel functions of all configurations, and inputs the size of the matrix multiplication during runtime. The kernel function with the best configuration is selected through the above-mentioned matrix multiplication slice configuration selection method to perform the matrix multiplication operation.

[0060] This embodiment is implemented on the Tianshu Zhixin GPGPU chip, but the present invention is not limited to the Tianshu Zhixin GPGPU chip, and is also applicable to some general programming chips or platforms with specific matrix multiplication acceleration units.

[0061] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A matrix multiplication slice configuration selection method, characterized in that: The following steps are involved: Step 1: List all possible slicing configurations of thread blocks based on hardware characteristics; Step 2: Divide the thread blocks in each slicing configuration scheme into multiple threads in the M direction and the N direction, and preliminarily screen each slicing configuration scheme based on data reuse performance; Step 3: From the sharding configuration schemes that are initially screened and retained, select and retain the sharding configuration scheme that can fully utilize the hardware computing resources; Step 4: Calculate the computational memory access ratio of each shard configuration scheme retained in step 3, select the shard configuration scheme with the largest computational memory access ratio and retain it; Step 5: Calculate the number of kernel function transmissions of each sharding configuration scheme retained in step 4, and find the sharding configuration scheme with the minimum number of kernel function transmissions as the optimal sharding configuration scheme; In the step 2, If general multiply-add instructions are used for matrix multiplication, the number of threads in the M direction and the N direction meets the following conditions: 16≤(block_m_threads,block_n_threads)≤max(block_m_threads) / 16; If the matrix multiplication hardware acceleration unit is used to perform matrix multiplication operations, the number of threads in the M direction and the N direction must meet the following conditions: Wherein, block_m_threads and block_n_threads represent the number of threads in the M direction and the number of threads in the N direction respectively, and TCUmin shape size represents the minimum number of threads in the M direction and the N direction supported by the matrix multiplication acceleration unit TCU; The step 3 specifically comprises the following steps: S1. Obtain the minimum number of thread blocks min_theory_block_num required for each shard configuration scheme to fully occupy the hardware computing resources; S2. For the matrix multiplication between the left matrix of m*k and the right matrix of k*n, input the size of the matrix multiplication (m, k, n) and calculate the number of thread blocks Thread_Blocks under each slicing configuration scheme. Among them, m_tiles and n_tiles represent the number of data multiplexing times in the M direction and the N direction respectively; S3. If Thread_Blocks is not less than min_theory_block_num, the sharding configuration scheme is retained; if the Thread_Blocks of all sharding configuration schemes are less than min_theory_block_num, it means that the size of the input matrix multiplication is too small to fully utilize the computing resources. The data reuse times in all sharding configuration schemes are set to 1 and retained.

2. A matrix multiplication slice configuration selection method as claimed in claim 1, characterized in that: In step 2, the method for preliminarily screening each shard configuration scheme according to data reuse performance is: Exclude sharding configurations with fewer active threads than a set threshold based on register usage; Alternatively, you can first list the data reuse solutions for the sharding configuration solution, and then perform performance tests using matrix multiplication test cases of various sizes to exclude data reuse solutions whose performance is lower than the set threshold in all tests.

3. A matrix multiplication slice configuration selection method as claimed in claim 1, characterized in that: The calculation formula for calculating the memory access ratio TLP described in step 4 is: Among them, the slice shape of the m*k left matrix is ​​(block_m_threads*m_tiles,block_k_threads), and the slice shape of the k*n right matrix is ​​(block_k_threads,block_n_threads*m_tiles), where block_k_threads represents the slice length in the K direction.

4. A matrix multiplication slice configuration selection method as claimed in claim 1, characterized in that: The calculation formula for the kernel function launch times Launch_kernel_times in step 5 is: Where block_per_cu represents the number of thread blocks that can exist in parallel on each computing unit, and maxavailable register num of a thread indicates the maximum number of registers that each thread can use due to chip hardware limitations; num of registers actually used indicates the number of registers actually used by each thread; maxshared memory size of a SM indicates the maximum shared memory size that each stream processor cluster can use; shared memory sizeused by per thread block indicates the shared memory size actually used by each thread block.

5. A matrix multiplication slice configuration selection system, characterized in that: The invention comprises an input module, a compilation module and an implementation module of the matrix multiplication slice configuration selection method described in any one of claims 1 to 4, wherein the input module is used to input the size of the matrix multiplication, and the compilation module is used to use the configuration parameters of each slice configuration scheme as the template parameters of the matrix multiplication kernel function and compile and generate the kernel functions of all slice configuration schemes; during operation, the size of the matrix multiplication is input, and the implementation module of the matrix multiplication slice configuration selection method selects the best slice configuration scheme and the corresponding kernel function to perform the matrix multiplication operation.

Images