A deep neural network acceleration method and system based on GEMM

Abstract

The present invention belongs to the field of GEMM operation acceleration technology, and is a GEMM-based deep neural network acceleration method and system, the method comprising the steps of: firstly judging whether the input matrix multiplication is regular matrix multiplication or irregular matrix multiplication; adopting different slicing methods for different types of matrix multiplication: if it is regular matrix multiplication, traverse the pre-established slicing strategy to select the best strategy; if it is irregular matrix multiplication, first generate the slicing strategy according to the pre-established strategy and then select the strategy; when selecting the slicing strategy, take Kernel Occupancy as the basis; slice the matrix according to the selected slicing strategy; calculate the matrix slices and merge the calculation results. The present invention utilizes two different dynamic slicing methods and Kernel Occupancy through GEMM to improve the adaptability of the slicing size to reduce the number of unnecessary memory loading times while increasing the CU occupancy.

Description

Technical Field

[0001] The present invention relates to the field of GEMM operation acceleration, and in particular to a GEMM-based deep neural network acceleration method and system. Background Art

[0002] GEMM (General Matrix Multiplication) is a widely used linear algebra operation, and its calculation form is: CαAB+βC, where A, B, and C are matrices, and α and β are scalars. As a basic module for high-performance computing, GEMM has a wide range of applications, including traditional fields such as statistics and scientific computing, as well as emerging fields such as deep learning and big data analysis.

[0003] The size and shape of GEMM operations involved in different fields and applications are usually different. For example, in scientific computing, big data analysis and other fields, large-size matrices usually need to be processed, while in deep neural network applications, small and medium-sized matrix multiplications usually appear.

[0004] cuBlas and rocBlas are currently commonly used linear algebra computing libraries and are the core libraries for most applications involving matrix operations. In practical applications, the GEMM operation performance of cuBlas and rocBlas is affected by the size and shape of the input matrix. In previous practices, the two performed well in large-scale regular matrix multiplication, but performed poorly in small and medium-sized matrices and irregular-shaped matrix multiplication (referring to the case where the number of rows and columns of the matrix differs greatly). The main reason is that the calculation scale at this time is far smaller than the computing power of the GPU and excessive interaction with the memory brings unnecessary memory access costs.

[0005] Batch matrix multiplication is a typical approach to optimize small and medium-sized matrix multiplication. For example, in the MAGMA-Batched scheme, a batch of small and medium-sized matrices are collected and input into the batch matrix multiplication kernel for calculation to improve the utilization of GPU computing resources. However, the batch matrix approach requires waiting for the accumulation of matrices, which has a certain delay, and the acceleration effect is not obvious in some scenarios. In addition, in deep neural networks, layers are interconnected, and the current layer uses the output of the previous layer as input, which makes it difficult to apply to batch processing methods. Therefore, in deep neural networks, optimizing single small and medium-sized matrix multiplication still has certain significance.

[0006] Currently, optimization for matrix multiplication usually defaults to optimization for regular matrix multiplication, but these optimization methods usually cannot achieve good performance on irregular matrix multiplication. Irregular matrix multiplication is very common in practical application scenarios, especially in the field of deep learning. The matrices in the GEMM operations involved in deep neural networks usually have small dimensions and irregular characteristics. Taking convolutional neural networks as an example, the convolution operation is usually converted into matrix multiplication for calculation. In the convolutional neural network ResNet, the ImtoCol algorithm is used to convert one of the convolution operations, and the resulting matrix multiplication has one dimension of more than 12,000 and the other dimension is only 64.

[0007] GEMM operations usually include matrix slices and operations. If regular matrix multiplication and irregular matrix multiplication are mixed together and the same slicing strategy is used, the matrix slices may be small and numerous due to incompatible slicing strategies, thereby introducing unnecessary memory loading times and affecting overall efficiency. Summary of the invention

[0008] In order to solve the technical problems existing in the prior art, the present invention provides a deep neural network acceleration method and system based on GEMM, which improves the adaptability of the slice size by adopting two different dynamic slicing methods for regular matrix multiplication and irregular matrix multiplication and combining the Kernel Occupancy concept to reduce the number of unnecessary memory loading times and improve the CU occupancy rate.

[0009] The first object of the present invention is to provide a deep neural network acceleration method based on GEMM.

[0010] The second object of the present invention is to provide a deep neural network acceleration system based on GEMM.

[0011] The first object of the present invention can be achieved by adopting the following technical solutions:

[0012] A deep neural network acceleration method based on GEMM, the method comprising:

[0013] S1. Determine whether the input matrix multiplication is regular matrix multiplication or irregular matrix multiplication;

[0014] S2. If it is a regular matrix multiplication, the pre-defined regular matrix multiplication sharding strategy is traversed, and the best sharding strategy is selected based on Kernel Occupancy. If it is an irregular matrix multiplication, a sharding strategy is generated according to the matrix dimension and the pre-defined irregular matrix multiplication sharding strategy, and the generated sharding strategy is traversed, and the best sharding strategy is selected based on Kernel Occupancy.

[0015] S3. Slice the matrix according to the selected optimal slicing strategy, and calculate all matrix slices to obtain the operation result.

[0016] Preferably, the selecting to generate a sharding strategy according to the matrix dimension and a pre-defined irregular matrix multiplication sharding strategy specifically includes:

[0017] Define the slice size of the small dimension of the matrix to be 2 n , n is a positive integer, when T min ≤L, 2 n Need to meet:

[0018] T min ≤2 n ≤L

[0019] Among them, T min is the minimum shard size of the pre-defined regular matrix multiplication shard strategy, and L is the size of the small dimension of the matrix;

[0020] When T min >L, 2 n Need to meet:

[0021] 0≤2 n ≤L

[0022] Where L is the size of the small dimension of the matrix;

[0023] For matrices that meet any of the above conditions, the slice size of the small dimension is 2 n , select a slice size greater than 2 in the pre-defined irregular matrix multiplication slice strategy n The entry with 2 n Form a new sharding strategy.

[0024] Preferably, the calculation formula of the Kernel Occupancy is:

[0025]

[0026] Where O is the Kernel Occupancy value, WF block , W.F. LDS , W.F. VGPR The maximum number of active wavefronts, WF, and max The maximum number of parallel wavefronts in a single CU. In the AMD GCN5 and CDNA architectures, WF max =40.

[0027] The second object of the present invention can be achieved by adopting the following technical solutions:

[0028] A GEMM-based deep neural network acceleration system, comprising:

[0029] An input judgment module, used to judge whether the input matrix multiplication is a regular matrix multiplication or an irregular matrix multiplication;

[0030] A strategy selection module is used to traverse the pre-defined regular matrix multiplication sharding strategies if it is a regular matrix multiplication, and select the best sharding strategy based on Kernel Occupancy; if it is an irregular matrix multiplication, a sharding strategy is generated according to the matrix dimension and the pre-defined irregular matrix multiplication sharding strategy, and the generated sharding strategies are traversed, and the best sharding strategy is selected based on Kernel Occupancy;

[0031] The slicing operation module is used to slice the matrix according to the selected optimal slicing strategy and calculate all matrix slices to obtain the operation results.

[0032] The selection strategy module generates a sharding strategy according to the matrix dimension and the pre-defined irregular matrix multiplication sharding strategy, specifically including:

[0033] Define the slice size of the small dimension of the matrix to be 2 n , n is a positive integer, when T min ≤L, 2 n Need to meet:

[0034] T min ≤2 n ≤L

[0035] Among them, T min is the minimum shard size of the pre-defined regular matrix multiplication shard strategy, and L is the size of the small dimension of the matrix;

[0036] When T min >L, 2 n Need to meet:

[0037] 0≤2 n ≤L

[0038] Where L is the size of the small dimension of the matrix;

[0039] For matrices that meet any of the above conditions, the slice size of the small dimension is 2 n , select a slice size greater than 2 in the pre-defined irregular matrix multiplication slice strategy n The entry with 2 n Form a new sharding strategy.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] 1. The present invention provides a GEMM-based deep neural network acceleration method, which fully considers the characteristics of matrix multiplication in deep neural networks, accelerates matrix multiplication by improving the deficiencies of previous optimization methods for small and medium matrix multiplication in irregular matrix multiplication, thereby reducing the time used for matrix multiplication in deep neural networks.

[0042] 2. The present invention adopts two different dynamic sharding methods for regular matrix multiplication and irregular matrix multiplication and combines the concept of Kernel Occupancy to improve the adaptability of the sharding size to reduce the number of unnecessary memory loading times and improve the CU occupancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0044] Figure 1 This is a flow chart of a deep neural network acceleration method based on GEMM in Example 1 of the present invention;

[0045] Figure 2 is a flowchart of the GEMM operation accelerator in Example 1 of the present invention;

[0046] Figure 3 This is a structural diagram of a GEMM-based deep neural network acceleration system in Example 2 of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is obvious that the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and the implementation of the present invention is not limited to this. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] Example 1

[0049] like Figure 1As shown, a flowchart of a deep neural network acceleration method based on GEMM, a deep neural network acceleration method based on GEMM, including the following steps: determine whether the input matrix multiplication is regular matrix multiplication or irregular matrix multiplication; if it is regular matrix multiplication, traverse the pre-established regular matrix multiplication sharding strategy, and select the best sharding strategy based on Kernel Occupancy; if it is irregular matrix multiplication, generate a sharding strategy based on the matrix dimension and the pre-established irregular matrix multiplication sharding strategy and traverse the generated sharding strategy, and select the best sharding strategy based on Kernel Occupancy; slice the matrix according to the selected best sharding strategy, and calculate all matrix slices to obtain the operation result. The workflow diagram of the GEMM operation accelerator, as shown Figure 2 shown.

[0050] The GEMM operation accelerator in the embodiment of the present invention is described below in conjunction with the convolutional neural network ResNet.

[0051] ResNet (Residual Neural Network, see: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778) was proposed by four Chinese researchers from Microsoft Research and won the championship in ILSRC 2015. ResNet neural network forms residual learning through a short-circuit mechanism, which solves the degradation problem of deep networks and greatly increases the number of neural network layers. Before this, VGGNet only had 19 layers, while ResNet can train up to 152 layers of neural networks. The increase in the number of neural network layers increases the number of GEMM operations, so using an optimized GEMM operation accelerator can significantly accelerate ResNet.

[0052] The ResNet neural network has five different network structures: 18-layer, 34-layer, 50-layer, 101-layer, and 152-layer. The specific structures are shown in Table 1.

[0053] Table 1

[0054]

[0055] The ImtoCol algorithm is used to convert the convolution operation into a GEMM operation, that is, C = α × A × B + β × C, where A, B, and C are matrices and α and β are scalars. Taking the 18-layer and 34-layer conv2_x layers as an example, the input feature map size is 112 × 112, the number of channels is 64, the convolution kernel size is 3 × 3, and the number of convolution kernels is 64. The ImtoCol algorithm is used to convert the convolution operation into a matrix multiplication, and the number of rows and columns of the two matrices obtained are:

[0056] A row =(112-3+1)*(112-3+1)=12100

[0057] B row =A col =3×3×64=576

[0058] B col =64

[0059] Among them, A row is the number of rows in matrix A, B row is the number of rows in matrix B, A col is the number of columns of matrix A, B col is the number of columns of matrix B, that is, the size of matrix A is 12100×576, and the size of matrix B is 576×64.

[0060] According to the rules of matrix operations: A×B=C, the number of rows of C is equal to the number of rows of A, and the number of columns of C is equal to the number of columns of B. The operation process is: the rows in A are multiplied and added with the corresponding columns in B, and the i-th row of A is multiplied and added with the j-th column of B to obtain the element in the i-th row and j-th column of C. Therefore, there are as many rows as there are rows in A, and there are as many columns as there are columns in B. row =12100, B col =64, we can get the size of matrix C to be 12100×64.

[0061] In a preferred embodiment, the ratio k of the number of rows and columns of the input matrix multiplication is calculated. If 0.5<k<2, it is a regular matrix multiplication, otherwise it is an irregular matrix multiplication. The GEMM operation accelerator receives the matrix input of each layer of the ResNet neural network, and determines whether the input matrix multiplication is a regular matrix multiplication or an irregular matrix multiplication by calculating the ratio k of the number of rows and columns of the input matrix multiplication. Different slicing methods are used for different types of matrix multiplication. The calculation formula of the ratio k is:

[0062]

[0063] Among them, N row is the number of matrix rows, N col is the number of matrix columns.

[0064] Pre-defined sharding strategies are formulated for regular matrix multiplication and irregular matrix multiplication according to GPU related parameters. The GPU related parameters include the maximum number of work items that a single block can contain, the maximum LDS capacity in the CU, the number of VGPRs in the SIMD, etc. These parameters are used to pre-determine the sharding strategy and calculate the Kernel Occupancy.

[0065] In a preferred embodiment, the pre-formulated sharding strategy for a regular matrix is ​​shown in Table 2, and the pre-formulated sharding strategy for an irregular matrix is ​​shown in Table 3.

[0066] Table 2

[0067] strategies <![CDATA[T M ]]> <![CDATA[T N ]]> <![CDATA[T K ]]> work items / block small 16 16 8 128 medium 32 32 8 128 large 64 64 8 256

[0068] In Table 2: T M , T N , T k Refers to the number of rows of matrix slices A and C, the number of columns of matrix slices B and C, the number of columns of matrix slice A, and the number of rows of matrix slice B in the subsequent GEMM operation (i.e., in the form of C = α × A × B + β × C, A, B, C are the matrix slices after slicing, and α and β are scalars). work items / block refers to the number of work items in each block.

[0069] Table 3

[0070] strategies T <![CDATA[T K ]]> work items / block level_1 32 8 128 level_2 64 8 128 level_3 128 8 128

[0071] In Table 3: T refers to the larger dimensional slice size in the subsequent GEMM operation (i.e., C=α×A×B+β×C, A, B, C are matrix slices after slices, α and β are scalars), taking the original matrix C with more rows than columns as an example, T and Tk refer to the number of rows of the matrix slice C and the number of rows of the matrix slice A, the number of columns of the matrix slice A and the number of rows of the matrix slice B after slices.

[0072] For regular matrix multiplication, the process of selecting a sharding strategy includes the steps of traversing pre-defined regular matrix multiplication sharding strategies, calculating Kernel Occupancy, and selecting a sharding strategy corresponding to a maximum Kernel Occupancy.

[0073] For irregular matrix multiplication, the process of selecting a sharding strategy includes: generating a sharding strategy according to the matrix dimension and a pre-defined irregular matrix multiplication sharding strategy, traversing the generated sharding strategies, calculating the kernel occupancy, and selecting the sharding strategy corresponding to the maximum kernel occupancy.

[0074] Specifically, for regular matrix multiplication and irregular matrix multiplication, the pre-defined sharding strategy and the previously generated sharding strategy are traversed respectively. During the traversal process, the Kernel Occupancy value corresponding to each sharding strategy is calculated and compared. After the traversal is completed, the sharding strategy with the maximum Kernel Occupancy is selected as the sharding strategy for the current GEMM operation. For irregular matrix multiplication, the sharding strategy needs to be generated first.

[0075] For irregular matrix multiplication, the process of generating a sharding strategy includes:

[0076] First, define the slice size of the small dimension of the matrix as 2 n , n is a positive integer;

[0077] When T min ≤L, 2 n Need to meet:

[0078] T min ≤2 n ≤L

[0079] Among them, T min is the minimum shard size of the pre-defined regular matrix multiplication shard strategy, and L is the size of the small dimension of the matrix;

[0080] When T min >L, 2 n Need to meet:

[0081] 0≤2 n ≤L

[0082] Where L is the size of the small dimension of the matrix;

[0083] For matrices that meet any of the above conditions, the slice size of the small dimension is 2 n , select a slice size greater than 2 in the pre-defined irregular matrix multiplication slice strategy n The entry with 2 n Form a new sharding strategy.

[0084] For example, the size of the matrix C in the input GEMM operation is 8092×32, and the number of columns is less than the number of rows. When n=4, 5, 16≤2 is satisfied. n ≤32, that is, the column sharding size can be 16, 32. Based on the sharding strategy in Table 3, the generated sharding strategy is shown in Table 4. If only the sharding strategy of regular matrix multiplication is used to slice the matrix, due to the limitation of the number of columns, the sharding size can only be 32×32 or 16×16, which will increase the number of unnecessary memory loads.

[0085] Table 4

[0086] <![CDATA[T M ]]> <![CDATA[T N ]]> <![CDATA[T K ]]> work items / block 32 16 8 128 64 16 8 128 128 16 8 128 64 32 8 128 128 32 8 128

[0087] Kernel Occupancy indicates the resource usage of CU in GPU. It is the ratio of the number of active wavefronts to the maximum number of wavefronts supported by a single CU. The calculation formula of Kernel Occupancy is:

[0088]

[0089] Among them, O is the Kernel Occupancy value, WF block 、WF LDS 、WF VGPR The maximum number of active wavefronts, WF, and max The maximum number of parallel wavefronts in a single CU. In AMD's GCN5 and CDNA architectures, WF max =40.

[0090] The calculation of Kernel Occupancy includes the following steps:

[0091] (1) Calculate WF based on the number of blocks block

[0092] In the AMDGPU architecture, if the number of wavefronts contained in each block is greater than 1, the number of blocks that each CU can accommodate cannot exceed 16. At this time, the maximum number of active wavefronts WF block for:

[0093] WF block =N block ×N wf

[0094] Among them, N block is the maximum number of blocks in a single CU, N wf is the number of wavefronts in a single block;

[0095] (2) Calculate WF based on LDS usage LDS

[0096] In GPU, LDS is a very important and limited resource. All threads in the same block share LSD resources and different blocks cannot access each other's LDS resources. Therefore, considering LDS usage, the maximum number of active wavefronts WF is calculated. LDS :

[0097]

[0098] Among them, LDS max The maximum LDS capacity in each CU, LDS used LDS usage per block.

[0099] (3) Calculate WF based on VGPR usage VGPR

[0100] VGPR is also a very limited resource in GPU. For example, in GCN architecture, each SIMD has 256 VGPRs, and in CDNA architecture, each SIMD has 512 VGPRs. Therefore, the maximum number of active wavefronts WF is calculated considering VGPR usage. VGPR :

[0101]

[0102]

[0103] Among them, WF′ VGPR Indicates the number of wavefronts calculated only from the perspective of VGPR usage, N wf is the number of wavefronts in a single block, VGPR used VGPRPerSIMD is the number of VGPRs in each SIMD, and SIMDPerCU is the number of SIMDs in each CU.

[0104] The matrix is ​​sliced ​​according to the selected slicing strategy, and the operation results are obtained by calculating all matrix slices. After determining the matrix block size, the input matrix C is sliced. If the rows and columns of the slices are T M , T N Since the blocks of matrix C are calculated from the corresponding rows and columns of matrix A and matrix B, the number of rows of matrix A is TM, and the number of columns of matrix B is T. N Since the LDS in the CU is limited, it is impossible to load all the data required for calculation from the system memory at one time. Therefore, the row data of matrix A and the column data of matrix B need to be further divided each time the calculation is performed. The slice sizes of the divided matrices A and B are T M ×T K , T K ×T N .

[0105] The work items in the block are responsible for calculating some elements in the matrix slice. When performing specific calculations, preferably, each block is responsible for calculating a matrix slice, and the sub-slices are divided according to the number of work items in the block, and each workitem is responsible for calculating a sub-slice. After the work item completes the calculation of the sub-slice, the result is written back to the LDS. After all sub-slices are calculated, the calculation results are obtained and written back to the system memory.

[0106] In summary, the GEMM-based deep neural network acceleration method of this embodiment fully considers the characteristics of matrix multiplication involved in deep neural networks, improves the shortcomings of previous optimization methods for small and medium-sized matrix multiplication in irregular matrix multiplication, and adopts two different dynamic sharding methods for regular matrix multiplication and irregular matrix multiplication. At the same time, it combines the concept of Kernel Occupancy to improve the adaptability of the sharding size to reduce unnecessary memory loading times while increasing the CU occupancy.

[0107] Example 2

[0108] like Figure 3 As shown, a structure diagram of a deep neural network acceleration system based on GEMM is provided. This embodiment provides a deep neural network acceleration system based on GEMM, which includes an input judgment module, a selection strategy module and a sharding operation module. The specific functions of each module are as follows:

[0109] An input judgment module, used to judge whether the input matrix multiplication is a regular matrix multiplication or an irregular matrix multiplication;

[0110] A strategy selection module is used to traverse the pre-defined regular matrix multiplication sharding strategies if it is a regular matrix multiplication, and select the best sharding strategy based on Kernel Occupancy; if it is an irregular matrix multiplication, a sharding strategy is generated according to the matrix dimension and the pre-defined irregular matrix multiplication sharding strategy, and the generated sharding strategies are traversed, and the best sharding strategy is selected based on Kernel Occupancy;

[0111] The slicing operation module slices the matrix according to the selected optimal slicing strategy and calculates all matrix slices to obtain the operation results.

[0112] The selection strategy module generates a sharding strategy according to the matrix dimension and the pre-defined irregular matrix multiplication sharding strategy, specifically including:

[0113] First, define the slice size of the small dimension of the matrix as 2 n , n is a positive integer;

[0114] When T min ≤L, 2 nNeed to meet:

[0115] T min ≤2 n ≤L

[0116] Among them, T min is the minimum shard size of the pre-defined regular matrix multiplication shard strategy, and L is the size of the small dimension of the matrix;

[0117] When T min >L, 2 n Need to meet:

[0118] 0≤2 n ≤L

[0119] Where L is the size of the small dimension of the matrix;

[0120] For matrices that meet any of the above conditions, the slice size of the small dimension is 2 n , select a slice size greater than 2 in the pre-defined irregular matrix multiplication slice strategy n The entry with 2 n Form a new sharding strategy.

[0121] The calculation formula of Kernel Occupancy is:

[0122]

[0123] Among them, O is the Kernel Occupancy value, WF block 、WF LDS 、WF VGPR The maximum number of active wavefronts, WF, and max The maximum number of parallel wavefronts in a single CU. In AMD's GCN5 and CDNA architectures, WF max =40.

[0124] The calculation of Kernel Occupancy includes the following steps:

[0125] (1) Calculate WF based on the number of blocks block

[0126] In the AMDGPU architecture, if the number of wavefronts contained in each block is greater than 1, the number of blocks that each CU can accommodate cannot exceed 16. At this time, the maximum number of active wavefronts WF block for:

[0127] WF block =N block ×Nwf

[0128] Among them, N block is the maximum number of blocks in a single CU, N wf is the number of wavefronts in a single block;

[0129] (2) Calculate WF based on LDS usage LDS

[0130] In GPU, LDS is a very important and limited resource. All threads in the same block share LSD resources and different blocks cannot access each other's LDS resources. Therefore, considering LDS usage, the maximum number of active wavefronts WF is calculated. LDS :

[0131]

[0132] Among them, LDS max The maximum LDS capacity in each CU, LDS used LDS usage per block.

[0133] (3) Calculate WF based on VGPR usage VGPR

[0134] VGPR is also a very limited resource in GPU. For example, in GCN architecture, each SIMD has 256 VGPRs, and in CDNA architecture, each SIMD has 512 VGPRs. Therefore, the maximum number of active wavefronts WF is calculated considering VGPR usage. VGPR :

[0135]

[0136]

[0137] Among them, WF′ VGPR Indicates the number of wavefronts calculated only from the perspective of VGPR usage, N wf is the number of wavefronts in a single block, VGPR used VGPRPerSIMD is the number of VGPRs in each SIMD, and SIMDPerCU is the number of SIMDs in each CU.

[0138] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A GEMM-based deep neural network acceleration method, characterized in that: It includes the following steps: S1. Determine whether the input matrix multiplication is regular matrix multiplication or irregular matrix multiplication; The specific content of step S1 includes: calculating the ratio k of the number of rows to the number of columns of the input matrix multiplication. If 0.5 < k < 2, it is regular matrix multiplication; otherwise, it is irregular matrix multiplication. The calculation formula for the ratio k is: Among them, N row is the number of matrix rows, N col is the number of matrix columns; S2. If it is regular matrix multiplication, traverse the pre-defined regular matrix multiplication sharding strategy and select the best sharding strategy based on Kernel Occupancy; if it is irregular matrix multiplication, generate a sharding strategy according to the matrix dimensions and the pre-defined irregular matrix multiplication sharding strategy, traverse the generated sharding strategies, and select the best sharding strategy based on Kernel Occupancy; The calculation formula for the Kernel Occupancy is: Among them, O is the Kernel Occupancy value, WF block , W.F. LDS , W.F. VGPR The maximum number of active wavefronts, WF, and max The maximum number of parallel wavefronts in a single CU. In the AMD GCN5 and CDNA architectures, WF max =40; The specific calculation of the Kernel Occupancy includes the following steps: Calculate WF based on the number of blocks block , W.F. block The calculation formula is: WF block =N block ×N wf Among them, N block N is the maximum number of blocks in a single CU. When in the AMDGPU architecture, if the number of wavefronts contained in each block is greater than 1, the number of blocks contained in each CU does not exceed 16. wf is the number of wavefronts in a single block; Calculate WF based on LDS usage LDS , W.F. LDS The calculation formula is: Among them, LDS max The maximum LDS capacity in each CU, LDS used The LDS usage for each block, N wf is the number of wavefronts in a single block; Calculate WF based on VGPR usage VGPR , W.F. VGPR The calculation formula is: Among them, WF′ VGPR is the number of wavefronts calculated only from the perspective of VGPR usage, N wf is the number of wavefronts in a single block, VGPR used VGPRPerSIMD is the number of VGPRs used for each wavefront, VGPRPerSIMD is the number of VGPRs in each SIMD, and SIMDPerCU is the number of SIMDs in each CU; S3. Shard the matrix according to the selected best sharding strategy, and calculate all matrix shards to obtain the operation result.

2. A GEMM-based deep neural network acceleration method according to claim 1, characterized in that: The pre-defined regular matrix multiplication sharding strategy and the pre-defined irregular matrix multiplication sharding strategy are both formulated according to the GPU architecture and related parameters. The related parameters of the GPU include the maximum number of work items contained in a single block, the maximum LDS capacity in the CU, and the number of VGPRs in the SIMD.

3. The GEMM-based deep neural network acceleration method according to claim 1, characterized in that: The specific content of step S2 includes: for regular matrix multiplication, traverse the pre-defined regular matrix multiplication sharding strategy, calculate Kernel Occupancy, and select the sharding strategy corresponding to the maximum Kernel Occupancy; for irregular matrix multiplication, generate a sharding strategy according to the matrix dimensions and the pre-defined irregular matrix multiplication sharding strategy, traverse the generated sharding strategies, calculate Kernel Occupancy, and select the sharding strategy corresponding to the maximum Kernel Occupancy.

4. The GEMM-based deep neural network acceleration method according to claim 3, characterized in that: The method for generating a sharding strategy according to the matrix dimensions and the pre-defined irregular matrix multiplication sharding strategy includes: Define the slice size of the small dimension of the matrix to be 2 n , n is a positive integer, when T min ≤L, 2 n Need to meet: T min ≤2 n ≤L Among them, T min is the minimum shard size of the pre-defined regular matrix multiplication shard strategy, and L is the size of the small dimension of the matrix; When T min >L, 2 n Need to meet: 0≤2 n ≤L where L is the size of the smaller dimension of the matrix; For matrices that meet any of the above conditions, the slice size of the small dimension is 2 n , select a slice size greater than 2 in the pre-defined irregular matrix multiplication slice strategy n The entry with 2 n Form a new sharding strategy.

5. The GEMM-based deep neural network acceleration method according to claim 1, characterized in that: When performing calculations on all matrix shards in step S3, each block is responsible for calculating one matrix shard, and the matrix shard is divided according to the number of work items in the block. Each work item is responsible for calculating one sub-shard.

6. A GEMM-based deep neural network acceleration system, characterized in that: It includes: An input judgment module, used to judge whether the input matrix multiplication is regular matrix multiplication or irregular matrix multiplication; Calculate the ratio k of the number of rows to the number of columns of the input matrix multiplication. If 0.5 < k < 2, it is regular matrix multiplication; otherwise, it is irregular matrix multiplication. The calculation formula for the ratio k is: Among them, N row is the number of matrix rows, N col is the number of matrix columns; A strategy selection module, used to, if it is regular matrix multiplication, traverse the pre-defined regular matrix multiplication sharding strategy and select the best sharding strategy based on Kernel Occupancy; if it is irregular matrix multiplication, generate a sharding strategy according to the matrix dimensions and the pre-defined irregular matrix multiplication sharding strategy, traverse the generated sharding strategies, and select the best sharding strategy based on Kernel Occupancy; The calculation formula of the Kernel Occupancy is: Among them, O is the Kernel Occupancy value, WF block , W.F. LDS , W.F. VGPR The maximum number of active wavefronts, WF, and max The maximum number of parallel wavefronts in a single CU. In the AMD GCN5 and CDNA architectures, WF max =40; The specific calculation of Kernel Occupancy includes the following steps: Calculate WF based on the number of blocks block , W.F. block The calculation formula is: WF block =N block ×N wf Among them, N block N is the maximum number of blocks in a single CU. When in the AMDGPU architecture, if the number of wavefronts contained in each block is greater than 1, the number of blocks contained in each CU does not exceed 16. wf is the number of wavefronts in a single block; Calculate WF based on LDS usage LDS , W.F. LDS The calculation formula is: Among them, LDS max The maximum LDS capacity in each CU, LDS used The LDS usage for each block, N wf is the number of wavefronts in a single block; Calculate WF based on VGPR usage VGPR , W.F. VGPR The calculation formula is: Among them, WF′ VGPR is the number of wavefronts calculated only from the perspective of VGPR usage, N wf is the number of wavefronts in a single block, VGPR used VGPRPerSIMD is the number of VGPRs used for each wavefront, VGPRPerSIMD is the number of VGPRs in each SIMD, and SIMDPerCU is the number of SIMDs in each CU; The slicing operation module is used to slice the matrix according to the selected optimal slicing strategy and calculate all matrix slices to obtain the operation results.

7. A GEMM-based deep neural network acceleration system according to claim 6, characterized in that: The selection strategy module generates a sharding strategy according to the matrix dimension and the pre-defined irregular matrix multiplication sharding strategy, specifically including: Define the slice size of the small dimension of the matrix to be 2 n , n is a positive integer, when T min ≤L, 2 n Need to meet: T min ≤2 n ≤L Among them, T min is the minimum shard size of the pre-defined regular matrix multiplication shard strategy, and L is the size of the small dimension of the matrix; When T min >L, 2 n Need to meet: 0≤2 n ≤L Where L is the size of the small dimension of the matrix; For matrices that meet any of the above conditions, the slice size of the small dimension is 2 n , select a slice size greater than 2 in the pre-defined irregular matrix multiplication slice strategy n The entry with 2 n Form a new sharding strategy.

Images