Method and computing device for implementing matrix multiplication operation

Abstract

The present disclosure provides a method and computing device for implementing matrix multiplication operations. The method includes: converting a first matrix of the matrix multiplication operation into a first two-dimensional matrix and loading it into a first buffer for the first matrix in a tensor core, wherein the first matrix is ​​a three-dimensional matrix; determining whether a second matrix of the matrix multiplication operation is a two-dimensional matrix or a three-dimensional matrix; in response to the second matrix being a two-dimensional matrix, loading the second matrix into a second buffer for the second matrix in the tensor core; in response to the second matrix being a three-dimensional matrix, converting the second matrix into a second two-dimensional matrix and loading it into a second buffer for the second matrix in the tensor core; and performing matrix multiplication on the first two-dimensional matrix and the second matrix or the second two-dimensional matrix in the tensor core.

Description

Technical Field

[0001] The present disclosure generally relates to the field of processors, and more particularly, to a method and a computing device for implementing a matrix multiplication operation. Background Art

[0002] In modern science and technology such as weather forecasting, oil exploration, and nuclear physics, a large number of simulation calculations are dependent on computers. The core of simulation calculations is the matrix calculation that represents state transitions. Computer graphics processing and deep learning also involve a large number of matrix multiplication operations.

[0003] When performing a matrix multiplication operation of two matrices, due to hardware requirements (such as GEMM (GEneral Matrix to Matrix Multiplication) units or Tensor Cores), it is necessary to partition the input matrices into blocks and perform the matrix multiplication operation based on the block matrices.

[0004] However, when the remainder value of the corresponding dimension of the input matrix (such as the batch dimension) is small relative to the hardware multiplication granularity, the hardware usage efficiency is low. Summary of the invention

[0005] In view of the above problems, the present disclosure provides a method for performing matrix multiplication by performing dimension folding for input matrices of different shapes.

[0006] According to one aspect of the present disclosure, a method for implementing a convolution operation is provided. The method includes: converting a first matrix of the matrix multiplication operation into a first two-dimensional matrix and loading it into a first buffer for the first matrix in a tensor core, wherein the first matrix is ​​a three-dimensional matrix; determining whether the second matrix of the matrix multiplication operation is a two-dimensional matrix or a three-dimensional matrix; in response to the second matrix being a two-dimensional matrix, loading the second matrix into a second buffer for the second matrix in the tensor core; in response to the second matrix being a three-dimensional matrix, converting the second matrix into a second two-dimensional matrix and loading it into a second buffer for the second matrix in the tensor core; and performing matrix multiplication on the first two-dimensional matrix and the second matrix or the second two-dimensional matrix in the tensor core.

[0007] In some implementations, converting the first matrix of the matrix multiplication operation into a first two-dimensional matrix and loading it into a first buffer for the first matrix in the tensor core includes: loading the first matrix from a first memory outside the tensor core to a vector core through a multiple data loading operation; determining the first dimension and the second dimension of the first two-dimensional matrix in the vector core based on the second dimension and the third dimension of the first matrix; storing the first two-dimensional matrix from the vector core to a second memory near the tensor core through a multiple data storage operation; and loading the first two-dimensional matrix into the first buffer based on the size of the first buffer.

[0008] In some implementations, determining the first dimension and the second dimension of the first two-dimensional matrix based on the second dimension and the third dimension of the first matrix in the vector core includes: determining whether the dimension in the second dimension and the third dimension of the first matrix that needs to be merged with the first dimension of the first matrix meets a predetermined condition; in response to determining that the dimension that needs to be merged with the first dimension of the first matrix meets the predetermined condition, determining the remainder value of the dimension that needs to be merged with the first dimension of the first matrix modulo the hardware multiplication granularity of the tensor core; determining whether the remainder value is greater than or equal to a predetermined threshold, wherein the predetermined threshold is associated with the hardware multiplication granularity of the tensor core; in response to determining that the remainder value is greater than or equal to the predetermined threshold, dividing the first matrix into blocks according to the first dimension of the first matrix and the dimension that needs to be merged with the first dimension of the first matrix, and reading the first matrix as the first two-dimensional matrix according to the second dimension and the other dimension of the third dimension of the first matrix; and in response to determining that the remainder value is less than the predetermined threshold, splicing the first matrix along the first dimension of the first matrix to form the first two-dimensional matrix.

[0009] In some implementations, converting the second matrix into a second two-dimensional matrix and loading it into a second cache for the second matrix within the tensor core includes: loading the second matrix from a first memory outside the tensor core to a vector core through a multiple data loading operation; determining the first dimension and the second dimension of the second two-dimensional matrix in the vector core based on the first dimension and the second dimension of the first two-dimensional matrix; storing the second two-dimensional matrix from the vector core to a second memory near the tensor core through a multiple data storage operation; and loading the second two-dimensional matrix into the second cache based on the size of the second cache.

[0010] In some implementations, determining the first dimension and the second dimension of the second two-dimensional matrix in the vector kernel based on the first dimension and the second dimension of the first two-dimensional matrix includes: determining whether the second dimension of the first matrix is ​​merged with the first dimension of the first matrix based on the first dimension of the first two-dimensional matrix; if it is determined that the second dimension of the first matrix is ​​merged with the first dimension of the first matrix, blocking the first matrix by the first dimension and the third dimension of the second matrix, and reading the second matrix as the second two-dimensional matrix according to the second dimension of the second matrix; and if it is determined that the second dimension of the first matrix is ​​not merged with the first dimension of the first matrix, blocking the second matrix by the first dimension and the second dimension of the second matrix, and reading the second matrix as the second two-dimensional matrix according to the third dimension of the second matrix.

[0011] According to another aspect of the present invention, a computing device is provided. The computing device includes: a tensor core, which includes a general matrix multiplication unit, a first cache for a first matrix for matrix multiplication operation, and a second cache for a second matrix for matrix multiplication operation, wherein the first matrix is ​​a three-dimensional matrix; a first memory located outside the tensor core, which is configured to store the first matrix and the second matrix before the matrix multiplication operation; a vector core, which is configured to convert the first matrix and the second matrix into a first two-dimensional matrix and a second two-dimensional matrix; and a second memory located near the tensor core, which is configured to store the first two-dimensional matrix and the second two-dimensional matrix, wherein the tensor core is configured to: load the first two-dimensional matrix into the first cache, when the second matrix is ​​a two-dimensional matrix, load the second matrix into the second cache, or when the second matrix is ​​a three-dimensional matrix, load the second two-dimensional matrix into the second cache, and perform matrix multiplication on the first two-dimensional matrix and the second matrix or the second two-dimensional matrix.

[0012] In some implementations, the vector core is configured to: load the first matrix from the first memory through a multiple data load operation; determine the first dimension and the second dimension of the first two-dimensional matrix based on the second dimension and the third dimension of the first matrix; and store the first two-dimensional matrix to the second memory through a multiple data storage operation.

[0013] In some implementations, the vector core is configured to: determine whether the dimension in the second dimension and the third dimension of the first matrix that needs to be merged with the first dimension of the first matrix meets a predetermined condition; in response to determining that the dimension that needs to be merged with the first dimension of the first matrix meets the predetermined condition, determine the remainder value of the dimension that needs to be merged with the first dimension of the first matrix modulo the hardware multiplication granularity of the tensor core; determine whether the remainder value is greater than or equal to a predetermined threshold, wherein the predetermined threshold is associated with the hardware multiplication granularity of the tensor core; in response to determining that the remainder value is greater than or equal to the predetermined threshold, block the first matrix according to the first dimension of the first matrix and the dimension that needs to be merged with the first dimension of the first matrix, and read the first matrix as the first two-dimensional matrix according to the second dimension and the other dimension of the third dimension of the first matrix; and in response to determining that the remainder value is less than the predetermined threshold, splice the first matrix along the first dimension of the first matrix to form the first two-dimensional matrix.

[0014] In some implementations, the vector core is configured to: load the second matrix from the first memory through a multiple data loading operation; determine the first dimension and the second dimension of the second two-dimensional matrix based on the first dimension and the second dimension of the first two-dimensional matrix; and store the second two-dimensional matrix in the second memory through a multiple data storage operation.

[0015] In some implementations, the vector core is configured to: determine whether the second dimension of the first matrix is ​​merged with the first dimension of the first matrix based on the first dimension of the first two-dimensional matrix; if it is determined that the second dimension of the first matrix is ​​merged with the first dimension of the first matrix, block the first matrix by the first dimension and the third dimension of the second matrix, and read the second matrix as the second two-dimensional matrix according to the second dimension of the second matrix; and if it is determined that the second dimension of the first matrix is ​​not merged with the first dimension of the first matrix, block the second matrix by the first dimension and the second dimension of the second matrix, and read the second matrix as the second two-dimensional matrix according to the third dimension of the second matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure will be better understood and other objects, details, features and advantages of the present disclosure will become more apparent through the description of specific embodiments of the present disclosure given with reference to the following drawings.

[0017] Figure 1A and Figure 1B Schematic diagram showing matrix multiplication operations for two different cases.

[0018] Figure 2 A schematic diagram of a computing device for performing a matrix multiplication operation according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic flow chart of a method for implementing a matrix multiplication operation according to an embodiment of the present invention is shown.

[0020] Figure 4 A further detailed flow chart of the process of converting and loading the first matrix according to some embodiments of the present invention is shown.

[0021] Figure 5 A further detailed schematic diagram of a process of determining a first dimension and a second dimension of a first two-dimensional matrix according to some embodiments of the present invention is shown.

[0022] Figure 6 A further detailed flow chart of the process of converting and loading the second matrix according to some embodiments of the present invention is shown.

[0023] Figure 7 A further detailed schematic diagram of a process of determining a first dimension and a second dimension of a second two-dimensional matrix according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0024] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "one embodiment" and "some embodiments" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects.

[0026] Figure 1A and Figure 1B Schematic diagram showing matrix multiplication operations for two different cases.

[0027] exist Figure 1AIn the example shown, of the two input matrices of the matrix multiplication operation, the first matrix A is a three-dimensional matrix (Batch, M, K), and the second matrix B is a two-dimensional matrix (K, N), and the output matrix C after the matrix multiplication operation is a three-dimensional matrix (Batch, M, N). Among them, the last dimension (third dimension K) of the first matrix A is the same as the first dimension K of the second matrix B (that is, the last dimension of the former is the same as the first dimension of the latter) so that the matrix multiplication operation is feasible.

[0028] exist Figure 1B In the example shown, of the two input matrices of the matrix multiplication operation, the first matrix A is a three-dimensional matrix (Batch, M, K), and the second matrix B is a three-dimensional matrix (Batch, K, N). After performing the matrix multiplication operation, the output matrix C is a two-dimensional matrix (M, N). Here, out of the habit of practical application, the last dimension (third dimension K) of the first matrix A is expressed as being the same as the second dimension K of the second matrix B. However, those skilled in the art can understand that such a statement is only for the convenience of description. In fact, in the actual matrix multiplication operation, the second matrix B can be transposed so that the matrix multiplication operation between the first matrix A and the transposed second matrix B is feasible, that is, the last dimension of the former is the same as the first dimension of the latter. The description of this transposition operation may be omitted in this article.

[0029] In addition, more generally, at least one dimension of the two matrices performing the matrix multiplication operation can be the same, and the specific order of the dimensions is not required. In this case, before performing the matrix multiplication operation, at least one of the two original matrices can be subjected to matrix transposition so that the last dimension of the first matrix A obtained after the transposition is the same as the first dimension of the second matrix B, which will not be repeated here.

[0030] There are two main ways to perform the above matrix multiplication operation. One is to loop the batch dimension in the outer layer and perform MxNxK matrix operations in the inner layer. For example, for Figure 1A In the case of batch, the first matrix A can be sliced ​​into batch two-dimensional matrices (M, K) according to the batch dimension, and then each two-dimensional matrix (M, K) is multiplied with the second matrix B (K, N) to obtain batch two-dimensional matrices (M, N). These batch two-dimensional matrices (M, N) are stacked into the output three-dimensional matrix C (Batch, M, N). Figure 1BIn the case of batches, the first matrix A and the second matrix B can be sliced ​​into batches of two-dimensional matrices (M, K) and (K, N) according to the batch dimension, and then the two-dimensional matrices (M, K) and (K, N) of each batch are matrix multiplied to obtain batches of two-dimensional matrices (M, N). These batches of two-dimensional matrices (M, N) are superimposed to generate the output two-dimensional matrix (M, N). However, in this method, the dimensions (M, K, and N) of the two-dimensional matrix do not change, and when the remainder value of the modulus relative to the hardware multiplication granularity is small, the hardware utilization efficiency is still low.

[0031] Another method is to reshape the input three-dimensional matrix into a two-dimensional matrix by software before performing the matrix multiplication operation on the first matrix A and the second matrix B, and then perform the matrix multiplication operation on the converted two-dimensional matrix in hardware. Figure 1A In this case, the first matrix A can be reshaped into a two-dimensional matrix (Batch*M,K), and then the two-dimensional matrix (Batch*M,K) and the second matrix B(K,N) are matrix multiplied to obtain the output two-dimensional matrix (Batch*M,N). The output two-dimensional matrix (Batch*M,N) can be reshaped into the output three-dimensional matrix C(Batch,M,N) as needed. In this way, the input matrix is ​​converted from a three-dimensional matrix to a two-dimensional matrix through the reshaping operation, so that the case where a certain dimension of the input matrix is ​​small relative to the modulus of the hardware multiplication granularity is greatly reduced (at most one), and the hardware utilization efficiency is improved. However, in this method, the reshaping operation performed by the software will bring more memory access operations, and the total consumption time will increase, and the overall performance will deteriorate.

[0032] In the above two matrix multiplication operations, if M and K are not aligned with the hardware multiplication granularity, the batch dimension information cannot be fully utilized. In particular, when the remainder value of M or K modulo the hardware multiplication granularity is a small number such as 2, 4, 8, etc., the utilization rate of the matrix multiplication hardware and the utilization rate of the cache will be very low, resulting in poor overall performance. If the reshaping process is inserted, a large memory access will be generated, making the overall performance worse.

[0033] In view of the above problems, the present invention proposes a method for optimizing matrix multiplication operations by folding the dimensions of two input matrices of the matrix multiplication operations, which can fully utilize the processing power of matrix multiplication hardware and improve the utilization rate of the cache.

[0034] Figure 2 FIG. 2 is a schematic diagram of a computing device 200 for performing a matrix multiplication operation according to an embodiment of the present invention. Figure 2As shown in , the computing device 200 may include a tensor core 210 for performing matrix multiplication operations, and the tensor core 210 has a given hardware multiplication granularity, which limits the maximum dimension of the input matrix that the tensor core 210 can process (ie, the maximum value of M or K).

[0035] The computing device 200 may further include a first memory 220 outside the tensor core 210. The first memory 220 is used to store the first matrix A and the second matrix B before the tensor core 210 performs the matrix multiplication operation, and to store the output matrix C after the matrix multiplication operation is completed. The first memory 220 may be, for example, a high bandwidth memory (HBM), which has the advantages of high throughput and high bandwidth, and can be widely used in fields with high requirements for large data volume access, such as artificial intelligence (AI) computing, etc.

[0036] The computing device 200 may further include a vector core 230. The vector core 230 is used to convert a three-dimensional input matrix into a two-dimensional matrix.

[0037] The computing device 200 may further include a second memory 240 near the tensor core 210. The second memory 240 may be composed of a plurality of registers, and is used to store a two-dimensional matrix generated by the vector core 230 converting the three-dimensional input matrix. The second memory 240, as the main memory of the tensor core 210, is usually directly connected to the tensor core 210 without any other intermediate conversion, so that the data required for the calculation of the tensor core 210 is stored as close to the tensor core 210 as possible.

[0038] In some embodiments, the tensor core 210, the vector core 230, and the second memory 240 may be part of an independent stream processing unit 250. Figure 2 Only one stream processing unit 250 is shown in the figure, but those skilled in the art will appreciate that the computing device 200 may include multiple such stream processing units 250. In this case, the first memory 220 may serve multiple stream processing units 250, and may exchange data and instructions with each stream processing unit 250 of the computing device 200 through a high-speed bus or an on-chip network.

[0039] In some embodiments, the tensor core 210 may further include a general matrix multiplication unit (GEMM) 212, a first buffer (A_buffer) 214 for the first matrix A, and a second buffer (B_buffer) 216 for the second matrix B. The first buffer 214 and the second buffer 216 may be memories dedicated to the tensor core 210 within the tensor core 210, wherein the first buffer 214 is used to cache one matrix of the first matrix A at a time according to the hardware multiplication granularity of the GEMM 212, and the second buffer 216 is used to cache one vector of the second matrix B at a time according to the hardware multiplication granularity of the GEMM 212. That is, the sizes of the first buffer 214 and the second buffer 216 are determined by the hardware multiplication granularity of the GEMM 212 or the tensor core 210. For example, in the case where the GEMM 212 is a two-dimensional accumulator array of nxn, the first buffer 214 and the second buffer 216 cache one vector of length N*n of the first matrix A and the second matrix B, respectively, at a time. The GEMM 212 is configured to perform a matrix multiplication operation on the vectors input from the first buffer 214 and the second buffer 216 each time when triggered by a matrix multiplication instruction received by the tensor core 210 .

[0040] Furthermore, in some embodiments, in the tensor core 210 , the two-dimensional matrix from the second memory 240 may be transposed before the first buffer 214 and the second buffer 216 (not shown in the figure).

[0041] Figure 3 2 shows a schematic flow chart of a method 300 for implementing a matrix multiplication operation according to an embodiment of the present invention. The method 300 may be executed by or in the computing device 200. Here, it is assumed that the first matrix A and the second matrix B of the matrix multiplication operation have been pre-stored in the first memory 220, and the first matrix A is as follows: Figure 1A and Figure 1B The three-dimensional matrix (Batch, M, K) shown in FIG. 1 can be the second matrix B as follows: Figure 1A The two-dimensional matrix (K, N) shown in Figure 1B The three-dimensional matrix (Batch, K, N) is shown. Depending on whether the second matrix B is a two-dimensional matrix or a three-dimensional matrix, different processing is performed on the second matrix B in the method 300 .

[0042] like Figure 3As shown in FIG. 3 , in block 310, the first matrix A of the matrix multiplication operation may be converted into a first two-dimensional matrix A' and the first two-dimensional matrix A' may be loaded into the first buffer 214 for the first matrix A in the tensor core 210. For example, assuming that the first matrix A is a three-dimensional matrix (Batch, M, K), the converted first two-dimensional matrix A' is a two-dimensional matrix (Batch*M, K) or (M, Batch*K). Here, a more specific embodiment of converting and loading the first matrix A is described below with reference to FIG. Figure 4 Give a description.

[0043] At block 320, it may be determined whether the second matrix B of the matrix multiplication operation is a two-dimensional matrix or a three-dimensional matrix, for example Figure 1A The matrix shown is still Figure 1B The matrix shown.

[0044] If the second matrix B is a two-dimensional matrix (e.g. Figure 1A If the two-dimensional matrix (K, N) is shown, then in box 330, the second matrix B can be directly loaded into the second buffer 216 for the second matrix B in the tensor core 210.

[0045] If the second matrix B is a three-dimensional matrix (e.g. Figure 1B If the three-dimensional matrix (Batch, K, N) shown in FIG. 3 is a matrix of the first matrix B, the second matrix B can be converted into a second two-dimensional matrix B' and loaded into the second buffer 216 for the second matrix B in the tensor core 210. Here, a more specific embodiment of converting and loading the second matrix B is referred to below. Figure 6 A description is given, which corresponds to the conversion and loading method of the first matrix A.

[0046] Then, in block 350 , a matrix multiplication is performed in the tensor core 210 on the first two-dimensional matrix A′ and the second matrix B (in the case of block 330 ) or the second two-dimensional matrix B′ (in the case of block 340 ).

[0047] Note that the conversion and loading of the first matrix A may be performed simultaneously with the loading of the second matrix B (block 330 ) or the conversion and loading (block 340 ) may be performed in sequence, and the specific execution order is not limited herein.

[0048] Figure 4 A further detailed flow chart of the process of converting and loading the first matrix A (block 310) according to some embodiments of the present invention is shown.

[0049] like Figure 4 As shown in FIG. 3 , in block 312 , the first matrix A may be loaded from the first memory 220 outside the tensor core 210 to the vector core 230 through a multiple data load (Load Much, LDM) operation.

[0050] In block 314, in the vector core 230, the first dimension and the second dimension of the first two-dimensional matrix A' may be determined based on the second dimension M and the third dimension K of the first matrix A. Here, the vector core 230 converts the first matrix A by performing address conversion on each element in the first matrix A. Depending on the different situations of the second dimension M and the third dimension K of the first matrix A, the converted first two-dimensional matrix A' may be (Batch*M,K) or (M,Batch*K), as described in detail below.

[0051] Figure 5 A further detailed schematic diagram of the process of determining the first dimension and the second dimension of the first two-dimensional matrix A′ (block 314 ) according to some embodiments of the present invention is shown.

[0052] like Figure 5 As shown in, in block 3141, it can be determined whether the dimensions in the second dimension M and the third dimension K of the first matrix A that need to be merged with the first dimension Batch meet a predetermined condition. Here, the predetermined condition may refer to the absolute value of the second dimension M or the third dimension K being smaller or smaller relative to the other.

[0053] For example, assume that the second dimension M of the first matrix A is specified as a dimension that needs to be merged with the first dimension Batch. In this case, if the second dimension M<< the third dimension K, or the absolute value of the second dimension M is smaller (for example, when the hardware multiplication granularity of the tensor core is 64, the second dimension M is 2, 4, or 8), then the second dimension M of the first matrix A is considered to meet the predetermined condition.

[0054] For another example, assume that the third dimension K of the first matrix A is designated as a dimension that needs to be merged with the first dimension Batch. In this case, if the third dimension K<<the second dimension M, or the absolute value of the third dimension K is small (for example, when the hardware multiplication granularity of the tensor core is 64, the third dimension K is 2, 4 or 8), then the third dimension K of the first matrix A is considered to meet the predetermined condition.

[0055] Those skilled in the art can understand that the second dimension M and the third dimension K here are merely exemplary and are not used to limit the order or size of the second dimension and the third dimension.

[0056] In block 3142, if it is determined that the dimension (second dimension M or third dimension K) that needs to be merged with the first dimension Batch of the first matrix A meets the predetermined condition, the remainder value k of the dimension that needs to be merged modulo the hardware multiplication granularity Kd of the tensor core 210 can be determined, that is, k=M%Kd or k=K%Kd, where % represents a remainder operation. Here, it is assumed that the hardware multiplication granularity Kd of the tensor core 210 is 64, that is, the tensor core 210 can process a matrix multiplication of size Kd*Kd each time, but those skilled in the art can understand that the present invention is not limited to this, and depending on the different hardware implementations of the computing device 200 or the tensor core 210, the hardware multiplication granularity Kd can be other values, such as 8*8, 16*16, etc.

[0057] In block 3143, it may be determined whether the remainder value k is greater than or equal to a predetermined threshold, where the predetermined threshold is associated with the hardware multiplication granularity Kd of the tensor core 210. For example, the predetermined threshold may be half or less of the hardware multiplication granularity Kd of the tensor core 210.

[0058] If it is determined that the remainder value k is greater than or equal to the predetermined threshold, then in block 3144, the first matrix A may be divided into blocks according to the first dimension Batch of the first matrix A and the dimension M or K to be merged, and the first matrix A may be read as a first two-dimensional matrix A' according to the other dimension of the dimension M or K of the first matrix A. That is, in this case, the first matrix A is actually directly regarded as a two-dimensional matrix of (Batch*M, K) or (M, Batch*K) for data reading, and the first matrix A after block division is not spliced, that is, the storage method of the matrix elements is not changed.

[0059] On the other hand, if it is determined that the remainder value k is less than the predetermined threshold, then in block 3145, the first matrix A may be spliced ​​along the first dimension Batch of the first matrix A to form a first two-dimensional matrix A'. That is, when k is small, splicing needs to be performed along the first dimension Batch of the first matrix A, that is, the storage method of the matrix elements is changed.

[0060] Here, the method of dividing and splicing the first matrix A(Batch, M, K) in box 3144 can be referred to the description in the Chinese invention patent application with application number 202210440010.1, for example.

[0061] In the above manner, the first matrix A may be converted into a first two-dimensional matrix A′(Batch*M, K) or (M, Batch*K) suitable for the hardware processing capability of the tensor core 210 .

[0062] continue Figure 4In block 316 , the first two-dimensional matrix A′ may be stored from the vector core 230 to the second memory 240 near the tensor core 210 through a Store Much (STM) operation.

[0063] Here, the LDM operation and the STM operation are single-instruction read and write operations for multiple registers, wherein the LDM operation can load data of a segment of addresses in the address space to the vector core 230 in sequence through a single LDM instruction, and the STM operation can write data of multiple registers to multiple registers of the second memory 240 in sequence through a single STM instruction.

[0064] At block 318 , the first two-dimensional matrix A′ may be loaded into the first buffer 214 based on the size of the first buffer 214 .

[0065] In this way, a three-dimensional input matrix of a matrix multiplication operation can be converted into an appropriate two-dimensional matrix and the method of loading data into the corresponding cache in the tensor core 210 is improved, making full use of the cache space, reducing the number of data loading times, and improving computing performance.

[0066] Figure 6 A further detailed flow chart of the process of converting and loading the second matrix B (block 340) according to some embodiments of the present invention is shown. In the implementation of block 340, the second matrix B is a three-dimensional matrix, and the conversion and loading process of the second matrix B depends on the above combination Figure 4 and Figure 5 The conversion and loading process of the first matrix A described above is performed accordingly.

[0067] like Figure 6 As shown in FIG. 3 , in block 342 , the second matrix B may be loaded from the first memory 220 outside the tensor core 210 to the vector core 230 through a multiple data load (LDM) operation.

[0068] In box 344, in the vector kernel 230, the first dimension (K or Batch*K) and the second dimension (Batch*N or N) of the second two-dimensional matrix B' can be determined in the vector kernel 230 based on the first dimension (Batch*M or M) and the second dimension (K or Batch*K) of the first two-dimensional matrix A'.

[0069] Specifically, as described above, the first two-dimensional matrix A' can be determined as (Batch*M, K) or (M, Batch*K). When the first two-dimensional matrix A' is (Batch*M, K), the first dimension Batch of the first matrix A is merged with the second dimension M, so the second matrix B should merge the first dimension Batch with the third dimension N. On the other hand, when the first two-dimensional matrix A' is (M, Batch*K), the first dimension Batch of the first matrix A is merged with the third dimension K, so the second matrix B should merge the first dimension Batch with the second dimension K to ensure that matrix multiplication operations can be performed between the first two-dimensional matrix A' and the second two-dimensional matrix B'.

[0070] Figure 7 A further detailed schematic diagram of the process of determining the first dimension and the second dimension of the second two-dimensional matrix B′ (block 344 ) according to some embodiments of the present invention is shown.

[0071] like Figure 7 As shown in, in block 3442, it can be determined whether the second dimension M of the first matrix A is merged with the first dimension Batch of the first matrix A based on the first dimension Batch*M or M of the first two-dimensional matrix A'. That is, it is equivalent to determining whether the second dimension M or the third dimension K of the first matrix A is merged with the first dimension Batch.

[0072] If it is determined that the second dimension M of the first matrix A is merged with the first dimension Batch of the first matrix A (i.e., the first two-dimensional matrix A' is (Batch*M,K)), then in block 3444, the second matrix B may be divided into blocks according to the first dimension Batch and the third dimension N of the second matrix B, and the second matrix B may be read as the second two-dimensional matrix B' according to the order of the second dimension K of the second matrix B. That is, in this case, the second matrix B is actually directly regarded as a two-dimensional matrix of (K, Batch*N) or (Batch*K, N) for data reading, and the second matrix B after block division is not spliced, that is, the storage method of the matrix elements is not changed.

[0073] On the other hand, if it is determined that the second dimension M of the first matrix A is not merged with the first dimension Batch of the first matrix A (that is, the first dimension Batch of the first matrix A is merged with the third dimension K), then in box 3446, the second matrix B can be divided into blocks according to the first dimension Batch and the second dimension K of the second matrix B, and the second matrix B can be read according to the third dimension N of the second matrix B as a second two-dimensional matrix B'.

[0074] In the above manner, the second matrix B can be converted into a second two-dimensional matrix B′ suitable for the hardware processing capability of the tensor core 210 .

[0075] continue Figure 6 At block 346 , the second two-dimensional matrix B′ may be stored from the vector core 230 to the second memory 240 near the tensor core 210 through a multiple data storage (STM) operation.

[0076] At block 348 , the second two-dimensional matrix B′ may be loaded into the second buffer 216 based on the size of the second buffer 216 .

[0077] In this way, a three-dimensional input matrix of a matrix multiplication operation can be converted into an appropriate two-dimensional matrix and the method of loading data into the corresponding cache in the tensor core 210 is improved, making full use of the cache space, reducing the number of data loading times, and improving computing performance.

[0078] The above describes the computing device and method for implementing matrix multiplication operations according to the present disclosure in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that the execution of the above method is not limited to the order shown in the figure and described above, but may be executed in any other reasonable order. In addition, the computing device does not necessarily include all the components shown in the figure, and may only include some of the components or more components necessary to perform the functions described in the present disclosure, and the connection mode of these components is not limited to the form shown in the figure.

[0079] The present disclosure may be implemented as a method, a computing device, a system and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for executing various aspects of the present disclosure. The computing device may include at least one processor and at least one memory coupled to the at least one processor, the memory may store instructions for execution by the at least one processor. When the instructions are executed by the at least one processor, the computing device may execute the above method.

[0080] In one or more exemplary designs, the functions described in the present disclosure may be implemented using hardware, software, firmware, or any combination thereof. For example, if implemented using software, the functions may be stored as one or more instructions or codes on a computer-readable medium, or transmitted as one or more instructions or codes on a computer-readable medium.

[0081] The various units of the apparatus disclosed herein may be implemented using discrete hardware components or may be integrated on a hardware component, such as a processor. For example, the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof for performing the functions described herein.

[0082] Those skilled in the art should also understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments of the present disclosure may be implemented as electronic hardware, computer software, or a combination of the two.

[0083] The above description of the present disclosure is intended to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications of the present disclosure are obvious to those of ordinary skill in the art, and the general principles defined herein may also be applied to other variations without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope of the principles and novel features disclosed herein.

Claims

1. A method for implementing a matrix multiplication operation, comprising: Converting a first matrix of the matrix multiplication operation into a first two-dimensional matrix via a vector core of a computing device and loading the first two-dimensional matrix into a first buffer for the first matrix in the tensor core via a tensor core of the computing device, wherein the first matrix is ​​a three-dimensional matrix; Determining whether a second matrix of the matrix multiplication operation is a two-dimensional matrix or a three-dimensional matrix; In response to the second matrix being a two-dimensional matrix, loading the second matrix into a second buffer for the second matrix within the tensor core via the tensor core; In response to the second matrix being a three-dimensional matrix, converting the second matrix into a second two-dimensional matrix via the vector core and loading the second matrix into a second buffer for the second matrix in the tensor core via the tensor core; as well as A matrix multiplication is performed on the first two-dimensional matrix and the second matrix or the second two-dimensional matrix in the tensor core.

2. The method of claim 1, wherein converting the first matrix of the matrix multiplication operation into a first two-dimensional matrix and loading the first matrix into a first buffer for the first matrix in the tensor core comprises: Loading the first matrix from a first memory outside the tensor core to a vector core through a multiple data loading operation; determining, in the vector kernel, a first dimension and a second dimension of the first two-dimensional matrix based on a second dimension and a third dimension of the first matrix; Storing the first two-dimensional matrix from the vector core to a second memory near the tensor core through a multi-data storage operation; as well as The first two-dimensional matrix is ​​loaded into the first buffer based on a size of the first buffer.

3. The method of claim 2, wherein determining the first dimension and the second dimension of the first two-dimensional matrix based on the second dimension and the third dimension of the first matrix in the vector kernel comprises: determining whether a dimension in the second dimension and the third dimension of the first matrix that needs to be merged with the first dimension of the first matrix meets a predetermined condition; In response to determining that the dimension that needs to be merged with the first dimension of the first matrix satisfies the predetermined condition, determining a remainder value of the dimension that needs to be merged with the first dimension of the first matrix modulo a hardware multiplication granularity of the tensor core; determining whether the remainder value is greater than or equal to a predetermined threshold, wherein the predetermined threshold is associated with a hardware multiplication granularity of the tensor core; In response to determining that the remainder value is greater than or equal to the predetermined threshold, dividing the first matrix into blocks according to a first dimension of the first matrix and a dimension that needs to be merged with the first dimension of the first matrix, and reading the first matrix as the first two-dimensional matrix according to the other of the second dimension and the third dimension of the first matrix; as well as In response to determining that the remainder value is less than the predetermined threshold, the first matrix is ​​concatenated along a first dimension of the first matrix to form the first two-dimensional matrix.

4. The method of claim 2, wherein converting the second matrix into a second two-dimensional matrix and loading the second matrix into a second buffer for the second matrix in the tensor core comprises: Loading the second matrix from the first memory outside the tensor core to the vector core through a multiple data loading operation; Based on the first dimension or the second dimension of the first two-dimensional matrix, determining the first dimension and the second dimension of the second two-dimensional matrix in the vector kernel; storing the second two-dimensional matrix from the vector core to a second memory near the tensor core through a multi-data storage operation; as well as The second two-dimensional matrix is ​​loaded into the second buffer based on the size of the second buffer.

5. The method of claim 4, wherein determining the first dimension and the second dimension of the second two-dimensional matrix in the vector kernel based on the first dimension and the second dimension of the first two-dimensional matrix comprises: determining, based on the first dimension of the first two-dimensional matrix, whether the second dimension of the first matrix is ​​merged with the first dimension of the first matrix; If it is determined that the second dimension of the first matrix is ​​merged with the first dimension of the first matrix, dividing the first matrix into blocks according to the first dimension and the third dimension of the second matrix, and reading the second matrix as the second two-dimensional matrix according to the second dimension of the second matrix; as well as If it is determined that the second dimension of the first matrix is ​​not merged with the first dimension of the first matrix, the second matrix is ​​divided into blocks according to the first dimension and the second dimension of the second matrix, and the second matrix is ​​read according to the third dimension of the second matrix as the second two-dimensional matrix.

6. A computing device comprising: A tensor core comprising a general matrix multiplication unit, a first buffer for a first matrix for matrix multiplication operation, and a second buffer for a second matrix for matrix multiplication operation, wherein the first matrix is ​​a three-dimensional matrix; a first memory located outside the tensor core, configured to store the first matrix and the second matrix before the matrix multiplication operation; a vector kernel configured to convert the first matrix and the second matrix into a first two-dimensional matrix and a second two-dimensional matrix; as well as a second memory located near the tensor core, configured to store the first two-dimensional matrix and the second two-dimensional matrix, Where the tensor core is configured as: loading the first two-dimensional matrix into the first buffer, When the second matrix is ​​a two-dimensional matrix, the second matrix is ​​loaded into the second buffer, or when the second matrix is ​​a three-dimensional matrix, the second two-dimensional matrix is ​​loaded into the second buffer, and A matrix multiplication is performed on the first two-dimensional matrix and the second matrix or the second two-dimensional matrix.

7. The computing device of claim 6, wherein the vector core is configured to: Loading the first matrix from the first memory through a multiple data loading operation; determining a first dimension and a second dimension of the first two-dimensional matrix based on a second dimension and a third dimension of the first matrix; and The first two-dimensional matrix is ​​stored in the second memory through a multiple data storage operation.

8. The computing device of claim 7, wherein the vector core is configured to: determining whether a dimension in the second dimension and the third dimension of the first matrix that needs to be merged with the first dimension of the first matrix meets a predetermined condition; In response to determining that the dimension that needs to be merged with the first dimension of the first matrix satisfies the predetermined condition, determining a remainder value of the dimension that needs to be merged with the first dimension of the first matrix modulo a hardware multiplication granularity of the tensor core; determining whether the remainder value is greater than or equal to a predetermined threshold, wherein the predetermined threshold is associated with a hardware multiplication granularity of the tensor core; In response to determining that the remainder value is greater than or equal to the predetermined threshold, dividing the first matrix into blocks according to a first dimension of the first matrix and a dimension that needs to be merged with the first dimension of the first matrix, and reading the first matrix as the first two-dimensional matrix according to the other of the second dimension and the third dimension of the first matrix; as well as In response to determining that the remainder value is less than the predetermined threshold, the first matrix is ​​concatenated along a first dimension of the first matrix to form the first two-dimensional matrix.

9. The computing device of claim 6, wherein the vector core is configured to: Loading the second matrix from the first memory through a multiple data loading operation; determining the first dimension and the second dimension of the second two-dimensional matrix based on the first dimension or the second dimension of the first two-dimensional matrix; and The second two-dimensional matrix is ​​stored in the second memory through a multiple data storage operation.

10. The computing device of claim 9, wherein the vector core is configured to: determining, based on the first dimension of the first two-dimensional matrix, whether the second dimension of the first matrix is ​​merged with the first dimension of the first matrix; If it is determined that the second dimension of the first matrix is ​​merged with the first dimension of the first matrix, dividing the first matrix into blocks according to the first dimension and the third dimension of the second matrix, and reading the second matrix as the second two-dimensional matrix according to the second dimension of the second matrix; as well as If it is determined that the second dimension of the first matrix is ​​not merged with the first dimension of the first matrix, the second matrix is ​​divided into blocks according to the first dimension and the second dimension of the second matrix, and the second matrix is ​​read according to the third dimension of the second matrix as the second two-dimensional matrix.

Images