Acceleration method of matrix multiplication, electronic equipment and storage medium

Abstract

The invention relates to the technical field of chip design, in particular to a matrix multiplication acceleration method, electronic equipment and a storage medium, and the method comprises the steps: obtaining an expected storage type of a first input matrix block in a matrix multiplication accumulator MMA; obtaining an actual storage type of matrix data of the first input matrix block stored in a memory; when the expected storage type is different from the actual storage type, reading the matrix data from the memory and storing the matrix data into an intermediate register, and converting the matrix data stored according to the actual storage type in the intermediate register into matrix data stored according to the expected storage type to obtain expected matrix data; storing the expected matrix data stored in the intermediate register into a shared memory; according to the method, the corresponding matrix data is taken out from the shared memory through the high-wide-bit data transfer instruction, loaded into the register and subjected to MMA calculation processing, the high-wide-bit data transfer instruction can be directly used for transferring the data, and the purpose of improving the data transfer efficiency is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of chip design, and in particular to an acceleration method, electronic device and storage medium for matrix multiplication. Background Art

[0002] In graphics processing units (GPUs), matrix multiplication is one of the core operations of high-performance computing applications. Matrix multiply-accumulate (MMA) is a device used in computer architecture to efficiently perform matrix multiplication and accumulation operations. The core of matrix multiplication is to calculate the cumulative sum of the products of the corresponding elements of two matrices. Matrix multiplication accumulators usually adopt a pipeline design, and the matrix multiplication process is decomposed into multiple stages, including data reading, multiplication operation, accumulation operation, etc. Each stage is completed in a different clock cycle, so that different operation stages can process different data elements in parallel at the same time, improving the operation efficiency. In order to accelerate the implementation of matrix multiplication, GPUs usually adopt a block processing strategy, that is, the input matrices A and B are divided into smaller sub-matrices, such as multiple M×K sub-matrices and K×N sub-matrices. These sub-matrix data are initially stored in global memory, and then read into shared memory or directly into registers for efficient matrix operations. When data is stored in shared memory, it needs to be read from the shared memory into registers again for matrix multiplication and accumulator (MMA) operations. However, based on various considerations, users store matrices in memory in the following ways: NT type storage, NN type storage, TN type storage, and TT type storage. NT type storage means that matrix A is stored in row-major order and matrix B is stored in column-major order, and NN type storage means that both matrix A and matrix B are stored in row-major order. However, the ordering of data read from shared memory or registers needs to meet the input conditions of MMA. Since MMA can only arrange data in a specific thread data manner when processing data, directly using the high-width lds (load from share memory) data movement instruction will result in incorrect processing results; the lds instruction is used to load data from shared memory into registers. Since the data movement process when storing data in registers is very time-consuming in GPUs, a method to improve the movement efficiency is urgently needed. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is: a matrix multiplication acceleration method, the method comprising the following steps: S100, obtaining a desired storage type of a first input matrix block in a matrix multiplication accumulator MMA.

[0004] S200, obtaining an actual storage type of matrix data storing a first input matrix block in a memory.

[0005] S300, when the expected storage type of the first input matrix block is different from the actual storage type, the matrix data is read from the memory and stored in the intermediate register, and the matrix data stored according to the actual storage type in the intermediate register is converted into matrix data stored according to the expected storage type through hardware instructions to obtain the expected matrix data.

[0006] S400, storing the desired matrix data stored in the intermediate register into a shared memory.

[0007] S500, fetching corresponding matrix data from the shared memory through a data moving instruction, and loading the corresponding matrix data into an input register specifically used to provide matrix data for the MMA.

[0008] In addition, the present invention also provides a non-transitory computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the above method.

[0009] In addition, the present invention also provides an electronic device, including a processor and the above-mentioned non-transitory computer-readable storage medium.

[0010] The present invention has at least the following beneficial effects: An embodiment of the present invention provides an acceleration method, electronic device and storage medium for matrix multiplication, which converts the matrix data stored in the intermediate register into the matrix data expected by the MMA, so that the data stored in the shared memory is the expected matrix data. When the data in the shared memory is loaded into the register of the MMA, Q data move instructions of the high-width bit can be used directly to move the data. When the amount of data moved each time increases, the value of T increases, and the value of Q decreases, that is, the number of moves of the data move instruction is reduced. At the same time, the conversion step S300 is used to ensure that the calculation result of the matrix multiplier will not be wrong, thereby achieving the purpose of improving data transfer efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1A flowchart of a matrix multiplication acceleration method provided by an embodiment of the present invention; Figure 2 A schematic diagram of an arrangement of threads provided in an embodiment of the present invention; Figure 3 Based on Figure 2 A schematic diagram showing the arrangement of matrix elements before and after executing the corresponding hardware instructions; Figure 4 A schematic diagram of another arrangement of threads provided in an embodiment of the present invention; Figure 5 Based on Figure 4 A schematic diagram showing the arrangement of matrix elements before and after executing the corresponding hardware instructions; Figure 6 A schematic diagram of another arrangement of threads provided in an embodiment of the present invention; Figure 7 Based on Figure 6 Schematic diagram comparing the arrangement of matrix elements before and after executing the corresponding hardware instructions. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0014] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meanings as commonly understood by those skilled in the art.

[0015] See also Figure 1 , which shows a method for accelerating matrix multiplication, the method comprising the following steps: S100, obtaining a desired storage type of a first input matrix block in a matrix multiplication accumulator MMA.

[0016] Among them, the first input matrix block is a first D-level matrix block, the second input matrix block is a second D-level matrix block, D is the maximum division level of the input matrix in the matrix multiplication into matrix blocks, and the value range of D is 1 to D. It should be noted that the matrix multiplication of MMA includes two input matrices. In order to improve the efficiency of matrix multiplication, for the matrix multiplication operation of the first matrix A and the second matrix B, according to the standard matrix multiplication rules, the first matrix A is divided into a plurality of first matrix blocks {A1, A2, …, A i ,…,A L}, A iis the i-th first matrix block, and the value range of i is 1 to L; the second matrix B is also divided into multiple second matrix blocks {B1, B2,…, B i ,…,B L}, B i is the i-th second matrix block, where i ranges from 1 to L; the first matrix block A i With the second matrix block B i Multiply to obtain an intermediate result, and then calculate based on the intermediate result to obtain the final result. i It will be further divided into multiple first and second level matrix blocks {a i1 ,a i2 ,…,a ir ,…,a iL}, a ir A i Similarly, B will be divided into a plurality of second-level matrix blocks. Other embodiments of dividing matrix blocks at more levels also fall within the protection scope of the present invention.

[0017] The conversion steps provided in the embodiment of the present invention and applicable to the first input matrix block are also applicable to the second input matrix block.

[0018] In one implementation, the storage type of each input matrix block in the memory is row-major storage or column-major storage.

[0019] Among them, MMA includes two input matrices, and the calculation of the two input matrices in MMA follows the standard matrix multiplication rules.

[0020] In one embodiment, the combined storage mode of the two input matrices of the MMA is NT type storage, NN type storage, TN type storage and TT type storage. Wherein N and T are respectively used to represent the storage type of the corresponding input matrix, wherein N refers to row-major storage, and T refers to column-major storage. For example, when the combined storage mode of the two input matrices of the MMA is NT type storage, it means that one of the two input matrices is row-major storage and the other input matrix is ​​column-major storage; when the combined storage mode of the two input matrices of the MMA is NN type storage, both input matrices are row-major storage; and so on, which will not be repeated.

[0021] In one embodiment, the expected storage type of the first input matrix block of the MMA is row-major storage, and the expected storage type of the second input matrix block is column-major storage, and the expected combined storage mode is NT type storage. The expected storage type of the first input matrix block of the MMA is column-major storage, and the storage type of the second input matrix block is row-major storage, and the expected combined storage mode is TN type storage. The expected storage types of the first input matrix block and the second input matrix block of the MMA are both row-major storage or both column-major storage, and the expected combined storage mode is NN type storage or TT type storage. Combinations of other storage types expected by the first input matrix block and the second input matrix block of the MMA also fall within the protection scope of the present invention.

[0022] S200, obtaining an actual storage type of matrix data storing a first input matrix block in a memory.

[0023] In one implementation, the actual storage type of the first input matrix block is row-major storage or column-major storage, wherein the actual storage type is a storage type configured by the user, which may be an actual storage type configured based on chip performance or storage space considerations.

[0024] S300, when the expected storage type of the first input matrix block is different from the actual storage type, the matrix data is read from the memory and stored in the intermediate register, and the matrix data stored according to the actual storage type in the intermediate register is converted into matrix data stored according to the expected storage type through hardware instructions to obtain the expected matrix data.

[0025] Among them, the purpose of the embodiment of the present invention is to directly use the high-width data move instruction to move data. If the order of the currently read matrix elements is correct, it can move more matrix data through one data move instruction, and can ensure that the calculation result of the matrix multiplication is correct. The number of moves of the high-width data move instruction is greatly reduced relative to the low-width data move instruction, so as to achieve the purpose of improving the data move efficiency. However, when the expected storage type is different from the actual storage type, if the storage type is not converted, it is easy to cause matrix multiplication errors. For example, the storage type of the first input matrix block originally moved is row-major storage, and each data move instruction moves 32 bits of data, while MMA expects matrix data stored in column-major order; at this time, if the data move instruction moves 64 bits of data each time, it will cause the calculation result of the matrix multiplier to be wrong. In order to directly use the high-width data move instruction and ensure that the calculation result of the matrix multiplier is correct, it is necessary to convert the storage type of the matrix data in the intermediate register according to S300.

[0026] When reading data from the memory to the intermediate register, the data storage type in the intermediate register is the same as the memory, which is the actual storage type; in order to make the data storage type in the shared memory the expected storage type, before moving the matrix data in the intermediate register to the shared memory, it is necessary to convert the matrix data stored in the intermediate register according to the actual storage type into matrix data stored according to the expected storage type.

[0027] S400, storing the desired matrix data stored in the intermediate register into a shared memory.

[0028] S500, taking out the matrix data of the first input matrix block and the matrix data of the second input matrix block of each MMA from the shared memory through Q data moving instructions, and loading the matrix data of the first input matrix block and the second input matrix block into an input register specifically providing matrix data for the MMA; wherein Q satisfies: Q=U / (T×H), wherein U is the amount of data of the first input matrix block and the second input matrix block, T is the amount of data moved each time by each data moving instruction, and H is the number of threads in the thread group of each execution unit.

[0029] It should be noted that, since the corresponding matrix data has been converted into the matrix data expected by the MMA in the intermediate register, the data stored in the shared memory is already the expected matrix data. When the data in the shared memory is loaded into the register of the MMA, the Q data move instructions of the high-width bits can be used directly to move the data. When the amount of data moved each time increases, that is, when the value of T increases, the value of Q decreases, that is, the number of moves of the data move instructions is reduced, and at the same time, the conversion step of S300 is used to ensure that the calculation result of the matrix multiplier will not be wrong.

[0030] Among them, in S300, the matrix data stored in the intermediate register according to the actual storage type is converted into matrix data stored according to the expected storage type, and the conversion step is configured according to the thread arrangement. In one embodiment, the thread arrangement includes one thread reading matrix data in N groups of intermediate registers, N threads reading matrix data in one intermediate register, and one thread reading matrix data in two intermediate registers. Other types of thread arrangements also fall within the protection scope of the present invention.

[0031] In one embodiment, when an execution unit processes the matrix multiplication of N MMAs in parallel, the memory stores the matrix data of N×M×K matrix elements of the N MMAs according to the actual storage type, wherein the matrix data of the N×M matrix elements is the matrix data of the first input matrix block of an MMA; the matrix data of the N MMAs are read in parallel by a thread group of an execution unit, and the matrix data of each MMA is stored in N groups of intermediate registers, and the conversion step includes: S310, obtaining the N×M matrix elements stored in the N groups of intermediate registers. S320, replacing the N×M matrix elements stored in the N groups of intermediate registers processed by the current thread through hardware instructions, converting the actual storage type of the N×M matrix elements into the expected storage type of the N MMAs, and obtaining the expected matrix data. Then U in S500 is the data amount of the M×K matrix elements.

[0032] In one embodiment, the hardware instruction is a permutation instruction, and the permutation instruction is used to recombine the data of the two intermediate registers. The combination method can be determined according to the specific hardware design requirements, for example, it may be a simple splicing, or it may be combined in a specific bit selection or crossover manner. As an example, when the actual storage type is row-major storage, the expected storage type is column-major storage, and each intermediate register stores two matrix elements, and each row of matrix elements in the input matrix block is stored through two intermediate registers, the positions of the matrix elements stored in the intermediate registers are exchanged through the permutation instruction, so that the storage type of the matrix data of the intermediate register is the expected column-major storage.

[0033] In one embodiment, N=4, M=4, each group includes two intermediate registers, each intermediate register stores 2 matrix elements, the actual storage type is row-major storage, and the expected storage type is column-major storage. The replacement instruction is a perm instruction, and the matrix elements stored in the intermediate register are replaced by the perm instruction.

[0034] In another implementation, N=4, M=4, each group includes two intermediate registers, each intermediate register stores a maximum of 32-bit data, each intermediate register stores two 16-bit matrix elements, the actual storage type is row-major storage, and the expected storage type is column-major storage. The matrix elements stored in the intermediate register are replaced by the perm instruction.

[0035] As an example, see Figure 2 , Figure 2 In is the arrangement of threads and registers, Figure 2T0 is thread 0, T1 is thread 1, and so on; r[0:1] is the intermediate registers r0 and r1, r[2:3], r[4:5], r[6:7] represent r2~r7 respectively, a total of 6 intermediate registers, and so on; each intermediate register stores 32-bit data. When each matrix element is 16 bits, the data volume is 16 rows and 64 columns of 16-bit matrix elements, that is, a 16×(16×4) matrix. Figure 3 In the example, A16 and B16 before the perm instruction are stored in r0, C16 and D16 are stored in r1, E16 and F16 are stored in r2, and so on, P16 and Q16 are stored in r7. The matrix data before the perm instruction is a row storage type. The matrix data and itself are used as two inputs of the per instruction for conversion, and the storage type of the converted matrix data is the desired column-major storage. Finally, the 64-bit, or 4 matrix elements, are taken out through the data movement instruction and placed into the shared memory.

[0036] In another embodiment, N=4, M=4, each group includes two intermediate registers, each intermediate register stores a maximum of 32-bit data, each intermediate register stores four 8-bit matrix elements, the actual storage type is row-major storage, and the expected storage type is column-major storage. The matrix data of the N×M matrix elements stored in the intermediate register and itself are used as two inputs of the perm instruction to perform the first permutation to obtain the permuted matrix data; and then the permuted matrix data and itself are used as two inputs of the perm instruction to perform the second permutation to obtain the expected matrix data.

[0037] As an example, when each matrix element is 16 bits and the data moving instruction is 64 bits, each data moving instruction moves 4 matrix elements. When each matrix element is 16 bits and the data moving instruction is 128 bits, each data moving instruction moves 8 matrix elements.

[0038] In one embodiment, when the memory stores matrix data of T×M matrix elements according to the actual storage type, and the matrix data of the T×M matrix elements is the matrix data of the first input matrix block of an MMA, the matrix elements are read by T threads and each intermediate register stores M matrix elements, then S300 also includes: reading the matrix elements stored continuously in the memory in parallel by T threads respectively, and storing the M matrix elements read by each thread in an intermediate register; the conversion step includes: S310, obtaining the T×M matrix elements stored in T groups of intermediate registers. The hardware instructions in S320 include a swap instruction (shfl instruction) and a permute instruction, through which the matrix elements stored in the intermediate registers processed by adjacent threads are exchanged to obtain the exchanged matrix elements; the exchanged matrix elements and the matrix elements before the exchange are permuted by the permute instruction to obtain the desired matrix data. Then U in S500 is the data amount of T×M matrix elements. Among them, the swap instruction is used to exchange the matrix elements stored in the intermediate registers corresponding to the two threads.

[0039] In one implementation, T=2, M=2, and each intermediate register stores two half-precision matrix elements; the hardware instructions include a swap instruction (shfl instruction) and a permute instruction.

[0040] As an example, see Figure 4 , Figure 4 Another way to arrange threads and registers is in Figure 4 In the example, threads T0 and T1 read two half-precision matrix elements b0 and b1 and store them in an intermediate register. They are also used to read two half-precision matrix elements b2 and b3 in another intermediate register. When each intermediate register stores 32-bit data and each matrix element is 16 bits, the data volume is 16 rows and 16 columns of 16-bit matrix elements, that is, a 16×16 matrix, where T0:b0 means that the data in the intermediate register read by thread 0 is b0. Figure 5 This is a schematic diagram showing the comparison of matrix elements before and after executing the corresponding hardware instructions. Figure 5 In the example, A16 and B16 before the shlf instruction are executed correspond to Figure 4 T0:b0 and T0:b1 in ; C16 and D16 before executing the shlf instruction, corresponding to Figure 4T1:b0 and T1:b1 in the data structure. When the matrix elements stored in the intermediate registers corresponding to T0 and T1 are exchanged through the shlf instruction, the exchanged matrix elements are obtained; after the exchange, T0 corresponds to the matrix elements C16 and D16, and T1 corresponds to the matrix elements A16 and B16; then the exchanged matrix elements and the original matrix elements before the exchange are exchanged through the perm instruction to obtain the expected matrix data; in the expected matrix data, T0 corresponds to the two half-precision matrix elements A16 and C16 stored in the intermediate register, and T1 corresponds to the two half-precision matrix elements B16 and D16 stored in the intermediate register, that is, the original row storage type of A16 and B16 is converted to the column storage type. Finally, the two half-precision matrix elements are taken out through the data movement instruction and placed in the shared memory.

[0041] In one implementation, the maximum precision of the intermediate register is 32 bits, each intermediate register stores two matrix elements, and each matrix element is 16 bits of data.

[0042] In one embodiment, the maximum precision of the intermediate register storage is 32 bits, and each intermediate register stores four matrix elements, each matrix element is 8 bits of data. Then, the step of performing substitution by the hardware instruction in S320 also includes: exchanging the matrix elements stored in the intermediate registers corresponding to two adjacent groups of threads by means of a swap instruction to obtain T×4 matrix elements after the swap; and performing substitution by means of a substitution instruction on the T×4 matrix elements after the swap and the T×4 matrix elements before the swap to obtain processed matrix data. The processed matrix data and itself are again permuted to obtain the desired matrix data, and the data moving instruction in S500 is a 32-bit instruction.

[0043] In one embodiment, when the memory stores matrix data of T×M matrix elements according to the actual storage type, wherein the matrix data of the T×M matrix elements is the matrix data of the first input matrix block of an MMA; when the matrix data of K intermediate registers are read through T threads and each intermediate register stores M / K matrix elements, the S300 also includes: reading the M matrix elements continuously stored in the memory in sequence through N threads; the conversion step includes: S310, obtaining the T×M matrix elements stored in N groups of intermediate registers; S320, the hardware instructions include exchange instructions, permutation instructions and move instructions (mov instructions), through which the matrix elements stored in the intermediate registers processed by adjacent threads are exchanged to obtain exchanged matrix elements; through the permutation instructions, the exchanged matrix elements and the matrix elements before the exchange are permuted to obtain permuted matrix data; through the exchange instructions, the permuted matrix data are exchanged again to obtain the matrix elements exchanged again; through the move instructions, the matrix elements exchanged again and the permuted matrix data are moved to obtain the desired matrix data. Then, U in S500 is the data amount of T×M matrix elements.

[0044] In one implementation, T=4, M=4, K=2.

[0045] As an example, see Figure 6 , Figure 6 Another way to arrange threads and registers is in Figure 6 In the example, T0:b0, T0:b1, T0:b2, and T0:b3 respectively indicate that the T0 thread processes the matrix elements b0, b1, b2, and b3, and so on. When each matrix element is 16 bits, the data volume is 16 rows and 16 columns of 16-bit matrix elements, that is, a 16×16 matrix. Figure 7 This is a schematic diagram showing the comparison of matrix elements before and after executing the corresponding hardware instructions. Figure 7 In the matrix before the first execution of the shlf instruction, A16, B16, C16, and D16 corresponding to T0 correspond to Figure 6b0, b1, b2 and b3 in . When the matrix elements stored in the intermediate registers corresponding to T0 and T1 are exchanged through the shlf instruction, the exchanged matrix elements are obtained; after the exchange, the matrix elements corresponding to T0 are E16, F16, G16 and H16, and the matrix elements corresponding to T1 are A16, B16, C16 and D16, and so on. Then the perm instruction is used to replace the exchanged matrix elements and the original matrix elements before the exchange to obtain the replaced matrix data; in the replaced matrix data, T0 corresponds to A16, E16, C16 and G16, and T1 corresponds to B16, F16, D16 and H16; then the shlf instruction and mov instruction are executed again to convert the matrix data into the desired column storage type. Finally, the 64-bit, i.e., 4 matrix elements, are taken out through the data movement instruction and placed in the shared memory.

[0046] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0047] An embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned non-transitory computer-readable storage medium.

[0048] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0049] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0050] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A matrix multiplication acceleration method, characterized in that: The method comprises the following steps: S100, obtaining a desired storage type of a first input matrix block in a matrix multiplication accumulator MMA; S200, obtaining an actual storage type of matrix data storing a first input matrix block in a memory; S300, when the expected storage type of the first input matrix block is different from the actual storage type, reading matrix data from the memory and storing it in an intermediate register, converting the matrix data stored in the intermediate register according to the actual storage type into matrix data stored in the expected storage type through hardware instructions, to obtain expected matrix data; S400, storing the desired matrix data stored in the intermediate register into a shared memory; S500, taking out the matrix data of the first input matrix block and the matrix data of the second input matrix block of each MMA from the shared memory through Q data moving instructions, and loading the matrix data of the first input matrix block and the second input matrix block into an input register specifically providing matrix data for the MMA; wherein Q satisfies: Q=U / (T×H), wherein U is the amount of data of the first input matrix block and the second input matrix block, T is the amount of data moved each time by each data moving instruction, and H is the number of threads in the thread group of each execution unit.

2. The method according to claim 1, characterized in that When one execution unit processes matrix multiplication of N MMAs in parallel, the memory stores matrix data of N×M×K matrix elements of the N MMAs according to the actual storage type, wherein the matrix data of the M×K matrix elements are matrix data of a first input matrix block of one MMA; the matrix data of the N MMAs are read in parallel by a thread group of one execution unit, and the matrix data of each MMA is stored in N groups of intermediate registers; The conversion step in S300 includes: S310, obtaining the N×M matrix elements stored in N groups of intermediate registers; S320, replacing the N×M matrix elements stored in the N groups of intermediate registers processed by the current thread through hardware instructions, converting the actual storage types of the N×M matrix elements into the expected storage types of the N MMAs, and obtaining expected matrix data; Then U in S500 is the data amount of M×K matrix elements.

3. The method according to claim 2, characterized in that N=4, M=4, each group includes two intermediate registers, each intermediate register stores a maximum of 32-bit data, each intermediate register stores two 16-bit matrix elements, the actual storage type is row-major storage, and the expected storage type is column-major storage; then the replacement instruction is the perm instruction, and the matrix elements stored in the intermediate register are replaced by the perm instruction.

4. The method according to claim 2, characterized in that: Said N=4, M=4, each group includes two intermediate registers, each intermediate register stores a maximum of 32-bit data, each intermediate register stores four 8-bit matrix elements, the actual storage type is row-major storage, and the expected storage type is column-major storage; the matrix data of the N×M matrix elements stored in the intermediate register and itself are used as two inputs of the perm instruction for the first permutation to obtain the permuted matrix data; and then the permuted matrix data and itself are used again as two inputs of the perm instruction for the second permutation to obtain the expected matrix data.

5. The method according to claim 1, characterized in that When the memory stores matrix data of T×M matrix elements according to the actual storage type, and the matrix data of the T×M matrix elements is the matrix data of the first input matrix block of an MMA, the matrix data of the same intermediate register is read through T threads and each intermediate register stores M matrix elements, then S300 further includes: reading the matrix elements stored continuously in the memory in parallel through the T threads respectively, and each thread reads M matrix elements and stores them in an intermediate register; the conversion step includes: S310, obtaining the T×M matrix elements stored in T groups of intermediate registers; The hardware instructions in S320 include a swap instruction and a permute instruction, wherein the matrix elements stored in the intermediate registers processed by adjacent threads are swapped by the swap instruction to obtain swapped matrix elements; and the matrix elements after the swap are permute with the matrix elements before the swap by the permute instruction to obtain desired matrix data; Then U in S500 is the data amount of T×M matrix elements.

6. The method according to claim 5, characterized in that Said T=2, M=2, each intermediate register stores two half-precision matrix elements; the hardware instructions include exchange instructions and permute instructions, each data movement instruction is used to fetch all matrix elements in an intermediate register; wherein, the exchange instruction is used to exchange the matrix elements stored in the intermediate registers corresponding to two threads.

7. The method according to claim 5, characterized in that The maximum precision of the intermediate register is 32 bits, and each intermediate register stores four matrix elements, each of which is 8 bits of data; The step of replacing by the hardware instruction in S320 also includes: replacing the matrix elements stored in the intermediate registers corresponding to two adjacent groups of threads by means of an exchange instruction to obtain T×4 matrix elements after the exchange; replacing the T×4 matrix elements after the exchange with the T×4 matrix elements before the exchange by means of a replacement instruction to obtain processed matrix data; and replacing the processed matrix data and itself again to obtain the desired matrix data.

8. The method according to claim 1, characterized in that When the memory stores matrix data of T×M matrix elements according to the actual storage type, wherein the matrix data of the T×M matrix elements is matrix data of a first input matrix block of an MMA; when the matrix data of K intermediate registers are read through T threads and each intermediate register stores M / K matrix elements, the S300 further includes: sequentially reading the M matrix elements continuously stored in the memory through the T threads; the conversion step includes: S310, obtaining the T×M matrix elements stored in N groups of intermediate registers; S320, the hardware instructions include a swap instruction, a permute instruction, and a move instruction, wherein the matrix elements stored in the intermediate registers processed by the adjacent threads are swapped by the swap instruction to obtain swapped matrix elements; the swapped matrix elements and the matrix elements before the swap are permute by the permute instruction to obtain permute matrix data; the permute matrix data are swapped again by the swap instruction to obtain re-swapped matrix elements; the re-swapped matrix elements and the permute matrix data are moved by the move instruction to obtain desired matrix data; Then, U in S500 is the data amount of T×M matrix elements.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images