Matrix multiplication operators, operational methods, devices, graphics processors, and storage media

Abstract

This invention provides a method, apparatus, graphics processor, and storage medium for matrix multiplication operators, relating to the field of artificial intelligence technology. The method includes: reading two operation matrices from high-bandwidth memory; determining a first target input matrix in the general matrix buffer (GMB) based on the two operation matrices, and identifying the other operation matrix as a second target input matrix; the first target input matrix is ​​arranged in a transposed manner; and writing the first and second target input matrices into a tensor core for matrix multiplication to obtain the target output matrix from the tensor core. This invention, by identifying the data arrangement of the two operation matrices, prioritizes inputting the first target input matrix (with a transposed arrangement) into the GMB, effectively avoiding hardware performance limitations associated with non-transposed readings, ensuring the tensor core maintains full-bandwidth data input, and significantly improving the computational efficiency of the matrix multiplication operator.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, graphics processor and storage medium for matrix multiplication. Background Technology

[0002] As one of the most commonly used operators in large model training and inference, the performance of the MMA (Matrix Multiply and Accumulate) operator has a significant impact on the overall performance of the large model. Therefore, in GPU (Graphics Processing Unit) programming, it is generally necessary to utilize GMB (Gemm Main Buffer) or tensor buffer high-speed caching resources to improve the performance of the MMA operator.

[0003] In the MMA operator, since the data volume of the two input matrices is large, the Tensor Core (Tcore) cannot read the input matrices at once. Usually, the two input matrices are split into input block matrices corresponding to each input matrix. Then, at least one of the two input block matrices is written into the GMB. The Tensor Core reads the input block matrices from the GMB and performs matrix multiplication, which can improve the performance by several times.

[0004] However, when the input block matrix in GMB is a non-transpose matrix and the data type is FP32, the bandwidth of the tensor core reading the non-transpose input block matrix from GMB will be significantly reduced compared to reading the transpose input block matrix, which cannot meet the computational capabilities of the tensor core, and the computation process does not achieve the expected performance. Summary of the Invention

[0005] This invention provides a matrix multiplication operator operation method, apparatus, graphics processor, and storage medium to solve the problem in the prior art where the bandwidth of the tensor core reading the untransposed input block matrix from the GMB is greatly reduced, which cannot meet the computing power of the tensor core and the calculation process does not achieve the expected performance.

[0006] This invention provides a method for operating a matrix multiplication operator, comprising the following steps.

[0007] Read two operation matrices from high-bandwidth memory.

[0008] Based on the two operation matrices, a first target input matrix is ​​determined in the input general matrix main cache (GMB), and the other operation matrix is ​​determined as a second target input matrix; the data arrangement of the first target input matrix is ​​a transpose arrangement.

[0009] The first target input matrix and the second target input matrix are written into the tensor core and multiplied to obtain the target output matrix output by the tensor core.

[0010] According to the matrix multiplication operator operation method provided by the present invention, determining the first target input matrix in the input general matrix main cache (GMB) using the two operation matrices includes: If the operation matrix input to the GMB is determined to be the first operation matrix, the first target input matrix input to the GMB is determined based on the data arrangement and data type of the first operation matrix. Without determining the operation matrix to be input to the GMB, the first target input matrix to be input to the GMB is determined based on the data arrangement corresponding to the two operation matrices.

[0011] According to the matrix multiplication operator operation method provided by the present invention, determining the first target input matrix for the GMB based on the data arrangement and data type of the first operation matrix includes: When the data arrangement of the first operation matrix is ​​a non-transposed arrangement and the data type of the first operation matrix is ​​a floating-point type, the non-transposed arrangement of the first operation matrix is ​​converted to a transposed arrangement by using a transpose write method to obtain the first target input matrix input to the GMB.

[0012] According to the matrix multiplication operator operation method provided by the present invention, determining the first target input matrix for the GMB based on the data arrangement of the two operation matrices respectively includes: If there is no transpose arrangement in the data arrangement of the two operation matrices, the non-transpose arrangement of any operation matrix is ​​converted to a transpose arrangement by transpose writing, and the first target input matrix is ​​obtained by inputting the GMB. If a transpose arrangement exists in the data arrangement of the two operation matrices, then any operation matrix corresponding to the transpose arrangement is determined as the first target input matrix for the GMB.

[0013] According to the matrix multiplication operator operation method provided by the present invention, the step of converting the non-transpose permutation corresponding to the first operation matrix into a transpose permutation using a transpose writing method to obtain the first target input matrix input to the GMB includes: The first operation matrix is ​​read into the registration register; the registration register is used to convert the non-transpose arrangement of the first operation matrix into a transpose arrangement using a transpose write method. Write the first target input matrix output from the registration register into the GMB.

[0014] According to the matrix multiplication operator operation method provided by the present invention, before performing matrix multiplication, the method further includes: If the data arrangement of the first operation matrix is ​​a non-transposed arrangement and the data type of the first operation matrix is ​​a floating-point type, or if there is no transposed arrangement in the data arrangement of the two operation matrices, the transposed configuration mode corresponding to the tensor core is adjusted to a transposed-non-transposed mode.

[0015] The present invention also provides a computational apparatus for a matrix multiplication operator, comprising the following modules.

[0016] The read module is used to read two operation matrices from high-bandwidth memory.

[0017] The determination module is used to determine a first target input matrix in the input general matrix main cache (GMB) based on the two operation matrices, and to determine the other operation matrix as a second target input matrix; the data arrangement of the first target input matrix is ​​a transpose arrangement.

[0018] The multiplication module is used to write the first target input matrix and the second target input matrix into the tensor core for matrix multiplication to obtain the target output matrix output by the tensor core.

[0019] The present invention also provides a graphics processor, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the operation method of the matrix multiplication operator as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the operation method of the matrix multiplication operator as described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the operation method of any of the matrix multiplication operators described above.

[0022] The matrix multiplication operator operation method, apparatus, graphics processor, and storage medium provided by this invention determine a first target input matrix with transposed data arrangement based on two operation matrices in the video memory (GMB). This first target input matrix is ​​then input into the GMB, and the other operation matrix is ​​designated as a second target input matrix. The first and second target input matrices are then written into the tensor core for matrix multiplication to obtain the target output matrix from the tensor core. In this invention, by identifying the data arrangement of the two operation matrices, the first target input matrix with transposed data arrangement is preferentially input into the GMB. This ensures that the tensor core always maintains transposed readings when reading data from the GMB, effectively avoiding hardware performance defects associated with non-transposed readings. This allows the tensor core to maintain full bandwidth data input, significantly improving the computational efficiency of the matrix multiplication operator. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is an example diagram illustrating matrix multiplication using existing matrix multiplication operators.

[0025] Figure 2 This is a flowchart illustrating the operation method of the matrix multiplication operator provided in the embodiments of the present invention.

[0026] Figure 3 This is a flowchart illustrating the transpose writing method provided in an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the operation device for the matrix multiplication operator provided in an embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram of the structure of the graphics processor provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] In MMA operators, taking the formula C = A × B as an example, A and B represent the input matrices, and C represents the output matrix. Input matrices A, B, and C are all stored in High Bandwidth Memory (HBM), also known as GPU memory. Input matrix A has dimensions M × K, input matrix B has dimensions K × N, and output matrix C has dimensions M × N. Generally, the input matrices A and B are large, and the tensor core cannot read them all at once. Typically, the two input matrices are split into tile matrices, resulting in corresponding input block matrices. This splitting strategy is called the Tiling strategy. Then, at least one of the two input block matrices is written into the GMB. The tensor core reads the input block matrices from the GMB, or from both the GMB and HBM, and performs matrix multiplication. For example... Figure 1 This is an example diagram illustrating matrix multiplication using existing matrix multiplication operators, such as... Figure 1 As shown, the data block matrix tileA of input matrix A is written into GMB. The dimension of data block matrix tileA is tileM×tileK. The tensor core reads data block matrix tileA from GMB and reads data block matrix tileB of input matrix B from HBM for matrix multiplication. The dimension of data block matrix tileB is tileK×tileN. The result of the multiplication is written into data block matrix tileC of output matrix C in HBM. The dimension of data block matrix tileC is tileM×tileN.

[0031] However, if the Tensor Core reads the data block matrix tileA of the input matrix A in non-transpose (N) and the data type of tileA is FP32, the bandwidth for the Tensor Core to read the non-transpose tileA from the GMB will decrease by about 50% compared to reading the transpose (T) version. This data throughput cannot meet the Tensor Core's computational capabilities, failing to achieve the expected performance. It should be noted that for a given original matrix, non-transpose means the matrix elements are stored according to the original matrix's own row and column order, while transpose means the matrix elements are stored according to the row and column order of the transpose of the original matrix. For example, if the original matrix is ​​M rows and K columns, non-transpose means storing the original matrix in row-major order, i.e., storing the K elements of the first row first, then the K elements of the second row, and so on. The transpose arrangement is to store the transpose of the original matrix in row-major order or to store the original matrix in column-major order. That is, first store the M elements of the first column, then store the M elements of the second column, and so on.

[0032] Therefore, in the existing technology, the bandwidth of the tensor core reading the untransposed input block matrix from the GMB will drop significantly, which cannot meet the computing power of the tensor core and the computing process does not achieve the expected performance. The present invention provides a matrix multiplication operator operation method, which effectively avoids the hardware performance defects when reading untransposed data by intelligently scheduling and reconstructing the hardware data path.

[0033] It should be noted that the execution entity of the matrix multiplication operator operation method provided in this embodiment of the invention can be a processor such as a GPU (Graphics Processing Unit), GPGPU (General purpose computing on Graphics Processing Units), or TPU (Tensor Processing Unit). This processor includes at least a high-bandwidth memory (HBM), a general-purpose matrix main cache (GMB), a tensor core (Tcore), and a registry register. In this embodiment of the invention, a registry register is added to the traditional hardware architecture. The registry register is used to identify the data arrangement and perform transpose write operations, actively managing the arrangement of the operation matrix flowing into the GMB. This ensures that the operation matrix ultimately stored in the GMB always exists in a transposed arrangement that the tensor core can read at full bandwidth, maximizing the utilization of the GMB cache resource and always supplying data to the tensor core in the most efficient way.

[0034] It should be noted that the matrix multiplication operator operation method provided in this embodiment of the invention can be applied to artificial intelligence training and inference scenarios that rely on matrix multiplication calculations, such as the training and inference scenarios of models like Large Language Models (LLM), computer vision models, and speech recognition and generation models. In the self-attention mechanism and feedforward neural network of the Transformer architecture of Large Language Models, there are numerous matrix multiplication operations. Using the operation method provided in this embodiment of the invention can accelerate the execution of core operators, thereby improving the training and inference speeds of tasks such as text generation, dialogue, and translation. In the Convolutional Neural Networks (CNN) and Vision Transformers (ViT) of computer vision models, both convolutional operations and fully connected layers include matrix multiplication operations. Using the operation method provided in this embodiment of the invention can accelerate the model training and image processing processes for tasks such as image classification, object detection, and semantic segmentation. In the acoustic models of speech processing, matrix multiplication is a fundamental operation. Using the operation method provided in this embodiment of the invention helps improve the response speed and efficiency of applications such as speech-to-text and text-to-speech.

[0035] Figure 2 This is a flowchart illustrating the operation method of the matrix multiplication operator provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes steps 210 to 230.

[0036] Step 210: Read the two operation matrices from the high-bandwidth memory.

[0037] It should be noted that each operation matrix can be the complete input matrix, or a tile matrix consisting of data blocks obtained by partitioning the input matrix. Furthermore, initially, both operation matrices are stored in video memory by default.

[0038] Step 220: Based on the two operation matrices, determine the first target input matrix in the input general matrix main cache (GMB), and determine the other operation matrix as the second target input matrix; the data arrangement of the first target input matrix is ​​a transpose arrangement.

[0039] Specifically, after obtaining the two operation matrices, a first target input matrix with a transposed data arrangement is determined from the two operation matrices based on their data arrangement. This first target input matrix can be one of the two operation matrices, or it can be obtained by transforming one of the operation matrices. Then, the first target input matrix is ​​written into the GMB, and the other operation matrix is ​​determined as the second target input matrix.

[0040] Optionally, the second target input matrix can be stored in HBM or written to GMB. It is important to note that when writing the second target input matrix to GMB, the data arrangement of the second target input matrix must be transposed to avoid hardware performance defects caused by the second target input matrix.

[0041] Step 230: Write the first target input matrix and the second target input matrix into the tensor core and perform matrix multiplication to obtain the target output matrix output by the tensor core.

[0042] Specifically, after determining the first target input matrix and the second target input matrix, the first target input matrix is ​​written from the GMB to the tensor core, and the second target input matrix is ​​written from the GMB or HBM to the tensor core. Then, the tensor core performs matrix multiplication on the first and second target input matrices to obtain the target output matrix, which is then written to the HBM. This target output matrix can be a complete output matrix or a data block matrix (tile) within a complete output matrix.

[0043] It should be noted that the Tensor Core includes four transpose configuration modes: non-transpose-non-transpose (NN), non-transpose-transpose (NT), transpose-non-transpose (TN), and transpose-transpose (TT). The data arrangement to the left of the "-" indicates the arrangement of the left matrix, and the data arrangement to the right of the "-" indicates the arrangement of the right matrix. For example, in calculating C = A × B, input matrix A represents the left matrix, and input matrix B represents the right matrix. The calculation methods and results differ across the four transpose configuration modes. Furthermore, the output matrix after matrix multiplication in the Tensor Core is in a non-transpose arrangement.

[0044] The matrix multiplication operator operation method provided in this invention involves determining a first target input matrix with a transposed data arrangement based on two operation matrices in the video memory (GMB). This first target input matrix is ​​then input into the GMB, and the other operation matrix is ​​designated as a second target input matrix. The first and second target input matrices are then written into the tensor core for matrix multiplication to obtain the target output matrix from the tensor core. In this invention, by identifying the data arrangement of the two operation matrices, the first target input matrix with a transposed data arrangement is preferentially input into the GMB. This ensures that the tensor core always reads data from the GMB in a transposed manner, effectively avoiding hardware performance defects associated with non-transposed readings. This allows the tensor core to maintain full bandwidth data input, significantly improving the computational efficiency of the matrix multiplication operator.

[0045] In one embodiment, determining the first target input matrix in the input general matrix main cache (GMB) based on the two operation matrices includes: If the operation matrix input to the GMB is determined to be the first operation matrix, the first target input matrix input to the GMB is determined based on the data arrangement and data type of the first operation matrix. Without determining the operation matrix to be input to the GMB, the first target input matrix to be input to the GMB is determined based on the data arrangement corresponding to the two operation matrices.

[0046] Specifically, both operation matrices can be written into the GMB, or one operation matrix can be written into the GMB and the other operation matrix stored in the HBM. In this embodiment of the invention, the example of one operation matrix being written into the GMB and the other operation matrix being stored in the HBM will be described.

[0047] If it is pre-specified that the first operation matrix of the two operation matrices is written to the GMB, the system determines whether there is a hardware performance defect of bandwidth reduction when the tensor core reads data from the GMB after the first operation matrix is ​​written to the GMB, based on the data arrangement and data type of the first operation matrix. If so, the first operation matrix is ​​converted into the first target data matrix with the data arrangement as transpose, so as to bypass the hardware performance defect of bandwidth reduction.

[0048] If it is not specified which of the two operation matrices to write to the GMB, the system determines whether there is an operation matrix with a transposed arrangement based on the data arrangement of the two operation matrices, thereby determining the first target input matrix for the GMB to bypass the hardware performance defect of bandwidth reduction.

[0049] In one embodiment, determining the first target input matrix for the GMB based on the data arrangement and data type of the first operational matrix includes: When the data arrangement of the first operation matrix is ​​a non-transposed arrangement and the data type of the first operation matrix is ​​a floating-point type, the non-transposed arrangement of the first operation matrix is ​​converted to a transposed arrangement by using a transpose write method to obtain the first target input matrix input to the GMB.

[0050] Specifically, when writing the first operand of two operands to the GMB, if the data arrangement of the first operand is non-transposed and its data type is floating-point, such as a 32-bit single-precision floating-point number (FP32), a transpose write method is used to convert the non-transposed arrangement of the first operand to a transposed arrangement. That is, when writing to the GMB, the row-major writing order is changed to a column-major writing order. Based on this, the data arrangement of the first operand itself remains unchanged; what changes is the physical address layout of the first operand written to the GMB. For example, the first operand is... The first target input matrix after being written to GMB using the transpose method is: .

[0051] In one embodiment, the step of converting the non-transpose permutation of the first operational matrix into a transpose permutation using a transpose writing method to obtain the first target input matrix input to the GMB includes: The first operation matrix is ​​read into the registration register; the registration register is used to convert the non-transpose arrangement of the first operation matrix into a transpose arrangement using a transpose write method. Write the first target input matrix output from the registration register into the GMB.

[0052] For example, Figure 3 This is a flowchart illustrating the transpose write method provided in an embodiment of the present invention, as shown below. Figure 3 As shown, with the first operation matrix tileA as... Taking a first operation matrix tileA with a dimension of 3×2 as an example, before writing the first operation matrix tileA into the GMB, it is written from the HBM into the register register in row-major order, and then transposed and written out from the register register in column-major order. The resulting first target input matrix tileAT is... The first target input matrix, tileAT, is written into the GMB, and its dimension is adjusted to 2×3. Then, the tensor core reads the first target input matrix, tileAT, from the GMB and the second target input matrix, tileB, from the HBM. It then performs matrix multiplication based on the first target input matrix, tileAT, and the second target input matrix, tileB, to obtain the target output matrix, tileC.

[0053] It should be noted that, compared to the prior art which directly writes the first operation matrix into the GMB, this embodiment of the invention involves an additional transpose write step before writing to the GMB, resulting in a certain performance loss. Therefore, in this embodiment of the invention, a high-speed registration register can be used to compensate for this performance loss.

[0054] In one embodiment, determining the first target input matrix for the GMB based on the data arrangement corresponding to each of the two operation matrices includes: If there is no transpose arrangement in the data arrangement of the two operation matrices, the non-transpose arrangement of any operation matrix is ​​converted to a transpose arrangement by transpose writing, and the first target input matrix is ​​obtained by inputting the GMB. If a transpose arrangement exists in the data arrangement of the two operation matrices, then any operation matrix corresponding to the transpose arrangement is determined as the first target input matrix for the GMB.

[0055] Specifically, if it is not specified which of the two operation matrices to write to the GMB, it is determined whether there is at least one transpose arrangement of the data in the two operation matrices. If the data arrangement of each of the two operation matrices is not transpose, then regardless of which operation matrix is ​​written to the GMB, there will be a hardware performance defect of reduced bandwidth when the tensor core reads data from the GMB. In this case, referring to the transpose writing method described above, either of the two operation matrices can be converted from a non-transpose arrangement to a transpose arrangement to obtain the first target input matrix, and the obtained first target input matrix is ​​written to the GMB.

[0056] If at least one of the two operation matrices has a transpose, the operation matrix corresponding to the transpose can be determined as the first target input matrix, and this first target input matrix can be written into the GMB to circumvent the hardware performance defect of bandwidth reduction. For example, if the data arrangement of the left matrix is ​​transpose, then the left matrix is ​​determined as the first target input matrix; if the data arrangement of the right matrix is ​​transpose, then the right matrix is ​​determined as the first target input matrix.

[0057] It should be noted that if the data arrangement of both operation matrices is transpose, then after either or both operation matrices are written to the GMB, there is no hardware performance defect of bandwidth reduction when the tensor core reads data from the GMB.

[0058] In one embodiment, the method further includes, prior to matrix multiplication: If the data arrangement of the first operation matrix is ​​a non-transposed arrangement and the data type of the first operation matrix is ​​a floating-point type, or if there is no transposed arrangement in the data arrangement of the two operation matrices, the transposed configuration mode corresponding to the tensor core is adjusted to a transposed-non-transposed mode.

[0059] Specifically, when the data arrangement of the first operation matrix is ​​non-transposed and the data type of the first operation matrix is ​​floating-point, or when neither of the two operation matrices has a transposed arrangement, it indicates that a transpose write operation is required to obtain a first target input matrix in a transposed arrangement. In this case, the dimensions of the first target input matrix change, and directly multiplying the first and second target input matrices may not be possible. In this situation, the transpose configuration mode of the tensor core can be adjusted to a transpose-non-transpose mode, allowing the tensor core to read the first and second target input matrices column-by-column, perform matrix multiplication based on the read data, and calculate based on the results. This ensures that the target output matrix after transpose writing, multiplying the first and second target input matrices in the transpose-non-transpose mode, is the same as the output matrix after directly multiplying the two operation matrices before transpose, but with better bandwidth performance.

[0060] For example, let's take operation matrix A as... The operation matrix B is Taking the calculation of the target output matrix C = A × B as an example, when both operation matrices A and B are in a non-transposed arrangement, operation matrix A is written to the GMB, and operation matrix B is written to the HBM. The tensor core reads operation matrix A from the GMB. This reading process suffers from a hardware performance defect that reduces bandwidth. The transpose configuration mode in the tensor core is NN mode. In NN mode, operation matrix A is not transposed and written. The tensor core reads operation matrix A row-wise and operation matrix B column-wise, and multiplies the read data to obtain the output matrix. In this embodiment of the invention, the transpose method is used to transform the data arrangement of the operation matrix A, resulting in the first target input matrix. The first target input matrix is ​​written into the GMB. The operation matrix B is then designated as the second target input matrix. Next, the tensor core reads the first target input matrix from the GMB and the second target input matrix from the HBM. At this point, both the first and second target input matrices have a dimension of 2×3, making matrix multiplication impossible. Therefore, the transpose configuration mode in the tensor core is switched from NN mode to TN mode. In TN mode, the tensor core reads the column data of both the first and second target input matrices, performs matrix multiplication on the read column data, and repeats the above operation. The final target output matrix C remains the same. By adjusting the transpose writing method and transpose configuration mode, the accuracy of the matrix multiplication result after transpose is ensured.

[0061] The following describes the arithmetic apparatus for the matrix multiplication operator provided by the present invention. The arithmetic apparatus for the matrix multiplication operator described below can be referred to in correspondence with the arithmetic method for the matrix multiplication operator described above.

[0062] This invention provides a computational apparatus for a matrix multiplication operator. Figure 4 This is a schematic diagram of the structure of the arithmetic device for the matrix multiplication operator provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the operation device 400 of the matrix multiplication operator includes: a reading module 410, a determining module 420, and a multiplication module 430.

[0063] The read module 410 is used to read two operation matrices from high-bandwidth memory.

[0064] The determining module 420 is used to determine a first target input matrix in the input general matrix main cache (GMB) based on the two operation matrices, and to determine the other operation matrix as a second target input matrix; the data arrangement of the first target input matrix is ​​a transpose arrangement.

[0065] The multiplication module 430 is used to write the first target input matrix and the second target input matrix into the tensor core for matrix multiplication to obtain the target output matrix output by the tensor core.

[0066] The matrix multiplication operator processing device provided in this embodiment of the invention determines a first target input matrix with a transposed data arrangement based on two operation matrices in the video memory (GMB). This first target input matrix is ​​then input into the GMB, and the other operation matrix is ​​determined as a second target input matrix. The first and second target input matrices are then written into the tensor core for matrix multiplication to obtain the target output matrix from the tensor core. In this embodiment, by identifying the data arrangement of the two operation matrices, the first target input matrix with a transposed data arrangement is preferentially input into the GMB. This ensures that the tensor core always reads data from the GMB in a transposed manner, effectively avoiding hardware performance defects during non-transposed readings. This allows the tensor core to maintain full bandwidth data input, significantly improving the computational efficiency of the matrix multiplication operator.

[0067] Optionally, the determining module 420 is specifically used for: If the operation matrix input to the GMB is determined to be the first operation matrix, the first target input matrix input to the GMB is determined based on the data arrangement and data type of the first operation matrix. Without determining the operation matrix to be input to the GMB, the first target input matrix to be input to the GMB is determined based on the data arrangement corresponding to the two operation matrices.

[0068] Optionally, the determining module 420 is specifically used for: When the data arrangement of the first operation matrix is ​​a non-transposed arrangement and the data type of the first operation matrix is ​​a floating-point type, the non-transposed arrangement of the first operation matrix is ​​converted to a transposed arrangement by using a transpose write method to obtain the first target input matrix input to the GMB.

[0069] Optionally, the determining module 420 is specifically used for: If there is no transpose arrangement in the data arrangement of the two operation matrices, the non-transpose arrangement of any operation matrix is ​​converted to a transpose arrangement by transpose writing, and the first target input matrix is ​​obtained by inputting the GMB. If a transpose arrangement exists in the data arrangement of the two operation matrices, then any operation matrix corresponding to the transpose arrangement is determined as the first target input matrix for the GMB.

[0070] Optionally, the determining module 420 is specifically used for: The first operation matrix is ​​read into the registration register; the registration register is used to convert the non-transpose arrangement of the first operation matrix into a transpose arrangement using a transpose write method. Write the first target input matrix output from the registration register into the GMB.

[0071] Optionally, the operation device 400 of the matrix multiplication operator further includes: a mode adjustment module, which is specifically used for: Before performing matrix multiplication, if the data arrangement of the first operation matrix is ​​non-transposed and the data type of the first operation matrix is ​​floating-point, or if there is no transposed arrangement in the data arrangement of the two operation matrices, the transposed configuration mode corresponding to the tensor core is adjusted to transposed-non-transposed mode.

[0072] Figure 5 This is a schematic diagram of the structure of the graphics processor provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the graphics processing unit (GPU) may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a matrix multiplication operator operation. This method includes: reading two operation matrices from high-bandwidth memory; determining a first target input matrix in the input general matrix main cache (GMB) based on the two operation matrices, and determining the other operation matrix as a second target input matrix; the first target input matrix is ​​arranged in a transposed arrangement; and writing the first target input matrix and the second target input matrix into a tensor core for matrix multiplication to obtain the target output matrix output by the tensor core.

[0073] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the matrix multiplication operator operation method provided by the above methods. The method includes: reading two operation matrices in high-bandwidth memory; determining a first target input matrix in the input general matrix main cache (GMB) based on the two operation matrices, and determining the other operation matrix as a second target input matrix; the data arrangement of the first target input matrix is ​​a transpose arrangement; writing the first target input matrix and the second target input matrix into a tensor core for matrix multiplication to obtain the target output matrix output by the tensor core.

[0075] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for performing the matrix multiplication operator provided by the above methods. The method includes: reading two operation matrices from high-bandwidth memory; determining a first target input matrix in the input general matrix main cache (GMB) based on the two operation matrices, and determining the other operation matrix as a second target input matrix; the data arrangement of the first target input matrix is ​​a transpose arrangement; writing the first target input matrix and the second target input matrix into a tensor core for matrix multiplication to obtain a target output matrix output by the tensor core.

[0076] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for operating a matrix multiplication operator, characterized in that, include: Read two operation matrices from high-bandwidth memory; Based on the two operation matrices, a first target input matrix is ​​determined in the input general matrix main cache (GMB), and the other operation matrix is ​​determined as the second target input matrix; the data arrangement of the first target input matrix is ​​a transpose arrangement. The first target input matrix and the second target input matrix are written into the tensor core and multiplied to obtain the target output matrix output by the tensor core.

2. The method for operating the matrix multiplication operator according to claim 1, characterized in that, The step of determining the first target input matrix in the input general matrix main cache (GMB) based on the two operation matrices includes: If the operation matrix input to the GMB is determined to be the first operation matrix, the first target input matrix input to the GMB is determined based on the data arrangement and data type of the first operation matrix. Without determining the operation matrix to be input to the GMB, the first target input matrix to be input to the GMB is determined based on the data arrangement corresponding to the two operation matrices.

3. The method for operating the matrix multiplication operator according to claim 2, characterized in that, The step of determining the first target input matrix for the GMB based on the data arrangement and data type of the first operation matrix includes: When the data arrangement of the first operation matrix is ​​a non-transposed arrangement and the data type of the first operation matrix is ​​a floating-point type, the non-transposed arrangement of the first operation matrix is ​​converted to a transposed arrangement by using a transpose write method to obtain the first target input matrix input to the GMB.

4. The method for operating the matrix multiplication operator according to claim 3, characterized in that, The step of determining the first target input matrix for the GMB based on the data arrangement of the two operation matrices includes: If there is no transpose arrangement in the data arrangement of the two operation matrices, the non-transpose arrangement of any operation matrix is ​​converted to a transpose arrangement by transpose writing, and the first target input matrix is ​​obtained by inputting the GMB. If a transpose arrangement exists in the data arrangement of the two operation matrices, then any operation matrix corresponding to the transpose arrangement is determined as the first target input matrix for the GMB.

5. The method for operating the matrix multiplication operator according to claim 3, characterized in that, The step of converting the non-transpose permutation of the first operation matrix into a transpose permutation using a transpose writing method to obtain the first target input matrix for the GMB includes: The first operation matrix is ​​read into the registration register; the registration register is used to convert the non-transpose arrangement of the first operation matrix into a transpose arrangement using a transpose write method. Write the first target input matrix output from the registration register into the GMB.

6. The method for operating the matrix multiplication operator according to claim 4, characterized in that, Before performing matrix multiplication, the method further includes: If the data arrangement of the first operation matrix is ​​a non-transposed arrangement and the data type of the first operation matrix is ​​a floating-point type, or if there is no transposed arrangement in the data arrangement of the two operation matrices, the transposed configuration mode corresponding to the tensor core is adjusted to a transposed-non-transposed mode.

7. A computational apparatus for a matrix multiplication operator, characterized in that, include: The read module is used to read two operation matrices from high-bandwidth memory. The determination module is used to determine a first target input matrix in the input general matrix main cache (GMB) based on the two operation matrices, and to determine the other operation matrix as a second target input matrix; the data arrangement of the first target input matrix is ​​a transpose arrangement. The multiplication module is used to write the first target input matrix and the second target input matrix into the tensor core for matrix multiplication to obtain the target output matrix output by the tensor core.

8. A graphics processor, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the operation method of the matrix multiplier as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the operation method of the matrix multiplier as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the operation method of the matrix multiplier as described in any one of claims 1 to 6.

Images