Matrix multiply accumulation operation unit and operation method, and hardware accelerator and electronic device

Abstract

Provided in the embodiments of the present disclosure are a matrix multiply accumulation (MMA) operation unit and operation method, and a hardware accelerator and an electronic device. The MMA operation unit comprises a data load / store engine, a tensor register file and a matrix multiplication engine, wherein the data load / store engine is used for loading data of a plurality of matrixes to be subjected to MMA computation; the tensor register file is used for storing the data of the plurality of matrixes that is acquired from the data load / store engine; the tensor register file comprises at least three tensor register groups, each tensor register group comprising a plurality of tensor registers, and different tensor register groups being used for storing data of different matrixes among the plurality of matrixes; and the matrix multiplication engine is used for performing MMA computation on the basis of the data of the plurality of matrixes that is stored in the tensor register file. By means of the embodiments of the present disclosure, more efficient MMA computation is realized with low cost, low power consumption and a smaller footprint.

Description

Matrix multiplication and accumulation operation unit and operation method, hardware accelerator, and electronic device

[0001] This disclosure claims priority to a Chinese patent application filed with the Patent Office of China on March 28, 2024, with application number 202410371734.4 and application name “Matrix multiplication and accumulation operation unit and operation method, hardware accelerator, electronic device,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The embodiments of the present disclosure relate to the field of chip technology, and in particular to a matrix multiplication and accumulation operation unit, a matrix multiplication and accumulation operation method, a hardware accelerator, and an electronic device. Background Art

[0003] MMA (Matrix Multiply Accumulation) is widely used in the computation of machine learning models such as LLMs (Large Language Models). An example of an MMA calculation is shown in Figure 1. In Figure 1, A is the input matrix, m*k dimensions; B is the weight matrix, k*n dimensions; and C is the accumulation matrix, m*n dimensions. MMA is implemented as AB+C. In practice, the result of AB+C is used to update C. However, in Figure 1, for ease of presentation, the result of AB+C is represented as D, forming the MMA form of D=AB+C.

[0004] Because machine learning models involve numerous MMA calculations like the one above, they require high bandwidth and computing power. To this end, one related technique uses multiple memories to store the A, B, and C matrices used in the MMA calculations. During the calculations, the corresponding data is read from these memories to perform the MMA calculations.

[0005] However, with this approach, multiple memories take up a lot of space, and accessing them consumes significant bandwidth and computing power, posing a challenge to the implementation of the components they reside in. Furthermore, multiple memories also increase implementation costs. Therefore, finding hardware support for MMA computing at a low cost and minimal footprint, while providing sufficient computing power, has become a pressing issue. Summary of the Invention

[0006] In view of this, an embodiment of the present disclosure provides an MMA implementation solution to at least partially solve the above problems.

[0007] According to a first aspect of an embodiment of the present disclosure, a matrix multiplication and accumulation operation unit is provided, comprising: a data loading and storage engine, a tensor register file, and a matrix multiplication engine; wherein: the data loading and storage engine is used to load data of multiple matrices to be subjected to matrix multiplication and accumulation calculations; the tensor register file is used to store the data of the multiple matrices obtained from the data loading and storage engine; wherein the tensor register file comprises at least three tensor register groups, each tensor register group comprises multiple tensor registers, and different tensor register groups are used to store data of different matrices among the multiple matrices; the matrix multiplication engine is used to perform matrix multiplication and accumulation calculations based on the data of the multiple matrices stored in the tensor register file.

[0008] According to a second aspect of an embodiment of the present disclosure, a matrix multiplication and accumulation method is provided, including: obtaining a machine instruction for a matrix multiplication and accumulation process, the machine instruction containing information about registers where operands are located, the registers being registers in at least three tensor register groups in a tensor register file, different tensor register groups being used to store data of different matrices in a plurality of matrices to be subjected to matrix multiplication and accumulation calculations; and executing the operation indicated by the machine instruction.

[0009] According to a third aspect of an embodiment of the present disclosure, a matrix multiplication and accumulation method is provided, comprising: loading data of multiple matrices to be subjected to matrix multiplication and accumulation calculations into a tensor register file, wherein the tensor register file comprises at least three tensor register groups, each tensor register group comprises multiple tensor registers, and different tensor register groups are used to store data of different matrices among the multiple matrices; and performing matrix multiplication and accumulation calculations based on the data of the multiple matrices stored in the tensor register file.

[0010] According to a fourth aspect of an embodiment of the present disclosure, a hardware accelerator is provided, which includes at least an instruction decoder and a matrix multiplication and accumulation operation unit as described in the first aspect; wherein the instruction decoder is used to decode a machine instruction indicating a matrix multiplication and accumulation calculation, and control the matrix multiplication and accumulation operation unit to perform a matrix multiplication and accumulation calculation based on the decoding result.

[0011] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising at least: a processor and a hardware accelerator as described in the fourth aspect; wherein the processor is configured to send an instruction to the hardware accelerator to perform matrix multiplication and accumulation calculations; and the hardware accelerator performs matrix multiplication and accumulation calculations according to the instruction.

[0012] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions, and the computer execution is used to implement the method described in the second aspect and / or the third aspect when specified by a processor.

[0013] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct an electronic device to execute the method described in the second aspect and / or the third aspect through the hardware accelerator described in the fourth aspect, especially a matrix multiplication and accumulation operation unit in the hardware accelerator.

[0014] According to the solution provided by the embodiment of the present disclosure, a tensor register file is set in the MMA operation unit, and the tensor register file is divided into at least three tensor register groups to store data of different matrices to be calculated by MMA. Therefore, when MMA calculation is required, the data loading and storage engine in the MMA operation unit can load and send the data of multiple matrices to be calculated to different tensor register groups respectively; on this basis, the matrix multiplication engine can read the data of different matrices from these tensor register groups at the same time to perform MMA calculation. Compared with the solution of storing matrix data in multiple memories, on the one hand, this method reads data from tensor registers instead of directly from memory, which not only improves data access efficiency but also avoids the power consumption caused by memory access data; on the other hand, since there is no need to use multiple memories to store data, the space occupied by them is also reduced; on the other hand, using a tensor register file also reduces the implementation cost of the solution compared to using multiple memories. Therefore, the solution of the embodiment of the present disclosure achieves more efficient MMA calculation under the conditions of low cost, low power consumption and less space occupation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] FIG1 is a schematic diagram of an MMA calculation;

[0017] FIG2 is a schematic diagram of the structure of an exemplary system applicable to the embodiment of the present disclosure;

[0018] FIG3 is a schematic diagram of the structure of a hardware accelerator according to an embodiment of the present disclosure;

[0019] FIG4A is a schematic structural diagram of an MMA operation unit according to an embodiment of the present disclosure;

[0020] FIG4B is a schematic diagram of the structure of the tensor register file in the MMA operation unit shown in FIG4A ;

[0021] FIG4C is a schematic diagram illustrating storage of tensor registers in the tensor register file shown in FIG4B ;

[0022] FIG4D is a schematic diagram of the structure of an exemplary matrix multiplication engine in the MMA operation unit shown in FIG4A ;

[0023] FIG5 is a flowchart of the steps of an MMA operation method according to an embodiment of the present disclosure;

[0024] FIG6 is a flowchart of another MMA operation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure should fall within the scope of protection of the embodiments of the present disclosure.

[0026] The specific implementation of the embodiment of the present disclosure is further explained below with reference to the accompanying drawings of the embodiment of the present disclosure.

[0027] Figure 2 illustrates an exemplary system applicable to embodiments of the present disclosure. As shown in Figure 2 , the system 100 may include one or more electronic devices 102 and a communication network 104. Figure 2 illustrates multiple electronic devices. The electronic devices may be implemented as server-side devices, cloud-based devices, or terminal devices, such as desktop computers, tablet computers, servers, and the like. The embodiments of the present disclosure do not limit the specific form of the electronic devices.

[0028] The communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The electronic device 102 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112). The communication link can be any communication link suitable for transmitting data between the electronic devices 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0029] In the disclosed embodiment, the electronic device 102 may execute a machine learning algorithm, and the execution process of the machine learning algorithm may include MMA calculation. In one example, the electronic device 102 includes at least one or more processors 1022, a memory 1024, a communication interface 1026, a communication bus 1028, and a hardware accelerator 1020.

[0030] in:

[0031] The processor 1022 , the memory 1024 , the communication interface 1026 , and the hardware accelerator 1020 communicate with each other via a communication bus 1028 .

[0032] The communication interface 1026 is used to communicate with other electronic devices.

[0033] The memory 1024 can store instructions executable by the processor 1022, as well as other data and information, such as matrix data and information for performing MMA calculations, etc. The memory 1024 may include a high-speed RAM memory, and may also include a non-volatile memory.

[0034] The processor 1022 may be a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present disclosure. When the electronic device 102 includes one or more processors, the multiple processors may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0035] In this example, the hardware accelerator 1020 is used to accelerate data calculations, such as accelerating various data calculations involved in machine learning models (including but not limited to MMA calculations). For example, a structure of the hardware accelerator 1020 is shown in FIG3 .

[0036] As can be seen from Figure 3, the hardware accelerator 1020 includes at least: an instruction queue, an instruction decoder, and an arithmetic component for performing computational acceleration. In the embodiment of the present disclosure, the arithmetic component is implemented as an MMA arithmetic unit to achieve MMA computational acceleration. The instruction queue can receive machine instructions that need to be executed by the hardware accelerator 1020 from outside the hardware accelerator 1020 (such as from a CPU or GPU or external memory, etc.). Furthermore, the instruction decoder can decode the machine instructions and send control signals to other components in the hardware accelerator 1020 to execute the machine instructions. In the embodiment of the present disclosure, the machine instructions may include instructions for instructing to perform MMA calculations, which can be executed by the MMA arithmetic unit.

[0037] Optionally, the hardware accelerator 1020 may further include a memory, which may be located outside or inside the MMA operation unit and is used to receive data used by the hardware accelerator 1020 when executing machine instructions, such as matrix data, etc.; and may also store data generated during the execution of machine instructions.

[0038] In this exemplary framework, in order to effectively support MMA calculations, an embodiment of the present disclosure provides an MMA operation unit, as shown in FIG4A , which includes: a data loading and storage engine, a tensor register file, and a matrix multiplication engine.

[0039] Among them, the data loading and storage engine is used to load the data of multiple matrices to be calculated by MMA; the tensor register file is used to store the data of multiple matrices read from the data loading and storage engine, and the tensor register file includes at least three tensor register groups, each tensor register group includes multiple tensor registers, and different tensor register groups are used to store data of different matrices in the multiple matrices; the matrix multiplication engine is used to perform MMA calculation based on the data of multiple matrices stored in the tensor register file.

[0040] A register file, also known as a register file, includes an array of multiple registers. Optionally, the register file can be implemented by a fast SRAM (Static Random-Access Memory) or by a trigger. In the embodiment of the present disclosure, a register file is used to access tensors, so it is also called a tensor register file. Tensors come in many forms. In the embodiment of the present disclosure, a matrix form is used. Therefore, the tensor register file can store data of multiple matrices used for MMA calculations.

[0041] An exemplary implementation of a tensor register file for storing data of multiple matrices used for MMA calculations is shown in FIG4B .

[0042] As can be seen from Figure 4B, the tensor register file includes three tensor register groups, each of which includes eight tensor registers, and different tensor register groups are used to store data of different matrices. Specifically in Figure 4B, the three tensor register groups are respectively referred to as tensor register groups A, B, and C, to respectively store matrices A, B, and C used for MMA calculations as shown in Figure 1 (it should be noted that in the embodiment of the present disclosure, matrices A, B, and C are used to indicate the matrices at various positions in the MMA calculation of Figure 1, and do not represent a specific matrix). In the embodiment of the present disclosure, the number of bits of the multiple tensor registers included in each tensor register group is the same. Optionally, the number of bits of all tensor registers included in the entire tensor register file is the same. In the embodiment of the present disclosure, there is no specific limitation on the number of bits of tensor registers. It should be noted that although three tensor register groups are used as an example in Figure 4B, those skilled in the art should understand that in actual applications, more tensor register groups can be set to meet actual computing requirements. When these additional tensor register groups are used for MMA calculations, each tensor register group is used in conjunction with the tensor register group storing the matrix for the same MMA calculation position. Assuming there are six tensor register groups, namely tensor register groups A1, A2, B1, B2, C1, and C2, then tensor register groups A1 and A2 are used in conjunction, B1 and B2 are used in conjunction, and C1 and C2 are used in conjunction. The number of tensor registers in each tensor register group can also be appropriately set by those skilled in the art based on actual needs, and the embodiments of the present disclosure do not impose any restrictions on this.

[0043] In addition, the MMA operation unit of the embodiment of the present disclosure may also include a memory, which can be used to store at least the data of multiple matrices to be subjected to MMA calculations obtained from outside the MMA operation unit (such as a hardware accelerator or an element outside the hardware accelerator), so as to be subsequently provided to the data loading and storage engine. Optionally, MMA operation results and / or process data may also be stored. However, this is not limited to this, and the memory may also be provided outside the MMA operation unit. When storing, the memory may store the obtained matrix data in row-major order, or it may store the obtained matrix data in column-major order.

[0044] Still taking MMA as an example to calculate the matrix A*B+C, as shown in Figure 4A, after receiving the instruction sent by the instruction decoder to instruct to perform the MMA calculation, the data load storage engine can read the data of multiple matrices to be calculated by MMA, namely matrices A, B, and C from the memory; then, the tensor register file reads the data of matrices A, B, and C from the data load storage engine. In the embodiment of the present disclosure, the data of matrices A, B, and C will be stored in the corresponding tensor register groups A, B, and C respectively; the tensor register file has three read ports and one write port. The matrix multiplication engine can read the data of matrices A, B, and C used in the current calculation from the tensor register groups A, B, and C respectively through the three read ports and perform the MMA calculation. After the calculation is completed, the matrix multiplication engine will return the MMA calculation result to the tensor register file for storage, such as storing it in tensor register group C. Then, when the set time is reached or the instruction is received, the data load storage engine reads the data from the tensor register file and stores it in the memory.

[0045] The data loading and storage engine can be implemented as any suitable component, including but not limited to a loader or hardware logic for loading and / or storing data. The matrix multiplication engine can be implemented based on components including adders and multipliers.

[0046] To ensure efficient matrix data access and improve the overall efficiency of MMA calculations, one feasible approach is to store the data of multiple matrices in the tensor register file. After receiving the data, the tensor register file will store the data of each matrix in the tensor registers of the corresponding tensor register groups according to the numerical precision of each matrix, following the rule of storing from the lowest bit to the lowest bit. Different data types correspond to different numerical precisions, but for a matrix, all elements in the matrix have the same type and therefore the same numerical precision.

[0047] Because in actual calculations, the tensor registers store the data of the matrix to be calculated, that is, the matrix elements, so a tensor register can store all the elements of a matrix at a time. For larger matrices, they can be divided into sub-matrices suitable for storage in tensor registers, and then all the elements of the sub-matrix are stored in a tensor register of its corresponding tensor register group. It can be seen that a tensor register group can store the data of multiple matrices or multiple sub-matrices at the same time to facilitate subsequent parallel calculations. For example, in Figure 4B, a tensor register group includes 8 tensor registers, which means that the tensor register group can read the matrix elements of 8 matrices at a time. For example, assuming that matrices A-1, A-2, A-3, and A-4 are all 4*5-dimensional matrices, the tensor register group A can store the matrix elements of the read matrices A-1, A-2, A-3, and A-4 respectively in four of the tensor registers; assuming that there are 8 matrices A-1, A-2, A-3,...A-8, the tensor register group A can store the data of these 8 matrices respectively in the 8 tensor registers included in the tensor register group A, and each tensor register stores the matrix elements of one of the matrices.

[0048] For a given matrix, the data type and size, i.e., the numerical precision, of its matrix elements are the same. Data types typically include: 4-bit precision (FP4 / NF4, 4bit), 8-bit precision (FP8, 8bit), half-precision (FP16 / BF16, 16bit), single-precision (FP32 / TF32, 32bit), double-precision (FP64, 64bit), etc. To this end, after receiving data from multiple matrices, the tensor register file stores the matrix data in the tensor registers of the corresponding tensor register group, starting from the low bits of the tensor registers according to the numerical precision corresponding to the data type. Unused high bits can be set to 0 to make the storage method more convenient for storing matrices of different element sizes.

[0049] For example, suppose the tensor register is a 64-bit register and the matrix A is a The matrix element data type is 8 bits. Assuming that matrix A is stored in register 01 of tensor register bank A, a11 will be stored in bits 0-7 of register 01, a12 will be stored in bits 8-15 of register 01, a21 will be stored in bits 16-23 of register 01, and a22 will be stored in bits 24-31 of register 01. Bits 32-63 of register 01 will be set to 0. Storage in halfword or fullword units is similar.

[0050] Through this storage method, on the one hand, because data reading and writing both start from the low bit, the storage and reading of data stored in the tensor register can be more in line with actual needs, reducing the storage and reading costs; on the other hand, after the storage unit is determined, the subsequent reading of these data from the tensor register can also be more efficient, thereby improving the efficiency of data storage and reading.

[0051] Below, the storage method is explained in conjunction with Figure 4C. For the tensor register, a corresponding storage rule is set for it according to the data type of the matrix element. As can be seen from Figure 4C, no matter which storage rule is used, data is stored from the least significant bit LSB of the tensor register to the most significant bit MSB. For a matrix with a dimension of (8*8), if the numerical precision corresponding to its matrix elements is 8 bits (such as FP8 (8-bit Floating Point)), a low-bit storage method with 8-bit precision as the unit of storage will be used. In this way, the elements of the matrix read into a tensor register at one time will occupy 64 bytes of register bit width. If the numerical precision of the matrix elements of the matrix is ​​16 bits (such as FB16 / BF16 (Brain Floating Point 16)), a low-bit storage method with half precision as the unit of storage will be used. In this way, the elements of the matrix read into a tensor register at one time will occupy 128 bytes of register bit width. If the numerical precision of the matrix elements of the matrix is ​​32 bits (such as FP32), a low-bit storage method with full precision will be used. In this method, the elements of the matrix read into a tensor register at one time will occupy a register bit width of 256 bytes.

[0052] 4B , assuming that the matrix elements of matrix A have been read into a tensor register in tensor register group A, the matrix elements of matrix B have been read into a tensor register in tensor register group B, and the matrix elements of matrix C have been read into a tensor register in tensor register group C, then these read-in matrix elements will be used as operands for MMA calculations and read out from each tensor register group by the matrix multiplication engine for MMA calculations.

[0053] In addition, it should be noted that the above examples all take the matrix elements of the same matrix as an example when reading into the tensor register group. However, in actual applications, a tensor register group may read matrix elements of different matrices in one read operation. In this case, different tensor registers in the same tensor register group may have different storage methods. For example, some tensor registers use 8-bit precision as the low-order storage method, some tensor registers use half-precision as the low-order storage method, and so on.

[0054] After the tensor register file has read the corresponding matrix data, the matrix multiplication engine can perform MMA calculations based on these matrix data. To further improve the efficiency of MMA calculations, the present disclosure provides a more efficient matrix multiplication engine architecture, as shown in Figure 4D.

[0055] As can be seen from FIG4D , the matrix multiplication engine performs matrix multiplication and accumulation calculations based on the data of multiple matrices stored in the tensor register file through the column selector array, the row selector array, the multiplier array, and the adder array.

[0056] Specifically, when the tensor register file receives a read data signal (such as the "MMA Read signals" shown in Figure 4D), the data therein will be sent to the corresponding arrays in the matrix multiplication engine. For example, the data of matrix A (m*k) stored in tensor register group A is sent to the column selector array, the data of matrix B (k*n) stored in tensor register group B is sent to the row selector array, and the data of matrix C (m*n) stored in tensor register group C is sent to the adder array. The column selector array and the row selector array then send the corresponding data to the 3D multiplier array respectively to perform matrix multiplication calculations on multiple matrix elements at the same time. The calculation results are then sent to the adder array and added to the data of matrix C. After obtaining the MMA calculation accumulation result, it can be written back to the tensor register file, such as tensor register group C of the tensor register file.

[0057] The column selector array includes k columns of multiplexers, and the i-th column multiplexer will convert the i-th column data of matrix A into 1i ,a 2i ,…,a mi >Sent to the i-th array in the 3D multiplier array, i = {1, 2, ... k}. The row selector array includes k rows of multiplexers, and the i-th row multiplexer will send the i-th row data of matrix B to i1 ,b i2 ,…,b in >Sent to the i-th array in the 3D multiplier array, as described above, i = {1, 2, ... k}. The 3D multiplier array includes k arrays, and the i-th array is calculated by the outer product of the mxn-th MUL (multiplication) unit: MUL_Result mn = 1i ,a 2i ,…,a mi > T x i1 ,b i2 ,…,b in >. The adder array includes mxn adder units, the first<i,j> Adder ADD ij ​​​​The sum of the k arrays from the 3D multiplier array corresponding to the same multiplier position MUL_Result ij The data C in the matrix C of the input adder array ij Perform addition calculation (for example, in FIG4D , the MUL of the kth array 1n Combine the MUL of the corresponding positions in the k-1th array, k-2th array, ..., the 1st array 1n Add and get the sum result MUL_Result 1n , the MUL_Result 1n In the adder array, through ADD 1n Added to the data at the corresponding position in the matrix C); the tensor accumulator register Tensor ACC reduces all the values ​​in the adder array to form a tensor data ACC in row-major order m*n , and writes it back to the tensor register file. As can be seen, because the column selector array, row selector array, and 3D multiplier array are all 3D structures, they are easier to implement using a 3D stacking architecture, which can make the chip area smaller and save chip space.

[0058] The above content describes the MMA operation unit in the embodiment of the present disclosure from a hardware perspective, but the execution of MMA calculations requires not only hardware support, but also compatible ISA (Instruction Set Architecture) support. To this end, a preset instruction set compatible with the hardware implementation is provided in the embodiment of the present disclosure, so that the MMA operation unit can control at least one of the data loading storage engine, the tensor register file and the matrix multiplication engine (optionally, it can also include a memory) based on the machine instructions in the preset instruction set to perform the operation indicated by the machine instruction. The machine instruction carries information about the tensor register group to be used and information about the tensor registers of the tensor register group. According to different implementation functions, the preset instruction set is divided into: a data loading instruction subset, a matrix multiplication and accumulation instruction subset, and a data storage instruction subset. They are explained below.

[0059] (1) Data loading instruction subset

[0060] The instruction subset may include: an instruction for instructing to load matrix data from a memory into a tensor register of a corresponding tensor register group.

[0061] For example, the machine instructions in the instruction subset may be shown in Table 1 below:

[0062] (2) MMA instruction subset

[0063] The instruction subset may include: instructions for instructing to perform matrix multiplication and accumulation calculations based on matrix data stored in a tensor register.

[0064] For example, the machine instructions in the instruction subset may be shown in Table 2 below:

[0065] (3) Data storage instruction subset

[0066] The instruction subset includes: an instruction for instructing to store data of a tensor register storing a result of an MMA calculation into a memory.

[0067] For example, the machine instructions in the instruction subset may be shown in Table 3 below:

[0068] According to the solution of this embodiment, a tensor register file is set in the MMA operation unit, and the tensor register file is divided into at least three tensor register groups to store data of different matrices to be calculated by MMA. Therefore, when MMA calculation is required, the data loading and storage engine in the MMA operation unit can load and send the data of multiple matrices to be calculated to different tensor register groups respectively; on this basis, the matrix multiplication engine can read the data of different matrices from these tensor register groups at the same time to perform MMA calculation. Compared with the solution of storing matrix data in multiple memories, on the one hand, this method reads data from tensor registers instead of directly from memory, which not only improves data access efficiency but also avoids the power consumption caused by memory access data; on the other hand, since there is no need to use multiple memories to store data, the space occupied by them is also reduced; on the other hand, using a tensor register file also reduces the implementation cost of the solution compared to using multiple memories. Therefore, the solution of the embodiment of the present disclosure achieves more efficient MMA calculation under the conditions of low cost, low power consumption and less space occupation.

[0069] Based on the above hardware architecture, an MMA operation method according to an embodiment of the present disclosure is described below, and its steps are shown in FIG5 .

[0070] 5 , the MMA operation method includes the following steps:

[0071] Step S502: Loading data of multiple matrices to be subjected to MMA calculation into a tensor register file.

[0072] When the hardware accelerator described above, especially the MMA operation unit in the hardware accelerator, is used to implement the steps of this embodiment, this step can be implemented as follows: loading the data of multiple matrices to be subjected to MMA calculation into the tensor register file through the data loading storage engine in the MMA unit.

[0073] During a specific calculation, the data of multiple matrices to be subjected to matrix multiplication and accumulation calculations can be loaded into a tensor register file based on a first machine instruction. The first machine instruction carries information about the tensor register group to be used and information about the tensor registers in the tensor register group. For example, the machine instructions are shown in Table 1.

[0074] For example, after the data load storage engine in the MMA operation unit receives an instruction for instructing to read data, it will read the machine instructions for the data of multiple matrices to be calculated by MMA from the memory. Then, after the tensor register file receives the instruction for instructing to read data, i.e., the first machine instruction, it will read the relevant data from the data load storage engine and read it into each tensor register group of the tensor register file. The tensor register file includes at least three tensor register groups, each of which includes multiple tensor registers, and different tensor register groups are used to store data of different matrices in the multiple matrices. As shown in Figure 4B, the tensor register file includes three tensor register groups, each of which includes 8 tensor registers. However, it should be noted that Figure 4B is only an example. In actual applications, the tensor register file may include a larger number of tensor register groups to meet the needs of calculation. The number of tensor registers included in each tensor register group can also be more, or less, and can be set by those skilled in the art according to actual needs.

[0075] When a tensor register file is specifically stored, after receiving the data of multiple matrices, the data of each matrix is ​​stored in the tensor registers of the corresponding different tensor register groups according to the size of the bytes occupied by the data of each matrix and the rule of storing from the low bit.

[0076] Still using MMA to calculate the matrix A*B+C as an example, the data load storage engine reads multiple data in matrices A, B, and C from memory. Then, based on the triggering of the first machine instruction to read the data, the tensor register file reads the matrices A, B, C and data obtained by the data load storage engine into the corresponding tensor register groups A, B, and C. Tensor register groups A, B, and C each store the data of matrices A, B, and C through their respective tensor registers, according to the byte size of the data of each matrix, and according to the rule of storing from the lowest bit.

[0077] Step S504: performing MMA calculation based on the data of the multiple matrices stored in the tensor register file.

[0078] As mentioned above, when the aforementioned hardware accelerator, especially the MME operation unit therein, is used to execute the solution of this embodiment, this step can perform MMA calculations based on the data of multiple matrices stored in the tensor register through the matrix multiplication engine in the MME operation unit.

[0079] When performing specific calculations, the matrix multiplication engine can perform MMA calculations based on the data of multiple matrices stored in the tensor register file according to the second machine instruction. The second machine instruction carries information about the tensor register group to be used and information about the tensor registers of the tensor register group. For example, the instructions are shown in Table 2. The matrix multiplication engine can adopt the structure shown in Figure 4D, based on the column selector array, row selector array, multiplier array, and adder array, and based on the data of multiple matrices stored in the tensor register file, to perform MMA calculations.

[0080] This embodiment achieves efficient MMA computation. Compared to solutions that store matrix data in multiple memories, this embodiment reads data from tensor registers rather than directly from memory, improving data access efficiency while also avoiding the power consumption associated with memory access. Furthermore, eliminating the need for multiple memories to store data reduces the space occupied by these memories. Furthermore, using a single tensor register file reduces implementation costs compared to using multiple memories. Consequently, this embodiment achieves more efficient MMA computation while maintaining low cost, power consumption, and space.

[0081] It should be noted that the specific implementation description of each step in this embodiment is relatively simple. Those skilled in the art may refer to the relevant description in the aforementioned hardware embodiment when implementing the specific steps, and no further details will be given here.

[0082] Referring to Figure 6 , another embodiment of the present disclosure provides another MMA operation method. This method is based on a preset instruction set and describes the MMA operation method from the perspective of instruction execution. However, it should be noted that the execution of the aforementioned instructions is based on the MMA operation unit provided in the embodiment of the present disclosure, but those skilled in the art should understand that this instruction set can also be executed on other devices having the tensor register file of the embodiment of the present disclosure.

[0083] The MMA operation method of this embodiment includes the following steps:

[0084] Step S602: Obtain machine instructions for the matrix multiplication and accumulation operation process.

[0085] Generally speaking, a machine instruction includes an opcode and an operand. The opcode typically indicates the operation to be performed, such as data loading, data storage, or data calculation. The operand is the object of the opcode's operation. In this embodiment, the machine instruction includes information about the registers where the operands are located. The registers are registers in at least three tensor register groups in a tensor register file. Different tensor register groups are used to store data for different matrices in the multiple matrices to be calculated by matrix multiplication and accumulation.

[0086] Among them, the specific settings and implementations of the tensor register file, the tensor register group in the tensor register file, and the tensor registers in the tensor register group can all refer to the descriptions of the corresponding parts in the aforementioned embodiments, and will not be repeated here.

[0087] In this embodiment, the machine instructions include at least: load instructions, calculation instructions and storage instructions.

[0088] in:

[0089] A load instruction is used to instruct the loading of the matrix data to be subjected to the matrix multiplication and accumulation operation from the memory into the tensor register of the corresponding tensor register group; a calculation instruction is used to instruct the matrix multiplication and accumulation calculation to be performed based on the matrix data loaded in the tensor register; a storage instruction is used to instruct the storage of the data of the tensor register storing the result of the matrix multiplication and accumulation calculation into the memory.

[0090] but,

[0091] When the machine instruction is a load instruction, the operands in the load instruction include: the address of the memory where the data to be loaded is located, the identifier of the tensor register group into which the data is loaded, and the identifier of the tensor register in the tensor register group. Taking "LD_TensorA reg_id, mem_addr" in Table 1 as an example, "LD" is the opcode indicating a data load operation; "mem_addr" is the address of the memory where the data to be loaded is located; "TensorA" is the identifier of the tensor register group into which the data is loaded; and "reg_id" is the identifier of the tensor register in "TensorA".

[0092] When the machine instruction is a calculation instruction, the operands in the calculation instruction include: the identifier of the tensor register group where the data of the multiple matrices to be calculated by matrix multiplication and accumulation are located and the identifier of the tensor register in the tensor register group. Taking "IMMA reg_a_id,reg_b_id,reg_c_id,e_size,m,k,n" in Table 2 above as an example, "IMMA" is an opcode indicating an integer MMA calculation; "reg_a_id" represents A[a_id], where "A" is the identifier of the tensor register group, that is, TensorA, and "a_id" represents the identifier of the tensor register in the tensor register group, that is, the register with the identifier id in TensorA. "reg_b_id" and "reg_c_id" are similar to "reg_a_id" and will not be described in detail one by one.

[0093] When the machine instruction is a store instruction, the operands in the store instruction include: the identifier of the tensor register group where the data to be stored is located, and the identifier of the tensor register in the tensor register group. Taking "ST_TensorC reg_id,mem_addr" in Table 2 as an example, "ST" is the opcode indicating that data should be stored; "TensorC" is the identifier of the tensor register group where the data to be stored is located; and "reg_id" is the identifier of the specific register in the tensor register group where the data to be stored is located, that is, the register identified by "reg_id" in TensorC.

[0094] Step S604: Execute the operation indicated by the machine instruction.

[0095] After acquiring the aforementioned machine instruction, the corresponding device can perform the corresponding operation based on the machine instruction, such as performing the operation corresponding to the opcode on the operand according to the opcode in the machine instruction. As previously mentioned, in this embodiment, the corresponding device is not limited to the MMA arithmetic unit in the embodiment of the present disclosure, but can also be other devices.

[0096] However, if the machine instruction is still executed by the MMA operation unit in the embodiment of the present disclosure, then the aforementioned acquisition of the machine instruction for the matrix multiplication and accumulation operation process can be implemented by decoding the machine instruction for the matrix multiplication and accumulation operation process through the instruction decoder. In addition, the aforementioned execution of the operation indicated by the machine instruction can be implemented by controlling the matrix multiplication and accumulation operation unit to perform the operation indicated by the machine instruction according to the machine instruction.

[0097] As mentioned above, the matrix multiplication and accumulation unit includes: a data loading and storage engine, a tensor register file, and a matrix multiplication engine. Specifically, according to the machine instruction, controlling the matrix multiplication and accumulation unit to perform the operation indicated by the machine instruction will be implemented as follows:

[0098] When the machine instruction is a load instruction, the load instruction is sent to the data load storage engine to instruct the data load storage engine to load the data of the matrix to be subjected to the matrix multiplication and accumulation operation from the memory into the tensor register of the corresponding tensor register group;

[0099] When the machine instruction is a calculation instruction, the calculation instruction is sent to the matrix multiplication engine to instruct the matrix multiplication engine to perform a matrix multiplication and accumulation calculation based on the matrix data loaded in the tensor register;

[0100] When the machine instruction is a storage instruction, the storage instruction is sent to the data loading storage engine to instruct the data loading storage engine to store the data of the tensor register storing the result of the matrix multiplication and accumulation calculation into the memory.

[0101] The specific process of implementing the machine instruction by the MMA operation unit can refer to the description of the relevant parts in the above embodiment, which will not be repeated here.

[0102] It can be seen from this embodiment that the machine instructions suitable for MMA operations provided by the embodiment of the present disclosure can be generally applicable to any appropriate device with a corresponding tensor register file structure to implement MMA operations and improve MMA memory access and computing efficiency.

[0103] An embodiment of the present disclosure also provides a hardware accelerator, which includes at least an instruction decoder and a matrix multiplication and accumulation operation unit; wherein the instruction decoder is used to decode a machine instruction indicating a matrix multiplication and accumulation calculation, and control the matrix multiplication and accumulation operation unit to perform a matrix multiplication and accumulation calculation based on the decoding result.

[0104] An embodiment of the present disclosure further provides an electronic device, which includes at least: a processor and a hardware accelerator; wherein the processor is used to send an instruction to the hardware accelerator to instruct it to perform matrix multiplication and accumulation calculations; the hardware accelerator performs matrix multiplication and accumulation calculations according to the instruction.

[0105] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer execution is used to implement operations corresponding to the MME operation method in the above method embodiment when specified by the processor.

[0106] The embodiments of the present disclosure further provide a computer program product, including computer instructions, which instruct an electronic device to perform operations corresponding to the MME operation method in the above method embodiment through a hardware accelerator, especially an MME operation unit in the hardware accelerator.

[0107] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for calculation, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0108] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present disclosure can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present disclosure.

[0109] The above-mentioned method according to the embodiment of the present disclosure can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., random access memory (RAM), read-only memory (ROM), flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0110] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure.

[0111] The above implementation methods are only used to illustrate the embodiments of the present disclosure, and are not intended to limit the embodiments of the present disclosure. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the embodiments of the present disclosure. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present disclosure, and the scope of patent protection of the embodiments of the present disclosure should be defined by the claims.

Claims

1. A matrix multiplication and accumulation unit, comprising: Data loading and storage engine, tensor register file, matrix multiplication engine; in: The data loading and storage engine is used to load data of multiple matrices to be subjected to matrix multiplication and accumulation calculations; The tensor register file is used to store the data of the multiple matrices obtained from the data load storage engine; wherein the tensor register file includes at least three tensor register groups, each tensor register group includes a plurality of tensor registers, and different tensor register groups are used to store data of different matrices in the multiple matrices; The matrix multiplication engine is used to perform matrix multiplication and accumulation calculations based on the data of multiple matrices stored in the tensor register file.

2. The matrix multiplication and accumulation unit according to claim 1, wherein: Also includes: Memory; The memory is used to store data of multiple matrices to be subjected to matrix multiplication and accumulation calculations, obtained from outside the matrix multiplication and accumulation operation unit, so as to provide the data to the data loading and storage engine.

3. The matrix multiplication and accumulation unit according to claim 1 or 2, wherein: The tensor register file is used to receive the data of the multiple matrices, and store the data of each matrix into the tensor registers of the corresponding different tensor register groups according to the numerical precision of each matrix and the rule of storing from the low bit.

4. The matrix multiplication and accumulation unit according to claim 3, wherein: The tensor register file is used to receive the data of the multiple matrices, and for each matrix data, store the matrix data into the tensor register of the corresponding tensor register group starting from the low bit of the tensor register according to the numerical precision corresponding to the data type.

5. The matrix multiplication and accumulation unit according to any one of claims 1 to 4, wherein: The matrix multiplication and accumulation operation unit controls at least one of the data loading and storage engine, the tensor register file, and the matrix multiplication engine based on machine instructions in a preset instruction set to perform the operation indicated by the machine instructions; the machine instructions carry information about the tensor register group to be used and information about the tensor registers of the tensor register group.

6. The matrix multiplication and accumulation unit according to any one of claims 1 to 4, wherein: The matrix multiplication engine performs matrix multiplication and accumulation calculations based on the data of multiple matrices stored in the tensor register file through a column selector array, a row selector array, a multiplier array and an adder array.

7. A matrix multiplication and accumulation method, comprising: Obtaining a machine instruction for a matrix multiplication-accumulation operation process, the machine instruction including information about registers where operands are located, the registers being registers in at least three tensor register groups in a tensor register file, different tensor register groups being used to store data of different matrices among a plurality of matrices to be subjected to the matrix multiplication-accumulation operation; Execute the operation indicated by the machine instruction.

8. The method according to claim 7, wherein: The machine instructions include: A load instruction is used to instruct to load the data of the matrix to be subjected to the matrix multiplication and accumulation operation from the memory into the tensor register of the corresponding tensor register group; Computation instructions, used to instruct matrix multiplication and accumulation calculations based on the matrix data loaded in the tensor register; The store instruction is used to instruct to store the data of the tensor register storing the result of the matrix multiplication and accumulation calculation to the memory.

9. The method according to claim 8, wherein When the machine instruction is a load instruction, the operands in the load instruction include: an address of a memory where data to be loaded is located, an identifier of a tensor register group into which the data is loaded, and an identifier of a tensor register in the tensor register group; When the machine instruction is a calculation instruction, the operands in the calculation instruction include: an identifier of a tensor register group where data of a plurality of matrices to be subjected to matrix multiplication and accumulation calculations are located and an identifier of a tensor register in the tensor register group; When the machine instruction is a store instruction, the operand in the store instruction includes: an identifier of a tensor register group where the data to be stored is located and an identifier of a tensor register in the tensor register group.

10. The method according to claim 8 or 9, wherein: The obtaining of the machine instruction for the matrix multiplication and accumulation operation process comprises: decoding and obtaining the machine instruction for the matrix multiplication and accumulation operation process through an instruction decoder; The executing the operation indicated by the machine instruction includes: controlling a matrix multiplication and accumulation operation unit to execute the operation indicated by the machine instruction according to the machine instruction.

11. The method according to claim 10, wherein: The matrix multiplication and accumulation operation unit includes: a data loading and storage engine, a tensor register file, and a matrix multiplication engine; According to the machine instruction, controlling the matrix multiplication and accumulation operation unit to perform the operation indicated by the machine instruction includes: When the machine instruction is a load instruction, sending the load instruction to the data load storage engine to instruct the data load storage engine to load the data of the matrix to be subjected to the matrix multiplication and accumulation operation from the memory into the tensor register of the corresponding tensor register group; When the machine instruction is a calculation instruction, sending the calculation instruction to the matrix multiplication engine to instruct the matrix multiplication engine to perform a matrix multiplication and accumulation calculation based on the matrix data loaded in the tensor register; When the machine instruction is a store instruction, the store instruction is sent to the data load storage engine to instruct the data load storage engine to store the data of the tensor register storing the result of the matrix multiplication and accumulation calculation into the memory.

12. A matrix multiplication and accumulation method, comprising: Loading data of a plurality of matrices to be subjected to matrix multiplication and accumulation calculations into a tensor register file, wherein the tensor register file includes at least three tensor register groups, each tensor register group includes a plurality of tensor registers, and different tensor register groups are used to store data of different matrices in the plurality of matrices; Matrix multiplication and accumulation calculations are performed based on the data of the multiple matrices stored in the tensor register file.

13. A hardware accelerator comprising at least an instruction decoder and a matrix multiplication-accumulation unit as claimed in any one of claims 1 to 6; in, The instruction decoder is used to decode the machine instruction indicating the matrix multiplication and accumulation calculation, and control the matrix multiplication and accumulation operation unit to perform the matrix multiplication and accumulation calculation according to the decoding result.

14. An electronic device comprising at least: A processor and a hardware accelerator as claimed in claim 13; The processor is configured to send an instruction to the hardware accelerator to perform matrix multiplication and accumulation calculations; and the hardware accelerator performs matrix multiplication and accumulation calculations according to the instruction.

15. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions, when specified by a processor, are used to implement the method according to any one of claims 7 to 12.

16. A computer program product comprising computer instructions, wherein the computer instructions instruct an electronic device to execute the method according to any one of claims 7 to 12 through the hardware accelerator according to claim 13, in particular a matrix multiplication and accumulation unit in the hardware accelerator.

Images