An outer product-based matrix multiplication operation processor, method, and medium

Abstract

The application provides an outer product-based matrix multiplication operation processor, method and medium, which are applied to a GPGPU. The processor comprises a register stack, a matrix buffer and a calculation unit. The register stack is used for storing to-be-processed matrices of different data types. The matrix buffer is used for caching each to-be-processed matrix in a preset arrangement mode. The calculation unit is used for performing outer product operation and accumulation operation on each to-be-processed matrix of different data types to obtain a target matrix, and writing the target matrix back to the matrix buffer. The outer product-based matrix multiplication operation processor provided by the application proposes a general design scheme for a low-precision operation mode commonly used in neural network calculation, can support matrix multiplication operation operations of different precisions and corresponding sizes, and increases on-chip storage and a calculation unit in a calculation core to improve operation efficiency.

Description

Technical Field

[0001] This application relates to the technical field of general-purpose processors, and specifically to a processor, method, and medium for matrix multiplication operations based on outer product. Background Technology

[0002] Deep neural networks (DNNs) have numerous applications in speech recognition, image recognition, computer vision, recommender systems, natural language processing, and brain-computer interfaces. DNNs require the execution of a large number of multidimensional matrix (or tensor) calculations. In convolutional neural networks (CNNs), such as AlexNet, VGGNet, and ResNet, the main computational cost is in multi-layer convolution operations. There are various methods for convolution calculations, with the mainstream approaches including direct convolution, im2col-gemm convolution, Winograd convolution, and FFT convolution. Among these, the im2col-gemm convolution scheme is commonly used on GPGPUs because of the design characteristics of GPUs and their significant advantages in supporting and optimizing matrix multiplication operations over many years.

[0003] In GPGPU, the convolution operations in CNNs and the attention modules in Transformer models can both be unified into GEMM operations. Therefore, the computational focus of deep learning models on GPGPU is how to optimize GEMM operations. Mathematically, matrix multiplication can be performed in two different ways: inner product method and outer product method.

[0004] GPUs are the common choice for deep learning computations because they offer orders of magnitude higher performance compared to CPUs. Much work has been dedicated to optimizing GPU hardware and software to accelerate tensor operations in DNNs. Modern GPU architectures consist of a number of streaming multiprocessors (SMs), which serve as the core processing units of the GPU. These SMs are further connected to GPU device memory / global memory, which is shared by all SMs. Each SM has its own internal cache, shared memory, register file, floating-point core, load memory, and other specialized units.

[0005] On the software side, many excellent parallel algorithm designs have been optimized for matrix multiplication, especially NVIDIA's related mathematical computing libraries, such as cuBlas, cuDNN, and CUTLASS. On the hardware architecture side, NVIDIA has also proposed Tensor Cores, core units specifically designed to accelerate matrix multiplication. Compared to ordinary floating-point cores, Tensor Cores offer significantly improved computational throughput. Furthermore, Tensor Cores support various precision and mixed-precision operations, further enhancing their throughput. However, these related hardware and software designs and calculations are primarily based on the principle of inner product matrix multiplication; there is currently no hardware architecture design in GPGPUs specifically for outer product matrix multiplication.

[0006] In the research and development of deep learning, the sparsity of neural networks is increasingly being used as a common technique to accelerate computation. NVIDIA proposed a structured 2:4 sparse Tensor Core to accelerate computation, but due to the limitations of its inner product calculation, it only supports one-sided sparsity. The outer product matrix multiplication is different; due to its mathematical principles, it provides more flexible support for sparse computation, including both one-sided and two-sided sparsity. Therefore, designing a matrix multiplication core based on outer products in GPGPUs is essential. Summary of the Invention

[0007] The purpose of this application is to provide a processor, method and medium for matrix multiplication based on outer product, to solve the problem that the existing technology cannot perform matrix outer product operations in GPGPU through hardware devices.

[0008] In a first aspect, this application provides a matrix multiplication processor based on outer product, applied in a GPGPU. The processor includes: a register file, a matrix buffer, and a computation unit; the register file is used to store matrices to be processed; the matrices to be processed include: a first matrix to be processed, a second matrix to be processed, and a third matrix to be processed; the matrix buffer is used to cache each of the matrices to be processed according to a preset arrangement; the computation unit is used to perform outer product operations and accumulation operations on each of the matrices to be processed of different types to obtain a target matrix, and write the target matrix back to the matrix buffer.

[0009] In one implementation of the first aspect, the processor includes multiple thread bundles, and a single outer product operation within any thread bundle is completed jointly by all threads within the thread bundle; the elements of each matrix to be processed within any thread bundle are shared.

[0010] In one implementation of the first aspect, any of the computing units performs an outer product operation on the rows of the first matrix to be processed and the columns of the second matrix to be processed according to the data type of the matrix to be processed, to obtain a temporary matrix; and performs an accumulation operation on the temporary matrix and the third matrix to be processed to obtain the rows of the target matrix.

[0011] Furthermore, any one of the computing units includes multiple MAC units; the MAC units are used to perform outer product operations on each of the matrices to be processed of different data types and then perform single or multiple accumulation operations to obtain the target matrix.

[0012] In one implementation of the first aspect, the MAC unit is applicable to multi-precision matrix operations. The MAC unit performs external product operations and accumulation operations on each of the matrices to be processed based on the general computing pattern DTYPE-STYPE-MNK-MUL. Here, DTYPE represents different data types, STYPE represents different precision levels, MNK represents the shape and size of the input matrix, and MUL represents the number of parallel multiplications performed by a single thread in a single operation.

[0013] In one implementation of the first aspect, any of the MAC units includes: a multi-precision multiplier and at least one adder; the multi-precision multiplier is used to determine the number of rows and columns of each of the matrices to be processed that require a single outer product operation based on the precision of each of the matrices to be processed; the adder is used to write the result of the single outer product operation directly back to the matrix buffer or to write it back to the matrix buffer after performing a single or multiple accumulation operations.

[0014] Furthermore, each of the MAC units also includes a controller, which is used to determine the number of accumulation operations based on the precision of each of the matrices to be processed.

[0015] Secondly, this application provides a matrix multiplication method based on outer product, which is applied to the matrix multiplication processor based on outer product as described in any of the first aspects of this application; wherein, the method includes: acquiring and storing a matrix to be processed, and caching the matrix to be processed in a matrix buffer according to a preset arrangement; performing outer product operation and accumulation operation on the matrix to be processed according to the data type of the matrix to be processed to obtain a target matrix, and writing the target matrix back to the matrix buffer.

[0016] In one implementation of the second aspect, the step of performing outer product and accumulation operations on the matrix to be processed according to the data type of the matrix to be processed includes: determining the data type and data precision of the matrix to be processed; determining the number of rows and columns to be performed in a single outer product operation based on the data type and data precision, and performing the outer product operation; determining the number of addition operations based on the data type and data precision, and writing the result of the outer product operation directly back to the matrix buffer or writing it back to the matrix buffer after one or more addition operations.

[0017] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the matrix multiplication method based on outer product as described in any of the second aspects of this application.

[0018] As described above, the matrix multiplication processor, method, and medium based on outer product described in this application have the following characteristics:

[0019] Beneficial effects:

[0020] First, this application proposes a matrix multiplication operation core based on outer product to accelerate matrix multiplication operations in GPGPU. The efficiency of the operation is improved by adding on-chip storage and computing units to the computing core.

[0021] Second, this application proposes a general design scheme for the low-precision operation methods commonly used in neural network computing, which can support matrix multiplication operations of different precisions and corresponding sizes. Attached Figure Description

[0022] Figure 1 The diagram shown is a schematic representation of the structure of the matrix multiplication processor based on outer product as described in an embodiment of this application.

[0023] Figure 2 This is the preset arrangement of the matrix buffer described in the embodiments of this application.

[0024] Figure 3 The diagram shows the detailed steps of the first outer product operation as described in the embodiments of this application.

[0025] Figure 4 The diagram shown is a structural schematic of multiple MAC units within a computing unit as described in an embodiment of this application.

[0026] Figure 5 The diagram shown is a structural schematic of the MAC unit described in an embodiment of this application.

[0027] Figure 6 The flowchart shown is a matrix multiplication operation method based on outer product as described in the embodiments of this application.

[0028] Figure 7 The flowchart shown is a matrix multiplication operation method based on outer product as described in the embodiments of this application.

[0029] Component designation explanation

[0030] 10 Register File

[0031] 20 Matrix Buffers

[0032] 201 Matrix Buffer A

[0033] 202 Matrix Buffer B

[0034] 203 Matrix Buffer C

[0035] 204 Matrix Buffer D

[0036] 30 computing units

[0037] 301 MAC Unit

[0038] 3011 Multiprecision Multiplier

[0039] 3012 Adder

[0040] 3013 Controller

[0041] Steps S1 to S2

[0042] Steps S21 to S23 Detailed Implementation

[0043] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0044] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0045] The following embodiments of this application provide a matrix multiplication processor, method, and medium based on outer product operations. The outer product-based matrix multiplication processor is applied in a GPGPU. The processor includes: a register file, a matrix buffer, and a computation unit. The register file stores matrices to be processed. The matrices to be processed include: a first matrix to be processed, a second matrix to be processed, and a third matrix to be processed. The matrix buffer caches each matrix to be processed according to a preset arrangement. The computation unit performs outer product operations and accumulation operations on different types of matrices to be processed to obtain a target matrix and writes the target matrix back to the matrix buffer. This application solves the problem in the prior art that GPGPUs cannot perform matrix outer product operations through hardware devices.

[0046] The following embodiments of this application provide a hardware architecture for matrix multiplication based on outer product, including: a register file, a matrix buffer, and a computing unit.

[0047] The register file stores the first matrix to be processed, the second matrix to be processed, and the third matrix to be processed. When matrix multiplication is required, the first matrix to be processed, the second matrix to be processed, and the third matrix to be processed are cached in matrix cache A, matrix cache B, and matrix cache C according to the arrangement of each matrix cache. According to the arrangement of each matrix cache, the data in each matrix cache is called for the calculation unit to perform outer product operation and addition operation. Finally, the operation result is written back to the preset matrix cache D to obtain the target matrix.

[0048] In this system, any computing unit shares all elements of the matrix to be processed within matrix buffer A, matrix buffer B, and matrix buffer C.

[0049] It should be noted that each outer product operation is completed jointly by all threads within the thread bundle.

[0050] It's important to note that in GPGPU computation, multiple threads are distributed across the SM (Scheduled Threading) and concurrently execute their unique operations. From a thread scheduling perspective, these threads are executed uniformly at the thread bundle granularity. Typically, the thread bundle size is 32 or 64. It's crucial to understand that during thread execution, before and after using OMMA Core hardware resources, the original SIMT scheduling method remains intact. However, when calling OMMA Core, resource scheduling should be performed at the full thread bundle size, avoiding branching operations within the thread bundle; otherwise, execution errors will occur.

[0051] The following will describe in detail, with reference to the accompanying drawings, the principle and implementation of a matrix multiplication processor, method and medium based on outer product in this embodiment, so that those skilled in the art can understand the matrix multiplication processor, method and medium based on outer product in this embodiment without creative effort.

[0052] refer to Figure 1 The diagram shows the structure of a matrix multiplication processor based on outer product as described in an embodiment of this application. The processor includes: a register file 10, a matrix buffer, and a computation unit 30.

[0053] Specifically, register file 10 is used to store matrices to be processed; the matrices to be processed include: a first matrix to be processed, a second matrix to be processed, and a third matrix to be processed; matrix buffer 20 is used to buffer each matrix to be processed according to a preset arrangement; calculation unit 30 is used to perform outer product operation and accumulation operation on each matrix to be processed of different types to obtain the target matrix, and write the target matrix back to matrix buffer 20.

[0054] The matrix buffer 20 includes: matrix buffer A 201, matrix buffer B 202, matrix buffer C 203 and matrix buffer D 204; wherein matrix buffer A 201 is used to receive and buffer the first matrix to be processed, matrix buffer B 202 is used to receive and buffer the second matrix to be processed, matrix buffer C 203 is used to receive and buffer the third matrix to be processed, and matrix buffer D 204 is used to receive and buffer the target matrix to be written back.

[0055] In one embodiment, such as Figure 2 As shown, this is the preset arrangement of the matrix buffer described in the embodiments of this application.

[0056] Specifically, according to a preset arrangement, the first matrix to be processed is cached in matrix cache A 201, the second matrix to be processed is cached in matrix cache B 202, and the third matrix to be processed is cached in matrix cache C 203. Matrix cache D 204 is used to cache the target matrix to be written back. Taking the first matrix to be processed being cached in matrix cache A 201 as an example, the value in T0 represents the first 8 elements of the first row of the first matrix to be processed, and T1 represents the last 8 elements of the first row of the first matrix to be processed. The second matrix to be processed is being cached in matrix cache B 202, where the value in T0 represents the first 8 elements of the first column of the second matrix to be processed, and T2 represents the last 8 elements of the first column of the second matrix to be processed.

[0057] Furthermore, the processor includes multiple thread bundles, and a single outer product operation within any thread bundle is completed jointly by all threads within the thread bundle; the elements of each matrix to be processed within any thread bundle are shared.

[0058] Furthermore, any computing unit 30 performs an outer product operation on the rows of the first matrix to be processed and the columns of the second matrix to be processed according to the data type of the matrix to be processed, so as to obtain a temporary matrix; and performs an accumulation operation on the temporary matrix and the third matrix to be processed to obtain the rows of the target matrix.

[0059] In one embodiment, such as Figure 3 The diagram shows the detailed steps of the first outer product operation described in the embodiment of this application.

[0060] Specifically, taking the matrices to be processed with data type FP32 as an example, this operation takes the first column v_A of the first matrix A to be processed and the first row v_B of the second matrix B to be processed. v_A has 16 elements, and each element of v_A is shared by two threads. For example, the first element of v_A is shared by thread 0 and thread 1, and can be accessed by both threads simultaneously. The second element of v_A is shared by thread 2 and thread 3, and so on. The 16th element of v_A is shared by thread 30 and thread 31. v_B also has 16 elements; the first 8 elements are shared by even-numbered threads (threads 0, 2, 4, 6, 8…), and the last 8 elements are shared by odd-numbered threads (threads 1, 3, 5, 7, 9…).

[0061] Taking the calculations of threads 0 and 1 as an example, threads 0 and 1 simultaneously take the first element of the first matrix A to be processed, as well as a complete row of v_B. In this row, the first 8 elements belong to thread 0, and the last 8 elements belong to thread 1. After obtaining the input required for the operation, taking thread 0 as an example, v_A[0] and the 8 elements of v_B[0] to v_B[7] are multiplied in parallel to obtain the result of the first 8 elements of the first row in the Temp matrix. Similarly, thread 1 obtains the result of the last 8 elements of the first row in the Temp matrix. By analogy, 32 threads can obtain the values ​​of all elements of a Temp matrix in this way (for example, thread 2 calculates the first 8 elements of the second row of the Temp matrix, etc.).

[0062] After that, the Temp matrix is ​​added to the third matrix C to be processed. Taking thread 0 as an example, the addition calculation method is to perform parallel addition operations on the eight elements obtained by thread 0 in the Temp calculation and the corresponding eight elements of the third matrix C to be processed, so as to obtain the first eight elements of the first row of the target matrix D_0. The 32 threads perform parallel operations to obtain all the values ​​of the final target matrix D_0.

[0063] Furthermore, any computing unit 30 includes multiple MAC units 301; the MAC units 301 are used to perform outer product operations on each matrix to be processed of different data types and then perform single or multiple accumulation operations to obtain the target matrix.

[0064] like Figure 4 As shown in one embodiment, a schematic diagram of the structure of a plurality of MAC units 301 within a computing unit 30 is presented.

[0065] Specifically, taking each matrix to be processed as an example with data type FP32, the number of multiplication operations performed by a single thread in parallel is 8.

[0066] Furthermore, the MAC unit 301 is applicable to multi-precision matrix operations. The MAC unit 301 performs external product operations and accumulation operations on each matrix to be processed based on the general computing mode DTYPE-STYPE-MNK-MUL. Here, DTYPE represents different data types, STYPE represents different precision levels, MNK represents the shape and size of the input matrix, and MUL represents the number of parallel multiplications in a single thread.

[0067] Table 1 shows the general calculation scheme for multi-precision matrix operations in the embodiments of this application.

[0068]

[0069] Specifically, taking FP32 as an example, if the data type of each matrix to be processed is FP32 and the data precision is FP16, then the size of each matrix to be processed is 16×16, and the number of multiplication operations performed by a single thread in parallel is 8.

[0070] Furthermore, any MAC unit 301 includes: a multi-precision multiplier 3011 and at least one adder 3012.

[0071] Specifically, the multi-precision multiplier 3011 is used to determine the number of rows and columns of each matrix to be processed that need to be subjected to a single outer product operation based on the precision of each matrix to be processed; the adder 3012 is used to write the result of the single outer product operation directly back to the matrix buffer or to write it back to the matrix buffer after performing a single or multiple accumulation operations.

[0072] In one embodiment, such as Figure 5 The diagram shown is a schematic representation of the MAC unit according to an embodiment of this application. The MAC unit 301 includes a multi-precision multiplier 3011 and multiple adders 3012.

[0073] Referring to Table 1, in this embodiment, when the matrix to be processed is FP16 precision, one column of the first matrix to be processed and one row of the second matrix to be processed are selected for calculation. When the matrix is ​​INT8 precision, two columns of the first matrix to be processed and two rows of the second matrix to be processed are selected for simultaneous calculation, and so on. Regardless of the data type of the input, the first matrix to be processed before entering the MAC unit 301 is 16 bits. When STYPE is FP16, in_A represents one element in the first matrix to be processed, and in_B0 represents one element in the second matrix to be processed. When STYPE is INT8, in_A represents two consecutive elements in the same row of the first matrix to be processed, and in_B0 represents two consecutive elements in the same column of the second matrix to be processed. This matches the two columns of the first matrix to be processed and the two rows of the second matrix to be processed mentioned above. The arrangement of in_A and in_B0 when STYPE is INT4 can be deduced by analogy. After passing through the multi-precision multiplier 3011, one, two, or four results can be obtained. Since they are partial sums of the same point in the result matrix, they can be accumulated.

[0074] Furthermore, in one embodiment, any MAC unit 301 further includes a controller 3013, which is used to determine the number of accumulation operations based on the precision of each matrix to be processed.

[0075] Specifically, such as Figure 5 As shown, the calculation method is controlled according to the DTYPE and STYPE types used in this calculation. For example, under the FP16 configuration, the result of one element is directly transmitted. Under the INT8 configuration, two elements should be added together before transmitting the result, and so on.

[0076] In one embodiment, this application also provides an instruction set for matrix multiplication based on outer product, as shown in Table 2.

[0077]

[0078] Specifically, this set of instructions describes the OMMA Core calculation scheme for different data precisions. OPMA is the opcode, indicating that this instruction needs to call the OMMA Core for calculation. .DTYPE.STYPE indicates the data precision for this calculation. Rd is the destination register number, indicating the location where the target matrix will be stored. Ra, Rb, and Rc are the source register numbers, representing the data at the corresponding positions of the first, second, and third matrices to be processed, respectively. Each instruction should be executed in parallel at the thread bundle granularity; otherwise, calculation errors will occur.

[0079] It should be noted that the above division of modules is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, module x can be a separate processing element, or it can be integrated into a chip in the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and its function can be called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0080] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Furthermore, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Additionally, these modules can be integrated together as a system-on-a-chip (SOC). The matrix multiplication method based on outer products described in this application is not limited to the execution order of the steps listed in the above embodiments. Any solutions implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this invention are included within the scope of protection of this invention.

[0081] like Figure 6 As shown, this illustrates the matrix multiplication method based on outer product as described in an embodiment of this application.

[0082] Specifically, the outer product-based matrix multiplication processor described in any embodiment of this application is applied. The outer product-based matrix multiplication method includes:

[0083] S1. Obtain and store the matrix to be processed, and cache the matrix to be processed in the matrix cache according to the preset arrangement.

[0084] Specifically, the register retrieves the matrix to be processed from main memory, calculates the position of each element in the matrix to be processed in the matrix cache according to the preset arrangement of the matrix cache, and loads the elements in each matrix to be processed into the corresponding matrix cache through instructions.

[0085] It should be noted that if the size of the matrix to be processed exceeds the capacity of the matrix buffer, the matrix to be processed needs to be divided into multiple basic blocks that meet the requirements of the matrix buffer for separate reading and caching.

[0086] S2. Based on the data type of the matrix to be processed, perform outer product and accumulation operations on the matrix to be processed to obtain the target matrix, and write the target matrix back to the matrix buffer.

[0087] Specifically, the matrices to be processed include: a first matrix to be processed, a second matrix to be processed, and a third matrix to be processed; the matrix buffers include: a first matrix buffer, a second matrix buffer, a third matrix buffer, and a target matrix buffer.

[0088] Specifically, in one embodiment, such as Figure 7 As shown, step S2 includes:

[0089] S21. Determine the data type and precision of the matrix to be processed.

[0090] S22. Determine the number of rows and columns to be used in a single outer product operation based on the data type and data precision, and then perform the outer product operation.

[0091] Specifically, the data types of the matrices to be processed include: FP32, FP16, INT32, UINT32, etc.

[0092] Taking the data type of the matrix to be processed as FP16 as an example, and its data precision as FP16, then during the first outer product operation, a column of the first matrix to be processed and a row of the second matrix B to be processed are selected for the outer product operation, and the result of the outer product operation is returned to the preset position in the target matrix buffer.

[0093] It should be noted that when performing outer product operations on a matrix of data type FP16, a single thread performs 8 multiplication operations in parallel at a time; a single thread needs to perform 16 parallel multiplication operations.

[0094] S23. Determine the number of addition operations based on the data type and data precision, and write the result of the outer product operation directly back to the matrix buffer or write it back to the matrix buffer after one or more addition operations.

[0095] Specifically, regardless of the input data type, the first matrix A to be processed before entering the MAC unit 301 is 16 bits. When STYPE is FP16, in_A represents one element in the first matrix A, and in_B0 represents one element in the second matrix B. When STYPE is INT8, in_A represents two consecutive elements in the same row of the first matrix, and in_B0 represents two consecutive elements in the same column of the second matrix. This matches the two columns of the first matrix and the two rows of the second matrix. The arrangement of in_A and in_B0 can be deduced similarly when STYPE is INT4. After passing through the multi-precision multiplier 3011, one, two, or four results can be obtained. Since they are partial sums of the same point in the result matrix, they can be accumulated.

[0096] This embodiment provides an electronic device, which includes at least a memory and a processor connected via a bus. The memory stores a computer program, and the processor executes the computer program stored in the memory to perform all or part of the steps in the aforementioned method embodiment.

[0097] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0098] This application embodiment may also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of this application are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0099] When the computer program product is executed by a computer, the computer performs the methods described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing methods are required, the computer program product can be downloaded and executed on the computer.

[0100] In summary, compared with existing technologies, the outer product-based matrix multiplication processor, method, and medium of this application offer several advantages. First, this application proposes an outer product-based matrix multiplication core to accelerate matrix multiplication operations within a GPGPU, improving operational efficiency by adding on-chip storage and computing units to the core. Second, this application proposes a general design scheme to address the low-precision computation methods commonly used in neural network computing, supporting matrix multiplication operations of varying precision and sizes. Therefore, this application effectively overcomes the various shortcomings of existing technologies and possesses significant industrial applicability.

[0101] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A matrix multiplication processor based on outer product, applied in a GPGPU, characterized in that, The processor includes: a register file, a matrix cache, and a computing unit; The register file is used to store matrices to be processed; the matrices to be processed include: a first matrix to be processed, a second matrix to be processed, and a third matrix to be processed. The matrix buffer is used to cache each of the matrices to be processed according to a preset arrangement. The computing unit is used to perform outer product and accumulation operations on each of the matrices to be processed of different types to obtain the target matrix, and write the target matrix back to the matrix buffer; The matrix buffer includes matrix buffer A, matrix buffer B, matrix buffer C and matrix buffer D. Matrix buffer A is used to buffer the first matrix to be processed, matrix buffer B is used to buffer the second matrix to be processed, matrix buffer C is used to buffer the third matrix to be processed, and matrix buffer D is used to buffer the target matrix. The preset arrangement method includes: grouping the first matrix to be processed along the row direction according to consecutive elements, and caching each group of data to the corresponding position in the matrix cache A; grouping the second matrix to be processed along the column direction according to consecutive elements, and caching each group of data to the corresponding position in the matrix cache B; caching the third matrix to be processed to the corresponding position in the matrix cache C; when the size of any matrix to be processed exceeds the capacity of the corresponding matrix cache, the matrix to be processed is divided into multiple basic blocks that meet the capacity requirements of the corresponding matrix cache, and read and cached respectively. The different types are those defined by both data type DTYPE and data precision STYPE, and the combination of data type DTYPE and data precision STYPE includes at least FP32-FP16, FP16-FP16, INT32-INT8, INT32-INT4, UINT32-UINT8, and UINT32-UINT4; The computing unit is used to perform outer product and accumulation operations on each of the matrices to be processed according to the general computing mode DTYPE-STYPE-MNK-MUL, where MNK represents the shape and size of the input matrix and MUL represents the number of parallel multiplications in a single thread. The computing unit reads data for a single outer product operation from the first and second matrices to be processed according to the MNK corresponding to the data type DTYPE and the data precision STYPE, performs parallel multiplication operations according to the corresponding MUL to obtain a temporary matrix, determines the number of accumulation operations according to the data precision STYPE, and directly accumulates the temporary matrix with the corresponding data of the third matrix to be processed or performs one or more accumulation operations to obtain the target matrix, and writes the target matrix back to the matrix buffer D.

2. The matrix multiplication processor based on outer product according to claim 1, characterized in that, The processor includes multiple thread bundles, and a single outer product operation within any thread bundle is completed jointly by all threads within the thread bundle. Elements of the matrices to be processed within any thread bundle are shared.

3. The matrix multiplication processor based on outer product according to claim 1, characterized in that, Each of the computing units performs an outer product operation on the rows of the first matrix to be processed and the columns of the second matrix to be processed according to the data type of the matrix to be processed, so as to obtain a temporary matrix; The temporary matrix and the third matrix to be processed are summed to obtain the rows of the target matrix.

4. The matrix multiplication processor based on outer product according to claim 3, characterized in that, Each of the aforementioned computing units includes multiple MAC units; The MAC unit is used to perform an outer product operation on each of the matrices to be processed of different data types, and then perform a single or multiple accumulation operation to obtain the target matrix.

5. The matrix multiplication processor based on outer product according to claim 4, characterized in that, The MAC unit is applicable to multi-precision matrix operations. The MAC unit performs external product and accumulation operations on each matrix to be processed based on the general computing mode DTYPE-STYPE-MNK-MUL. Here, DTYPE represents different data types, STYPE represents different precision levels, MNK represents the shape and size of the input matrix, and MUL represents the number of parallel multiplications in a single thread.

6. The matrix multiplication processor based on outer product according to claim 4, characterized in that, Each of the MAC units includes: a multi-precision multiplier and at least one adder; The multi-precision multiplier is used to determine the number of rows and columns of each matrix to be processed that requires a single outer product operation based on the precision of each matrix to be processed. The adder is used to write the result of a single outer product operation directly back to the matrix buffer, or to perform a single or multiple accumulation operations and then write it back to the matrix buffer.

7. The matrix multiplication processor based on outer product according to claim 6, characterized in that, Each of the MAC units further includes a controller, which is used to determine the number of accumulation operations based on the precision of each of the matrices to be processed.

8. A matrix multiplication method based on outer product, characterized in that, The processor for outer product-based matrix multiplication operations according to any one of claims 1-7; wherein the method comprises: Acquire and store the matrix to be processed, and cache the matrix to be processed in a matrix cache according to a preset arrangement; According to the data type of the matrix to be processed, the outer product operation and accumulation operation are performed on the matrix to be processed to obtain the target matrix, and the target matrix is ​​written back to the matrix buffer; The matrix buffer includes matrix buffer A, matrix buffer B, matrix buffer C and matrix buffer D. Matrix buffer A is used to buffer the first matrix to be processed, matrix buffer B is used to buffer the second matrix to be processed, matrix buffer C is used to buffer the third matrix to be processed, and matrix buffer D is used to buffer the target matrix. The preset arrangement method includes: grouping the first matrix to be processed along the row direction according to consecutive elements, and caching each group of data to the corresponding position in the matrix cache A; grouping the second matrix to be processed along the column direction according to consecutive elements, and caching each group of data to the corresponding position in the matrix cache B; caching the third matrix to be processed to the corresponding position in the matrix cache C; when the size of any matrix to be processed exceeds the capacity of the corresponding matrix cache, the matrix to be processed is divided into multiple basic blocks that meet the capacity requirements of the corresponding matrix cache, and read and cached respectively. The different types are those defined by both data type DTYPE and data precision STYPE, and the combination of data type DTYPE and data precision STYPE includes at least FP32-FP16, FP16-FP16, INT32-INT8, INT32-INT4, UINT32-UINT8, and UINT32-UINT4; The computing unit is used to perform outer product and accumulation operations on each of the matrices to be processed according to the general computing mode DTYPE-STYPE-MNK-MUL, where MNK represents the shape and size of the input matrix and MUL represents the number of parallel multiplications in a single thread. The computing unit reads data for a single outer product operation from the first and second matrices to be processed according to the MNK corresponding to the data type DTYPE and the data precision STYPE, performs parallel multiplication operations according to the corresponding MUL to obtain a temporary matrix, determines the number of accumulation operations according to the data precision STYPE, and directly accumulates the temporary matrix with the corresponding data of the third matrix to be processed or performs one or more accumulation operations to obtain the target matrix, and writes the target matrix back to the matrix buffer D.

9. The matrix multiplication method based on outer product according to claim 8, characterized in that, The step of performing outer product and accumulation operations on the matrix to be processed according to the data type of the matrix to be processed includes: Determine the data type and precision of the matrix to be processed; The number of rows and columns to be used in a single outer product operation is determined based on the data type and data precision, and the outer product operation is then performed. The number of addition operations is determined based on the data type and data precision. The result of the outer product operation is written back to the matrix buffer directly or after one or more addition operations.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the matrix multiplication method based on outer product as described in any one of claims 8 to 9.

Images