Operational Units and Matrix Transpose and Operation Fusion Execution Method
By introducing a transpose unit into the operation unit, the problems of numerous data loading times and complex transfers in the matrix transpose and operation fusion process in the prior art are solved, and a more efficient matrix transpose and operation fusion process is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIBABA DAMO (HANGZHOU) TECH CO LTD
- Filing Date
- 2021-10-08
- Publication Date
- 2026-05-26
AI Technical Summary
In existing technologies, the matrix transposition and operation fusion process involves multiple data loading operations and complex data transfer processes, resulting in low efficiency.
A transpose unit is introduced in the arithmetic unit, located between the register and the execution unit. The source matrix is loaded by the scheduler and transposed into a transposed matrix by the transpose unit. The execution unit performs subsequent operations on the transposed matrix and stores the results in the register and main memory.
It reduces the number of data loading operations and the complexity of the transfer process, and improves the efficiency of matrix transposition and subsequent operation fusion processes.
Smart Images
Figure CN115964085B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of chips, and more specifically, to a computation unit and a matrix transpose and computation fusion execution method. Background Technology
[0002] In deep neural networks (DNNs), matrix transpose operations and multiplication operations between transpose matrices and other matrices are commonly involved. For example, in the activation gradient calculation task during the backpropagation stage of a DNN, the transpose matrix corresponding to the node weight matrix is multiplied with other matrices.
[0003] Currently, within the arithmetic unit, the execution process of matrix transposition and operation fusion is typically as follows: First, a general-purpose execution unit such as CUDACore retrieves the source matrix from the register, performs a transposition operation on the source matrix to obtain the transposed matrix, and stores the transposed matrix in shared memory. Then, the general-purpose execution unit loads the transposed matrix from the shared memory into the register. The register then sends the transposed matrix to another execution unit dedicated to performing matrix multiplication and other operations, so that the execution unit dedicated to performing matrix multiplication and other operations can perform multiplication and other operations on the transposed matrix and other matrices.
[0004] In the above methods, the number of data loading operations is relatively large and the data transfer process is relatively complex, resulting in low efficiency in the matrix transposition and operation fusion process. Summary of the Invention
[0005] In view of this, the present disclosure aims to improve the efficiency of matrix transpose and computation fusion processes.
[0006] According to one aspect of this disclosure, a computational unit is provided, comprising: a scheduler, a register, a transpose unit, and an execution unit, wherein the transpose unit is located between the register and the execution unit, wherein...
[0007] The scheduler receives a matrix transpose and operation fusion instruction, which includes a first source address, a first destination address, an identifier for the transpose operation, and an identifier for the first operation.
[0008] The scheduler loads the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address;
[0009] The transpose unit obtains the first source matrix from the register and transposes the first source matrix into the transposed matrix.
[0010] The execution unit performs the first operation on the transposed matrix and stores the operation result in the register according to the first destination address;
[0011] The scheduler loads the calculation results into the main memory.
[0012] Optionally, the first source matrix includes a data portion before transpose and an index portion before transpose, and the first source address includes the source address of the data portion before transpose and the source address of the index portion before transpose.
[0013] The scheduler loads the transposed data portion from the main memory into the register according to the source address of the transposed data portion, and loads the transposed index portion from the main memory into the register according to the source address of the transposed index portion;
[0014] The transpose unit obtains the data portion before transpose and the index portion before transpose from the register, and transposes the data portion before transpose and the index portion before transpose into the data portion after transpose and the index portion after transpose;
[0015] The execution unit performs the first operation on the transposed data portion and the transposed index portion.
[0016] Optionally, the transpose unit includes:
[0017] A data transpose subunit is used to obtain the transpose data portion based on the data portion before transpose and the index portion before transpose;
[0018] An index transpose subunit is used to obtain the transpose index portion based on the data portion before transpose and the index portion before transpose.
[0019] Optionally, the matrix transpose and operation fusion instruction further includes a second source address;
[0020] The scheduler loads the second source matrix from the main memory into the register according to the second source address;
[0021] The execution unit performs the first operation on the transposed matrix and the second source matrix.
[0022] Optionally, the first source matrix is divided into multiple matrix blocks, and the multiple matrix blocks have a preset positional relationship;
[0023] The scheduler loads the plurality of matrix blocks one by one from the main memory outside the arithmetic unit into the register;
[0024] The transpose unit retrieves the plurality of matrix blocks from the register one by one and transposes each matrix block into a transposed matrix block.
[0025] The execution unit merges the transposed matrix blocks according to the preset positional relationship, and performs the first operation on the merged transposed matrix.
[0026] Optionally, the execution unit is connected to the register;
[0027] The scheduler also receives a matrix direct operation instruction, which includes a third source address, a second destination address, and an identifier of a second operation.
[0028] The scheduler loads the third source matrix from the main memory outside the arithmetic unit into the register according to the third source address;
[0029] The execution unit obtains the third source matrix from the register, performs the second operation on the third source matrix, and stores the operation result in the register according to the second destination address.
[0030] Optionally, the connection relationship between the execution unit and the register and the transpose unit includes one of the following:
[0031] The execution unit is connected to a multiplexer, which is connected to the register and the transpose unit respectively. The strobe signal of the multiplexer determines whether to turn on the register or the transpose unit.
[0032] The execution unit is connected to the register via a first switch and to the transposition unit via a second switch. The opening and closing of the first and second switches determines whether to connect the register or the transposition unit.
[0033] The execution unit is connected to both the register and the transpose unit, and its internal logic controls whether to use the third source matrix received from the register or the transpose matrix received from the transpose unit.
[0034] According to a second aspect of this disclosure, a computing device is provided, including the arithmetic unit described in the first aspect above.
[0035] According to a third aspect of this disclosure, a system-on-a-chip is provided, including the computing unit described in the first aspect above.
[0036] According to a third aspect of this disclosure, a matrix transposition and operation fusion execution method is provided for an operation unit, the operation unit including a scheduler, a register, a transposition unit, and an execution unit, wherein the transposition unit is located between the register and the execution unit, wherein...
[0037] The scheduler receives a matrix transpose and operation fusion instruction, which includes a first source address, a first destination address, an identifier for the transpose operation, and an identifier for the first operation.
[0038] The scheduler loads the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address.
[0039] The first source matrix is obtained from the register through the transpose unit, and the first source matrix is transposed into the transposed matrix.
[0040] The execution unit performs the first operation on the transposed matrix to store the operation result in the register according to the first destination address.
[0041] The scheduler loads the computation results into the main memory.
[0042] Optionally, the first source matrix includes a data portion before transpose and an index portion before transpose, and the first source address includes the source address of the data portion before transpose and the source address of the index portion before transpose.
[0043] The step of loading the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address includes: loading the data portion before transpose into the register according to the source address of the data portion before transpose, and loading the index portion before transpose into the register according to the source address of the index portion before transpose;
[0044] The step of obtaining the first source matrix from the register and transposing the first source matrix into a transposed matrix includes: obtaining the data portion before transposition and the index portion before transposition from the register, and transposing the data portion before transposition and the index portion before transposition into a data portion after transposition and an index portion after transposition.
[0045] The operation of performing the operation on the transposed matrix includes performing the first operation on the transposed data portion and the transposed index portion.
[0046] Optionally, transposing the pre-transpose data portion and the pre-transpose index portion into a transpose data portion and a transpose index portion includes:
[0047] Based on the data portion before transpose and the index portion before transpose, the data portion after transpose is obtained;
[0048] The transposed index portion is obtained based on the data portion before transposition and the index portion before transposition.
[0049] Optionally, the matrix transpose and operation fusion instruction further includes a second source address; the method further includes:
[0050] The scheduler loads the second source matrix from the main memory into the register according to the second source address.
[0051] The operation performed on the transposed matrix includes:
[0052] The first operation is performed on the transposed matrix and the second source matrix.
[0053] Optionally, the first source matrix is divided into multiple matrix blocks, and the multiple matrix blocks have a preset positional relationship;
[0054] The step of loading the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address includes:
[0055] According to the first source address, the plurality of matrix blocks are loaded one by one from the main memory outside the arithmetic unit into the register;
[0056] The step of obtaining the first source matrix from the register and transposing the first source matrix into the transposed matrix includes:
[0057] The plurality of matrix blocks are retrieved one by one from the register, and each matrix block is transposed into a transposed matrix block.
[0058] Performing the first operation on the transposed matrix includes:
[0059] The transposed matrix blocks are merged according to the preset positional relationship, and the first operation is performed on the merged transposed matrix.
[0060] Optionally, the execution unit is connected to the register; the method further includes:
[0061] The scheduler receives direct matrix operation instructions, which include a third source address, a second destination address, and an identifier for a second operation.
[0062] The scheduler loads the third source matrix from the main memory outside the arithmetic unit into the register according to the third source address.
[0063] The execution unit obtains the third source matrix from the register and performs the second operation on the third source matrix to store the operation result in the register according to the second destination address.
[0064] In the arithmetic unit provided in this embodiment, a transpose unit located between the register and the execution unit transposes the first source matrix stored in the register into a transposed matrix. Then, the execution unit performs subsequent arithmetic operations on the transposed matrix to store the result in the destination address in the register, and then transfers it to the main memory. Compared with existing arithmetic units, the arithmetic unit provided in this embodiment performs fewer data loading operations and a simpler data transfer process during matrix transposition and subsequent arithmetic operations, thus improving the efficiency of the matrix transposition and subsequent arithmetic fusion process. Attached Figure Description
[0065] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:
[0066] Figure 1 This is a schematic diagram of the structure of a computing unit according to an embodiment of the present disclosure;
[0067] Figure 2 This is a schematic diagram of the structure of a transpose unit according to an embodiment of the present disclosure;
[0068] Figure 3 This is a schematic diagram of dividing the first source matrix into matrix blocks;
[0069] Figure 4 This is a schematic diagram of the structure of a computing unit according to another embodiment of the present disclosure;
[0070] Figure 5 This is a schematic diagram of the structure of the computing unit according to yet another embodiment of the present disclosure;
[0071] Figure 6 This is a schematic diagram of the structure of the computing unit according to yet another embodiment of the present disclosure;
[0072] Figure 7 This is a flowchart illustrating a matrix transpose and operation fusion execution method according to an embodiment of the present disclosure. Detailed Implementation
[0073] The present disclosure is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present disclosure below, certain specific details are described in detail. Those skilled in the art will fully understand the present disclosure even without these details. To avoid obscuring the substance of the present disclosure, well-known methods, processes, and procedures are not described in detail. Furthermore, the accompanying drawings are not necessarily drawn to scale.
[0074] General Overview of this Disclosure
[0075] A computational unit is a unit that can be used to perform computational tasks and output computational structures. For example, the computational cores contained in a GPU can be used to perform a series of operations such as multiplication and addition in a DNN.
[0076] DNNs typically involve matrix transposition and other operations. Specifically, in the activation gradient calculation task during the backpropagation stage, it is necessary to transpose the weight matrix of the node and then perform multiplication and other operations on the transposed matrix with other matrices.
[0077] Existing computational units used for performing matrix transpose and operation fusion typically have two different execution units: a general-purpose execution unit that can perform various operations, and a dedicated execution unit for operations such as matrix multiplication. The steps for implementing matrix transpose and operation fusion in these existing computational units are as follows: First, the general-purpose execution unit retrieves the source matrix from its internal register, performs a transpose operation on the source matrix to obtain the transposed matrix, and stores the transposed matrix in shared memory. Then, the general-purpose execution unit loads the transposed matrix from shared memory into its internal register. The internal register then sends the transposed matrix to the dedicated execution unit, which performs multiplication and other operations on the transposed matrix and other matrices to obtain the final result. Performing matrix transpose and operation fusion using existing computational units involves numerous data loading operations and a complex data transfer process, resulting in low efficiency.
[0078] In the computation unit of this embodiment, a transpose unit for transposing the source matrix is provided between the register and the execution unit that performs subsequent operations. This allows the transpose unit to directly send the transposed matrix to the execution unit, which then performs subsequent operations on the transposed matrix to obtain the final result. In this embodiment, the number of data loading operations is reduced, and the data transfer process is simpler, thus effectively improving the efficiency of the matrix transpose and subsequent operation fusion process.
[0079] Detailed implementation process of the embodiments disclosed herein
[0080] See Figure 1 , Figure 1 This is a schematic diagram of the structure of a computing unit according to an embodiment of the present disclosure. The computing unit 100 includes a scheduler 110, a register 120, a transpose unit 130, and an execution unit 140, wherein the transpose unit 130 is located between the register 120 and the execution unit 140.
[0081] Specifically, as described above, the arithmetic unit 100 refers to a unit that can perform arithmetic tasks and output arithmetic structures, such as the arithmetic cores contained in a GPU. These arithmetic cores can be used to perform a series of operations such as multiplication and addition in a DNN. The scheduler 110 refers to a unit used for scheduling and controlling arithmetic instructions. The scheduler 110 can efficiently allocate various received arithmetic instructions to the units within the arithmetic unit, enabling each unit to perform the corresponding operation. The register 120 is a unit inside the arithmetic unit 100 used to store data (operands).
[0082] The matrix transpose and operation fusion instruction is used to transpose the first source matrix at the first source address of the main memory 200 outside the operation unit 100, perform the first operation on the transposed matrix to obtain the operation result, and store the operation result in the first destination address of the register 120 so as to return to the main memory 200.
[0083] In one embodiment of this disclosure, the scheduler 110 receives a matrix transpose and operation fusion instruction, which includes a first source address, a first destination address, an identifier for the transpose operation, and an identifier for the first operation. The scheduler 110 loads the first source matrix from the main memory 200 outside the arithmetic unit 100 into register 120 according to the first source address. The transpose unit 130 retrieves the first source matrix from register 120 and transposes it into a transposed matrix. The execution unit 140 performs the first operation on the transposed matrix, stores the operation result in register 120 according to the first destination address, and the scheduler 110 loads the operation result into the main memory 200.
[0084] The aforementioned matrix transpose and operation fusion instruction may include at least: a first source address, a first destination address, an identifier for the transpose operation, and an identifier for the first operation. Further, the aforementioned matrix transpose and operation fusion instruction may also include: the type of the first source matrix stored in the first source address, such as a sparse matrix, or a dense matrix, the size of the first source matrix, etc. In this embodiment of the disclosure, the specific format of the matrix transpose and operation fusion instruction is not limited. For example, the format of the matrix transpose and operation fusion instruction may be: "HMMA.SP.TRANS.16.32.Rd.Ra", where "Ra" represents the first source address; "Rd" represents the destination address; "16.32" indicates that the size of the first source matrix stored in the first source address is 16×32; "TRANS" is the identifier for the transpose operation; "SP" indicates that the type of the first source matrix is a sparse matrix; and "HMMA" is the identifier for the matrix multiplication operation (first operation).
[0085] In the aforementioned matrix transpose and operation fusion instruction, the first source address can represent the address of the first source matrix in the main memory 200 outside the operation unit 100, wherein the first source matrix is the matrix to be transposed by the transpose unit 130. The first destination address can represent the storage address of the operation result to be stored in the register 120 of the operation unit 100; the first operation can be an operation performed on the transposed matrix. In this disclosure, the specific type of the first operation is not limited. For example, it can be a matrix multiplication operation between the transposed matrix and other matrices, or a matrix addition operation between the transposed matrix and other matrices, or a matrix multiplication operation can be performed between the transposed matrix and a certain matrix (e.g., matrix A) to obtain the multiplication result, and then a matrix addition operation can be performed between the multiplication result and another matrix (e.g., matrix B), etc.
[0086] The arithmetic unit 100 of this embodiment transposes the first source matrix stored in register 120 into a transposed matrix via a transposition unit 130 located between register 120 and execution unit 140. Then, execution unit 140 performs subsequent arithmetic operations on the transposed matrix to store the result at the destination address in register 120, and then transfers it to main memory 200. Compared with existing arithmetic units, the arithmetic unit 100 provided in this embodiment performs fewer data loading operations and a simpler data transfer process during matrix transposition and subsequent arithmetic operations, thus improving the efficiency of the matrix transposition and subsequent arithmetic fusion process.
[0087] Optionally, in some embodiments, the first source matrix includes a pre-transpose data portion and a pre-transpose index portion, and the first source address includes the source address of the pre-transpose data portion and the source address of the pre-transpose index portion. Scheduler 110 loads the pre-transpose data portion from main memory 200 into register 120 according to the source address of the pre-transpose data portion, and loads the pre-transpose index portion from main memory 200 into register 120 according to the source address of the pre-transpose index portion. Transpose unit 130 obtains the pre-transpose data portion and the pre-transpose index portion from register 120, and transposes the pre-transpose data portion and the pre-transpose index portion into a transpose data portion and a transpose index portion. Execution unit 140 performs a first arithmetic operation on the transpose data portion and the transpose index portion.
[0088] Specifically, matrices can be classified into sparse matrices and dense matrices based on the ratio of zero elements to non-zero elements. A sparse matrix is one where the number of zero elements far exceeds the number of non-zero elements. For sparse matrices, to facilitate storage, they can be divided into a data part and an index part, stored separately. The data part consists of the elements in the matrix, while the index part indicates the position of each element within the matrix. Correspondingly, during subsequent matrix loading and transposition processes, the data part and the index part can also be processed separately to improve the speed of matrix loading and transposition.
[0089] Therefore, in this embodiment, if the first source matrix is a sparse matrix, it can be divided into a pre-transpose data portion and a pre-transpose index portion, which are stored separately. Correspondingly, the loading process of the subsequent scheduler 110 is also divided into two sub-processes: loading the pre-transpose data portion into a register, and loading the pre-transpose index portion into register 120. The transpose process of the transpose unit 130 includes transposing the pre-transpose data portion and the pre-transpose index portion into the transpose data portion and the transpose index portion, respectively. Thus, this embodiment can effectively improve the speed of matrix loading and transpose, further improving the overall efficiency of the matrix transpose and operation fusion process.
[0090] As described above, since the matrix transpose process can be divided into a subprocess for obtaining the transposed data part and a subprocess for obtaining the transposed index part, the transpose unit 130 may also include: a data transpose subunit 1301 for obtaining the transposed data part and an index transpose subunit 1302 for obtaining the transposed index part.
[0091] Further, see Figure 2 , Figure 2 This is a schematic diagram of the structure of a transpose unit according to an embodiment of the present disclosure. The transpose unit 130 includes: a data transpose subunit 1301, used to obtain a transpose data portion based on the data portion before transpose and the index portion before transpose; and an index transpose subunit 1302, used to obtain a transpose index portion based on the data portion before transpose and the index portion before transpose.
[0092] Specifically, for example, the data part and the index part can be combined. First, the first source matrix stored in the data part and the index part can be restored to a matrix in its regular form. Then, for the restored matrix, the symmetrically distributed elements are swapped along the diagonal to obtain the swapped matrix. After that, the data transpose subunit 1301 obtains the transposed data part based on the swapped matrix, and the index transpose subunit 1302 obtains the transposed index part based on the swapped matrix.
[0093] In some cases, the matrix to be subjected to the first operation with the transposed matrix may be the first source matrix or the transposed matrix itself. In this case, it is not necessary to load other source matrices from the main memory 200. However, in normal cases, the matrix to be subjected to the first operation with the transposed matrix is not the first source matrix or the transposed matrix itself, but another matrix, such as the second source matrix. In this case, the second source matrix needs to be obtained before the execution unit 140 performs the first operation.
[0094] Optionally, in some embodiments, the matrix transpose and operation fusion instruction further includes a second source address; the scheduler 110 loads the second source matrix from the main memory 200 into the register 120 according to the second source address; and the execution unit 140 performs a first operation on the transposed matrix and the second source matrix.
[0095] Specifically, the aforementioned second source address can represent the address of the second source matrix in the main memory 200 outside the arithmetic unit 100. The second source matrix can represent the matrix to be multiplied by the transposed matrix. For example, assuming the transposed matrix is AT and the first operation is matrix multiplication, the second source matrix is the matrix to be multiplied by AT.
[0096] Optionally, in some embodiments, the first source matrix is divided into multiple matrix blocks, and the multiple matrix blocks have a preset positional relationship. The scheduler 110 loads the multiple matrix blocks one by one from the main memory 200 outside the arithmetic unit into the register 120; the transpose unit 130 retrieves the multiple matrix blocks one by one from the register 120 and transposes each matrix block into a transposed matrix block; the execution unit 140 merges the transposed matrix blocks according to the preset positional relationship and performs a first arithmetic operation on the merged transposed matrix.
[0097] Specifically, when the size of the first source matrix is large, if the entire matrix is loaded and transposed directly, that is, the first source matrix is loaded as a whole into the transpose unit 130, and after loading is completed, the transpose unit 130 transposes the entire first source matrix to obtain the transposed matrix, then the loading and transpose operations are inefficient.
[0098] To improve the efficiency of loading and transpose operations and enable them to be performed in parallel, in this embodiment, the first source matrix is divided into multiple matrix blocks with preset positional relationships. Then, on a single matrix block basis, a loading operation is performed first, followed by a transpose operation, resulting in transposed matrix blocks. These transposed matrix blocks are then merged according to the preset relationships to obtain the transposed matrix. Thus, while the scheduler 110 loads matrix blocks, the transpose unit 130 can perform the transpose operation, improving the overall efficiency of the loading and transpose processes, and consequently, the efficiency of the entire matrix transpose and computation fusion process.
[0099] When partitioning the first source matrix, the positional relationships can be set according to the actual situation. This disclosure does not limit the positional relationships between the partitioned matrix blocks. For example, the first source matrix can be partitioned by column, or it can be partitioned by column, and so on.
[0100] In one embodiment, see Figure 3 , Figure 3 A schematic diagram of matrix block partitioning for the first source matrix. Figure 3 The matrix on the right is the first source matrix C, which contains 8 columns. The first 4 columns of C can be divided into the first matrix block C1, and the last 4 columns into the second matrix block C2. The positional relationship between the first matrix block C1 and the second matrix block C2 is that the first matrix block C1 is to the left of the second matrix block C2, that is, the second matrix block C2 is to the right of the first matrix block C1. Then, loading and transposing operations are performed on a single matrix block basis to obtain the first transposed matrix block C1T corresponding to the first matrix block C1, and the second transposed matrix block C2T corresponding to the second matrix block C2. Then, since the first matrix block C1 is to the left of the second matrix block C2, the first transposed matrix block C1T should be above the second transposed matrix block C2T. At this time, the first transposed matrix block C1T and the second transposed matrix block C2T can be merged in the same way that the first transposed matrix block C1T is above the second transposed matrix block C2T to obtain the transposed matrix CT corresponding to the complete first source matrix.
[0101] In some applications, the transpose of a source matrix is used as the input for the operations performed by the arithmetic unit 100. However, in other applications, it may be necessary to perform certain operations directly on the source matrix stored in the main memory 200. For example, the source matrix may be directly multiplied by another matrix, or added to another matrix. Alternatively, the source matrix may be multiplied by another matrix (e.g., matrix A) to obtain the multiplication result, and then the result may be added to another matrix (e.g., matrix B), and so on.
[0102] To meet the second requirement mentioned above, optionally, in some embodiments, the execution unit 140 can also be connected to the register 120. Correspondingly, the scheduler 110 also receives a matrix direct operation instruction, which includes a third source address, a second destination address, and an identifier of the second operation. The scheduler 110 loads the third source matrix from the main memory 200 outside the arithmetic unit 100 into the register 120 according to the third source address; the execution unit 140 obtains the third source matrix from the register 120, performs the second operation on the third source matrix, and stores the operation result in the register 120 according to the second destination address. Specifically, the execution unit 140 receives a matrix direct operation instruction to perform the second operation on the third source matrix at the third source address in the main memory 200, and stores the operation result in the second destination address in the register 120 for returning to the main memory 200. The third source address can represent the address of the third source matrix in the main memory 200 outside the arithmetic unit 100, and the third source matrix is the matrix to be directly subjected to the second operation. The second destination address can represent the storage address of the operation result to be stored in register 120 of the arithmetic unit 100; the second arithmetic operation can be an arithmetic operation performed on the aforementioned third source matrix. In this disclosure, the specific type of the third arithmetic operation is not limited. For example, it can be a matrix multiplication operation between the third source matrix and other matrices, or a matrix addition operation between the aforementioned third source matrix and other matrices, or a matrix multiplication operation can be performed between the aforementioned third source matrix and a certain matrix (e.g., matrix A) to obtain the multiplication result, and then a matrix addition operation can be performed between the aforementioned multiplication result and another matrix (e.g., matrix B), and so on.
[0103] In this disclosure, the operation unit can not only perform matrix transposition and operation fusion instructions, as well as the operation tasks of transposed matrix and other matrices, but also perform operation tasks between the source matrix and other matrices directly based on matrix direct operation instructions. In other words, the operation unit in the embodiments of this disclosure can perform at least the two different operation tasks described above, making it more applicable to a wider range of scenarios and more efficient in operation.
[0104] In order for the arithmetic unit 100 to determine the arithmetic task corresponding to a specific instruction from the two types of arithmetic tasks based on the different instructions received by the scheduler 110, in this embodiment of the disclosure, the connection relationship between the execution unit and the register 120 and the transpose unit 130 can be one of the following relationships:
[0105] The first type of connection: the execution unit 140 is connected to the multiplexer 150, the multiplexer 150 is connected to the register 120 and the transpose unit 130 respectively, and the strobe signal of the multiplexer 150 determines whether to turn on the register 120 or the transpose unit 130.
[0106] The second type of connection: the execution unit 140 is connected to the register 120 through the first switch 160 and to the transpose unit 130 through the second switch 170. The opening and closing of the first switch 160 and the second switch 170 determines whether the register 120 or the transpose unit 130 is turned on.
[0107] The third type of connection: the execution unit 140 is connected to both the register 120 and the transpose unit 130. The internal logic of the execution unit 100 controls whether to use the third source matrix received from the register 120 or the transposed matrix received from the transpose unit 130.
[0108] Specifically, for the first type of connection, please refer to... Figure 4 A multiplexer 150 is added between the transpose unit 130 and the execution unit 140, with one end of the multiplexer 150 connected to both the transpose unit 130 and the register 120, and the other end connected to the execution unit 140. The strobe signal of the multiplexer 150 determines whether register 120 or transpose unit 130 is activated. In this disclosure, the specific setting of the strobe signal and how the value of the strobe signal determines whether register 120 or transpose unit 130 is activated are not limited. For example, the strobe signal can be represented by a variable if_trans, and when if_trans = 1, execution unit 140 activates transpose unit 130; conversely, when if_trans = 0, execution unit 140 activates register 120, and so on.
[0109] For the second type of connection, please refer to Figure 5A first switch 160 is provided between the execution unit 140 and the register 120, and a second switch 170 is provided between the execution unit 140 and the transpose unit 130. The opening and closing of the first switch 160 and the second switch 170 determine whether the register 120 or the transpose unit 130 is activated. Specifically, if the first switch 160 is open and the second switch 170 is closed, the execution unit 140 activates the transpose unit 130; conversely, if the first switch 160 is closed and the second switch 170 is open, the execution unit 140 activates the register 120.
[0110] For the third type of connection, please refer to Figure 6 The execution unit 140 is connected to the register 120, and is also connected to the transpose unit 130. The execution unit 140 determines, based on its internal logic control, whether to obtain the third source matrix from the register 120 or the transposed matrix from the transpose unit 130.
[0111] like Figure 7 As shown, one embodiment of this disclosure also provides a matrix transpose and operation fusion execution method for use in... Figure 1 The arithmetic unit 100 shown includes a scheduler 110, a register 120, a transpose unit 130, and an execution unit 140, wherein the transpose unit 130 is located between the register 120 and the execution unit 140. The method shown includes:
[0112] Step 701: Receive matrix transpose and operation fusion instructions through the scheduler. The matrix transpose and operation fusion instructions include a first source address, a first destination address, an identifier for the transpose operation, and an identifier for the first operation.
[0113] Step 702: Load the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address through the scheduler.
[0114] Step 703: Obtain the first source matrix from the register through the transpose unit, and transpose the first source matrix into the transposed matrix.
[0115] Step 704: The execution unit performs the first operation on the transposed matrix to store the operation result in the register according to the first destination address.
[0116] Step 705: Load the calculation results into the main memory through the scheduler.
[0117] Optionally, the first source matrix includes a data portion before transpose and an index portion before transpose, and the first source address includes the source address of the data portion before transpose and the source address of the index portion before transpose. Loading the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address includes: loading the data portion before transpose into the register according to the source address of the data portion before transpose, and loading the index portion before transpose into the register according to the source address of the index portion before transpose;
[0118] Obtaining the first source matrix from the register and transposing the first source matrix into the transposed matrix includes: obtaining the data part and the index part before transposition from the register, and transposing the data part and the index part before transposition into the data part and the index part after transposition.
[0119] Perform operations on the transposed matrix, including: performing a first operation on the transposed data part and the transposed index part.
[0120] Optionally, transposing the data portion before transpose and the index portion before transpose into the data portion after transpose and the index portion after transpose includes: obtaining the data portion after transpose based on the data portion before transpose and the index portion before transpose; and obtaining the index portion after transpose based on the data portion before transpose and the index portion before transpose.
[0121] Optionally, the first source matrix is a sparse matrix.
[0122] Optionally, the matrix transpose and operation fusion instruction further includes a second source address; the method further includes: loading a second source matrix from main memory into a register according to the second source address through the scheduler.
[0123] Perform operations on the transposed matrix, including: perform a first operation on the transposed matrix and the second source matrix.
[0124] Optionally, the first source matrix is divided into multiple matrix blocks, and the multiple matrix blocks have a preset positional relationship; loading the first source matrix into the register from the main memory outside the arithmetic unit according to the first source address includes: loading multiple matrix blocks into the register one by one from the main memory outside the arithmetic unit according to the first source address.
[0125] Obtaining the first source matrix from the register and transposing the first source matrix into the transposed matrix includes: retrieving multiple matrix blocks one by one from the register and transposing each matrix block into the transposed matrix block.
[0126] Performing a first operation on the transposed matrix includes: merging the transposed matrix blocks according to a preset positional relationship, and performing the first operation on the merged transposed matrix.
[0127] Optionally, the execution unit is connected to the register; the method further includes: receiving a matrix direct operation instruction via a scheduler, the matrix direct operation instruction including a third source address, a second destination address, and an identifier of a second operation; loading a third source matrix from the main memory outside the operation unit into the register according to the third source address via the scheduler; obtaining the third source matrix from the register via the execution unit, performing a second operation on the third source matrix, and storing the operation result in the register according to the second destination address.
[0128] Since the implementation details of the above process have been described in detail in the description of the foregoing device embodiments, they will not be repeated here.
[0129] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the method embodiments, since they are basically similar to the methods described in the apparatus and system embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions of other embodiments.
[0130] It should be understood that the foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0131] It should be understood that the use of a singular form to describe an element or to show only one element in the accompanying drawings does not imply that the number of such element is limited to one. Furthermore, modules or elements described or shown as separate herein may be combined into a single module or element, and modules or elements described or shown as single herein may be broken down into multiple modules or elements.
[0132] It should also be understood that the terminology and expressions used herein are for descriptive purposes only, and one or more embodiments described herein should not be limited to these terms and expressions. The use of these terms and expressions does not exclude any illustrative and descriptive equivalent features (or parts thereof), and it should be recognized that various modifications that may exist should also be included within the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be considered to cover all such equivalents.
Claims
1. An arithmetic unit comprising a scheduler, a register, a transpose unit and an execution unit, wherein, The transpose unit is located between the register and the execution unit; the arithmetic unit further includes a multiplexer, one end of which is connected to both the transpose unit and the register, and the other end of which is connected to the execution unit; the strobe signal of the multiplexer determines whether to activate the register or the transpose unit; wherein... When the multiplexer's strobe signal determines to activate the transpose unit, the scheduler receives a matrix transpose and operation fusion instruction, which includes a first source address, a first destination address, an identifier for the transpose operation, and an identifier for the first operation. The scheduler loads the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address; The transpose unit obtains the first source matrix from the register, transposes the first source matrix into a transposed matrix, and sends the transposed matrix to the execution unit; The execution unit performs the first operation on the transposed matrix and stores the operation result in the register according to the first destination address; The scheduler loads the calculation result into the main memory; When the multiplexer's strobe signal determines that the register is turned on, the scheduler receives a direct matrix operation instruction, which includes a third source address, a second destination address, and an identifier for a second operation. The scheduler loads a third source matrix from the main memory outside the arithmetic unit into the register according to the third source address; The execution unit obtains the third source matrix from the register, performs the second operation on the third source matrix, and stores the operation result in the register according to the second destination address.
2. The arithmetic unit of claim 1, wherein, The first source matrix includes a data portion before transpose and an index portion before transpose, and the first source address includes the source address of the data portion before transpose and the source address of the index portion before transpose. The scheduler loads the transposed data portion from the main memory into the register according to the source address of the transposed data portion, and loads the transposed index portion from the main memory into the register according to the source address of the transposed index portion; The transpose unit obtains the data portion before transpose and the index portion before transpose from the register, and transposes the data portion before transpose and the index portion before transpose into the data portion after transpose and the index portion after transpose; The execution unit performs the first operation on the transposed data portion and the transposed index portion.
3. The arithmetic unit of claim 2, wherein, The transposition unit includes: A data transpose subunit is used to obtain the transpose data portion based on the data portion before transpose and the index portion before transpose; An index transpose subunit is used to obtain the transpose index portion based on the data portion before transpose and the index portion before transpose.
4. The arithmetic unit according to claim 1, wherein, The matrix transpose and operation fusion instruction also includes a second source address; The scheduler loads the second source matrix from the main memory into the register according to the second source address; The execution unit performs the first operation on the transposed matrix and the second source matrix.
5. The arithmetic unit according to claim 1, wherein, The first source matrix is divided into multiple matrix blocks, and the multiple matrix blocks have a preset positional relationship; The scheduler loads the plurality of matrix blocks one by one from the main memory outside the arithmetic unit into the register; The transpose unit retrieves the plurality of matrix blocks from the register one by one and transposes each matrix block into a transposed matrix block. The execution unit merges the transposed matrix blocks according to the preset positional relationship, and performs the first operation on the merged transposed matrix.
6. A computing device, comprising: The arithmetic unit according to any one of claims 1-5; The main memory.
7. A system-on-a-chip, comprising a computing unit according to any one of claims 1-5.
8. A matrix transpose and operation fusion execution method, used in an arithmetic unit, the arithmetic unit comprising a scheduler, registers, a transpose unit, and an execution unit, wherein, The transpose unit is located between the register and the execution unit; the arithmetic unit further includes a multiplexer, one end of which is connected to both the transpose unit and the register, and the other end of which is connected to the execution unit; the strobe signal of the multiplexer determines whether to activate the register or the transpose unit; wherein... When the multiplexer's strobe signal determines to activate the transpose unit, the scheduler receives a matrix transpose and operation fusion instruction, which includes a first source address, a first destination address, an identifier for the transpose operation, and an identifier for the first operation. The scheduler loads the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address. The transpose unit retrieves the first source matrix from the register, transposes the first source matrix into a transposed matrix, and sends the transposed matrix to the execution unit. The execution unit performs the first operation on the transposed matrix to store the operation result in the register according to the first destination address. The scheduler loads the computation result into the main memory. When the multiplexer's strobe signal determines that the register is turned on, the scheduler receives a direct matrix operation instruction, which includes a third source address, a second destination address, and an identifier for a second operation. The scheduler loads the third source matrix from the main memory outside the arithmetic unit into the register according to the third source address. The execution unit obtains the third source matrix from the register and performs the second operation on the third source matrix to store the operation result in the register according to the second destination address.
9. The method according to claim 8, wherein, The first source matrix includes a data portion before transpose and an index portion before transpose, and the first source address includes the source address of the data portion before transpose and the source address of the index portion before transpose. The step of loading the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address includes: loading the data portion before transpose into the register according to the source address of the data portion before transpose, and loading the index portion before transpose into the register according to the source address of the index portion before transpose; The step of obtaining the first source matrix from the register and transposing the first source matrix into a transposed matrix includes: obtaining the data portion before transposition and the index portion before transposition from the register, and transposing the data portion before transposition and the index portion before transposition into a data portion after transposition and an index portion after transposition. The operation of performing the operation on the transposed matrix includes performing the first operation on the transposed data portion and the transposed index portion.
10. The method according to claim 9, wherein, The step of transposing the pre-transpose data portion and the pre-transpose index portion into a transpose data portion and a transpose index portion includes: Based on the data portion before transpose and the index portion before transpose, the data portion after transpose is obtained; The transposed index portion is obtained based on the data portion before transposition and the index portion before transposition.
11. The method according to claim 8, wherein, The matrix transpose and operation fusion instruction further includes a second source address; the method further includes: The scheduler loads the second source matrix from the main memory into the register according to the second source address. The operation performed on the transposed matrix includes: The first operation is performed on the transposed matrix and the second source matrix.
12. The method according to claim 8, wherein, The first source matrix is divided into multiple matrix blocks, and the multiple matrix blocks have a preset positional relationship; The step of loading the first source matrix from the main memory outside the arithmetic unit into the register according to the first source address includes: According to the first source address, the plurality of matrix blocks are loaded one by one from the main memory outside the arithmetic unit into the register; The step of obtaining the first source matrix from the register and transposing the first source matrix into the transposed matrix includes: The plurality of matrix blocks are retrieved one by one from the register, and each matrix block is transposed into a transposed matrix block. Performing the first operation on the transposed matrix includes: The transposed matrix blocks are merged according to the preset positional relationship, and the first operation is performed on the merged transposed matrix.