Block matrix multiplication system

Abstract

A block matrix multiplication operation system relates to a processor, which comprises: an N-bit data bus configured to access a memory; a central processing unit CPU connected to the data bus; a coprocessor coupled to the CPU, the coprocessor comprising a register file having an N-bit register (V); an instruction processing unit in the CPU configured to read access to a memory address in response to a load-spreading machine instruction received by the CPU, and to delegate processing of a corresponding N-bit word (W) presented on the data bus to the coprocessor; and a register control unit in the coprocessor, configured by the CPU in response to the load-spreading instruction to divide the word (W) presented on the data bus into K segments, and write the K segments into the same position in K corresponding registers (V0-V3), the position and the register being specified by the load-spreading instruction.

Description

Technical Field

[0001] The present invention relates to digital matrix multiplication operations, and in particular to a processor that performs matrix operations with the assistance of a dedicated hardware accelerator. Background Art

[0002] Artificial intelligence technology, especially deep learning, has particularly high requirements for multiplication operations of large matrices that can have hundreds of rows and columns. Therefore, hardware accelerators dedicated to mixed-precision matrix multiplication operations have emerged.

[0003] Multiplication operations on large matrices are usually performed in blocks, i.e. by decomposing the matrix into sub-matrices of a size suitable for the computational resources. Accelerators are therefore designed to efficiently compute the products of these sub-matrices.

[0004] For example, NVidia provides hardware operators called tensor cores in its Volta graphics processors, each of which monopolizes the resources of several graphics cores of the processor to perform many calculations involved in sub-matrix multiplication operations in parallel. NVidia also provides NVDLA ("NVidia Deep Learning Accelerator") units in its integrated Xavier processors that have tensor cores independent of the graphics cores and are dedicated to accelerating neural inference.

[0005] This accelerator is called "weakly coupled" because it receives batches of tasks prepared by the host processor and then manages the tasks independently of the host processor, including accessing shared memory to read matrix data. The accelerator and host processor thus operate asynchronously, whereby the host processor and accelerator compete for memory access and must be synchronized periodically so that the host processor can recover task results and transmit new tasks. Summary of the invention

[0006] A method for matrix multiplication is generally provided, comprising the following steps:

[0007] a) storing a first source matrix in a shared memory in a row-major format, whereby consecutive elements of a row of the matrix are stored at consecutive memory addresses;

[0008] b) storing the second source matrix in a shared memory in a column-major format, whereby consecutive elements of a column of the matrix are stored at consecutive memory addresses;

[0009] c) In a single transfer cycle from shared memory to a processor register over an N-bit memory bus:

[0010] Read the N-bit word in the source matrix;

[0011] Write K consecutive segments of the word into specified locations in K corresponding N-bit registers, K such that N is divisible by K;

[0012] d) performing step c) on K=R different rows of the first source matrix so as to fill R first registers, whereby each of the R first registers contains: a different sub-matrix of R rows of N / R bits organized in R consecutive segments of the register;

[0013] e) performing step c) on K=Q different columns of the second source matrix so as to fill Q second registers, whereby each of the Q second registers contains: a different sub-matrix of Q columns of N / Q bits organized in Q consecutive segments of the register; and

[0014] f) in response to execution of a sub-matrix multiplication machine instruction, performing in parallel a scalar product of each segment of a first operand register with each segment of a second operand register using a dedicated hardware operator, the sub-matrix multiplication machine instruction designating one of the R first registers as a first operand and one of the Q second registers as a second operand, the segments of the operand registers being treated as vectors.

[0015] The method may also include the following steps: receiving a load spreading machine instruction in a central processing unit (CPU) of a processor; in response to the load spreading instruction, delegating step c) to a coprocessor coupled to the CPU, wherein the locations of registers and segments are specified by the load spreading instruction; receiving a sub-matrix multiplication instruction in the CPU; and in response to the sub-matrix multiplication instruction, delegating step f) to the coprocessor.

[0016] The CPU and the coprocessor may be coupled according to a VLIW architecture, the method comprising the steps of simultaneously executing a load spread instruction and a sub-matrix multiplication instruction contained in the same instruction group received by the CPU, whereby the load spread instruction prepares a next operand sub-matrix while the sub-matrix multiplication instruction operates on a current operand sub-matrix.

[0017] The hardware operator can be configured to accumulate each scalar product in a corresponding segment of a destination register specified by an instruction with the previous contents of the corresponding segment, the method comprising the following steps: executing a sub-matrix multiplication instruction R times, each time specifying a different operand register pair of the same rank among the same set of R first registers and Q second registers and the destination register.

[0018] Each source matrix element has a configurable precision between 8, 16 and 32 bits.

[0019] A processor is provided, comprising: an N-bit data bus configured to access a memory; a central processing unit (CPU) connected to the data bus; a coprocessor coupled to the CPU, the coprocessor comprising a register file having N-bit registers; an instruction processing unit in the CPU configured to, in response to a load-spread machine instruction received by the CPU, read access to a memory address and delegate processing of a corresponding N-bit word presented on the data bus to the coprocessor; and a register control unit in the coprocessor, configured by the CPU in response to the load-spread instruction to divide the word presented on the data bus into K segments and write the K segments to the same location in K corresponding registers, the location and register being specified by the load-spread instruction.

[0020] The coprocessor may include an array of dot-product hardware operators, each operator being configured to simultaneously perform a scalar product of two vectors contained in two segments of two corresponding operand registers, the rank of the two segments in the operand registers being determined by the position of the operator in the operator array, and the operand registers being specified by a sub-matrix multiplication machine instruction received by the CPU, whereby the operand registers contain corresponding operand sub-matrices, and the operator array produces a result sub-matrix.

[0021] The operator may be configured to accumulate scalar products in corresponding segments of the destination register specified by the sub-matrix multiplication instruction.

[0022] The elements of each operand submatrix can have a configurable precision among 8, 16, and 32 bits. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments will be described in the following non-limiting description with reference to the accompanying drawings, in which:

[0024] Figure 1 is a block diagram of a processor integrating a coprocessor strongly coupled to a central processing unit;

[0025] FIG. 2A to FIG. 2D shows the different steps of filling coprocessor registers from matrix data stored in memory;

[0026] Figure 3 is a block diagram of an embodiment of a coprocessor hardware computing unit designed to implement a multiplication operation of two sub-matrices in response to a single machine instruction;

[0027] Figure 4 is Figure 3 A block diagram of the dot product operator used in the computational unit of ;

[0028] Figure 5is a block diagram of another embodiment of a coprocessor hardware computing unit designed to implement a multiplication operation of two sub-matrices in response to a single machine instruction;

[0029] Figure 6 is Figure 5 A block diagram of the dot product operator used in the computational unit of ;

[0030] Figure 7 is a block diagram of an embodiment of a configurable dot product operator; and

[0031] Figure 8 is a block diagram of an embodiment of a coprocessor hardware computation unit designed to implement multiplication operations of sub-matrices of different sizes in response to a single machine instruction. DETAILED DESCRIPTION

[0032] Hardware accelerators dedicated to matrix multiplication operations face difficulties in feeding matrix data stored in shared memory to the computational units of the accelerator without causing scarcity of the computational units or underutilization of these units. For example, the format used to store the data in the memory may not be suitable for the format required by the computational units, so that latency and data buffers may be introduced to reorganize the data.

[0033] These difficulties are exacerbated when the accelerator is weakly coupled and operates asynchronously with the main processor that directs the accelerator's tasks. In practice, the main processor and accelerator, each operating independently, may conflict with each other when accessing shared memory. Furthermore, the synchronization mechanisms between the main processor and accelerator may cause one or the other to stall while waiting for a synchronization state to occur.

[0034] Additional difficulties arise when the matrix elements have configurable precision among several possibilities.

[0035] To alleviate these difficulties, this paper discloses a processor architecture that integrates a strongly coupled coprocessor with its own register file and a special mechanism for transferring data between memory and the coprocessor registers.

[0036] Figure 1 is a block diagram of such a processor architecture. It comprises a general purpose central processing unit (CPU) 10 strongly coupled to a coprocessor 12, which integrates hardware operators dedicated to the calculation of matrix products. By "strong coupling", it is understood that the coprocessor obeys cycle by cycle machine instructions executed in the CPU and implemented by the hardware execution unit 14.

[0037] More specifically, some of the machine instructions in the processor instruction set incorporate commands specific to the coprocessor. When these instructions reach the corresponding execution unit 14 of the CPU, the execution unit configures the coprocessor operation via the control lines CTRL. The coprocessor is wired to immediately obey the signals present on these control lines. In fact, the coprocessor is an extension of the execution unit 14 of the CPU, obeying an extension of the general instruction set of the processor. Thus, in addition to adapting the execution unit to the coprocessor control, the CPU 10 can be of general type, in particular allowing the execution of an operating system or a program compiled from a general programming language.

[0038] The coprocessor 12 includes a hardware algebraic computation unit 16 including hardware operators dedicated to the computation of matrix multiplication operations. The coprocessor also incorporates its own set of working registers, or register file 18, separate from the conventional register file 20 of the CPU 10.

[0039] Register files 18 and 20 are connected to shared memory 22 by an N-bit data bus D. Not shown are the address and memory control buses that obey conventional CPU execution units. The coprocessor's registers 18 have the same size N as the data bus and are configured to obey commands from the execution units 14 of the CPU.

[0040] The two matrices [a] and [b] to be multiplied are initially stored in shared memory 22. Depending on the programming language used, the matrices are stored by default in row-major format, i.e., elements of the same row are located at consecutive addresses, or in column-major format, i.e., elements of the same column are located at consecutive addresses. The C programming language uses the first format, while Fortran uses the second format. In any case, the Standard Linear Algebra Library (BLAS) used by these programming languages ​​provides conversion parameters to switch the matrix from one format to another format according to the needs of the calculation.

[0041] For the needs of the current architecture, the two matrices to be multiplied are stored in complementary format, for example, the first matrix [a] is stored in row-major format, while the second matrix [b] is stored in column-major format. Matrix [b] is therefore stored in transposed form. Figure 1 The memory contents of a matrix [a] having x+1 rows and y+1 columns and a matrix [b] having y+1 rows and z+1 columns are shown. Matrix [b] has the same number of rows as the number of columns of matrix [a], and matrix [b] can be multiplied by matrix [a].

[0042] The coprocessor 12 is designed to multiply two sub-matrices of the source matrix (a first sub-matrix [A] with a fixed number of rows Q, and a second sub-matrix [B] with a fixed number of columns, assuming that the fixed number of columns is equal to Q) in a completely hardware manner. The remaining size of the sub-matrix (hereinafter referred to as the depth) can be configured according to the desired accuracy of the matrix elements. Therefore, the multiplication operation of these sub-matrices produces a result sub-matrix [C] with QxQ elements.

[0043] The number Q determines the hardware resources required to perform the multiplication operation. For AI applications, the value Q = 4 provides a reasonable compromise and will be used as an example below. In fact, in AI calculations, the use of 8-bit or 16-bit numbers (rarely 32-bit) in floating-point, fractional or integer representations makes the operators less complex than those required to process the "single-precision" and "double-precision" floating-point numbers conventionally used in general-purpose CPUs (encoded as 32 bits and 64 bits, respectively).

[0044] Furthermore, each submatrix to be multiplied is considered to have a total size of N bits, i.e. the size of the data bus D, which, as an example below, will be assumed to be 256 bits. Thus, in the example considered below, the submatrix has 4 rows or 4 columns and a depth of 64 bits. Depending on the application, this depth is occupied by eight bytes, four 16-bit words, two 32-bit words or one 64-bit word. They can be integers, fixed numbers or floating point numbers.

[0045] Given this structure, one goal is to provide the coprocessor with a series of sub-matrices to be multiplied while making full use of the memory bus D. In theory, only two read operations are taken to read the amount of data corresponding to two sub-matrices. However, in general, the organization of the source matrix in memory does not directly allow this. In practice, the 256 consecutive bits in the memory overlap with consecutive rows (or columns) of arbitrary depth and do not contain 4 aligned sub-rows (or sub-columns) that can form a sub-matrix.

[0046] To address this complication, the processor instruction set is extended by a special machine instruction, which is called a "load-scatter" instruction and has K variants labeled load.0 to load.3 for K=4. The load-scatter instruction specifies K registers 18 of the coprocessor and the memory address from which the 256-bit word W is read, and for K=4, the K registers 18 are labeled V0 to V3. When the instruction is executed in the CPU 10 and the memory places the word W on the bus, the register control unit of the coprocessor 12 is configured to divide the word W into K segments of N / K bits and write these segments into the K designated registers respectively. In other words, the word W is "scattered" in the registers. The location in the designated register where the segment is written is determined by the variant of the load-scatter instruction executed.

[0047] FIG. 2A to FIG. 2D A variant of executing the load spread instruction is shown for source matrices [a] and [b] having a depth exceeding 256 bits. As an example, it is assumed that the matrix elements are 16-bit words.

[0048] exist Figure 2A In the example above, the load.0 instruction is executed. The memory address transmitted in the instruction refers to, for example, the first element of the first row of matrix [a], i.e., element a[0][0]. Therefore, the word W0 returned on the bus contains elements a[0][0] to a[0]

[15] . The instruction variant configures registers V0 to V3 to receive the corresponding four segments of word W0 in their first quarter.

[0049] exist Figure 2B In the example above, the load.1 instruction is executed. The memory address refers to element a[1][0]. Therefore, the word W1 returned on the bus contains elements a[1][0] to a[1]

[15] . The instruction variant configures registers V0 to V3 to receive the corresponding four segments of word W1 in their second quarter.

[0050] exist Figure 2C In the example above, the load.2 instruction is executed. The memory address refers to element a[2][0]. Therefore, the word W2 returned on the bus contains elements a[2][0] to a[2]

[15] . The instruction variant configures registers V0 to V3 to receive the corresponding four segments of word W2 in their third quarters.

[0051] exist Figure 2D , the load.3 instruction is ultimately executed. The memory address refers to element a[3][0]. Therefore, the word W3 returned on the bus contains elements a[3][0] to a[3]

[15] . The instruction variant configures registers V0 to V3 to receive the corresponding four segments of word W3 in their fourth quarter.

[0052] At the end of these four load scatters, each of the registers V0 to V3 contains a sub-matrix [A] of consecutive 4x4 16-bit words (also called 4x4x16) along the rows of the source matrix [a]. The rows of each sub-matrix are contained in four corresponding segments of the corresponding register V. The registers V0 to V3 containing the sub-matrix [A] will be designated as VA0 to VA3.

[0053] The second matrix [b] is processed in a similar manner by four registers VB0 to VB3, each of which receives a succession of 4x4x16 sub-matrices [B] along the columns of the source matrix [b]. The columns of each sub-matrix are contained in four respective sections of the corresponding register VB.

[0054] Using this process, four sub-matrices [A] and four sub-matrices [B] that can be multiplied are loaded into the coprocessor's registers in eight consecutive clock cycles. This achieves a transfer rate of one sub-matrix per cycle, which is optimal because the full bus width is used in each cycle.

[0055] The segments of registers VA and VB containing rows or columns of a submatrix will be considered as vectors in the following. Thus, register VAi contains four vectors A[0] to A[3], and register VBj contains four vectors B[0] to B[3], each vector having four 16-bit components.

[0056] Figure 3 is a block diagram of an embodiment of a hardware computation unit 16 of a coprocessor designed to perform a multiplication operation of two sub-matrices [A] and [B] in response to a single dedicated machine instruction which will be labeled mm4a. This embodiment corresponds to FIG. 2A to FIG. 2B The example considered in , where the matrix elements have a size of 16 bits.

[0057] The computation unit includes an array of 4x4 hardware dot-product-and-accumulate operators (named DP4A), where the number 4 represents the dimension of the vector. The DP4A operators of row i and column j are wired to perform the scalar product of vector A[i] and vector B[j] contained in registers VA and VB, respectively, and accumulate the scalar product in the cell Cij of the result submatrix [C]. The cell Cij can be all or part of a segment of the appropriate size of the register VC of the coprocessor. A convenient size is 64 bits, which involves four registers VCa to VCd to store the result submatrix [C]. Registers VA, VB and VC are specified by the instruction mm4a executed in CPU 10, here registers VA0, VB0 and VC0a-VC0d.

[0058] Register VC is configured to preserve the organization of the Q segments of registers VA and VB. In this example, with Q=4 and each result element encoded on 64 bits, each of the 4 segments of register VC contains a single element. As shown in the example, register VC is organized so that four segments of the same register store elements from different rows.

[0059] Once the calculation of the result sub-matrix is ​​completed, in this specific case of a segment containing a single element, the contents of the four registers VC can be transferred to the memory within four cycles at addresses corresponding to the positions of the columns of the sub-matrix [C] in the complete result matrix [c], thereby obtaining the result matrix [c] stored in column-major format.

[0060] The registers VC can also be organized in a complementary manner so that four segments of the same register store different column elements. In this case, the contents of the four registers VC can be transferred to the memory in four cycles at addresses corresponding to the positions of the rows of the submatrix [C] in the complete result matrix [c], thereby obtaining a result matrix [c] stored in a row-major format.

[0061] Preferably, the processor instruction set is extended by a machine instruction for transposing a tuple containing Q registers. In the case of Q=4, the instruction designated as mt4 obtains the contents of four registers designated as VCa, VCb, VCC, VCd, and fills the other four registers designated as VDa, VDb, VDc, VDd, such that:

[0062] VDa receives in its segments the first segments of VCa, VCb, VCC and VCd respectively.

[0063] VDb receives the second segments of VCa, VCb, VCC and VCd in its segments respectively.

[0064] VDc receives the third segments of VCa, VCb, VC and VCd in its segments respectively.

[0065] VDd receives the fourth segments of VCa, VCb, VCC and VCd respectively in its segments.

[0066] After executing the instruction, each of the registers VCa, VCb, VCC, VCd may be stored in memory to facilitate storing a row of the result matrix [c] in a row-major format.

[0067] Figure 4 is a block diagram of a dot product operator DP4A at position (i, j) in the operator array. The operator processes vectors A[i] and B[j]. More specifically, the operator includes four 16-bit multipliers MUL, each of which receives as a first multiplicand the corresponding components (Ai0 to Ai3) of the vector A[i] and as a second multiplicand the corresponding components (B0j to B3j) of the vector B[j]. The four products encoded on 32 bits are provided to a multi-adder 40, which is connected to accumulate the sum of the products in a 64-bit register segment Cij. By "accumulate", it is meant that segment Cij receives the sum of the products and the sum of the previous contents of segment Cij. The multi-adder 40 is therefore wired to add four 32-bit numbers and one 64-bit number.

[0068] In theory, Figure 3A computation unit, in the form of wired combinatorial logic, produces a result matrix [C] immediately in response to an instruction mm4a specifying registers VA, VB and VC to be used. In practice, such a computation unit will take more than one processor clock cycle to build up the result, taking into account the propagation delay of the combinatorial logic. Therefore, the computation unit, in particular each operator DP4A, is usually designed as a pipeline structure, which means that the calculation is broken down into several steps that are synchronized in consecutive clock cycles, but also means that a new calculation can be started, that is, a new instruction mm4a is executed at each clock cycle.

[0069] Therefore, the pipeline structure is not optimal for performing independent computations. However, by properly organizing the consecutive computations to keep the pipeline permanently supplied, the pipeline structure can be 100% efficient after the initial pipeline startup phase and before the final pipeline drain phase.

[0070] In fact, the eight sub-matrices are preloaded into the coprocessor registers using the load-spread instruction ( Figure 2A-2D ), and according to the organization of the computational units immediately available, the sub-matrices to be multiplied are therefore available in registers, which makes it possible to maintain a pipeline provided for the calculation of at least four sub-matrix products. These four sub-matrix products, which can be performed by executing four consecutive mm4a instructions specifying the same destination register VC, provide a 4x4 result matrix [C], which is the product of the 4x16 source sub-matrix and the 16x4 source sub-matrix.

[0071] In addition, the highly coupled configuration between the coprocessor and the CPU allows the processor to be organized in a VLIW (Very Large Instruction Word) architecture. The load spread instruction and the mm4a instruction can then be executed in the same VLIW grouping, thereby executing the two instructions in parallel, which occupies the memory bus to retrieve the next sub-matrix while performing calculations on the current sub-matrix.

[0072] Table 1 below presents a sequence of instructions that implements the multiplication of an 8x32 matrix [a] with a 32x8 matrix [b], providing an 8x8 matrix [c] without dead time and utilizing all bandwidth of the computational units and memory bus in parallel. Instructions placed on the same line correspond to instructions executed in the same VLIW grouping. For reasons of compactness and clarity of representation, the coprocessor registers are designated by Ax and By for the multiplicand submatrix and by Cz for the result submatrix, keeping in mind that all these registers selected from the register file 18 have the same properties.

[0073] The first parameter of the instruction identifies the destination register. Subsequent parameters of the instruction typically identify memory addresses or operand registers - here interpretation symbols are used, which specify what the destination register receives. It is assumed that all registers C are set to 0 before the operation, or the operator is configured by the flags of the mm4a instruction to disable accumulation when writing to register C for the first time, which is shown by the symbol "=" (direct write) instead of the symbol "+=" (accumulate).

[0074] Phases 0 and 1 correspond to Figure 2A-2D The content of the description.

[0075] Table 1

[0076] Stage 0: Load submatrices a[0..3][0..3] to a[0..3][12..15]

[0077] load.0A0A1A2A3<-a[0][0..15]

[0078] load.1A0A1A2A3<-a[1][0..15]

[0079] load.2A0A1A2A3<-a[2][0..15]

[0080] load.3A0A1A2A3<-a[3][0..15]

[0081] Stage 1: Load submatrices b[0..3][0..3] to b[12..15][0..3]

[0082] load.0B0B1B2B3<-b[0..15][0]

[0083] load.1B0B1B2B3<-b[0..15][1]

[0084] load.2B0B1B2B3<-b[0..15][2]

[0085] load.3B0B1B2B3<-b[0..15][3]

[0086] Stage 2: Load submatrix a[4..7][0..3] to a[4..7][12..15] and calculate c[0..3][0..3]

[0087]

[0088] After stage 9 and some cycles needed to flush the compute unit pipeline, registers C0 to C15 contain the matrix resulting from the multiplication operation, i.e. a 8x8x64 matrix.

[0089] It should be noted that stages 6 to 9 reuse the C registers of stages 2 to 5, namely the same result sub-matrices c[0..3][0..3], c[0..3][4..7], c[4..7][0..3] and c[4..7][4..7]. In fact, during stages 2 to 5, the sub-matrix c[][] is calculated only on the first half of the source matrix depth (elements 0 to 15). In stages 6 to 9, the sub-matrix c[][] is updated with the influence of the second half of the source matrix depth (elements 16 to 31). This calculation naturally extends to any depth by updating the c[][] sub-matrix with the influence of the following 256-bit slices in the depth of matrices [a] and [b].

[0090] As an example the depth of the source matrix mentioned is a multiple of the width N of the memory bus. When this property does not apply, a load spreading operation affecting the last part of a row (or column) reads a word whose first part contains the end of the row (or column) and whose second part contains unusable data (the beginning of a misaligned row or column, or any data after the end of the matrix). In this case the data in the second part of the word is replaced by zeros and the thus realigned word is processed normally as described above.

[0091] As mentioned before, the depth of the matrix varies depending on the precision chosen for the elements. In the above, matrices with 16-bit elements were considered. The elements can also be 8 or 32 bits long without affecting the organization of the source matrix in the memory or their transfer to the coprocessor registers. However, some modifications can be implemented in the calculation unit 16 of the coprocessor.

[0092] Figure 5 is a block diagram of an embodiment of a coprocessor's compute unit 16 designed to perform a multiplication operation of two sub-matrices [A] and [B] of 8-bit elements in response to a single dedicated machine instruction to be designated mm8a.

[0093] The transfer of the elements of the source matrix from memory to the coprocessor registers VA and VB is performed in the same manner as before, using a series of four load-scatter instructions. As a result, the four segments of the registers, as shown in operand registers VA0 and VB0, now contain eight vectors of 8-bit elements instead of four vectors of 16-bit elements.

[0094] The dot product operator (here called DP8A) is then configured to compute eight products and their sum simultaneously. The precision of the result is 32 bits instead of 64 bits, so only two registers VC0a and VC0b are needed to store the result submatrix [C].

[0095] As shown in the example, registers VC are organized so that each segment stores two row elements of the result submatrix. For example, register VC0a contains the pairs (C00, C01), (C10, C11), (C20, C21) and (C30, C31) in its corresponding segment. In this case, the contents of the two registers VC cannot be directly transferred to the memory in the correct order to form a 4x4 submatrix of the result matrix [c].

[0096] Using a conventional store instruction, it would take 8 cycles to store the contents of registers VC0a and VC0b to form the result matrix [c] in row-major format, i.e., four cycles using only a quarter of the bus width to transfer the four segments of register VC0a, and then four cycles using only a quarter of the bus width to transfer the four segments of register VC0b. This results in a 75% bandwidth waste.

[0097] Preferably, the processor instruction set is extended by an instruction complementary to the load spread instruction, which will be referred to as a "gather-store" instruction. Execution of this instruction by the CPU configures the coprocessor to cascade Q segments fetched from Q specified registers on a memory bus. The location of the segments in the registers is defined by the corresponding variant of the instruction.

[0098] In this example, with Q=4, four gather-store instructions are executed to process the four registers VC0a, VC0b, VC1a and VC1b containing the two result sub-matrices. This mechanism is similar to Figure 2A-2D mechanism.

[0099] In the first cycle, word W0 contains elements C00, C01 of the first segment of register VC0a, followed by elements C02, C03 of the first segment of register VC0b, followed by four elements of the first row of the second submatrix stored in the first segment of registers VC1a and VC1b (not shown). The remaining three cycles similarly transfer the second to fourth rows of the two submatrices. Thus, in the result matrix stored in row-major format, two adjacent submatrices are formed in row-major format, fully occupying the bus bandwidth.

[0100] about Figure 3 , points out that the contents of registers VC0a to VC0d can be transferred to memory in four cycles using conventional store instructions to form a result matrix stored in column-major format. In contrast, the sub-matrices are written to memory in row-major format using four gather-store instructions. In other words, in the specific case where each register VC contains exactly four elements, the sub-matrices can be transferred to memory by row or column, depending on the type of store instruction selected, which is achieved by fully occupying the bus bandwidth.

[0101] In the case of Q=4, an alternative to using the gather-store instruction is to use the previously described mt4 transpose instruction, which allows the four registers VD0a, VD0b, VD1a, VD1b to be filled from registers VC0a, VC0b, VC1a, VC1b in one cycle. Registers VD0a, VD0b, VD1a, VD1b can then be written in a conventional manner to the result matrix stored in row-major format by fully occupying the bus bandwidth.

[0102] Figure 6 is a block diagram of a dot product operator DP8A at position (i, j) in the operator array. It comprises eight 8-bit multipliers, each receiving as a first multiplicand a corresponding component (Ai0 to Ai7) of a vector A[i] and as a second multiplicand a corresponding component (B0j to B7j) of a vector B[j]. The eight products encoded on 16 bits are provided to a multi-adder 40b, which is connected to accumulate the sum of the products in a 32-bit register segment Cij.

[0103] Similarly, when the matrix elements are 32-bit words, the transfer from memory to the coprocessor registers is again performed in the same manner, using a series of four load-scatter instructions. The register segment then contains two vectors of 32-bit elements. The dot product operators are then designated as DP2A, and they accumulate the product of two 32-bit numbers in a 64- or 128-bit register segment. The matrix multiplication instructions are then designated as mm2a.

[0104] Figure 7 It is a block diagram of an embodiment of a configurable dot product operator for processing matrix elements with adjustable precision among 8, 16 and 32 bits. It includes two 32-bit digital multipliers, two 16-bit digital multipliers and four 8-bit digital multipliers, the outputs of which are provided to a multi-adder 40c, which is connected to accumulate the sum of the products in a 64- or 128-bit register segment Cij.

[0105] Only the connections associated with the first operand vector Ai are shown. The used bits of the corresponding register segment are indicated by square brackets.

[0106] The 32x32 multipliers are connected to process the first two components of the vector regardless of their precision. Each of the two components of the 2x32 bit vector is applied to all input lines of a corresponding one of the 32x32 multipliers. The other multipliers produce zero results.

[0107] The 16x16 multipliers are connected to process the third and fourth components of the 4x16 or 8x8 bit vectors. For the 4x16 bit vectors, each of the first two components is applied to the first 16 input lines of the corresponding one of the two 32x32 multipliers, and each of the next two components is applied to all input lines of the corresponding one of the two 16x16 multipliers.

[0108] For an 8x8 bit vector, the first four components are applied to the first 8 input lines of two 32x32 multipliers and two 16x16 multipliers, respectively, and the remaining components are applied to all input lines of the 8x8 multiplier, respectively.

[0109] The interconnect structure between the multiplier inputs and the registers is determined by the nature of the matrix multiplication instruction being executed (ie, mm2a, mm4a, or mm8a).

[0110] As mentioned above, the matrix elements can be integers or floating point numbers (in the latter case, 16-bit and 32-bit precision are preferably used). The multipliers and adders are configured accordingly. If it is desired to process both integers and floating point numbers, the multiplier will have a reconfigurable structure based on the floating point multiplication operation structure.

[0111] When matrix elements have 32 bits of precision, it has been suggested that the precision of the result can be 64 or 128 bits. If a certain degree of error is acceptable, such as saturation when dealing with integers, or rounding errors when dealing with fixed or floating point numbers, 64 bits of precision may be sufficient.

[0112] In the examples presented, it is considered that the two source matrices to be multiplied have elements of the same precision. The disclosed architecture can also multiply mixed precision matrices. For example, in the case of a multiplication operation of a matrix [a] of 16-bit elements and a matrix [b] of 8-bit elements, the following operation can be provided.

[0113] The four registers A0-A3 are filled by the first set of four load-scatter instructions to contain four 4x4x16 sub-matrices taken from the first four rows of the source matrix [a]. In other words, these sub-matrices contain rows a[0][0...15] to a[3][0...15].

[0114] The other four registers A4-A7 are filled by the second set of four load-scatter instructions to contain four 4x4x16 sub-matrices taken from the next four rows of matrix [a]. In other words, these sub-matrices contain rows a[4][0..15] to a[7][0..15].

[0115] The four registers B0-B3 are filled by the third set of four load-scatter instructions to contain eight 4x4x8 sub-matrices taken from the first four columns of matrix [b]. In other words, these sub-matrices contain columns b[0..31][0] to b[0..31][3].

[0116] The two halves of register B0 contain two sub-matrices, one of which, b[0..3][0..3], can be multiplied by the sub-matrix a[0..3][0..3] of register A0 to obtain the result sub-matrix c[0..3][0..3]. The other sub-matrix b[4..7][0..3] of register B0 can be multiplied by the sub-matrix a[4..7][0..3] of register A4 to obtain the result sub-matrix c[4..7][0..3]. In other words, the multiplication operation of the two halves of register B0 is organized into two different sub-matrices that obtain the result matrix [c].

[0117] The two halves of register B1 contain two sub-matrices that can be multiplied with the sub-matrices of registers A1 and A5 to obtain the same result sub-matrices c[0..3][0..3] and c[4..7][0..3], respectively.

[0118] Similarly, the two halves of register B2 contain the two sub-matrices to be multiplied with the sub-matrices of registers A2 and A6, respectively, and the two halves of register B3 contain the two sub-matrices to be multiplied with the sub-matrices of registers A3 and A7, respectively, each time obtaining the same resulting sub-matrices c[0..3][0..3] and c[4..7][0..3], i.e. register VC is not changed during these operations.

[0119] To multiply the contents of registers, two variants of the multiplication operation mm4a can be used in two consecutive cycles, specifying two different registers A (e.g., A0 and A4), a corresponding register B (e.g., B0), and two sets of two registers C to contain the results. Figure 3 In the 4x4x16 sub-matrix multiplication unit in FIG. 1 , two variants first configure the DP4A operator to use only the 8 least significant bits of the B input (called half-input) of the multiplier. The first variant additionally reconfigures the operator to connect the sixteen 8-bit half-inputs to the first four bytes contained in each of the segments Bi0 to Bi3 of register B0. In contrast, the second variant reconfigures the operator to connect the sixteen 8-bit half-inputs to the last four bytes contained in each of the segments Bi0 to Bi3 of register B0.

[0120] In mixed precision matrix multiplication applications, it is conceivable that the two multiplicand matrices are the same size in terms of the number of bits, and therefore the number of elements of the low precision matrix is ​​larger than the high precision matrix. Therefore, the number of rows or columns of the multiplicand matrix of 8-bit elements may be twice that of the corresponding multiplicand matrix of 16-bit elements. In this case, the calculation unit can be designed to multiply the QxQx16 submatrix by the Qx(2Q)x8 submatrix, or conversely by the (2Q)xQx8 matrix.

[0121] Figure 8 is from Figure 3 An example of a corresponding computation unit for Q=4 derived from a computation unit of is configured to respond to the simultaneous execution of two previously indicated variants of the mm4a instruction by producing two sets of two result 4x4 sub-matrices in register VC from two 4x4x16 sub-matrices in registers VA0 and VA1 and a 4x8x8 sub-matrix in register VB0. Consider that register VB0 is subdivided into two halves, each half having 4 segments, each segment containing a 4-byte vector, and the DP4A dot product operator is organized into two 4x4 arrays that process the corresponding two halves of register VB0 simultaneously.

[0122] In a more general case, a computation unit may be provided that includes a dot product operator organized in an RxQ array, where R>Q. In this case, the load-spread instruction is configured to fill R different registers A in a first series of R cycles and to fill Q different registers B in a second series of Q cycles. Each register A then contains a sub-matrix of Q rows with elements of size N / Q bits, and each register B contains an R / Q sub-matrix of Q columns with elements of size N / R bits, where N is the width of the memory bus and the size of the register, and is divisible by Q and R. Each multiplication instruction reads the contents of R / Q registers A and one register B to produce a sub-matrix of size RxQ.

[0123] In order to enable the sub-matrices of register A and register B to be multiplied, their vectors should have the same size. The value N is preferably a constant equal to the memory bus size, which means that if Q≠R, the two sub-matrices have different precision elements. In practice, Q and R can vary in a ratio of 1, 2 or 4, depending on the precision combination chosen for matrices among 8, 16 and 32 bits.

Claims

1. A method for matrix multiplication, comprising the following steps: a) storing a first source matrix in a shared memory in a row-major format, whereby consecutive elements of a row of the matrix are stored at consecutive memory addresses; b) storing the second source matrix in a shared memory in a column-major format, whereby consecutive elements of a column of the matrix are stored at consecutive memory addresses; c) In a single transfer cycle from shared memory to a processor register over an N-bit memory bus: Read the N-bit word in the source matrix; writing K consecutive segments of the word into designated locations in K corresponding N-bit registers, K such that N is divisible by K; d) performing step c) on K=R different rows of the first source matrix so as to fill R first registers, whereby each of the R first registers contains: a different sub-matrix of R rows of N / R bits organized in R consecutive segments of the register; e) performing step c) on K=Q different columns of the second source matrix so as to fill Q second registers, whereby each of the Q second registers contains: a different sub-matrix of Q columns of N / Q bits organized in Q consecutive segments of the register; and f) in response to execution of a sub-matrix multiplication machine instruction, performing in parallel a scalar product of each segment of a first operand register with each segment of a second operand register using a dedicated hardware operator, wherein the sub-matrix multiplication machine instruction specifies one of the R first registers as a first operand and one of the Q second registers as a second operand, the segments of the operand registers being treated as vectors.

2. The method according to claim 1, comprising the steps of: Receiving a load-spreading machine instruction in a central processing unit CPU of a processor; In response to a load spread instruction, delegating step c) to a coprocessor coupled to the CPU, the locations of the registers and the segments being specified by the load spread instruction; receiving a sub-matrix multiplication instruction in a CPU; and In response to the sub-matrix multiplication instruction, step f) is delegated to the coprocessor.

3. The method according to claim 2, wherein: The CPU and the coprocessor are coupled according to a VLIW architecture, and the method includes the steps of simultaneously executing a load spread instruction and a sub-matrix multiplication instruction contained in the same instruction group received by the CPU, whereby the load spread instruction prepares the next operand sub-matrix while the sub-matrix multiplication instruction operates on the current operand sub-matrix.

4. The method according to claim 1, wherein: The hardware operator is configured to accumulate each scalar product in a corresponding segment of a destination register specified by the instruction with the previous contents of the corresponding segment, the method comprising the steps of: The sub-matrix multiplication instruction is executed R times, each time specifying a different pair of operand registers of the same rank among the same set of R first registers and Q second registers and a destination register.

5. The method according to claim 1, wherein: Each source matrix element has a configurable precision between 8, 16 and 32 bits.

6. A processor, comprising: An N-bit data bus configured to access the memory; A central processing unit, CPU, is connected to the data bus; a coprocessor coupled to the CPU, the coprocessor comprising a register file having N-bit registers; an instruction processing unit in the CPU configured to, in response to a load-spread machine instruction received by the CPU, read access to a memory address and delegate processing of a corresponding N-bit word presented on a data bus to a coprocessor; and A register control unit in the coprocessor is configured by the CPU in response to a load-spread instruction to divide a word presented on the data bus into K segments and write the K segments to the same location in K corresponding registers, the locations and registers being specified by the load-spread instruction.

7. A processor according to claim 6, comprising an array of dot product hardware operators in the coprocessor, each operator being configured to simultaneously perform a scalar product of two vectors contained in two segments of two corresponding operand registers, the rank of the two segments in the operand registers being determined by the position of the operator in the operator array, and the operand registers being specified by a sub-matrix multiplication machine instruction received by the CPU, whereby the operand registers contain corresponding operand sub-matrices and the operator array produces a result sub-matrix.

8. The processor according to claim 7, wherein: The operator is configured to accumulate scalar products in corresponding segments of a destination register specified by a sub-matrix multiplication instruction.

9. The processor according to claim 7, wherein: The elements of each operand submatrix have a configurable precision among 8, 16, and 32 bits.

Images