An arithmetic unit compatible with multi-dimensional matrix multiplication

Abstract

This disclosure proposes a PE array structure, arithmetic unit, and its associated MPU compatible with multi-dimensional matrix multiplication. This disclosure expands the functionality of the PE array, enabling it to support multi-dimensional matrix multiplication operations by transmitting control signals in different modes. This improves PE array utilization while shortening computation time and reducing energy consumption associated with data movement. Furthermore, this disclosure designs both low-power and high-performance modes for the same vector-matrix multiplication operation to meet the needs of diverse application scenarios.

Description

Technical Field

[0001] The present disclosure belongs to the field of processors and computing technologies, and in particular relates to a PE array structure compatible with multi-dimensional matrix multiplication, an operation unit, and an MPU thereof. Background Art

[0002] In the arithmetic units of modern processors, matrix multiplication and vector multiplication by matrix are common arithmetic types. Among them, operations of square matrix multiplication, vector multiplication by square matrix, and matrix multiplication by square matrix appear in large numbers in the convolutional layers and fully connected layers of neural networks. Therefore, traditional scalar arithmetic units can no longer meet today's computing power requirements.

[0003] Heterogeneous processors have become a growing trend in recent years, with the MPU (Matrix Processing Unit) within these processors specifically dedicated to matrix multiplication and convolution operations. While matrix multiplication on modern processors can be improved through explicit software programming (subword parallelism, instruction set parallelism, and unrolling) and hardware cache partitioning, hardware resource utilization can be increased and matrix operation time shortened. However, significant hardware resource waste still occurs when calculating asymmetric matrix multiplications, such as vector-matrix multiplications. Summary of the Invention

[0004] In view of this, the present disclosure provides a PE array structure compatible with multi-dimensional matrix multiplication, including:

[0005] There are 64 PE units, and the address of each PE unit in the PE array is recorded as (i, j), where i represents the row and j represents the column;

[0006] The PE array has two inputs: eight A-direction inputs A0 to A7 and 64 W-direction inputs W00 to W77. The A and W directions are orthogonal. W00 to W07 have eight sub-directions, W10 to W17 have eight sub-directions, and so on. W70 to W77 have eight sub-directions, for a total of 64 W-direction inputs.

[0007] Each of A0 to A7 contains 8 numbers, and the 8 numbers form a [1,8] vector. Each of them is a [1,8] vector, which is called A-direction input. The 8 A-direction inputs A0 to A7 can be the same or different.

[0008] Each of W00 to W77 contains 8 numbers, which form a [1,8] vector. Each of these 8 numbers is a [1,8] vector, called the W-direction input. Each of the 64 W-direction inputs W00 to W77 is sent to the corresponding PE unit. The 64 W-direction inputs W00 to W77 can be the same or different.

[0009] For each PE unit (Processing Element), where:

[0010] As a basic processing unit in the PE array, it has two inputs (e.g., one A-direction input and one W-direction input) and one output;

[0011] Taking the PE unit at position (i, j) as an example, the input is Ai and Wij, and the output is recorded as Psum(i, j);

[0012] One PE unit can complete 1 [1,8] × [1,8] in one cycle T Vector multiplication operation of ;

[0013] For the combination of 8 PE units in the first row of the PE array, vector A0 is input to the 8 PE units simultaneously in the horizontal direction;

[0014] Treat the vectors W00 to W07 in the first row of the PE array as 8 column vectors of an [8,8] matrix, and then input them into the PE units at the corresponding positions respectively;

[0015] One cycle of one row of the PE array can complete a [1,8]×[8,8] T Vector multiplication matrix operation;

[0016] By performing different combinations of vector multiplication matrices on the 8 rows in the PE array, vector multiplication matrix and matrix multiplication matrix operations of different dimensions can be achieved.

[0017] In addition, the present disclosure also discloses an operation unit, which includes the PE array structure described above.

[0018] In addition, the present disclosure also discloses an MPU, which includes the PE array structure described above, or the computing unit described above.

[0019] Preferably,

[0020] In addition to the PE_array (PE array structure), the MPU also includes the following modules: control, ACC, Buf, lm_A (local memory A), lm_W (local memory W);

[0021] control, used to generate various control signals to control other modules;

[0022] lm_A and lm_W are used to store the input in the A and W directions;

[0023] PE_array, used to implement matrix operations in various modes;

[0024] ACC, used to accumulate the output of the PE array in different time domains;

[0025] Buf is used to store the accumulated results of ACC. In addition, the results of unfinished operations must be returned to ACC, and the results of completed operations must be output from MPU.

[0026] When the control signal is the first control signal, the PE array structure operates in a low power consumption mode.

[0027] Preferably,

[0028] When the control signal is the first control signal, the PE array structure operates in a high-performance mode.

[0029] Preferably,

[0030] The default mode of the PE array structure is the high performance mode.

[0031] Preferably,

[0032] By transmitting control signals in different modes, it supports matrix multiplication operations in multiple dimensions.

[0033] Preferably,

[0034] For the same vector multiplication matrix operation, the PE array structure can operate in different modes of low power consumption or high performance, and the default mode is high performance mode, and the mode can be switched.

[0035] Therefore, the present disclosure proposes a PE array structure, arithmetic unit, and its MPU compatible with multi-dimensional matrix multiplication. The present disclosure expands the functionality of the PE array, enabling it to support multi-dimensional matrix multiplication operations by transmitting control signals in different modes. This improves PE array utilization while shortening computation time and saving energy consumed by data movement. Furthermore, the present disclosure designs two modes for the same vector-matrix multiplication operation: low-power and high-performance. These modes can be switched as needed to meet the needs of different application scenarios, with the default mode being high-performance.

[0036] Compared with explicit software programming, which requires programmers to have certain knowledge of the underlying hardware, the present disclosure can significantly improve the computational efficiency of matrix multiplication and reduce the workload of programmers when programming by improving the PE array structure from the hardware level. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of a PE array structure in one embodiment of the present disclosure;

[0038] Figure 1A This is a schematic diagram of a conventional PE array in the prior art when calculating matrix multiplication of a non-square matrix;

[0039] Figure 1B It is a schematic diagram of the time domain accumulation of a row of PE array outputs;

[0040] Figure 1C It is a schematic diagram of the time domain accumulation of a column of PE array output;

[0041] Figure 2 In one embodiment of the present disclosure, [1,8*s]×[8*s,8*s] T Schematic diagram of vector multiplication matrix operation;

[0042] Figure 3 2 is a schematic diagram of a first method for copying A[m] and W[m][n] (low power consumption mode) in one embodiment of the present disclosure;

[0043] Figure 4 is a schematic diagram of the second method (high performance mode) for copying A[m] and W[m][n] in one embodiment of the present disclosure;

[0044] Figure 5 Schematic diagram of the combination of A[m] and W[m][n] in low power mode and the PE array input per cycle when s=8 in one embodiment of the present disclosure;

[0045] Figure 6 1 is a schematic diagram of the combination of A[m] and W[m][n] in high-performance mode and the PE array input per cycle when s=8 in one embodiment of the present disclosure;

[0046] Figure 7 is a schematic diagram of a standard PE array matrix multiplication in one embodiment of the present disclosure;

[0047] Figure 8 In one embodiment of the present disclosure, [1,64]×[64,64] T Schematic diagram of vector multiplication matrix operation;

[0048] Figure 9 In one embodiment of the present disclosure, [1,64]×[64,64] T Schematic diagram of dividing a vector into 8 equal rows when multiplying a matrix;

[0049] Figure 10 In one embodiment of the present disclosure, [1,64]×[64,64] T Schematic diagram of low-power vector-matrix multiplication operation;

[0050] Figure 11 In one embodiment of the present disclosure, in the (k+1)th cycle, the 8 [8,8] in W[k] are TThe matrix is ​​input into the corresponding row in the PE array respectively; after one cycle, a schematic diagram of a vector C[k] with a dimension of [1,8] is obtained;

[0051] Figure 12 In one embodiment of the present disclosure, [1,64]×[64,64] T Schematic diagram of high-performance vector-matrix multiplication operations;

[0052] Figure 13 In one embodiment of the present disclosure, in the (k+1)th cycle, a vector A[k] of dimension [1,8] is broadcast to A0-A7 of the PE array, and eight [8,8] matrices W[i][k] (i=0,1,…,7) in W are respectively input to the i-th row of the PE array; after one cycle, each PE unit outputs one number, and the outputs of the eight PEs in the i-th row are concatenated into a vector to be accumulated of dimension [1,8];

[0053] Figure 14 In one embodiment of the present disclosure, [4,16]×[16,16] T Schematic diagram of matrix multiplication;

[0054] Figure 15 In one embodiment of the present disclosure, [4,16]×[16,16] T The splitting and combination of A and W in the matrix multiplication low power mode;

[0055] Figure 16 In one embodiment of the present disclosure, the PE array is [4,16]×[16,16] T Schematic diagram of the PE array in the first cycle of the vector-matrix low-power operation;

[0056] Figure 17 In one embodiment of the present disclosure, the PE array is [4,16]×[16,16] T Schematic diagram of the second cycle of the vector-matrix low-power operation when the W input remains unchanged;

[0057] Figure 18 In one embodiment of the present disclosure, [4,16]×[16,16] T Schematic diagram of the splitting and combination of A and W in the high-performance mode of matrix multiplication;

[0058] Figure 19 In one embodiment of the present disclosure, the PE array is [4,16]×[16,16] T Schematic diagram of the PE array in the first cycle of high-performance vector-matrix multiplication operations;

[0059] Figure 20In one embodiment of the present disclosure, the PE array is [4,16]×[16,16] T Schematic diagram of the second cycle of a high-performance vector-matrix multiplication operation, where the A-direction input remains unchanged and the W-direction input is replaced by the eight matrix inputs of the second cycle in W.

[0060] Figure 21 In one embodiment of the present disclosure, [2,32]×[32,32] T Schematic diagram of matrix multiplication;

[0061] Figure 22 In one embodiment of the present disclosure, [2,32]×[32,32] T Schematic diagram of the splitting and combination of A and W in the vector-matrix multiplication low-power mode;

[0062] Figure 23 In one embodiment of the present disclosure, the PE array is [2,32]×[32,32] T Schematic diagram of the PE array in the first cycle of high-performance vector-matrix multiplication operations;

[0063] Figure 23A In one embodiment of the present disclosure, [2,32]×[32,32] T Schematic diagram of the splitting and combination of A and W in the high-performance mode of vector multiplication matrix;

[0064] Figure 23B In one embodiment of the present disclosure, the PE array performs [4,16]×[16,16] in high performance mode. T Schematic diagram of vector multiplication matrix operation;

[0065] Figure 23C In one embodiment of the present disclosure, when k=0, the PE array performs [4,16]×[16,16] in high performance mode. T Schematic diagram of vector multiplication matrix operation;

[0066] Figure 24 Schematic diagram of an MPU in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] To further describe the present invention, the following Figures 1 to 24 Further explanation will be given.

[0068] Various embodiments of the present disclosure will be described in detail below.

[0069] like Figure 1As shown, in one embodiment, the present disclosure discloses an operation unit compatible with multi-dimensional matrix multiplication, which includes a PE array, wherein the PE array includes at least 64 PE units (preferably, 64 PE units, which conforms to binary and is also a multiple of 8), and the address of each PE unit in the array is recorded as (i, j), where i represents a row and j represents a column;

[0070] The PE array has two types of inputs: A0 to A7, each of which is a [1,8] vector, called the A-direction input; W00 to W77, each of which is a [1,8] vector, called the W-direction input. Each of the 64 inputs is sent to the PE unit at the corresponding position.

[0071] The PE array has 8 A-direction inputs (A0-A7) and 64 W-direction inputs (W00-W77). Each of A0-A7 contains 8 numbers (8 numbers form a [1,8] vector), and the 8 A-direction inputs (A0-A7) can be the same or different. Each of W00-W77 contains 8 numbers (8 numbers form a [1,8] vector), and the 64 W-direction inputs (W00-W77) can be the same or different.

[0072] The PE unit (Processing Element) is a basic processing unit in the PE array. It has two inputs and one output. Taking the PE unit at position (i, j) as an example, the inputs are Ai and Wij, and the output is recorded as Psum(i, j). One PE unit can complete one [1,8]×[1,8] T Vector multiplication operation of ;

[0073] Taking the combination of 8 PE units in the first row of the PE array as an example, vector A0 is input to the 8 PE units horizontally at the same time; the vectors W00 to W07 in the first row of the PE array can be regarded as 8 column vectors of an [8,8] matrix, and then input to the PE units at the corresponding positions respectively; one cycle of one row of the PE array can complete a [1,8]×[8,8] T A vector-matrix multiplication operation.

[0074] The present disclosure implements vector-matrix multiplication and matrix-matrix multiplication operations of different dimensions by performing different combinations of 8-row vector-matrix multiplications in the PE array.

[0075] Those skilled in the art know that the W-direction input of the traditional PE array has only 64 numbers, and the W-direction input of each column of 8 PE units is the same. Compared with the prior art, the W-direction input of each PE unit in the present invention is different. When the W-direction input of the 8 PE units in each column of the PE array is set to be the same, all the functions of the traditional PE array can be completed. When the traditional PE array calculates matrix multiplication of non-square matrices, a large amount of hardware resources will be idle, such as Figure 1A As shown, when performing vector multiplication matrix operations, the PE array can only calculate one group [1,8]×[8,8] per cycle. T At this time, only one row of PE units is in working state, and the computing power is only 1 / 8 of the peak computing power of the PE array.

[0076] The following details the PE array and the splitting and combining method of the input data disclosed in the present invention:

[0077] The PE array of the present disclosure increases 56 units of storage resources (data input in the W direction). Assuming that the size of each input data number is 1 Byte, the storage space is increased by 448 Bytes. The 8 PE units in each row of the PE array can complete a [1,8]×[8,8] T By controlling 8 different input combinations, matrix multiplication operations of different dimensions and vector multiplication matrix operations can be completed. When performing vector multiplication matrix operations, the PE array can calculate 8 different sets of [1,8]×[8,8] per cycle. T Operation, multi-dimensional matrix multiplication operation can be completed by performing time domain accumulation on the output of the PE array.

[0078] The ACC (Accumulation) unit can be used to achieve time domain accumulation of PE array output, such as Figure 1B As shown in the figure, the time domain accumulation of the output of a row of PE arrays requires one vector adder; the ACC unit contains eight vector adders, which perform time domain accumulation on the outputs of eight rows of PE units respectively; the eight PE units in each column of the PE array can complete a [1,64]×[8,64] T By controlling the 8 columns of different input combinations, matrix multiplication operations of different dimensions and vector multiplication matrix operations can be completed. When performing vector multiplication matrix operations, the PE array can calculate 8 different sets of [1,64]×[8,64] per cycle. T Operation, multi-dimensional matrix multiplication operation can be completed by performing spatial summation on the output of the PE array;

[0079] The spatial summation of the PE array output can be achieved through the internal adder of the PE array, such as Figure 1CAs shown in the figure, the spatial summation of the outputs of a column of PE units requires 7 vector adders. By enabling a specified adder, the output of a specified PE unit can be summed. The PE array contains 8*7 adders, which perform spatial summation on the outputs of 8 columns of PE units respectively.

[0080] When performing matrix multiplication operations of any dimension, the input data is split and combined based on the control signal and input to the specified PE unit in a time-sharing manner, so that all 64 PE units in the PE array are in working state. In theory, the PE array disclosed in the present invention can maintain peak computing power for matrix multiplication operations of any dimension.

[0081] The following describes the splitting and combining methods of input data in more detail:

[0082] For any vector multiplied by a matrix, it can be expanded to [1,8*s]×[8*s,8*s] by zero padding. T In the form of (s is a positive integer), [1,30]×[28,30] T For example, after zero padding and expansion, it can be [1,32]×[32,32] T Mode operation;

[0083] like Figure 2 Shown as [1,8*s]×[8*s,8*s] T Vector multiplication matrix operation;

[0084] Split vector A into s [1,8] vectors (8 numbers), one of which is A[m]; split matrix W into s*s [8,8] matrices, one of which is W[m][n], where m represents the row and n represents the column; one cycle of a row of the PE array can just complete a vector-matrix multiplication operation of A[m]×W[m][n]; one cycle of the entire PE array can just complete 8 A[m]×W[m][n] T Vector multiplication matrix operation;

[0085] Therefore, to complete a [1,8*s]×[8*s,8*s] T The vector multiplication matrix operation is to multiply A[m]×W[m][n] T Evenly distributed on the PE array in different time and space domains;

[0086] In matrix multiplication operations, the critical path that affects the PE array clock frequency is the time required to complete a "multiplication and addition" operation per cycle, that is, the time required to complete a vector multiplication and vector addition operation within one cycle.

[0087] In addition, the present disclosure mainly provides two ways to combine A[m] and W[m][n]. The first focuses on low power consumption. After reading the data once, it is reused as much as possible to reduce the energy consumption caused by data movement. The second focuses on high performance. The operation time of data in the PE array is shortened as much as possible. Operations unrelated to vector multiplication (such as accumulation) are placed in the external ACC unit for processing, thereby increasing the clock frequency. For matrix multiplication with a dimension not exceeding [8,8], this type of operation can be completed in one cycle. It does not involve the problem of reducing data movement or the problem of accumulating the PE array operation results in different time domains in the ACC unit. Therefore, the high-performance mode and low-power mode of this type of matrix multiplication are consistent.

[0088] (1) When s = 1, 2, or 4, the original input data is too small to fully utilize the PE array bandwidth. Therefore, the PE array disclosed in the present invention broadcasts the A-direction input or the W-direction input to improve the utilization of the PE array and shorten the operation time.

[0089] like Figure 3 The figure shows the first method for copying A[m] and W[m][n] (low power consumption). The original A-direction input (A[0] to A[s-1]) is copied 8 / s times, each copy forming a group with 8*s elements in total. The original W-direction input is divided into groups of 8 columns each, for a total of s groups with 64*s elements in total. Therefore, the vector-matrix multiplication operation of one group of A-direction input and one group of W-direction input can be completed in one cycle of s rows of the PE array. The entire PE array can complete 8 / s groups of such operations, and a total of s cycles are required to complete all operations.

[0090] like Figure 4 The figure shows the second method for copying A[m] and W[m][n] (high performance). The A-direction input in the first copying method is transposed to obtain s [1,8*8 / s] vectors, with each row forming a group and each group having 8*8 / s elements. The original W-direction input is divided into groups of 8 rows each, for a total of s groups, with each group having 64*s elements. Therefore, the PE array's 8 / s rows can complete the vector-matrix multiplication operation of one group of A-direction input and one group of W-direction input in one cycle. The entire PE array can complete s groups of such operations, and a total of s cycles are required to complete all operations.

[0091] (2) Figure 5 The figure shows the combination of A[m] and W[m][n] in low-power mode when s=8, as well as the PE array input per cycle:

[0092] In the first cycle, 8 vectors (A[0] to A[7]) are sequentially taken from A and used as the input of PE arrays A0 to A7 respectively, and the input of A remains unchanged for 8 cycles; in the kth cycle, 8 matrices (W[k][0] to W[k][7]) are taken from W in column order and used as the W input of 8 rows of the PE array respectively;

[0093] After 8 cycles, all operations related to A[0] to A[7] are completed;

[0094] like Figure 6 The figure shows the combination of A[m] and W[m][n] in high-performance mode when s=8, as well as the PE array input per cycle:

[0095] In the (k+1)th cycle, the kth vector A[k] in A is input and broadcast to A0 to A7 of the PE array. Eight matrices (W[0][k] to W[7][k]) are taken from W in row order and used as the W-direction input of the eight rows of the PE array respectively. After eight cycles, all operations on A are completed.

[0096] (3) When s = 3, 5, 6, or 7, the A-direction input and the W-direction input can be expanded to the form of s = 4 or 8, respectively, and then the operation is performed according to the rules in (1) and (2);

[0097] (4) When s>8, A is divided into s / 8 segments and W is divided into s*s / 8 blocks. Then the split A and W are operated according to the operation rules when s=8.

[0098] The following is described in conjunction with other embodiments:

[0099] Example 1: [8,8]×[8,8] T Matrix multiplication operation

[0100] [8,8]×[8,8] T Matrix multiplication is common in neural network convolution operations, such as Figure 7 The following is a standard PE array matrix multiplication. Matrix A is the input data of A0 to A7. The dimension of A is [8,8]. It is divided into eight equal parts A[0] to A[7] by row. Each of them is a [1,8] vector. One of them is denoted as A[i].

[0101] The matrix W is the input data of W00 to W77. The dimension of W is [8,8]. It is divided into eight equal parts W[0] to W[7] by column. Each of them is a [1,8] vector. One of them is denoted as W[j].

[0102] Matrix C is the output data, the dimension of C is [8,8], and the value of C at position (i,j) is recorded as C[i][j];

[0103] The PE array of the present disclosure is [8,8]×[8,8] T When performing vector multiplication matrix operations, the data input method and calculation steps are as follows:

[0104] (1) Input A[0] to A[7] into A0 to A7 of the PE array, respectively. That is, the A-direction inputs of the eight PE units in each row of the PE array are the same. Input W[0] to W[7] into Wi0 to Ai7 of the PE array, respectively. That is, the W-direction inputs of the eight PE units in each column of the PE array are the same.

[0105] (2) The output of each PE unit is:

[0106] Psum(i,j)=A[i]×W[j]

[0107] It is easy to know that the value of C[i][j] is Psum(i,j);

[0108] (3) After one cycle, the calculations are performed to obtain C[0][0] to C[7][7], completing [8,8]×[8,8] T =[8,8] matrix multiplication calculation.

[0109] For matrix multiplication operations that also require only one cycle to complete (matrix multiplication with matrix dimensions not exceeding [8,8]), the data input in the A direction and the data input in the W direction are both used for only one cycle of operation. Therefore, the operations in high-performance mode and low-power mode of this type of operation are the same and are not differentiated.

[0110] Example 2: [1,64]×[64,64] T Vector multiplication matrix operation

[0111] To meet the requirements of different application scenarios, the hardware structure designed in this paper supports both low-power and high-performance modes for vector-matrix multiplication operations;

[0112] [1,64]×[64,64] T In low-power mode, the PE array can read eight different A-direction inputs at a time, highly multiplexing them over eight cycles. This significantly reduces the energy consumption caused by data movement during vector-matrix multiplication. In low-power mode, an additional 8x7 adders (two inputs and one output) are required in the PE array to sum the calculation results of each PE array cycle.

[0113] In high-performance mode, the A- and W-direction inputs to the PE array vary from cycle to cycle. However, the calculation results do not need to be summed within the PE array for each cycle; instead, they are directly output to the external ACC unit. Therefore, the PE array can perform high-frequency operations. After the calculation results output in the current cycle are accumulated by the ACC unit, the accumulated results are temporarily stored in the external Buf unit, awaiting accumulation with the calculation results of the PE unit in the next cycle in a different time domain.

[0114] like Figure 8As shown, vector A is the input data of A0~A7, the dimension of A is [1,64], and it is divided into eight equal parts A[0]~A[7], each of which is a [1,8] vector, and one of them is denoted as A[i];

[0115] The matrix W is the input data of W00 to W77. The dimension of W is [64,64]. It is divided into 64 [8,8] matrices, one of which is denoted as W[k][i], where k represents the row and i represents the column.

[0116] Vector C is the output, and its dimension is [1,64]. Divide it into eight equal parts, and record one of them as C[k], whose dimension is [1,8]. Each number in C[k] is recorded as C[k][j].

[0117] like Figure 9 As shown, W[k][i] is divided into 8 equal rows, W[k][i][0] to W[k][i][7], where each row is a vector of [1,8], and one of them is denoted as W[k][i][j];

[0118] Low Power Mode:

[0119] The vector multiplication matrix that needs to be calculated in the (k+1)th cycle of the PE array is A[i]×W[k][i] T , i=0,1,…,7; 8 cycles PE array needs to calculate the vector multiplication matrix in each cycle as follows Figure 10 As shown;

[0120] The PE array of the present disclosure performs [1,64]×[64,64] in low power mode. T When performing vector multiplication matrix operations, the data input method and calculation steps are as follows:

[0121] (1) In the first cycle, A[0] to A[7] are input into A0 to A7 of the PE array respectively and remain unchanged (this saves 7 cycles of data movement of 8 [1,8] vectors); Figure 11 As shown, in the (k+1)th cycle, the 8 [8,8] in W[k] are T The matrix is ​​input into the corresponding row of the PE array respectively; after one cycle, a vector C[k] with a dimension of [1,8] is obtained;

[0122] (2) Taking k=0 as an example, A[0]~A[7] are input into A0~A7 of the PE array respectively; the eight W[0][i] are input into the i-th row of the PE array respectively;

[0123] (3) The output of each PE unit is:

[0124] Psum(i,j)=A[i]×W[0][i][j] T

[0125] (4) Sum the Psums with the same j in the 8 rows of the PE array to obtain 8 values;

[0126]

[0127] (5) Concatenate these 8 values ​​to obtain a vector C[0] with a dimension of [1,8];

[0128] C[0]={C[0][0],C[0][1],…,C[0][j],…,C[0][7]}

[0129] (6) The inputs of A0 to A7 remain unchanged. Repeat steps (2) to (5) for W[1] to W[7]. After 8 cycles, we get C[0] to C[7], completing [1,64]×[64,64]. T =[1,64] vector multiplication matrix calculation.

[0130] High Performance Mode:

[0131] The vector multiplication matrix that needs to be calculated in the (k+1)th cycle of the PE array is A[k]×W[i][k] T , i=0,1,…,7; 8 cycles PE array needs to calculate the vector multiplication matrix in each cycle as follows Figure 12 As shown;

[0132] The PE array of the present disclosure performs [1,64]×[64,64] in high performance mode. T When performing vector multiplication matrix operations, the data input method and calculation steps are as follows:

[0133] (1) Figure 13 As shown, in the (k+1)th cycle, the vector A[k] of dimension [1,8] is broadcast to A0~A7 of the PE array, and the 8 [8,8] matrices W[i][k] (i=0,1,…,7) in W are input to the i-th row of the PE array respectively; after 1 cycle, each PE unit outputs 1 number, and the outputs of the 8 PEs in the i-th row are spliced ​​into a vector to be accumulated with dimension [1,8], which is recorded as C k [i];

[0134] (2) Taking k=0 as an example, broadcast A[0] to A0~A7 of the PE array; input the eight W[i][0] into the i-th row of the PE array respectively;

[0135] (3) The output of each PE unit is:

[0136] Psum(i,j)=A[0]×W[i][0][j] T

[0137] (4) The output Psum of the 8 PE units in the i-th row is spliced ​​to obtain a vector of dimension [1,8]. This vector is input to the ACC unit for accumulation, that is, it is added with the vector stored in the Buf unit in the previous cycle. The accumulated result is C k [i], C k [i] Output to the external Buf unit, waiting to be accumulated with the calculation results of the PE array in the next cycle in different time domains;

[0138] (5) Repeat steps (2) to (4) for A[1] to A[7], and in each cycle the newly obtained C k [i] Accumulate with the output of the PE unit in the previous cycle:

[0139]

[0140] (6) After 8 cycles, the 8 C[i] are concatenated to obtain a vector C with a dimension of [1,64]; 8 cycles complete [1,64]×[64,64] T =[1,64] vector multiplication matrix calculation.

[0141] Example 3: [4,16]×[16,16] T Matrix multiplication operation

[0142] like Figure 14 The shown is [4,16]×[16,16] T Matrix multiplication, matrix A is the input data of A0~A7, the dimension of A is [4,16]. First, it is divided into four equal parts A[0]~A[3] by row, and one of them is recorded as A[m]. Then A[m] is divided into two equal parts A[m][0] and A[m][1] by column, each of which is a [1,8] vector, and one of them is recorded as A[m][n].

[0143] The matrix W is the input data of W00 to W77. The dimension of W is [16,16]. It is split into four [8,8] matrices, one of which is denoted as W[i][j], where i represents the row and j represents the column.

[0144] Matrix C is the output of matrix multiplication. Its dimensions and numbering are the same as those of matrix A. Each small block in the figure is a [1,8] vector, and one of them is denoted as C[m][n];

[0145] Low Power Mode:

[0146] The PE array can complete 8 [1,8]×[8,8] T The vector multiplication matrix operation is as follows: Figure 15 The figure shows the splitting and combination of A and W in low power mode;

[0147] The PE array of the present disclosure is [4,16]×[16,16] T When performing a low-power vector-matrix multiplication operation, the data input method and calculation steps are as follows:

[0148] (1) In the first cycle, if Figure 16 As shown, according to Figure 15 The splitting and combining method of A and W shown in the figure inputs the 4 [1,8] vectors of the first cycle in A into A0~A7 of the PE array respectively (each [1,8] vector is broadcast to 2 rows in the PE array); the 8 [8,8] vectors in W are T The matrix is ​​input into the PE array W00~W77 respectively; after 1 cycle, C[0][0]~C[1][1] are obtained;

[0149] (2) The second cycle, such as Figure 17 The W-direction input shown remains unchanged (here, the movement of 8 [8,8] matrix data in one cycle is saved), and the A-direction input is replaced by the 64 inputs in the second cycle of A; after one cycle, C[2][0]~C[3][1] are obtained;

[0150] (3) The calculation formula of C[m][n] obtained in each cycle is:

[0151]

[0152] (4) After 2 cycles, the complete output matrix C is obtained, completing [4,16]×[16,16] T =[4,16] matrix multiplication calculation.

[0153] High Performance Mode:

[0154] like Figure 18 The figure shows the split combination of A and W in high performance mode;

[0155] The PE array of the present disclosure is [4,16]×[16,16] T For high-performance vector-matrix multiplication operations, the data input method and calculation steps are as follows:

[0156] (1) In the first cycle, if Figure 19 As shown, according to Figure 18 As shown, the 8 [1,8] vectors in the first cycle of A are input into A0~A7 of the PE array respectively; the 2 [8,8] matrices in W are input into W00~W77 of the PE array respectively (each [8,8] matrix is ​​broadcast to 4 rows in the PE array); after one cycle, 8 [1,8] vectors to be accumulated are obtained, which are recorded as C`[m][n];

[0157] (2) The second cycle, such as Figure 20The input to A remains unchanged, and the input to W is replaced by the 8 matrix inputs of the second cycle in W. After one cycle, 8 [1,8] vectors are obtained. After accumulating them with the C`[m][n] of the previous cycle in the ACC unit, all C[m][n] are obtained, completing [4,16]×[16,16] T =[4,16] matrix multiplication calculation.

[0158] Example 4: [2,32]×[32,32] T Matrix multiplication operation

[0159] like Figure 21 The shown is [2,32]×[32,32] T Matrix multiplication, matrix A is the input data of A0~A7, the dimension of A is [2,32]. First, it is divided into two equal parts by row, A[0] and A[1], one of which is recorded as A[m]. Then A[m] is divided into four equal parts by column, A[m][0]~A[m][3]. Each of them is a [1,8] vector, one of which is recorded as A[m][n].

[0160] The matrix W is the input data of W00 to W77. The dimension of W is [32,32]. It is divided into four equal parts by row and then by column, splitting into 16 [8,8] matrices. One of them is denoted as W[i][j], where i represents the row and j represents the column.

[0161] Matrix C is the output of matrix multiplication. Its dimensions and numbering are the same as those of matrix A. Each small block in the figure is a [1,8] vector, and one of them is denoted as C[m][n];

[0162] Low Power Mode:

[0163] The PE array can complete 8 [1,8]×[8,8] T Vector multiplication matrix operation, such as Figure 22 The figure shows the splitting and combination of A and W in low power mode;

[0164] The PE array of the present disclosure is [4,16]×[16,16] T When performing vector multiplication matrix operations, the data input method and calculation steps are as follows:

[0165] (1) In the first cycle, if Figure 23 As shown, according to Figure 22 The splitting and combining method of A and W shown in the figure inputs the four [1,8] vectors in the first cycle of A into A0 to A7 of the PE array (each [1,8] vector is broadcast to two rows in the PE array); inputs the eight [8,8] matrices in W into W00 to W77 of the PE array respectively; after one cycle, C[0][0] and C[0][1] are obtained;

[0166] (2) In the second cycle, the W input remains unchanged (this saves 1 cycle 8 [8,8] T Matrix data movement), the input to A is replaced by the 8 [1,8] vectors of the second cycle in A; after one cycle, C[1][0] and C[1][1] are obtained;

[0167] (3) Similarly, according to Figure 22 The splitting and combining method of A and W shown in the figure, the calculation formula of C[m][n] obtained in each cycle is:

[0168]

[0169] (4) After 4 cycles, the complete output matrix C is obtained, completing [2,32]×[32,32] T =[2,32] matrix multiplication calculation.

[0170] High Performance Mode:

[0171] The vector multiplication matrix that needs to be calculated in the (k+1)th cycle of the PE array is A[m][k]×W[i][k]T (m=0,1;i=0,1,2,3); the vector multiplication matrix that needs to be calculated in each cycle of the 4-cycle PE array is as follows Figure 23A As shown;

[0172] When the PE array of the present disclosure performs a [4,16]×[16,16]T vector-matrix multiplication operation in high-performance mode, the data input method and calculation steps are as follows:

[0173] (1) Figure 23B As shown, in the (k+1)th cycle, the vector A[0][k] of dimension [1,8] is broadcast to A0~A3 of the PE array; similarly, the vector A[1][k] is broadcast to A4~A7 of the PE array; the matrix W[i][k] (i=0,1,2,3) is input to the i-th row and i+4-th row of the PE array; after 1 cycle, each PE unit outputs 1 number, and the outputs of the 8 PEs in the same row are concatenated into a vector to be accumulated with dimension [1,8], recorded as Ck[m][i];

[0174] (2) Figure 23C As shown, taking k = 0 as an example, A[0][0] is broadcast to A0~A3 of the PE array; A[1][0] is broadcast to A4~A7 of the PE array; 4 W[i][0] (i=0,1,2,3) are input into the first 4 rows and the last 4 rows of the PE array respectively;

[0175] (3) The output Psum of the 8 PE units in the (m*4+i)th row is concatenated to obtain a vector of dimension [1,8]. This vector is input to the ACC unit for accumulation, that is, it is added with the vector stored in the Buf unit in the previous cycle. The accumulated result is Ck[m][i];

[0176] (4) Output Ck[m][i] to the external Buf unit, waiting to be accumulated with the output results of the PE array in the next cycle in different time domains;

[0177] (5) Repeat steps (2) to (4) for A[m][n] (m=0,1;n=1,2,3). Each cycle, the output Ck[m][i] of the (m*4+i)th row of the PE array is accumulated in the ACC unit. The accumulated result of 4 cycles is:

[0178]

[0179] (6) After 4 cycles, the 4 C[0][i] (i=0,1,2,3) are concatenated to obtain a vector C[0] of dimension [1,32]. Similarly, a vector C[1] of dimension [1,32] is obtained. The vector multiplication matrix calculation of [2,32]×[32,32]T=[2,32] is completed in 4 cycles.

[0180] The matrix multiplication operations of different dimensions that can be implemented by the PE array of the present disclosure are shown in Table 1 below:

[0181] Table 1

[0182]

[0183]

[0184] In addition to supporting the matrix multiplication operations shown in the table above, the PE array of the present disclosure can also pad any W-direction input that is not a square matrix to a square matrix closest to its size in the table above; at the same time, pad the A-direction input to the same column width, [1,5]×[5,

[0185] 6] T For example, after zero padding and expansion, it can be [1,8]×[8,8] T Mode operation;

[0186] For any [N,s]×[s,s] T Matrix multiplication operations can be split into N [1,s]×[s,s] T The vector multiplication matrix operation can be completed in a maximum of N*cycle cycles (cycle is the number of cycles required for the original vector multiplication matrix operation).

[0187] In addition, the present disclosure also discloses the application of the arithmetic unit, in particular the PE array thereof, in a typical MPU. Furthermore, the present disclosure also discloses an MPU, the MPU including the arithmetic unit.

[0188] The layout of the PE array of the present disclosure in a typical MPU is as follows: Figure 24 As shown, where:

[0189] (1) Control is mainly used to generate various control signals to realize the control of each module;

[0190] In the present invention, the working modes are divided into high-performance mode and low-power mode. The default mode is high-performance mode. The working mode signals of the control are exemplarily distinguished by the following Table 2:

[0191] Table 2

[0192]

[0193] Assume that the two matrices to be multiplied are in the form of [m, s] × [n, s]T. In the present invention, the possible values ​​of the three dimensions m, s, and n are 1, 2, 4, and 8 (when the dimension is 3, the original dimension can be expanded to a matrix of dimension 4 by zero padding; when the dimension is 5, 6, or 7, the original dimension can be expanded to a matrix of dimension 8 by zero padding). Therefore, there are 64 matrix multiplication forms in total. The matrix dimension signal of control requires 6 bits to distinguish and represent, see Table 3 below;

[0194] Table 3

[0195]

[0196] The control's working mode signal and matrix dimension signal need to be transmitted to lm_A and lm_W to jointly determine the data acquisition method of matrices A and W in each cycle;

[0197] The working mode signal needs to be transmitted to PE_array, ACC and Buf to control the output result Psum of the PE unit for time domain accumulation or spatial domain summation;

[0198] The matrix dimension signal is transmitted to PE_array, ACC and Buf to determine the number of cycles of matrix multiplication operations of different dimensions of the PE array, thereby judging whether the matrix multiplication operation is completed and deciding whether Buf can transmit the operation results to the outside;

[0199] (2) lm_A (i.e., local memory A) and lm_W (i.e., local memory W) are storage resources of 64 bytes and 512 bytes respectively (assuming that the size of each input data number is 1 byte). They are mainly used to temporarily store the input in the A direction and the input in the W direction, that is, the data of the two matrices A and W to be multiplied;

[0200] lm_A and lm_W read the specified data in matrix A and matrix W based on the matrix dimension signal and working mode signal of control, and input them to the specified PE unit;

[0201] (3) PE_array is the PE array, which is mainly used to implement matrix operations in various modes. The adder inside the PE array is enabled based on the working mode signal of the control. In the default mode (high-performance mode), the output of the PE unit is directly transmitted to the ACC unit for time domain accumulation; in the low-power mode, the adder inside the PE array is enabled for spatial domain summation, and the operation results of the PE array do not need to be accumulated in the ACC unit;

[0202] (4) The ACC unit is mainly used to accumulate the output of the PE array in different time domains. In high-performance mode, the output of the 8 rows of PE units in the PE array is used as the first input of the 8 adders of the ACC unit; the 8 vectors stored in the Buf unit are used as the second input of the 8 adders of the ACC unit; finally, the ACC unit inputs the calculation results of the 8 adders into the Buf unit; in low-power mode, the ACC unit directly inputs the 8 vectors after the spatial summation of the PE array into the Buf unit;

[0203] (5) The Buf (Buffer) unit is mainly used to temporarily store the results of ACC accumulation, with an initial value of 0. Based on the control's working mode signal, the Buf unit determines whether to input the stored data into the ACC unit for accumulation. In high-performance mode, the value stored in the Buf unit serves as the second input of the ACC unit for time-domain accumulation. In low-power mode, the Buf unit does not transmit data to the ACC unit.

[0204] Based on the matrix dimension signal of control, the number of cycles of matrix multiplication operations of different dimensions can be determined. The Buf unit can judge whether the matrix multiplication operation is completed based on this count. When the operation is completed, the Buf unit transfers the stored data from the MPU.

[0205] It can be understood that, as is commonly understood in the processor field, control stands for control module, PE_array stands for PE array structure, ACC stands for accumulator, and Buf stands for cache.

[0206] In addition, it should be noted that:

[0207] In the high-performance mode of this disclosure, the PE array can calculate 8 different sets of [1,8]×[8,8] per cycle. T Operation, the time domain accumulation of calculation results is realized through the ACC unit. Therefore, the addition operation in the matrix multiplication is transferred to the ACC unit. The ACC unit performs an addition operation per cycle, thereby shortening the time of the PE array critical path and increasing the clock frequency;

[0208] Assuming that the latency to complete a vector multiplication operation (performed by the PE unit) is 1 ns and the latency to complete a vector addition operation is 0.3 ns, the critical path delays and clock frequencies of the matrix operations in Examples 2, 3, and 4 shown in the table in the two modes are shown in Table 4 below:

[0209] Table 4

[0210]

[0211] Among them, the addition operation in high-performance mode is performed in the ACC unit, not in the PE array, so its addition delay is not accumulated in the PE array critical path delay.

[0212] The main power consumption of the processor when working comes from data movement. Therefore, the present disclosure uses spatial locality to improve the utilization rate of single-time data reading, thereby effectively reducing power consumption. In the present disclosure, the amount of data moved in the W direction input per cycle is much larger than the amount of data moved in the A direction input. Therefore, the data input in the W direction should be reused first to reduce power consumption.

[0213] In the low-power mode of the present disclosure, the PE array can calculate eight different sets of [1,64]×[8,64]T operations per cycle. The control signals based on the matrix multiplications of different dimensions enable the specified adders in the PE array, which can realize the spatial summation of the calculation results.

[0214] Spatial summation maximizes the reuse of input data in the W direction. Therefore, for the same matrix multiplication operation, the low-power mode can effectively reduce the amount of data moved within the same number of operation cycles. The spatial summation result is directly transmitted to the ACC unit without accumulation, further reducing power consumption.

[0215] Assuming that the size of each input data number is 1 byte, the data transfer amount for one input in the A direction is 64 bytes, and the data transfer amount for one input in the W direction is 512 bytes. For the same matrix operations in Examples 2, 3, and 4, the data transfer amount in the two modes is shown in Table 5 below:

[0216] Table 5

[0217]

[0218] Taking Example 4 low power mode as an example, 4*64 means that the input data in the A direction is moved in 4 cycles, with a movement amount of 64 bytes per cycle; 2*512 means that the input data in the W direction is moved in 2 cycles, with a movement amount of 512 bytes per cycle.

[0219] It should be emphasized that for matrix multiplication with dimensions not exceeding [8,8], this type of operation can be completed in one cycle. There is no need to reduce data movement or accumulate the results of PE array operations in different time domains in the ACC unit. Therefore, the high-performance mode and low-power mode of this type of matrix multiplication are consistent.

[0220] In summary, the present disclosure has the following features:

[0221] In addition to being applied to end-side neural network acceleration chips in mobile phones, monitoring equipment, and automotive electronics, it can also be applied to acceleration boards on servers.

[0222] The PE operation unit in the present disclosure can be implemented as a fixed-point multiplication-accumulation operation or a floating-point multiplication-accumulation operation;

[0223] In addition to supporting matrix multiplication operations of 64 elements, the PE array of the present disclosure can also support operations of similar structures such as 4×4, 16×16, and rectangular matrix operations such as 4×8, 8×4, 8×16, and 16×8.

[0224] Although the embodiments of the present disclosure have been described above with reference to the accompanying drawings, the present disclosure is not limited to the specific embodiments and application areas described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. A person of ordinary skill in the art, guided by this specification and without departing from the scope of protection of the claims of the present disclosure, may devise various other forms, all of which fall within the scope of protection of the present disclosure.

Claims

1. An MPU, comprising: An arithmetic unit, comprising a PE array structure compatible with multi-dimensional matrix multiplication, The PE array structure includes: There are 64 PE units, and the address of each PE unit in the PE array is recorded as (i, j), where i represents the row and j represents the column; The PE array has two inputs: eight A-direction inputs A0 to A7 and 64 W-direction inputs W00 to W77. The A and W directions are orthogonal. W00 to W07 have eight sub-directions, W10 to W17 have eight sub-directions, and so on. W70 to W77 have eight sub-directions, for a total of 64 W-direction inputs. Each of A0 to A7 contains 8 numbers, which form a [1, 8] vector. Each of these 8 numbers is a [1, 8] vector, called the A-direction input. The 8 A-direction inputs A0 to A7 can be the same or different. Each of W00-W77 contains 8 numbers, which form a [1, 8] vector. Each of these 8 numbers is a [1, 8] vector, called the W-direction input. Each of the 64 W-direction inputs W00-W77 is sent to the corresponding PE unit. The 64 W-direction inputs W00-W77 can be the same or different. For each PE unit (Processing Element), where: As a basic processing unit in the PE array, it has two inputs, one A-direction input and one W-direction input, and one output; Taking the PE unit at position (i, j) as an example, the input is Ai and Wij, and the output is recorded as Psum(i, j); For the combination of 8 PE units in the first row of the PE array, vector A0 is input to the 8 PE units simultaneously in the horizontal direction; The vectors W00 to W07 in the first row of the PE array are regarded as eight column vectors of an [8, 8] matrix, and then input into the corresponding PE units respectively; One cycle of one row of the PE array can complete a [1, 8]×[8, 8] T Vector multiplication matrix operation; By performing different combinations of vector multiplication matrices on the 8 rows in the PE array, vector multiplication matrix and matrix multiplication matrix operations of different dimensions can be realized; The MPU also includes the following modules: control, ACC, Buf, lm_A (local memory A), lm_W (local memory W); control, used to generate various control signals to control other modules; lm_A and lm_W are used to store the input in the A and W directions; PE_array, used to implement matrix operations in various modes; ACC, used to accumulate the output of the PE array in different time domains; Buf is used to store the accumulated results of ACC. In addition, the results of unfinished operations need to be returned to ACC, and the results of completed operations are output from MPU. The default mode of the PE array structure is high performance mode; When performing matrix multiplication operations of any dimension, the input data is split and combined based on the control signal and input to the specified PE unit in a time-sharing manner, so that all 64 PE units in the PE array are in working state.

2. The MPU according to claim 1, wherein: When the control signal is the first control signal, the PE array structure operates in a low power consumption mode.

3. The MPU according to claim 1, wherein: By transmitting control signals in different modes, it supports matrix multiplication operations in multiple dimensions.

4. The MPU according to claim 1, wherein: For the same vector multiplication matrix operation, the PE array structure can operate in different modes of low power consumption or high performance, and the modes are switchable.

Images