Matrix multiplication array architecture

Abstract

The disclosure provides a matrix multiplication array architecture, which can be applied to the technical field of machine learning. The matrix multiplication array architecture comprises: a matrix multiplication array comprising M*P matrix multiplier arrays; a matrix multiplier array comprising N*Q matrix multiplication array units; a matrix multiplication array unit comprising a first multiplier, a second multiplier, an adder, and a partial sum accumulator; in each sub-period within a period, the first multiplier is configured to multiply a first matrix element of a first matrix and a second matrix element of a second matrix to obtain a first product result; the second multiplier is configured to multiply a third matrix element of the first matrix and a fourth matrix element of the second matrix to obtain a second product result; the adder is configured to add a partial sum result obtained in a previous sub-period, the first product result, and the second product result to obtain a partial sum result; and the partial sum accumulator is further configured to store the partial sum result obtained in the present sub-period until a matrix element of a target matrix is obtained after the end of the period.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and in particular to a matrix multiplication array architecture. Background Technology

[0002] Matrix multiplication is the most frequently occurring operator in deep neural networks, and it is often the bottleneck consuming hardware computing power and energy in the entire deep neural network computation. The matrix multiplication architecture based on systolic arrays is considered to be a high-energy-efficiency matrix multiplication solution. However, existing technologies with fixed-output matrix multiplication systolic arrays suffer from low multiplier utilization and a large number of required registers, resulting in high dynamic power consumption.

[0003] Therefore, how to realize a high-energy-efficiency pulsating array for matrix multiplication is a technical problem that needs to be solved in the existing technology. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a matrix multiplication array architecture.

[0005] According to a first aspect of this disclosure, a matrix multiplication array architecture is provided, comprising:

[0006] The matrix multiplication array consists of M×P matrix multiplication subarrays, where M is an integer greater than or equal to 1 and P is an integer greater than or equal to 1.

[0007] The above matrix multiplier array includes N×Q matrix multiplication array elements, where N is an integer greater than or equal to 1 and Q is an integer greater than or equal to 1;

[0008] Within a sub-cycle, the matrix elements of the first matrix in the same horizontal matrix multiplication array unit of the above matrix multiplication array are the same; the matrix elements of the second matrix in the same vertical matrix multiplication array unit of the above matrix multiplication array are the same.

[0009] The aforementioned matrix multiplication array unit includes a first multiplier, a second multiplier, an adder, and a partial sum accumulator;

[0010] In each sub-cycle within a period

[0011] The first multiplier described above is used to multiply the first matrix element of the first matrix and the second matrix element of the second matrix to obtain the first product result;

[0012] The second multiplier is used to multiply the third matrix element of the first matrix and the fourth matrix element of the second matrix to obtain the second product result.

[0013] The above part and accumulator are used to feed back the part and result obtained in the previous sub-cycle to the above adder;

[0014] The aforementioned adder is used to add the partial sum result obtained in the previous sub-cycle, the aforementioned first product result, and the aforementioned second product result to obtain the partial sum result;

[0015] The aforementioned part and accumulator are also used to store the part and result obtained in this sub-cycle until the matrix elements of the target matrix are obtained after one cycle ends, wherein the target matrix is ​​obtained by multiplying the aforementioned first matrix and the aforementioned second matrix.

[0016] According to an embodiment of this disclosure, within a sub-cycle, the first input data of the matrix multiplication array is a first matrix block in the first matrix, and the second input data of the matrix multiplication array is a second matrix block in the second matrix;

[0017] The first matrix block is obtained by dividing the first matrix into blocks of every M×N rows. The first matrix block is obtained by dividing the i-th first matrix block into blocks of every 2 columns, where i is an integer greater than or equal to 1. The second matrix block is obtained by dividing the second matrix into blocks of every P×Q columns. The second matrix block is obtained by dividing the j-th second matrix block into blocks of every 2 rows, where j is an integer greater than or equal to 1.

[0018] The number of sub-cycles within a period, the number of first matrix blocks corresponding to the i-th first matrix block, and the number of second matrix blocks corresponding to the j-th second matrix block are all the same.

[0019] According to an embodiment of this disclosure, the first matrix element input in the matrix multiplication array is the first matrix element in each row of the first matrix block, the third matrix element is the second matrix element in each row of the first matrix block, the second matrix element is the first matrix element in each column of the second matrix block, and the fourth matrix element is the second matrix element in each column of the second matrix block.

[0020] According to embodiments of this disclosure, the matrix multiplier arrays are connected by 2×N or 2×Q registers. Horizontally adjacent matrix multiplier arrays pulsate the first and third matrix elements through the registers, and vertically adjacent matrix multiplier arrays pulsate the second and fourth matrix elements through the registers.

[0021] According to embodiments of this disclosure, the matrix multiplication array unit further includes a bypass register, an AND gate, and a multiplexer. The AND gate is connected to the clock terminal of the bypass register, the output terminal of the multiplexer is connected to the bypass register, the first input terminal of the multiplexer is connected to the partial sum accumulator, and the second input terminal of the multiplexer is connected to the bypass register in the vertically adjacent previous matrix multiplication array unit.

[0022] According to an embodiment of this disclosure, when the matrix elements of the target matrix are obtained by the partial sum accumulator in the matrix multiplication array unit, the AND gate is used to control the bypass register to be in the open state, the multiplexer is used to output the matrix elements input from the first input terminal to the bypass register, and the AND gate is also used to control the bypass register to be in the closed state.

[0023] According to an embodiment of this disclosure, when the bypass register in each matrix multiplication array unit stores the matrix elements of the target matrix, the AND gate in the matrix multiplication array unit is used to control the bypass register to be in the open state, the multiplexer is used to output the matrix elements input from the second input terminal to the bypass register, and the bypass register is used to transfer at least one matrix element of the stored target matrix to the bypass register in the next vertically adjacent matrix multiplication array unit.

[0024] According to an embodiment of the present disclosure, when the bypass register in the matrix multiplication array unit passes at least one matrix element of the target matrix to the bypass register in the next vertically adjacent matrix multiplication array unit, the AND gate in the matrix multiplication array unit is used to control the bypass register to be in a closed state.

[0025] According to embodiments of this disclosure, the matrix multiplication array architecture further includes:

[0026] A first memory, wherein the first memory is used to store the first matrix;

[0027] The second memory is used to store the second matrix.

[0028] The buffer is connected to the first memory and is used to retrieve multiple first matrix blocks corresponding to the i-th first matrix block from the first memory and cache the multiple first matrix blocks corresponding to the first matrix block.

[0029] The encoding module is connected to the second memory. The encoding module is used to retrieve the second matrix block corresponding to the j-th second matrix block from the second memory and encode the matrix elements in the second matrix block.

[0030] The buffer is connected to the matrix multiplication array and is used to input the first matrix into the matrix multiplication array in blocks; the encoding module is connected to the matrix multiplication array and is used to input the encoded second matrix into the matrix multiplication array in blocks.

[0031] According to embodiments of this disclosure, the target matrix includes matrix blocks corresponding to each first matrix block; in the next period of calculating the matrix blocks in the target matrix using the i-th first matrix block,

[0032] The buffer described above is used to divide the multiple first matrices corresponding to the i-th first matrix block into blocks, and to sequentially input the first matrix blocks into the matrix multiplication array in each sub-cycle.

[0033] The encoder described above is used to divide the multiple second matrices corresponding to the (j+1)th second matrix block into blocks, and to input the second matrix blocks into the matrix multiplication array in each sub-cycle.

[0034] The matrix multiplication array described above is used to obtain the matrix block corresponding to the (j+1)th second matrix block in the target matrix based on the multiple first matrix blocks corresponding to the i-th first matrix block and the multiple second matrix blocks corresponding to the (j+1)-th second matrix block. The matrix block corresponding to the i-th first matrix block in the target matrix includes the matrix block corresponding to each second matrix block.

[0035] According to the matrix multiplication array architecture provided in this disclosure, the partial sum of the two-dimensional dot product is completed by two multipliers and an adder in the matrix multiplication array unit. The partial sum accumulator adjacent to the adder can be used to accumulate the partial sum results of different sub-cycles within one cycle to obtain a matrix element in the target matrix. By using the matrix multiplication array, the time interval between the completion of the partial sum calculation results of the first row and first column and the last row and first column is reduced. Only one sub-cycle is needed to enter the next round of matrix multiplication calculation. Due to the use of two multipliers in each matrix multiplication array unit, the utilization rate of the multipliers is improved, and the calculation of partial sums is also reduced, which improves the computing power of the matrix multiplication array. The number of partial sum accumulators used is reduced while the power consumption of the partial sum accumulators is reduced, thereby reducing the power consumption of the matrix multiplication array to a certain extent and further improving energy efficiency. Attached Figure Description

[0036] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0037] Figure 1 A schematic diagram illustrating matrix multiplication of an array according to an embodiment of the present disclosure is shown.

[0038] Figure 2 A schematic diagram illustrating matrix partitioning according to an embodiment of the present disclosure is shown.

[0039] Figure 3 A schematic diagram illustrating the structure of a matrix multiplier array according to an embodiment of the present disclosure is shown.

[0040] Figure 4 A schematic diagram of a matrix multiplication array architecture according to an embodiment of the present disclosure is shown.

[0041] Figure 5 A schematic diagram of the structure of a matrix multiplication array unit according to an embodiment of the present disclosure is shown. Detailed Implementation

[0042] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0043] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0044] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0045] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0046] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0047] In the process of implementing this disclosure, it was discovered that, INT8 multiplication Let's discuss the traditional architecture of a fixed-output matrix-multiplication systolic array based on the conditions of each multiplier. Under the aforementioned fixed conditions, due to each new round... Matrix multiplication calculations require the PE (Processing Element) in row 16, column 1 to complete the calculation. The multiplier utilization of this fixed-output matrix multiplication systolic array is low, at only 80.9%. Traditional fixed-output matrix multiplication systolic arrays only use systolic multiplexing for spatial multiplexing, not broadcasting. Furthermore, the multiplexing of partial sums only uses time multiplexing, not spatial multiplexing, resulting in a large number of required registers and high dynamic power consumption. The registers are defined as follows: b is a bit, and 22b and 8b are the bit widths of the registers in the array.

[0048] Based on the above, the low multiplier utilization in this fixed-output matrix multiplication systolic array will significantly impact the hardware's computing power, and the large number of required registers will significantly impact the hardware's power consumption. Therefore, the energy efficiency of this fixed-output matrix multiplication systolic array needs further improvement. To address this, embodiments of this disclosure provide a matrix multiplication array architecture capable of realizing a high-energy-efficiency systolic array for calculating fixed-point matrix multiplication.

[0049] Figure 1 A schematic diagram of a matrix multiplication array according to an embodiment of the present disclosure is shown.

[0050] like Figure 1 As shown, the matrix multiplication array architecture includes a matrix multiplication array 100, which can include M×P matrix multiplier arrays, where M is an integer greater than or equal to 1 and P is an integer greater than or equal to 1. When both M and P are 2, the matrix multiplication array 100 can include 2×2 matrix multiplier arrays Tile, namely matrix multiplier array 110_1, matrix multiplier array 110_2, matrix multiplier array 110_3, and matrix multiplier array 110_4.

[0051] According to embodiments of this disclosure, each matrix multiplier array may include N×Q matrix multiplier array units 120, where N is an integer greater than or equal to 1 and Q is an integer greater than or equal to 1; when both N and Q are 8, each matrix multiplier array may include 8×8 matrix multiplier array units DPE.

[0052] According to embodiments of this disclosure, the matrix elements of the first matrix can be input from the left side of the matrix multiplication array 100, and the matrix elements of the second matrix can be input from the top side of the matrix multiplication array 100. Based on the principle of matrix multiplication, within one sub-cycle, the matrix elements of the first matrix in the same horizontal matrix multiplication array unit of the input matrix multiplication array 100 are the same; the matrix elements of the second matrix in the same vertical matrix multiplication array unit of the input matrix multiplication array are the same.

[0053] For example, when N and Q are both 2 and M and P are both 8, the matrix multiplication array 100 includes 16 rows and 16 columns of matrix multiplication array units. Within one sub-cycle, the matrix elements of the first matrix in each row of the matrix multiplication array unit of the input matrix multiplication array 100 are the same; the matrix elements of the second matrix in each column of the input matrix multiplication array 100 are the same.

[0054] According to embodiments of the present disclosure, each matrix multiplication array unit 120 may include a first multiplier 121, a second multiplier 122, an adder 123, and a partial sum accumulator 124.

[0055] According to embodiments of this disclosure, in each sub-cycle within a cycle, the first multiplier 121 can be used to multiply the first matrix element of the input first matrix and the second matrix element of the second matrix to obtain a first product result. The second multiplier 122 can be used to multiply the third matrix element of the input first matrix and the fourth matrix element of the second matrix to obtain a second product result. According to embodiments of this disclosure, the partial sum accumulator 124 can be used to feed back the partial sum result obtained in the previous sub-cycle to the adder, and the adder 123 can be used to add the partial sum result obtained in the previous sub-cycle, the first product result, and the second product result to obtain the partial sum result of the current sub-cycle; the partial sum accumulator 124 can also be used to store the partial sum result obtained in the current sub-cycle until a matrix element of the target matrix is ​​obtained after a cycle ends, wherein the target matrix is ​​obtained by multiplying the first matrix and the second matrix, and the partial sum accumulator is also a register.

[0056] According to an embodiment of this disclosure, when M and P are both 2 and N and Q are both 8, the matrix multiplication array 100 includes 16×16 matrix multiplication array units. When matrix multiplication is performed using the matrix multiplication array 100, after one cycle, 16×16 matrix elements in the target matrix can be obtained through the matrix multiplication array 100. That is, each matrix multiplication array unit can obtain one matrix element in the target matrix within one cycle.

[0057] According to embodiments of this disclosure, the partial sum of a two-dimensional dot product is calculated using two multipliers and an adder in a matrix multiplication array unit. The partial sum accumulator adjacent to the adder can be used to accumulate the partial sums of different sub-cycles within a cycle to obtain a matrix element in the target matrix. By using a matrix multiplication array, the time interval between calculating the partial sums of the first row and first column and the last row and first column is reduced. Only one sub-cycle is needed before entering the next round of matrix multiplication calculation. Due to the use of two multipliers in each matrix multiplication array unit, the utilization rate of the multipliers is improved, while the calculation of partial sums is reduced, thus improving the computing power of the matrix multiplication array. The number of partial sum accumulators used is reduced, and the power consumption caused by the partial sum accumulators is also reduced, thereby reducing the power consumption of the matrix multiplication array to a certain extent and further improving energy efficiency.

[0058] Figure 2 A schematic diagram of matrix partitioning according to an embodiment of the present disclosure is shown.

[0059] like Figure 2 As shown, with Taking a case where M and P are both 2, and N and Q are both 8, the first matrix is ​​matrix A, the second matrix is ​​matrix B, and the target matrix is ​​matrix C.

[0060] According to an embodiment of this disclosure, within a sub-cycle, the first input data of the matrix multiplication array is a first matrix block in the first matrix, and the second input data of the matrix multiplication array is a second matrix block in the second matrix.

[0061] According to an embodiment of this disclosure, the first matrix block is obtained by dividing each M×N row of the first matrix into blocks, and the first matrix block is obtained by dividing each 2 columns of the i-th first matrix block into blocks, where i is an integer greater than or equal to 1.

[0062] According to embodiments of this disclosure, for the matrix Dividing the matrix into blocks of 16 rows yields 32 first matrix blocks, each with a size of 16×64 (16 rows and 64 columns). Since the matrix multiplication array unit includes two multipliers, the first matrix blocks can be divided into blocks of 2 columns each, resulting in 32 first matrix sub-blocks, each with a size of 16×2 (16 rows and 2 columns).

[0063] According to an embodiment of this disclosure, the second matrix block is obtained by dividing each P×Q column of the second matrix into blocks, and the second matrix block is obtained by dividing each 2 rows of the j-th second matrix block into blocks, where j is an integer greater than or equal to 1.

[0064] According to embodiments of this disclosure, for the matrix Dividing the matrix into blocks of 16 columns yields 32 second matrix blocks, each 64×16 in size (64 rows and 16 columns). Since the matrix multiplication array unit includes two multipliers, each second matrix block can be divided into blocks of 2 rows, resulting in 32 second matrix sub-blocks, each 2×16 in size (2 rows and 16 columns).

[0065] According to embodiments of this disclosure, within a sub-cycle, a first matrix block in the i-th first matrix block can be used as the first input data of the matrix multiplication array, and the corresponding second matrix block in the j-th second matrix block can be used as the second input data of the matrix multiplication array. Specifically, the s-th first matrix block from left to right in the i-th first matrix block corresponds to the s-th second matrix block from top to bottom in the j-th second matrix block; that is, when the s-th first matrix block in the i-th first matrix block is used as the first input data of the matrix multiplication array, the s-th second matrix block in the j-th second matrix block is used as the second input data of the matrix multiplication array, where s is an integer greater than or equal to 1.

[0066] According to embodiments of this disclosure, for When M and P are both 2, and N and Q are both 8, one period consists of 32 sub-periods. That is, the number of sub-periods, the number of first matrix blocks corresponding to the i-th first matrix block, and the number of second matrix blocks corresponding to the j-th second matrix block are the same in a period. This is so that in a certain sub-period, the first matrix block corresponding to the i-th first matrix block and the second matrix block corresponding to the j-th second matrix block are used as the input of the matrix multiplication array for this sub-period.

[0067] For example, in the 10th sub-cycle, the 10th first matrix block in the i-th first matrix block is used as the first input data of the matrix multiplication array, and the 10th second matrix block in the j-th second matrix block is used as the second input data of the matrix multiplication array.

[0068] According to an embodiment of this disclosure, within a sub-cycle, each matrix multiplication array unit can obtain a partial sum of a matrix element in matrix C. After a cycle ends, each matrix multiplication array unit can obtain a matrix element in matrix C. The matrix multiplication array can obtain a matrix block in matrix C, and the size of the matrix block is 16×16.

[0069] According to embodiments of this disclosure, based on the 16×16 matrix multiplication array units included in the matrix multiplication array and the two multipliers included in each matrix multiplication array unit, the first matrix and the second matrix are reasonably divided into blocks so that in each sub-cycle, the corresponding first matrix block and the second matrix block are input into the matrix multiplication array, so that the matrix multiplication array can perform the correct matrix multiplication operation to obtain the corresponding matrix elements in the target matrix.

[0070] Figure 3 A schematic diagram of the structure of a matrix multiplier array according to an embodiment of the present disclosure is shown.

[0071] like Figure 3 As shown, when both N and Q are 8, the matrix multiplier array includes 8×8 matrix multiplier array units (DPEs). The matrix multiplier array has 1 to 16 connecting lines that input matrix elements from the left side of the first matrix and 1 to 16 connecting lines that input matrix elements from the top side of the second matrix. These connecting lines are used to transmit data.

[0072] Each horizontal connection line is 8 bits, meaning each horizontal connection line includes 8 data lines, and each vertical connection line is 12 bits, meaning each vertical connection line includes 12 data lines.

[0073] According to an embodiment of this disclosure, within one sub-cycle, the first matrix block is divided into 16 rows and 2 columns, and the second matrix block is divided into 2 rows and 16 columns. Then, the two matrix elements of each row in the first matrix block are input into the matrix multiplication array unit of the corresponding row in the matrix multiplication array. That is, the two matrix elements of the first row in the first matrix block can be used as the input of the 16 matrix multiplication array units of the first row in the matrix multiplication array. Similarly, the two matrix elements of each column in the second matrix block are input into the matrix multiplication array unit of the corresponding column in the matrix multiplication array. That is, the two matrix elements of the first column in the second matrix block can be used as the input of the 16 matrix multiplication array units of the first column in the matrix multiplication array.

[0074] According to an embodiment of this disclosure, the first matrix element input in the matrix multiplication array is the first matrix element in each row of the first matrix block, the third matrix element is the second matrix element in each row of the first matrix block, the second matrix element is the first matrix element in each column of the second matrix block, and the fourth matrix element is the second matrix element in each column of the second matrix block.

[0075] According to embodiments of this disclosure, based on the calculation rules of matrix multiplication, the first matrix element input in each matrix multiplication array unit is the first matrix element of the corresponding row in the first matrix block, the third matrix element is the second matrix element of the corresponding row in the first matrix block, the second matrix element is the first matrix element of the corresponding column in the second matrix block, and the fourth matrix element is the second matrix element of the corresponding column in the second matrix block.

[0076] In one matrix multiplier array, the first and third matrix elements are broadcast to the eight matrix multiplier array cells in the corresponding row via corresponding connecting lines; the second and fourth matrix elements are broadcast to the eight matrix multiplier array cells in the corresponding column via corresponding connecting lines. That is, the matrix multiplier array achieves spatial reuse by broadcasting matrix elements.

[0077] According to an embodiment of this disclosure, taking matrix multiplier array 110_1 as an example, for the 8 matrix multiplication array units in the first row of the matrix multiplier array 110_1, the first matrix element in the first row of the first matrix block can be input as the first matrix element through the first horizontal connection line, and the second matrix element in the first row of the first matrix block can be input as the third matrix element through the second horizontal connection line; for the 8 matrix multiplication array units in the first column of the matrix multiplier array 110_1, the first matrix element in the first column of the second matrix block can be input as the second matrix element through the first vertical connection line, and the second matrix element in the first column of the second matrix block can be input as the fourth matrix element through the second vertical connection line.

[0078] According to embodiments of this disclosure, based on the calculation rules of matrix multiplication, the matrix elements in the first matrix block and the second matrix block are correctly input into the corresponding matrix multiplication array unit, so that the matrix multiplication array unit can perform calculations based on the calculation rules of matrix multiplication to obtain the matrix elements in the accurate target matrix.

[0079] With matrix As the first matrix, matrix Take the second matrix as an example.

[0080] in, , .

[0081] According to embodiments of this disclosure, in the first matrix block is The second matrix block is In the case where, in the first sub-period within a period, the first matrix is ​​divided into blocks as follows: The second matrix is ​​divided into blocks. Then, for the matrix multiplication array cell in the first row and first column, the first matrix element is... The second matrix element is The third matrix element is The fourth matrix element is Then the partial sum generated by the matrix multiplication array unit in the first sub-period is: The partial sum and result generated in the second sub-cycle are as follows: Then, at the end of the period, the result of multiplying the matrix by the array cells is: That is, the target matrix Matrix elements .

[0082] in, .

[0083] Figure 4 A schematic diagram of a matrix multiplication array architecture according to an embodiment of the present disclosure is shown.

[0084] like Figure 4 As shown, the matrix multiplication array architecture 400, in addition to the matrix multiplication array 100, also includes a first memory (ASRAM) 410, a buffer (A buffer) 420, a second memory (B SRAM) 430, and an encoding module (Booth encoding module) 440.

[0085] According to embodiments of this disclosure, the first memory 410 can be used to store a first matrix, such as a matrix. The second memory 430 can be used to store a second matrix, such as a matrix. .

[0086] According to an embodiment of the present disclosure, the buffer 420 is connected to the first memory 410 and can be used to retrieve multiple first matrix blocks corresponding to the i-th first matrix block from the first memory and cache the multiple first matrix blocks corresponding to the first matrix block in the buffer 420.

[0087] According to embodiments of this disclosure, with Figure 2 For example, within one period, based on the first matrix block in matrix A and the second matrix block in matrix B, a 16×16 matrix block in matrix C can be obtained. Based on the calculation rules of matrix multiplication, in order to obtain... Figure 2The remaining 16×16 matrix blocks to the right of the 16×16 matrix block shown in matrix C should have their first matrix block in the input matrix multiplication array unchanged. The 2nd to 32nd second matrix blocks from left to right in the second matrix are used as input to the matrix multiplication array in different periods to obtain a 16×512 matrix block in matrix C. Thus, the i-th matrix block in the first matrix can be buffered using buffer 420 until all the matrix elements of the corresponding 16 rows in the target matrix are obtained.

[0088] According to an embodiment of this disclosure, the encoding module 440 is connected to the second memory 430. The encoding module 440 is used to retrieve the second matrix block corresponding to the j-th second matrix block from the second memory 430 and encode the matrix elements in the second matrix block.

[0089] According to an embodiment of this disclosure, when M and P are both 2 and N and Q are both 8, since the 16 matrix multiplication array units in the corresponding column of the matrix multiplication array use the same matrix elements in the second matrix within one sub-cycle, the encoding module 440 may include 32 encoding circuits. Within one sub-cycle, the 32 encoding circuits are used to encode the 32 matrix elements in the second matrix block respectively, that is, to change the data from 8 bits to 12 bits.

[0090] The buffer is connected to the matrix multiplication array and is used to input the first matrix into the matrix multiplication array in blocks; the encoding module is connected to the matrix multiplication array and is used to input the encoded second matrix into the matrix multiplication array in blocks.

[0091] According to embodiments of this disclosure, since the i-th first matrix block is cached in buffer 420, which is a smaller memory than the first memory 410, fetching data from buffer 420 reduces power consumption compared to fetching data from the first memory, and realizes time multiplexing of matrix elements in the first matrix. In a traditional output fixed matrix multiplication pulse array, each multiplier has an encoding circuit to encode one of the matrix elements before multiplication. However, since each column of matrix multiplication array units in this disclosure uses the same matrix elements and encodes them before inputting them into the matrix multiplication array, that is, the multipliers in each column of matrix multiplication array units share a single encoding circuit, each matrix element in the second matrix block only needs to be encoded once before being broadcast, reducing the number of encoding circuits, and reducing the area, delay and power consumption of the multipliers, further improving energy efficiency.

[0092] According to embodiments of this disclosure, the matrix multiplier arrays are connected by 2×N or 2×Q registers. Horizontally adjacent matrix multiplier arrays pulsate the first and third matrix elements through registers, and vertically adjacent matrix multiplier arrays pulsate the second and fourth matrix elements through registers.

[0093] According to embodiments of this disclosure, horizontally adjacent matrix multiplier arrays are connected through 2×N registers, and vertically adjacent matrix multiplier arrays are connected through 2×Q registers.

[0094] According to embodiments of this disclosure, such as Figure 4 As shown, the matrix multiplier arrays are connected through register 450. With M and P both set to 2 and N and Q both set to 8, taking matrix multiplier arrays 110_1 and 110_2 as examples, matrix multiplier arrays 110_1 and 110_2 are connected through 16 registers to pulsately transmit matrix elements from the first matrix block input from the left side of matrix multiplier array 110_1 to matrix multiplier array 110_2.

[0095] For example, in Figure 3 In the case shown as matrix multiplier array unit 110_1, the 16 registers between matrix multiplier array 110_1 and matrix multiplier array 110_2 are sequentially connected to the right sides of horizontal connecting lines 1 to 16; Figure 3 In the case shown as matrix multiplier array unit 110_1, the 16 registers between matrix multiplier array 110_1 and matrix multiplier array 110_3 are sequentially connected to the lower side of vertical connecting lines 1 to 16.

[0096] According to embodiments of this disclosure, adjacent matrix multiplier arrays pulsate matrix elements through registers to achieve spatial reuse, maintaining the scalability of the array and reducing the number of registers used, thereby reducing the power consumption caused by registers and further improving energy efficiency.

[0097] Figure 5 A schematic diagram of the structure of a matrix multiplication array unit according to an embodiment of the present disclosure is shown.

[0098] like Figure 5 As shown, the matrix multiplication array unit 120 also includes a bypass register 510, an AND gate 520, and a multiplexer 530. The AND gate 520 is connected to the clock terminal of the bypass register 510, the output terminal of the multiplexer 530 is connected to the bypass register 510, the first input terminal of the multiplexer 510 is connected to the partial sum accumulator 124, and the second input terminal of the multiplexer 530 is connected to the bypass register in the vertically adjacent matrix multiplication array unit above it.

[0099] According to an embodiment of this disclosure, the bypass register 510 can be used to copy the matrix elements of the target matrix obtained in the copy section and the accumulator 124. The clock terminal of the bypass register 510 is connected to an AND gate 520 to implement clock gating. By eliminating the matrix elements of the overlapping period and calculating the first sub-cycle of the next period, the bypass register reduces the idle cycle of the multiplier and improves the utilization of the multiplier.

[0100] According to embodiments of this disclosure, when the matrix elements of the target matrix are obtained by the partial sum accumulator in the matrix multiplication array unit, an AND gate is used to control the bypass register to be in the open state, a multiplexer is used to output the matrix elements input from the first input terminal to the bypass register, and the AND gate is also used to control the bypass register to be in the closed state.

[0101] According to an embodiment of this disclosure, when the matrix elements of the target matrix are obtained in the partial sum accumulator 124 of the matrix multiplication array unit, the AND gate 520 can control the bypass register 510 to be in the open state, and the multiplexer 530 is used to output the matrix elements input by the partial sum accumulator 124 through the first input terminal to the bypass register, which is used to store a matrix element of the target matrix obtained by the corresponding matrix multiplication array unit.

[0102] According to an embodiment of this disclosure, after storing matrix elements in the bypass register, since there are cases where the bypass registers in other matrix multiplication array units in the matrix multiplication array have not yet stored matrix elements in the target matrix, AND gate 520 can first control the bypass registers in the matrix multiplication array unit to be in a closed state.

[0103] According to embodiments of this disclosure, since the bypass register remains active for a relatively short period of time, an AND gate connected to the clock input of the bypass register can be used to clock-gated the bypass register to prevent it from remaining on for extended periods. The bypass register is only turned on during the time it is used, thereby reducing the dynamic power consumption of the bypass register.

[0104] According to an embodiment of this disclosure, when the bypass register in each matrix multiplication array unit stores the matrix elements of the target matrix, the AND gate in the matrix multiplication array unit is used to control the bypass register to be in the open state, the multiplexer is used to output the matrix elements input from the second input terminal to the bypass register, and the bypass register is used to pass at least one matrix element of the stored target matrix to the bypass register in the next vertically adjacent matrix multiplication array unit.

[0105] According to an embodiment of this disclosure, when the bypass register in each matrix multiplication array unit stores matrix elements of the target matrix, the AND gate can control the bypass register to be turned on again, and the multiplexer is used to output the matrix elements input from the second input terminal to the bypass register. The bypass register can pass at least one matrix element in the stored target matrix down to the bypass register in the next vertically adjacent matrix multiplication array unit.

[0106] At least one matrix element includes the matrix element originally stored in the bypass register and the matrix elements stored in other bypass registers vertically upwards from the bypass register.

[0107] According to embodiments of this disclosure, with Figure 5 Taking the matrix multiplication array unit shown as an example, this matrix multiplication array unit can be the matrix multiplication array unit in the 2nd row and 1st column. When the bypass register in each matrix multiplication array unit stores the matrix elements of the target matrix, the multiplexer in this matrix multiplication array unit can output the matrix elements stored in the bypass register of the matrix multiplication array unit in the 1st row and 1st column to the bypass register in this matrix multiplication array unit through the second input terminal; this bypass register can pass the two matrix elements generated by the matrix multiplication array units in the 1st row and 1st column and the 2nd row and 1st column to the bypass register in the 3rd row and 1st column matrix multiplication array unit, until 16 matrix elements are output from the bypass register in the 16th row and 1st column of the matrix multiplication array.

[0108] According to embodiments of this disclosure, the matrix elements generated by the matrix multiplication array unit are pulsed out by a bypass register instead of by a partial sum accumulator, so that the time interval between two cycles of matrix multiplication calculation is 0, which improves the utilization of the multiplier. For example, the utilization of the multiplier in the matrix multiplication array architecture of this disclosure is 97%, which is 19.9% ​​higher than that of the traditional fixed output matrix multiplication pulse array.

[0109] According to embodiments of this disclosure, when a bypass register in a matrix multiplication array unit passes at least one matrix element of the target matrix to a bypass register in the next vertically adjacent matrix multiplication array unit, an AND gate in the matrix multiplication array unit is used to control the bypass register to be in a closed state.

[0110] According to embodiments of this disclosure, when a bypass register passes a matrix element to a bypass register in the next vertically adjacent matrix multiplication array unit, an AND gate can control the bypass register to be in a closed state, reducing unnecessary dynamic power consumption of the bypass register.

[0111] According to embodiments of this disclosure, the target matrix includes matrix blocks corresponding to each first matrix block; in the next cycle of calculating matrix blocks in the target matrix using the i-th first matrix block, a buffer is used to divide the multiple first matrix blocks corresponding to the i-th first matrix block into blocks, and sequentially input the first matrix blocks into a matrix multiplication array in each sub-cycle; an encoder is used to divide the multiple second matrix blocks corresponding to the (j+1)-th second matrix block into blocks, and sequentially input the second matrix blocks into a matrix multiplication array in each sub-cycle; the matrix multiplication array is used to obtain the matrix block corresponding to the (j+1)-th second matrix block in the target matrix based on the multiple first matrix blocks corresponding to the i-th first matrix block and the multiple second matrix blocks corresponding to the (j+1)-th second matrix block, wherein the matrix block corresponding to the i-th first matrix block in the target matrix includes matrix blocks corresponding to each second matrix block.

[0112] According to embodiments of this disclosure, with Figure 2 For example, the 16×512 matrix block in the target matrix corresponds to the i-th first matrix block. That is, based on the i-th first matrix block and the 32 second matrix blocks in the second matrix, the 16×512 matrix block in the target matrix can be obtained. In other words, based on the i-th first matrix block, the 16×512 matrix block in the target matrix can be obtained after 32 cycles.

[0113] According to an embodiment of this disclosure, in the next cycle of calculating the matrix block in the target matrix using the i-th first matrix block, the buffer can reuse multiple first matrix blocks corresponding to the i-th first matrix block to sequentially input the first matrix blocks into the matrix multiplication array in each sub-cycle of the cycle; while the encoder sequentially encodes the matrix elements in multiple second matrix blocks corresponding to the j+1-th second matrix block and inputs them into the matrix multiplication array in each sub-cycle of the cycle.

[0114] According to embodiments of this disclosure, within this period, matrix multiplication arrays, based on the i-th first matrix block and the (j+1)-th second matrix block, can obtain a matrix block in the target matrix corresponding to the (j+1)-th second matrix block. For example, the matrix block in the target matrix corresponding to the (j+1)-th second matrix block is a 16×16 matrix block within a 16×512 matrix block corresponding to the i-th first matrix block in the target matrix.

[0115] According to embodiments of this disclosure, based on the calculation rules of matrix multiplication, the i-th first matrix block in the first matrix is ​​reused, and multiple cycles are performed using a matrix multiplication array architecture according to each second matrix block in the second matrix, so that a matrix block of size 16×512 in the target matrix can be obtained. Since the buffer makes the i-th first matrix block reused, the power consumption of the matrix multiplication array architecture is reduced.

[0116] According to embodiments of this disclosure, when M and P are both 2, and N and Q are both 8, a matrix is ​​used... As the first matrix, with Taking the second matrix as an example, within one cycle, based on the first matrix block of size 16×64 and the corresponding second matrix block of size 64×16, a 16×16 matrix block in the target matrix can be obtained through the matrix multiplication array architecture 400. That is, each matrix multiplication array unit in the matrix multiplication array 100 can obtain one matrix element of the target matrix within one cycle. The matrix elements generated by each of the 16×16 matrix multiplication array units in the matrix multiplication array correspond one-to-one with the positions of the matrix elements in the 16×16 matrix block of the target matrix. After 32×32 cycles, the target matrix, i.e., matrix... .

[0117] For example, the matrix element generated by the matrix multiplication array cell in the 10th row and 6th column of the matrix multiplication array is the matrix element in the 10th row and 6th column of the corresponding 16×16 matrix block in the target matrix.

[0118] According to embodiments of this disclosure, in order to ensure a low total access volume for matrix multiplication based on a matrix multiplication array architecture, it is preferable to set M and P to 2 and N and Q to 8.

[0119] According to embodiments of this disclosure, based on the matrix multiplication array architecture 400 described above, the multiplication data type used in the matrix multiplication array architecture 400 can be INT8. When M and P are both 2 and N and Q are both 8, the number of multipliers is 2×16×16. In the matrix multiplication array architecture of this disclosure, the three types of space multiplexing in the matrix multiplier array—broadcast, systolic, partial, and reduced—are combined, which greatly reduces the number of registers, reduces the power consumption caused by registers, and improves energy efficiency. The number of registers required in this disclosure is 16×2×12b + 16×2×8b + 22×2×16×16b = 11904b, which is 38.8% less than that of the traditional fixed-output matrix multiplication systolic array. In this disclosure, since the matrix elements are encoded before being input into the matrix multiplication array, each column of multipliers shares one encoding circuit. The traditional fixed-output matrix multiplication systolic array requires 16×32 = 512 encoding circuits, while this disclosure only requires 16×2 = 32 encoding circuits, which is 93.75% less than that of the traditional systolic array.

[0120] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0121] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A matrix multiplication array architecture, comprising: A matrix multiplication array, comprising M×P matrix multiplication subarrays, where M is an integer greater than or equal to 1 and P is an integer greater than or equal to 1; The matrix multiplier array comprises N×Q matrix multiplier array units, where N is an integer greater than or equal to 1 and Q is an integer greater than or equal to 1. Within a sub-cycle, the matrix elements of the first matrix in the same horizontal matrix multiplication array unit of the input matrix multiplication array are the same; the matrix elements of the second matrix in the same vertical matrix multiplication array unit of the input matrix multiplication array are the same. The matrix multiplication array unit includes a first multiplier, a second multiplier, an adder, and a partial sum accumulator; In each sub-cycle within a period The first multiplier is used to multiply the first matrix element of the input first matrix and the second matrix element of the second matrix to obtain a first product result; The second multiplier is used to multiply the third matrix element of the input first matrix and the fourth matrix element of the second matrix to obtain a second product result; The portion and accumulator are used to feed back the portion and result obtained in the previous sub-cycle to the adder; The adder is used to add the partial sum result obtained in the previous sub-cycle, the first product result, and the second product result to obtain the partial sum result; The portion and accumulator are also used to store the portion and result obtained in this sub-cycle until the matrix elements of the target matrix are obtained after one cycle ends, wherein the target matrix is ​​obtained by multiplying the first matrix and the second matrix.

2. The architecture according to claim 1, wherein, Within a sub-period, the first input data of the matrix multiplication array is the first matrix block in the first matrix, and the second input data of the matrix multiplication array is the second matrix block in the second matrix; The first matrix block is obtained by dividing the first matrix into blocks of every M×N rows, and the first matrix block is obtained by dividing the i-th first matrix block into blocks of every 2 columns, where i is an integer greater than or equal to 1; the second matrix block is obtained by dividing the second matrix into blocks of every P×Q columns, and the second matrix block is obtained by dividing the j-th second matrix block into blocks of every 2 rows, where j is an integer greater than or equal to 1. The number of sub-cycles within a period, the number of first matrix blocks corresponding to the i-th first matrix block, and the number of second matrix blocks corresponding to the j-th second matrix block are the same.

3. The architecture according to claim 2, wherein, The first matrix element input in the matrix multiplication array is the first matrix element in each row of the first matrix block, the third matrix element is the second matrix element in each row of the first matrix block, the second matrix element is the first matrix element in each column of the second matrix block, and the fourth matrix element is the second matrix element in each column of the second matrix block.

4. The architecture according to claim 1, wherein, The matrix multiplier arrays are connected by 2×N or 2×Q registers. Horizontally adjacent matrix multiplier arrays pulsate the first matrix element and the third matrix element through the registers, and vertically adjacent matrix multiplier arrays pulsate the second matrix element and the fourth matrix element through the registers.

5. The architecture according to claim 1, wherein, The matrix multiplication array unit further includes a bypass register, an AND gate, and a multiplexer. The AND gate is connected to the clock terminal of the bypass register, the output terminal of the multiplexer is connected to the bypass register, the first input terminal of the multiplexer is connected to the partial sum accumulator, and the second input terminal of the multiplexer is connected to the bypass register in the vertically adjacent previous matrix multiplication array unit.

6. The architecture according to claim 5, wherein, When the partial sum accumulator in the matrix multiplication array unit obtains the matrix elements of the target matrix, the AND gate is used to control the bypass register to be in the open state, the multiplexer is used to output the matrix elements input from the first input terminal to the bypass register, and the AND gate is also used to control the bypass register to be in the closed state.

7. The architecture according to claim 6, wherein, When the bypass register in each matrix multiplication array unit stores the matrix elements of the target matrix, the AND gate in the matrix multiplication array unit is used to control the bypass register to be in the open state, the multiplexer is used to output the matrix elements input from the second input terminal to the bypass register, and the bypass register is used to pass at least one matrix element of the stored target matrix to the bypass register in the next vertically adjacent matrix multiplication array unit.

8. The architecture according to claim 7, wherein, When the bypass register in the matrix multiplication array unit passes at least one matrix element of the target matrix to the bypass register in the next vertically adjacent matrix multiplication array unit, the AND gate in the matrix multiplication array unit is used to control the bypass register to be in the closed state.

9. The architecture according to claim 3, further comprising: A first memory, used to store the first matrix; A second memory, used to store the second matrix; A buffer, connected to the first memory, is used to retrieve multiple first matrix blocks corresponding to the i-th first matrix block from the first memory and cache the multiple first matrix blocks corresponding to the first matrix block; An encoding module is connected to the second memory. The encoding module is used to retrieve the second matrix block corresponding to the j-th second matrix block from the second memory and encode the matrix elements in the second matrix block. The buffer is connected to the matrix multiplication array and is used to input the first matrix into the matrix multiplication array in blocks; The encoding module is connected to the matrix multiplication array and is used to input the encoded second matrix into the matrix multiplication array in blocks.

10. The architecture according to claim 9, wherein, The target matrix includes matrix blocks corresponding to each first matrix block; in the next period of calculating the matrix blocks in the target matrix using the i-th first matrix block, The buffer is used to divide the multiple first matrices corresponding to the i-th first matrix block into blocks, and to sequentially input the first matrix blocks into the matrix multiplication array in each sub-cycle. The encoder is used to divide the multiple second matrices corresponding to the (j+1)th second matrix block into blocks, and input the second matrix blocks into the matrix multiplication array in each sub-cycle. The matrix multiplication array is used to obtain the matrix block corresponding to the (j+1)th second matrix block in the target matrix based on multiple first matrix blocks corresponding to the i-th first matrix block and multiple second matrix blocks corresponding to the (j+1)-th second matrix block, wherein the matrix block corresponding to the i-th first matrix block in the target matrix includes the matrix block corresponding to each second matrix block.

Images