calculator

Abstract

To improve area and power efficiency in large-scale computing units. [Solution] An arithmetic unit 1 that performs matrix multiplication operation C=A×B or matrix product sum operation C=A×B+Cin, wherein the matrix A is a block A which is an l×1 column vector. ik Divide the matrix B into blocks B, which are 1 × m row vectors. kj Divided into Block A ik and Block B kj The l × m cross product of the two is accumulated and summed over a processing tile 100 containing l × m processing elements 10, and output stationary systolic array arithmetic is performed in the entire arithmetic unit 1, using the processing tile as the unit.

Description

[Technical Field]

[0001] This invention relates to an arithmetic unit. [Background technology]

[0002] In particular, in applications such as AI (Artificial Intelligence), speeding up and reducing the power consumption of matrix multiply-accumulate (MMA) operations on low-precision elements, such as FP16 (half-precision floating-point numbers), has become important. In recent years, matrix multiply-accumulate units have tended to become larger in scale due to their use in LLMs (Large Language Models) and other applications.

[0003] Conventional matrix multiplication and summation operators include SIMD (Single Instruction / Multiple Data Stream) and Systolic Array (SA) types.

[0004] The SIMD type using the cross product method and the SA type using the output stationary method perform calculations using the outer product (or direct product) as shown in the following equation, rather than the cross product. Hereafter, the SIMD type using the cross product method and the SA type using the output stationary method will simply be referred to as SIMD type and SA type.

number

[0005] Both SIMD and SA types are two-dimensional arrays where FMA (Fused Multiply-Add) operators for matrix elements are used as Processing Elements (PEs). They share the common feature of performing matrix multiplication operations C=A×B or matrix sum operations C=A×B+Cin by inputting A and B from the edge of the array and accumulating and summing C using the accumulators (Acc) of each PE. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Special Publication No. 2021-508125 [Overview of the project] [Problems that the invention aims to solve]

[0007] A multi-array (SA) type matrix multiplier unit propagates input data (A, B) sequentially to adjacent PEs (in other words, a bucket brigade), resulting in a repeating structure for each PE. This allows the efficiency to remain nearly constant even when scaled up, making it scalable. On the other hand, an SA type matrix multiplier unit requires a flip-flop (FF) for each PE to propagate the input data, increasing area and power consumption (in other words, reducing efficiency). It also suffers from the problem of high latency proportional to the matrix size.

[0008] SIMD matrix multiplier units broadcast input data (A, B) to all PEs in the rows and columns respectively, eliminating the need for flip-flops (FFs) that relay input data to each PE in SA-type systems. This allows for smaller area and power consumption (in other words, higher efficiency) in small-scale applications. Latency is also reduced. However, SIMD matrix multiplier units face the challenge of not being scalable at large scales, as the area and power consumption required for broadcasting increase.

[0009] One aspect of this approach is to improve area and power efficiency in large-scale computing units. [Means for solving the problem]

[0010] In one aspect, the arithmetic unit is an arithmetic unit that performs a matrix multiplication operation C = A × B or a matrix product sum operation C = A × B + Cin, where matrix A is an l × 1 column vector block A ik Divide the matrix B into blocks B, which are 1 × m row vectors. kj Divided into, Block A ikand the block B kj The l×m outer product with it is cumulatively summed in a processing tile (PT) including l×m processing elements, and in the entire arithmetic unit, systolic array arithmetic in an output stationary method is performed in units of the processing tile.

Advantages of the Invention

[0011] On one side, in a large-scale arithmetic unit, the area and power efficiency can be improved.

Brief Description of the Drawings

[0012] [Figure 1] It is a diagram showing an L×M×N matrix product sum. [Figure 2] It is a diagram illustrating an outer product method SIMD type matrix product sum arithmetic unit in a related example. [Figure 3] It is a diagram illustrating an output stationary method SA type matrix product sum arithmetic unit in a related example. [Figure 4] It is a diagram showing matrix product sum arithmetic and block matrix product sum arithmetic. [Figure 5] It is a diagram for explaining an operation example of a block SA arithmetic unit in an example of an embodiment. [Figure 6] It is a diagram schematically showing a configuration example of a block SA arithmetic unit in an example of an embodiment. [Figure 7] It is a diagram for comparing the pipeline operations of matrix product sum arithmetic units of each method. [Figure 8] It is a diagram schematically showing a configuration example of a rectangular outer product type matrix multiplier. [Figure 9] It is a diagram schematically showing a configuration example of a rectangular outer product type matrix multiplier including a PT (Processing Tile) in a modified example. [Figure 10] It is a graph comparing the area and power efficiency of a matrix product sum arithmetic unit in a related example and a block SA arithmetic unit in an embodiment.

Modes for Carrying Out the Invention

[0013] [A] Related examples Figure 1 is a diagram representing the sum of products of L×M×N matrices.

[0014] The matrix operation formula illustrated in Figure 1 shows that matrix A is L x N, matrix B is N x M, and matrix C is L x M, and the matrix obtained by the operation A × B + C is substituted into the next matrix C.

[0015] The arrows labeled A1 in matrix A and A2 in matrix B will be described later in Figures 2 and 3.

[0016] Note that in the example shown in Figure 1, L=M (=4), but L≠M is also acceptable.

[0017] Figure 2 illustrates a SIMD (cross product) matrix product summation unit 6 in a related example.

[0018] In the SIMD-type matrix multiply-sum arithmetic unit 6 shown in Figure 2, the FMA arithmetic unit PE (Processing Element) 60 is arranged in L×M. Note that the shaded and diagonal rectangles in Figure 2 represent FF (Functional File-Based) operations.

[0019] a *k 1 row and b k* Each row and column are broadcast in the row and column directions, respectively. The arrows labeled A1 and A2 in Figure 2 are the same as those labeled A1 and A2 in Figure 1, indicating that matrices A and B are being broadcast in reverse.

[0020] Multiplication is performed at the PE60 of the broadcast intersection, c ij The process of outputting the cumulative sum is repeated N times. PE60 includes a multiplier 61, an adder 62, and an Acc (Accumulator) 63.

[0021] Figure 3 illustrates an output-stationary type SA matrix multiply-accumulate arithmetic unit 6a in a related example.

[0022] In the SA type matrix multiply-accumulate unit 6a shown in FIG. 3, the PE60 which is an FMA unit is arranged in L×M. Note that the shaded rectangles and the slanted rectangles in FIG. 3 represent FFs.

[0023] a *k one column of and b k* one row of are shifted and input in the row direction and the column direction respectively, and are bucket-brigaded between PEs. The arrows with symbols A1 and A2 in FIG. 3 are the same as the symbols A1 and A2 shown in FIG. 1, and represent that the A matrix and the B matrix are inverted and broadcasted.

[0024] Multiplication is performed at the PE60 at the intersection of the broadcast, and the cumulative sum (accumulate) is output to c ij The process of outputting is repeated until the A and B matrices pass through the array of PE60. The PE60 includes a multiplier 61, an adder 62, and an Acc (Accumulator) 63.

[0025] 〔B〕Embodiment Hereinafter, an embodiment will be described with reference to the drawings. However, the embodiment shown below is merely an example, and there is no intention of excluding various modifications and applications of technologies not explicitly shown in the embodiment. That is, the present embodiment can be variously modified and implemented without departing from its gist. Also, each figure does not mean that it includes only the components shown in the figure, and can include other functions and the like.

[0026] Hereinafter, in the drawings, since the same reference numerals denote the same parts, the description thereof will be omitted.

[0027] 〔B-1〕An example of the embodiment The matrix multiply-accumulate unit in an example of the embodiment has a two-layer structure of SIMD type and SA type. The lower layer is of SIMD type composed of l×m PEs, and this is called a processing tile (PT). The upper layer is SA with this PT as an element.

[0028] In one example of the embodiment, the matrix multiplication and summation unit is a "block matrix SA" that treats each matrix as a block matrix, the matrix product of each block is calculated using a SIMD type PT, and the entire matrix is ​​calculated using the SA type on a block-by-block basis.

[0029] By setting the PT to the size that provides the best efficiency (for example, l=m=8~16) and using SA in units of PT, power consumption, area saving, and scalability can be achieved.

[0030] Figure 4 illustrates matrix multiplication and summation operations and block matrix multiplication and summation operations. In Figure 4, the symbol B1 represents the matrix multiplication and summation operation, and the symbol B2 represents the block matrix multiplication and summation operation.

[0031] In symbol B1, matrix A is L rows and N columns, matrix B is N rows and M columns, and matrix C is L rows and M columns.

[0032] The matrix operation formula shown in sign B1 is for an l × 1 column vector A ik (See symbol B11) and the l × m row vector B kj (See symbol B12) The matrix is ​​divided into blocks, and the matrix operation formula shown in symbol B2 is obtained.

[0033] In symbol B2, matrix A is L / l rows and N columns, matrix B is N rows and M / m columns, and matrix C is L / l rows and M / m columns.

[0034] In block matrix multiplication operations, Block A ik ,B kj By recognizing them as elements and performing matrix multiplication and sum operations, recursive operations can be realized. The multiplication of elements is performed in block A. ik ,B kj This is the cross product.

[0035] Figure 5 illustrates an example of the operation of the block SA arithmetic unit 1 in one embodiment.

[0036] The block SA arithmetic unit 1 shown in Figure 5 is an example of an arithmetic unit and contains multiple (four in the example shown in Figure 5) PT100s. The shaded and diagonal rectangles in Figure 5 represent FFs.

[0037] In Figure 5, PT100 is arranged in rows L / l and columns M / m. For example, if L=M=4 and l=m=2, it is in rows 2 and columns 2.

[0038] A *k Column 1 and B k* The data is entered in a staggered manner and passed between PTs in a bucket brigade fashion, causing adjacent PT100s to be activated sequentially step by step. Each PT100 performs a cross product calculation in SIMD format, and C is accumulated.

[0039] In other words, in the case of block SA arithmetic unit 1, between adjacent PT100(Tu) and PT100(Td), Tu sends the block to Td before Tu finishes the cumulative sum of the cross product with respect to the block.

[0040] Figure 6 is a schematic diagram showing an example of the configuration of the block SA arithmetic unit 1 in one embodiment.

[0041] In the example shown in Figure 6, block SA arithmetic unit 1 includes PE10, which is an FMA arithmetic unit, in row L and column M (4 rows and 4 columns in the example shown in Figure 6). Of the PE10 in row L and column M, PE10 in row l and column m (2 rows and 2 columns in the example shown in Figure 6) is placed within one PT100. Note that the shaded and diagonal rectangles in Figure 6 represent FF.

[0042] PE10 comprises a multiplier 11, an adder 12, and an Acc 13.

[0043] The block SA arithmetic unit 1 shown in Figure 6 has fewer FFs in PT100 (or, in other words, l × m PE10) compared to the SA-type matrix product / sum arithmetic unit 6a in the related example shown in Figure 3.

[0044] Figure 7 is a diagram comparing the pipeline operation of matrix multiply-accumulate units of each type.

[0045] The symbol C1 represents the pipeline operation of the SA-type matrix multiply-accumulate unit 6a in the related example, the symbol C2 represents the pipeline operation of the block SA unit 1 in one example of the embodiment, and the symbol C3 represents the pipeline operation of the SIMD-type matrix multiply-accumulate unit 6 in the related example. The symbol C4 represents the pipeline operation of the SA-type matrix multiply-accumulate unit that has been shallowed to half its original depth.

[0046] In Figure 7, the horizontal direction represents the pipeline between data a and data b (for simplicity, only the data path for a is shown), and the vertical direction represents the pipeline within the FMA.

[0047] In the SA-type matrix multiplication and summation unit 6a, indicated by symbol C1, FF7s are arranged horizontally at intervals of PE60.

[0048] In the block SA arithmetic unit 1 shown in code C2, an FF2 is placed for every few PE10s (in other words, PT100s). Data is broadcast only to the PE10s within each PT100.

[0049] In the SIMD-type matrix multiply-accumulate unit 6 shown in code C3, no FF7s are placed between PE10s in the horizontal direction, and data is broadcast to all PE60s.

[0050] In the matrix multiplicand operator shown in code C4, shallow pipelining is employed. FF7 is omitted, resulting in a stage count of 1 / s, and the clock speed and throughput also become 1 / s.

[0051] In the diagram, the horizontal signal propagation direction is a bucket brigade of a and b. Although only a is shown in Figure 7 for simplicity, b is similar. In the horizontal direction of the diagram, there is no logic; it is only propagation and not critical.

[0052] On the other hand, the signal propagation method in the vertical direction in the diagram is critical, as it involves FMA logic. One cycle is allocated to each of the multiplier (×) and adder (+) within PE60 of code C4.

[0053] Block SA arithmetic unit 1, indicated by symbol C2, acts as a bucket brigade between a and b horizontally in the diagram, is not critical, and FF2 is thinned out. The placement of FF2 is skipped to the adjacent PT100 instead of the adjacent PE10. The adjacent PT100 operates with a 1-cycle delay, reducing the latency to 1 / l, 1 / m. Note that PE10 within PT100 is not originally dependent, so it automatically becomes parallel.

[0054] Furthermore, the block SA arithmetic unit 1, indicated by symbol C2, becomes critical within the PT100 and FMA in the vertical direction shown in the figure, and FF2 is not downsampled. The clock speed and throughput remain unchanged.

[0055] [B-2] Variation In the modified version, the matrix multiplier / sum operator is based on a SIMD (cross product) type matrix multiplier / sum operator, but the PE array is made "rectangular". For example, without reducing the size of the target matrix (= total number of Accs), only one of the rows or columns of the PE array is reduced to 1 / r (r>1). Instead of reducing the number of PEs (= number of FMAs) to 1 / r, each PE has r Accs and performs r FMA operations for each cross product operation.

[0056] In the modified matrix multiply-accumulate unit, the computation throughput decreases to 1 / r, but the transfer throughput, where the input data is passed along in a bucket brigade fashion, also decreases to 1 / r for both a and b. However, if the number of FMAs is the same, and both rows and columns are set to 1 / √r instead of just one, the transfer throughput decreases to only 1 / √r.

[0057] Figure 8 schematically shows an example of the configuration of a rectangular cross product type matrix multiplier 1a.

[0058] Figure 8 shows L=M=2,r=2,[a ikAn example of splitting is shown. The number of PE10a which is an FMA arithmetic unit is 1 / r = 1 / 2, each PE10a has r = 2 Acc13, and r = 2 FMA operations are performed per outer product. [a ik is split into r = 2 and executed in an interleaved (alternate) manner.

[0059] FIG. 9 is a diagram schematically showing a configuration example of a rectangular outer product type matrix multiplier 1b including PT100a in a modified example.

[0060] In the example shown in FIG. 9, the rectangular outer product type matrix multiplier 1b is applied to the block SA type PT100a.

[0061] PT100a includes an array of λ × μ PE10a. The rectangular outer product type matrix multiplier 1b performs the cumulative sum of the l × m outer products in a plurality of steps when either λ or μ is λ = l or μ = m, and the other is an integer of 2 ≤ λ < l or 2 ≤ μ < m.

[0062] In the bucket relay of data a and data b, the bit width becomes 1 / r, and block A ik , B kj In terms of units, it takes r cycles to propagate to the adjacent PT100a. However, the adjacent PT100a still operates with a 1-cycle delay. Since PT100a is of the SIMD type, during the r cycles of interleaved operation on different r rows of data a, b needs to continuously supply the same data. Therefore, [b kj requires a buffer to synchronize the timing with [a ik .

[0063] [C] Effect FIG. 10 is a graph comparing the area and power efficiency between the matrix product sum arithmetic unit 6, 6a in a related example and the block SA arithmetic unit 1 in an embodiment.

[0064] In the related example shown by symbol D1, the SA-type matrix multiply-sum unit 6a requires an FF7 for bucket brigade purposes for each PE60, and the efficiency remains constant regardless of scale. In other words, the SA-type matrix multiply-sum unit 6a is relatively inefficient and scalable.

[0065] In the related example shown in code D2, the SIMD-type matrix multiply-sum operator 6 is more efficient than the SA-type for small scales because it requires fewer FF7s. However, for large scales, the number of broadcast PE60s increases, leading to decreased efficiency. In other words, the SIMD-type matrix multiply-sum operator 6 is not scalable.

[0066] In the block SA arithmetic unit 1 of the embodiment shown by symbol D3, the point of highest efficiency for SIMD types can be maintained even at large scales. In other words, the block SA arithmetic unit 1 is highly efficient and scalable.

[0067] The matrix multiply-sum arithmetic unit in the above-described embodiment can achieve, for example, the following effects.

[0068] Block SA arithmetic unit 1 processes matrix A as an l×1 column vector of block A ik Divide the matrix B into blocks B, which are 1 × m row vectors. kj It is divided into blocks. Also, block SA arithmetic unit 1 is block A ik and Block B kj The l × m cross product is cumulatively summed in a PT containing l × m PE, and output-stationary systolic array arithmetic is performed throughout the entire arithmetic unit.

[0069] This makes it possible to improve area and power efficiency in large-scale computing units.

[0070] When matrix A is L rows and N columns, matrix B is N rows and M columns, and matrix C is L rows and M columns, block A ik The partitioning results in matrix A becoming L / l rows and N columns, and block B kj The partition results in matrix B having N rows and M / m columns, and matrix C having L / l rows and M / m columns.

[0071] As a result, the block division can be accurately executed.

[0072] Between adjacent PT100(Tu) and PT100(Td), the block SA calculator 1 causes Tu to send the block to Td before Tu finishes the cumulative sum of the outer product regarding the block. Typically, PT100(Tu) sends the block to PT100(Td) in the next cycle after receiving the block, and PT100(Tu) and PT100(Td) perform pipeline operation with a one-cycle shift.

[0073] As a result, pipeline operation in units of PT can be performed, and latency can be reduced.

[0074] PT100a includes an array of λ×μ PE10a. The rectangular outer product type matrix multiplier 1b performs the cumulative sum of the l×m outer product in a plurality of steps when either one of λ or μ is λ = l or μ = m and the other is an integer of 2 ≤ λ < l or 2 ≤ μ < m.

[0075] As a result, the transfer throughput of the bucket relay can be decreased.

[0076] 〔D〕Others The disclosed technology is not limited to the above-described embodiments, and can be implemented with various modifications without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment can be selectively adopted as necessary, or can be appropriately combined.

Explanation of Reference Numerals

[0077] 1: Block SA calculator 1a, 1b: Rectangular outer product type matrix multiplier 10, 10a: PE 11, 61: Multiplier 12, 62: Adder 13, 63: Acc 100, 100a: PT 6, 6a: Matrix product sum calculator

Claims

1. An arithmetic unit that performs matrix multiplication operation C = A × B or matrix product sum operation C = A × B + Cin, Matrix A is a column vector of block A ｉｋ Divided into, Matrix B is a 1 × m row vector, block B ｋｊ Divided into, Block A ｉｋ and Block B ｋｊ The cross product of l × m is cumulatively summed in a processing tile containing l × m processing elements. In the entire arithmetic unit, output-stationary systolic array operations are performed on the processing tile as the unit. Arithmetic unit.

2. When matrix A has L rows and N columns, matrix B has N rows and M columns, and matrix C has L rows and M columns, Block A ｉｋ The division into matrix A results in L / l rows and N columns, and block B ｋｊ By partitioning, matrix B becomes N rows and M / m columns, and matrix C becomes L / l rows and M / m columns. The arithmetic unit according to claim 1.

3. Between adjacent processing tiles Tu and Td, Before the processing tile Tu finishes calculating the cumulative sum of the cross product of the block, the processing tile Tu sends the block to the processing tile Td. The arithmetic unit according to claim 1 or 2.

4. The processing tile comprises an array of λ × μ processing elements, Either λ or μ is such that λ = l or μ = m, and the other is an integer such that 2 ≤ λ < l or 2 ≤ μ < m. The cumulative sum of the l × m cross product is performed in multiple steps. The arithmetic unit according to claim 1 or 2.

Images