A computer architecture for matrix-matrix multiplication

Abstract

A computer architecture for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprises: a systolic array of processing elements configured to: load stationary values from a permutated matrix A, which is based on a matrix A having columns with indexes 0 to N-1 and the permutated matrix A is formed by shifting and rotating each column, by the column index; load, in a first direction, each row of values from a matrix B; multiply the stationary value from the permutated matrix A with the value from the matrix B to produce a product; accumulate the product with a partial sum input received from another processing element; and output a partial sum output resulting from the accumulation; wherein the processing elements are configured to transfer the values from the matrix B to an adjacent row via a diagonal connection with respect to the first direction.

Description

[0001] A Computer Architecture for Matrix-Matrix Multiplication

[0002] FIELD OF DISCLOSURE

[0003] The disclosure relates to a computer architecture for matrix-matrix multiplication.

[0004] BACKGROUND

[0005] Natural Language Processing (NLP) is an emerging application of artificial intelligence (Al). NLPs use transformer models as they can provide a highly effective and scalable architecture for language understanding and generation.

[0006] However, transformer models are extremely data intensive. They deal with massive workloads in comparison to Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs). A clear example of the new challenging level of scalability is Generative Pre-trained Transformer (GPT) models that incorporate billions of parameters that allow matrix-matrix multiplication. Conventional Von Neumann architectures are struggling to meet these new performance demands due to the bottleneck resulting from data-movement between the memory and the processor in addition to the general purpose processing capabilities of these architectures.

[0007] Some specific Al architectures try to address these problems by using a Tensor Processing Unit (TPU) as an Al accelerator to handle large scale matrix-matrix multiplications with higher performance and energy efficiency than Central Processing Units (CPUs) and Graphics Processing Units (GPUs).

[0008] In the core of TPU models are systolic arrays that aim to maximise data utilisation of both weights and inputs to mitigate the memory / data-movement problem. Systolic arrays usually consist of two-dimensional (2D) interconnecting processing elements (PEs) comprising multiplication and accumulation functions along with register units. They can enhance local data utilisation by increasing the number of computational operations per each memory access. Data flows among different PEs in a systolic fashion, while the communication with an external buffer (or main memory) occurs only at the boundary PEs.

[0009] 55613365-1 Weights are kept stationary in the systolic array, and the inputs are reused and shifted in one direction (for example horizontally), while partial sums are propagated in the other direction (for example vertically). As the dataflow is based on a conventional weight-stationary systolic array, synchronization First-In-First-Out (FIFO) buffers are required to synchronize both inputs and outputs which adds a huge hardware penalty from area, power and energy perspectives. Moreover, the computations propagate as a diagonal wavefront from one corner (top-left) to the opposite corner (bottom-right) of the systolic array due to the weight-stationary dataflow, which means many PEs are idle for several clock cycles at the beginning and end of each matrix-matrix multiplication. Thus, the overall hardware utilization is significantly reduced resulting in a degraded performance with decreased throughput and increased latency.

[0010] Eyeriss is a reconfigurable Al accelerator with more focus on Convolutional Neural Networks (CNNs). It consists of a systolic array of 168 PEs with a Row Stationary (RS) dataflow that is more suitable for convolutional operations. Eyeriss has a coarsegrained PE that has local memory blocks to capture inputs, filters (weights) and partial sums internally. This RS dataflow depends on broadcasting inputs diagonally and weights horizontally to different PEs, which means multiple copies of the same data are saved by multiple PEs. This decreases local data utilization and increases the hardware penalty (in term of area, energy and power) associated with storing the data in multiple PEs.

[0011] Meissa is a systolic array that separates multipliers from adders rather than combining them in a unified array of multiply-add components (MACs). Like TPUs, Meissa has adopted a weight-stationery dataflow for the systolic array, but eliminates the input synchronization FIFOs to reduce the overall latency. However, it has bulky adder trees for each column of the systolic array instead of having the partial sums accumulated on-the-fly through the PEs. These adder trees impose scalability limitations due to serious implementation challenges. The larger the adder trees, the deeper pipelines required to achieve good operational frequency (speed). This increases not only the overall latency but also area and energy consumption. The routing congestion is another expensive challenge, caused by delivering all products from all PEs in the same column to the adder tree. Consequently, Meissa is not scalable to large NxN dimensions, which is vital for large-scale Al models. Moreover, it is not able to remove

[0012] 55613365-1 the output synchronization FIFOs which still add a considerable area, energy and power penalty.

[0013] Thus, all known weight-stationary architectures require input and output, or at least output, FIFOs for dataflow synchronization and they have scalability challenges due to the high cost of synchronization FIFOs and bulky adder trees.

[0014] It is an aim of the present disclosure to provide a computer architecture to ameliorate one or more of the problems above or at least provide a useful alternative.

[0015] SUMMARY

[0016] In general, this disclosure proposes a novel Diagonal-Input Permutated (DiP) Weight- Stationary (WS) Systolic Array (SA) that overcomes the main challenges of conventional weight-stationery systolic arrays, as detailed above. The DiP WS SA comprises a spatial systolic array architecture of processing elements (PEs) configured for diagonal-input movement and permutated weight-stationary dataflow for the acceleration of Matrix-Matrix Multiplication (MMM). Notably, the proposed architecture eliminates the input and output interfacing complexity of known weight-stationary (e.g. Google™ TPU-like) systolic-arrays.

[0017] According to a first aspect of the present disclosure, there is provided a computer architecture for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprising: a systolic array of processing elements, the processing elements being configured to: load stationary values from a permutated matrix A, wherein the permutated matrix A is based on a matrix A having columns with indexes 0 to N- 1 and wherein the permutated matrix A is formed by shifting and rotating each column in a rotation direction, by the column index; load, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements; multiply the stationary value from the permutated matrix A with the value from the matrix B to produce a product;

[0018] 55613365-1 accumulate the product with a partial sum input received in the first direction from another processing element, when available; and output, in the first direction, a partial sum output resulting from the accumulation; wherein the processing elements are configured to transfer the values from the matrix B to an adjacent row via a diagonal connection with respect to the first direction, and wherein a first processing element in a row is diagonally connected to a last processing element in the adjacent row.

[0019] Thus, the disclosure provides a computer architecture which is suitable for accelerating the matrix-matrix multiplication which is at the core of various Al models, including not only transformer models for Natural Language Processing (NLP) but also Deep Neural Networks (DNNs).

[0020] In contrast to TPUs’ weight-stationary systolic arrays, the proposed architecture eliminates the synchronization FIFOs for both inputs and outputs due to the diagonal propagation of values (e.g. inputs) from matrix B and the alignment of the permutated stationary values (e.g. weights) from matrix A.

[0021] By eliminating the need for input and output FIFOs, the architecture increases significantly, not only throughput, but also energy efficiency and hardware utilization. Consequently, the architecture decreases latency, area and power consumption.

[0022] The proposed architecture also eliminates the need for adder trees, resulting in increased scalability, performance and energy efficiency and a lower area footprint.

[0023] The proposed architecture maximises internal data utilization by saving only one copy of the data (values) at a time in the systolic array. In addition, the architecture depends on fine-grained PEs with no local memory blocks. Consequently, the proposed architecture decreases the area footprint of both the PE and the systolic array, therefore facilitating higher performance and energy efficiency.

[0024] Additionally, each PE comprises its own adder to accumulate the product with a partial sum input received in the first direction from another processing element, thereby enabling scalability to higher dimension arrays without the need for bulky adder trees.

[0025] 55613365-1 Consequently, the proposed dataflow introduces higher scalability and energy efficiency for large-scale Al models like transformer models.

[0026] Notably, the proposed architecture requires permutation of matrix A to be done before loading the values (e.g. weights) of matrix A into the systolic array. However, the values of matrix A may be pre-trained using software and can be permutated at the software level for use in the proposed architecture for effectively no cost. However, in some embodiments, hardware may be configured to form the permutated matrix A.

[0027] The shifting and rotating of each column by the column index may be performed in an upwards rotation direction by shifting each column upwardly by the column index and moving the values from the permutated matrix A that are shifted out of a top row of each column to a bottom row of each column.

[0028] Alternatively, the shifting and rotating of each column by the column index may be performed in a downwards rotation direction by shifting each column downwardly by the column index and moving the values from the permutated matrix A that are shifted out of a bottom row of each column to a top row of each column.

[0029] The adjacent row may be disposed in the first direction, with respect to the row of the first processing element. It will be understood that a row viewed in one orientation may be considered to be a column when viewed in another orientation and therefore the term row is used herein to mean a row or a column, and is not intended to be limited by a specific orientation of the array.

[0030] The first processing element may be located in a row i and a column j, wherein i and j are non-negative integers, and the first processing element may be configured to transfer the values from the matrix B to a second processing element located in a row i+1 or i-1 and a column j-1 or j+1. Notably, i and j are defined as non-negative integers because the first column has an index of 0, which is not positive.

[0031] The processing elements may be configured to load the first row of values from the matrix B at a same time as loading a last row of stationary values from the permutated matrix A.

[0032] 55613365-1 The computer architecture may further comprise a controller configured to generate the permutated matrix A based on the matrix A.

[0033] According to a second aspect of the present disclosure, there is provided a computer- implemented method for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprising: generating a permutated matrix A based on a matrix A having columns with indexes 0 to N-1 by shifting and rotating each column in a rotation direction, by the column index; loading, as stationary values, the permutated matrix A into a systolic array of processing elements; loading, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements; multiplying the stationary value from the permutated matrix A with the value from the matrix B to produce a product; accumulating the product with a partial sum input received in the first direction from another processing element, when available; outputting, in the first direction, a partial sum output resulting from the accumulation; and transferring the values from the matrix B to an adjacent row via a diagonal connection with respect to the first direction, wherein a first processing element in a row is diagonally connected to a last processing element in the adjacent row.

[0034] According to a third aspect of the present disclosure, there is provided a computer architecture for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprising: a systolic array of processing elements, the processing elements being configured to: load stationary values from a permutated transposed matrix A, wherein the permutated transposed matrix A is based on a transposed matrix A having columns with indexes 0 to N-1 and wherein the permutated transposed matrix A is formed by shifting and rotating each column in a rotation direction, by the column index;

[0035] 55613365-1 load, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements; multiply the stationary value from the permutated transposed matrix A with the value from the matrix B to produce a product; accumulate the product with a partial sum input received from another processing element, when available, to produce a partial sum output; and output, in the first direction, the values from the matrix B; wherein the processing elements are configured to transfer the partial sum output to an adjacent row via a diagonal connection, with respect to the first direction, and wherein a first processing element in a row is diagonally connected to a last processing element in an adjacent row.

[0036] Thus, the disclosure provides a further computer architecture which is suitable for accelerating the matrix-matrix multiplication by diagonal propagation of partial sums, instead of diagonal propagation of the values (e.g. inputs) from the matrix B. In this case, a different weight stationary permutation (based on a transposed matrix A) is required to eliminate the input and output FIFOs. However, similar benefits to those described above are also true of this further computer architecture.

[0037] The first processing element may be located in a row i and a column j, wherein i and j are non-negative integers, and the first processing element may be configured to transfer the partial sum output to a second processing element located in a row i+1 or i- 1 and a column j-1 or j+1.

[0038] The computer architecture may further comprise a controller configured to generate the permutated transposed matrix A based on the transposed matrix A.

[0039] The controller may be further configured to generate the transposed matrix A.

[0040] According to a fourth aspect of the present disclosure, there is provided a computer- implemented method for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprising:

[0041] 55613365-1 creating a permutated transposed matrix A based on a transposed matrix A having columns with indexes 0 to N-1 by shifting and rotating each column in a rotation direction, by the column index; loading, as stationary values, the permutated transposed matrix A into a systolic array of processing elements; loading, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements; multiplying the stationary value from the permutated transposed matrix A with the value from the matrix B to produce a product; accumulating the product with a partial sum input received from another processing element, when available, to produce a partial sum output; outputting, in the first direction, the values from the matrix B; and transferring the partial sum outputs to an adjacent row via a diagonal connection, with respect to the first direction, wherein a first processing element in a row is diagonally connected to a last processing element in an adjacent row.

[0042] Notably, the matrix A is first transposed and then secondly permutated.

[0043] Matrix transposing is a well-known process wherein each row of the matrix is converted to a column with the same index. For example,

[0044] Matrix A = a d g b e h c f i

[0045] Transposed Matrix A = a b c d e f g h i

[0046] 55613365-1 In any aspect of the disclosure, the permutation can be performed at the software level, after training, with zero penalty for inference or the permutation can be performed at the hardware level using a small controller, with minimal cost for training.

[0047] In any aspect of the disclosure, the systolic array may be square or non-square (e.g. rectangular).

[0048] In some cases, the systolic array may comprise NxN processing elements that can be utilised for a range of differently dimensioned square or non-square matrices. For example, non-square (e.g. rectangular) matrices may be accommodated by using known workload tiling techniques.

[0049] In some cases, non-square (e.g. rectangular) systolic arrays can be employed in accordance with the aspects described above, by padding a rectangular systolic array with a number of registers (e.g. FIFOs) whilst still maintaining a lower overhead than for conventional systolic arrays.

[0050] The disclosure relates generally to a spatial systolic-array architecture of processing elements (PEs) featuring diagonal-input (or diagonal-partial sum) and permutated weight-stationary dataflow for the acceleration of matrix-matrix multiplication (MMM). The architecture may comprise NxN PEs forming a square systolic array where the PEs mainly perform Multiply-Accumulation computations (using multiply-add components (MACs)) between inputs (from a first matrix, e.g. matrix B) and weights (from a second matrix, e.g. matrix A). The architecture uses a weight-stationary dataflow, where weights are captured by PEs to maximise the weight data utilisation. However, the architecture introduces a new diagonal input (or diagonal partial sum) dataflow with a permutated weight-stationary orientation. This arrangement eliminates the input and output synchronisation First-ln-first-Out (FIFO) buffers required by conventional weight-stationary systolic arrays.

[0051] The input (or partial sum) data propagates diagonally through the systolic array from one PE in a certain row to another PE in the next row. For example, if the systolic array is 3x3 and consists of three rows: a) Row 0: PE00, PE01 , PE02, b) Row 1 : PE10, PE11 , PE12, c) Row 2: PE20, PE21 , PE22. The input data is loaded in parallel to Row 0 including inputs for PE00, PE01 and PE02. After accomplishing the required MAC

[0052] 55613365-1 computations by Row 0, the input data is shifted diagonally from PEOO to PE12 and from PE01 to PE10 and from PE02 to PE11. Then after accomplishing the required MAC computations by Row 1 , the input data is shifted diagonally again from PE10 to PE22 and from PE11 to PE20 and from PE12 to PE21. Meanwhile, more new inputs are loaded into Row 0 and the processing continues until the end of the workload.

[0053] Notably, the diagonal movement of inputs (or partial sums) need not always flow in a downwardly and left configuration as per this example. In some cases, the diagonal movement may be downwardly and right, or upwardly and left or upwardly and right. Moreover, the orientation of the array is not limited and the diagonal movement may be, for example, right and downwardly, right and upwardly, left and downwardly or left and upwardly.

[0054] Advantageously, the proposed architecture significantly increases the utilisation of the compute resources (e.g. PEs). Thus, it has been demonstrated that the proposed architecture may outperform conventional weight-stationary systolic arrays by up to 50% in terms of throughput.

[0055] At least some of the above and other features of the invention are set out in the claims.

[0056] These and other aspects will be apparent from the embodiments described in the following. The scope of the present disclosure is not intended to be limited by this summary nor to implementations that necessarily solve any or all of the disadvantages noted.

[0057] Any features described in relation to one aspect of the disclosure may be applied to any one or more other aspect of the disclosure.

[0058] Brief Description of Certain Embodiments

[0059] Some embodiments of the disclosure will now be described by way of example only and with reference to the accompanying drawings, in which:

[0060] Figure 1(a) shows a block diagram for a known weight-stationary systolic array.

[0061] 55613365-1 Figure 1(b) shows a block diagram for a known weight-stationary systolic array, similar to that of Figure 1(a) but configured for a different orientation.

[0062] Figure 1(c) shows an illustration of a computational cycle 3 with resulting data flow for the multiplication of matrices when processed by the systolic array of Figure 1(b).

[0063] Figure 2(a) shows a block diagram for a proposed DiP systolic array architecture according to the disclosure.

[0064] Figure 2(b) shows a block diagram for an individual processing element (PE) used in the array of Figure 2(a).

[0065] Figure 3(a) shows a block diagram illustrating general matrix permutation.

[0066] Figure 3(b) shows pseudo-code for performing the matrix permutation of Figure 3(a).

[0067] Figure 4(a) shows a block diagram illustrating the diagonal input connections for a 3x3 systolic array according to the disclosure.

[0068] Figure 4(b) illustrates two matrices to be multiplied (matrix 1 and matrix 2) and the matrix permutation of matrix 2.

[0069] Figure 4(c) illustrates the computational cycles and resulting data flow for the multiplication of the matrices of Figure 4(b) when processed by the systolic array of Figure 4(a).

[0070] Figure 5 shows a comparison of latency calculated for a traditional WS systolic array and the proposed DiP systolic array for different dimensions NxN.

[0071] Figure 6 shows a comparison of throughput calculated for a traditional WS systolic array and the proposed DiP systolic array for different dimensions NxN.

[0072] Figure 7 shows a comparison of register count calculated for a traditional WS systolic array and the proposed DiP systolic array for different dimensions NxN.

[0073] 55613365-1 Figure 8 shows a flow-diagram for a computer-implemented method for matrix-matrix multiplication using diagonal input propagation.

[0074] Figure 9 shows a flow-diagram for a computer-implemented method for matrix-matrix multiplication using diagonal partial sum propagation.

[0075] Detailed Description of the Preferred Embodiments

[0076] Generally speaking, the disclosure proposes a novel Diagonal-Input Permutated (DiP) Weight-Stationary (WS) Systolic Array (SA) comprising a spatial systolic array architecture of processing elements (PEs) configured for diagonal-input movement and permutated weight-stationary dataflow for the acceleration of Matrix- Matrix Multiplication (MMM). Notably, the proposed architecture eliminates the input and output interfacing complexity of known weight-stationary (e.g. Google™ TPU-like) systolic-arrays.

[0077] In the following description, the computer architecture is generally described in the orientation shown in the Figures. However, it will be understood that the orientation of the elements is not limited to that shown and, in practice, for example, the computer architecture itself may be rotated or configured in a mirror image and / or data flowing through the computer architecture may be configured to flow in a different direction to that described.

[0078] Figure 1(a) illustrates a computer architecture 100 for a known weight-stationary systolic array 102 comprising NxN MAC processing elements (PE) labelled W(o,o> to W(N-I , N-1). Two groups of FIFO buffers are provided - Input FIFO buffers 104 and Output FIFO buffers 106 - required for input / output synchronisation.

[0079] As illustrated, the Input FIFO buffers 104 comprise no buffers in row 0; 1 buffer in row 1 ; 2 buffers in row 2; and N-1 buffers in row N-1. As illustrated, the Output FIFO buffers 106 comprise N-1 buffers in column 0; N-2 buffers in column 1 ; 1 buffer in column N-2; and no buffers in column N-1.

[0080] In operation, the weights are loaded (vertically downwardly as shown) into the array 102 and partial sums (psums) computed by each PE are moved through the array 102

[0081] 55613365-1 using vertical connections. The inputs labelled l(0) to I<N-I) are shifted horizontally (to the right as shown) through the array 102 via the input FIFO buffers 104, while the outputs labelled 0<o) to 0<N-I) are shifted vertically downwardly from the array 102, through the output FIFO buffers 106.

[0082] Such weight-stationary (WS) dataflow is widely used, where weights are initially loaded to the PEs and then remain stationary in the PEs while inputs and partial sums are moved sequentially through the PEs during processing, in a systolic fashion. This architecture addresses the Von Neumann bottleneck by reducing the number of memory accesses (e.g. to re-fetch weights), while increasing the reusability of the same piece of data. However, this WS dataflow is required to use input and output FIFOs 104, 106 to synchronize the dataflow for correct functionality.

[0083] Figure 1(b) shows a block diagram for a known weight-stationary systolic array 200, similar to that of Figure 1(a) but configured for a different orientation. In this case, the inputs are loaded vertically downwardly, the partial sums propagate horizontally to the right, the weights are loaded horizontally to the right and the outputs are provided horizontally to the right, as shown. Although not illustrated, the array 200 also includes input and output FIFOs similar to those of Figure 1(a) to synchronize the dataflow for correct functionality.

[0084] Figure 1(c) shows an illustration of a computational cycle 3 with resulting data flow for the multiplication of matrices when processed by the systolic array 200. This illustrates that an output Out11 is first generated in row 1, with no output from rows 2 or 3 during the same clock cycle. As illustrated in Table 1 below, it will be understood that in cycle 4, output Out21 will be generated in row 1 , output Out12 will be generated in row 2 and no output will be generated from row 3. It is only in cycle 5 that outputs are generated from each of rows 1 to 3. Thus, the output FIFO buffers of Figure 1(a) are required in order to synchronise the outputs from the different rows.

[0085] 55613365-1

[0086] Table 1 - Outputs from Computational Cycles for a Traditional Weight-Stationary Dataflow

[0087] Similarly, the inputs are loaded in a staggered fashion beginning with the upper left PE and then with the upper left and adjacent column PE and then with inputs in the PEs in all three columns. Thus, the input FIFO buffers of Figure 1(a) are required in order to synchronise the inputs into the columns in the array.

[0088] The use of FIFOs increases not only area, power and energy consumption, but also the latency for matrix-matrix multiplication. Consequently, FIFO-based systolic arrays suffer from larger area footprint requirements with more expensive clock trees, lower energy efficiency and lower PE utilization which leads to higher latencies and lower throughput.

[0089] The Applicant has undertaken some analytical modelling of WS systolic arrays to study latency, throughput, and FIFO overhead, offering insights into the dataflow performance. This has been considered for both a 1 -stage pipelined PE and a 2-stage pipelined PE.

[0090] For the 1 -stage pipelined PE, for the latency analytical model, the WS systolic array 100 consumes 2N - 1 cycles to reach full hardware utilization, and N-l cycles to complete processing, resulting in 3N -2 cycles, as shown in Equation (1). The throughput is approximated by a number of operations per cycle, as per Equation (2).

[0091] Latency for WSi-stage = 3N -2 (1 )

[0092] Throughput for WSi.stage = 2N3 / (3 / V -2) (2)

[0093] 55613365-1 For the registers overhead, the WS systolic array 100 uses two FIFO groups 104, 106 for input and output synchronization. As shown in Figure 1(a), each FIFO group consists of N N - 1J / 2 registers. Consequently, the total register overhead for a typical WS systolic array is calculated as shown in Equation (3), where N is the number of rows / columns per WS systolic array. Note, this equation holds regardless of whether 1- stage or 2-stage pipelined PEs are employed.

[0094] Registers overhead for WS = N N -1) (3)

[0095] However, for the 2-stage pipelined PE, for the latency analytical model, the WS systolic array consumes 2N - 1 cycles to reach full hardware utilization, and N cycles to complete processing, resulting in a total of 3 / V-l cycles, as shown in Equation (4). As above, the throughput is approximated by a number of operations per cycle, as per Equation (5).

[0096] Latency for WSz-stage = 3N -1 (4)

[0097] Throughput for WSz-stage = 2N3 / (3 / V -1) (5)

[0098] A new dataflow is proposed in this disclosure which uses diagonal input (or partial sum) propagation along with permutated weight-stationary loading. The new dataflow represents an improvement to the conventional weight-stationary dataflow, as the weights still are captured and held stationary by the PEs during processing. However, the proposed architecture makes the input data (e.g. matrix 1 data) propagate diagonally through the systolic array from one PE in a certain row to another PE in the next row, as illustrated in the proposed generic DiP systolic array architecture 300 of Figure 2(a).

[0099] The architecture 300 comprises a DiP systolic array 302 comprising NxN MAC processing elements (PE) labelled PE(o,o> to PE(N-I , N-I>. In this case, no input or output First-In-First-Out (FIFO) buffers are required for input / output synchronisation, as will be explained below.

[0100] In operation, the weights are loaded (vertically downwardly as shown by crossed arrow lines) into the array 302 via buses and partial sums (psums) computed by each PE are moved through the array 302 using the same or similar vertical connections or buses

[0101] 55613365-1 (shown by bold lines). The inputs labelled l(0) to I<N-I) are also initially input vertically downwardly into the array 302 via buses and are then propagated through the array 302 via diagonal (generally down and left) connections or buses illustrated by thin arrow lines. Notably, the first PE in each row, is diagonally connected to the last PE in the row below to transfer the inputs during processing. The outputs labelled pe_output(0) to pe_output(N-1) are shifted vertically downwardly from the array 302, without the need for output FIFO buffers.

[0102] Figure 2(b) shows a block diagram for an individual processing element (PE) used in the array of Figure 2(a). The PE consists of a MAC unit, comprising a multiplier 400 and an accumulator or adder 402; four registers 404 to store, respectively, a weight (w), an input (I), a product from the multiplier 400 and an output (pe_output) from the MAC unit. The weight is passed from the weight register 404 to the multiplier 400 for multiplying with the input (I) passed from the input register 404 to the multiplier 400. The product from the multiplier 400 is passed from the product register 404 to the accumulator 402. In addition, when available, a partial sum (psum) (pe_output) from a PE above the present PE is passed to the accumulator 402, where it is added to the product and stored in the output (pe_output) register 404. The output (pe_output) from its respective register 404 is then passed to the PE below as the partial sum (psum) input or it is output as a final column output (e.g. Row N-1). In addition, the input (I) from its respective register 404 is transferred diagonally to the row below (when the PE is not in the bottom row), for reuse in the next computational cycle. Also, when initial loading the weights into the PEs, the weight in its respective register 404 may be transferred to the PE in the row below (when available).

[0103] It will be understood that a controller may be provided to control the operation of each PE to ensure that they function as intended.

[0104] Figure 3(a) shows a block diagram illustrating an example of matrix permutation by column index. In this case, a weight matrix 500 comprises weights Wo.o to WN-I ,N-I in columns labelled 0 to N-1 and row labelled 0 to N-1. The permutation shown involves shifting and rotating each column in an upwardly direction, by its column index.

[0105] As shown in the permutated weight matrix 502, because the first column index is 0 there is no change to column 0. For column 1, the weights are all shifted upwardly by 1 and the top weight Wo.i, which is ejected from the top of column 1, is rotated and

[0106] 55613365-1 inserted at the bottom of column 1. Similarly, for column 2, the weights are all shifted upwardly by 2 and the top weights Wo, 2 and Wi,2, which are ejected from the top of column 2, are rotated and inserted at the bottom of column 2 (with the first ejected weight inserted first). This process is repeated for the remaining columns to form the permutated weight matrix 502.

[0107] Figure 3(b) shows pseudo-code for performing the matrix permutation of Figure 3(a).

[0108] The permutation of the weight matrix (or any one of the two matrices to be multiplied by the proposed architecture) eliminates the need for the input and output synchronization First-In-First-Out (FIFO) buffers required by conventional weight-stationary systolic arrays such as those in Figures 1(a)-(c). Moreover, a consequence of the permutation is increased PE utilization and throughput because entire rows of calculations can be performed at the same time, without the need to wait for each input to reach the required weight for it to be multiplied with, further down the columns.

[0109] Figure 4(a) shows a block diagram illustrating the connections (i.e. buses) for a 3x3 systolic array 600 in accordance with the generic architecture shown in Figure 2(a). The array 600 consists of three columns and three rows: a) Row 0: PE00, PE01, PEm, b) Row 1 : PEW, PEu, PE12, C) ROW 2: PE20, PE21, PE22. The array 600 is diagonally connected for reuse of inputs, as illustrated, and with the first PE in each row being connected to the last PE in the row below.

[0110] Figure 4(b) illustrates two matrices to be multiplied (matrix 1 and matrix 2) and the matrix permutation of matrix 2 into permutated matrix 2, using the same shift and rotate process described above in relation to Figure 3(a). Notably, in some cases, the permutation may be performed differently (e.g. by shifting and rotating the columns downwardly for use with a dataflow comprising upward loading of inputs).

[0111] In the present case, the permutation is performed by software. Alternatively, a hardware controller may be configured to perform the permutation. The permutated matrix 2 is then ready for loading into the systolic array 600.

[0112] 55613365-1 Figure 4(c) illustrates the computational cycles and resulting data flow for the multiplication of the matrices of Figure 4(b) when processed by the systolic array of Figure 4(a).

[0113] Cycles -2, -1 and 0 are required for sequential row-by-row loading of the permutated matrix 2 from the bottom row to the top row. For increased efficiency, Cycle 0 is also used for loading of the first row of matrix 1. Thus, the last weight row is loaded at the same time as the first row of the input matrix. The input data is loaded in parallel to all PEs in Row 0 including inputs for PFoo, PE01, and PEm.

[0114] Cycle 1 to Cycle 5 are for the matrix-matrix multiplication processing through the systolic array 600. After accomplishing the required computations by Row 0, the input data is shifted diagonally from PEmto PE12and from PE01to PEWand from PE02to PEu. Then after accomplishing the required computations by Row 1 , the input data is shifted diagonally again from PFio to PF22and from PFn to PF2o and from PE12to PE21. The first output row is generated at Cycle 3, and the last output row is generated at Cycle 5.

[0115] The complete process flow is as follows:

[0116] • Cycle -2: The last row of the permutated weight matrix (c, d, h) is loaded to the first PE row.

[0117] • Cycle -1 : The last row of the permutated weight matrix (c, d, h) is shifted to the second PE row, and the second row of the permutated weight matrix (b, f, g) is loaded to the first PE row.

[0118] • Cycle 0: The last row of the permutated weight matrix (c, d, h) is shifted to the last PE row, the second row of the permutated weight matrix (b, f, g) is shifted to the second PE row, and the first row of the permutated weight matrix (a, e, I) is loaded to the first PE row. To save one cycle, the first input matrix row (1, 2, 3) is loaded to the first PE row. In other cases, the first input matrix row (1, 2, 3) may be loaded to the first PE row in a separate cycle after loading of the permutated weight matrix is completed.

[0119] • Cycle 1 : The first PE row shifts the partial sums (1a, 2e, 3i) vertically downwardly to the second row; the second input matrix row (4, 5, 6) is loaded to the first PE row; and the first input matrix row (1, 2, 3) is transferred diagonally downwardly

[0120] 55613365-1 and left (e.g. permutated) to (2, 3, 1) and loaded to the second PE row at the same cycle.

[0121] Cycle 2: The first PE row shifts the partial sums (4a, 5e, 6i) vertically downwardly to the second row; the third input matrix row (7, 8, 9) is loaded to the first PE row; and the second input matrix row (4, 5, 6) is transferred diagonally downwardly and left (e.g. permutated) to (5, 6, 4) and loaded to the second PE row. Similarly, the second PE row shifts the partial sums (1a+2b, 2e+3f, 3i+1g) to the third row; and the first input matrix row (2, 3, 1) is transferred diagonally downwardly and left (e.g. permutated) to (3, 1, 2) and loaded to the third PE row.

[0122] Cycle 3: The first PE row shifts the partial sums (7a, 8e, 9i) vertically downwardly to the second row, and the third input matrix row (7, 8, 9) is transferred diagonally downwardly and left (e.g. permutated) to (8, 9, 7) and loaded to the second PE row. Similarly, the second PE row shifts the partial sums (4a+5b, 5e+6f, 6i+4g) to the third row; and the second input matrix row (5, 6, 4) is transferred diagonally downwardly and left (e.g. permutated) to (6, 4, 5) and loaded to the third PE row. In addition, the third PE row shifts the first output row (1a+2b+3c, 2e+3f+1d, 3i+1g+2h) downwardly, for example into output registers or to be stored in the memory (not shown).

[0123] Cycle 4: The first PE row becomes idle if there is no more workload, and the second PE row shifts the partial sums (7a+8b, 8e+9f, 9i+7g) vertically downwardly to the third row, and the third input matrix row (8, 9, 7) is transferred diagonally downwardly and left (e.g. permutated) to (9, 7, 8) and loaded to the third PE row. In addition, the third PE row shifts the second output row (4a+5b+6c, 5e+6f+4d, 6i+4g+5h) downwardly, for example into output registers or to be stored in the memory (not shown).

[0124] Cycle 5: The first and second PE rows become idle if there is no more workload, and the third PE row shifts the third output row (7a+8b+9c, 8e+9f+7d, 9i+7g+8h) downwardly, for example into output registers or to be stored in the memory (not shown).

[0125] When the inputs are no longer required for processing (i.e. when all calculations requiring the current inputs are complete), they will be overwritten by new inputs, when available.

[0126] 55613365-1 In some cases, further rows of inputs may be loaded into row 1 if the size of the input data matrix is larger than 3x3 and the processing continues in line with the above, until the end of the workload. This approach significantly reduces the latency and the number of registers required for each systolic array 600 by eliminating the input and output synchronisation FIFOs. Thus, the proposed architecture addresses the problem of the FIFO components adding a considerable energy, power and area overhead due to their complex clock trees and high switching activity.

[0127] The analytical models discussed above for latency and throughput of a traditional systolic array have been adapted for the proposed DiP systolic array described above. For the latency analytical model, the DiP systolic array consumes 2N-1 cycles for processing, and zero cycles for synchronization, resulting in only 2N-1 cycles for 1- stage MAC pipelined PEs, as per Equation (6). The throughput of DiP is approximated by a number of operations per cycle, as per Equation (7) for the 1 -stage MAC pipelined PEs.

[0128] Latency for DiPi.stage = 2N -1 (6)

[0129] Throughput for DiPi-stage = 2N3 / (2?V -1) (7)

[0130] Where N is the number of rows / columns per DiP systolic array.

[0131] For the latency analytical model for a 2-stage pipelined PE, the DiP systolic array consumes 2N cycles for processing, and zero cycles for synchronization, resulting in only 2N cycles, as per Equation (8). The throughput of DiP is approximated by a number of operations per cycle, as per Equation (9) for the 2-stage MAC pipelined PEs.

[0132] Latency for DiP2-stage= 2N (8)

[0133] Throughput for DiPg-sta e = 2 N3 / 2 / V = N2(9)

[0134] The DiP systolic array eliminates the FIFO overhead by enabling a whole input row to be loaded and processed in parallel, without using any input synchronization FIFOs. Correspondingly, the output is generated row by row without any need of output synchronization FIFOs. Consequently, no FIFOs are required for input / output synchronization regardless of whether 1-stage or 2-stage pipelined PEs are employed.

[0135] 55613365-1 Each PE consists of one multiplier, one adder, and four registers, as shown in Figure 2(b). The partial sums and weights may be transferred through the array using separate buses or they may be configured to use the same bus since they are not required to be transferred at the same time. In which case, a multiplexer may be used to select between the adder output register or the weight register based on a workflow control signal provided by a controller (not shown).

[0136] The PE shown in Figure 2(b) is a 2-stage pipelined PE, where the multiplication output is registered before the addition is performed. In some cases, a 1-stage pipelined PE may be used. However, it may be advantageous to use more stages in order to reduce delay, operate at higher frequencies and thereby increase speed.

[0137] Each of the weight and input connections may be configured as 16-bit buses, while the multiplier output and the adder output may be transferred using a 32-bit bus. In other cases, different bus sizes may be selected to suit the size of the data being transferred.

[0138] The scalability of the proposed systolic array is important to meet the acceleration requirements for current and future Al applications. The proposed systolic array may be gradually scaled up, for example, from 3x3 to 64x64 and more with sizes (3x3, 4x4, 8x8, 16x16, 32x32, 64x64). In some case, the proposed systolic array may be scaled to handle the whole Al model workloads. An Al accelerator may be built including multiple, e.g. 64x64 PE slices based on the proposed systolic array, global buffers, micro-controllers and interfacing logic.

[0139] Figure 5 shows the latency calculated for the conventional WS systolic array and the proposed DiP systolic array based on the above array sizes, and using 2-stage pipelined PEs. In all cases, the latency of the WS systolic array is greater than the latency of the proposed DiP systolic array. The percentage of latency improvement is calculated as the difference between the WS and DiP latency, divided by the WS latency. The percentage of latency improvement starts at 25% for a 3x3 DiP systolic array and reaches 33% for the 64x64 DiP systolic array.

[0140] In addition, the throughput for both the conventional WS systolic array and the proposed DiP systolic array is compared, as shown in Figure 6, for 2-stage pipelined

[0141] 55613365-1 PEs. The proposed architecture for the DiP systolic array significantly increases the compute resources (PE) utilization. Thus, the DiP systolic array outperforms the conventional WS weight-stationary systolic array in terms of throughput by up to 50%.

[0142] Figure 7 shows the number of registers required and the percentage of registers saving by the proposed DiP systolic array compared to the conventional WS systolic array, using 2-stage pipelined PEs. The proposed design eliminates input / output synchronisation FIFOs leading to considerable area, power and energy savings. The registers saving reaches up to 20% for 64x64 DiP systolic array. The register saving percentage is calculated as the difference between the WS and DiP register count, divided by the WS register count. The registers of WS systolic array are distributed between input synchronisation FIFOs, output synchronization FIFOs, and internal PE registers. In comparison, the proposed DiP systolic array uses only internal registers for each PE as it eliminates the need for any input / output synchronisation FIFOs. The register count is normalised by 8-bit. In this example, the input registers are represented in 8-bits and the output registers are represented in 16bits. The PE may also comprise enable registers for controlling the operation of the PE. Such enable registers are optional and are therefore not included in the analysis relating to Figure 7. However, they can be useful for power saving and may be shared among different PEs.

[0143] In summary, the disclosure provides a computer-implemented method 800 for matrixmatrix multiplication as illustrated in Figure 8. The method 800 comprises a step 802 of generating a permutated matrix A based on a matrix A having columns with indexes 0 to N-1 by shifting and rotating each column in a rotation direction, by the column index. The method 800 comprises a step 804 of loading, as stationary values, the permutated matrix A into a systolic array of processing elements and a step 806 of loading, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements. The method 800 comprises a step 808 of multiplying the stationary value from the permutated matrix A with the value from the matrix B to produce a product; and a step 810 of accumulating the product with a partial sum input received in the first direction from another processing element, when available. The method 800 comprises a step 812 of outputting, in the first direction, a partial sum output resulting from the accumulation; and a step 814 of transferring the values from the matrix B to an adjacent row via a diagonal connection with respect to the first direction, wherein a

[0144] 55613365-1 first processing element in a row is diagonally connected to a last processing element in the adjacent row.

[0145] Thus, a diagonal-input and permutated (DiP) weight-stationary systolic array is proposed to accelerate matrix-matrix multiplication. As an example, a spatial square systolic array architecture of NxN processing elements (PEs) has been described, where the PEs comprise MACs to perform multiply-accumulate operations. The proposed systolic array adopts a new dataflow named diagonal-input and permutated weight (DiP) dataflow, which eliminates the input and output synchronization First-In- First-Out (FIFO) buffers required by conventional weight-stationary systolic arrays. The proposed architecture significantly increases the compute resources (PE) utilization. Latency, throughput, and FIFO overhead analytical models for the conventional weight stationery and the proposed DiP systolic arrays have been considered. Design space exploration was shown for scaling the proposed architecture for different dimensions (e.g. from 3x3 to 4x4, 16x16, 32x32 and 64x64). The proposed architecture outperformed the conventional weight-stationary counterpart in terms of throughput by up to 50% in addition to achieving area, power and energy savings by eliminating the input / output synchronisation FIFOs.

[0146] Although only the diagonal input case has been described above in detail, it will be understood that the same principals and outcomes can be achieved for the proposed diagonal partial sum propagation. This case is described in relation to Figure 9 which shows a computer-implemented method 900 for matrix-matrix multiplication comprising a step 902 of creating a permutated transposed matrix A based on a transposed matrix A having columns with indexes 0 to N-1 by shifting and rotating each column in a rotation direction, by the column index. The method 900 comprises a step 904 of loading, as stationary values, the permutated transposed matrix A into a systolic array of processing elements; and a step 906 of loading, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements. The method 900 comprises a step 908 of multiplying the stationary value from the permutated transposed matrix A with the value from the matrix B to produce a product; and a step 910 of accumulating the product with a partial sum input received from another processing element, when available, to produce a partial sum output. The method 900 comprises a step 912 of outputting, in the first direction, the values from the matrix B; and a step 914 of transferring the partial sum outputs to an adjacent row via a diagonal

[0147] 55613365-1 connection, with respect to the first direction, wherein a first processing element in a row is diagonally connected to a last processing element in an adjacent row.

[0148] Thus, the method 900 described above achieves the same result as that of the method 800 in increasing PE utilization and eliminating the need for any input or output synchronisation FIFOs. As such, the latency and throughput are similarly improved by the method 900.

[0149] In the method 900, the first processing element may be located in a row i and a column j, wherein i and j are non-negative integers, and the first processing element may be configured to transfer the partial sum output to a second processing element located in a row i+1 or i-1 and a column j-1 or j+1.

[0150] A controller may be configured to generate the transposed matrix A and to generate the permutated transposed matrix A based on the transposed matrix A.

[0151] Embodiments of the present disclosure can be employed in many different applications including, for example, image recognition and natural language processing.

[0152] The memory referred to herein can be based on conventional CMOS SRAMs, Flash memories, dynamic random access memories (DRAMs), resistive-based memories (like memristor-based memory arrays) or any other type of memories, including those using emerging technologies (e.g. RRAM, spin transfer torque (STT), spin orbit torque (SOT), spin orbit torque field effect transistor (SOTFET), ferroelectric field effect transistor (FeFET), .. etc.).

[0153] The skilled person will understand that in the preceding description and appended claims, positional terms such as ‘above’, ‘along’, ‘side’, etc. are made with reference to conceptual illustrations, such as those shown in the appended drawings. These terms are used for ease of reference but are not intended to be of limiting nature. These terms are therefore to be understood as referring to an object when in an orientation as shown in the accompanying drawings.

[0154] Although the disclosure has been described in terms of preferred embodiments as set forth above, it should be understood that these embodiments are illustrative only and

[0155] 55613365-1 that the claims are not limited to those embodiments. Those skilled in the art will be able to make modifications and alternatives in view of the disclosure which are contemplated as falling within the scope of the appended claims. Each feature disclosed or illustrated in the present specification may be incorporated in any embodiments, whether alone or in any appropriate combination with any other feature disclosed or illustrated herein.

[0156] References

[0157] [1] B. Mondal, ’’Artificial Intelligence: State of the Art,” Recent Trends and Advances in Artificial Intelligence and Internet of Things, vol. 172, pp. 389-425, 2020.

[0158] [2] G. G. Chowdhury, ’’Natural language processing,” Annu. Rev. Inf Sci. Technol., vol. 37, no. 1, pp. 51-89, 2003.

[0159] [3] D. Khurana et al., ’’Natural language processing: State of the art current trends and challenges,” Multimedia Tools and Applications, Vol. 82, pp 3713-3744, 2023.

[0160] [4] T. Lin et al., ”A survey of transformers,” Al Open, vol. 3, pp. 111-132, 2022.

[0161] [5] L. Floridi and M. Chiriatti, ”GPT-3: Its nature scope limits and consequences,” Minds Mach., vol. 30, pp. 681-694, 2020.

[0162] [6] H.-T. Kung, ’’Why systolic architectures?,” IEEE Comput., vol. 15, no. 1 , pp. 37-46, Jan. 1982.

[0163] [7] Yu Hen Hu and S. Kung, ’’Systolic arrays,” Handbook of Signal Processing Systems,, pp. 939-977, 2018.

[0164] [8] Z. Yang et al., ’’Systolic array based accelerator and algorithm mapping for deep learning algorithms,” Proc. Netw. Parallel Comput, pp. 153-158, 2018.

[0165] [9] M. Soltaniyeh, R. P. Martin and S. Nagarakatte, ”An Accelerator for Sparse Convolutional Neural Networks Leveraging Systolic General Matrix-Matrix Multiplication,” ACM Trans. Archit. Code Optim., Vol. 19, No. 3, Article 42, 2022.

[0166]

[0010] Bo Wang, et al., ”A novel systolic array processor with dynamic dataflows,” Integration, Vol. 85, Pages 42-47, 2022. in] H. Waris, et al., ”AxSA: On the design of high-performance and powerefficient approximate systolic arrays for matrix multiplication,” J. Signal Process. Syst., vol. 93, no. 6, pp. 605-615, Jun. 2021.

[0167]

[0012] G. Shomron, T. Horowitz and U. Weiser, et al., ’’SMT-SA: Simultaneous multithreading in systolic arrays,” IEEE Comput. Archit. Lett., vol. 18, no. 2, pp. 99- 102, Jul. 2019.

[0168] 55613365-1

[0013] M. A. Hanif et al., ”MPNA: A massively-parallel neural array accelerator with dataflow optimization for convolutional neural networks,” arXiv:1810. 12910, 2018.

[0169]

[0014] C. Peltekis, et al., ’’ArrayFlex: A Systolic Array Architecture with Configurable Transparent Pipelining,” Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1-6, 2023.

[0170] [is] N. P. Jouppi et al., ” In-datacenter performance analysis of a tensor processing unit,” Proc. 44th Annu. Int. Symp. Comput. Architecture, pp. 1-12, Jun. 2017.

[0171]

[0016] B. Asgari et al., ’’Meissa: Multiplying matrices efficiently in a scalable systolic architecture,” 2020 IEEE 38th International Conference on Computer Design (ICCD), pp. 130-137, 2020.

[0172]

[0017] Xu, et al., ”A Survey of Design and Optimization for Systolic ArrayBased DNN Accelerators,” ACM Comput. Surv., Vol. 56, no. 1, pp. 1-37, 2023.

[0173] [is] E. Yago et al., ’’Impact of the Array Shape and Memory Bandwidth on the Execution Time of CNN Systolic Arrays,” 23rd Euromicro Conference on Digital System Design (DSD),, pp. 510-517, 2020.

[0174] 55613365-1

Claims

CLAIMS:

1. A computer architecture for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprising: a systolic array of processing elements, the processing elements being configured to: load stationary values from a permutated matrix A, wherein the permutated matrix A is based on a matrix A having columns with indexes 0 to N- 1 and wherein the permutated matrix A is formed by shifting and rotating each column in a rotation direction, by the column index; load, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements; multiply the stationary value from the permutated matrix A with the value from the matrix B to produce a product; accumulate the product with a partial sum input received in the first direction from another processing element, when available; and output, in the first direction, a partial sum output resulting from the accumulation; wherein the processing elements are configured to transfer the values from the matrix B to an adjacent row via a diagonal connection with respect to the first direction, and wherein a first processing element in a row is diagonally connected to a last processing element in the adjacent row.

2. The computer architecture of claim 1 wherein the shifting and rotating of each column by the column index is performed in an upwards rotation direction by shifting each column upwardly by the column index and moving the values from the permutated matrix A that are shifted out of a top row of each column to a bottom row of each column.55613365-13. The computer architecture of claim 1 wherein the shifting and rotating of each column by the column index is performed in a downwards rotation direction by shifting each column downwardly by the column index and moving the values from the permutated matrix A that are shifted out of a bottom row of each column to a top row of each column.

4. The computer architecture of any preceding claim wherein the adjacent row is disposed in the first direction, with respect to the row of the first processing element.

5. The computer architecture of any preceding claim wherein the first processing element is located in a row i and a column j, wherein i and j are non-negative integers, and the first processing element is configured to transfer the values from the matrix B to a second processing element located in a row i+1 or i-1 and a column j-1 or j+1.

6. The computer architecture of any preceding claim wherein the processing elements are configured to load the first row of values from the matrix B at a same time as loading a last row of stationary values from the permutated matrix A.

7. The computer architecture of any preceding claim further comprising a controller configured to generate the permutated matrix A based on the matrix A.

8. A computer-implemented method for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprising: generating a permutated matrix A based on a matrix A having columns with indexes 0 to N-1 by shifting and rotating each column in a rotation direction, by the column index; loading, as stationary values, the permutated matrix A into a systolic array of processing elements; loading, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements; multiplying the stationary value from the permutated matrix A with the value from the matrix B to produce a product;55613365-1accumulating the product with a partial sum input received in the first direction from another processing element, when available; outputting, in the first direction, a partial sum output resulting from the accumulation; and transferring the values from the matrix B to an adjacent row via a diagonal connection with respect to the first direction, wherein a first processing element in a row is diagonally connected to a last processing element in the adjacent row.

9. A computer architecture for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprising: a systolic array of processing elements, the processing elements being configured to: load stationary values from a permutated transposed matrix A, wherein the permutated transposed matrix A is based on a transposed matrix A having columns with indexes 0 to N-1 and wherein the permutated transposed matrix A is formed by shifting and rotating each column in a rotation direction, by the column index; load, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements; multiply the stationary value from the permutated transposed matrix A with the value from the matrix B to produce a product; accumulate the product with a partial sum input received from another processing element, when available, to produce a partial sum output; and output, in the first direction, the values from the matrix B; wherein the processing elements are configured to transfer the partial sum output to an adjacent row via a diagonal connection, with respect to the first direction, and wherein a first processing element in a row is diagonally connected to a last processing element in an adjacent row.55613365-110. The computer architecture of claim 9 wherein the first processing element is located in a row i and a column j, wherein i and j are non-negative integers, and the first processing element is configured to transfer the partial sum output to a second processing element located in a row i+1 or i-1 and a column j-1 orj+1.

11. The computer architecture of any of claims 9 to 10 further comprising a controller configured to generate the permutated transposed matrix A based on the transposed matrix A.

12. The computer architecture of any of claims 9 to 11 wherein the controller is further configured to generate the transposed matrix A.

13. A computer-implemented method for matrix-matrix multiplication, where an output C equals matrix B multiplied by matrix A, comprising: creating a permutated transposed matrix A based on a transposed matrix A having columns with indexes 0 to N-1 by shifting and rotating each column in a rotation direction, by the column index; loading, as stationary values, the permutated transposed matrix A into a systolic array of processing elements; loading, in a first direction, each row of values, in turn, from a matrix B, into a first row of processing elements; multiplying the stationary value from the permutated transposed matrix A with the value from the matrix B to produce a product; accumulating the product with a partial sum input received from another processing element, when available, to produce a partial sum output; outputting, in the first direction, the values from the matrix B; and transferring the partial sum outputs to an adjacent row via a diagonal connection, with respect to the first direction, wherein a first processing element in a row is diagonally connected to a last processing element in an adjacent row.55613365-1

Images