Method for accelerating a general matrix multiplication on a computer system

Abstract

The present invention relates to a computer-implemented method, computer program, non-transitory, computer-readable medium comprising a program code and computer system for accelerating a General Matrix Multiplication (GEMM) on a computer system, and to methods, systems and computer programs / computer-readable media using such a computer-implemented method. A computer-implemented method for accelerating a General Matrix Multiplication, GEMM, on a computer system, comprises obtaining (310) a data structure comprising a representation of a first set of matrix layout variants known to be more efficient than a second set of matrix layout variants when being used to perform, by the computer system, a GEMM, with each matrix layout variant defining whether at least one of the matrices involved in the GEMM are to be transposed, and selecting (350) a matrix layout variant of the first set of matrix layout variants based on matrices involved in a GEMM to be performed, and performing (370) the GEMM on the matrices using the selected matrix layout variant.

Description

[0001] METHOD FOR ACCELERATING A GENERAL MATRIX MULTIPLICATION ON A COMPUTER SYSTEM

[0002] The project leading to this application has received funding from the European Union’s Horizon Europe research and innovation program under grant agreement No. 101096466.

[0003] The present invention relates to a computer-implemented method, computer program, non-transitory, computer-readable medium comprising a program code and computer system for accelerating a General Matrix Multiplication (GEMM) on a computer system.

[0004] The present invention relates to computer-implemented methods, computer programs, non-transitory, computer-readable media comprising program codes and systems using the computer-implemented method, for accelerating a General Matrix Multiplication on a computer system, e.g., as part of machine learning inference or training, or as part of a scientific calculation.

[0005] General Matrix Multiplication (GEMM, e.g., as shown in Figs. 1 a and 1 b), is used in nearly all Al (Artificial Intelligence), scientific and engineering applications today. Accordingly, there are several publications that relate to optimizing the performance of GEMM, such as Springer et al.: “A Design of a High-Performance GEMM-like Tensor-Tensor Multiplication” and Matthias Reumann: “Transpose-Free Contraction of Complex Tensors”.

[0006] As GEMM is mission critical, hardware vendors provide optimized implementations for them in their Basic Linear Algebra Subprograms (BLAS) libraries. Examples for these are Intel MKL (Math Kernel Library), OpenBLAS, AMD rocBLAS, NVIDIA CUBLAS or NEC NLC. The GEMM operation allow the user to perform the following operation: With op(X, Tx) indicating whether to transpose / not transpose the input matrix, a and p can be used to scale the multiplication and / or to add it to the existing values stored in C. The API (Application Programming Interface) call is usually defined as (order of arguments might be different):

[0007] Xgemm(Ta, Tb, m, n, k, a, A, lda, B, ldb, ft, C, ldcy,

[0008] X determines the datatype, usually h=half, s=float, d=double m, n, k: are the matrix dimensions lda, ldb, ldcare the data strides of the matrices

[0009] GEMM are used throughout computation-heavy applications, such as machine learning and scientific computations / calculations, and often make up a large part of the computations being performed.

[0010] It is therefore an object of the present invention to improve and further develop a method, computer program, non-transitory, computer-readable medium comprising a program code and system for accelerating a General Matrix Multiplication (GEMM) on a computer system.

[0011] In accordance with the invention, the aforementioned object is accomplished by a computer-implemented method comprising the features of claim 1. The method comprises obtaining a data structure comprising a representation of a first set of matrix layout variants known to be more efficient than a second set of matrix layout variants when being used to perform, by the computer system, a GEMM. Each matrix layout variant defines whether at least one of the matrices involved in the GEMM are to be transposed. The method comprises selecting a matrix layout variant of the first set of matrix layout variants based on matrices involved in a GEMM to be performed. The method comprises performing the GEMM on the matrices using the selected matrix layout variant.

[0012] According to the proposed concept, it has been recognized that the most performance critical part of GEMM is to find the right way of loading / storing the input and output matrices, as the operation is mainly memory bandwidth bound. It has also been recognized that there can be a major difference in performance depending on how the data stored in a matrix is being accessed, as the memory chips being used to read and write the matrices do not read and write single data values, but entire cache lines. If only a small portion of a cache line is being used when reading or writing a matrix, the memory transfers lot of data to the processor with low utilization. In the optimal case, on the other hand, the data layout allows to utilize the entire cache line, resulting in peak memory bandwidth utilization. For each computer system, there are some matrix layout variants, i.e. , orientations in which matrices are read and / or written, that yield an improved performance, as the operations to operate on the memory more closely align with the cache lines of the. Some of these matrix layout variants can be used, for a given GEMM, by transposing one or both of the input matrices and / or the output matrix. While transposing the matrix yields a computational overhead, this overhead can be outweighed by the performance benefits of using a matrix layout variant with an improved performance. In the proposed concept, a set of matrix layout variants known to be more efficient than other matrix layout variants is used as a starting point. Based on the matrices involved in the GEMM, and the overhead caused by potential transpositions, one of the matrix layout variants known to be more efficient is selected (or not, if the cost of transposition is too high) based on the matrices at hand, and used to perform the GEMM. The selection of a most efficient matrix layout variant among the matrix layout variants known to be more efficient and use of the selected matrix layout variant can, in many cases, accelerate the GEMM, leading to an improved performance of the computer system with respect to performing the GEMM, e.g., as part of a machine learning inference or training operation, or as part of a scientific computation / calculation.

[0013] To evaluate, which of the matrix layout variants of the first set of matrix layout variants is likely to yield the best performance, the cost of performing the necessary transposition(s) can be evaluated. This cost is based on the respective sizes (i.e., dimensions, and thus numbers of items of the matrix) of the respective matrices. Thus, the method may comprise determining respective sizes of a first input matrix, a second input matrix and of an output matrix of the matrices involved in the GEMM. The method may comprise calculating or estimating, for the matrix layout variants of the first set of matrix layout variants, a number of elements to be transposed when performing GEMM on the matrices based on the sizes of the respective matrices. The method may comprise selecting the matrix layout variant of the first set of matrix layout variants based on the number of elements to be transposed. In particular, as the matrix layout variants of the first set of matrix layout variants tend to yield a similar performance (excluding the necessary transpositions), the matrix layout variant can be selected based on the determined (i.e., exactly calculated) or estimated (e.g., based on orders of magnitude) number of items to be transposed. If the overall number of items to be transposed is too high (for each of the matrix layout variants of the first set), e.g., higher than a threshold number, the GEMM may be performed without modifying the layout of the respective matrices, i.e., without introducing an additional transposition.

[0014] In some cases, transposition of the respective matrices may have a downstream effect that can be considered when selecting the matrix layout variant. In particular, during the training of a machine learning model, there often is a forward pass and a backward pass. In particular, a forward pass refers to the process of passing input data through the network to obtain the output predictions. When using a neural network, this involves calculating the activations of the neurons layer-by-layer from the input layer to the output layer. A backward pass, also known as backpropagation, involves calculating gradients of the loss function with respect to each parameter in the network. This is done by propagating the error from the output layer back through the network, allowing the model to update its parameters using these gradients and an optimization algorithm, such as gradient descent. In many cases, the matrices being used as input matrices of the GEMM and the output matrix are also used during the backward pass, such that the transposition of one or more of the matrices during the forward pass also has an effect on the backward pass. Therefore, the cumulative number of transpositions during the forward pass and backward pass, and whether both the GEMM used during the forward pass and the backward pass can be mapped to a matrix layout variant that is part of the first set, may be used to select the matrix layout variant (for the forward pass). Thus, the method may comprise, if the GEMM is performed as part of a training operation being performed on a machine-learning model, calculating or estimating, for the matrix layout variants of the first set of matrix layout variants, an additional number of elements to be transposed when performing one or more additional GEMM involving the first input matrix, the second input matrix and / or the output matrix during a backward pass based on the sizes of the respective matrices. The method may comprise selecting the matrix layout variant of the first set of matrix layout variants based on the number of elements to be transposed and based on the additional number of elements to be transposed. The selection of the matrix layout variant may be applied to the backward pass GEMM(s) as well. This may avoid cases in which the performance of the forward pass is improved at the detriment of the backward pass, which usually involves twice the number of GEMMs.

[0015] Depending on the matrix layout variant, none of the matrices (first input matrix, second input matrix, output matrix), one, two or three of the matrices may be transposed before performing the GEMM. Thus, each matrix layout variant may define whether a first input matrix and a second input matrix to be multiplied by the GEMM and an output matrix being the result of the GEMM are to be transposed.

[0016] The GEMM APIs exposed by the respective libraries usually expose a mechanism to specify, for the input matrices, whether the input matrices are to be transposed, resulting in four different matrix layout variants (C = A ** B, C = A ** BT, C = AT** B, C = AT** BT, with XTindicating that matrix X is transposed). Four additional matrix layout variants, which may yield better performance than the above-mentioned matrix layout variants, can be achieved by swapping the first and second input matrix. This results in variants CT= B ** A, CT= B ** AT, CT= BT** A, CT= BT** AT. Thus, at least one of the first and the second set of matrix layout variants includes a matrix layout variant having a transposed output matrix and / or swapped input matrices. This extends the number of matrix layout variants, and thus increases the likelihood that a matrix layout variant yielding improved performance can be selected while reducing or minimizing the number of transpositions necessary for using the respective matrix layout variant.

[0017] In many cases, the hardware vendor or library vendor will be able to specify which of the matrix layout variants yields an improved performance / efficiency on a given computer system. Thus, the data structure may be a data structure provided by a hardware vendor of hardware of the computer system, with the hardware being used to perform the GEMM, or a data structure provided by a provider of a library using the GEMM. This way, an appropriate set of first matrix layout variants can be used that is available regardless of whether the sizes of the matrices are known in advance or not.

[0018] In an alternative approach, at least if the sizes of the respective matrices are known or partially known, automatic performance measurements may be conducted to identify the first set of matrix layout variants. Thus, the method may comprise executing, using the computer system, a series of benchmarks to determine the data structure. Alternatively, the data structure may be based on a previously executed series of benchmarks having been executed using the computer system. This way, there is no need for vendor-provided data structures.

[0019] To avoid having to execute the series of benchmark using the very computer system being used to perform the GEMM, benchmarks that have been executed on computer systems having equivalent hardware may be used. In other words, the data structure may be based on a previously executed series of benchmarks having been executed using at least one hardware component being equivalent to a hardware component included in the computer system, with the hardware component being used to perform the GEMM. This way, the need for benchmarking may be avoided for most computer systems of a fleet of computer systems comprising equivalent hardware.

[0020] In many cases, the selected matrix layout variant may feature at least one matrix that is transposed (when compared to the initial matrices). Accordingly, the method may comprise transposing, if indicated by the selected matrix layout variant, at least one of the first input matrix, the second input matrix and the output matrix based on the selected matrix layout variant. For example, the transposition may be specified explicitly, i.e., as parameter, when execution of the GEMM is initiated via a respective API. Alternatively, the transposition may be performed prior to the GEMM being initiated via the API. In case the transposition is performed on the output matrix, a transposition may be performed on the result of the GEMM, to re-transpose the result. There are several use cases that benefit from the proposed technique. For example, the GEMM may be performed as part of an inference operation being performed using a machine-learning model or as part of a training operation being performed on a machine-learning model. Both training a machine learning model and performing inference using the machine learning model uses a large number of GEMMs, so that applying the proposed method is most likely to result in a substantial performance increase.

[0021] Another use case of the proposed technique is in scientific computing. For example, the GEMM may be performed as part of a scientific or engineering calculation (computation), such as a simulation in the physical, chemical or biological domain. For example, the GEMM may be performed as part a calculation in the field of molecular dynamics, a Markov-chain Monte Carlo calculation, a calculation in the field of coupling flows, or a calculation in the field of equivariant flows. Many scientific computations rely on performing a large number of GEMMs, so that applying the proposed method is most likely to result in a substantial performance increase.

[0022] Accordingly, another aspect of the present disclosure relates to a computer- implemented method for performing inference on, or for training of, a machine learning model. The method comprises obtaining a first input matrix and a second input matrix, performing a GEMM using a selected matrix layout variant using the above method for accelerating a GEMM on a computer system, and using an output matrix provided by the GEMM as part of the inference or training. Both training a machine learning model and performing inference using the machine learning model uses a large number of GEMMs, so that applying the proposed method is most likely to result in a substantial performance increase.

[0023] Similarly, another aspect of the present disclosure relates to a computer- implemented method for performing a scientific or engineering calculation. The method comprises obtaining a first input matrix and a second input matrix. The method comprises performing a GEMM using a selected matrix layout variant using the above method for accelerating a GEMM on a computer system. The method comprises using an output matrix provided by the GEMM as part of the scientific or engineering calculation. Many scientific computations rely on performing a large number of GEMMs, so that applying the proposed method is most likely to result in a substantial performance increase.

[0024] Another aspect of the present disclosure relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out at least one of the above methods.

[0025] Another aspect of the present disclosure relates to a non-transitory, computer- readable medium comprising a program code that, when the program code may be executed on a processor, a computer, or a programmable hardware component, causes the processor, computer, or programmable hardware component to perform at least one of the above methods.

[0026] Another aspect of the present disclosure relates to a computer system comprising interface circuitry, machine-readable instructions, and processor circuitry to execute the machine-readable instructions to carry out at least one of the above methods.

[0027] There are several ways how to design and further develop the teaching of the present invention in an advantageous way. To this end it is to be referred to the dependent claims on the one hand and to the following explanation of preferred embodiments of the invention by way of example, illustrated by the figure on the other hand. In connection with the explanation of the preferred embodiments of the invention by the aid of the figure, generally preferred embodiments and further developments of the teaching will be explained. In the drawing

[0028] Fig. 1 a and 1 b show examples of a General Matrix Multiplication (GEMM) using non-transposed and transposed versions of a second input matrix;

[0029] Fig. 2 shows a performance comparison of different Basic Linear

[0030] Algebra Subprogram (BLAS) libraries with eight different layouts;

[0031] Fig. 3a shows an example of a computer-implemented method for accelerating a GEMM on a computer system; Fig. 3b shows an example of a computer system being configured to perform the computer-implemented method for accelerating the GEMM on the computer system;

[0032] Fig. 4a shows an example of a computer-implemented method for performing inference on, or for training of, a machine learning model;

[0033] Fig. 4b shows an example of a computer-implemented method for performing a scientific or engineering calculation; and

[0034] Fig. 4c shows an example of a computer system being configured to perform a computer-implemented method for performing inference on, or for training of, a machine learning model and / or a computer- implemented method for performing a scientific or engineering calculation.

[0035] In many cases, the most performance-critical part of GEMM is to find the right way of loading / storing the input and output matrices, as the operation is mainly memory bandwidth bound. The following code describes a naive implementation of GEMM. As can be seen, the inner-most loop requires 2 memory loads compared to a single fused-multiply-add operation. for n in range(N): for m in range(M): sum = 0 for k in range( sum +=

[0036] C[n, m] = sum * alpha + C[n,m] * beta

[0037] The GEMM API allows four different input / output layout variants for executing GEMM (for simplicity, (op(...)), alpha, beta were removed). These change how the matrices A and B get read from the memory. In the following, XTdenotes the transposition of matrix X.

[0038] Fig. 1 a and 1 b show examples of a General Matrix Multiplication (GEMM) using non-transposed and transposed versions of a second input matrix, representing layout variants 1 (Fig. 1 a) and 2 (Fig. 1 b). In Fig. 1a, the first input matrix has the dimensions k x m and the second input matrix has the dimensions n x k. Accordingly, the resulting output matrix has dimensions n x m. In Fig. 1 b, the first input matrix has the dimensions k x m and the transposed second input matrix has the dimensions k x n. Also in this case, the resulting output matrix has dimensions n x m.

[0039] Additionally, four more layout variants can be achieved by swapping the inputs A and B. This causes the sizes of the matrices (m, n and k) to change, which can have an impact on performance. However, this results in transposition of the output C.

[0040] Fig. 2 shows an example of different BLAS libraries with the eight different layouts. As can be seen, although the operation itself is mathematically always identical, the chosen layouts can have a significant impact on the performance.

[0041] The main reason for this difference is how the memory access within the device works. Memory chips usually do not read and write single data values, but entire cache lines. If the layout being used only reads a few values from the cache line, the memory transfers a lot of data to the processor with low utilization. In the optimal case, the data layout allows to utilize 100% of the cache line, resulting in peak memory bandwidth utilization. Different layouts and how they get accessed are visualized at the URL: https: / / www.adityaagrawal.net / blog / architecture / matrix_rnultiplication

[0042] Further, there are different ways of implementing the matrix multiplications, using inner or outer products. Inner products accumulate all results in a register before writing the output, while the outer product computes partial results and accumulates them in the output memory. It depends on the underlying hardware architecture which algorithm works best. Parallelizing outer products might require hardware support for atomic operations.

[0043] In summary, General Matrix Multiplication (GEMM) are one of the most relevant and performance critical operations in Al and many scientific or engineering applications today. However, due to syntactic limitations of high level frameworks and non- optimal implementations of vendor libraries, the performance of the implementation might not reach peak performance on the hardware. In the present disclosure, a method is proposed that can improve upon or solve this kind of performance critical problem.

[0044] In some concepts, explicit transpositions are applied to speed up GEMM. Depending on the hardware architecture and exact implementation, it is known to be beneficial to explicit transpose one of the inputs a of GEMM (see StackOverflow Discussion 19580456: “Why does transposing matrix before multiplication results in great speed up” or Issue 3197: “Performance: speed up GEMM by pre-faulting memory” of the OpenBLAS project on GitHub). As explained, this explicit transposition changes how the memory is read, so that the memory access can most efficiently be used.

[0045] This consideration is based on the fact that the computational effort / complexity of performing a GEMM is O(NA3), while transposing data only has a computational effort / complexity of O(N). However, depending on the matrix sizes and capabilities of the device, transposing first can be slower than running a suboptimal layout. It is assumed that there is no general rule that is always best, and decisions thus are made depending on the application (matrix sizes) and used hardware.

[0046] In state of the art frameworks such as PyTorch, the memory layouts cannot be freely chosen. E.g., C = torch. matmul(A, B) expects C = A * B. To apply transposition the matrices need to be explicitly transposed using:

[0047] A, B = ..., ...

[0048] Bt = B.t()

[0049] C = torch. matmul(A, Bt)

[0050] The PyTorch developers have identified that this can be a performance issue, if the matrix is very big and gets explicitly transposed, so the ‘B.t()’ operation is implemented as a meta data optimization. Instead of explicitly applying the transposition, it just changes the data strides. This meta data change then gets detected by the ‘torch. matmul(...)’ operation, and then does not execute the explicit transposition, but instead sets the transposition flags of the GEMM call accordingly. This meta data also allows the python compiler / interpreter to treat ‘B. t().t()’ as a no op (no operation), as they both cancel out each other.

[0051] However, as can be seen in Fig. 2, doing an explicit transpose can yield a significant speedup, because a much better suited implementation of GEMM can be used. But as PyTorch cannot be sure, as PyTorch does not have a decision mechanism in place, PyTorch uses the “safe route” to not transpose with the risk in choosing an inefficient layout. The user however can enforce transposition manually with ‘B.t(). contiguous ()’. However, there is no framework or compiler that automatically detects if explicitly executing a transposition before GEMM yields a better performance or not. Known frameworks rather try to optimize-out the transposition, similar to PyTorch, as this behavior seems more intuitive to remove work from the code. This behavior assumes that all layouts work identically well in GEMM, which is not fulfilled as shown in Fig. 2.

[0052] Also, manually optimized implementations such as FlashAttention (Dao et al: “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”) or FlashAttention-2 (Tri Dao: “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning”) do not consider this kind of optimization and instead only rely on the tensor cores of GPUs and to keep data within the registers / cache of the device.

[0053] The resulting downside can be seen in LLM (Large Language Model) inference (e.g., using a version of the Llama Large Language Model) on GPUs (Graphics Computing Units), but also in machine learning training pipelines, in particular within the backward pass (verified on machine-learning focused GPUs), where the backward pass is 6-9x slower than the forward pass, although mathematically only 1.5-2. Ox more computations are expected.

[0054] Examples of the present disclosure rely on deciding, whether executing an explicit transposition of the data before calling the GEMM library, yields a better performance or not. For example, this decision is performed as part of the method of Fig. 3a.

[0055] Fig. 3a shows an example of a computer-implemented method for accelerating a GEMM on a computer system, such as computer system 30 of Fig. 3b. The method comprises obtaining 310 a data structure comprising a representation of a first set of matrix layout variants known to be more efficient than a second set of matrix layout variants when being used to perform, by the computer system, a GEMM. Each matrix layout variant defines whether at least one of the matrices involved in the GEMM are to be transposed. Preferably, each matrix layout variant defines whether a first input matrix and a second input matrix to be multiplied by the GEMM and an output matrix being the result of the GEMM are to be transposed. The method comprises selecting 350 a matrix layout variant of the first set of matrix layout variants based on matrices involved in a GEMM to be performed. If indicated by the selected matrix layout variant, the method may comprise transposing 360 at least one of a first input matrix, a second input matrix and an output matrix based on the selected matrix layout variant. Alternatively, transposition may be performed by the respective computation framework or BLAS. The method comprises performing 370 the GEMM on the matrices using the selected matrix layout variant. The method of Fig. 3a may be performed by the corresponding computer system 30 shown in Fig. 3b. Fig. 3b shows an example of a computer system 30 being configured to perform the computer-implemented method for accelerating the GEMM on the computer system. The computer system 30 shown in Fig. 3b comprises interface circuitry 32, processor circuitry 34, machine-readable instructions, and, optionally, memory and / or storage circuitry 36. The interface circuitry 32 may be used to facilitate communication with other components or other computer systems. This communication may occur via a bus or a network, for example. The processor circuitry 34 may be used to provide the functionality of the system, for example, in conjunction with the interface circuitry 32 (for exchanging information) and / or optional memory or storage circuitry 36 (for storing information, such as machine-readable instructions). The processor circuitry 34 is therefore coupled with the interface circuitry 32 and, optionally, with the memory or storage circuitry 36. For instance, the system 30 may comprise machine-readable instructions, i.e., a computer program, which may prompt the one or more processors to execute the method introduced in connection with Fig. 3a. In accordance with the aforementioned, the method introduced in connection with Fig. 3a may be carried out by the one or more processors executing the machine- readable instructions.

[0056] In the following, four GEMM cases are evaluated within PyTorch, with the inputs x[100, 11008] (as first input matrix) and y[4096, 11008] (as second input matrix), or their transposed inputs xt / yt. x @ y x @ yt.t() xt.t() @ y xt.t() @ yt.t()

[0057] The profiler of a GPU chip provider reports the following results (n = no transpose, t = transpose, where the notation of X,Y is reversed in the function name!):

[0058] As described before, PyTorch can inline the transpose operations into the matrix multiplication, using a different implementations ‘nn’, ‘tn’, ‘nt’ or ‘tt’. However, all cases where ‘yt’ is used as input perform significantly slower than the other combinations.

[0059] The time required for transposing a matrix can be only a fraction of total execution time. When using proper explicit transposition, significant speedups can be achieved:

[0060] Testcase PyTorch Proposed method Speedup

[0061] 1 299.55ms 300.59ms 1.00x

[0062] 2 695.03ms 367.33ms 1.89x

[0063] 3 303.72ms 304.97ms 1.00x

[0064] 4 672.69ms 368.81 ms 1.82x

[0065] As can be seen, in case 1 and 3, the original layout already performs best (minor differences come from measurement differences). However, in 2 and 4, speedups of >1 ,8x can be seen.

[0066] As described before, it is an objective of the present disclosure to improve the performance of GEMM. This is achieved by finding improved combinations of data layouts to minimize the overhead of explicit transpositions, thus maximizing the performance of GEMM operations.

[0067] It is evident that it can be useful to explicitly transpose the inputs of a GEMM. However, depending on the size of A and B, this transposition might be very expensive. Accordingly, the number of transpositions may be used to determine which of the layout variants (that make up the first set) to choose, or whether to forego the layout variants of the first set, as the necessary transpositions would outweigh the benefits of the better matrix layout variant. Accordingly, the method may comprise determining 320 respective sizes of a first input matrix, a second input matrix and of an output matrix of the matrices involved in the GEMM, and calculating or estimating 330, for the matrix layout variants of the first set of matrix layout variants, a number of elements to be transposed when performing GEMM on the matrices based on the sizes of the respective matrices. The matrix layout variant of the first set of matrix layout variants can then be selected based on the number of elements to be transposed. If the likely speedup of the matrix layout variants of the first set is outweighed by the computational effort of transposing the respective matrix, a matrix layout variant from the second set of matrix layout variants may be selected instead (if the total effort is reduced).

[0068] In some cases, in an extension of the above approach, the output matrix C may be transposed instead or in addition to transposition of the output matrices. As previously described, when, starting from C = A ** B, the inputs are swapped, CT= B ** A is the result. This changes the values of m, n and k, which, as shown in Fig. 2, can have a significant impact on the performance. An explicit transpose may be applied to the output matrix to get the correct result of C. Depending on the shapes of A and B, C can be a very small matrix, so that the transposition can be really cheap, compared to transposing large input matrices. Accordingly, the matrix layout variant may be selected also among matrix layout variants that feature a transposed output matrix, due to swapping the input matrices. In other words, at least one of the first and the second set of matrix layout variants may include a matrix layout variant having a transposed output matrix and / or swapped input matrices.

[0069] In the following, a good example (in which transposing the output matrix yields a low effort) and a bad example (in which transposing the output matrix yields a high effort) are shown:

[0070] Good Example: A[100, 10000] ** £[10000, 10] = C[100, 10]

[0071] Bad Example: A[100, 10] ** £[10, 10000] = C[100, 10000] In other approaches used in Al frameworks or compilers, such output transposes are not applied, neither are they mentioned in guidelines for optimal GEMM performance. Further, in cases such as ‘(A @ B).t()’, the user defined output transposition can be singled-out by this method.

[0072] In some examples, not only the GEMM at hand may be considered, but also further GEMM that have to be performed down the line and that are impacted by the selection of the matrix layout variant. Thus, in case of machine learning training, another relevant aspect for selecting the optimal layouts is to also consider the backward pass for training of the neural network.

[0073] In particular, the forward pass C = A ** B will result in two matrix multiplications to compute the gradients for dA and dB in the backward pass, dA = dC ** BTand dB = A ** dC. As can be seen, ‘B’ is used in the forward pass, we ‘BT’ is used in the backward pass. Transposing ‘B’ explicitly in the forward pass moves the negative performance impact to the backward pass. However, transposing A can have a positive impact on C = AT** B and dB = AT** dC, or transposing C, which also will result in dCT. Thus, if the GEMM is performed as part of a training operation being performed on a machine-learning model, the method may comprise calculating or estimating 340, for the matrix layout variants of the first set of matrix layout variants, an additional number of elements to be transposed when performing one or more additional GEMM involving the first input matrix, the second input matrix and / or the output matrix during a backward pass based on the sizes of the respective matrices. The method may comprise selecting the matrix layout variant of the first set of matrix layout variants based on the number of elements to be transposed and based on the additional number of elements to be transposed.

[0074] The impact onto the backward pass is thus far not considered Al frameworks or compilers. Further, as the backward pass requires two GEMMs, a wrong decision in the forward pass can double the negative impact on the backward pass. So, it might be more efficient to choose a bad layout for the forward pass, if then the two GEMM in backward pass are more efficient. Finding the correct layout depends on the user application. The required shapes of the matrices can be unknown, partially known or fully known at compile time. Unknown happens, if two inputs of the model get multiplied, as the shape of the inputs can be unknown. Partially known is the usual case for Al, where one input and one weight matrix get multiplied. The weight matrix is part of the model and therefore the sizes are known. Fully known occurs when the shape of the input is fixed.

[0075] For the decision making on selecting the matrix layout variant, a list of optimal layouts is used, i.e. the first set of matrix layout variants, which is part of the data structure. As can be seen in Fig. 2, the difference between the best layouts is neglectable, but the difference between best and worst can be significant.

[0076] In the case the input shapes are unknown, the hardware / library vendor may provide a list of “expected to be optimal” layouts. As explained, as the hardware’s memory has specific properties, such a list can be created by knowledgeable developers. Thus, the data structure may be a data structure provided by a hardware vendor of hardware of the computer system, with the hardware being used to perform the GEMM, or a data structure provided by a provider of a library using the GEMM.

[0077] In case of the partially or fully known shapes, automated performance measurements may be used, given the known sizes of the matrices that allow to compile a list that is optimized for the particular application test and hardware. In some cases, the automated performance measurements may be part of the method. Thus, the method may comprise executing 315, using the computer system, a series of benchmarks to determine the data structure. Alternatively, the benchmarks may have been performed prior to performing the method, using the computer system or using hardware that is equivalent to the hardware being used in the computer system. Thus, the data structure may be based on a previously executed series of benchmarks having been executed using the computer system. Alternatively o additionally, the data structure may be based on a previously executed series of benchmarks having been executed using at least one hardware component, such as a GPU or Al accelerator (e.g., tensor core or neural engine) being equivalent to a hardware component included in the computer system, with the hardware component being used to perform the GEMM.

[0078] Making better decisions at runtime may then include looking at the shapes of the matrices and our list of optimal layouts. Assuming this list is [C = A * BT, CT= B ** i4T]. Using the inputs A and B, the number of elements within all input / output matrices can be computed. Assuming [100, 10000] ** BflOOOO, 10] = C [100, 10], A has 1000000 elements, B has 100000 elements and C has 1000 elements.

[0079] As a result, C = A * BTrequires 100000 elements to be transposed, and CT= B ** ATrequires 1000000 + 1000 = 1001000 elements to be transposed, which is 10x more data that would need to be moved within memory and therefore is more expensive. Accordingly, as outlined above, the method may comprise determining 320 respective sizes of a first input matrix, a second input matrix and of an output matrix of the matrices involved in the GEMM, and calculating or estimating 330, for the matrix layout variants of the first set of matrix layout variants, a number of elements to be transposed when performing GEMM on the matrices based on the sizes of the respective matrices. In the example, the first layout would be selected, as it requires less items to be transposed. As an aside: If one dimension of a matrix is 1 , e.g., B [1, 1000], then the transpose is for free.

[0080] The proposed concept can be used in different use cases.

[0081] In particular, the proposed concept may be used for AI / ML inference and training. GEMM is the major building block in modern neural network architectures such as LLMs. These are the foundation for applications such as ChatGPT. Thus, the GEMM may be performed as part of an inference operation being performed using a machine-learning model or as part of a training operation being performed on a machine-learning model. The urge to provide optimal performance in very deep neural networks with billions of parameters is a key challenge, as GEMMs make up to 90% of the total execution time. Experiments have shown, a speedup of >1 ,8x for a single GEMM can have a significant impact on the overall performance of the ML model, such as the LLM. The proposed concept allows to significantly go beyond the performance of existing Al frameworks and compilers.

[0082] This is also reflected in the method of Fig. 4a. Fig. 4a shows an example of a computer-implemented method for performing inference on, or for training of, a machine learning model. For example, the method may comprise obtaining 410 a first input matrix and a second input matrix. For example, one of the first and second input matrix may be part of the machine learning model, and the other of the first and second input matrix may be data being processed by the machine learning mode. The method comprises performing 310-370 a GEMM (e.g., as shown in Fig. 3a) using a selected matrix layout variant using the method of Fig. 3a. The method comprises using 420 an output matrix provided by the GEMM as part of the inference or training.

[0083] In connection with AI / ML, the proposed concept may be used as part of the Al compiler, to allow (High Performance Computing) end users to run Al more efficiently on given hardware, e.g., hardware provided by a specific vendor. Higher performance over the competition may help differentiate the solution provided by the vendor from the competition. The proposed concept may also be used by specific hardware vendors to improve their combined hardware and software stack.

[0084] The proposed concept can also be used to improve the performance of existing prediction and training pipelines. A speedup of 1.8x can reduce training time, allowing customers to be quicker to the market, which is key differentiator, e.g., in the medical domain.

[0085] The proposed concept can also be used as part of an LLM technology stack. It can be used to accelerate prediction times, to a) answer user requests faster and b) to serve more user requests in parallel. Using the proposed concept, both goals can be achieved, reducing the operational costs of the Al infrastructure and serving more customer requests in shorter time.

[0086] The proposed concept may also be used as part of scientific Computing / Simulations (HPC, High Performance Computing), Engineering (CAD, Computer-Aided Design.) GEMM is a major operation also in all kinds of scientific and engineering applications, used to simulate physical, chemical, biological and other domains. Prominent examples are Molecular Dynamics, Markov-Chain Monte Carlo, Coupling flows, or Equivariant Flows. Thus, the GEMM may be performed as part of a scientific or engineering calculation, such as a simulation in the physical, chemical or biological domain, for example a calculation in the field of molecular dynamics, a Markov-chain Monte Carlo calculation, a calculation in the field of coupling flows or a calculation in the field of equivariant flows.

[0087] Accordingly, Fig. 4b shows an example of a computer-implemented method for performing a scientific or engineering calculation. The method comprising obtaining 410 a first input matrix and a second input matrix. The method comprises performing 310-370 (as shown in Fig. 3a) a GEMM using a selected matrix layout variant using the method of Fig. 3a. The method comprises using 430 an output matrix provided by the GEMM as part of the scientific or engineering calculation.

[0088] Especially due to the hype of Al technology, major players only focus mainly on the Al market, developing tools such as PyTorch, JAX or TensorFlow for Al, and putting little effort in assisting scientists of other areas to make advantage of peak performance on their hardware. With the proposed concept, non-AI scientists and engineers can be assisted automatically, as they rely on toolkits such as Torch MD, JAX MD or BGFIow.

[0089] The methods of Fig. 4a and / or Fig. 4b may be performed by the corresponding computer system 40 shown in Fig. 4c. Fig. 4c shows an example of a computer system being configured to perform a computer-implemented method for performing inference on, or for training of, a machine learning model and / or a computer- implemented method for performing a scientific or engineering calculation. The computer system 40 shown in Fig. 4c comprises interface circuitry 42, processor circuitry 44, machine-readable instructions, and, optionally, memory and / or storage circuitry 46. The interface circuitry 42 may be used to facilitate communication with other components or other computer systems. This communication may occur via a bus or a network, for example. The processor circuitry 44 may be used to provide the functionality of the system, for example, in conjunction with the interface circuitry 42 (for exchanging information) and / or optional memory or storage circuitry 46 (for storing information, such as machine-readable instructions). The processor circuitry 44 is therefore coupled with the interface circuitry 42 and, optionally, with the memory or storage circuitry 46. For instance, the system 40 may comprise machine- readable instructions, i.e. , a computer program, which may prompt the one or more processors to execute at least one of the methods introduced in connection with Fig. 4a and / or 4b. In accordance with the aforementioned, the methods introduced in connection with Fig. 4a and / or 4b may be carried out by the one or more processors executing the machine-readable instructions.

[0090] The proposed concept is based on adding or removing explicit transpositions before or after GEMM operations by selecting improved or optimal GEMM input / output data layouts based on a list of optimal layouts. In some examples, the proposed concept includes considering transposing the output of the GEMM to select better working implementation (as A and B get swapped), if transposing the output is cheaper than any input. In some examples, the proposed concept includes considering the impact of transposing A, B or C for forward and backward pass onto the overall computation time.

[0091] The present disclosure provides methods and systems for framework identical pseudo random number generation in compiled tensor computation graphs, comprising one or more of the following operations / components. For example, the method may comprise determining list of optimal layouts. In case of unknown shapes (i.e., dimensions of the respective matrices), the list of optimal layouts (i.e., first set of matrix layout variants) may be provided by the hardware / library vendor. In case of (partially) known shapes, measuring the execution time may be used to identify best layouts for the particular use case. The method further comprises deciding at runtime which layout to use, i.e., selecting a matrix layout variant among the first set of matrix layout variants, based on the matrix shapes, and list of optimal layouts. The method may include considering to transpose the output instead of the inputs, if this transposition is cheaper. The method may include, in training, considering the influence of layouts of the forward pass onto the performance of the backward pass. The present invention may be implemented as a computer-implemented method, computer system (comprising one or more processors and one or more storage devices) configured to perform the computer-implemented method and / or as a computer program for performing the computer-implemented method. For example, the computer-implemented method may include one or more steps and / or operations discussed above.

[0092] Various aspects of the present invention relates to machine learning. Machine learning is a branch of artificial intelligence that involves the development of algorithms and models that allow computers to learn and make predictions or decisions without being explicitly programmed. It focuses on creating systems that can improve their performance over time by learning from data.

[0093] Training a machine-learning model refers to the process of teaching the model to make accurate predictions or decisions. During training, the model is exposed to a large amount of data, which is used to adjust the model's internal parameters or weights. The model learns patterns, relationships, or rules from the training data, allowing it to generalize and make predictions on new, unseen data.

[0094] Training data is the set of examples or instances that is used to teach a machinelearning model. It is often labeled data, meaning that each example is associated with a known outcome or target value. The training data consists of both input features and the corresponding output or target variable. The model learns from this data by analyzing the patterns and relationships between the input features and the target variable. Training algorithms, such as supervised learning, semi-supervised learning, unsupervised learning or reinforcement learning may be used for training the machine-learning model.

[0095] Machine-learning models are often implemented as Artificial Neural Networks (ANNs), and in particular Deep Neural Networks, Support Vector Machines, Decision Tree models, or Random Forest models.

[0096] Examples may involve or relate to computer programs, including program codes to execute one or more of the mentioned methods when the program is executed on a computer, processor, or other programmable hardware component. As a result, steps, operations, or processes from various methods described above can also be executed by computers, processors, or other programmable hardware components. Examples may additionally cover program storage devices, such as digital data storage media, which are machine-, processor-, or computer-readable and encode and / or contain machine-executable, processor-executable, or computer-executable programs and instructions. These devices may include or be digital storage devices, magnetic storage media like magnetic disks and tapes, hard disk drives, or optically readable digital data storage media, for instance. Other examples encompass computers, processors, control units, field programmable logic arrays (FPLAs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), applicationspecific integrated circuits (ASICs), integrated circuits (ICs), or system-on-a-chip (SoC) systems that are programmed to carry out the steps of the aforementioned methods. In simpler terms, examples may involve computer programs and storage media comprising computer programs, as well as hardware components like processors and control units, which can be programmed to execute the methods described above.

[0097] For example, the interface circuitry 32, 42 of the systems 30, 40 introduced in connection with Figs. 3b and / or 4c may include or correspond to a network interface circuitry and / or a device interface circuitry configured to be communicatively coupled to one or more other devices, such as the one or more processors. For example, the interface circuitry may include a transmitter, a receiver, or a combination thereof (e.g., a transceiver), and may enable wired communication, wireless communication, or a combination thereof. For example, the processor circuitry 34, 44 of the systems 130, 40 introduced in connection with Figs. 3b and / or 4c may include or correspond to one or more of a digital signal processor circuitry (DSP), a graphical processing unit (GPU), and / or a central processing unit (CPU). For example, the memory and / or storage circuitry 36, 46 of the systems 30, 40 introduced in connection with Figs. 3b and / or 4c may include or correspond to volatile or nonvolatile storage circuitry, such as Random Access Memory (RAM), magnetic disks, optical disks, or flash memory devices. The one or more memory / storage devices may include both removable and non-removable memory devices.

[0098] When certain aspects are mentioned in relation to a device or system, they should also be considered as descriptions of the corresponding methods. For example, a block, component, or functional aspect of the device or system may correspond to a method step or feature of the related method. Therefore, aspects described regarding a method should also be understood as depicting a corresponding element, property, or functional feature of the corresponding device or system. In simpler terms, if something is described in relation to a device or system, it can also be applied to the corresponding method, and vice versa.

[0099] As shown before, embodiments of the present invention allow to improve the performance of GEMM operations beyond the performance provided by other systems. The state of the art considers “less operations” to be optimal and tries to remove the user defined transpositions from the computation graph. However, as GEMM is an expensive operation (O( / V3)) compared to transpose (O(AQ), explicitly transposing can be more efficient.

[0100] The present invention itself does not necessarily be limited to Al or Al compilers / optimizers, but embodiments can even be directly integrated into the BLAS libraries, for example.

[0101] Many modifications and other embodiments of the invention set forth herein will come to mind the one skilled in the art to which the invention pertains having the benefit of the teachings presented in the foregoing description and the associated drawings. Therefore, it is to be understood that the invention is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the invention. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation. L i s t o f r e f e r e n c e s i g n s , 40 Computer system , 42 Interface circuitry , 44 Processor circuitry , 36 Memory or storage circuitry

[0102] 310 Obtaining a data structure

[0103] 315 Performing a series of benchmarks

[0104] 320 Determining respective sizes of the input matrices and of the output matrix

[0105] 330 Determining or estimating a number of elements to be transposed

[0106] 340 Determining or estimating an additional number of elements to be transposed in a backward pass

[0107] 350 Selecting a matrix layout variant

[0108] 360 Transposing a matrix

[0109] 370 Performing a General Matrix Multiplication

[0110] 410 Obtaining a first input matrix and a second input matrix

[0111] 420 Using an output matrix provided by a General Matrix Multiplication as part of machine learning training or inference

[0112] 430 Using an output matrix provided by a General Matrix Multiplication as part of a scientific or engineering calculation

Claims

C l a i m s1. A computer-implemented method for accelerating a General Matrix Multiplication, GEMM, on a computer system, comprising: obtaining (310) a data structure comprising a representation of a first set of matrix layout variants known to be more efficient than a second set of matrix layout variants when being used to perform, by the computer system, a GEMM, with each matrix layout variant defining whether at least one of the matrices involved in the GEMM are to be transposed; and selecting (350) a matrix layout variant of the first set of matrix layout variants based on matrices involved in a GEMM to be performed; and performing (370) the GEMM on the matrices using the selected matrix layout variant.

2. The method according to claim 1 , comprising determining (320) respective sizes of a first input matrix, a second input matrix and of an output matrix of the matrices involved in the GEMM, calculating or estimating (330), for the matrix layout variants of the first set of matrix layout variants, a number of elements to be transposed when performing GEMM on the matrices based on the sizes of the respective matrices, and selecting the matrix layout variant of the first set of matrix layout variants based on the number of elements to be transposed.

3. The method according to claim 2, comprising, if the GEMM is performed as part of a training operation being performed on a machine-learning model, calculating or estimating (340), for the matrix layout variants of the first set of matrix layout variants, an additional number of elements to be transposed when performing one or more additional GEMM involving the first input matrix, the second input matrix and / or the output matrix during a backward pass based on the sizes of the respective matrices, and selecting the matrix layout variant of the first set of matrix layout variants based on the number of elements to be transposed and based on the additional number of elements to be transposed.

4. The method according to one of the claims 1 to 3, wherein each matrix layout variant defines whether a first input matrix and a second input matrix to be multipliedby the GEMM and an output matrix being the result of the GEMM are to be transposed.

5. The method according to one of the claims 1 to 4, wherein at least one of the first and the second set of matrix layout variants includes a matrix layout variant having a transposed output matrix and / or swapped input matrices.

6. The method according to one of the claims 1 to 5, wherein the data structure is a data structure provided by a hardware vendor of hardware of the computer system, the hardware being used to perform the GEMM, or a data structure provided by a provider of a library using the GEMM.

7. The method according to one of the claims 1 to 6, wherein the method comprises executing (315), using the computer system, a series of benchmarks to determine the data structure, or wherein the data structure is based on a previously executed series of benchmarks having been executed using the computer system.

8. The method according to one of the claims 1 to 6, wherein the data structure is based on a previously executed series of benchmarks having been executed using at least one hardware component being equivalent to a hardware component included in the computer system, with the hardware component being used to perform the GEMM.

9. The method according to one of the claims 1 to 8, comprising transposing (360), if indicated by the selected matrix layout variant, at least one of a first input matrix, a second input matrix and an output matrix based on the selected matrix layout variant10. The method according to one of the claims 1 to 9, wherein the GEMM is performed as part of an inference operation being performed using a machinelearning model or as part of a training operation being performed on a machinelearning model.

11. The method according to one of the claims 1 to 10, wherein the GEMM is performed as part of a scientific or engineering calculation, such as a simulation in the physical, chemical or biological domain, for example a calculation in the field of molecular dynamics, a Markov-chain Monte Carlo calculation, a calculation in the field of coupling flows or a calculation in the field of equivariant flows.

12. A computer-implemented method for performing inference on, or for training of, a machine learning model, the method comprising: obtaining (410) a first input matrix and a second input matrix; performing (310-370) a GEMM using a selected matrix layout variant using the method according to one of the claims 1 to 10; and using (420) an output matrix provided by the GEMM as part of the inference or training.

13. A computer-implemented method for performing a scientific or engineering calculation, the method comprising: obtaining (410) a first input matrix and a second input matrix; performing (310-370) a GEMM using a selected matrix layout variant using the method according to one of the claims 1 to 9, 11 ; and using (430) an output matrix provided by the GEMM as part of the scientific or engineering calculation.

14. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out at least one of the methods according to one of the claims 1 to 13.

15. A computer system (30, 40) comprising interface circuitry (32, 42), machine- readable instructions, and processor circuitry (34, 44) to execute the machine- readable instructions to carry out at least one of the methods according to one of the claims 1 to 13.

Images