Two-dimensional vector masking for boosting matrix multiplication performance

Abstract

Two-dimensional vector masking for boosting matrix multiplication performance is disclosed. A processor can receive a request to perform a matrix multiplication operation between a first matrix and a second matrix, determine a vector length of a register provided by the processor, and perform the matrix multiplication operation to obtain a third matrix. The matrix multiplication operation can include a plurality of iterations, each iteration including determining a number of remaining rows based on a total number of rows in the first matrix and the vector length, determining a number of remaining columns based on a total number of columns in the second matrix and the vector length, creating a first mask based on the number of remaining rows, creating a second mask based on the number of remaining columns, and transferring data associated with the first matrix and the second matrix from a memory associated with the processor to the register based on the first mask and the second mask, respectively.

Description

Technical Field

[0001] This disclosure relates to optimizing computer performance. More specifically, this disclosure relates to improving the performance of matrix multiplication operations in computer systems. Background Technology

[0002] Matrix multiplication is a fundamental operation in numerous high-performance computing (HPC) workloads, encompassing traditional applications such as the Millennium Car-Parrinello Code (CP2K) and the Vienna Ab initio Simulation Package (VASP), as well as contemporary machine learning (ML) and deep learning (DL) tasks. The performance of matrix multiplication directly impacts the training and inference times of many ML / DL models, particularly in the field of deep neural networks. Therefore, improving the computational efficiency of matrix multiplication is crucial for the scalability of ML / DL models, enabling them to handle larger datasets and adapt to increasingly complex architectures.

[0003] two The typical algorithm for matrix multiplication exhibits a time complexity of O(n). O(n³) This makes matrix multiplication a computationally demanding operation, especially for large matrices. One optimization approach is based on Single Instruction Multiple Data (SIMD) instructions, which facilitate the vectorization of matrix multiplication operations. This vectorization allows the resulting matrix to be partitioned into sub-blocks or tiles, enabling parallel processing. However, the resulting matrix may have arbitrary dimensions that are not divisible by the tiles. Attached Figure Description

[0004] Figure 1A An example register block method for matrix multiplication according to one aspect of this disclosure is illustrated.

[0005] Figure 1B An example register block scenario with partially filled registers is illustrated according to one aspect of this disclosure.

[0006] Figure 2 A table is illustrated that compares the number of condition checks required by conventional unmasked fragmentation techniques with the number of condition checks required by register masking techniques, according to one aspect of this disclosure.

[0007] Figure 3 A table illustrating a comparison of the number of condition checks required by conventional unmasked fragmentation techniques with the number of condition checks required by register masking techniques, according to one aspect of this disclosure, is provided.

[0008] Figure 4A A flowchart illustrating an example process for performing matrix multiplication operations according to one aspect of this disclosure is presented.

[0009] Figure 4B A flowchart illustrating example operations within each iteration of one aspect of this disclosure is presented.

[0010] Figure 5 The illustration shows a measured relative performance improvement of a register masking scheme according to one aspect of the present disclosure compared to a conventional scheme without register masking.

[0011] Figure 6 An example computer system is illustrated that facilitates matrix multiplication using register masking according to one aspect of this disclosure.

[0012] Figure 7 The illustration depicts a computer-readable medium (CRM) that facilitates matrix multiplication using register masking, according to one aspect of this disclosure.

[0013] In these accompanying drawings, the same reference numerals refer to the same elements. Detailed Implementation

[0014] Matrix multiplication is a fundamental operation in linear algebra that combines two matrices to produce a new matrix. The process involves multiplying the rows of the first matrix (called the multiplier matrix) with the columns of the second matrix (called the multiplicand matrix). The resulting matrix (called the product matrix) encapsulates the combined information from the two input matrices, making this operation crucial in various fields such as computer graphics, data analysis, and machine learning. An example matrix multiplication operation can be defined as... , where the matrix , and The dimensions are respectively , and Each element of the product matrix (represented as) ) is a multiplier matrix The i Row and multiplicand matrix The j The dot product of the columns.

[0015] The standard implementation of matrix multiplication typically employs nested loops, which operate along the three dimensions of the matrix. , and Up iteration. These dimensions are usually referred to as... m Dimensions n Dimensions and k Dimension. Dimensions and Dimensions can also be called row dimensions and column dimensions, respectively, because they correspond to the rows and columns in a product matrix. The order of nested loops can be determined based on data access patterns. For example, when a choice is made... When sorting, nested loops can include the outermost loop. Intermediate layer loop and the innermost loop ,in, , and It is an iterator, and the item This indicates the number of iterations used for loop unrolling, a technique that improves performance by reducing loop overhead and increasing instruction-level parallelism.

[0016] SIMD instructions significantly improve the computational efficiency of matrix operations, including matrix multiplication, by enabling parallelization. SIMD processors are equipped with wider registers, often called vector registers, capable of holding multiple data elements simultaneously. A key advantage of SIMD is that a single instruction can be applied to all elements in the vector register at once, thus vectorizing matrix multiplication operations. This parallelism greatly improves processing speed and efficiency.

[0017] The above presentation Matrix multiplication in this form can be extended to: ,in, , and There are two main methods for storing matrices in local memory: row-major order or column-major order. In row-major order, the elements of each row are stored contiguously, creating an ordered arrangement of rows. Conversely, in column-major order, the elements of each column are stored contiguously, producing an ordered arrangement of columns.

[0018] Taking column-major order as an example, a matrix (e.g., matrix...) A Each column of the matrix can be stored in a contiguous block of memory, which may also include additional unused data, resulting in redundant rows in the matrix. In one example, variables can be defined... t , as a continuous column (e.g., and The stride between the first elements of dimension ). For dimension matrix A The memory address of the first element in a consecutive column can follow the following rules: ,in, and .

[0019] Matrix multiplication implementations must handle four combinations of transposed (T) and non-transposed (N) input matrices: NN, TN, NT, and TT. While the mathematical results are consistent across these combinations, the data access patterns and cache efficiencies vary considerably. NN represents non-transposed access to both matrices; TN represents non-transposed access to the matrix... A It is a transpose access and for matrices B It is a non-transpose access; NT represents a matrix. A Is it a non-transpose access and for the matrix B It is a transpose access; and TT means that both matrices are non-transpose accesses.

[0020] Assuming to adopt Circular sorting, (N) or (T) matrix A and (N) or (T) matrix B Multiplication can include multiplying matrices A of m Elements and Matrices B of n Multiplying elements by elements, because we are multiplying an m × 1 vector by a 1 × n vector, resulting in one... Matrix (i.e., matrix) C ), represented as: ,exist The iteration proceeds upwards. In other words, within the innermost loop, The matrix was updated in total. k The above mathematical representation is independent of the input, and the input matrix... A and B The N / T direction only affects the data loading implementation method of the algorithm, and does not affect the mathematical results.

[0021] To further optimize the performance of matrix multiplication operations, register blocking or tiling techniques can be employed. This reduces the number of iterations, thereby improving overall efficiency. In this method, the resulting product matrix (matrix...) C The matrix is ​​divided into sub-blocks or tiles within registers suitable for the processor. The size of these sub-blocks or tiles is determined by two main factors: the vector length of each register (i.e., the number of matrix elements a single register can hold) and the total number of available registers. Furthermore, the arrangement of the registers (e.g., row-based or column-based) defines the product matrix. The partitioning method. Strategic partitioning of the product matrix allows for more efficient use of processor resources, especially in terms of cache utilization and register management.

[0022] Register partitioning can be applied to matrices of any size (where...) , and It can have any value and is particularly beneficial for small matrices, which are prevalent in machine learning (ML) and deep learning (DL) workflows. For small matrices, loop management (e.g., index computation and branch prediction) consumes a disproportionate amount of execution time. Furthermore, resource utilization has a greater impact on small matrices. In ML / DL applications, transformers and recurrent networks involve batches of small matrix multiplications (e.g., attention heads and token embeddings), and matrix multiplication optimization is crucial for latency-sensitive training. The logical complexity of matrix multiplication algorithms can significantly impact ML / DL training performance due to potential branch misprediction causing pipeline stalls. Additionally, condition checks (such as if-else statements) introduce additional execution overhead.

[0023] In a simple example, we can assume m and n It is the register vector length (represented as) l () multiples of. Vector and Multiplication between vectors produces The sub-block is represented as: Product matrix C The sub-blocks in the middle can target and and with size as l The step size is calculated, and... The previous iteration. In other words, the input matrix (i.e., the matrix...) A sum matrix B Multiplication can be achieved by multiplying by... Product matrix (i.e., matrix) C Each of ) Sub-blocks perform total k This will be executed in the next update.

[0024] Based on some aspects, regarding the matrix C In The transformation of the above mathematical expression of the sub-block into a vectorized kernel can begin with... A vector multiplied by a scalar produces vector: Then in all columns Repeat this operation. Then for... It can Sub-block update k Next. Applying this definition to a vector register allows you to store matrices...A of l A vector of consecutive elements and its containing matrix B The vector multiplication of the broadcast values, expressed in vector multiplication form, is as follows: .

[0025] The above vector operations on matrices A and B All N / T directions are valid. Data preprocessing can be performed if necessary. Regarding the access patterns of the input matrix stored in memory, a total of four cases must be considered, including the untransposed multiplier matrix. A (present) (matrix form), transpose multiplier matrix A (present) (matrix form), non-transpose multiplicand matrix B (present) (in matrix form), and the transpose of the multiplicand matrix B (present) (In matrix form). More specifically, for non-transpose matrices. A Access involves from the first k Column loading m consecutive elements, while for the transpose matrix A Access involves from the first k line collection m Non-continuous elements. On the other hand, for the non-transpose matrix... B Access involves from the first k line collection n Non-continuous elements, while for the transpose matrix B Access involves from the first k Column loading n There are consecutive elements. Of the four cases mentioned above, only the transpose matrix... A Non-transpose matrix B Additional processing is required (e.g., collecting scattered matrix elements).

[0026] To simplify the discussion, consider The scenario in which the matrix A and B The multiplication generation dimension is product matrix C For an untranspose matrix A or transpose matrix B Each matrix's column elements can be stored contiguously in memory. During matrix multiplication, the column elements from the corresponding matrix are... k Columns l A series of consecutive elements can be transferred to a vector register. On the other hand, for a transpose matrix... A or nontranspose matrix BThe row elements of each matrix can be arranged with intervals of 1 / 2. t The nth unit. During matrix multiplication, the nth unit from the corresponding matrix. k OK l Non-contiguous elements can be transferred to the vector register. Contiguous elements can be transferred to the vector register using standard load instructions, while the transfer of non-contiguous elements may require intermediate steps (e.g., using collect instructions).

[0027] For dimensions m and n yes l A large input matrix that is a multiple of the given matrix, used to extract from the untransposed matrix A The pseudocode for transmitting data can be represented as: And used from the transpose matrix A The pseudocode for transmitting data can be represented as: On the other hand, it is used to extract from an untransposed matrix. B The pseudocode for transmitting data can be represented as: ; Used from the transpose matrix B The pseudocode for transmitting data can be represented as: ; .Notice, and These are respectively containing from the matrix A and B The register of data, which is used to calculate the product matrix. C of This is necessary for updating the column-major order of sub-blocks. Load instructions copy contiguous data from a specified memory location to a vector register, collect instructions copy non-contiguous data (e.g., data with strafing access patterns) from a specified memory location to a vector register, and broadcast instructions copy data to all elements within the vector register. This applies to matrix... B Broadcasting one element to the entire vector register is equivalent to... Vector (i.e., from a matrix) A A column) and a scalar (i.e., a matrix) B Multiply the elements in the array.

[0028] Figure 1A An example register block method for matrix multiplication according to one aspect of this disclosure is illustrated. Figure 1A In the middle, the dimension is Matrix 102 (matrix) A ) and dimension Matrix 104 (matrix) B Multiplying them together produces a product matrix of 106 (matrix). CThe product matrix is ​​divided into sub-blocks or tiles, including tile 110. Each tile (e.g., tile 110) can be obtained from the matrix. A and B Loading required elements and step-by-step traversal k Dimensions are used for calculation. Tile 110 can have... The dimension, in which, This corresponds to the number of rows of the matrix elements stored in the tile, and Corresponding to the number of columns.

[0029] exist Figure 1A In the example shown, m and n yes l Multiples of, and . Figure 1A An example of the arrangement of registers in tile 110 is also illustrated, as shown in enlarged view 120 of tile 110. When represented using registers, the size of tile 110 is... ,in, h It is the row dimension (also known as m The number of registers in the dimension), and i It is a column dimension (also known as n The number of registers in the dimension. Considering that each register can store... l The dimension of a 110-tile matrix can also be represented by the number of matrix elements. In this example, and .

[0030] As discussed previously, the matrix A and B Multiplication between them can involve stepping register tiles in the column and row dimensions until the entire matrix is ​​traversed. C The following is an example pseudocode for performing matrix multiplication:

[0031] Depending on the layout of the input matrix in memory, either a load instruction or a collect instruction will be used to transfer matrix elements from memory to registers. In the last line, a store instruction will move data from a register. Copy to the specified storage location.

[0032] Considering that in practice, the input matrix (i.e., matrix) A and B It may have any dimension. m and n Maybe not l Multiples of. Therefore, in the product matrix CThe column and / or row step register tiles may produce partially filled vector registers. Figure 1B An example register block scenario with partially filled registers is illustrated according to one aspect of this disclosure.

[0033] exist Figure 1B In the middle, the dimension is Matrix 122 (matrix) A ) and dimension Matrix 124 (matrix) B Multiplying them together produces a product matrix 126 (matrix). C Similar to Figure 1A In the example shown, the product matrix 126 is divided into sub-blocks or tiles, such as tile 130, which has a dimension of ,in, In this example, m and n Not vector length l Multiples of this. Therefore, as register tiles step to the edges in both column and row dimensions, some registers may be partially filled or filled with elements that do not belong to the matrix. A , B and C The data.

[0034] In traditional methods, vector operations on partially padded vector registers may involve reducing the tile size at the edges, which can involve checking for several additional cases. These additional cases refer to applying smaller tiles to transfer remaining rows or columns from memory to the vector register. To ensure coverage of all possible cases regarding the number of bits in the partially padded register, the algorithm needs to check for tiles whose dimensions are powers of 2 and less than a certain value in descending order. l Additional cases. For any implementation, this requires additional cases to be powers of 2, a consequence of using a binary representation architecture in the processor. For the sake of simplicity without loss of generality, it can be assumed that... and A complete list of additional cases may include tile sizes from the following set: ,in, Note that only when l It is only included in the list if it is not a power of 2.

[0035] The additional cases to be checked may differ along the row and column dimensions. Figure 1B In the example shown, it can be assumed that the maximum number of rows and columns is equal to the vector length, i.e., and Therefore, the conditions to be checked in the row dimension can be defined as follows: ,in, Only when It is only included in the list if it is not a power of 2. Similarly, the case along the column dimension can be defined as follows: ,in, Only when It is only included in the list if it is not a power of 2. or If it is a power of 2, then the total number of cases needs to be checked. ) can be ;if or If it is not a power of 2, then the total number of cases needs to be checked. ) can be In some examples, the vector length of the vector register in the processor can be eight. In this case, matrix multiplication operations may require checking for additional cases 4, 2, and 1 in both the row and column dimensions. These checks may need to be performed within nested loops (including...). m cycle, n Cycles and k Executing if-else statements within a loop can negatively impact matrix multiplication performance.

[0036] To improve performance (e.g., by reducing code complexity and overhead associated with additional cases), two-dimensional (2D) vector register masking schemes can be used, depending on several aspects, to transfer matrix data from memory to registers. Register masking allows selective modification or reading of specific bits within a register while leaving other bits unchanged. Depending on several aspects, the mask can include a binary pattern for isolating or modifying specific bits, and applying the mask can involve bitwise operations (e.g., AND, OR, XOR) on the mask and register values. For example, if a specific bit in the mask is set to zero, and the mask is applied using an AND operation, that specific bit will be read as zero by the processor.

[0037] Assumption and or and ,and and The following is sample pseudocode for creating a register mask in the row or column dimension:

[0038] The generated mask can be applied to vector instructions (e.g., load or collect instructions). For example, an AND operation between the transferred data and the mask can force a specific bit in a register to zero. In this way, the data transfer operation effectively ignores extra data channels in the current register tile. Figure 1B In the middle, when updating the matrix CWhen the shadowed bits are occluded (represented by the shaded area), they will be ignored because they are all zero.

[0039] Assuming to adopt Circular sorting is used to implement register masking to update the product matrix. C The pseudocode for the sub-block in the code is shown below:

[0040]

[0041] The first two lines of pseudocode will m and n The maximum value is set to l Additionally, the row and column dimensions of the register block (i.e., mblk and nblk ) is set to not greater than l These requirements allow the vector mask to be applied to all iterations without requiring additional logic. The pseudocode above includes calls to the mask creation function discussed earlier to create register masks in both the row and column dimensions, i.e., respectively. and These created masks are applied to data operations (e.g., load, collect, and store). More specifically, the mask is applied as a bitwise AND operation to fill unused channels in registers with zeros, so that load or collect operations only collect data belonging to the input matrix from memory. Filling unused vector channels with zeros does not affect the results of arithmetic operations (e.g., vector addition), so masking is not required for such instructions. The number of cycles required to perform the operation does not depend on the number of active channels (i.e., the number of unmasked channels) because vector instructions operate on multiple data elements simultaneously.

[0042] The above pseudocode does not require m and n yes l Multiples of, and including the input matrix A and B All combinations of data direction (N or T). As discussed previously, when m and n no l When dealing with multiples of a certain value, the algorithm needs to check for several additional cases in both the row and column dimensions. Each of these cases requires multiple condition checks. The number of condition checks is determined by iterating through the list of cases for each item in descending order until a case is found where the sum equals... m or n It is determined by the correct combination of cases. For example, for m or The algorithm needs to check the following combinations of additional cases: A total of 9 condition checks are required. In another example, for m or The algorithm needs to check additional cases. A total of 6 condition checks are required. Odd sizes (e.g., when...) m or n (When the number is odd) all cases need to be checked. Register masking can significantly reduce the number of checks required because a single vector mask can represent all cases. The algorithm only needs to create a suitable mask, which is equivalent to a single condition check.

[0043] Figure 2 A table illustrating a comparison of the number of condition checks required by conventional unmasked fragmentation techniques with the number of condition checks required by register masking techniques, according to one aspect of this disclosure, is provided. Figure 2 As can be seen from this, for dimensions in For an input matrix within a given range, traditional unmasked piecewise operations require checking four possible cases, averaging 1.63 cases and 4.63 checks across all sizes. Register masking, on the other hand, only requires checking one case and only one check across all sizes. In some examples, an 8-bit register mask can include bits 0, 1, ..., and 7. For vector length... The processor can be based on The vector mask bits (represented as) b The size of the mapping to the input matrix (represented as) s ).

[0044] exist Figure 1A and Figure 1B In the example shown, the register tiles are configured as follows: This makes the dimension of the tile, in terms of matrix elements, be... In practice, register tiles can have different configurations depending on the dimension of the input matrix and the number of registers available in the processor. In one example, a register tile configuration could be... This makes the dimension of the tile, in terms of matrix elements, be... Cases to be considered in row and column dimensions (e.g., tile dimensions) include and Using register masking, tile size (exist m Dimension or n Each case in the dimension can be extended to a range. A set of sizes within. Therefore, for the above configuration as The register tiles, the case under consideration can be extended to and Note that inn The cases considered in the dimension will not be extended because .

[0045] The number of checks required for any given tile configuration can be further minimized, and overall performance can be optimized by including additional cases (other than powers of 2) to obtain the complete set. ,in, This refers to the maximum allowed tile dimension. For example, if the maximum allowed tile dimension is... m If the dimension is 48, you might want to check all cases ranging from 48 to 1. However, by using register masking, for cases within the range... The cases within each case only need to be checked once, thus greatly reducing the number of checks. The complete set of cases can be described by the following formula: and By definition, the relationship between successive occlusion cases is given by the following equation: and , Therefore, the set of results required for this implementation can include... .

[0046] Figure 3 A table illustrating a comparison of the number of condition checks required by a conventional unmasked fragmentation technique and the number of condition checks required by a register masking technique, according to one aspect of this disclosure, is shown. In this example, the configuration of the register tiles is... The vector length is 8, which leads to the following conditions that need to be checked: Furthermore, the dimensions of the input matrix can be within the range As can be seen from the table, traditional unmasked fragmentation techniques require checking seven possible cases, averaging 3.38 cases and requiring 14.38 checks across all sizes, while register masking techniques only require checking one case and only one check across all sizes.

[0047] Figure 4A A flowchart illustrating an example process for performing matrix multiplication operations according to one aspect of this disclosure is presented. Figure 4A All or any part of the operations shown can be executed by the processor executing the instructions. Although Figure 4A The example process in the flowchart illustrates a specific order in which certain operations are performed, but the process is not limited to this order. Operations shown consecutively in the flowchart can be performed in different orders, and can be performed simultaneously, partially simultaneously, or in combination.

[0048] During operation, the processor may receive a request (operation 402) to perform matrix multiplication between the first and second matrices. The two matrices may have arbitrary dimensions (e.g., rows and columns), provided that the number of columns in the multiplier matrix (e.g., the first matrix) is equal to the number of rows in the multiplicand matrix (e.g., the second matrix). If this condition is not met, the processor may return an error message.

[0049] The processor can determine the vector length included in the registers within the processor (Operation 404). The vector length (represented as...) l This corresponds to the number of elements in the first or second matrix stored in the corresponding register. Note that the size of the register can depend on the type of processor (e.g., manufacturer and model) used to perform matrix multiplication. Determining the vector length can involve dividing the register size by the data type size. In some examples, the processor may use 512-bit registers, and the data type of the matrix elements may be 64-bit integers / floating-point, resulting in a vector length of eight.

[0050] The processor can then perform matrix multiplication to obtain a third matrix, which involves multiple iterations (operation 406). The third matrix is ​​the product of the first and second matrices. The number of rows in the third matrix is ​​equal to the number of rows in the first matrix, and the number of columns in the third matrix is ​​equal to the number of columns in the second matrix. In some examples, the order of iteration can follow... Circular sort, where the outermost loop is in the row dimension (i.e., m In the dimension), the intermediate layer loops through the column dimension (i.e., n (in dimension), and the innermost loop is k Within the dimension. Each m / n A loop defines a sub-block in the product matrix, and each k The sub-block is updated iteratively. Performing matrix multiplication involves updating each sub-block of the product matrix in total. k Second-rate.

[0051] Figure 4B A flowchart illustrating example operations within each iteration of one aspect of this disclosure is presented. Figure 4B All or any part of the operations shown can be executed by the processor executing the instructions. Although Figure 4B The example process in the flowchart illustrates a specific order in which certain operations are performed, but the process is not limited to this order. Operations shown consecutively in the flowchart can be performed in different orders, and can be performed simultaneously, partially simultaneously, or in combination.

[0052] During the operation, the processor can determine the number of remaining rows based on the total number of rows in the first matrix and the vector length (operation 412). For which... The number of remaining rows in a small matrix can be determined by... It is confirmed that, among them, Corresponding to the number of rows in the first matrix, and Corresponding to the vector length. For which... m Greater than l In this case, the algorithm can first update the sub-blocks that do not have remaining rows without applying a register mask, until... m The value is less than l .

[0053] The processor can further determine the number of remaining columns based on the total number of columns in the second matrix and the vector length (operation 414). For which The number of remaining columns in a small matrix can be determined by... It is confirmed that, among them, Corresponding to the number of columns in the second matrix, and Corresponding to the vector length. For which... n Greater than l In this case, the algorithm can first update the sub-blocks that do not have remaining columns without applying a register mask, until... n The value is less than l .

[0054] The processor can create a first mask based on the number of remaining rows (operation 416) and a second mask based on the number of remaining columns (operation 418). Creating the corresponding mask may include initializing a mask with a length equal to the vector length, making it consist of all ones. For example, if the vector length is eight, the initial mask may consist of eight ones. Creating the mask may also include determining the difference between the vector length and the number of remaining rows or columns, and shifting and zeroing the initial mask based on that difference. In one example, there are seven remaining rows, and the difference between the vector length of eight and the number of remaining rows of seven is one. Therefore, the initial mask may be shifted one bit to the right and padded with zeros to generate the following mask. If there are no remaining rows or columns, the initialized mask will be used. .

[0055] The processor can then transfer data associated with the first and second matrices from processor-associated memory to registers based on a first mask and a second mask, respectively (operation 420). Depending on some aspects, the data associated with the first and second matrices is stored in memory in row-major or column-major order. Furthermore, the data access mode can be transposed or non-transposed. Transferring data may include applying the first mask to transfer data associated with a first number of rows of the first matrix, and applying the second mask to transfer data associated with a second number of columns of the second matrix. Applying the mask may involve a bitwise AND operation between the transferred data and the mask.

[0056] The processor can update a sub-block of the third matrix based on the data associated with the first and second matrices transferred to the register (operation 422). The dimension of this sub-block can be represented as... ,in, and According to some aspects, in order to update the sub-blocks of the third matrix, the processor performs a vectorized dot product on the transmitted data associated with the first and second matrices and accumulates the results. In a specific... Within the sub-block, the processor can perform calculations To determine the number of remaining rows, where the remainder is within the range Internally. Similarly, the processor can perform calculations. To determine the number of remaining columns, where the remainder is within a certain range. Within a given input matrix (e.g., the first matrix and the second matrix), the processor can also determine the number of rows corresponding to a sub-block (i.e., the number of rows), and determine the number of remaining rows by dividing the total number of rows in the first matrix by that number. Similarly, the processor can determine the number of columns corresponding to a sub-block (i.e., the number of columns), and determine the number of remaining columns by dividing the total number of columns in the second matrix by that number. The processor can iterate repeatedly along the column and row dimensions until all sub-blocks within the third matrix (e.g., ...) are considered. Figure 1A The sub-block 110 shown Figure 1B Sub-block 130 shown has been updated. Figure 1B As shown, most sub-blocks can be updated using an all-one mask because the remainder will be zero. Only edge sub-blocks (e.g., sub-blocks with remaining rows and columns) may require a zero-bit mask.

[0057] The following is an example pseudocode for performing matrix multiplication using register masking:

[0058]

[0059] The first two lines of the pseudocode above define the number of rows and columns of a set of register tiles used in each loop, where, and These represent the configuration of the maximum tiles (also known as base register tiles) in the row and column dimensions, respectively. The number of rows and columns have been sorted in descending order. Row dimensions are cyclic or... m The loop is the outermost loop, and the column dimension loop is... n The loop is an intermediate loop, and k The loop is the innermost loop, with a step size of . .exist k Within the loop, masked load / collect instructions are used to load data from matrices A and B, with the mask on the outside. m Cycles and n The pseudocode is generated within a loop. Based on several aspects, the above pseudocode can be implemented as a computational kernel for matrix multiplication. Compared to traditional methods using unmasked register fragmentation techniques, the proposed matrix multiplication method reduces the number of required condition checks, thereby lowering computational overhead.

[0060] In the example above, the register tiles have a fixed size (i.e., for all...). m Dimension value, n (The dimension values ​​are all fixed). In some examples, variable register tiles can be used to improve register utilization, where the maximum... n Dimension values ​​can depend on the current m Dimension value. Depending on several aspects, for a given... m The maximum dimension value can be determined based on the following constraints. n Dimension values: ,in, N It is the total number of registers provided by the processor. It is used to store data from the product matrix. C The number of elements in the register, It is used to store data from the multiplier matrix. A The number of registers containing the elements, and It is used to store data from the multiplicand matrix. B The number of registers containing the elements. For in-place / copy-free algorithms, the constraint can be expressed as... .

[0061] Figure 5 The illustration shows a measured relative performance improvement of a register masking scheme according to one aspect of this disclosure compared to a conventional scheme without register masking. In this example, matrix multiplication is implemented in-place / copy-free to load data from memory. The term "copy-free" means that the process does not call external code to pack the matrix data. The underlying register tile configuration is... Performance is measured by the Intel Xeon Gold 6330 CPU's speed in executing input matrices of various sizes. More specifically, the matrix... A The number of rows in a set can be determined from the set. Select from matrix B The number of columns in a set can be determined from the set. Choose from, and .from Figure 5 As can be seen, significant performance gains are achieved with input matrices of various sizes. In some cases (e.g., , and The performance gain is over 100%. The average performance improvement is 66.39% across all matrix dimensions tested.

[0062] Figure 6 An example computer system facilitating matrix multiplication using register masking according to one aspect of this disclosure is illustrated. Computer system 600 may include one or more processing resources (e.g., processing resource 602), one or more storage devices (e.g., storage device 604), and memory 606. According to some aspects, processing resource 602 may include multiple registers ( Figure 6 (not shown in the image), such as floating-point registers, which can temporarily store data loaded from memory 606.

[0063] Processing resources may include, for example, one or more processors included in a single computing device or distributed across multiple computing devices. In some examples, concurrent processes may execute on a single computing device or multiple computing devices. As used herein, a “processor” may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to fetch and execute instructions, other electronic circuitry suitable for fetching and executing instructions stored on a computer-readable storage medium, or combinations thereof. In the examples described herein, processing resources may fetch, decode, and execute instructions stored on a storage medium to perform the function described with respect to the instructions stored on the computer-readable medium. In other examples, the function described with respect to any instructions described herein may be implemented in the form of electronic circuitry, executable instructions encoded on a computer-readable medium, or a combination thereof. The computer-readable storage medium may be located in or away from the computing device executing the instructions but accessible (e.g., via a computer network) for execution. In the examples illustrated herein, a node may be implemented by one or more computer-readable storage media.

[0064] Computer system 600 may be coupled to peripheral I / O user equipment 610 (e.g., display device 612, keyboard 614, and fixed-point device 616). Storage device 604 includes a non-transitory computer-readable storage medium and stores operating system 618, register-masked matrix multiplication instructions 620, and data 640. Computer system 600 may include... Figure 6 The entities shown are fewer or more entities. Depending on some aspects, computer system 600 can be implemented by standalone devices or computing nodes within a data center.

[0065] The register-masked matrix multiplication instruction 620 may include instructions that, when executed by the computer system 600, cause the computer system 600 to perform the methods and / or procedures described in this disclosure. The register-masked matrix multiplication instruction 620 may be executed at least on the processing resource 602.

[0066] The register-masked matrix multiplication instruction 620 may include instruction 622 for receiving a request to perform a matrix multiplication operation between a first matrix and a second matrix, as described above regarding... Figure 4A The operation 402 shown is described in the diagram. The first matrix is ​​usually called the matrix. A And the second matrix is ​​usually called the matrix. B Matrix multiplication can be part of a machine learning workflow. For example, in machine learning, features are often represented as matrices, which can be multiplied by weight matrices to transform a dataset into a new representation.

[0067] The matrix multiplication instruction 620 with register masking may include instruction 624 for determining the length of a vector included in registers in the processor, as described above regarding... Figure 4A The operation 404 shown is described. The processor may be part of the processing resource 602. The vector length corresponds to the number of elements in the first or second matrix stored in the respective register.

[0068] The register-masked matrix multiplication instruction 620 may include instruction 626 for performing matrix multiplication to obtain a third matrix, as described above. Figure 4A The operation 406 shown in the diagram describes matrix multiplication. Matrix multiplication can include multiple iterations.

[0069] Instruction 626 may further include instruction 628 for determining the number of remaining rows based on the total number of rows in the first matrix and the vector length, and for determining the number of remaining columns based on the total number of columns in the second matrix and the vector length, as described above regarding... Figure 4B Operations 412 and 414 are shown in the diagram. Register blocks or slices can be used for matrix multiplication, where the product matrix (matrix...) CA matrix can be divided into multiple sub-blocks. Determining the number of remaining rows or columns may involve dividing the total number of rows in the first matrix or the number of columns in the second matrix by the number of rows or columns in the sub-block. In one example, the number of remaining rows can be determined by dividing the total number of rows in the first matrix by the number of rows in the sub-block to satisfy a constraint. The largest The value is determined, and the number of remaining columns can be satisfied by dividing the total number of columns in the second matrix by the constraint. The largest The value determines this.

[0070] Instruction 626 may include instruction 630 for creating a first mask and a second mask based on the number of remaining rows and columns, respectively, as described above regarding... Figure 4B Operations 416 and 418 are shown in the diagram. The first mask is a row-dimensional mask, while the second mask is a column-dimensional mask. Each mask can be created by shifting multiple predetermined bits from an initial mask that is all ones and padding it with zeros. The number of bits shifted is equal to the difference between the vector length and the number of remaining rows or columns.

[0071] Instruction 626 may include instruction 632 for transferring data associated with the first matrix and the second matrix from processor-associated memory to a register based on the first mask and the second mask, as described above regarding... Figure 4B The operation 420 shown is described. The memory may be part of memory 606. Depending on the layout of the input matrix in the memory, the data transfer may include masked loading or collecting operations.

[0072] Instruction 626 may include instruction 634 for updating a sub-block of the third matrix based on data transferred to the register that is associated with the first and second matrices, as described above. Figure 4B As described in operation 422, updating this sub-block may include performing a vectorized dot product on the transmitted data associated with the first and second matrices and accumulating the results.

[0073] The matrix multiplication instruction 620 with register masking can include ratios Figure 6 The instructions shown are further examples. For instance, the matrix multiplication instruction 620 with register masking may include instructions for defining a set of register tiles used for register partitioning during a matrix multiplication operation. In some examples, the set of register tiles may be fixed-size tiles. In alternative examples, the set of register tiles may be variable-size tiles, where the maximum number of columns per tile depends on the number of rows in the tile.

[0074] Figure 7A computer-readable medium (CRM) facilitating matrix multiplication using register masking, according to one aspect of this disclosure, is illustrated. The CRM 700 may be a non-transitory computer-readable medium or device storing instructions that, when executed by a computer or processing resource, cause the computer or processing resource to perform a method. As used herein, a “computer-readable storage medium” can be any electronic, magnetic, optical, or other physical storage device for containing or storing information such as executable instructions, data, etc. For example, any computer-readable storage medium described herein can be any or a combination of RAM, EEPROM, volatile memory, non-volatile memory, flash memory, storage drives (e.g., HDD, SSD), any type of storage disk (e.g., compact disk, DVD, etc.). Furthermore, any computer-readable storage medium described herein may be non-transitory.

[0075] CRM 700 can store instructions 710 for receiving requests to perform matrix multiplication between the first and second matrices, as described above. Figure 4A The operation 402 shown describes the instruction 720 for determining the vector length of registers included in the processor, as described above regarding... Figure 4A The operation 404 shown is described; and the instruction 730 for performing matrix multiplication to obtain a third matrix, as described above regarding... Figure 4A The operation 406 shown is described.

[0076] Instruction 730 may further include instruction 732 for determining the number of remaining rows based on the total number of rows in the first matrix and the vector length, and for determining the number of remaining columns based on the total number of columns in the second matrix and the vector length, as described above regarding... Figure 4B Operations 412 and 414 are described herein; instruction 734, for creating a first mask and a second mask based on the number of remaining rows and columns respectively, as described above regarding... Figure 4B The operations 416 and 418 shown herein describe; instruction 736 for transferring data associated with the first matrix and the second matrix from processor-associated memory to a register based on the first mask and the second mask, as described above regarding... Figure 4B Operation 420 is described in the diagram; and instruction 738 is used to update a sub-block of the third matrix based on data transferred to the register that is associated with the first and second matrices, as described above regarding... Figure 4B The operation 422 shown is described.

[0077] CRM 700 can include more Figure 7The instructions shown are further instructions. For example, the CRM 700 may include instructions for defining a set of register tiles used for register partitioning during matrix multiplication operations. In some examples, this set of register tiles may be fixed-size tiles. In alternative examples, the set of register tiles may be variable-size tiles, where the maximum number of columns per tile depends on the number of rows in the tile.

[0078] Typically, aspects of this disclosure provide a solution to improve the performance of matrix multiplication, particularly for small matrices widely used in machine learning applications. More specifically, when register blocks are used for matrix multiplication operations, the disclosed system can use register masking techniques to handle the partial data loading and storage associated with the remaining rows and columns. Matrix multiplication operations typically involve nested iterations. Within each iteration, the system can determine the number of remaining rows and columns to create masks for the row and column dimensions, respectively. The row-dimensional mask can be used to transmit the elements of the multiplicand matrix, while the column-dimensional mask can be used to transmit the elements of the multiplicand matrix. Compared to traditional methods that require separate conditional checks for different remainder cases, vector masks are flexible and can represent all remainder cases.

[0079] One aspect of this disclosure provides a system and method for improving the performance of matrix multiplication operations. During operation, a processor may receive a request to perform a matrix multiplication operation between a first matrix and a second matrix, determine a vector length provided by the processor corresponding to the number of elements of the first or second matrix stored in the respective register, and perform the matrix multiplication operation to obtain a third matrix. The matrix multiplication operation may include multiple iterations, each iteration including: determining the number of remaining rows based on the total number of rows in the first matrix and the vector length; determining the number of remaining columns based on the total number of columns in the second matrix and the vector length; creating a first mask based on the number of remaining rows; creating a second mask based on the number of remaining columns; and transferring data associated with the first and second matrices from memory associated with the processor to registers based on the first and second masks, respectively.

[0080] In a variation of this, transmitting data may include applying a first mask to transmit data associated with a first number of rows of a first matrix, and applying a second mask to transmit data associated with a second number of columns of a second matrix.

[0081] In a further variation, applying the first mask or the second mask may include: initializing the first mask or the second mask to include all ones; determining the difference between the vector length and the number of remaining rows or columns; and shifting the mask based on the difference and filling the shifted bits with zeros.

[0082] In a variation thereof, the data associated with the first matrix and the second matrix is ​​stored in memory in row-major or column-major order, and transmitting the data associated with a first number of rows of the first matrix or a second number of columns of the second matrix may include accessing the data to be transmitted in a transposed or non-transposed manner.

[0083] In a variation of this, each iteration of the matrix multiplication operation may further include updating a sub-block of the third matrix based on data associated with the first and second matrices transferred to a register, the dimension of which is limited by the number of rows and columns.

[0084] In a further variation, determining the number of remaining rows may include determining the number of rows corresponding to the sub-block, and determining the number of remaining rows by dividing the total number of rows in the first matrix by that number of rows.

[0085] In a further variation, determining the number of remaining columns may include determining the number of columns corresponding to the sub-block, and determining the number of remaining columns by dividing the total number of columns in the second matrix by that number of columns.

[0086] The methods and processes described in the Detailed Description section can be embodied in code and / or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied in data structures and code and stored within the computer-readable storage medium.

[0087] The methods and processes described above may be included in hardware modules or devices. Hardware modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute specific software modules or code at specific times, and other programmable logic devices now known or developed later. When a hardware module or device is activated, it executes the methods and processes included therein.

[0088] The foregoing description is presented to enable any person skilled in the art to make and use the aspects and examples, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of this disclosure. Therefore, the aspects described herein are not limited to those shown, but are intended to be consistent with the maximum scope of the principles and features disclosed herein.

[0089] Furthermore, the foregoing descriptions of the various aspects have been presented solely for illustrative and descriptive purposes. These descriptions are not intended to be exhaustive or to limit the aspects described herein to the disclosed forms. Accordingly, many modifications and variations will be apparent to those skilled in the art. Additionally, the foregoing disclosure is not intended to limit the aspects described herein. The scope of the aspects described herein is defined by the appended claims.

Claims

1. A computer-implemented method, comprising: The processor receives a request to perform matrix multiplication between the first and second matrices. Determine the vector length of the registers provided by the processor, the vector length corresponding to the number of elements of the first matrix or the second matrix stored in the respective registers; as well as The processor performs the matrix multiplication operation to obtain a third matrix. The matrix multiplication operation includes multiple iterations, each iteration including: The number of remaining rows is determined based on the total number of rows in the first matrix and the length of the vector. The number of remaining columns is determined based on the total number of columns in the second matrix and the length of the vector. A first mask is created based on the number of remaining rows; A second mask is created based on the number of the remaining columns; and Data associated with the first matrix and the second matrix are transferred from the memory associated with the processor to the register based on the first mask and the second mask, respectively.

2. The computer-implemented method as described in claim 1, wherein, The data transmission includes: The first mask is applied to transmit data associated with a first number of rows of the first matrix; and The second mask is applied to transmit data associated with a second number of columns of the second matrix.

3. The computer-implemented method as described in claim 2, wherein, Applying the first mask or the second mask includes: Initialize either the first mask or the second mask to all values ​​of 1; Determine the difference between the vector length and the number of remaining rows or columns; and The mask is shifted based on the difference, and the shifted bits are filled with zeros.

4. The computer-implemented method as described in claim 1, in, Data associated with the first matrix and the second matrix is ​​stored in the memory in either row-major or column-major order; and The transmission of data associated with the first number of rows of the first matrix or the second number of columns of the second matrix includes accessing the data to be transmitted in a transposed or non-transposed manner.

5. The computer-implemented method as described in claim 1, wherein, Each iteration of the matrix multiplication operation further includes updating a sub-block of the third matrix based on the data associated with the first and second matrices transmitted to the register, the dimension of the sub-block being defined by the number of rows and columns.

6. The computer-implemented method as described in claim 5, wherein, Determining the number of remaining rows includes: Determine the row number corresponding to the sub-block; and The number of remaining rows is determined by dividing the total number of rows in the first matrix by the number of rows.

7. The computer-implemented method as described in claim 5, wherein, Determining the number of the remaining columns includes: Determine the number of columns corresponding to the sub-block; and The number of remaining columns is determined by dividing the total number of columns in the second matrix by the total number of columns.

8. A computer system, comprising: At least one processor; as well as A non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform a method comprising: Receive a request to perform matrix multiplication between the first matrix and the second matrix; Determine the vector length of the registers provided by the processor, the vector length corresponding to the number of elements of the first matrix or the second matrix stored in the respective registers; and The matrix multiplication operation is performed to obtain a third matrix. The matrix multiplication operation includes multiple iterations, each iteration including: The number of remaining rows is determined based on the total number of rows in the first matrix and the length of the vector. The number of remaining columns is determined based on the total number of columns in the second matrix and the length of the vector. A first mask is created based on the number of remaining rows; A second mask is created based on the number of the remaining columns; and Data associated with the first matrix and the second matrix are transferred from the memory associated with the processor to the register based on the first mask and the second mask, respectively.

9. The computer system of claim 8, wherein, The data transmission includes: The first mask is applied to transmit data associated with a first number of rows of the first matrix; and The second mask is applied to transmit data associated with a second number of columns of the second matrix.

10. The computer system of claim 9, wherein, Applying the first mask or the second mask includes: Initialize either the first mask or the second mask to all values ​​of 1; Determine the difference between the vector length and the number of remaining rows or columns; and The mask is shifted based on the difference, and the shifted bits are filled with zeros.

11. The computer system as described in claim 8, in, Data associated with the first matrix and the second matrix is ​​stored in the memory in either row-major or column-major order; and The transmission of data associated with the first number of rows of the first matrix or the second number of columns of the second matrix includes accessing the data to be transmitted in a transposed or non-transposed manner.

12. The computer system of claim 8, wherein, Each iteration of the matrix multiplication operation further includes updating a sub-block of the third matrix based on the data associated with the first and second matrices transmitted to the register, the dimension of the sub-block being defined by the number of rows and columns.

13. The computer system of claim 12, wherein, Determining the number of remaining rows includes: Determine the row number corresponding to the sub-block; and The number of remaining rows is determined by dividing the total number of rows in the first matrix by the number of rows.

14. The computer system of claim 12, wherein, Determining the number of the remaining columns includes: Determine the number of columns corresponding to the sub-block; and The number of remaining columns is determined by dividing the total number of columns in the second matrix by the total number of columns.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method comprising: The processor receives a request to perform matrix multiplication between the first matrix and the second matrix; Determine the vector length of the registers provided by the processor, the vector length corresponding to the number of elements of the first matrix or the second matrix stored in the respective registers; as well as The matrix multiplication operation is performed to obtain a third matrix. The matrix multiplication operation includes multiple iterations, each iteration including: The number of remaining rows is determined based on the total number of rows in the first matrix and the length of the vector. The number of remaining columns is determined based on the total number of columns in the second matrix and the length of the vector. A first mask is created based on the number of remaining rows; A second mask is created based on the number of the remaining columns; and Data associated with the first matrix and the second matrix are transferred from the memory associated with the processor to the register based on the first mask and the second mask, respectively.

16. The non-transitory computer-readable storage medium of claim 15, wherein, The data transmission includes: The first mask is applied to transmit data associated with a first number of rows of the first matrix; and The second mask is applied to transmit data associated with a second number of columns of the second matrix.

17. The non-transitory computer-readable storage medium of claim 16, wherein, Applying the first mask or the second mask includes: Initialize either the first mask or the second mask to all values ​​of 1; Determine the difference between the vector length and the number of remaining rows or columns; and The mask is shifted based on the difference, and the shifted bits are filled with zeros.

18. The non-transitory computer-readable storage medium of claim 15, wherein, The data associated with the first matrix and the second matrix are stored in the memory in either row-major or column-major order; and The transmission of data associated with the first number of rows of the first matrix or the second number of columns of the second matrix includes accessing the data to be transmitted in a transposed or non-transposed manner.

19. The non-transitory computer-readable storage medium of claim 15, wherein, Each iteration of the matrix multiplication operation further includes updating a sub-block of the third matrix based on the data associated with the first and second matrices transmitted to the register, the dimension of the sub-block being defined by the number of rows and columns.

20. The non-transitory computer-readable storage medium as claimed in claim 19, in, Determining the number of remaining rows includes: Determine the row number corresponding to the sub-block; and The number of remaining rows is determined by dividing the total number of rows in the first matrix by the number of rows; and Determining the number of remaining columns includes: Determine the number of columns corresponding to the sub-block; and The number of remaining columns is determined by dividing the total number of columns in the second matrix by the total number of columns.

Images