Variable vector register slicing for improved resource utilization of matrix multiplication

Abstract

The present disclosure relates generally to variable vector register tiling for improved resource utilization of matrix multiplication. One aspect of the present disclosure provides a system and method for improving register utilization in a processor. During operation, the processor can receive a request to perform a matrix multiplication operation between a first matrix and a second matrix; and determine a set of valid configurations of register tiles for the matrix multiplication operation based on a number of registers provided by the processor, a respective configuration including a number of rows and a number of columns. The number of columns is a function of the number of rows. The processor can then perform the matrix multiplication operation based on the determined set of valid configurations of register tiles to obtain a third matrix.

Description

Background of the Invention Technical Field

[0001] This disclosure relates to a system and method for improving the performance of matrix multiplication operations performed by a vector processor. More specifically, the system employs variable vector register tiling to perform matrix multiplication operations. Background Technology

[0002] Matrix multiplication is a fundamental operation in many high-performance computing (HPC) workloads, encompassing traditional applications such as the Millennium Car-Parrinello code (CP2K) and the Vienna Ab initio Simulation Package (VASP) as well as contemporary machine learning (ML) and deep learning (DL) tasks. The performance of matrix multiplication directly impacts the training and inference times of many ML / DL models, particularly in the field of deep neural networks. Therefore, improving the computational efficiency of matrix multiplication is crucial for the scalability of ML / DL models, enabling them to handle larger datasets and adapt to increasingly complex architectures.

[0003] two Typical matrix multiplication algorithms exhibit a time complexity of O(n³), making it a computationally demanding operation, especially for large matrices. One optimization approach is based on Single Instruction Multiple Data (SIMD) instructions, which facilitate the vectorization of matrix multiplication. This vectorization allows the resulting matrix to be partitioned into sub-blocks or tiles, enabling parallel processing. The dimensions of these sub-blocks or tiles affect the number of iterations required to complete the multiplication operation, thus playing a crucial role in determining the algorithm's performance. Attached Figure Description

[0004] Figure 1 An example register block method for matrix multiplication according to one aspect of this disclosure is illustrated.

[0005] Figure 2 The illustration shows a table indicating the register utilization of a set of fixed tile configurations for a remote implementation of matrix multiplication.

[0006] Figure 3A A table illustrating a list of effective tile configurations for a remote implementation of a processor equipped with 32 registers, according to one aspect of this disclosure, is shown.

[0007] Figure 3B The illustration shows a table presenting a list of effective tile configurations for a remote implementation and their corresponding register utilization rates, according to one aspect of this disclosure.

[0008] Figure 4AA flowchart illustrating an example process for performing matrix multiplication operations according to one aspect of this disclosure is presented.

[0009] Figure 4B A flowchart illustrating an example process for obtaining a product matrix according to one aspect of this disclosure is presented.

[0010] Figure 5A The illustration shows a table indicating the register utilization of a set of fixed tile configurations for an in-situ implementation.

[0011] Figure 5B A table illustrating a list of effective tile configurations for an in-situ implementation of a processor equipped with 32 registers, according to one aspect of this disclosure, is shown.

[0012] Figure 5C The illustration shows a table presenting a list of effective tile configurations for in-situ implementations and their corresponding register utilization rates, according to one aspect of this disclosure.

[0013] Figure 6 The illustration shows the relative performance improvement of variable register fragmentation technology compared to conventional fixed register fragmentation technology for an in-situ implementation according to one aspect of the present disclosure.

[0014] Figure 7 An example computer system for facilitating matrix multiplication using variable register slices according to one aspect of this disclosure is illustrated.

[0015] Figure 8 The illustration depicts a computer-readable medium (CRM) that facilitates matrix multiplication via variable register fragmentation according to one aspect of this disclosure.

[0016] In these accompanying drawings, the same reference numerals refer to the same elements. Detailed Implementation

[0017] Matrix multiplication is a fundamental operation in linear algebra that combines two matrices to produce a new matrix. The process involves multiplying the rows of the first matrix (called the multiplier matrix) by the columns of the second matrix (called the multiplicand matrix). The resulting matrix (called the product matrix) encapsulates the combined information from the two input matrices, making this operation crucial in various fields such as computer graphics, data analysis, and machine learning. An example of matrix multiplication can be defined as... , where the matrix , and The dimensions are respectively , and Each element of the product matrix (represented as) ) is a multiplier matrix The i Row and multiplicand matrix The j The dot product of the columns.

[0018] The standard implementation of matrix multiplication typically employs nested loops, which operate along the three dimensions of the matrix. , and Up iteration. These dimensions are usually referred to as... m Dimensions n Dimensions and k Dimension. Dimensions and Dimensions can also be called row dimensions and column dimensions, respectively, because they correspond to the rows and columns in a product matrix. The order of nested loops can be determined based on data access patterns. For example, when a choice is made... When sorting, nested loops can include the outermost loop. Intermediate layer loop and the innermost loop ,in, , and It is an iterator, and the item This indicates the number of iterations used for loop unrolling, a technique that improves performance by reducing loop overhead and increasing instruction-level parallelism.

[0019] SIMD instructions significantly improve the computational efficiency of matrix operations, including matrix multiplication, by enabling parallelization. SIMD processors are equipped with wider registers, often called vector registers, capable of holding multiple data elements simultaneously. A key advantage of SIMD is that a single instruction can be applied to all elements in the vector register at once, thus vectorizing matrix multiplication operations. This parallelism greatly improves processing speed and efficiency.

[0020] To further optimize performance, register partitioning or slicing techniques can be used, which can reduce the number of memory load / store instructions and improve overall efficiency. In this method, the resulting matrix (matrix) C The matrix is ​​divided into sub-blocks or tiles. The size of these sub-blocks or tiles is determined by two main factors: the vector length of each register (i.e., the number of matrix elements a single register can hold) and the total number of available registers. Furthermore, the arrangement of the registers defines the product matrix. This partitioning strategy of the product matrix allows for more efficient use of processor resources, particularly in terms of cache utilization and register management.

[0021] Register partitioning can be applied to matrices of any size (where...) , and (Can have any value). This disclosure focuses on smaller matrices for several reasons. Loop management overhead and resource utilization have a more significant impact on smaller matrices. Furthermore, optimizing matrix multiplication for smaller matrices is crucial for ML / DL training workflows and presents unique challenges. The logical complexity of matrix multiplication algorithms can significantly affect ML / DL training performance due to potential branch misprediction leading to pipeline stalls. Additionally, condition checks (such as if-else statements) introduce additional execution overhead. Conventional register partitioning methods typically use fixed-size tiles, which optimize register usage only for a finite subset of input parameters. Therefore, for... , and For input matrices with arbitrary values, overall performance can be negatively impacted by very low register utilization. To improve the performance of matrix multiplication over a wide input range, according to aspects of this disclosure, a matrix multiplication kernel can be implemented using variable-size tiles. This approach aims to more effectively adapt to different input dimensions, optimizing register usage and overall performance for a wider range of matrix sizes and shapes.

[0022] Figure 1 An example register block method for matrix multiplication according to one aspect of this disclosure is illustrated. Figure 1 In the middle, the dimension is Matrix 102 (matrix) A ) and dimension Matrix 104 (matrix) B Multiplying them together produces a product matrix of 106 (matrix). C The product matrix is ​​divided into sub-blocks or tiles, such as tile 110. Each tile (e.g., tile 110) can be obtained by analyzing the matrix... A and B Load the required values ​​and perform step-by-step traversal. k Dimensions are used for calculation. Tile 110 can have... The dimension, in which, This corresponds to the number of rows of the matrix elements stored in the tile, and Corresponding to the number of columns.

[0023] Figure 1 An example of the arrangement of registers in tile 110 is also illustrated, as shown in enlarged view 120 of tile 110. In this example, when represented using registers, the size of tile 110 is... ,in, h It is the row dimension (also known as mThe number of registers in the dimension), and i It is a column dimension (also known as n The number of registers in the dimension. Considering that each register can store... l The dimension of a 110-tile matrix can also be represented by the number of matrix elements. In one example, a processor performing matrix multiplication may include 32 registers, each capable of storing 512 bits. If each matrix element is represented using a 64-bit data type, then the vector length (i.e., l The maximum number of registers is eight (i.e., each register can hold a maximum of eight matrix elements).

[0024] Considering the input matrix (i.e., matrix) A and B It can have any dimension, for a dimension of Given tiles, the product matrix (i.e., matrix) C It may not be possible to be counted in the row dimension. Or cannot be measured on the column dimension i Integer division. Therefore, matrix multiplication operations need to consider the cycle remainder to handle the remaining rows or columns, such as... Figure 1 As shown. Conventional matrix multiplication algorithms typically begin with a base tile having a predetermined configuration. This configuration can be determined heuristically or empirically based on the average performance of a set of input parameters. For example, in a processor with 32 available registers, a conventional approach might choose a configuration as follows: and The basic register tiles, corresponding to dimensions of The matrix sub-blocks, the number of rows in the sub-blocks corresponds to And the number of columns in the sub-block corresponds to .

[0025] For a matrix whose dimensions are not divisible by the base tiles (e.g., row dimension 24 or column dimension 8) C Conventional methods handle remainder cases by checking smaller tiles whose dimension is a power of two (up to the base tile dimension). At the row dimension, the algorithm might need to consider the number of rows (e.g., ...). mblk The values ​​are 24, 16, 8, 4, 2, and 1. Similarly, in the column dimension, the number of columns needs to be considered (e.g., nblk The formula is 8, 4, 2, 1. This method leads to a large number of condition checks. More specifically, for each row dimension value (i.e., m The algorithm may need to check all column dimension values ​​(i.e., values). n (value), which in many cases can lead to low register utilization.

[0026] Vector register utilization (expressed as) The total number of registers used to calculate each tile can be defined as the total number of registers used to calculate the product matrix divided by the total number of available registers. The total number of registers used to calculate the tiles in the product matrix can be based on... It is confirmed that, among them, Defines a method for storing product matrices C of The number of registers required for the sub-block, and and Define the storage matrix respectively A and B The number of registers required for each element. Function and Depending on the implementation (e.g., off-site / copying or in-site / copyless implementation of matrix multiplication).

[0027] If matrix multiplication is performed using a local buffered off-site / copy algorithm, then and It defines a method for loading matrices via loop expansion. A and B The number of registers containing the data. (Item) Describes the matrix B A common implementation of loading data involves one register storing data from the matrix. B The currently needed elements are obtained, and additional registers are used to preload the matrix. B exist k The next element in the dimension. If Then only one register is needed, because there is no need to access the matrix. B Get other elements from it.

[0028] Therefore, register utilization can be calculated as ,in, N This is the total number of registers provided by the processor. For different tile configurations (e.g., different...), h and i (Value), register utilization may differ. Figure 2 A table illustrating register utilization for a set of fixed-tile configurations in a remote implementation of matrix multiplication is illustrated. In this example, N The value is 32, and the base tile configuration is set to 32. and , corresponding to dimension The matrix sub-blocks. The matrix multiplication algorithm uses... Circular sorting, where the innermost loop iterates through different... k Iterate over the value. From Figure 2As can be seen, except for the case of the base tile, register utilization is below 100% in most cases. This means that while the base tile achieves optimal register utilization, the remainder loop (where additional smaller tiles are used to compute the product matrix) is not optimized. C The remaining rows and columns (of data) typically lead to suboptimal register utilization. For example, and The tile configuration resulted in 63% register utilization, while and The tile configuration resulted in 69% register utilization. These examples highlight the inefficiencies that can occur when dealing with matrix dimensions that are not perfectly aligned with the base tile size.

[0029] Figure 2 The average register utilization of the set of tile configurations shown is expressed as E It can be calculated using the following formula: ,in, This refers to the total number of tiles in this group of tiles. Figure 2 In the example shown, the average register utilization is only 32%. Figure 2 In the example shown, the configuration of the base tile (i.e., the maximum tile) is typically determined heuristically or empirically based on average performance (e.g., average register utilization) across a range of inputs. Once the base tile is selected, the configuration of the smaller attached tiles is fixed, where each row dimension value (i.e., h The values ​​will use the same set of column dimension values ​​(i.e., (Value). For example Figure 2 As shown, for different h Value (i.e., The same group will always be applied. Value (i.e., While this fixed structure is easy to implement, it can lead to unsatisfactory register utilization for many matrix sizes, especially those that are not well aligned with the dimensions of the base tiles.

[0030] To improve the performance of matrix multiplication operations, particularly increasing average register utilization, this disclosure proposes a variable partitioning technique instead of using fixed base tiles with associated fixed-size tiles. Register partitioning logic units can implement this adaptive approach to partition the product matrix more efficiently. More specifically, all valid tile configurations can be considered, and the configuration with the largest row or column dimension can be selected for register partitioning, depending on the specific cyclic sorting used. According to a further aspect, each row dimension value can correspond to a set of column dimension values, where the largest column dimension value is a function of the row dimension values. Figure 1As shown, the size of tile 110 can be determined based on the dimensions of input matrices 102 and 104. Furthermore, it can be determined based on the row dimension values ​​(i.e., mblk ) Calculate column dimension values ​​(i.e., nblk This allows for better register utilization.

[0031] The optimal configuration of register tiles can be determined based on certain constraints, depending on several factors. These constraints stipulate that the total number of registers required for each load / store operation (i.e., the number of registers used to store matrix elements in the tile and related elements in the matrix to be multiplied) must be less than or equal to the total number of registers provided by the processor. ,in, N It is the total number of registers provided by the processor. It is used to store data from the product matrix. C The number of registers containing the elements. It is used to store data from the multiplier matrix. A The number of registers for the element, and It is used to store data from the multiplicand matrix. B The number of registers for the element. This constraint ensures that the selected tile configuration can be executed efficiently within the processor's available resources. It prevents the selection of tile sizes that would lead to register overflows, which occur when there are not enough registers to hold all the necessary data, resulting in costly memory operations.

[0032] The following is an example pseudocode for determining all valid configurations of register tiles given the total number of registers in a processor:

[0033] The example code above includes two nested loops designed to identify valid tile configurations. The outer loop operates in the row dimension (i.e., h From 1 to N-1 Iterate through a certain number of registers, and the inner loop iterates along the column dimension (i.e., i The algorithm iterates through a certain number of registers. The goal is to identify registers that satisfy the constraints (i.e., ...). All possible h- i Yes. For the matrix multiplication implementation using the local / replica algorithm with local buffering described above, the constraint becomes: There can be multiple valid tile configurations. For example, given... and , i The possible values ​​can range from 1 to 28. Since smaller tile sizes may result in a larger number of iterations, it is best to choose tiles with the largest dimension that still satisfy the constraints. For using... In implementing cyclic sorting, the register block logic can be determined first. m Tile size in dimension (i.e., the maximum size of a given multiplier matrix) h (value), then determine n The size of the tiles in the dimension (i.e., the size relative to the determined dimensions) h The maximum value corresponding to the value i value).

[0034] Figure 3A A table illustrating a list of effective tile configurations for a remote implementation of a processor equipped with 32 registers, according to one aspect of this disclosure, is shown. The first column of the table lists... h All possible values, and the second column lists the values ​​related to... h The maximum value corresponding to the value i Value. For example, when hour ;when hour, And so on. The third column shows the number of registers used per load / store operation for a given tile configuration. From Figure 3A As can be seen, all 32 registers are used in the five tile configurations. Figure 2 This is a significant improvement compared to the fixed-tile method shown (where only one configuration provides optimal utilization). Even in the worst-case scenario under this adaptive method, register utilization is still significantly better than... Figure 2 The situation shown is quite similar. This demonstrates that variable tile techniques can maintain or surpass the efficiency of fixed-tile methods across all matrix sizes.

[0035] As discussed earlier, the input multiplier and multiplicand matrices may not be divisible by the tiles of the maximum allowed dimension, and the register block logic needs to handle the loop remainder. According to some aspects, the column dimension tile size of the loop remainder can be determined based on the current row dimension tile size (i.e., i The value depends on the current h Value). The dependency between row and column dimensions allows for more adaptive sharding. Within a given loop of row dimensions (e.g., m (Loop), register block logic can first determine the current... h All possible values i Value. From the maximum i Starting with a value, logic can check each hi Can combinations be used for tile configurations? Depending on the alternative aspect, for a given... h Value, corresponding i Values ​​can include the maximum value and values ​​that are powers of two and less than the maximum value. For example, when hour, And the rest of the possiblei Values ​​can include , , , as well as In other words, it is used for partitioning product matrices. C The possible register tiles may have , , , , and The configuration. Similarly, when hour, And the rest i Values ​​can include , , as well as This method strikes a balance between flexibility and simplicity in tile size.

[0036] Figure 3B The illustration shows a table presenting a list of effective tile configurations for a remote implementation and their corresponding register utilization rates, according to one aspect of this disclosure. Figure 3B The table shows that there are five possible tile configurations. h- i The combination (i.e., 6-3, 5-4, 3-8, 2-13, and 1-28) can provide 100% register utilization. The average register utilization is 56%, significantly higher than... Figure 2 The example shown.

[0037] Figure 4A A flowchart illustrating an example process for performing matrix multiplication operations according to one aspect of this disclosure is presented. Figure 4A All or any part of the operations shown can be executed by the processor executing the instructions. Although Figure 4A The example process in the flowchart illustrates a specific order in which certain operations are performed, but the process is not limited to this order. Operations shown consecutively in the flowchart can be performed in different orders, and can be performed simultaneously, partially simultaneously, or in combination.

[0038] During operation, the processor may receive a request (operation 402) to perform matrix multiplication between the first and second matrices. The two matrices may have arbitrary dimensions (e.g., rows and columns), provided that the number of columns in the multiplier matrix (e.g., the first matrix) is equal to the number of rows in the multiplicand matrix (e.g., the second matrix). If this condition is not met, the processor may return an error message.

[0039] For matrix multiplication, the processor can determine a set of effective configurations of register tiles based on the number of registers provided by the processor (Operation 404). Register tiles can be used in register partitioning schemes, which divide the product matrix into sub-blocks and compute each sub-block independently to improve performance. The corresponding configuration of the register tiles can include the number of rows (denoted as ) that defines the number of rows in the tile. mblk ), and the number of columns that define the number of columns in a tile (represented as ). nblk The number of columns is a function of the number of rows. Based on certain aspects, it can be determined by finding all columns that satisfy the following constraints. hi To determine the valid configuration of register tiles: in, N It is the total number of registers provided by the processor. It is used to store data from the product matrix. C The number of elements in the register, It is used to store data from the multiplier matrix. A The number of registers containing the elements, and It is used to store data from the multiplicand matrix. B The number of registers for each element. If the memory access used to load the data of the input matrix is ​​a local / copy algorithm with local buffering, then the constraint becomes... In one example, the processor may include 32 floating-point registers, and a valid set of register tiles can be configured similarly to... Figure 3B The configuration shown.

[0040] Each register can have a predetermined vector length (denoted as ) corresponding to the number of matrix elements stored in the register. l According to some sources, the number of rows in a register tile is equal to the number of registers in a column multiplied by the vector length (i.e., The number of columns in a register tile is equal to the number of registers in a row (i.e., ).

[0041] The processor can then perform matrix multiplication based on a determined set of valid configurations of the register tiles to obtain a third matrix (operation 406). The third matrix is ​​the product of the first and second matrices. The number of rows in the third matrix is ​​equal to the number of rows in the first matrix, and the number of columns in the third matrix is ​​equal to the number of columns in the second matrix.

[0042] Figure 4B A flowchart illustrating an example process for obtaining a product matrix according to one aspect of this disclosure is presented. Figure 4B All or any part of the operations shown can be executed by the processor executing the instructions. Although Figure 4BThe example process in the flowchart illustrates a specific order in which certain operations are performed, but the process is not limited to this order. Operations shown consecutively in the flowchart can be performed in different orders, and can be performed simultaneously, partially simultaneously, or in combination.

[0043] During operation, the processor can select a configuration of register tiles from this set of valid configurations (Operation 412). Depending on the cyclic order, the processor may first select the maximum number of rows or the maximum number of columns in the valid configuration. In one example, using... Circular sorting (meaning the outermost loop iterates over the number of rows), and the processor can first select the row that satisfies the constraints. The maximum number of rows. The processor can further determine a set of column numbers corresponding to the maximum number of rows and select the maximum number of columns from that set. The number of rows with the selected register tiles is equal to the maximum number of rows, and the number of columns is equal to the maximum number of columns.

[0044] The processor can update the third matrix one sub-block at a time (operation 414). Each sub-block may include a subset of the rows and columns of the third matrix corresponding to the selected configuration of the register tiles, i.e., the number of rows and columns in the sub-block is equal to the number of rows and columns in the register tiles, respectively. Depending on some aspects, to update the sub-blocks of the third matrix, the processor may allocate an appropriate number of registers in the processor to load matrix elements of the first and second matrices from memory, perform vectorized dot products on the matrix elements, and accumulate the results. The allocation of registers can be determined based on the selected configuration of the register tiles.

[0045] In response to a third matrix including remaining rows and / or columns, the processor can select additional configurations for register tiles and update the remaining rows and columns based on these additional configurations (operation 416). Depending on some aspects, the processor can sort the number of rows with valid configurations in descending order. Furthermore, for each row number, the processor can sort the number of columns corresponding to that row number in descending order. The processor can then select each additional configuration for the register tiles based on the sorted number of rows and columns. In some examples, the processor can first update the third matrix along the column dimension, updating one sub-block at a time. In response to detecting a remaining column, the processor can select the next maximum number of columns corresponding to the current row number to form an additional smaller tile and use this additional smaller tile to update the remaining columns. The processor can repeat this process until all remaining columns within the current row have been updated. The processor can then update the third matrix along the row dimension. In response to detecting a remaining row, the processor can select the next maximum number of rows and repeat this process until all remaining rows have been updated.

[0046] The following is sample pseudocode for performing matrix multiplication:

[0047]

[0048]

[0049] The first two lines of the pseudocode define the number of rows and corresponding columns for a set of valid register tile configurations. The row numbers are sorted in descending order, and each row number corresponds to a set of column numbers, also sorted in descending order. The row-dimensional loop, or m-loop, is the outermost loop, and... k The loop is the innermost loop, with a step size of . These loops begin with the maximum number of rows and the corresponding maximum number of columns in the sorted list, and terminate when a remaining row or column is detected (e.g., m < and / or n < The algorithm gradually descends to the next row / column number. Depending on several aspects, the above pseudocode can be implemented as a computational kernel for matrix multiplication. Compared to conventional methods using a fixed base tile and a set of additional tiles associated with that base tile, the proposed matrix multiplication method can provide higher register utilization over a wider range of input matrices.

[0050] In addition to off-site / copying using a local buffering scheme, processors can also load data from memory using an in-place / copyless scheme, which can be optimized for small input matrices. Although often referred to as a copyless process, performance can be improved using local data buffering. The term copyless means that the process does not call external code to pack the matrix data. Function calls incur additional overhead. Input matrices can be loaded on demand (streaming) or packed into a local buffer for contiguous memory access. The data loading method depends on the data layout of the input matrices (column-wise or row-wise) and the order in which the process accesses the matrix elements. Data reuse is another important consideration. The two input matrices can be streamed or buffered, resulting in four possible combinations, but for simplicity, libraries typically stream or buffer both matrices. However, the data layout of the input matrices is not critical, as register tiles are related to the chunking of the product matrix.

[0051] If an in-place / copy-free algorithm is used to load the input matrix data into the processor's registers, the constraints on register tile configuration can be expressed as follows: ,in, N It is the total number of registers provided by the processor. This is the number of registers used to store elements from the product matrix, and additional terms. This is the number of registers required to load data from the input matrix without performing loop unrolling. This additional term is implementation-specific and may have different definitions.

[0052] In some examples, the total number of registers in the processor is 32, and the constraint on the effective tile configuration becomes... . Figure 5A The illustration shows a table indicating register utilization for a set of fixed tile configurations used in an in-situ implementation. In this example, the size of the base tile is... Furthermore, the average register utilization rate is only 29%. Figure 5B A table illustrating a list of effective tile configurations for an in-situ implementation of a processor equipped with 32 registers, according to one aspect of this disclosure, is shown. Figure 5C A table illustrating one aspect of this disclosure includes a list of effective tile configurations for in-situ implementations and their corresponding register utilization. For example... Figure 5C As shown, six tile configurations can provide 100% register utilization. The average tile utilization is 56%, which is almost twice as efficient as the conventional implementation using fixed base tiles.

[0053] In addition to improving register utilization, the variable register tile scheme also improves the performance of matrix multiplication (in terms of speed) by reducing the number of iterations. Figure 6 The illustration shows the relative performance improvement of variable register fragmentation technology compared to conventional fixed register fragmentation technology for an in-situ implementation according to one aspect of this disclosure. In this example, the input matrix has a wide range (e.g., m The range is from 8 to 56), and the fixed register fragmentation scheme uses a size of The base tiles. Performance of matrix multiplication using these two sharding schemes was measured on an Intel Gold 6330 CPU. From Figure 6 As can be seen, variable register slicing can improve the performance of matrix multiplication operations across different dimensions of the input matrix. Due to memory alignment, the performance gain varies, with an average performance improvement of 16.81% across all matrix dimensions.

[0054] Figure 7 An example computer system for facilitating matrix multiplication using variable register fragmentation according to one aspect of this disclosure is illustrated. Computer system 700 may include one or more processing resources (e.g., processing resource 702), one or more storage devices (e.g., storage device 704), and memory 706. According to some aspects, processing resource 702 may include multiple registers ( Figure 7 (not shown in the image), such as floating-point registers, which can temporarily store data loaded from memory 706.

[0055] Processing resources may include, for example, one or more processors included in a single computing device or distributed across multiple computing devices. In some examples, concurrent processes may execute on a single computing device or multiple computing devices. As used herein, a “processor” may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to fetch and execute instructions, other electronic circuitry suitable for fetching and executing instructions stored on a computer-readable storage medium, or combinations thereof. In the examples described herein, processing resources may fetch, decode, and execute instructions stored on a storage medium to perform the function described with respect to the instructions stored on the computer-readable medium. In other examples, the function described with respect to any instructions described herein may be implemented in the form of electronic circuitry, executable instructions encoded on a computer-readable medium, or a combination thereof. The computer-readable storage medium may be located in or away from the computing device executing the instructions but accessible (e.g., via a computer network) for execution. In the examples illustrated herein, a node may be implemented by one or more computer-readable storage media.

[0056] Computer system 700 may be coupled to peripheral I / O user equipment 710 (e.g., display device 712, keyboard 714, and fixed-point device 716). Storage device 704 includes a non-transitory computer-readable storage medium and stores operating system 718, instructions 720 for matrix multiplication via variable register slices, and data 740. Computer system 700 may include... Figure 7 The entities shown are fewer or more entities. Depending on some aspects, computer system 700 can be implemented by standalone devices or computing nodes within a data center.

[0057] Instructions 720 for performing matrix multiplication via variable register slices may include instructions that, when executed by computer system 700, cause computer system 700 to perform the methods and / or procedures described in this disclosure. Instructions 720 for performing matrix multiplication via variable register slices may be executed at least on processing resource 702.

[0058] Instructions 720 for matrix multiplication via variable register slices may include instructions 722 for receiving a request to perform matrix multiplication between a first matrix and a second matrix, as described above. Figure 4A The operation 402 shown is described in the diagram. The first matrix is ​​usually called the matrix. A And the second matrix is ​​usually called the matrix. BMatrix multiplication can be part of a machine learning workflow. For example, in machine learning, features are often represented as matrices, which can be multiplied by weight matrices to transform a dataset into a new representation.

[0059] Instructions 720 for matrix multiplication via variable register slices may include instructions 724 for determining a set of effective configurations of register tiles for matrix multiplication operations based on the number of registers provided by the processor, as described above regarding... Figure 4A The operation 404 shown is described. The processor may be part of processing resource 702. The corresponding configuration of the register tiles may include the number of rows defining the number of rows in the tile (in the product matrix). C (in terms of rows), and the number of columns that define the number of columns in a tile (in terms of the product matrix). C (Regarding columns in the table). The number of columns can be a function of the number of rows. Depending on some aspects, it can be determined by finding all rows that satisfy the following constraints. hi To determine the valid configuration of register tiles: in, N It is the total number of registers provided by the processor. It is used to store data from the product matrix. C The number of elements in the register, It is used to store data from the multiplier matrix. A The number of registers containing the elements, and It is used to store data from the multiplicand matrix. B The number of elements in the register.

[0060] Instruction 720 for matrix multiplication via variable register slices may include instruction 726 for performing matrix multiplication based on a determined set of valid configurations of register tiles to obtain a third matrix, as described above regarding... Figure 4A The operation 406 shown is described. Instruction 726 may further include instruction 728 for selecting a register tile from the set of valid configurations, as described above regarding... Figure 4B The operation 412 shown describes this. The selected configuration can have a maximum number of rows or columns, depending on whether the register tiles traverse the product matrix in the row or column dimension. If a number of rows is selected, a set of column numbers corresponding to that number of rows can be determined, and the maximum number of columns in that set can be selected.

[0061] Instruction 726 may include instruction 730 for updating the third matrix one sub-block at a time, as described above regarding... Figure 4BAs described in operation 414, each sub-block in the third matrix (i.e., the product matrix) has the same size and dimensions as the register tile. Instruction 730 may include instructions for allocating registers for loading matrix elements of the first and second matrices from memory (e.g., memory 706), instructions for performing vectorized dot products on the matrix elements, and instructions for accumulating the results to update the sub-blocks.

[0062] Instruction 726 may include instruction 732 for updating the remaining rows and columns in response to additional configuration of a third matrix selection register tile including the remaining rows and / or columns, as described above regarding... Figure 4B The operation 416 shown describes this. Since the product matrix may not be divisible by the previously selected register tile configuration, there may be remaining columns or rows after the register tiles have been traversed along the column or row dimensions respectively. To update the remaining rows and / or columns, additional smaller register tiles can be used. In some examples, the number of rows or columns in the smaller tile may be a power of two.

[0063] Instructions 720, which perform matrix multiplication using variable register slices, can include... Figure 7 The instructions shown are further instructions. For example, instruction 720 for matrix multiplication via variable register slices may include instructions for sorting the number of rows of the effective tile configuration in descending order, and instructions for sorting a set of corresponding column numbers for each row in descending order.

[0064] Figure 8 A computer-readable medium (CRM) facilitating matrix multiplication via variable register fragmentation according to one aspect of this disclosure is illustrated. The CRM 800 may be a non-transitory computer-readable medium or device storing instructions that, when executed by a computer or processing resource, cause the computer or processing resource to perform a method. As used herein, a “computer-readable storage medium” can be any electronic, magnetic, optical, or other physical storage device for containing or storing information such as executable instructions, data, etc. For example, any computer-readable storage medium described herein may be any one or a combination of RAM, EEPROM, volatile memory, non-volatile memory, flash memory, storage drives (e.g., HDD, SSD), any type of storage disk (e.g., compact disk, DVD, etc.). Furthermore, any computer-readable storage medium described herein may be non-transitory.

[0065] CRM 800 can store instructions 810 for receiving requests to perform matrix multiplication operations between the first and second matrices, as described above. Figure 4AThe operation 402 shown describes instructions 820 for determining a set of effective configurations of register tiles based on the number of registers provided by the processor for matrix multiplication operations, as described above regarding... Figure 4A The operation 404 shown is described; and the instruction 830 for performing matrix multiplication to obtain a third matrix based on a determined set of valid configurations of register tiles, as described above regarding... Figure 4A The operation 406 shown is described. Instruction 830 may further include instruction 832 for selecting a register tile from the set of valid configurations, as described above regarding... Figure 4B Operation 412 shown describes the instruction 834 for updating the third matrix one sub-block at a time, as described above regarding... Figure 4B The operation 414 shown is described; and the instruction 836 for updating the remaining rows and columns in response to an additional configuration of a third matrix selection register tile including the remaining rows and / or columns, and based on the additional configuration of the register tile, as described above regarding... Figure 4B The operation 416 shown is described.

[0066] CRM 800 can include more Figure 8 The instructions shown are further instructions. For example, the CRM 800 may include instructions for sorting the number of rows of valid tile configurations in descending order, and instructions for sorting a set of corresponding column numbers for each row in descending order.

[0067] Generally, aspects of this disclosure provide a solution to improve the performance of matrix multiplication, particularly for small matrices widely used in machine learning applications. More specifically, the disclosed system may use a variable register sharding technique that takes into account all available tile configurations when partitioning the product matrix into sub-blocks. In one example, the available tile configuration may be determined based on a constraint that the number of registers required to store the tiles of the product matrix and the relevant elements in the input matrix is ​​less than the total number of registers available from the processor. For a tile configuration with a given number of rows, the maximum number of columns may be determined based on an implementation-specific (e.g., off-site / copied or in-site / no-copied) register allocation formula. When performing a matrix multiplication operation, the system may update the product matrix one tile at a time. In response to the detection of remaining rows and / or columns, the disclosed system may select an additional configuration of register tiles and update the remaining rows and columns based on this additional configuration. This approach improves register utilization, a major factor affecting the performance of multiplication of small matrices. In some examples, this solution may be implemented as a computational kernel for matrix multiplication. The computational kernel may include assembly language statements for allocating registers in the processor.

[0068] One aspect of this disclosure provides a system and method for improving register utilization in a processor. During operation, the processor may receive a request to perform a matrix multiplication operation between a first matrix and a second matrix; and for the matrix multiplication operation, determine a set of valid configurations of register tiles based on a number of registers provided by the processor, the corresponding configurations including the number of rows and columns. The number of columns is a function of the number of rows. The processor may then perform the matrix multiplication operation based on the determined set of valid configurations of the register tiles to obtain a third matrix. The matrix multiplication operation may include selecting a configuration of register tiles from the set of valid configurations; updating the third matrix one sub-block at a time, the sub-block including a subset of rows and columns of the third matrix corresponding to the selected configuration of the register tiles; and in response to the third matrix including the remaining rows and columns, selecting an additional configuration of the register tiles and updating the remaining rows and columns based on the additional configuration of the register tiles.

[0069] In a variation of this, the configuration of the select register tile may include selecting the maximum number of rows, determining a set of column numbers corresponding to the maximum number of rows, and selecting the maximum number of columns based on that set of column numbers.

[0070] In a variation of this aspect, additional configurations for selecting register tiles may include sorting the number of rows of valid configurations in descending order; sorting the number of columns corresponding to each row in descending order; and selecting each additional configuration of register tiles based on the sorted number of rows and columns.

[0071] In a variation of this aspect, determining a valid set of register tiles may include determining the number of registers required to store elements in a sub-block and the number of registers required to load elements of a first matrix and a second matrix.

[0072] In a further variation, determining a set of effective configurations for the register tiles may further include identifying the number of rows and columns such that the sum of the number of registers required to store elements in the sub-block and the number of registers required to load elements of the first and second matrices is not greater than the number of registers provided by the processor.

[0073] In a further variation, performing matrix multiplication may include applying an in-place algorithm; and for each valid configuration having the identified number of rows and columns, the number of registers allocated for loading the elements of the first matrix is ​​equal to the number of rows, and the number of registers allocated for loading the elements of the second matrix is ​​equal to the smaller of the number of rows and columns.

[0074] In a further variation, performing matrix multiplication may include applying a remote algorithm; and for each valid configuration with the identified number of rows and columns, the number of registers required to load the elements of the first matrix is ​​equal to twice the number of rows, while the number of registers required to load the elements of the second matrix is ​​equal to the smaller of the number of rows and columns.

[0075] The methods and processes described in the Detailed Description section can be embodied in code and / or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied in data structures and code and stored within the computer-readable storage medium.

[0076] The methods and processes described above may be included in hardware modules or devices. Hardware modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute specific software modules or code at specific times, and other programmable logic devices now known or developed later. When a hardware module or device is activated, it executes the methods and processes included therein.

[0077] The foregoing description is presented to enable any person skilled in the art to make and use the aspects and examples, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of this disclosure. Therefore, the aspects described herein are not limited to those shown, but are intended to be consistent with the maximum scope of the principles and features disclosed herein.

[0078] Furthermore, the foregoing descriptions of the various aspects have been presented solely for illustrative and descriptive purposes. These descriptions are not intended to be exhaustive or to limit the aspects described herein to the disclosed forms. Accordingly, many modifications and variations will be apparent to those skilled in the art. Additionally, the foregoing disclosure is not intended to limit the aspects described herein. The scope of the aspects described herein is defined by the appended claims.

Claims

1. A computer-implemented method for improving register utilization in a processor, the method comprising: The processor receives a request to perform matrix multiplication between the first matrix and the second matrix; For the matrix multiplication operation, a set of valid configurations of register tiles is determined based on the number of registers provided by the processor. The corresponding configurations include the number of rows and the number of columns, wherein the number of columns is a function of the number of rows. as well as The matrix multiplication operation is performed based on a determined set of valid configurations of the register tiles to obtain a third matrix, the matrix multiplication operation including: Select the configuration of the register tile from the set of valid configurations; The third matrix is ​​updated one sub-block at a time, each sub-block comprising a subset of the rows and columns of the third matrix corresponding to the selected configuration of the register tiles; and In response to the third matrix including the remaining rows and columns, an additional configuration of the register tiles is selected, and the remaining rows and columns are updated based on the additional configuration of the register tiles.

2. The computer-implemented method as described in claim 1, wherein, The configuration for selecting the register tile includes: Select the maximum number of rows; Determine a set of column numbers corresponding to the maximum number of rows; and Select the maximum number of columns based on the given set of column numbers.

3. The computer-implemented method as described in claim 1, wherein, The additional configuration for selecting the register tile includes: Sort the number of rows of the valid configurations in descending order; For each row number, sort the column numbers corresponding to that row number in descending order; and Each additional configuration of the register tile is selected based on the sorted number of rows and columns.

4. The computer-implemented method as described in claim 1, wherein, Determining the set of valid configurations for the register tiles includes determining the number of registers required to store elements in the sub-block and the number of registers required to load elements of the first matrix and the second matrix.

5. The computer-implemented method as described in claim 4, wherein, Determining the set of valid configurations for the register tiles further includes: The identification ensures that the sum of the number of registers required to store the elements in the sub-block and the number of registers required to load the elements of the first and second matrices is not greater than the number of rows and columns of registers provided by the processor.

6. The computer-implemented method as described in claim 4, in, Performing the matrix multiplication operation includes applying an in-place algorithm; and For each valid configuration having an identified number of rows and columns, the number of registers allocated for loading elements of the first matrix is ​​equal to the number of rows, and the number of registers allocated for loading elements of the second matrix is ​​equal to the smaller of the number of rows and the number of columns.

7. The computer-implemented method as described in claim 4, in, Performing the matrix multiplication operation includes applying a remote algorithm; and For each valid configuration having an identified number of rows and columns, the number of registers required to load the elements of the first matrix is ​​equal to twice the number of rows, and the number of registers required to load the elements of the second matrix is ​​equal to the smaller of the number of rows and columns.

8. A computer system, comprising: At least one processing resource; as well as A non-transitory computer-readable storage medium storing instructions that, when executed by the processing resource, cause the computer system to perform the following operations: Receive a request to perform matrix multiplication between the first matrix and the second matrix; For the matrix multiplication operation, a set of valid configurations of register tiles is determined based on the number of registers provided by the at least one processing resource, the corresponding configurations including the number of rows and the number of columns, the number of columns being a function of the number of rows; as well as The matrix multiplication operation is performed based on a determined set of valid configurations of the register tiles to obtain a third matrix, the matrix multiplication operation including: Select the configuration of the register tile from the set of valid configurations; The third matrix is ​​updated one sub-block at a time, each sub-block comprising a subset of the rows and columns of the third matrix corresponding to the selected configuration of the register tiles; and In response to the third matrix including the remaining rows and columns, an additional configuration of the register tiles is selected, and the remaining rows and columns are updated based on the additional configuration of the register tiles.

9. The computer system as claimed in claim 8, wherein, The configuration for selecting the register tile includes: Select the maximum number of rows; Determine a set of column numbers corresponding to the maximum number of rows; and Select the maximum number of columns based on the given set of column numbers.

10. The computer system of claim 8, wherein, The additional configuration for selecting the register tile includes: Sort the number of rows of the valid configurations in descending order; For each row number, sort the column numbers corresponding to that row number in descending order; and Each additional configuration of the register tile is selected based on the sorted number of rows and columns.

11. The computer system of claim 8, wherein, Determining the set of valid configurations for the register tiles includes determining the number of registers required to store elements in the sub-block and the number of registers required to load elements of the first matrix and the second matrix.

12. The computer system of claim 11, wherein, Determining the set of valid configurations for the register tiles further includes: The identification ensures that the sum of the number of registers required to store the elements in the sub-block and the number of registers required to load the elements of the first and second matrices is not greater than the number of rows and columns of registers provided by the processor.

13. The computer system as described in claim 11, in, Performing the matrix multiplication operation includes applying an in-place algorithm; and For each valid configuration having an identified number of rows and columns, the number of registers allocated for loading elements of the first matrix is ​​equal to the number of rows, and the number of registers allocated for loading elements of the second matrix is ​​equal to the smaller of the number of rows and the number of columns.

14. The computer system as described in claim 11, in, Performing the matrix multiplication operation includes applying a remote algorithm; and For each valid configuration having the identified number of rows and columns, the number of registers required to load the elements of the first matrix is ​​equal to twice the number of rows, and the number of registers required to load the elements of the second matrix is ​​equal to the smaller of the number of rows and columns.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method comprising: The processor receives a request to perform matrix multiplication between the first matrix and the second matrix; For the matrix multiplication operation, a set of valid configurations of register tiles is determined based on the number of registers provided by the processor. The corresponding configurations include the number of rows and the number of columns, wherein the number of columns is a function of the number of rows. as well as The matrix multiplication operation is performed based on a determined set of valid configurations of the register tiles to obtain a third matrix, the matrix multiplication operation including: Select the configuration of the register tile from the set of valid configurations; The third matrix is ​​updated one sub-block at a time, each sub-block comprising a subset of the rows and columns of the third matrix corresponding to the selected configuration of the register tiles; and In response to the third matrix including the remaining rows and columns, an additional configuration of the register tiles is selected, and the remaining rows and columns are updated based on the additional configuration of the register tiles.

16. The non-transitory computer-readable storage medium as described in claim 15, in, The configuration for selecting the register tile includes: Select the maximum number of rows; Determine a set of column numbers corresponding to the maximum number of rows; and Select the maximum number of columns based on the aforementioned set of column numbers; and The additional configuration for selecting the register tile includes: Sort the number of rows of the valid configurations in descending order; For each row number, sort the column numbers corresponding to that row number in descending order; and Each additional configuration of the register tile is selected one by one based on the sorted row and column numbers.

17. The non-transitory computer-readable storage medium of claim 15, wherein, Determining the set of valid configurations for the register tiles includes determining the number of registers required to store elements in the sub-block and the number of registers required to load elements of the first matrix and the second matrix.

18. The non-transitory computer-readable storage medium of claim 17, wherein, Determining the set of valid configurations for the register tiles further includes: The identification ensures that the sum of the number of registers required to store the elements in the sub-block and the number of registers required to load the elements of the first and second matrices is not greater than the number of rows and columns of registers provided by the processor.

19. The non-transitory computer-readable storage medium of claim 17, in, Performing the matrix multiplication operation includes applying an in-place algorithm; and For each valid configuration having an identified number of rows and columns, the number of registers allocated for loading elements of the first matrix is ​​equal to the number of rows, and the number of registers allocated for loading elements of the second matrix is ​​equal to the smaller of the number of rows and the number of columns.

20. The non-transitory computer-readable storage medium of claim 17, in, Performing the matrix multiplication operation includes applying a remote algorithm; and For each valid configuration having the identified number of rows and columns, the number of registers required to load the elements of the first matrix is ​​equal to twice the number of rows, and the number of registers required to load the elements of the second matrix is ​​equal to the smaller of the number of rows and columns.

Images