Gpu-oriented batch variable-size matrix multiplication adaptive computing method

Abstract

The application relates to a GPU-oriented adaptive computing method for batch variable-size matrix multiplication. The method comprises the following steps: receiving a batch variable-size matrix multiplication problem, obtaining size information of each matrix calculation instance in a batch, determining a target Tile division scheme from a plurality of preset Tile division schemes for each matrix calculation instance based on the size information, and sorting the execution sequence of the matrix calculation instances in the batch to improve the load balancing of a GPU computing unit. By traversing fine-grained Tile division schemes, combining effective work item proportion and work group utilization rate threshold to filter a target scheme, and dynamically adjusting Tile division in the case of insufficient utilization, the method ensures that as many work groups as possible become effective work groups, and solves the problem that a large number of threads are idle and computing units are idling due to irregular sizes.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an adaptive computation method, apparatus, computer device, storage medium, and computer program product for batch variable-size matrix multiplication on GPUs. Background Technology

[0002] With the rapid development of deep learning and high-performance parallel computing, GPU-accelerated batch matrix multiplication is increasingly widely used in image processing, scientific simulation, and recommender systems. As a core operation in these applications, batch matrix multiplication has expanded from fixed-size regularization computation to variable-size general matrix multiplication, enabling algorithms to flexibly handle data batches of different sizes. However, in practical applications, when faced with a batch of matrix multiplication instances of varying sizes, problems exist such as low GPU computing unit utilization and uneven load distribution among computing units. Especially in scenarios with extreme changes in matrix size or misaligned boundaries, traditional kernel scheduling methods struggle to meet the demands for efficient and high-throughput computation.

[0003] In related technologies, GPU kernels typically use a fixed tile size partitioning strategy to process all instances uniformly. This approach lacks the ability to adapt to irregular sizes. Once a boundary situation is encountered where the matrix dimension cannot be divided by the tile, it will lead to a large number of threads being idle and computing units running idle, resulting in the GPU computing resources not being fully utilized. At the same time, traditional methods do not optimize the execution order of batch instances. When instances with different computational loads are executed together, it is easy to have an uneven workload situation where some computing units are busy while others are waiting, which limits the overall parallel efficiency and throughput of batch matrix multiplication. Summary of the Invention

[0004] Therefore, it is necessary to provide an adaptive computation method, apparatus, computer device, computer-readable storage medium, and computer program product for batch variable-size matrix multiplication for GPUs to address the above-mentioned technical problems.

[0005] Firstly, this application provides an adaptive computation method for batch variable-size matrix multiplication on GPUs, including:

[0006] Receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and for each matrix calculation instance, determine the target tile partitioning scheme from multiple preset tile partitioning schemes based on the size information;

[0007] The execution order of matrix computation instances in the batch is sorted to improve load balancing of GPU computing units;

[0008] Based on the size information of the matrix computation instance and the target tile partitioning scheme, the utilization rate of the GPU workgroup is evaluated, and when the evaluation results meet the preset conditions, the tile partitioning scheme of at least some matrix computation instances is adjusted to increase the number of effective workgroups.

[0009] According to the adjusted execution order and the target tile partitioning scheme of each instance, the GPU kernel function is started to perform parallel calculations on the matrix calculation instances in the batch, and the calculation results are sent back to the host memory.

[0010] In one embodiment, for each matrix calculation instance, a target tile partitioning scheme is determined from a set of preset tile partitioning schemes based on size information, including:

[0011] Obtain the dimensions of the matrix computation instance, including the M, N, and K dimension values ​​of the matrix multiplication;

[0012] Iterate through each set of (Tm, Tn, Tk, workitems) parameter combinations in the preset set of fine-grained tile partitioning schemes, where Tm and Tn have values ​​ranging from {16, 32, 64}, Tk has values ​​ranging from {4, 8, 16}, and workitems has values ​​ranging from {128, 256}.

[0013] For each set of parameters, calculate the number of tiles along dimension M, Pm = ceil(M / Tm), and the number of tiles along dimension N, Pn = ceil(N / Tn), and use Pm × Pn as the total number of scheduling workgroups;

[0014] For tiles located at the M-dimensional and N-dimensional boundaries, the number of valid work items in each work group is determined based on the relationship between the remaining size and Tm and Tn. Work groups whose percentage of valid work items exceeds a preset threshold are included in the number of valid work groups.

[0015] Calculate the workgroup utilization rate under this parameter combination, where the workgroup utilization rate is the ratio of the number of effective workgroups to the total number of scheduled workgroups;

[0016] The parameter combination that satisfies the condition that the utilization rate of the workgroup is greater than the first threshold and the number of effective workgroups is the largest, is selected as the target Tile partitioning scheme; if multiple combinations satisfy the condition, the combination with the smallest value of the workitems parameter is selected; if multiple combinations still exist, the combination with the largest product of the Tm and Tn parameter values ​​is selected.

[0017] In one embodiment, the execution order of matrix computation instances in the batch is sorted to improve load balancing of GPU computing units, including:

[0018] Obtain the K-dimensional value Kᵢ of each matrix calculation instance, as well as Tmᵢ and Tnᵢ in the target Tile partitioning scheme corresponding to that instance;

[0019] The matrix calculation instances in the batch are sorted in two rounds based on the Kᵢ value and the Tmᵢ×Tnᵢ value to obtain the adjusted execution order. The two rounds of sorting prioritize the execution of instances with larger computational loads and postpone the execution of instances with smaller Tile size product values.

[0020] In one embodiment, the two rounds of sorting include:

[0021] The first round of sorting uses the Kᵢ value of each matrix calculation instance as the sorting key to sort in descending order, with a time complexity of O(Blog B), where B is the number of matrix calculation instances in the batch.

[0022] In the second round of sorting, the Tmᵢ×Tnᵢ values ​​of each matrix instance are used as the sorting key to sort the instance sequence obtained after the first round of sorting in descending order. The second round of sorting uses the radix sort algorithm, with a time complexity of O(B).

[0023] In one embodiment, based on the size information of the matrix computation instance and the target tile partitioning scheme, the GPU workgroup utilization is evaluated, and when the evaluation result meets preset conditions, the tile partitioning scheme of at least some matrix computation instances is adjusted to increase the number of effective workgroups, including:

[0024] Obtain the dimensions of each matrix calculation instance in the batch, including the M-dimensional value Mᵢ and the N-dimensional value Nᵢ, as well as Tmᵢ and Tnᵢ in the target Tile partitioning scheme determined for each instance;

[0025] The parameter θ is calculated and determined using the following formula:

[0026] θ = c·μ·N_CU / Σᵢ ( (Mᵢ × Nᵢ) / (Tmᵢ × Tnᵢ) )

[0027] Where c is a constant related to the number of workitems and the SIMD structure of the computing units in a single workgroup, μ is a hardware-related parallelism factor greater than 1, N_CU is the total number of computing units in the target GPU, and Σᵢ represents the summation of all matrix computing instances in the batch.

[0028] Compare θ with 1: When θ ≤ 1, it is determined that the total number of scheduling workgroups for each instance is sufficient to fully utilize the GPU computing units, the workgroup utilization meets the preset conditions, and the target Tile partitioning scheme for each instance remains unchanged; when θ > 1, it is determined that the workgroup utilization is insufficient, the number of effective workgroups is increased, and the Tile partitioning scheme for at least some instances is adjusted.

[0029] In one embodiment, the tile partitioning scheme for at least some instances is adjusted, including:

[0030] The instance sequence obtained after being sorted according to the execution order is traversed from back to front. Instances with smaller Tile size product Tmᵢ×Tnᵢ and smaller K dimension value Kᵢ are selected first. Their current target Tile partitioning scheme is replaced with a set of parameter combinations with smaller Tile size in the preset Tile partitioning scheme set. The judgment parameter θ is recalculated after each replacement until θ ≤ 1, thus completing the adaptive adjustment to extreme data input.

[0031] Secondly, this application also provides an adaptive computing device for batch variable-size matrix multiplication for GPUs, comprising:

[0032] The target partitioning module is used to receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and determine the target tile partitioning scheme from multiple preset tile partitioning schemes for each matrix calculation instance based on the size information.

[0033] An execution sorting module is used to sort the execution order of matrix calculation instances in the batch in order to improve the load balancing of GPU computing units;

[0034] The scheme adjustment module is used to evaluate the GPU workgroup utilization based on the size information of the matrix calculation instance and the target Tile partitioning scheme, and when the evaluation result meets the preset conditions, adjust the Tile partitioning scheme of at least some matrix calculation instances to increase the number of effective workgroups.

[0035] The instance computing module is used to start the GPU kernel function to perform parallel computing on the matrix computing instances in the batch according to the adjusted execution order and the target tile partitioning scheme of each instance, and to send the computing results back to the host memory.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0037] Receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and for each matrix calculation instance, determine the target tile partitioning scheme from multiple preset tile partitioning schemes based on the size information;

[0038] The execution order of matrix computation instances in the batch is sorted to improve load balancing of GPU computing units;

[0039] Based on the size information of the matrix computation instance and the target tile partitioning scheme, the utilization rate of the GPU workgroup is evaluated, and when the evaluation results meet the preset conditions, the tile partitioning scheme of at least some matrix computation instances is adjusted to increase the number of effective workgroups.

[0040] According to the adjusted execution order and the target tile partitioning scheme of each instance, the GPU kernel function is started to perform parallel calculations on the matrix calculation instances in the batch, and the calculation results are sent back to the host memory.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0042] Receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and for each matrix calculation instance, determine the target tile partitioning scheme from multiple preset tile partitioning schemes based on the size information;

[0043] The execution order of matrix computation instances in the batch is sorted to improve load balancing of GPU computing units;

[0044] Based on the size information of the matrix computation instance and the target tile partitioning scheme, the utilization rate of the GPU workgroup is evaluated, and when the evaluation results meet the preset conditions, the tile partitioning scheme of at least some matrix computation instances is adjusted to increase the number of effective workgroups.

[0045] According to the adjusted execution order and the target tile partitioning scheme of each instance, the GPU kernel function is started to perform parallel calculations on the matrix calculation instances in the batch, and the calculation results are sent back to the host memory.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0047] Receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and for each matrix calculation instance, determine the target tile partitioning scheme from multiple preset tile partitioning schemes based on the size information;

[0048] The execution order of matrix computation instances in the batch is sorted to improve load balancing of GPU computing units;

[0049] Based on the size information of the matrix computation instance and the target tile partitioning scheme, the utilization rate of the GPU workgroup is evaluated, and when the evaluation results meet the preset conditions, the tile partitioning scheme of at least some matrix computation instances is adjusted to increase the number of effective workgroups.

[0050] According to the adjusted execution order and the target tile partitioning scheme of each instance, the GPU kernel function is started to perform parallel calculations on the matrix calculation instances in the batch, and the calculation results are sent back to the host memory.

[0051] The aforementioned adaptive computation method, apparatus, computer device, storage medium, and computer program products for GPU-based batch variable-size matrix multiplication utilize a fine-grained tile partitioning scheme. This scheme filters target schemes by traversing fine-grained tile partitioning schemes, combining the proportion of effective work items and work group utilization thresholds, and dynamically adjusts tile partitioning for cases of insufficient utilization. This ensures that as many work groups as possible become effective work groups, resolving the problem of numerous idle threads and computational units running idle due to irregular sizes, thus enabling more efficient utilization of GPU computing resources. Subsequently, based on the computational load and tile granularity of each matrix instance, a two-round sorting process is performed: first, K-value descending order; then, Tm×Tn descending order followed by Tm. This prioritizes the execution of computationally intensive instances to release resources early, while delaying the execution of instances with small tile products to smooth out tail loads. This improves the uneven workload distribution among computational units when tasks of different sizes are mixed, thereby enhancing overall parallel efficiency and throughput. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating an adaptive computation method for batch variable-size matrix multiplication for GPUs in one embodiment.

[0054] Figure 2 This is a block diagram of a GPU-oriented batch variable-size matrix multiplication adaptive computing device in one embodiment.

[0055] Figure 3 This is an internal structural diagram of a computer device in one embodiment;

[0056] Figure 4 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In one exemplary embodiment, such as Figure 1 As shown, an adaptive computation method for batch variable-size matrix multiplication for GPUs is provided, including:

[0059] Step S102: Receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and for each matrix calculation instance, determine the target tile partitioning scheme from multiple preset tile partitioning schemes based on the size information.

[0060] The batch variable-size matrix multiplication problem refers to a group of matrix multiplication operations to be processed in a single GPU kernel call. The matrix dimensions of each multiplication instance in this group of operations are not completely the same, belonging to mixed-size batches. The size information of the matrix computation instance refers to the matrix size parameters of each independent multiplication instance in the batch, including the number of rows in M ​​dimensions, the inner product dimension in K dimensions, and the number of columns in N dimensions. The preset multiple tile partitioning schemes refer to a set of thread block partitioning strategies pre-defined for different combinations of matrix shapes and sizes. Each scheme defines the partitioning granularity and computational resource mapping method along the M and N dimensions to match different parallelism and memory access efficiency requirements. The target tile partitioning scheme refers to the single partitioning scheme suitable for the current matrix computation instance selected from the preset scheme set by rule matching or table lookup based on the size information of the current matrix computation instance.

[0061] For example, the GPU runtime receives a batch of matrix multiplication requests of mixed sizes, containing several multiplication instances with varying M, K, and N dimensions. The runtime iterates through the batch, extracting the M, K, and N dimension information for each instance. For instance with small M and N dimensions and aligned dimensions, the runtime matches a first-type tile scheme optimized for small-scale aligned matrices from a pre-defined scheme table. This scheme uses smaller block granularity along both the M and N dimensions to improve thread utilization. Another instance has a larger M dimension and is unaligned; the runtime matches a second-type tile scheme optimized for large M-dimensional unaligned matrices. This scheme uses larger granularity blocks along the M dimension to reduce boundary fragmentation overhead. After matching schemes for all instances, the runtime distributes the target tile partitioning scheme identifier bound to each instance to the subsequent block execution phase.

[0062] For example, the dimensions of the matrix calculation instance are obtained, including the M, N, and K dimension values ​​of the matrix multiplication; each set of (Tm, Tn, Tk, workitems) parameter combinations in the preset fine-grained tile partitioning scheme set is traversed, where Tm and Tn range from {16, 32, 64}, Tk ranges from {4, 8, 16}, and workitems ranges from {128, ... ... 256}; For each parameter combination, calculate the number of tiles along the M dimension Pm = ceil(M / Tm) and the number of tiles along the N dimension Pn = ceil(N / Tn), and use Pm × Pn as the total number of scheduled workgroups; For tiles located at the boundaries of the M and N dimensions, determine the number of effective work items in each workgroup based on the relationship between the remaining size and Tm and Tn, and include workgroups whose effective work item ratio exceeds a preset threshold in the number of effective workgroups; calculate the workgroup utilization rate under this parameter combination, which is the ratio of the number of effective workgroups to the total number of scheduled workgroups; select the parameter combination that satisfies the workgroup utilization rate being greater than the first threshold and has the largest number of effective workgroups as the target tile partitioning scheme; if multiple combinations satisfy the condition, select the combination with the smallest workitems parameter value; if multiple combinations still exist, select the combination with the largest product of Tm and Tn parameter values.

[0063] In this embodiment, the size of a matrix calculation instance is obtained at runtime as M=100, N=100, K=128. The runtime iterates through the preset scheme set, taking a parameter combination Tm=32, Tn=32, Tk=8, and workitems=256 as an example for calculation. The runtime calculates Pm=ceil(100 / 32)=4, Pn=ceil(100 / 32)=4, and the total number of scheduled workgroups is 4×4=16. The runtime checks the four workgroups at the M-dimensional boundary with i=3. Each workgroup has a remaining M-dimensional size of 100-3×32=4, and the effective work item ratio is 4 / 32=12.5%, which is lower than the preset threshold. These four workgroups are not counted as effective workgroups. Similarly, the four workgroups at the N-dimensional boundary with j=3 are also removed. The remaining nine workgroups are internally complete workgroups with an effective work item ratio of 100%, and are counted as effective workgroups, resulting in a total of 9 effective workgroups. The runtime calculation shows a workgroup utilization rate of 9 / 16 = 56.25%, which is below the first threshold, so this parameter combination is excluded. The runtime continues to iterate. When it reaches Tm = 16, Tn = 16, Tk = 8, and workitems = 256, Pm = ceil(100 / 16) = 7 and Pn = ceil(100 / 16) = 7, the total number of scheduled workgroups is 49. The remaining size of the M-dimensional boundary is 4, and the same applies to the N-dimensional boundary. The effective work item percentage of the boundary workgroups is 25%, still below the threshold and therefore excluded. There are 5 × 5 = 25 complete internal workgroups, with a workgroup utilization rate of approximately 51%, which are also excluded. At runtime, the parameter combination Tm=64, Tn=64, Tk=8, and workitems=128 was ultimately matched. Under this combination, Pm=2 and Pn=2, resulting in a total of 4 scheduling workgroups. Only the M-dimensional and N-dimensional boundaries of the workgroups showed partial filling. The percentage of valid work items exceeded the threshold, and all four workgroups were counted as valid workgroups. The workgroup utilization rate reached 100%, and all screening priority conditions were met. Therefore, this was selected as the target tile partitioning scheme.

[0064] Step S104: Sort the execution order of matrix computation instances in the batch to improve the load balancing of GPU computing units.

[0065] Among them, execution order sorting refers to rearranging the scheduling order of each matrix computing instance within the batch so that the total computing load borne by the GPU computing unit during execution is distributed as evenly as possible, avoiding idle computing resources caused by some computing units completing first while other computing units are still processing large-size instances; load balancing is based on the number of workgroups or floating-point operations required by each computing instance.

[0066] For example, after completing the tile partitioning of all instances, the runtime counts the total number of scheduling workgroups for each instance, sorts them in descending order of the number of workgroups, and places the instance with the largest number of workgroups at the head of the batch execution queue, the second largest at the next head, and so on. After the GPU starts execution, the larger instances with more workgroups occupy all computing units first, and then the smaller instances fill the computing resources released by the gradual completion of the larger instances in turn, minimizing the idle time of each computing unit during the entire batch execution and improving the overall load balancing.

[0067] For example, the K-dimensional value Kᵢ of each matrix calculation instance is obtained, as well as Tmᵢ and Tnᵢ in the target Tile partitioning scheme corresponding to the instance; the matrix calculation instances in the batch are sorted in two rounds based on the Kᵢ value and the Tmᵢ×Tnᵢ value to obtain the adjusted execution order; the two rounds of sorting make the instances with larger computational loads execute first, and make the instances with smaller Tile size product values ​​execute later.

[0068] In this embodiment, the runtime obtains the K value and Tile size of the three matrix computation instances in the batch: instance A has a K value of 256 and Tm×Tn of 1024, instance B has a K value of 128 and Tm×Tn of 2048, and instance C has a K value of 256 and Tm×Tn of 512. At runtime, they are first sorted in descending order of K value, with instances A and C at the front due to their largest K values, and instance B at the back. Between instances A and C with the same K value, they are then sorted in ascending order of Tm×Tn value, with instance C, having a smaller Tm×Tn, moving forward and instance A relatively moving backward. The final adjusted execution order is instance C, instance A, and instance B, ensuring that the instance with the highest computational load prioritizes GPU computing resources while preventing large-tile-size instances from prematurely blocking the scheduling of subsequent smaller-size instances.

[0069] Furthermore, exemplarily, the two-round sorting includes: a first round of sorting, which sorts the matrix computation instances in descending order using the Kᵢ value of each instance as the sorting key, with a time complexity of O(Blog B), where B is the number of matrix computation instances in the batch; and a second round of sorting, which sorts the instance sequence obtained after the first round of sorting in descending order using the Tmᵢ×Tnᵢ value of each instance as the sorting key, and the second round of sorting uses the radix sort algorithm, with a time complexity of O(B).

[0070] In this embodiment, four matrix calculation instances are obtained during runtime: instance A has a K value of 256 and a Tm×Tn value of 1024; instance B has a K value of 128 and a Tm×Tn value of 2048; instance C has a K value of 256 and a Tm×Tn value of 512; and instance D has a K value of 64 and a Tm×Tn value of 4096. The first round of sorting arranges the instances in descending order of K value, resulting in instances A and C tied at the beginning, instance B in the middle, and instance D at the end. The second round of sorting uses the Tm×Tn value as the key, performing radix sort on the results of the first round. Between instances A and C, both with a K value of 256, instance C, with its smaller Tm×Tn value, moves forward, while instance A moves backward; the remaining instances do not undergo positional swaps across K value intervals due to their different K values. The final adjusted execution order is instance C, instance A, instance B, and instance D.

[0071] Step S106: Based on the size information of the matrix computation instance and the target Tile partitioning scheme, evaluate the GPU workgroup utilization, and when the evaluation results meet the preset conditions, adjust the Tile partitioning scheme of at least some matrix computation instances to increase the number of effective workgroups.

[0072] The utilization rate of a workgroup refers to the ratio of the number of effective workgroups to the total number of scheduled workgroups, which has been calculated in the Tile partitioning scheme selection process in step S102. The preset condition refers to the utilization rate of a workgroup being lower than a preset second threshold. This second threshold can be set as another judgment threshold that is different from the first threshold in step S102. It is used to further identify instances where there are still too many boundary fragments after the initial selection of the Tile scheme. Adjusting the Tile partitioning scheme for at least some matrix calculation instances means that for instances where the utilization rate of a workgroup does not meet the preset condition, based on the currently selected target Tile partitioning scheme, the block size of the Tile at the M-dimensional or N-dimensional boundary is reduced, or the boundary Tile is switched to a preset spare Tile size with a smaller granularity, so as to reduce the proportion of invalid work items in the boundary Tile, thereby converting some of the workgroups that were originally judged to be invalid into effective workgroups.

[0073] For example, after the runtime completes the tile partitioning for a batch of matrix calculation instances, the utilization rate of the workgroups for each instance is reviewed one by one. For an instance with M=100, N=100, K=128, the currently selected target tile scheme is Tm=64, Tn=64, Pm=2, Pn=2, and the total number of scheduled workgroups is 4. Among them, the effective work items of the two workgroups at the M-dimensional boundary account for 36 / 64=56%, and the same applies to the two groups at the N-dimensional boundary. Of the four groups, only the one with internal overlapping area is a complete workgroup. The effective work item percentage of the other three groups is lower than the preset second threshold, and there is only 1 effective workgroup. The workgroup utilization rate is 25%. At runtime, if the instance's workgroup utilization does not meet the preset conditions, an adjustment process is triggered: the tile size at the M-dimensional boundary is reduced from 64 to 32, and the tile size at the N-dimensional boundary is reduced from 64 to 32. After the adjustment, the percentage of effective work items in the boundary workgroups increases to 100%, all three previously removed groups are restored to effective workgroups, the number of effective workgroups increases from 1 to 4, and the workgroup utilization increases from 25% to 100%.

[0074] For example, the dimensions of each matrix computation instance in the batch are obtained, including the M-dimensional value Mᵢ and the N-dimensional value Nᵢ, as well as Tmᵢ and Tnᵢ in the target tile partitioning scheme determined for each instance; the judgment parameter θ is calculated according to the following formula: θ = c·μ·N_CU / Σᵢ ( (Mᵢ × Nᵢ) / (Tmᵢ × Tnᵢ) ); where c is a constant related to the number of workitems contained in a single workgroup and the SIMD structure of the computing unit, μ is a hardware-related parallelism factor greater than 1, N_CU is the total number of computing units in the target GPU, and Σᵢ represents the summation over all matrix computation instances in the batch; θ is compared with 1: when θ ≤ 1, it is determined that the total number of scheduled workgroups for each instance is sufficient to fully occupy the GPU computing units, the workgroup utilization meets the preset conditions, and the target tile partitioning scheme for each instance remains unchanged; when θ > 1, it is determined that the workgroup utilization is insufficient, the number of effective workgroups is increased, and the tile partitioning scheme for at least some instances is adjusted.

[0075] In this embodiment, the runtime obtains the M-dimensional and N-dimensional dimensions of the three matrix computation instances in the batch, as well as the selected Tm and Tn values: Instance A has an M×N of 256×256, a Tm×Tn of 64×64, and a working group size of 16; Instance B has an M×N of 128×128, a Tm×Tn of 32×32, and a working group size of 16; Instance C has an M×N of 64×64, a Tm×Tn of 16×16, and a working group size of 16. The total is 48. The target GPU's N_CU is 80, the constant c is 1, and the parallelism factor μ is 2. Calculating θ = c·μ·N_CU / 48 = 1×2×80 / 48≈3.33, which is greater than 1, indicates that the current number of working groups is insufficient to fully utilize the GPU's computing units. During runtime, instance A's tile size was adjusted from 64×64 to 32×32, the number of workgroups increased from 16 to 64, and the total increased to 96. θ was recalculated as 160 / 96≈1.67, which is still greater than 1. Instance B was then adjusted to 16×16, increasing the total to 144, and θ was 160 / 144≈1.11, still slightly greater than 1. Instance C was adjusted to 8×8, increasing the total to 208, and θ was 160 / 208≈0.77, decreasing to below 1. Therefore, the sufficient occupancy condition was met, and the adjustment was terminated.

[0076] Furthermore, exemplarily, the tile partitioning scheme of at least some instances is adjusted, including: traversing the instance sequence obtained after sorting by execution order from back to front, prioritizing the selection of instances with smaller tile size product Tmᵢ×Tnᵢ and smaller K dimension value Kᵢ, replacing their current target tile partitioning scheme with a set of parameter combinations with smaller tile size in the preset tile partitioning scheme set, recalculating the judgment parameter θ after each replacement, until θ ≤ 1, thus completing the adaptive adjustment to extreme data input.

[0077] In this embodiment, the runtime traverses the sorted instance sequence from back to front, with instance D at the end of the sequence. Instance D has a small K value and a small Tm×Tn product. The runtime adjusts instance D's tile scheme from 16×16 to 8×8, increasing the number of workgroups to four times the original. After recalculating θ, it is still greater than 1. The runtime continues to select instance C, similarly performing a tile size reduction adjustment. After recalculating θ, it is still greater than 1. The runtime then performs adjustments on instance B sequentially, recalculating θ after each adjustment. When the adjustment of instance B is completed and θ drops below 1, the adjustment process terminates. Instance A, located at the beginning of the sequence, is not selected for adjustment due to its large K value and large Tm×Tn product; its original tile scheme is retained to ensure the scheduling efficiency of computationally intensive instances.

[0078] Step S108: According to the adjusted execution order and the target tile partitioning scheme of each instance, start the GPU kernel function to perform parallel calculations on the matrix calculation instances in the batch, and return the calculation results to the host memory.

[0079] The adjusted execution order refers to the scheduling order of each matrix calculation instance within the batch, determined after two rounds of sorting in step S104; the target tile partitioning scheme for each instance refers to the combination of block parameters for each instance, initially selected in step S102 and potentially adjusted in step S106; the GPU kernel function refers to the matrix multiplication kernel function executed on the device side, which can dynamically adjust the block calculation logic according to the independent tile parameters of each instance; parallel computing refers to starting the kernels of each instance sequentially or concurrently according to the execution order, and mapping the calculation tasks to the corresponding number of workgroups according to its tile partitioning scheme, which are then executed in parallel by the GPU computing units; and the calculation result feedback refers to the output matrix of each instance being fed back from the GPU global memory to the host system memory through an asynchronous data transmission interface after the calculation is completed on the device side.

[0080] For example, the runtime packages the matrix data pointers, dimension parameters, and tile partitioning parameters of each instance into kernel startup parameters according to the adjusted execution order, and starts the matrix multiplication kernel instance by instance. Taking the first instance C as an example, its tile scheme is Tm=32, Tn=32, Tk=8. The runtime divides its output matrix into a corresponding number of workgroup grids, with each workgroup responsible for the calculation of one output tile. The GPU scheduler allocates workgroups to various computing units for parallel execution, and the work items within each workgroup cooperate to complete the multiplication and accumulation operation of that tile. After the kernel of instance C is completed, the runtime immediately starts the kernel of the next instance, and asynchronously sends back the output matrix of instance C to the host memory. After all instances have been calculated, the runtime inserts a synchronization point to ensure that all asynchronous backhaul operations are completed, and the host obtains all matrix multiplication results in the batch.

[0081] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0082] Based on the same inventive concept, this application also provides a GPU-oriented batch variable-size matrix multiplication adaptive computing device for implementing the aforementioned GPU-oriented batch variable-size matrix multiplication adaptive computing method. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more GPU-oriented batch variable-size matrix multiplication adaptive computing device embodiments provided below can be found in the limitations of the GPU-oriented batch variable-size matrix multiplication adaptive computing method described above, and will not be repeated here.

[0083] In one exemplary embodiment, such as Figure 2 As shown, a batch variable-size matrix multiplication adaptive computing device for GPUs is provided, comprising:

[0084] The target partitioning module 202 is used to receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and determine the target tile partitioning scheme from multiple preset tile partitioning schemes based on the size information for each matrix calculation instance.

[0085] The sorting module 204 is used to sort the execution order of matrix computation instances in a batch in order to improve the load balancing of GPU computing units.

[0086] The scheme adjustment module 206 is used to evaluate the GPU workgroup utilization based on the size information of the matrix computing instance and the target Tile partitioning scheme, and when the evaluation result meets the preset conditions, adjust the Tile partitioning scheme of at least some matrix computing instances to increase the number of effective workgroups.

[0087] The instance computing module 208 is used to start the GPU kernel function to perform parallel computing on the matrix computing instances in the batch according to the adjusted execution order and the target tile partitioning scheme of each instance, and to send the computing results back to the host memory.

[0088] The modules in the aforementioned GPU-oriented batch variable-size matrix multiplication adaptive computing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in a computer device, or stored in software within the computer device's memory, so that the processor can invoke and execute the operations corresponding to each module.

[0089] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data collected by the terminal. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements an adaptive computation method for batch variable-size matrix multiplication targeting GPUs.

[0090] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a GPU-oriented batch variable-size matrix multiplication adaptive computation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0091] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0092] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: receiving a batch of variable-size matrix multiplication problems; obtaining size information of each matrix computation instance in the batch; for each matrix computation instance, determining a target tile partitioning scheme from a plurality of preset tile partitioning schemes based on the size information; sorting the execution order of the matrix computation instances in the batch to improve the load balancing of GPU computing units; evaluating the utilization rate of GPU workgroups according to the size information of the matrix computation instances and the target tile partitioning scheme, and adjusting the tile partitioning scheme of at least some matrix computation instances when the evaluation result meets preset conditions to increase the number of effective workgroups; starting GPU kernel functions to perform parallel computation on the matrix computation instances in the batch according to the adjusted execution order and the target tile partitioning scheme of each instance, and returning the computation results to host memory.

[0093] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0094] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An adaptive computation method for batch variable-size matrix multiplication for GPUs, characterized in that, The method includes: Receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and for each matrix calculation instance, determine the target tile partitioning scheme from multiple preset tile partitioning schemes based on the size information; The execution order of matrix computation instances in the batch is sorted to improve load balancing of GPU computing units; Based on the size information of the matrix computation instance and the target tile partitioning scheme, the utilization rate of the GPU workgroup is evaluated, and when the evaluation results meet the preset conditions, the tile partitioning scheme of at least some matrix computation instances is adjusted to increase the number of effective workgroups. According to the adjusted execution order and the target tile partitioning scheme of each instance, the GPU kernel function is started to perform parallel calculations on the matrix calculation instances in the batch, and the calculation results are sent back to the host memory.

2. The method according to claim 1, characterized in that, For each matrix calculation instance, a target tile partitioning scheme is determined from a set of preset tile partitioning schemes based on size information, including: Obtain the dimensions of the matrix computation instance, including the M, N, and K dimension values ​​of the matrix multiplication; Iterate through each set of (Tm, Tn, Tk, workitems) parameter combinations in the preset set of fine-grained tile partitioning schemes, where Tm and Tn have values ​​ranging from {16, 32, 64}, Tk has values ​​ranging from {4, 8, 16}, and workitems has values ​​ranging from {128, 256}. For each set of parameters, calculate the number of tiles along dimension M, Pm = ceil(M / Tm), and the number of tiles along dimension N, Pn = ceil(N / Tn), and use Pm × Pn as the total number of scheduling workgroups; For tiles located at the M-dimensional and N-dimensional boundaries, the number of valid work items in each work group is determined based on the relationship between the remaining size and Tm and Tn. Work groups whose percentage of valid work items exceeds a preset threshold are included in the number of valid work groups. Calculate the workgroup utilization rate under this parameter combination, where the workgroup utilization rate is the ratio of the number of effective workgroups to the total number of scheduled workgroups; The parameter combination that satisfies the condition that the utilization rate of the workgroup is greater than the first threshold and the number of effective workgroups is the largest, is selected as the target Tile partitioning scheme; if multiple combinations satisfy the condition, the combination with the smallest value of the workitems parameter is selected; if multiple combinations still exist, the combination with the largest product of the Tm and Tn parameter values ​​is selected.

3. The method according to claim 1, characterized in that, Sort the execution order of matrix computation instances in the batch to improve load balancing of GPU computing units, including: Obtain the K-dimensional value Kᵢ of each matrix calculation instance, as well as Tmᵢ and Tnᵢ in the target Tile partitioning scheme corresponding to that instance; The matrix calculation instances in the batch are sorted in two rounds based on the Kᵢ value and the Tmᵢ×Tnᵢ value to obtain the adjusted execution order. The two rounds of sorting prioritize the execution of instances with larger computational loads and postpone the execution of instances with smaller Tile size product values.

4. The method according to claim 3, characterized in that, The two rounds of sorting include: The first round of sorting uses the Kᵢ value of each matrix calculation instance as the sorting key to sort in descending order, with a time complexity of O(BlogB), where B is the number of matrix calculation instances in the batch. In the second round of sorting, the Tmᵢ×Tnᵢ values ​​of each matrix instance are used as the sorting key to sort the instance sequence obtained after the first round of sorting in descending order. The second round of sorting uses the radix sort algorithm, with a time complexity of O(B).

5. The method according to claim 1, characterized in that, Based on the size information of the matrix computation instance and the target tile partitioning scheme, the GPU workgroup utilization is evaluated. When the evaluation results meet preset conditions, the tile partitioning scheme of at least some matrix computation instances is adjusted to increase the number of effective workgroups, including: Obtain the dimensions of each matrix calculation instance in the batch, including the M-dimensional value Mᵢ and the N-dimensional value Nᵢ, as well as Tmᵢ and Tnᵢ in the target Tile partitioning scheme determined for each instance; The parameter θ is calculated and determined using the following formula: θ = c·μ·N_CU / Σᵢ ( (Mᵢ × Nᵢ) / (Tmᵢ × Tnᵢ) ) Where c is a constant related to the number of workitems and the SIMD structure of the computing units in a single workgroup, μ is a hardware-related parallelism factor greater than 1, N_CU is the total number of computing units in the target GPU, and Σᵢ represents the summation of all matrix computing instances in the batch. Compare θ with 1: When θ ≤ 1, it is determined that the total number of scheduling workgroups for each instance is sufficient to fully utilize the GPU computing units, the workgroup utilization meets the preset conditions, and the target Tile partitioning scheme for each instance remains unchanged; when θ > 1, it is determined that the workgroup utilization is insufficient, the number of effective workgroups is increased, and the Tile partitioning scheme for at least some instances is adjusted.

6. The method according to claim 5, characterized in that, Adjustments were made to the tile partitioning scheme for at least some instances, including: The instance sequence obtained after being sorted according to the execution order is traversed from back to front. Instances with smaller Tile size product Tmᵢ×Tnᵢ and smaller K dimension value Kᵢ are selected first. Their current target Tile partitioning scheme is replaced with a set of parameter combinations with smaller Tile size in the preset Tile partitioning scheme set. The judgment parameter θ is recalculated after each replacement until θ ≤ 1, thus completing the adaptive adjustment to extreme data input.

7. An adaptive computing device for batch variable-size matrix multiplication for GPUs, characterized in that, The device includes: The target partitioning module is used to receive batch variable-size matrix multiplication problems, obtain the size information of each matrix calculation instance in the batch, and determine the target tile partitioning scheme from multiple preset tile partitioning schemes for each matrix calculation instance based on the size information. An execution sorting module is used to sort the execution order of matrix calculation instances in the batch in order to improve the load balancing of GPU computing units; The scheme adjustment module is used to evaluate the GPU workgroup utilization based on the size information of the matrix calculation instance and the target Tile partitioning scheme, and when the evaluation result meets the preset conditions, adjust the Tile partitioning scheme of at least some matrix calculation instances to increase the number of effective workgroups. The instance computing module is used to start the GPU kernel function to perform parallel computing on the matrix computing instances in the batch according to the adjusted execution order and the target tile partitioning scheme of each instance, and to send the computing results back to the host memory.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images