TCU and CUDA core hybrid computing task scheduling method and device

Abstract

The application provides a TCU and CUDA core hybrid computing task scheduling method and device, the method comprising: determining the value of the theoretical execution time ratio according to the ratio between the estimated execution time of the CUDA core and the estimated execution time of the TCU; the CUDA core is used to perform matrix multiplication calculation on the CUDA core computing task in each computing task; the TCU is used to perform matrix multiplication calculation on the TCU computing task in each computing task; input the value of the theoretical execution time ratio into the TCU utilization prediction model to obtain the TCU utilization; if the TCU utilization is less than the TCU utilization threshold, each CUDA core computing task and each TCU computing task is sequentially and serially scheduled in a single computing flow. The application can accurately and without additional profiling overhead to predict and select the optimal execution mode of each computing task, thereby significantly improving the overall task computing throughput and hardware resource utilization efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and device for scheduling hybrid computing tasks on TCU and CUDA cores. Background Technology

[0002] A Graphics Processing Unit (GPU), also known as an image processor, is a coprocessor used for processing images and graphics computations, widely used in personal computers, workstations, and some mobile devices. With the rapid development of artificial intelligence and scientific computing, Sparse Matrix-Matrix Multiplication (SpMM) and Dense-Dense Matrix Multiplication (SDDMM) in sparse matrix multiplication have gradually become key computational cores in large-scale data processing and analysis tasks, widely used in machine learning, scientific computing, recommender systems, and other fields. Especially in GPUs, these computations typically involve two heterogeneous computing resources: CUDA cores (Compute Unified Device Architecture Cores) and Tensor Cores (TCUs).

[0003] Currently, some GPU implementations employ a hybrid computing strategy combining TCU and CUDA cores. This strategy can schedule the computational tasks of both TCUs and CUDA cores simultaneously using a single kernel function to improve overall parallelism and resource utilization for the partitioned hybrid computing load. However, this strategy faces two main problems. First, real-world testing shows that parallel execution of TCUs and CUDA cores may lead to a decrease in computation frequency (such as thermal throttling or resource contention), and in some scenarios, it may be less efficient than using only TCUs. Second, in unstructured sparse scenarios, the load between TCUs and CUDA cores is often difficult to distribute evenly within thread blocks. Furthermore, due to the difference in computational capabilities between the two types of computing units, the execution duration within a warp is difficult to align, easily leading to idle resources in some threads, thus reducing overall computational efficiency and still affecting GPU resource utilization. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method and apparatus for scheduling hybrid computing tasks on TCU and CUDA cores to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of this application provides a method for scheduling hybrid computing tasks on TCU and CUDA cores, including:

[0006] The preset theoretical execution time ratio is determined based on the ratio between the estimated execution time of the CUDA cores in the GPU and the estimated execution time of the Tensor Cores (TCUs) in the GPU; wherein, the CUDA cores are used to perform matrix multiplication calculations for the CUDA core computing tasks in each computing task; and the TCUs are used to perform matrix multiplication calculations for the TCU computing tasks in each computing task.

[0007] Input the value of the theoretical execution time ratio into the preset TCU utilization prediction model to obtain the corresponding TCU utilization.

[0008] If the TCU utilization rate is less than the preset TCU utilization rate threshold, then each CUDA core computing task and each TCU computing task are sequentially scheduled in a single computing stream.

[0009] In some embodiments of this application, it also includes:

[0010] If the TCU utilization rate is equal to or greater than the TCU utilization rate threshold, then the preset atomic operation ratio is determined based on the ratio between the total number of atomic operation instructions in the execution process of the CUDA core computing task and the TCU computing task and the total number of all computing operation instructions.

[0011] If the value of the atomic operation ratio is greater than the preset atomic index threshold, then each of the CUDA core computing tasks and each of the TCU computing tasks are sequentially scheduled in a single computing stream.

[0012] In some embodiments of this application, it also includes:

[0013] If the value of the atomic operation ratio is less than or equal to the atomic index threshold, then each of the TCU computing tasks and each of the CUDA core computing tasks are decoupled into different computing streams for parallel scheduling.

[0014] In some embodiments of this application, the TCU utilization prediction model is as shown in formula (1):

[0015] (1)

[0016] in, This represents the theoretical execution time ratio; This indicates the TCU utilization rate.

[0017] In some embodiments of this application, before inputting the value of the theoretical execution time ratio into a preset TCU utilization prediction model, the method further includes:

[0018] An exponential decay model is constructed between the theoretical execution time ratio and TCU utilization; wherein, the exponential decay model is shown in formula (2):

[0019] (2)

[0020] Where A, k, and C all represent undetermined coefficients;

[0021] The actual TCU utilization rate obtained by testing under different theoretical execution time ratios is used as observation data. The values ​​of each of the undetermined coefficients in the exponential decay model are determined by function fitting to obtain the corresponding TCU utilization rate prediction model.

[0022] In some embodiments of this application, before determining the value of the preset theoretical execution time ratio based on the ratio between the estimated execution time of the CUDA cores corresponding to the CUDA cores in the graphics processing unit (GPU) and the estimated execution time of the tensor cores corresponding to the TCUs in the GPU, the method further includes:

[0023] The partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication to be calculated are determined based on the type of the sparse matrix multiplication to be calculated; wherein, the types of sparse matrix multiplication include: sparse matrix multiplication with dense matrix multiplication (SpMM) and sampled dense matrix multiplication (SDDMM);

[0024] Based on the partitioning granularity and the non-zero element threshold, the submatrices corresponding to each window in the sparse matrix multiplication are respectively load-distributed to obtain the computation groups corresponding to each window; wherein, each computation group contains: the CUDA core computation block corresponding to the CUDA core, and / or, the TCU block corresponding to the TCU.

[0025] Load balancing is performed on each of the computing groups to obtain multiple computing tasks corresponding to each computing group; wherein each computing task includes: a CUDA core computing task corresponding to the CUDA core computing block, and / or a TCU computing task corresponding to the TCU block; so that the CUDA core performs matrix multiplication calculations for the CUDA core computing task and / or the TCU performs matrix multiplication calculations for the TCU computing task.

[0026] In some embodiments of this application, determining the partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication based on the type of the sparse matrix multiplication to be computed includes:

[0027] If the type of the sparse matrix multiplication to be calculated is SpMM, then the partitioning granularity corresponding to the SpMM is determined to be non-zero element granularity, and the first non-zero element threshold corresponding to the SpMM is obtained.

[0028] If the type of the sparse matrix multiplication to be calculated is the SDDMM, then the partitioning granularity corresponding to the SDDMM is determined to be the TCU block granularity, and the second non-zero element threshold corresponding to the SDDMM is obtained.

[0029] In some embodiments of this application, the step of distributing the load of the submatrices corresponding to each window in the sparse matrix multiplication according to the partitioning granularity and the non-zero element threshold to obtain the computation group corresponding to each window includes:

[0030] If the type of the sparse matrix multiplication to be calculated is SpMM, then based on the non-zero element granularity, it is determined whether the number of non-zero elements in each column of the sub-matrix corresponding to each window is equal to or greater than the first non-zero element threshold; for each sub-matrix, if there is a column in the sub-matrix where the number of non-zero elements is equal to or greater than the first non-zero element threshold, then the column in the sub-matrix where the number of non-zero elements is equal to or greater than the first non-zero element threshold is compressed into a TCU block; if there is a column in the sub-matrix where the number of non-zero elements is less than the first non-zero element threshold, then the column in each sub-matrix where the number of non-zero elements is less than the first non-zero element threshold is divided into CUDA core computing blocks to obtain the computing groups corresponding to each window;

[0031] If the type of the sparse matrix multiplication to be calculated is SDDMM, then for each window, the columns are sorted in descending order of the number of non-zero elements in each column of the sub-matrix corresponding to the window; based on the TCU block granularity, each sorted column in each window is divided into TCU judgment units, and it is determined whether the number of non-zero elements in each TCU judgment unit is equal to or greater than the second non-zero element threshold; for each sub-matrix, if there is a TCU judgment unit in the sub-matrix whose number of non-zero elements is equal to or greater than the second non-zero element threshold, then the TCU judgment units in each sub-matrix whose number of non-zero elements is equal to or greater than the second non-zero element threshold are assigned as TCU blocks; if there is a TCU judgment unit in the sub-matrix whose number of non-zero elements is less than the second non-zero element threshold, then each element in the TCU judgment unit in each sub-matrix whose number of non-zero elements is less than the second non-zero element threshold is divided into CUDA core computing blocks to obtain the computing groups corresponding to each window.

[0032] In some embodiments of this application, the step of performing load balancing on each of the computing groups to obtain multiple computing tasks corresponding to each computing group includes:

[0033] A preset splitting step is performed on each of the computing groups corresponding to each window to obtain multiple computing tasks corresponding to each computing group.

[0034] The splitting step includes: if the calculation group contains the TCU block, then determine whether the number of columns in the TCU block exceeds the first column number threshold; if so, then divide each TCU block in the calculation group into multiple TCU block calculation tasks based on the first column number threshold.

[0035] Furthermore, if the computing group contains the CUDA core computing blocks, it is determined whether the number of CUDA core computing blocks in the computing group exceeds a second threshold. If so, each CUDA core computing block in the computing group is divided into multiple CUDA core computing tasks based on the second threshold. Each CUDA core computing block in the CUDA core computing task whose number of CUDA core computing blocks exceeds a third threshold is identified as a long CUDA core computing block, and each CUDA core computing block in the CUDA core computing task whose number of CUDA core computing blocks does not exceed the third threshold is identified as a short CUDA core computing block.

[0036] In some embodiments of this application, the corresponding method for scheduling hybrid computing tasks on the TCU and CUDA cores further includes:

[0037] The splitting information corresponding to each calculation task obtained by splitting in the calculation group of each window is recorded based on multiple auxiliary arrays;

[0038] The auxiliary array includes:

[0039] The first array is used to record the number of TCU blocks corresponding to each computing task within each window;

[0040] The second array is used to record the number of non-zero elements corresponding to each computing task in each window;

[0041] The third array is used to record the index of each of the computing tasks in their respective original windows;

[0042] The fourth array is used to record the index of each computation task in its corresponding original row;

[0043] The fifth array is used to record whether each of the aforementioned computational tasks is to be performed as an atomic operation.

[0044] In some embodiments of this application, before determining the partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication based on the type of the sparse matrix multiplication to be computed, the method further includes:

[0045] Construct data access cost models for sparse matrix multiplication (SpMM) and sampled dense matrix multiplication (SDDMM); wherein the access cost model is used to represent the data access cost ratio between the CUDA core and the TCU.

[0046] The partitioning granularity of the SpMM is determined based on the data access cost model corresponding to the SpMM, and the partitioning granularity of the SDDMM is determined based on the data access cost model corresponding to the SDDMM.

[0047] Another aspect of this application provides a hybrid computing task scheduling device on a TCU and CUDA core, comprising:

[0048] The theoretical execution time ratio determination module is used to determine a preset theoretical execution time ratio based on the ratio between the estimated execution time of the CUDA cores corresponding to the CUDA cores in the GPU and the estimated execution time of the tensor cores corresponding to the TCUs in the GPU; wherein, the CUDA cores are used to perform matrix multiplication calculations for the CUDA core computing tasks in each computing task; and the TCUs are used to perform matrix multiplication calculations for the TCU computing tasks in each computing task.

[0049] The TCU utilization prediction module is used to input the value of the theoretical execution time ratio into a preset TCU utilization prediction model to obtain the corresponding TCU utilization.

[0050] A single-stream serial scheduling module is used to sequentially schedule each CUDA core computing task and each TCU computing task in a single computing stream if the TCU utilization rate is less than a preset TCU utilization rate threshold.

[0051] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the hybrid computing task scheduling method on the TCU and CUDA cores.

[0052] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the hybrid computing task scheduling method on the TCU and CUDA cores.

[0053] The fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the hybrid computing task scheduling method on the TCU and CUDA cores.

[0054] The hybrid computing task scheduling method on TCU and CUDA cores provided in this application determines a preset theoretical execution time ratio based on the ratio between the estimated execution time of the CUDA cores corresponding to the CUDA cores in the GPU and the estimated execution time of the tensor cores corresponding to the TCUs in the GPU. The CUDA cores are used to perform matrix multiplication calculations for the CUDA core computing tasks in each computing task; the TCUs are used to perform matrix multiplication calculations for the TCU computing tasks in each computing task. The value of the theoretical execution time ratio is input into a preset TCU utilization prediction model to obtain the corresponding TCU utilization. If the TCU utilization is less than a preset TCU utilization threshold, the CUDA core computing tasks and the TCU computing tasks are sequentially scheduled in a single computing stream. This method can accurately predict and select the optimal execution mode of each computing task kernel function without additional analysis overhead, thereby significantly improving the overall computational throughput of sparse matrix multiplication tasks and the utilization rate of GPU hardware computing resources, and improving the reliability and effectiveness of GPU operation.

[0055] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0056] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0057] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings:

[0058] Figure 1 This is a schematic diagram of the first process of a hybrid computing task scheduling method on the TCU and CUDA cores in one embodiment of this application.

[0059] Figure 2 This is a schematic diagram of the second process of the hybrid computing task scheduling method on the TCU and CUDA cores in one embodiment of this application.

[0060] Figure 3 This is a schematic diagram of the hybrid computing task scheduling process on the TCU and CUDA cores provided in this application.

[0061] Figure 4 This is a schematic diagram of the third process of the hybrid computing task scheduling method on the TCU and CUDA cores in one embodiment of this application.

[0062] Figure 5 This is an example schematic diagram of the original sparse matrix corresponding to the sparse matrix multiplication in one embodiment of this application.

[0063] Figure 6 A flowchart illustrating the hybrid computing task scheduling method on the TCU and CUDA cores provided as an application example of this application.

[0064] Figure 7 This is an example diagram illustrating load distribution for SpMM and SDDMM, provided as an application example of this application.

[0065] Figure 8 This is an example diagram illustrating load balancing for SpMM and SDDMM, provided as an application example of this application.

[0066] Figure 9 This diagram illustrates the relationship between the theoretical execution time ratio r and TCU utilization in an application example of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0068] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0069] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0070] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0071] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0072] CUDA cores are the fundamental computing units in GPUs, used to perform parallel computing tasks, including floating-point operations, integer operations, and memory access. As the basic unit for GPUs to handle massively parallel computing, they are suitable for scientific computing, deep learning, graphics rendering, and other fields. TCU is a new type of processing core that can achieve mixed-precision computing and dynamically adjust computing power according to the reduction in precision, improving throughput while maintaining accuracy.

[0073] Current methods generally lack a hybrid computing task scheduling method for TCU and CUDA cores and lack a clear execution mode. Therefore, even if the load is partitioned, it is difficult to fully unleash the computing potential of hybrid resources.

[0074] Based on this, in order to address the problems of existing scheduling methods that use a single kernel function to simultaneously schedule computing tasks on TCU and CUDA cores, which may lead to a decrease in computing frequency (such as hot throttling or resource contention), and the difficulty in evenly distributing the load between TCU and CUDA cores within thread blocks in unstructured sparse scenarios, and the difficulty in aligning execution durations in warps due to the difference in computing power between the two types of computing units, which can easily lead to some thread resources being idle, thereby reducing overall computing efficiency and still affecting GPU resource utilization, this application provides a method for scheduling hybrid computing tasks on TCU and CUDA cores, a device for scheduling hybrid computing tasks on TCU and CUDA cores to execute the method, a physical device, a computer-readable storage medium, and a computer program product, which can predict suitable execution modes to achieve efficient collaborative computing on TCU and CUDA cores and meet the actual needs of high-performance deep learning and scientific computing for accelerating sparse operators.

[0075] To alleviate the aforementioned single-kernel function scheduling problem, this application introduces a multi-stream parallel execution mode, decoupling the tasks of the TCU and CUDA core into different computational streams for parallel scheduling, thereby improving resource utilization efficiency. This mode exhibits good scalability in tasks with highly decoupled computational paths. However, based on actual test results, the designers of this application found that the multi-stream scheduling mode can also introduce new performance bottlenecks in certain situations: for example, for operators such as SpMM that require load partitioning by row or window, executing tasks across multiple computational streams significantly increases the number of atomic operations, resulting in additional synchronization overhead and reducing overall computational performance. Furthermore, the multi-stream parallel execution mode also suffers from competition for computational resources, leading to a decrease in computational frequency and limiting performance improvement. In contrast, the single-stream serial execution mode (scheduling TCU tasks and CUDA core tasks sequentially in one computational stream), while inherently having lower parallelism, effectively avoids the frequent atomic operation synchronization conflicts that occur in the multi-stream parallel execution mode and significantly reduces the decrease in computational frequency caused by resource contention. Therefore, under specific load structures and computational conditions, the serial execution mode can achieve higher hardware resource utilization efficiency and better overall computational performance.

[0076] Therefore, based on solving the aforementioned problems of scheduling TCU and CUDA core computing tasks simultaneously by a single kernel function, the designers of this application have further overcome the new technical problems caused by introducing a multi-stream parallel execution mode. The designers of this application have designed a new hybrid computing task scheduling method that can accurately evaluate and determine which execution mode is most suitable for a specific hybrid computing task, thereby achieving the best performance. This will be described in detail through the following embodiments.

[0077] Based on this, embodiments of this application provide a method for scheduling hybrid computing tasks on TCU and CUDA cores that can be implemented by a hybrid computing task scheduling device on TCU and CUDA cores. See [link to relevant documentation]. Figure 1 The hybrid computing task scheduling method on the TCU and CUDA cores specifically includes the following:

[0078] Step 100: Determine the preset theoretical execution time ratio based on the ratio between the estimated execution time of the CUDA core corresponding to the CUDA core in the GPU and the estimated execution time of the Tensor Core corresponding to the TCU in the GPU; wherein, the CUDA core is used to perform matrix multiplication calculations for the CUDA core computing tasks in each computing task; the TCU is used to perform matrix multiplication calculations for the TCU computing tasks in each computing task.

[0079] In the embodiments of this application, each computing task includes: a CUDA core computing task corresponding to the CUDA core computing block, and / or a TCU computing task corresponding to the TCU block, so that the CUDA core performs matrix multiplication calculations for the CUDA core computing task and / or the TCU performs matrix multiplication calculations for the TCU computing task. Before scheduling the hybrid computing tasks, the hybrid computing task scheduling device on the TCU and CUDA core can first determine the partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication to be calculated based on the type of the sparse matrix multiplication to be calculated; wherein, the type of sparse matrix multiplication includes: Sparse Matrix and Dense Matrix Multiplication (SpMM) and Sampled Dense Matrix Multiplication (SDDMM); according to the partitioning granularity and the non-zero element threshold, the submatrices corresponding to each window in the sparse matrix multiplication are respectively load-distributed to obtain the computing groups corresponding to each window; load balancing is performed on each computing group to obtain multiple computing tasks corresponding to each computing group.

[0080] It should be noted that a sparse matrix is ​​a matrix in which most elements are 0 (or default values).

[0081] Sparse Matrix-Matrix Multiplication (SpMM) can also be called sparse matrix-dense matrix multiplication or sparse matrix and dense matrix multiplication operations. In SpMM, the mathematical definition is C = A × B; A is a sparse matrix (most elements are 0); C and B are both dense matrices (all elements are non-zero). Its core calculation is to only calculate the product of the number of non-zero elements in the sparse matrix A with the corresponding row and column of B, avoiding zero element operations.

[0082] The mathematical definition of Dense-Dense Matrix Multiplication (SDDMM) is C = S⊙(A×B), where A and B are dense matrices; S is a sparse sampling matrix (mask matrix, specifying the positions of the elements to be calculated); ⊙ represents element-wise multiplication (Hadamard product). Its core calculation is to only calculate the values ​​of the non-zero positions of S in (A×B), and the remaining positions are 0.

[0083] In step 100, this application embodiment designs an index, namely the theoretical execution time ratio, which can also be called the theoretical execution time ratio index, and can be denoted as time_ratio, or simply r.

[0084] This can be achieved by first determining the total number L of floating-point operations allocated to the CUDA cores.cuda With the theoretical peak throughput F of CUDA cores cuda The ratio between (in TFLOPS) is used to calculate the estimated execution time T of the CUDA core. CUDA And based on the total number L of floating-point operations allocated to the TCU tcu Compared to the theoretical peak throughput F of TCU tcu The estimated execution time T of the TCU is calculated by the ratio between (in TFLOPS). TCU Then T CUDA With T TCU The ratio between them is used as the theoretical execution time ratio r.

[0085] Specifically, the formula for calculating the theoretical execution time ratio r (time_ratio) is as follows:

[0086] time_ratio = T CUDA / T TCU = (L cuda / F cuda ) / (L tcu / F tcu )

[0087] Step 200: Input the value of the theoretical execution time ratio into the preset TCU utilization prediction model to obtain the corresponding TCU utilization.

[0088] It is understood that the TCU utilization prediction model refers to a model used for predicting TCU utilization.

[0089] Step 300: If the TCU utilization rate is less than the preset TCU utilization rate threshold, then each CUDA core computing task and each TCU computing task are sequentially scheduled in a single computing stream.

[0090] In step 300, it is determined whether the TCU utilization rate is less than a preset TCU utilization threshold. If so, it indicates that the TCU utilization rate is significantly low, and a serial execution mode should be directly adopted. That is, each CUDA core computing task and each TCU computing task are sequentially scheduled in a single computing stream. In other words, when the TCU utilization rate is significantly low, the optimal execution mode for each CUDA core computing task and each TCU computing task is single-stream serial. By accurately predicting and selecting this optimal execution mode without additional analysis overhead, the overall task's computing throughput and GPU hardware resource utilization efficiency can be significantly improved.

[0091] In a preferred embodiment, to achieve effective task scheduling, this application establishes a TCU utilization threshold U based on extensive experiments (including 500 test matrices). thr =78%, to further improve the accuracy and reliability of determining the optimal execution mode when TCU utilization is significantly low.

[0092] As can be seen from the above description, the hybrid computing task scheduling method on the TCU and CUDA cores provided in this application embodiment can accurately predict and select the optimal execution mode of each computing task kernel function without additional analysis overhead, thereby significantly improving the overall computing throughput of sparse matrix multiplication computing tasks and the utilization rate of GPU hardware computing resources, and improving the reliability and effectiveness of GPU operation.

[0093] To further improve the effectiveness and accuracy of predicting and selecting the optimal execution mode for each computational task kernel function, a hybrid computational task scheduling method on a TCU and CUDA core is provided in this application embodiment, see [link to relevant documentation]. Figure 2 and Figure 3 The following steps after step 200 in the hybrid computing task scheduling method on the TCU and CUDA cores specifically include the following:

[0094] Step 400: If the TCU utilization rate is equal to or greater than the TCU utilization rate threshold, then determine the preset atomic operation ratio based on the ratio between the total number of atomic operation instructions in the execution process of the CUDA core computing task and the TCU computing task and the total number of all computing operation instructions.

[0095] Specifically, the atomic_ratio reflects the degree of synchronization overhead caused by atomic operations in a task, and its calculation formula is as follows:

[0096] atomic_ratio = N_atomic / N_total

[0097] Here, N_atomic represents the total number of atomic operation instructions during task execution, while N_total represents the total number of all computational operation instructions. The lower the atomic_ratio, the less synchronization overhead caused by atomic operations in the task, making multi-stream parallel execution mode more suitable for improving overall computational performance.

[0098] Step 500: If the value of the atomic operation ratio is greater than the preset atomic index threshold, then each of the CUDA core computing tasks and each of the TCU computing tasks are sequentially scheduled in a single computing stream.

[0099] In step 500, if the value of the atomic operation ratio (atomic_ratio) is greater than a preset atomic index threshold A... thr This indicates that the computational task has high synchronization overhead and should adopt a serial execution mode, that is, to sequentially schedule each CUDA core computational task and each TCU computational task in a single computation stream. In other words, when the computational task has high synchronization overhead, the optimal execution mode for each CUDA core computational task and each TCU computational task is single-stream serial. By accurately predicting and selecting this optimal execution mode without additional analysis overhead, the overall computational throughput and GPU hardware resource utilization efficiency of the task can be significantly improved.

[0100] In a preferred embodiment, to achieve efficient task scheduling, this application establishes an atomic index threshold A based on extensive experiments (including 500 test matrices). thr =0.6, to further improve the accuracy and reliability of determining the optimal execution mode for computational tasks with high synchronization overhead.

[0101] Based on this, in order to further improve the effectiveness and accuracy of predicting and selecting the optimal execution mode for each computational task kernel function, a hybrid computational task scheduling method on a TCU and CUDA core is provided in the embodiments of this application, see [link to relevant documentation]. Figure 2 and Figure 3 The following steps after step 300 in the hybrid computing task scheduling method on the TCU and CUDA cores specifically include the following:

[0102] Step 600: If the value of the atomic operation ratio is less than or equal to the atomic index threshold, then each of the TCU computing tasks and each of the CUDA core computing tasks are decoupled into different computing streams for parallel scheduling.

[0103] In step 600, if the value of the atomic operation ratio atomic_ratio is less than or equal to a preset atomic index threshold A... thr This indicates that the computational task has low synchronization overhead and should adopt a multi-stream parallel execution mode, that is, decoupling each TCU computational task and each CUDA core computational task into different computational streams for parallel scheduling. In other words, when the computational task has low synchronization overhead, the optimal execution mode for each CUDA core computational task and each TCU computational task is multi-stream parallelism. By accurately predicting and selecting this optimal execution mode without additional analysis overhead, the overall computational throughput and GPU hardware resource utilization efficiency of the task can be significantly improved.

[0104] To further improve the accuracy and reliability of TCU utilization prediction, in a hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment, the TCU utilization prediction model is as shown in formula (1):

[0105] (1)

[0106] in, This represents the theoretical execution time ratio; This indicates the TCU utilization rate.

[0107] Based on this, in order to further improve the application effectiveness and reliability of the above-mentioned TCU utilization prediction model, in a hybrid computing task scheduling method on TCU and CUDA cores provided in the embodiments of this application, see [link to relevant documentation]. Figure 2 The method for scheduling hybrid computing tasks on the TCU and CUDA cores may include the following content before step 200 or step 100:

[0108] Step 001: Construct an exponential decay model between the theoretical execution time ratio and TCU utilization; wherein, the exponential decay model is as shown in formula (2):

[0109] (2)

[0110] Where A, k, and C are all undetermined coefficients; undetermined coefficient A represents the additional increase in TCU utilization compared to the saturation value in the initial state (when the CUDA core load is very small), undetermined coefficient k describes the rate at which TCU utilization decreases as the CUDA core load ratio increases, and undetermined coefficient C represents the saturation value of TCU utilization (i.e., the stable lower limit of utilization) when the CUDA core load ratio is too high.

[0111] Step 002: The actual TCU utilization rate obtained by testing under different theoretical execution time ratios is used as observation data. The values ​​of each of the undetermined coefficients in the exponential decay model are determined by function fitting to obtain the corresponding TCU utilization rate prediction model.

[0112] Specifically, taking the NVIDIA RTX 4090 GPU as an example, experiments were conducted on the RTX 4090 GPU to test the actual TCU utilization under different theoretical execution time ratios r. Experimental data shows that when r gradually increases from 0.1 to 1, the TCU utilization decreases rapidly; while when r further increases to 6, the utilization almost stabilizes. This application uses the actual TCU utilization obtained from testing under different theoretical execution time ratios r as observation data. This application determines the undetermined coefficients (including A, k, and C) of the model through function fitting, thereby obtaining an accurate TCU utilization prediction model.

[0113] In other words, by comprehensively considering the above two heuristic indicators and the exponential decay function, the embodiments of this application can accurately predict and select the optimal execution mode for each computing task without additional analysis overhead, thereby significantly improving the overall computing throughput and hardware resource utilization efficiency of the task.

[0114] Based on this, the existing methods for sparse matrix multiplication mainly include the following implementation schemes:

[0115] (1) Sparse matrix multiplication is computed based on CUDA core. CUDA core has strong programming flexibility and can effectively handle irregular access to sparse data, but its peak computation performance is significantly lower than that of TCU, making it difficult to efficiently handle high-density computing tasks.

[0116] (2) TCU is used to accelerate sparse matrix calculation, but its effect on highly sparse data is limited, and there is a large amount of computational redundancy and data management overhead. That is, TCU is suitable for regular, high-density matrix multiplication operations and can provide extremely high peak performance, but when processing highly sparse and irregular data, it is easy to generate a large amount of redundant calculations, reduce resource utilization, and fail to achieve the ideal performance improvement.

[0117] (3) Although some solutions have proposed a coarse-grained TCU and CUDA core hybrid computing strategy, there is a lack of a clear sparse load partitioning mechanism for TCU and CUDA core. This results in problems such as coarse partitioning granularity, unbalanced load, and low data processing efficiency. It fails to effectively solve the problems of data redundancy and computational efficiency, thus limiting the overall computing performance.

[0118] In other words, existing sparse matrix multiplication methods lack efficient coordination mechanisms for TCU and CUDA cores on GPU platforms, particularly in dynamically and precisely partitioning computational tasks and optimizing resource allocation in SpMM and SDDMM tasks. Specifically, the technical problems of existing sparse matrix multiplication methods are as follows:

[0119] (1) Current methods for accelerating sparse matrix multiplication (SpMM and SDDMM) on GPUs mainly employ a single type of computing resource, namely CUDA cores or TCUs. However, both methods have significant drawbacks. For example, when TCUs process unstructured sparse data, the operands must be forcibly aligned and filled with a large number of zero values, resulting in significant computational redundancy and greatly reducing computational performance. While CUDA cores can flexibly adapt to data of different sparsities, their computational power is limited, failing to fully exploit the computational potential of GPU hardware. This limits the efficiency of sparse operators in practical scenarios such as high-performance computing and graph neural networks.

[0120] (2) Current technologies lack effective load partitioning strategies to guide fine-grained hybrid scheduling of tasks between CUDA cores and TCUs. Existing sparse load partitioning methods suffer from excessively coarse granularity, typically partitioning tasks based solely on the average sparsity of the sparse matrix or TCU block as a whole, while ignoring the local non-uniformity of the sparse patterns within the matrix or block. This coarse-grained partitioning makes it difficult to accurately match the performance advantages of CUDA cores and TCUs according to task characteristics, thus limiting the full utilization of GPU hardware computing resources.

[0121] (3) Manual parameter tuning is costly and lacks universality: A few studies have attempted to use CUDA cores and TCUs in combination, such as PCGCN and SparseTIR, but they rely on complex graph partitioning tools (such as METIS) or manual parameter tuning, making it difficult to automatically adapt to input data with different sparsity and sparse distribution. The optimization process is complex and the effect is unstable.

[0122] Therefore, in addition to being able to accurately predict and select the optimal execution mode for each computing task without additional analysis overhead, there is an urgent need to propose an efficient TCU and CUDA core hybrid load partitioning strategy to give full play to the advantages of the two resources, improve the overall efficiency and performance of sparse matrix related computing on the GPU platform, and further improve the utilization of GPU hardware computing resources.

[0123] Based on this, in the hybrid computing task scheduling method on the TCU and CUDA cores provided in the embodiments of this application, see... Figure 4 The method for scheduling hybrid computing tasks on the TCU and CUDA cores includes the following steps prior to step 100:

[0124] Step 010: Determine the partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication based on the type of the sparse matrix multiplication to be calculated; wherein, the types of sparse matrix multiplication include: Sparse Matrix Multiplication with Dense Matrix Multiplication (SpMM) and Sampled Dense Matrix Multiplication (SDDMM).

[0125] It is understood that the hybrid computing task scheduling device on the TCU and CUDA core can be set up in the GPU as a sparse matrix multiplication computing accelerator. It is used to receive the current sparse matrix multiplication computing instruction or request, and obtain the original sparse matrix corresponding to the sparse matrix multiplication from the instruction or request, with the missing type being SpMM or SDDMM. Then, it extracts the partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication from the correspondence data between the type, partitioning granularity and non-zero element threshold of the sparse matrix multiplication in the local pre-stored data.

[0126] Wherein, the partition granularity is used to represent the unit partition granularity for load distribution on the original sparse matrix corresponding to the sparse matrix multiplication. The non-zero element threshold is used to represent a threshold for comparison with the number of non-zero elements in the unit partition granularity.

[0127] The aforementioned granularity of division and the threshold for non-zero elements can be manually defined and then transmitted to the hybrid computing task scheduling device on the TCU and CUDA cores for storage. The threshold for non-zero elements can be set according to different GPU models, performance, and response speeds, and can be determined after a limited number of experiments based on actual application scenarios.

[0128] To further improve the effectiveness and reliability of load allocation for sparse matrix multiplication in the subsequent step 200, the partitioning granularity can be pre-generated based on the data access cost ratio between CUDA cores and TCUs, which will be explained in detail in subsequent embodiments.

[0129] Step 020: Based on the partitioning granularity and the non-zero element threshold, the submatrices corresponding to each window in the sparse matrix multiplication are respectively load-distributed to obtain the computation groups corresponding to each window; wherein, each computation group contains: the CUDA core computation block corresponding to the CUDA core, and / or, the TCU block corresponding to the TCU.

[0130] In one or more embodiments of this application, the size of the window can be predetermined by the user according to actual application requirements, and it has the number of rows of the original sparse matrix corresponding to the sparse matrix multiplication. For example, if the original sparse matrix corresponding to the sparse matrix multiplication has 9 rows of elements and the window size is 3, the matrix can be divided into three sub-matrices corresponding to each window. The sub-matrix of the first window is the first three rows of the original sparse matrix, the sub-matrix of the second window is the middle three rows of the original sparse matrix, and the sub-matrix of the third window is the last three rows of the original sparse matrix. If the number of rows of the original sparse matrix corresponding to the sparse matrix multiplication cannot be rounded up by the window size, then the number of rows of the sub-matrix of the last window is less than the window size.

[0131] In step 020, the load allocation of the submatrices corresponding to each window in the sparse matrix multiplication according to the partitioning granularity and the non-zero element threshold means: dividing the submatrices corresponding to each window in the sparse matrix multiplication into judgment units according to the partitioning granularity, and then comparing whether the non-zero elements in each judgment unit exceed the non-zero element threshold. If so, the judgment unit is allocated to the TCU block corresponding to the Tensor Core (TCU) in the GPU; otherwise, the judgment unit is allocated to the CUDA Core compute block corresponding to the CUDA Core in the GPU.

[0132] Therefore, within a window, the load allocation results (TCU block or CUDA core computing block) corresponding to each decision unit in its unique submatrix constitute the computing group corresponding to that window. Since the load allocation results of each decision unit in the same computing group may be consistent or inconsistent, the computing group may contain only TCU blocks, only CUDA core computing blocks, or both TCU blocks and CUDA core computing blocks.

[0133] After workload allocation is complete, the next issue to consider is how to evenly map these allocated tasks across the thread blocks of the parallel system. For the allocated workload, some windows may contain too many TCU blocks or long CUDA core compute blocks, so the windows need to be split through step 300 below to ensure load balance.

[0134] In one example, see Figure 5 For a raw coefficient matrix containing 8 rows and 12 columns, if the window size is 4, the raw coefficient matrix is ​​divided into two windows: the first window and the second window, each with its own submatrix. If the raw coefficient matrix is ​​the same as the SpMM raw coefficient matrix, each column containing non-zero raw elements in the submatrix of each window is a decision unit. Columns designated for TCU processing can be compressed into matrices consisting of multiple sets of columns containing non-zero raw elements, i.e., TCU blocks. For example, in the submatrix corresponding to W0 in the first window, columns 3, 4, 6, and 7 can form a 4×4 matrix, which is a TCU block; columns 0, 1, 2, 5, 8, 9, and 10 are designated as CUDA core computing blocks; and column 11, because it does not contain any non-zero elements, is not included in the decision unit for load allocation.

[0135] Step 030: Perform load balancing on each computing group to obtain multiple computing tasks corresponding to each computing group; wherein each computing task includes: a CUDA core computing task corresponding to the CUDA core computing block, and / or a TCU computing task corresponding to the TCU block; so that the CUDA core performs matrix multiplication calculations for the CUDA core computing task and / or the TCU performs matrix multiplication calculations for the TCU computing task.

[0136] In step 030, if each CUDA core computing block in the computing group obtained in step 020 needs to be split, then each CUDA core computing block in the computing group is divided into multiple CUDA core computing tasks, and each CUDA core computing task contains at least one CUDA core computing block; if each CUDA core computing block in the computing group obtained in step 020 does not need to be split, then each CUDA core computing block in the computing group belongs to the same CUDA core computing task.

[0137] Furthermore, in step 030, if each TCU block in the computing group obtained in step 020 needs to be split into data, then each TCU block in the computing group is divided into multiple TCU computing tasks, and each TCU computing task contains at least one TCU block; if each TCU block in the computing group obtained in step 020 does not need to be split, then each TCU block in the computing group belongs to the same TCU computing task.

[0138] It should be noted that the matrix multiplication calculation performed by the CUDA core for the CUDA core computing task and / or the matrix multiplication calculation performed by the TCU for the TCU computing task specifically refers to:

[0139] (1) If the multiple computation tasks corresponding to the sparse matrix multiplication only include CUDA core computation tasks, then each CUDA core computation task is mapped to the corresponding thread block so that the CUDA core performs matrix multiplication computation for the CUDA core computation task.

[0140] (2) If the multiple computation tasks corresponding to the sparse matrix multiplication only include TCU computation tasks, then each TCU computation task is mapped to the corresponding thread block so that the TCU performs matrix multiplication computation for the TCU computation task.

[0141] (3) If the multiple computation tasks corresponding to the sparse matrix multiplication include CUDA core computation tasks and TCU computation tasks, then each CUDA core computation task and each TCU computation task are mapped to the corresponding thread block, so that the CUDA core performs matrix multiplication computation for the CUDA core computation task and the TCU performs matrix multiplication computation for the TCU computation task.

[0142] As can be seen from the above description, the hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment can also achieve efficient collaborative allocation of sparse computing load between TCU and CUDA cores, realize fine partitioning of sparse load between CUDA cores and TCU under GPU architecture, give full play to the advantages of heterogeneous computing resources, thereby significantly improving the overall performance of sparse operators such as SpMM and SDDMM, making full use of GPU hardware performance, reducing computational redundancy and manual parameter tuning costs, meeting the actual needs of high-performance deep learning and scientific computing for sparse operator acceleration, and thus effectively improving the efficiency of sparse matrix multiplication calculation and GPU hardware computing resource utilization, and improving the reliability and effectiveness of GPU operation.

[0143] To further improve the effectiveness and computational efficiency of SpMM hybrid heterogeneous computing, in a hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment, step 010 of the hybrid computing task scheduling method on TCU and CUDA cores specifically includes the following:

[0144] Step 011: If the type of the sparse matrix multiplication to be calculated is SpMM, then determine that the partitioning granularity corresponding to the SpMM is non-zero element granularity, and obtain the first non-zero element threshold corresponding to the SpMM.

[0145] To further improve the effectiveness and reliability of load allocation for SpMM, in a hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment, step 020 of the hybrid computing task scheduling method on TCU and CUDA cores specifically includes the following:

[0146] Step 021: Based on the non-zero element granularity, determine whether the number of non-zero elements in each column of the sub-matrix corresponding to each window is equal to or greater than the first non-zero element threshold.

[0147] Step 022: For each of the sub-matrices, if there is a column in the sub-matrices with the number of non-zero elements equal to or greater than the first non-zero element threshold, then compress the column in the sub-matrices with the number of non-zero elements equal to or greater than the first non-zero element threshold into a TCU block; if there is a column in the sub-matrices with the number of non-zero elements less than the first non-zero element threshold, then divide the column in each sub-matrices with the number of non-zero elements less than the first non-zero element threshold into a CUDA core computing block to obtain the computing group corresponding to each window.

[0148] In step 022, the judgment unit is the column that has non-zero elements.

[0149] To further improve the effectiveness and computational efficiency of SDDMM hybrid heterogeneous computing, in a hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment, step 010 of the hybrid computing task scheduling method on TCU and CUDA cores specifically includes the following:

[0150] Step 012: If the type of the sparse matrix multiplication to be calculated is the SDDMM, then determine that the partitioning granularity corresponding to the SDDMM is the TCU block granularity, and obtain the second non-zero element threshold corresponding to the SDDMM.

[0151] To further improve the effectiveness and reliability of load allocation for SDDMM, in a hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment, step 020 of the hybrid computing task scheduling method on TCU and CUDA cores specifically includes the following:

[0152] Step 023: For each window, sort each column in descending order of the number of non-zero elements contained in each column of the submatrix corresponding to the window.

[0153] Step 024: Based on the TCU block granularity, divide each sorted column in each window into a TCU judgment unit, and determine whether the number of non-zero elements in each TCU judgment unit is equal to or greater than the second non-zero element threshold.

[0154] For example, the sorted columns in each window are combined according to the preset size of the TCU block to form a TCU block as a judgment unit, i.e., a TCU judgment unit.

[0155] Step 025: For each submatrix, if there are TCU judgment units in the submatrix whose number of non-zero elements is equal to or greater than the second non-zero element threshold, then the TCU judgment units in each submatrix whose number of non-zero elements is equal to or greater than the second non-zero element threshold are assigned as TCU blocks; if there are TCU judgment units in the submatrix whose number of non-zero elements is less than the second non-zero element threshold, then each element in the TCU judgment units in each submatrix whose number of non-zero elements is less than the second non-zero element threshold is divided into CUDA core computing blocks to obtain the computing groups corresponding to each window.

[0156] To further improve the effectiveness and reliability of load balancing for sparse matrix multiplication, in a hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment, step 030 specifically includes the following:

[0157] Step 031: Perform a preset splitting step on each of the computing groups corresponding to each window to obtain multiple computing tasks corresponding to each computing group; wherein, the splitting step includes: if the computing group contains the TCU block, determine whether the number of columns in the TCU block exceeds a first column number threshold; if so, divide each TCU block in the computing group into multiple TCU block computing tasks based on the first column number threshold; and if the computing group contains the CUDA core computing block, determine whether the number of CUDA core computing blocks in the computing group exceeds a second number threshold; if so, divide each CUDA core computing block in the computing group into multiple CUDA core computing tasks based on the second number threshold, and determine each CUDA core computing block in the CUDA core computing task whose number of CUDA core computing blocks exceeds a third number threshold as a long CUDA core computing block, and determine each CUDA core computing block in the CUDA core computing task whose number of CUDA core computing blocks does not exceed the third number threshold as a short CUDA core computing block.

[0158] It is understandable that in step 031, if the number of columns in the TCU block does not exceed the first column number threshold, the TCU block will not be split and will be directly identified as a TCU computing task; if the number of columns in the CUDA core computing block does not exceed the second column number threshold, the CUDA core computing block will not be split and will be directly identified as a CUDA core computing task.

[0159] In one example, the first column number threshold T s = 4; the second column number threshold C s= 5; The threshold for the third column is Short_len=2.

[0160] To further improve the application effectiveness and reliability of long CUDA core computing blocks, short CUDA core computing blocks, and TCU computing tasks, and to improve the efficiency and convenience of data querying during load allocation and balancing processes, in a hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment, step 030 of the hybrid computing task scheduling method on TCU and CUDA cores further includes the following:

[0161] Step 032: Record the splitting information corresponding to each computing task obtained from the splitting of the computing group in each window based on multiple auxiliary arrays; wherein, the auxiliary arrays include: a first array for recording the number of TCU blocks corresponding to each computing task in each window; a second array for recording the number of non-zero elements corresponding to each computing task in each window; a third array for recording the index of each computing task in its corresponding original window; a fourth array for recording the index of each computing task in its corresponding original row; and a fifth array for recording whether each computing task is to be executed an atomic operation.

[0162] Step 033: Map the TCU computing task, long CUDA core computing block, and short CUDA core computing block to different computing resources through three CUDA streams respectively.

[0163] In one example, the first array can be denoted as WindowOffset; the second array can be denoted as RowOffset; the third array can be denoted as CurWindow; the fourth array can be denoted as CurRow; and the fifth array can be denoted as Atomic.

[0164] To further improve the effectiveness and reliability of the partitioning granularity corresponding to sparse matrix multiplication, in the hybrid computing task scheduling method on TCU and CUDA cores provided in this application embodiment, the method before step 010 specifically includes the following:

[0165] Step 002: Construct the data access cost models corresponding to Sparse Matrix Multiplication (SpMM) and Sampled Dense Matrix Multiplication (SDDMM); wherein, the access cost model is used to represent the data access cost ratio between the CUDA core and the TCU.

[0166] Step 003: Determine the partitioning granularity of the SpMM based on the data access cost model corresponding to the SpMM, and determine the partitioning granularity of the SDDMM based on the data access cost model corresponding to the SDDMM.

[0167] To further illustrate the above embodiments, this application also provides a specific application example of a hybrid computing task scheduling method on TCU and CUDA cores. It proposes an efficient hybrid load partitioning strategy that considers both the non-zero element density of sub-blocks in the sparse matrix and the data reusability characteristics of different computing resources during task partitioning. This strategy precisely allocates sparse computing tasks to the most suitable computing unit (CUDA core or TCU), achieving an optimal balance between performance and hardware adaptability. This application proposes a hybrid computing task scheduling method based on dual indices (time_ratio and atomic_ratio) and an exponential decay model to predict suitable execution modes (multi-stream parallel or serial execution) to achieve efficient collaborative computing on TCU and CUDA cores. The core objective of this application is to achieve fine-grained partitioning of sparse loads between CUDA cores and TCUs under a GPU architecture, fully leveraging the advantages of heterogeneous computing resources, thereby significantly improving the overall performance of sparse operators such as SpMM and SDDMM, reducing computational redundancy and manual parameter tuning costs, and meeting the practical needs of high-performance deep learning and scientific computing for accelerating sparse operators.

[0168] This application proposes a novel hybrid load partitioning strategy to efficiently guide the collaborative allocation of sparse computing workloads between TCUs and CUDA cores. In sparse operations such as SpMM and SDDMM, performance bottlenecks often stem from data access to dense matrices, and TCUs and CUDA cores differ significantly in data reuse patterns and theoretical computational performance. Therefore, to fully leverage hardware potential and efficiently handle sparse computing tasks, the allocation strategy in this application considers two key dimensions: data reuse and the actual performance of specific sparse tasks on different hardware resources.

[0169] Specifically, see Figure 6This application example first determines the granularity of load partitioning based on the sparse task type (SpMM or SDDMM), and then determines whether the proportion of non-zero elements in the task or TCU block exceeds a preset threshold to determine the specific sparse load partitioning. The threshold is selected based on the peak performance difference between the TCU and CUDA core hardware resources. When the proportion of non-zero elements is higher than the threshold, it indicates that the sparse task is relatively dense and more suitable for processing by the TCU with higher computing performance; conversely, if the proportion of non-zero elements is lower than the threshold, it indicates that the task is relatively sparse and more suitable for processing by the CUDA core with better sparse flexibility. After the initial task partitioning is completed, this application example further adjusts it through a hybrid computing task scheduling method based on dual indicators and an exponential decay model. This method can accurately predict the performance of different computing paths, dynamically balance the overhead of atomic operations with the utilization of computing resources, and achieve fine-grained matching between task granularity and hardware architecture. In addition, by reasonably setting the resource decomposition threshold, resource contention is avoided to the greatest extent, ensuring the improvement of overall computing efficiency. Overall, this refined load allocation strategy effectively reduces computational redundancy caused by forced data alignment and zero-value filling in unstructured sparse computing. The efficient hybrid computing task scheduling method fully leverages the advantages of heterogeneous computing resources, significantly improving overall computing performance and efficiency.

[0170] The hybrid computing task scheduling method on the TCU and CUDA cores provided in this application example specifically includes the following:

[0171] (a) Determining the granularity

[0172] Regarding data reusability, compared to the CUDA core, the TCU has unique architectural features. After being loaded into registers, operands A1 (i.e., sparse TCU blocks) and B1 (i.e., dense TCU blocks) can be reused multiple times in a single matrix multiplication and accumulation (MMA) instruction. For simplicity, this application defines data access cost as the overhead of loading data from the storage hierarchy, without distinguishing between data sources (such as global memory or cache).

[0173] 1) For SpMM, the main data access cost comes from loading densely packed TCU blocks. The data access cost ratio between CUDA cores and TCUs in SpMM is... It can be represented as:

[0174] (3)

[0175] In equation (3), NNZ1 represents the number of non-zero elements in the sparse TCU block A1; m,n,k are the dimensions of the MMA operands on the TCU.

[0176] For CUDA cores, each non-zero element is processed individually, therefore the data access cost is NNZ. n.

[0177] For TCUs, each dense TCU block B1 is loaded into the register only once, and then reused by multiple non-zero elements in the same non-zero element.

[0178] Therefore, when NNZ1>k, the TCU can reduce data access costs. times.

[0179] Furthermore, this application uses density as an example. Representing the density of the sparse TCU block A1, NNZ1= Substituting into equation (1) above, it is further simplified to This refers to the average number of non-zero elements in a sparse TCU block. Intuitively, higher-density vectors can gain more benefits from data reuse when processed on a TCU.

[0180] Therefore, for SpMM, the application example in this application uses a non-zero column vector granularity (m 1) To allocate sparse workloads to the TCU and CUDA cores.

[0181] 2) For SDDMM, both input TCU blocks A2 and B2 are dense matrices. The data access cost ratio between CUDA cores and TCUs corresponding to SDDMM is... It can be represented as:

[0182] (4)

[0183] In equation (4), NNZ2 represents the number of non-zero elements in the sparse TCU block C.

[0184] When using the CUDA kernel, each non-zero element in a sparse TCU block C requires accessing a row in TCU block A and a column in TCU block B, resulting in a data access cost of 2. NNZ2 k.

[0185] On the TCU, TCU blocks A and B are loaded only once and can be reused by MMA instructions.

[0186] Therefore, when NNZ2> At that time, TCU can reduce data access costs to Unlike SpMM, the formula of SDDMM cannot be further simplified. Intuitively, the more non-zero elements in the sparse TCU block C, the more data reuse advantages the TCU can obtain.

[0187] Therefore, for SDDMM, the application example in this application uses TCU block granularity (m n) Distribute sparse workloads to TCU and CUDA cores.

[0188] (ii) Load Distribution

[0189] The above analysis determines the granularity of load allocation for different operations. In practice, it is easy to satisfy... >1 and The condition >1 means that vectors or blocks with low data access costs should be processed preferentially on the TCU. However, considering only data access cost is not enough. Although the theoretical peak performance of the TCU is much higher than that of the CUDA core, the lower... or This can lead to significant computational redundancy on the TCU because the TCU may process a large number of unnecessary zero elements, resulting in a degraded actual performance. Therefore, a higher number of non-zero elements is necessary to ensure the actual performance of the TCU while maintaining lower data access overhead. However, actual performance cannot be determined in advance; therefore, the application example in this application uses a threshold adjuster to guide load distribution. Based on a previously determined allocation granularity, a vector or block is allocated to the TCU only when the column vector NNZ of the SpMM or the TCU block NNZ of the SDDMM exceeds the threshold; otherwise, it is processed by the CUDA core. Since the actual performance of the TCU can be correlated with the theoretical peak performance... The estimated optimal threshold is highly correlated with the hardware architecture, but less so with the specific matrix.

[0190] See Figure 7 In the sparse load distribution examples for SpMM and SDDMM, for SpMM, this example uses a column vector threshold of 2. The NNZ of non-zero column vectors in each window is counted. If NNZ ≥ 2, the vector is allocated to the TCU; otherwise, it is allocated to a CUDA core. Vectors allocated to the TCU are typically compressed into TCU blocks (4). In step 4), the remaining vectors processed by the CUDA core are used to fill zero vectors to complete the TCU block. For SDDMM, this example uses a TCU block threshold of 4. The non-zero elements in each window are sorted in descending order of NNZ. The densest vectors are preferentially compressed into the TCU block. If the TCU block NNZ ≥ 4, the block is allocated to the TCU; otherwise, it is allocated to the CUDA core. In practice, this example uses the mma.m16n8k4 (TF32) and mma.m16n8k8 (FP16) instructions to process SpMM, and the mma.m16n8k8 (TF32) and mma.m16n8k16 (FP16) instructions to process SDDMM. Combined with the swap transpose strategy, 8 1 vector granularity, using 8 vectors for SDDMM The TCU block granularity is 16. Overall, this example provides an in-depth theoretical analysis of the core issues of load allocation on heterogeneous computing resources, offering clear guidance for precise hybrid computing load allocation.

[0191] (iii) Load balancing

[0192] After workload allocation, the next issue to consider is how to evenly map these allocated tasks across the thread blocks of the parallel system. For allocated workloads, some windows may contain too many TCU blocks or long CUDA core computation blocks, thus requiring window splitting to ensure load balance. However, in SpMM hybrid computation, if the workload within a window is split, each split portion requires atomic operations for accumulation. Therefore, the primary goal of this application example design is to minimize the atomic operation overhead caused by window splitting. This design is guided by two main principles: This application example observes that if a window has a small number of TCU blocks or CUDA core computation blocks, splitting it may not provide sufficient performance gain to offset the additional overhead of atomic operations. Therefore, this application example sets clear criteria to determine whether a window needs to be split.

[0193] This example strictly limits the splitting to within a single window, avoiding splitting across windows, in order to further reduce the overhead of atomic operations. Figure 8 An example of window splitting is shown, which includes TCU blocks (such as 2). 2) and CUDA core compute blocks, and specified splitting conditions (such as T) s = 4 and C s = 5). For CUDA core compute blocks, this example uses a long and short compute block partitioning method, setting Short_len=2 as the standard.

[0194] This example summarizes three window splitting scenarios:

[0195] (1) For window W0, since both the TCU block and the long CUDA core computing block need to be split, all segments within this window need to be atomically operated.

[0196] (2) For window W1, since the CUDA core computation blocks need to be split, the TCU blocks within the same window also need to undergo atomic operations. Similarly, the split CUDA core computation blocks also need to perform atomic operations;

[0197] (3) For windows W2 and W3, these windows contain only a single type of workload and do not meet the splitting conditions, so there is no need to perform atomic operations.

[0198] Furthermore, this application example introduces multiple auxiliary arrays to record splitting information: WindowOffset and RowOffset record the number of TCU blocks and non-zero elements in each split section within each window, respectively; CurWindow and CurRow track the index of each split section in the original window and row, respectively; and Atomic indicates whether each split section needs to perform atomic operations. Overall, the strategy of this application example effectively achieves load balancing while significantly reducing the overhead of atomic operations in hybrid computing.

[0199] (iv) Hybrid computing task scheduling method on TCU and CUDA cores

[0200] To dynamically select between multi-stream parallel execution mode and serial execution mode for optimal performance, this application proposes a hybrid computing task scheduling method on TCU and CUDA cores (by designing two heuristic metrics to accurately evaluate and determine which execution mode is most suitable for a specific hybrid computing task, thereby achieving optimal performance; the two heuristic metrics include:

[0201] 1) Theoretical execution time ratio (denoted as time_ratio, or r for short);

[0202] 2) Atomic operation ratio index (denoted as atomic_ratio).

[0203] This application example proposes a hybrid computing task scheduling method on TCU and CUDA cores, which includes the following 5 steps:

[0204] 1) Define the theoretical execution time ratio time_ratio: This metric is used to measure the ratio of the execution time of CUDA cores and TCUs under a specific task.

[0205] 2) Define atomic_ratio: This metric reflects the degree of synchronization overhead caused by atomic operations in a task.

[0206] 3) Selection of TCU utilization prediction model

[0207] Parallel execution of the TCU and CUDA cores may lead to a decrease in computation frequency (such as thermal throttling or resource contention), resulting in a reduction in the actual utilization of the TCU. To accurately characterize the impact of different computational load ratios on TCU utilization during parallel execution, this application example is based on experimentally observed data characteristics (such as...). Figure 9As shown in the figure) and the theoretical execution time ratio r defined in the application example of this application, the exponential decay model as shown in the above formula (2) was selected.

[0208] 4) Determine the sparsity of the TCU utilization prediction model.

[0209] Taking the NVIDIA RTX4090 GPU as an example, experiments were conducted on the RTX4090 GPU to test the actual TCU utilization under different theoretical execution time ratios r. Experimental data showed that when r gradually increased from 0.1 to 1, the TCU utilization decreased rapidly; while when r further increased to 6, the utilization almost stabilized. In this application example, the actual TCU utilization obtained under different theoretical execution time ratios r was used as observation data. In this application example, the undetermined coefficients (including A, k, and C) of the model were determined by function fitting, thereby obtaining an accurate TCU utilization prediction model as shown in the aforementioned formula (1). The fitting model is highly consistent with the experimental data and effectively depicts the change law of TCU utilization with load ratio. This model not only facilitates the formulation and optimization of subsequent hardware scheduling strategies, but also provides a theoretical basis for in-depth analysis of resource utilization of hybrid computing loads on GPU heterogeneous hardware.

[0210] (5) Task scheduling mode selection strategy

[0211] To achieve effective task scheduling, this application example establishes the TCU utilization threshold U based on extensive experiments (including 500 test matrices). thr= 78%, atomic index threshold A thr =0.6:

[0212] First, calculate the index r based on the defined theoretical execution time ratio formula;

[0213] Substitute the obtained execution time ratio r into the fitted exponential decay function to calculate the predicted TCU utilization U. TCU (r);

[0214] If U TCU (r) thr This indicates that the TCU utilization rate is significantly low, and the serial execution mode should be adopted directly.

[0215] If U TCU (r)>=U thr Then, the atomic operation ratio (atomic_ratio) of the task is further calculated and compared with the threshold A. thr Compare;

[0216] If atomic_ratio>A thr ​If the result is positive, it indicates that the task has high synchronization overhead and should be executed serially; otherwise, it indicates that the task has low synchronization overhead and should be executed in a multi-stream parallel manner.

[0217] In summary, by comprehensively considering the two heuristic indicators and the exponential decay function mentioned above, the application examples of this application can accurately predict and select the optimal execution mode of each computational task kernel function without additional analysis overhead, thereby significantly improving the overall computational throughput and hardware resource utilization efficiency of the task.

[0218] (V) Application Scenarios

[0219] The refined hybrid load partitioning and execution mode prediction strategy proposed in this application example is particularly suitable for sparse computation scenarios in Graph Neural Networks (GNNs). In typical GNN inference or training processes, the node feature aggregation stage involves large-scale sparse matrix multiplication (SpMM) operations and exhibits highly heterogeneous load characteristics, including strong unstructured sparsity and uneven local density distribution. Traditional scheduling strategies based on a single execution mode struggle to adequately adapt to this heterogeneity, easily leading to low resource utilization and computational bottlenecks. By introducing the feature extraction mechanism and execution mode prediction model described in this application example, the sparse computation task of each layer in the GNN can select a more suitable execution mode (multi-stream parallel execution or serial execution) based on the actual graph structure, achieving a high degree of matching between load and hardware resources. Experiments demonstrate that, compared to DGL, the method in this application example achieves a geometric mean speedup of 1.57 times on GCN (up to 1.89 times) and a geometric mean speedup of 2.9 times on AGNN (up to 3.9 times).

[0220] Furthermore, besides unstructured sparse computation scenarios in graph neural networks, the application examples in this application also have broad applicability in large-scale language model (LLM) inference. In recent years, large-scale language models have widely adopted weight pruning techniques for model compression. The pruned large models also involve large-scale sparse matrix multiplication (SpMM) operations. High-accuracy large model pruning techniques often introduce unstructured sparsity, making it difficult to evenly distribute the computational load between CUDA cores and TCUs. The hybrid load partitioning and hybrid computation method proposed in the application examples of this application can adaptively select the optimal execution path based on the distribution characteristics of the pruned sparse weights, achieving efficient scheduling of unstructured sparse operators and significantly improving the inference performance of large sparse models after weight pruning.

[0221] Therefore, whether in sparse adjacency multiplication in GNNs or in weight pruning scenarios in large-scale language models, this method can effectively improve throughput and resource utilization, demonstrating good versatility and scalability.

[0222] (vi) Experimental Testing

[0223] Referring to Table 1, experimental results on the NVIDIA H100 GPU show that, in tests involving 500 sparse matrices, the hybrid computing method using TCU and CUDA cores proposed in this application example covers more optimal matrices and delivers a significant performance improvement compared to using only CUDA cores or only TCUs. Specifically, the hybrid computing method achieves an average speedup of 1.59x and 1.22x in SpMM compared to using only CUDA cores and only TCUs, respectively, and 1.97x and 1.21x in SDDMM, respectively, with a maximum speedup of 10.38x. These results validate that the hybrid computing method proposed in this application example can significantly improve the computational performance of sparse matrix operators on heterogeneous computing resources of TCUs and CUDA cores.

[0224] Table 1

[0225]

[0226] The hybrid load partitioning strategy provided in this application example is a novel task partitioning method. Based on the density of non-zero elements in the sub-blocks of the sparse matrix and the data reuse characteristics, the sparse matrix operators are precisely and dynamically allocated to CUDA cores or TCUs for execution, thereby improving resource matching, reducing computational redundancy, and significantly improving computational performance. It also provides an efficient hybrid computing task scheduling method on a hybrid architecture of TCU and CUDA cores. The core idea of ​​the application example is to accurately analyze and predict the load ratio and computational efficiency of different computing resources based on a clear theoretical execution time ratio index (denoted as time_ratio, or r for short) and an atomic operation ratio index (atomic_ratio), combined with a fitted exponential decay model, so as to determine whether to adopt a single-stream serial execution mode or a multi-stream parallel execution mode.

[0227] The hybrid load partitioning strategy proposed in this application example can accurately allocate sparse operator tasks to suitable CUDA cores or TCUs for execution based on the density distribution of sparse matrix sub-blocks and the characteristics of computing resources. This effectively solves the problem of overly coarse task allocation in existing technologies, which leads to underutilization of computing resources and performance degradation. Through fine-grained task partitioning, this application example achieves a better match between sparse tasks and computing unit characteristics, reducing computational redundancy caused by task-resource mismatch. Furthermore, a hybrid computing task scheduling method for hybrid TCU and CUDA core architectures is proposed, which can automatically determine and select the optimal scheduling strategy (multi-stream parallel execution or serial execution) before execution, significantly improving the overall performance and stability of hybrid computing. Compared with traditional hybrid computing schemes that rely solely on single-core functions, this application example achieves dynamic adaptation of computation paths and hardware characteristics through execution mode matching, improving parallel efficiency, and is particularly suitable for heterogeneous computing tasks in unstructured sparse scenarios with large structural differences.

[0228] Another possible alternative to the application example in this application is to use a coarse-grained random task partitioning method. Instead of performing fine-grained analysis of the sparse matrix, a simpler partitioning strategy can be adopted, such as directly cutting the sparse matrix into multiple roughly uniform submatrices or subtasks of fixed size or arbitrarily, and then arbitrarily mapping these subtasks to a single kernel function or multiple computational flows for execution. Although this method is simpler to implement, it often fails to accurately partition the mixed sparse load due to its lack of consideration for the local density characteristics of the matrix and hardware resource adaptability, leading to significant computational redundancy and memory access conflicts. Furthermore, unreasonable task scheduling can also cause wasted thread resources and decreased execution unit utilization, thus limiting further improvements in overall performance. Compared to the above simple and coarse partitioning scheme, the application example in this application proposes a more refined mixed load partitioning and scheduling strategy. This strategy, based on a reasonable partitioning of the sparse computational load, further selects the optimal execution mode using a hybrid computational task scheduling method with dual indices (time_ratio and atomic_ratio) and an exponential decay model, mapping subtasks to appropriate computational paths to achieve efficient parallel execution. This method not only significantly reduces computational redundancy and memory access conflicts, but also fully leverages the computational advantages of both the CUDA cores and TCUs, thereby more effectively unleashing the collaborative computing potential of GPU heterogeneous resources. Therefore, although using coarse-grained random task partitioning and a simple execution strategy is a feasible alternative, it falls far short of the technical solution proposed in this application example in terms of performance, computational efficiency, and resource utilization.

[0229] From a software perspective, this application also provides a scheduling device for scheduling hybrid computing tasks on TCU and CUDA cores, which is used to execute all or part of the aforementioned method for scheduling hybrid computing tasks on TCU and CUDA cores. The scheduling device specifically includes the following components:

[0230] The theoretical execution time ratio determination module is used to determine a preset theoretical execution time ratio based on the ratio between the estimated execution time of the CUDA cores corresponding to the CUDA cores in the GPU and the estimated execution time of the tensor cores corresponding to the TCUs in the GPU; wherein, the CUDA cores are used to perform matrix multiplication calculations for the CUDA core computing tasks in each computing task; and the TCUs are used to perform matrix multiplication calculations for the TCU computing tasks in each computing task.

[0231] The TCU utilization prediction module is used to input the value of the theoretical execution time ratio into a preset TCU utilization prediction model to obtain the corresponding TCU utilization.

[0232] A single-stream serial scheduling module is used to sequentially schedule each CUDA core computing task and each TCU computing task in a single computing stream if the TCU utilization rate is less than a preset TCU utilization rate threshold.

[0233] The embodiments of the hybrid computing task scheduling device on TCU and CUDA cores provided in this application can be used to execute the processing flow of the embodiments of the hybrid computing task scheduling method on TCU and CUDA cores in the above embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the hybrid computing task scheduling method on TCU and CUDA cores in the above embodiments.

[0234] The part of the hybrid computing task scheduling on the TCU and CUDA cores in the hybrid computing task scheduling device can be completed in the GPU.

[0235] As can be seen from the above description, the hybrid computing task scheduling device on the TCU and CUDA cores provided in this application embodiment can accurately predict and select the optimal execution mode of each computing task kernel function without additional analysis overhead, thereby significantly improving the overall computing throughput of sparse matrix multiplication computing tasks and the utilization rate of GPU hardware computing resources, and improving the reliability and effectiveness of GPU operation.

[0236] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the hybrid computing task scheduling method on the TCU and CUDA cores mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0237] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0238] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the hybrid computing task scheduling method on the TCU and CUDA core in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the hybrid computing task scheduling method on the TCU and CUDA core in the above method embodiments.

[0239] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0240] The one or more modules are stored in the memory, and when executed by the processor, the hybrid computing task scheduling method on the TCU and CUDA cores in the implementation embodiment is executed.

[0241] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0242] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0243] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0244] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned hybrid computing task scheduling method on TCU and CUDA cores. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0245] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned hybrid computing task scheduling method on TCU and CUDA cores.

[0246] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0247] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0248] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0249] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for scheduling hybrid computing tasks on TCU and CUDA cores, characterized in that, include: The preset theoretical execution time ratio is determined based on the ratio between the estimated execution time of the CUDA cores in the GPU and the estimated execution time of the Tensor Cores (TCUs) in the GPU; wherein, the CUDA cores are used to perform matrix multiplication calculations for the CUDA core computing tasks in each computing task; and the TCUs are used to perform matrix multiplication calculations for the TCU computing tasks in each computing task. Input the value of the theoretical execution time ratio into the preset TCU utilization prediction model to obtain the corresponding TCU utilization. If the TCU utilization rate is less than the preset TCU utilization rate threshold, then each CUDA core computing task and each TCU computing task are sequentially scheduled in a single computing stream. If the TCU utilization rate is equal to or greater than the TCU utilization rate threshold, then the preset atomic operation ratio is determined based on the ratio between the total number of atomic operation instructions in the execution process of the CUDA core computing task and the TCU computing task and the total number of all computing operation instructions. If the value of the atomic operation ratio is greater than the preset atomic index threshold, then each of the CUDA core computing tasks and each of the TCU computing tasks are sequentially scheduled in a single computing stream. If the value of the atomic operation ratio is less than or equal to the atomic index threshold, then each of the TCU computing tasks and each of the CUDA core computing tasks are decoupled into different computing streams for parallel scheduling.

2. The method for scheduling hybrid computing tasks on TCU and CUDA cores according to claim 1, characterized in that, The TCU utilization prediction model is shown in formula (1): （1） in, This represents the theoretical execution time ratio; This indicates the TCU utilization rate.

3. The method for scheduling hybrid computing tasks on TCU and CUDA cores according to claim 2, characterized in that, Before inputting the value of the theoretical execution time ratio into the preset TCU utilization prediction model, the method further includes: An exponential decay model is constructed between the theoretical execution time ratio and TCU utilization; wherein, the exponential decay model is shown in formula (2): （2） Where A, k, and C all represent undetermined coefficients; The actual TCU utilization rate obtained by testing under different theoretical execution time ratios is used as observation data. The values ​​of each of the undetermined coefficients in the exponential decay model are determined by function fitting to obtain the corresponding TCU utilization rate prediction model.

4. The method for scheduling hybrid computing tasks on TCU and CUDA cores according to any one of claims 1 to 3, characterized in that, Before determining the preset theoretical execution time ratio based on the ratio between the estimated execution time of the CUDA cores corresponding to the CUDA cores in the GPU and the estimated execution time of the tensor cores corresponding to the TCUs in the GPU, the method further includes: The partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication to be calculated are determined based on the type of the sparse matrix multiplication to be calculated; wherein, the types of sparse matrix multiplication include: sparse matrix multiplication with dense matrix multiplication (SpMM) and sampled dense matrix multiplication (SDDMM); Based on the partitioning granularity and the non-zero element threshold, the submatrices corresponding to each window in the sparse matrix multiplication are load-distributed to obtain computation groups corresponding to each window; wherein each computation group contains: a CUDA core computation block corresponding to the CUDA core, and / or a TCU block corresponding to the TCU; load balancing is performed on each computation group to obtain multiple computation tasks corresponding to each computation group; wherein each computation task contains: a CUDA core computation task corresponding to the CUDA core computation block, and / or a TCU computation task corresponding to the TCU block; so that the CUDA core performs matrix multiplication computation for the CUDA core computation task and / or the TCU performs matrix multiplication computation for the TCU computation task.

5. The method for scheduling hybrid computing tasks on TCU and CUDA cores according to claim 4, characterized in that, The process of determining the partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication based on the type of the sparse matrix multiplication to be computed includes: If the type of the sparse matrix multiplication to be calculated is SpMM, then the partitioning granularity corresponding to the SpMM is determined to be non-zero element granularity, and the first non-zero element threshold corresponding to the SpMM is obtained. If the type of the sparse matrix multiplication to be calculated is the SDDMM, then the partitioning granularity corresponding to the SDDMM is determined to be the TCU block granularity, and the second non-zero element threshold corresponding to the SDDMM is obtained.

6. The method for scheduling hybrid computing tasks on TCU and CUDA cores according to claim 5, characterized in that, The step of distributing the load among the submatrices corresponding to each window in the sparse matrix multiplication according to the partitioning granularity and the non-zero element threshold to obtain the computation group corresponding to each window includes: If the type of the sparse matrix multiplication to be calculated is SpMM, then based on the non-zero element granularity, it is determined whether the number of non-zero elements in each column of the sub-matrix corresponding to each window is equal to or greater than the first non-zero element threshold; for each sub-matrix, if there is a column in the sub-matrix where the number of non-zero elements is equal to or greater than the first non-zero element threshold, then the column in the sub-matrix where the number of non-zero elements is equal to or greater than the first non-zero element threshold is compressed into a TCU block; if there is a column in the sub-matrix where the number of non-zero elements is less than the first non-zero element threshold, then the column in each sub-matrix where the number of non-zero elements is less than the first non-zero element threshold is divided into CUDA core computing blocks to obtain the computing groups corresponding to each window; If the type of the sparse matrix multiplication to be calculated is SDDMM, then for each window, the columns are sorted in descending order of the number of non-zero elements in each column of the sub-matrix corresponding to the window; based on the TCU block granularity, each sorted column in each window is divided into TCU judgment units, and it is determined whether the number of non-zero elements in each TCU judgment unit is equal to or greater than the second non-zero element threshold; for each sub-matrix, if there is a TCU judgment unit in the sub-matrix whose number of non-zero elements is equal to or greater than the second non-zero element threshold, then the TCU judgment units in each sub-matrix whose number of non-zero elements is equal to or greater than the second non-zero element threshold are assigned as TCU blocks; if there is a TCU judgment unit in the sub-matrix whose number of non-zero elements is less than the second non-zero element threshold, then each element in the TCU judgment unit in each sub-matrix whose number of non-zero elements is less than the second non-zero element threshold is divided into CUDA core computing blocks to obtain the computing groups corresponding to each window.

7. The method for scheduling hybrid computing tasks on TCU and CUDA cores according to claim 4, characterized in that, The process of performing load balancing on each of the computing groups to obtain multiple computing tasks corresponding to each computing group includes: A preset splitting step is performed on each of the computing groups corresponding to each window to obtain multiple computing tasks corresponding to each computing group. The splitting step includes: if the calculation group contains the TCU block, then determine whether the number of columns in the TCU block exceeds the first column number threshold; if so, then divide each TCU block in the calculation group into multiple TCU block calculation tasks based on the first column number threshold. Furthermore, if the computing group contains the CUDA core computing blocks, it is determined whether the number of CUDA core computing blocks in the computing group exceeds a second threshold. If so, each CUDA core computing block in the computing group is divided into multiple CUDA core computing tasks based on the second threshold. Each CUDA core computing block in the CUDA core computing task whose number of CUDA core computing blocks exceeds a third threshold is identified as a long CUDA core computing block, and each CUDA core computing block in the CUDA core computing task whose number of CUDA core computing blocks does not exceed the third threshold is identified as a short CUDA core computing block. Correspondingly, the hybrid computing task scheduling method on the TCU and CUDA cores also includes: The splitting information corresponding to each calculation task obtained by splitting in the calculation group of each window is recorded based on multiple auxiliary arrays; The auxiliary array includes: The first array is used to record the number of TCU blocks corresponding to each computing task within each window; The second array is used to record the number of non-zero elements corresponding to each computing task within each window; The third array is used to record the index of each of the computing tasks in their respective original windows; The fourth array is used to record the index of each computation task in its corresponding original row; The fifth array is used to record whether each of the aforementioned computational tasks is to be performed as an atomic operation.

8. The method for scheduling hybrid computing tasks on TCU and CUDA cores according to claim 4, characterized in that, Before determining the partitioning granularity and non-zero element threshold corresponding to the sparse matrix multiplication based on the type of the sparse matrix multiplication to be computed, the method further includes: Construct data access cost models for sparse matrix multiplication (SpMM) and sampled dense matrix multiplication (SDDMM); wherein the access cost model is used to represent the data access cost ratio between the CUDA core and the TCU. The partitioning granularity of the SpMM is determined based on the data access cost model corresponding to the SpMM, and the partitioning granularity of the SDDMM is determined based on the data access cost model corresponding to the SDDMM.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the hybrid computing task scheduling method on the TCU and CUDA cores as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hybrid computing task scheduling method on the TCU and CUDA cores as described in any one of claims 1 to 8.

Images