Kernel-level execution time prediction method for GPU deep learning task

Abstract

The invention belongs to the technical field of GPU performance prediction. The invention provides a kernel-level execution time prediction method for a GPU deep learning task. According to the embodiment of the invention, the cloud platform can quickly and accurately predict the execution time of any deep learning task on different GPU models before the GPU resources are allocated, and meanwhile, the dependence on large-scale data collection and a complex model retraining process are avoided. The kernel-level execution time prediction system adopts a training-free system design, kernel-level execution time prediction is cooperatively realized through a core component consisting of an analyzer (comprising a client analyzer and a server analyzer), a predictor, a kernel mapping table and a kernel cache, the analyzer is responsible for collecting general kernel information across clients and servers, and the predictor is responsible for predicting kernel execution time. The predictor generates execution time on the target GPU based on kernel information, the kernel mapping table is used for aligning kernel sequences under different GPU architectures, and the kernel cache continuously records and reuses observed kernel execution features.

Description

Technical Field

[0001] This disclosure relates to the field of GPU performance prediction technology, and in particular to a kernel-level execution time prediction method for GPU deep learning tasks. Background Technology

[0002] Deep neural networks have demonstrated outstanding performance in numerous applications such as image recognition, speech recognition, and autonomous driving. As model structures become increasingly complex and parameter sizes grow rapidly, deep neural networks are able to learn richer and higher-level feature representations from large-scale data. However, the increased complexity and scale of these models have also significantly increased the demand for GPU computing resources for deep learning training and inference tasks. Currently, public cloud platforms (such as Microsoft Azure and Amazon AWS) have become the primary runtime environment for deep learning tasks. These cloud platforms provide elastic GPU resources and robust infrastructure, enabling users to directly submit deep learning tasks, with the cloud platform handling the allocation and scheduling of GPU resources.

[0003] Modern public clouds utilize multiple generations of GPU architectures. The significant differences in computational performance and memory bandwidth among these GPUs result in varying execution times and throughputs for the same deep learning task across different GPU models. This discrepancy leads to a conflict in GPU resource allocation: users want to select suitable and cost-effective GPUs to meet specific task execution time requirements, while public clouds need to balance high-end and low-end GPU models to reduce operating costs. A natural solution to this conflict is to build predictors that can forecast the execution time of deep neural networks on different GPUs, thereby helping cloud platforms select appropriate GPUs to meet user-defined latency / throughput requirements and optimize operating costs.

[0004] An ideal predictor for public cloud platforms should possess high accuracy, low latency, and be training-free. Specifically, this means: 1) High accuracy: accurately estimating the execution time of any deep learning model on different GPU models; 2) Low latency: providing predictions within milliseconds, suitable for real-time scheduling on cloud platforms; and 3) Training-free: able to operate without relying on large-scale training data and complex model retraining when faced with new GPU models or deep learning architectures. However, existing predictors based on benchmarking, linear regression models, and neural network methods often struggle to simultaneously meet these requirements.

[0005] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0007] The purpose of this disclosure is to provide a kernel-level execution time prediction method for GPU deep learning tasks, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0008] According to a first aspect of the present disclosure, a kernel-level execution time prediction method for GPU deep learning tasks is provided, comprising: S1, Client-side pre-analysis: A complete iteration of a deep learning task is run on a local device. The GPU kernel-level information of this iteration is collected by the client analyzer, and the GPU kernel-level information is converted into general kernel-level information and then uploaded to the cloud asynchronously. S2, cloud platform prediction: The cloud-based predictor uses the received GPU kernel-level information and employs a general kernel-level prediction method or a kernel reuse prediction method to predict the estimated execution time of each kernel on the target GPU for deep learning tasks. S3, Cloud Platform GPU Allocation: Based on the predicted execution time, allocate appropriate GPU resources for the deep learning task; S4, Server-Side Kernel Analysis: When a deep learning task is actually run on the target GPU, the server-side analyzer collects the actual GPU kernel-level information and updates the kernel mapping table and kernel cache to optimize subsequent predictions.

[0009] Furthermore, GPU kernel-level information includes static startup configuration and runtime performance metrics, while general kernel-level information includes total arithmetic operations, total DRAM memory communication, kernel arithmetic strength, total number of SMs required, and number of execution waves.

[0010] Furthermore, kernel reuse prediction methods include: The kernel mapping table is queried to map the kernel sequence on the client GPU to the corresponding kernel sequence on the target GPU, and the kernel cache is queried to reuse the execution time of the same or similar kernels that have been executed.

[0011] Furthermore, general kernel-level prediction methods include: The number of execution waves is calculated based on kernel resource requirements and target GPU hardware specifications; among which, the target GPU hardware specifications include the number of streaming multiprocessors, the maximum number of concurrent threads per SM, the maximum register capacity, and the maximum shared memory capacity; The kernel's performance bottleneck type is determined by the ratio of kernel arithmetic strength to the target GPU's peak computing performance and peak memory bandwidth; among which, the performance bottleneck types include computing bottlenecks and memory bottlenecks. Computation scaling factor and memory scaling factor are constructed based on the peak performance ratio and actual efficiency of the client GPU and the target GPU; Calculate the execution time of a single wave and the execution time of the entire kernel on the target GPU based on the client kernel execution time, the number of execution waves, the calculation scaling factor, and the memory scaling factor.

[0012] Furthermore, the formula for calculating the number of execution waves includes:

[0013] Among them, max_threads is the maximum number of concurrent threads for each SM, threads_per_block is the number of threads per thread block, max_regs is the maximum register capacity, regs_per_block is the register usage per thread block, max_mem is the maximum shared memory capacity, mem_per_block is the shared memory usage per thread block, blocks_per_kernel is the number of thread blocks per kernel, max_blocks is the maximum number of blocks that a single SM can reside in at the same time, and sm_count is the number of streaming multiprocessors.

[0014] Furthermore, the scaling factor is calculated as follows:

[0015] in, This represents the peak computational power of the client's GPU. The peak computational power of the target GPU. For computational efficiency; The memory scaling factor is:

[0016] in, For the peak bandwidth of the client GPU, The peak bandwidth of the target GPU. For memory access efficiency; The formula for calculating the execution time of a single wave is:

[0017] The formula for calculating the execution time of the entire kernel is:

[0018] in, For client GPU execution time, The number of waves executed by the client GPU. The number of waves to execute for the target GPU.

[0019] Furthermore, the process of building and updating the kernel mapping table is as follows: Extract the kernel sequences of the same task on the client and target GPUs, denoted as sequence C and sequence T respectively. Establish a mapping M such that each target kernel corresponds to a continuous sub-interval of the client sequence, and satisfies the constraints of continuity, monotonicity and full coverage. The longest common subsequence algorithm is used to identify deterministic anchor pairs in sequences C and T. These anchor pairs must satisfy the following conditions: identical kernel name, thread block size, and grid size; and a deviation in total instruction count and arithmetic strength within a preset threshold. Inside; Sequence intervals are divided using deterministic anchor points, and functionally equivalent client subsequences and target terminal sequences are extracted within each interval. Construct a mapping entry using the client subsequence as the key and the target subsequence as the value, and update it to the kernel mapping table.

[0020] Furthermore, the matching and updating process of the kernel cache is as follows: During initialization, collect general kernel-level information about the actual kernel executing on the target GPU, including kernel name, thread block size, grid size, total number of instructions, arithmetic strength, and execution time. During cache matching, it is determined whether the kernel name, thread block size, and grid size are completely identical, and whether the deviation between the instruction count and arithmetic strength is within a preset threshold. If the conditions are met, it is considered a cache hit; When a cache hit occurs, the corresponding entry is updated using the average of the current execution time and the original cache duration; when a cache miss occurs, the kernel's general kernel-level information is inserted into the cache as a new entry.

[0021] According to a second aspect of the present disclosure, a kernel-level execution time prediction system for GPU deep learning tasks is provided, which is used to implement any of the above-described kernel-level execution time prediction methods for GPU deep learning tasks.

[0022] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the above-described method and system enable cloud platforms to quickly and accurately predict the execution time of any deep learning task on different GPU models before allocating GPU resources, while avoiding reliance on large-scale data collection and complex model retraining processes. The kernel-level execution time prediction system, Kernomic, employs a training-free system design. It achieves kernel-level execution time prediction through the collaborative efforts of core components consisting of an analyzer (including a client-side analyzer and a server-side analyzer), a predictor, a kernel mapping table, and a kernel cache. The analyzer is responsible for collecting general kernel information across clients and servers; the predictor generates the execution time on the target GPU based on the kernel information; the kernel mapping table aligns kernel sequences under different GPU architectures; and the kernel cache continuously records and reuses observed kernel execution features. Furthermore, the kernel-level execution time prediction system, Kernomic, can efficiently and accurately predict the execution time of various deep learning tasks on different GPUs, providing technical support for GPU resource allocation on cloud platforms. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 The diagram illustrates the steps of a kernel-level execution time prediction method for GPU deep learning tasks in an exemplary embodiment of this disclosure. Figure 2 This illustrates the operation flow of the Kernomic kernel-level execution time prediction system in an exemplary embodiment of this disclosure; Figure 3 This illustration shows an example of kernel sequence variation behavior identified by the kernel map in an exemplary embodiment of this disclosure; Figure 4 This illustrates the process by which the Kernel Map establishes an accurate correspondence between the kernel sequences of the client GPU and the target GPU in an exemplary embodiment of this disclosure; Figure 5 This illustrates the delayed prediction error of a large language model in an exemplary embodiment of this disclosure; Figure 6 This illustrates the delay prediction error of a deep neural network in an exemplary embodiment of this disclosure; Figure 7 This illustrates the predicted execution time delays generated by Kernomic and various benchmark methods in exemplary embodiments of this disclosure; Figure 8 The average prediction error across GPUs for Kernomic and various benchmark methods is shown under different load levels in an exemplary embodiment of this disclosure. Figure 9 The normalized throughput and power-normalized throughput prediction results of ShuffleNet v2 under different graphics card models in the exemplary embodiments of this disclosure are shown (based on T4 graphics card). Figure 10 The normalized throughput and power-normalized throughput prediction results of BERT-Large in an exemplary embodiment of this disclosure are shown (based on a T4 graphics card). Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0027] This example implementation provides a kernel-level execution time prediction method for GPU deep learning tasks. (Reference) Figure 1 As shown, this kernel-level execution time prediction method for GPU deep learning tasks may include: S1, Client-side pre-analysis: A complete iteration of a deep learning task is run on a local device. The GPU kernel-level information of this iteration is collected by the client analyzer, and the GPU kernel-level information is converted into general kernel-level information and then uploaded to the cloud asynchronously. S2, cloud platform prediction: The cloud-based predictor uses the received GPU kernel-level information and employs a general kernel-level prediction method or a kernel reuse prediction method to predict the estimated execution time of each kernel on the target GPU for deep learning tasks. S3, Cloud Platform GPU Allocation: Based on the predicted execution time, allocate appropriate GPU resources for the deep learning task; S4, Server-Side Kernel Analysis: When a deep learning task is actually run on the target GPU, the server-side analyzer collects the actual GPU kernel-level information and updates the kernel mapping table and kernel cache to optimize subsequent predictions.

[0028] The above methods enable cloud platforms to quickly and accurately predict the execution time of any deep learning task on different GPU models before allocating GPU resources, while avoiding reliance on large-scale data collection and complex model retraining processes. The kernel-level execution time prediction system, Kernomic, employs a training-free system design. It achieves kernel-level execution time prediction through the collaborative efforts of core components: an analyzer (including client and server analyzers), a predictor, a kernel mapping table, and a kernel cache. The analyzer collects general kernel information across clients and servers, the predictor generates the execution time on the target GPU based on this information, the kernel mapping table aligns kernel sequences across different GPU architectures, and the kernel cache continuously records and reuses observed kernel execution features. Furthermore, Kernomic enables efficient and accurate prediction of the execution time of various deep learning tasks on different GPUs, providing technical support for GPU resource allocation on cloud platforms.

[0029] Below, we will refer to Figures 1 to 10 The steps of the kernel-level execution time prediction method for GPU deep learning tasks described in this example implementation will be explained in more detail.

[0030] In one embodiment, this application discloses a kernel-level execution time prediction method for GPU deep learning tasks, involving fields such as cloud computing, deep neural networks, and GPU performance forecasting, especially achieving accurate and real-time prediction of deep learning task execution time in heterogeneous GPU clusters.

[0031] With the widespread deployment of deep neural networks in practical applications such as image recognition, speech recognition, and autonomous driving, their model complexity and parameter count continue to increase, leading to a significant increase in the demand for GPU resources. Public clouds have become the mainstream platform for deploying deep learning tasks, providing multiple generations and models of GPU instances (covering various GPU architectures from low-end to high-end) elastically to support Machine-Learning-as-a-Service (MLAS). However, the significant differences in computing performance and memory bandwidth among different GPUs result in significant fluctuations in the execution time of the same deep learning task on different GPUs, leading to a contradiction in GPU allocation: users want to choose GPUs with suitable cost-effectiveness to meet the task's execution time requirements, while cloud platforms need to balance low-end and high-end GPUs to reduce overall operating costs. Therefore, accurately predicting the execution time of deep learning tasks on different GPUs has become a key technical requirement. Existing prediction methods typically rely on massive amounts of training data collected for specific GPUs to build machine learning models, which suffers from high data collection costs, difficulty in adapting to new GPU models or deep learning architectures, and an inability to balance high accuracy with low prediction latency. To address this, this application designs a kernel-level execution time prediction method for GPU deep learning tasks, guiding cloud platforms to allocate GPU resources for these tasks. First, this application designs a kernel-level execution time prediction system called Kernomic, enabling cloud platforms to quickly and accurately predict the execution time of any deep learning task on different GPU models before allocating GPU resources, while avoiding reliance on large-scale data collection and complex model retraining processes. Kernomic employs a training-free system design, and its kernel-level execution time prediction function is implemented collaboratively through the following four core components.

[0032] The Profiler is primarily responsible for collecting fine-grained kernel-level performance metrics from the GPU and synthesizing them to generate general kernel-level information. This information forms the basis for cross-GPU execution time prediction. Through a two-stage design, the profiler can efficiently collect performance metrics asynchronously from both the client and server GPUs. The client profiler runs on the user's local consumer-grade GPU, quickly completing a single-iteration performance profile before task submission; the server profiler collects more precise performance characteristics when the target GPU in the cloud actually executes the task, providing a data source for continuous system optimization.

[0033] The Predictor's main function is to map kernels on the client GPU to equivalent kernels on the target GPU using a kernel mapping table, based on kernel information obtained from the client analyzer. If the kernel already exists in the kernel cache, its execution time is directly read from the cache as the prediction result; otherwise, a general kernel-level prediction method is called to calculate the kernel's execution time. Finally, the Predictor summarizes the execution time of the deep learning task according to the kernel execution order.

[0034] Kernel Map: The kernel map establishes an accurate correspondence between kernel sequences on the client GPU and the target GPU. This component uses the longest common subsequence to identify kernel sequence variation behavior across different GPU architectures, and aligns the kernel sequences at both ends to generate a one-to-one mapping. This mapping ensures that operator kernels on different GPUs can be accurately associated, thus providing a structured foundation for cross-GPU kernel-level prediction.

[0035] Kernel Cache: The kernel cache stores general kernel-level information observed on the target GPU, including the kernel's static boot configuration and execution duration. This component determines whether an entry matches the kernel's static boot configuration; if a match is found, the existing record is updated; otherwise, a new cache entry is created. With continuous usage, the kernel cache gradually forms a continuously evolving kernel-level benchmark library that requires no training.

[0036] In one specific embodiment, the above and other technical features and advantages of this application are described in detail below with reference to the accompanying drawings. A kernel-level execution time prediction method for GPU deep learning tasks includes the following: (I) Kernomic, a kernel-level execution time prediction system The operational flow of the Kernomic kernel-level execution time prediction system is as follows: Figure 2 As shown: Step S1 (Client Pre-analysis): Before submitting a deep learning task to the public cloud, users typically run a complete iteration of the task on their local device, collect GPU kernel-level information of the iteration through the Client-Side Profiler, and asynchronously upload it to the cloud.

[0037] Step S2 (Cloud Platform Prediction): The predictor in the cloud platform provides two complementary prediction options: one is general kernel-level prediction, which estimates the execution time of each kernel in the task based on the hardware differences between the client GPU and the target GPU on the cloud platform; the other is kernel reuse prediction, which reuses the execution time of the same or similar kernels that have been executed on the same target GPU before by querying the kernel map and utilizing the kernel cache, thereby further improving the prediction accuracy.

[0038] Step S3 (Cloud Platform GPU Allocation): The public cloud selects the most suitable GPU resources for the deep learning task based on the predicted execution time results of different GPU models.

[0039] Step S4 (Server-Side Profiler): When the deep learning task is actually running on the target GPU, the Server-Side Profiler will analyze at least one iteration (mainly for the training task), integrate kernel-level information collected from the client GPU and the target GPU on the cloud platform, update the kernel mapping table and kernel cache to reflect the real kernel behavior on the target GPU, thereby continuously improving the accuracy of subsequent predictions.

[0040] (ii) Profiler The Profiler in this application is based on performance analysis tools provided by GPU manufacturers. It generates general kernel information that can be used for cross-GPU execution time prediction by uniformly processing kernel-level performance metrics, thereby supporting high-precision prediction for any deep learning task. The Profiler employs a two-stage analysis process, with the specific steps as follows: Step S11 (Client Kernel Analysis): The client analyzer is deployed on the user's local consumer-grade GPU device. Utilizing performance analysis tools provided by the GPU manufacturer, it performs single-iteration performance profiling of the deep learning task, collecting fine-grained kernel-level performance metrics, including static startup configurations (such as block size and grid size) and runtime performance metrics (such as register usage per thread block (regs_per_block) and shared memory usage per thread block (mem_per_block)). The analyzer then synthesizes these raw metrics into general kernel-level information, such as total arithmetic operations (op_num), total DRAM memory communication (dram_bytes), and kernel arithmetic intensity (AI). This information forms the data foundation for cross-GPU prediction. Furthermore, based on runtime performance metrics, the maximum number of resident blocks per kernel on each streaming multiprocessor can be determined, while based on static startup configuration, the total number of SMs (sm_num) and the number of waves (wave_num) required to complete the kernel can be derived. After extraction, the analyzer asynchronously uploads the kernel-level general information and task data to the cloud. This design eliminates the need for cloud pre-analysis, avoiding resource contention and queuing delays.

[0041] Step S12 (Server-side Kernel Analysis): During the server-side performance analysis on the cloud platform, the analyzer runs on the target GPU of the cloud platform, primarily collecting data during the actual execution of long-duration deep learning training tasks. This stage also collects kernel-level performance metrics on the target GPU and converts them into the aforementioned general kernel-level information (including op_num, dram_bytes, AI, sm_num, wave_num, etc.). The collected information is dynamically updated in the system's kernel cache, gradually building a continuously evolving kernel-level performance benchmark library covering different GPU models. By continuously capturing the real execution characteristics of each kernel on various target hardware, this mechanism provides the prediction system with a training-free, continuously evolving kernel-level benchmark library, thereby ensuring the accuracy of long-term predictions.

[0042] (III) Predictor The predictor in this application is based on the GPU kernel execution mechanism, decomposing the execution time of a deep learning task into the sum of the times of multiple consecutively executed kernels. Based on kernel information obtained from the client analyzer, a kernel mapping table maps kernels on the client GPU to equivalent kernels on the target GPU. If the kernel already exists in the kernel cache, the execution time in the cache is directly read as the prediction result; otherwise, a general kernel-level prediction method is called to calculate the kernel's execution time. Finally, the predictor summarizes the execution time of the deep learning task according to the kernel execution order. The specific execution steps of the predictor are as follows: Step S21 (Kernel Sequence Transformation): The predictor first performs kernel sequence transformation. This step converts the kernel sequence C on the client GPU into the corresponding sequence T on the target GPU. In real-world scenarios, most kernels remain consistent across different GPU models, with only a small number undergoing structural changes due to GPU architecture-specific optimizations. The predictor scans the client sequence C sequentially and uses a stack structure to buffer consecutive kernels whose mappings have not yet been confirmed. For each scanned kernel, the system pushes it onto the stack and then attempts to match the consecutive kernels at the top of the stack with entries in the kernel mapping table, probing from the longest possible consecutive segment. If the kernel segment at the top of the stack successfully matches a mapping table entry, the predictor converts it into the corresponding kernel sequence on the target GPU and removes it from the stack; otherwise, it continues accumulating kernels. Through this mechanism, the system can accurately reconstruct the kernel sequence T corresponding to the deep learning workload submitted by the client when executed on the target GPU.

[0043] Step S22 (Cache-Based Kernel Execution Time Prediction): When a kernel has been executed on the target GPU, the predictor prioritizes querying the kernel cache. If the target kernel's name, thread block size, and grid size strictly match a cache entry, and the deviation of its instruction count and arithmetic strength is within a preset threshold θ, it is considered a cache hit. For a hit kernel, the predictor can directly retrieve the general kernel-level information recorded in the cache as the prediction result. If the kernel type matches multiple cache entries, the predictor can estimate the execution time of the current kernel by linearly scaling the execution times of multiple similar cached kernels, achieving accurate prediction based on historical data.

[0044] Step S23 (Theoretical Prediction Based on a General Kernel-Level Model): For target kernels that fail to hit the kernel cache, the predictor will use a general kernel-level prediction model for theoretical derivation. This model first calculates the number of execution waves based on kernel resource requirements and target GPU hardware specifications. Then, it determines the type of performance bottleneck by analyzing kernel arithmetic strength. Finally, it combines measured performance data from the client GPU with the peak performance difference of the target GPU to construct a time scaling model for computation and memory access, thereby deriving the estimated execution time of the kernel on the target GPU. This method allows the system to provide reliable theoretical predictions even in the absence of historical data. The predictor calculates the required number of execution waves for the kernel based on general kernel information provided by the client analyzer, including the number of thread blocks per kernel, the number of threads per thread block, the register usage per thread block (regs_per_block), and the shared memory usage per thread block (mem_per_block). This is combined with the fixed hardware specifications of the target GPU, such as the number of streaming multiprocessors (sm_count), the maximum number of concurrent threads per SM (max_threads), the maximum register capacity (max_regs), and the maximum shared memory capacity (max_mem). Specifically, through the formula:

[0045] The maximum number of blocks that a single SM can reside in simultaneously, `max_blocks`, is calculated. Then, the total number of SMs required to execute the kernel is calculated, ultimately determining the number of execution waves. This method comprehensively considers various hardware resource constraints such as threads, registers, and shared memory to accurately determine the scheduling scale of each kernel on the target GPU. After determining the number of execution waves, the predictor further predicts the execution time of a single wave. The general information provided by the client analyzer includes the kernel arithmetic operands `op_num`, the total DRAM communication volume `dram_bytes`, and the kernel arithmetic strength `AI = op_num / dram_bytes`. The predictor determines the performance bottleneck type of the kernel on the target GPU by comparing the kernel arithmetic strength `AI` with the ratio of the target GPU's peak computational performance `FLOPS` and peak memory bandwidth `memBW`. If `AI ≥ FLOPS / memBW`, the kernel is a computational bottleneck; otherwise, it is a memory bottleneck. Based on this determination, the predictor builds a model to estimate the duration of a single wave. It is assumed that the peak computational performance and peak bandwidth of the client GPU are FLOPS... c and memBW c The actual computing performance (FLOPS) achieved by the kernel on the client GPU achv and memory bandwidth of memBW achv The computational efficiency is Memory access efficiency is The scaling factor is constructed based on the peak performance ratio and actual efficiency of the client GPU and the target GPU.

[0046] Duration obtained through client-side analysis c and wave_num c Calculate the execution time of a single wave and the execution time of the entire kernel on the target GPU:

[0047] Step S24 (Synthesis of Deep Learning Task Execution Time Prediction Results): After completing the prediction of all kernels, the predictor accumulates the estimated execution times of each kernel according to the transformed sequence T to obtain the total predicted execution time of the entire deep learning task on the target GPU. As the system continues to run, the kernel cache accumulates and matures, and the predictor will increasingly rely on cached data for prediction, thereby gradually reducing its dependence on theoretical models and achieving simultaneous improvement in prediction accuracy and efficiency. This hybrid prediction mechanism ensures that the system maintains high accuracy and low latency during both initial deployment and long-term operation.

[0048] (iv) KernelCache The kernel cache in this application is used to store general kernel-level information on the target GPU and reuses existing kernel data in cross-GPU execution time prediction, thereby achieving high-precision, low-latency prediction without training. The specific steps for updating the kernel cache (KernelCache) are as follows: Step S31 (Kernel Information Cache Initialization): The kernel cache first collects general kernel-level information corresponding to each actually executed kernel on the target GPU, including key features such as kernel name, block size, grid size, total number of instructions, arithmetic intensity (AI), and kernel execution duration. During the execution of a deep learning task, the target GPU performs kernel parsing for one iteration and constructs a persistently manageable kernel cache based on the above features. This cache is used to store and accumulate stable execution behaviors of various kernels under a specific GPU model. Since a single iteration of a deep learning task contains a large number of kernels of limited types, this step effectively reduces the storage size while ensuring the representativeness of the cache content.

[0049] Step S32 (Kernel Cache Matching and Hit Determination): When the target GPU generates a new kernel execution record during an iteration of analyzing a deep learning task, the kernel cache first performs a strict cache matching operation on that kernel. Specifically, the system checks whether the kernel_name, block_size, and grid_size of the current kernel are completely consistent with a certain entry in the cache, and simultaneously determines whether the deviation between the instruction and AI of that entry and the corresponding values ​​of the current kernel are within a preset threshold θ (the default value is 10%, but it can be adjusted according to needs to achieve a stricter or more lenient matching strategy). If the kernel meets all of the above conditions, the system determines that it is a cache hit.

[0050] Step S33 (Dynamic Update of Kernel Cache Entry): When a kernel is determined to have a cache hit, the kernel cache (KernelCache) updates the general kernel information of the corresponding entry in the cache using the latest execution duration of that kernel, thereby achieving continuous adaptive calibration of the target GPU behavior. The update method uses the average of the current execution duration and the original duration in the cache, thus balancing historical information with new observations, enabling the cache to dynamically track the stable trend of kernel execution on the GPU side and improve the accuracy of future inferences. If a kernel fails to find a corresponding entry in the kernel cache that meets the strict matching conditions, the system directly inserts the general kernel-level information of that kernel as a new cache entry into the kernel cache (KernelCache). In this way, this application can continuously supplement previously unseen kernel types and continuously expand the cache coverage with the diversity of deep learning tasks, thus making the kernel cache a continuously evolving kernel-level benchmark library.

[0051] (v) Kernel Map In deep learning workloads, the forward propagation or training iteration of a deep learning task can be broken down into several GPU kernels. Ideally, if the client GPU and target GPU have the same architecture, the kernel sequence for the same DNN should be consistent. However, this is not the case in reality. Client GPUs and target GPUs are usually different, and deep learning frameworks (such as TensorFlow and PyTorch) and GPU compilers optimize for different hardware architectures based on their characteristics. This can lead to variations in the kernel sequence generated for the same deep learning task on different GPUs; this phenomenon is called kernel sequence misalignment. Figure 3 As shown. The main kernel change behaviors include the following: Single-kernel conversion: A kernel with the same functionality may be replaced with a functionally equivalent kernel but with a different architecture on different GPU architectures. For example, a volta_sgemm_tn kernel may be generated on the Volta architecture, while an ampere_sgemm_tn kernel may be generated on the Ampere architecture. The differences mainly come from the compiler's different tiling strategies for the kernel, memory layout (NHWC or NCHW), and optimizations for specific hardware instruction sets (IMMA or HMMA).

[0052] Kernel fusion / splitting: To improve data reuse and memory efficiency, the compiler may fuse multiple kernels into a single composite kernel or split a complex kernel into multiple smaller kernels. This can result in inconsistent kernel sequence lengths and potentially different kernel orders between the client GPU and the target GPU.

[0053] Kernel sequence reassembly: Some architectures may split and reassemble kernel sequences, transforming existing kernel subsequences into new sequences of equivalent size or different structure. This further increases the complexity of sequence misalignment, making simple cache reuse ineffective.

[0054] The KernelMap in this application is used to establish an accurate correspondence between kernel sequences of the client GPU and the target GPU. This component utilizes the longest common subsequence to identify kernel sequence variation behavior under different GPU architectures, and aligns the kernel sequences at both ends to generate a one-to-one mapping relationship. It automatically identifies and locates the "deterministic anchor points" and "variable subsequences" between the client kernel sequence and the target kernel sequence, constructing a reliable mapping relationship from the client to the target GPU while satisfying the constraints of continuity, monotonicity, and full coverage. This links functionally equivalent but structurally different kernels or kernel fragments, ensuring that subsequent cross-GPU execution time predictions based on cache or general models have the correct correspondence. Figure 4 As shown, the specific steps for updating the kernel cache mapping table are as follows: Step S41 (Extracting Kernel Sequences from Client and Target GPUs): Before mapping, perform one iteration of the same deep learning task on both the client and target GPUs, and extract common kernel-level information for each kernel using the Profiler, including kernel_name, block_size, grid_size, instruction, and AI. The extracted sequences are denoted as the target kernel sequences. and client kernel sequence The kernel mapping table formalizes the problem as a constrained sequence mapping problem, with the goal of establishing a mapping M such that each target kernel corresponds to a contiguous sub-interval in the client sequence. And satisfy the following three constraints: Continuity: per kernel Must be mapped to a contiguous subsequence of the client kernel sequence ; Monotonicity: All mapping intervals are either exactly the same as the previous mapping interval (i.e., ... ), or keep the indexes monotonically non-decreasing and non-overlapping (i.e. ); Full coverage: The union of all mapped intervals must completely cover the entire client kernel sequence without any omissions; Step S42 (Identification and Extraction of Deterministic Anchors): The KernelMap identifies identical kernel pairs in two sequences as deterministic anchors. An anchor pair must satisfy the following conditions: kernel_name, block_size, grid_size are identical, and the deviation between the total number of instructions (instructions) and the arithmetic strength (AI) is within a preset threshold θ. To find the longest set of deterministic anchors appearing sequentially in the client and target GPU kernel sequences, the KernelMap uses the longest common subsequence algorithm:

[0055] We use dynamic programming to find all the longest sequence of anchor points that maintain a consistent order. Assume we eventually find k deterministic anchor points, whose indices in the client sequence are... The index in the target sequence is And form a list of deterministic anchor pairs. This step ensures that the sequence index is monotonically increasing, providing a basis for continuous alignment between kernel sequences.

[0056] Step S43 (Kernel Change Behavior Localization): Kernel change behaviors may occur between deterministic anchor points, causing inconsistencies in the length and structure of their kernel subsequences on the client and target GPUs. The deterministic anchor points divide the sequences at both ends into several intervals. The system sequentially checks adjacent anchor point pairs. and If the client index is poor (Due to the inherent sequential nature of the kernel execution trajectory in deep learning tasks, the target-side index difference is at this point) If this is also necessarily true, then it indicates that there is kernel mutation behavior between these two anchor points. At this point, the client subsequence... With target terminal sequence They are extracted. The lengths of these two subsequences may differ (e.g., 4 kernels for the client and 3 kernels for the target), and their internal structures may also differ due to compilation optimizations, but they are functionally equivalent.

[0057] Step S44 (Dynamic Update of Kernel Mapping Entries): For each pair of extracted functionally equivalent subsequences, the kernel mapping table constructs a mapping entry: using the client subsequence as the key and the target subsequence as the value. This entry fully records the detailed information of each kernel in the subsequence, including kernel name, block size, grid size, total number of instructions, and arithmetic strength (AI). By establishing this precise mapping from client mutated subsequences to target equivalent subsequences, the kernel mapping table systematically solves the problem of inaccurate kernel sequence structure caused by compiler optimizations, providing a reliable sequence transformation foundation for accurate prediction across GPUs.

[0058] In one specific embodiment, the above-mentioned and other technical features and advantages of this application will be described in detail below with reference to specific experiments.

[0059] 1. Experimental setup Hardware and software configuration. As shown in Table 1, a Lenovo Y9000P laptop equipped with an RTX 4080 graphics card was used as the client. For the public cloud, T4, A10, A30, A100, V100 (16 GB), and V100 (32 GB) graphics cards were used, hosted on the following Alibaba Cloud instances: gn6ic8g1.2xlarge, gn7i-c8g1.2xlarge, gn7s-c8g 1.2xlarge, gn7e-c16g1.4xlarge, gn6v-c8g1.2xlarge, and gn6 ec12g1.3xlarge, covering Turing, Ampere, and Volta architectures. The runtime environment used the Alibaba Cloud Ubuntu 20.04 image as the operating system, and the GPU runtime environment included CUDA 12.4, cuDNN 9.2.0, PyTorch 2.4.1, and Nsight Compute 2024.1.1.0.

[0060] Table 1 List of GPU models used for evaluation

[0061] Model evaluation: Kernomic was evaluated by predicting the execution time of the DNN models (different batch sizes) listed in Table 2 on different GPUs.

[0062] Table 2 List of deep learning models used for evaluation

[0063] ImageNet and Wikipedia were used as datasets.

[0064] Comparing Kernomic with three baselines: (1) Roofline model, which is one of the most widely used performance prediction methods; (2) Habitat, which uses waveform scaling and pre-trained MLP to directly predict model execution time; (3) NeuSight uses an MLP to estimate the GPU utilization of the DNN model, which is then used to predict the model execution time. For Habitat and NeuSight, their open-source code libraries are used respectively. It should be noted that the NeuSig-ht open-source code library only provides training data for LLM and does not support other DNN models. Therefore, NeuSig-ht is specifically compared in the LLM experiments. Measurements and Metrics. Kernel information for each workload's single iteration is collected using NsightCompute. Training / inference iteration execution time is calculated by the actual runtime required to process a single input batch. Consistent with Habitat, three warm-up iterations are performed first, and then the average execution time of the next three iterations is recorded to mitigate the cold start effect. To demonstrate the practicality of Kernomic in the public cloud, prediction latency, i.e., the time required for Kernomic to generate predictions for newly submitted workloads, is measured. The throughput and power-normalized throughput of the DNN are further calculated. Here, throughput is measured by dividing the batch size by the iteration execution time, while power-normalized throughput is calculated by dividing the throughput by the GPU power consumption. These metrics guide GPU resource allocation in the public cloud to better meet user-defined latency / throughput requirements and reduce operating costs. Furthermore, since the storage costs of kernel cache and kernel mapping are both in the MB range, relatively small compared to workload storage costs, Kernomic's storage costs were not taken into consideration.

[0065] 2. End-to-end prediction accuracy Performance of large language models. Figure 5 This demonstrates the latency prediction error of Kernomic on a large language model. Percentage error (defined as absolute error divided by the true value) is used as the evaluation metric. The runtime of the DNN model on other heterogeneous GPUs is predicted using an RTX4080 laptop as the client. Figure 5Figures (a) and (b) show the prediction errors of Kernomic and three baseline methods on six different GPUs for four large language models (BERT, BERT-Large, GPT2-Large, and OPT-1.3B), respectively. Kernomic demonstrates strong generality, achieving a percentage error of less than 3.91% in all evaluations. Thanks to kernel mapping technology, Kernomic can accurately predict runtimes across heterogeneous GPU architectures, from client-side AdaLovelace to Turing, Ampere, and Volta. Kernomic outperforms baseline methods on almost all workloads, with an average percentage error 61.6 times higher for Roofline, 123.9 times higher for Habitat, and 33.7 times higher for NeuSight. Performance on Deep Neural Networks (DNNs) Figure 6 The percentage prediction error of Kernomic on deep neural network models is shown. As mentioned earlier, NeuSight was not included in this comparison because its open-source codebase only supports Large Language Models (LLM) and cannot be adapted to other types of deep neural networks. Figure 6 Figures (a) and (b) both present the percentage error of prediction for Kernomic and benchmark methods during training and inference on six GPUs and four types of deep neural networks (ResNet50, Inceptionv3, ShuffleNetv2, and SqueezeNet). The results show that Kernomic outperforms all benchmark methods in almost all deep neural network models: in comparison, Roofline has a 34.8-fold higher average percentage error, and Habitat has an 18.1-fold higher average percentage error.

[0066] In summary, Kernomic achieves accurate execution time prediction across graphics cards for Large Language Models (LLMs) and General Deep Neural Networks (DNNs) using only kernel information collected on the client RTX 4080 graphics card, outperforming various benchmark methods in almost all test scenarios. Regarding prediction errors: Kernomic achieves relatively high prediction accuracy of 96.09% under a single deep neural network load; its accuracy in cross-graphics card prediction tasks is even higher at 98.87%. Nevertheless, this method may still experience occasional errors in practical applications, especially during the initial deployment phase. Such issues can be effectively mitigated through the optimization methods described in this application.

[0067] Architectural differences For example, during BERT training, the same iteration step might be compiled as `volta_sgemm_32x128_tn` on a V100 (16GB) but as `ampere_sgemm_128x64_tn` on an A10. However, the kernel cache treats them as the same kernel, limiting prediction accuracy. Kernomic's kernel mapping accurately identifies such compilation differences, maps them to the correct kernel, and leverages cached kernel duration for prediction. Therefore, this approach achieves a low average cross-GPU prediction error of 1.13%, with 86.7% of cross-GPU cases having an error ≤1.5%.

[0068] 3. Kernomic's advantages in public cloud Case 1: Reducing Prediction Latency Suppose a cloud service provider needs to allocate GPU resources to a deep neural network (DNN) load based on the predicted execution time. The provider's goal is to achieve millisecond-level prediction latency while maintaining prediction accuracy. Figure 7 The presentation showcases latency data for predicting four Large Language Models (LLMs) on A100 and T4 graphics cards. Results show that Kernomic significantly reduces prediction latency, stabilizing at around 0.12 seconds, an average increase of only 0.06 seconds compared to Roofline. Habitat and NeuSight, on the other hand, exhibit latency that is 104.89 times (13.11 seconds) and 110.83 times (13.85 seconds) higher than Kernomic, respectively. Furthermore, Kernomic's performance analyzer employs a two-stage design, enabling asynchronous performance analysis of deep neural networks on both client and server graphics cards, ensuring that this analysis process does not introduce any additional prediction latency.

[0069] Case 2: Plug and Play Deployment Kernomic requires no training and has virtually no impact on user code. With just two lines of code added in a single training iteration for performance analysis, the analyzer can achieve cross-GPU prediction capabilities. Figure 8 As shown, Kernomic's initial prediction error is similar to Roofline's, at approximately 55.11%. However, as more kernels in the deep neural network complete performance analysis and caching, its kernel mapping and caching mechanisms gradually reduce the error, eventually converging it to below 1.02%. Although Roofline has a faster response speed, it can only provide coarse-grained predictions, with a prediction error exceeding 50%. Meanwhile, Habitat and NeuSight not only require collecting massive amounts of training data and performing complex model retraining, but their prediction errors are still higher than Kernomic's.

[0070] Case 3: Guiding Graphics Card Resource Allocation Another advantage of Kernomic is its ability to help public cloud service providers develop reasonable GPU allocation schemes, thereby reducing power costs. For example... Figure 9 and Figure 10 As shown, the throughput of the BERT-Large and ShuffleNetv2 models was tested on five different graphics cards, and normalized based on the throughput of a T4 graphics card (blue bars). If public clouds allocate graphics cards based solely on throughput, all tasks would be distributed to high-end graphics cards like the A100. However, this approach doesn't always minimize operating costs because the A100 has relatively higher power consumption. To address this issue, Kernomic further introduces a power-normalized throughput metric (i.e., throughput per unit of power consumption) to guide graphics card allocation. For the ShuffleNetv2 model, the power-normalized throughput of A10, A30, and A100 graphics cards reached 1.27 times, 1.29 times, and 1.16 times that of the T4 graphics card, respectively. For the BERT-Large model, the power-normalized throughput of A10, V100 (16GB), and V100 (32GB) graphics cards performed even better, reaching 1.43 times, 1.17 times, and 1.15 times that of the T4 graphics card, respectively. Furthermore, Kernomic demonstrates high accuracy in predicting throughput and power-normalized throughput, with average errors of only 3.77% and 1.04%, respectively. Therefore, public cloud service providers can leverage Kernomic to allocate GPUs to deep neural network loads based on accurate throughput and power-normalized throughput predictions, thereby minimizing operating costs while meeting users' customized latency / throughput requirements.

[0071] This application presents a kernel-level execution time prediction method for GPU deep learning tasks, designed to guide cloud platforms in allocating GPU resources for these tasks. The application designs a kernel-level execution time prediction system called Kernomic, enabling cloud platforms to quickly and accurately predict the execution time of any deep learning task on different GPU models before allocating GPU resources, while avoiding reliance on large-scale data collection and complex model retraining processes. Kernomic employs a training-free system design, collaboratively implementing kernel-level execution time prediction through four core components: a profiler, a predictor, a kernel map, and a kernel cache. The profiler collects general kernel information across clients and servers; the predictor generates the execution time on the target GPU based on this kernel information; the kernel map aligns kernel sequences across different GPU architectures; and the kernel cache continuously records and reuses observed kernel execution features. Based on this design, the Kernomic system can efficiently and accurately predict the execution time of various deep learning tasks on different GPUs, providing technical support for GPU resource allocation on cloud platforms.

[0072] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0073] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A kernel-level execution time prediction method for GPU deep learning tasks, characterized in that, include: S1, Client-side pre-analysis: A complete iteration of a deep learning task is run on a local device. The GPU kernel-level information of this iteration is collected by the client analyzer, and the GPU kernel-level information is converted into general kernel-level information and then uploaded to the cloud asynchronously. S2, cloud platform prediction: The cloud-based predictor uses the received GPU kernel-level information and employs a general kernel-level prediction method or a kernel reuse prediction method to predict the estimated execution time of each kernel on the target GPU for deep learning tasks. S3, Cloud Platform GPU Allocation: Based on the predicted execution time, allocate appropriate GPU resources for the deep learning task; S4, Server-Side Kernel Analysis: When a deep learning task is actually run on the target GPU, the server-side analyzer collects the actual GPU kernel-level information and updates the kernel mapping table and kernel cache to optimize subsequent predictions.

2. The kernel-level execution time prediction method for GPU deep learning tasks according to claim 1, characterized in that, GPU kernel-level information includes static startup configuration and runtime performance metrics. General kernel-level information includes total arithmetic operations, total DRAM memory communication, kernel arithmetic strength, total number of SMs required, and number of execution waves.

3. The kernel-level execution time prediction method for GPU deep learning tasks according to claim 2, characterized in that, Kernel reuse prediction methods include: The kernel mapping table is queried to map the kernel sequence on the client GPU to the corresponding kernel sequence on the target GPU, and the kernel cache is queried to reuse the execution time of the same or similar kernels that have been executed.

4. The kernel-level execution time prediction method for GPU deep learning tasks according to claim 3, characterized in that, General kernel-level prediction methods include: The number of execution waves is calculated based on kernel resource requirements and target GPU hardware specifications; among which, the target GPU hardware specifications include the number of streaming multiprocessors, the maximum number of concurrent threads per SM, the maximum register capacity, and the maximum shared memory capacity; The kernel's performance bottleneck type is determined by the ratio of kernel arithmetic strength to the target GPU's peak computing performance and peak memory bandwidth; among which, the performance bottleneck types include computing bottlenecks and memory bottlenecks. Computation scaling factor and memory scaling factor are constructed based on the peak performance ratio and actual efficiency of the client GPU and the target GPU; Calculate the execution time of a single wave and the execution time of the entire kernel on the target GPU based on the client kernel execution time, the number of execution waves, the calculation scaling factor, and the memory scaling factor.

5. The kernel-level execution time prediction method for GPU deep learning tasks according to claim 4, characterized in that, The formula for calculating the number of execution waves includes: Among them, max_threads is the maximum number of concurrent threads for each SM, threads_per_block is the number of threads per thread block, max_regs is the maximum register capacity, regs_per_block is the register usage per thread block, max_mem is the maximum shared memory capacity, mem_per_block is the shared memory usage per thread block, blocks_per_kernel is the number of thread blocks per kernel, max_blocks is the maximum number of blocks that a single SM can reside in at the same time, and sm_count is the number of streaming multiprocessors.

6. The kernel-level execution time prediction method for GPU deep learning tasks according to claim 5, characterized in that, The scaling factor is calculated as follows: in, This represents the peak computational power of the client's GPU. The peak computational power of the target GPU. For computational efficiency; The memory scaling factor is: in, For the peak bandwidth of the client GPU, The peak bandwidth of the target GPU. For memory access efficiency; The formula for calculating the execution time of a single wave is: The formula for calculating the execution time of the entire kernel is: in, For client GPU execution time, The number of waves executed by the client GPU. The number of waves to execute for the target GPU.

7. The kernel-level execution time prediction method for GPU deep learning tasks according to claim 1, characterized in that, The process of building and updating the kernel mapping table is as follows: Extract the kernel sequences of the same task on the client and target GPUs, denoted as sequence C and sequence T respectively. Establish a mapping M such that each target kernel corresponds to a continuous sub-interval of the client sequence, and satisfies the constraints of continuity, monotonicity and full coverage. The longest common subsequence algorithm is used to identify deterministic anchor pairs in sequences C and T. These anchor pairs must satisfy the following conditions: identical kernel name, thread block size, and grid size; and a deviation in total instruction count and arithmetic strength within a preset threshold. Inside; Sequence intervals are divided using deterministic anchor points, and functionally equivalent client subsequences and target terminal sequences are extracted within each interval. Construct a mapping entry using the client subsequence as the key and the target subsequence as the value, and update it to the kernel mapping table.

8. The kernel-level execution time prediction method for GPU deep learning tasks according to claim 1, characterized in that, The kernel cache matching and update process is as follows: During initialization, collect general kernel-level information about the actual kernel executing on the target GPU, including kernel name, thread block size, grid size, total number of instructions, arithmetic strength, and execution time. During cache matching, it is determined whether the kernel name, thread block size, and grid size are completely identical, and whether the deviation between the instruction count and arithmetic strength is within a preset threshold. If the conditions are met, it is considered a cache hit; When a cache hit occurs, the corresponding entry is updated using the average of the current execution time and the original cache duration; when a cache miss occurs, the kernel's general kernel-level information is inserted into the cache as a new entry.

9. A kernel-level execution time prediction system for GPU deep learning tasks, characterized in that, Used to implement the method as described in any one of claims 1 to 8.

Images