Fine-grained, interference-aware GPU scheduling method and system for deep learning

Abstract

The present invention proposes a fine-grained, interference-aware GPU scheduling method and system for deep learning, and relates to the technical field of GPU resource scheduling. The method includes collecting resource demand information of workloads in the offline stage; generating CUDA kernel tasks corresponding to each application program executed on the GPU; intercepting the start request of each application program CUDA kernel task, and caching the CUDA kernel task corresponding to the start request in the task queue corresponding to each application program; for high-priority applications, directly submitting the CUDA kernel task in the corresponding task queue to GPU scheduling; for low-priority applications, judging whether to submit the CUDA kernel task to the GPU for scheduling based on the interference score and resource demand. The present invention can reduce resource conflicts and interferences between tasks, realize GPU resource sharing, and improve GPU resource utilization.

Description

Technical Field

[0001] The present invention belongs to the technical field of GPU resource scheduling, and in particular, relates to a fine-grained, interference-aware GPU scheduling method and system for deep learning. Background Art

[0002] With the continuous advancement of computer technology, GPU, as a hardware device specially used for parallel computing, provides strong support for deep learning applications (DNN). The significance of GPU in deep learning applications lies in its powerful parallel computing capabilities, high throughput, optimized hardware architecture and good support for mainstream deep learning frameworks. DNN applications can significantly improve throughput when running on GPU, so they are widely used in various computing-intensive tasks. By using GPU, DNN applications can greatly improve computing efficiency, accelerate model training and reasoning processes, thereby improving productivity and research efficiency.

[0003] However, DNN applications often fail to fully utilize GPU resources. Due to the different memory and computing requirements of operators in the workload, when an operator saturates the GPU's computing unit or memory bandwidth, another resource will be idle. In order to improve the utilization of GPU resources, there are many existing GPU sharing technologies, but there are still problems with fine granularity and interference perception. The time slice rotation technology shares GPU resources by switching contexts between multiple tasks, but too small a time slice will lead to frequent process switching, and too large a time slice will affect the response speed. For low-priority tasks, the short time slice may cause starvation problems, and a single running task cannot consume all computing and memory resources. Although the space sharing technology allows multiple tasks to use the GPU at the same time, the current space sharing technology usually provides coarse-grained GPU partitioning and cannot fully utilize GPU resources. When multiple tasks share the same GPU, resource competition may occur in terms of computing units, memory bandwidth, cache, etc., so requirements are put forward for interference perception.

[0004] Therefore, how to achieve fine-grained and interference-aware scheduling of DNN applications and improve GPU resource utilization is the focus of current research. Summary of the invention

[0005] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a fine-grained, interference-aware GPU scheduling method and system for deep learning, which performs scheduling at the CUDA kernel level, prioritizes CUDA kernel tasks in high-priority task queues, and determines whether to schedule kernel tasks in low-priority task queues based on interference scores and the resource requirements of kernel tasks, thereby reducing resource conflicts and interference between tasks, realizing GPU resource sharing, and improving GPU resource utilization.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention provides a fine-grained, interference-aware GPU scheduling method for deep learning.

[0008] A fine-grained, interference-aware GPU scheduling method for deep learning includes the following steps:

[0009] Collecting workload resource requirement information during the offline phase;

[0010] Generate CUDA kernel tasks for each application to be executed on the GPU based on the application's computing requirements;

[0011] Intercept the launch request of each application's CUDA kernel task, and cache the CUDA kernel task corresponding to the launch request in the task queue corresponding to each application;

[0012] Determine whether each application has a high priority or a low priority;

[0013] Poll the task queue of each application. For high-priority applications, directly submit the CUDA kernel tasks in the corresponding task queue to the GPU for scheduling. For low-priority applications, determine whether to submit the CUDA kernel tasks to the GPU for scheduling based on the interference score and resource requirements.

[0014] A second aspect of the present invention provides a fine-grained, interference-aware GPU scheduling system for deep learning.

[0015] Fine-grained, interference-aware GPU scheduling system for deep learning, including:

[0016] The load analysis module is configured to: collect resource requirement information of the workload during the offline phase;

[0017] The task generation module is configured to: generate a CUDA kernel task corresponding to each application program to be executed on the GPU according to the computing requirements of the application program;

[0018] The kernel interception and management module is configured to: intercept the start request of the CUDA kernel task of each application, and cache the CUDA kernel task corresponding to the start request in the task queue corresponding to each application;

[0019] The priority determination module is configured to: determine whether the priority of each application is a high priority or a low priority;

[0020] The kernel task scheduling module is configured to: poll the task queue of each application, and for high-priority applications, directly submit the CUDA kernel tasks in the corresponding task queue to the GPU for scheduling; for low-priority applications, determine whether to submit the CUDA kernel tasks to the GPU for scheduling based on the interference score and resource requirements.

[0021] One or more of the above technical solutions have the following beneficial effects:

[0022] The present invention provides a fine-grained, interference-aware GPU scheduling method for deep learning. Different scheduling strategies are adopted for applications of different priorities. The kernel tasks of high-priority applications are directly submitted to GPU scheduling. For low-priority tasks, an interference score model is established. The interference score and kernel requirements are combined for scheduling, thereby ensuring the execution performance of high-priority tasks. At the same time, idle resources are used to process low-priority tasks under low interference, thereby improving resource utilization and realizing GPU sharing.

[0023] The present invention refines GPU resource scheduling to the kernel task level, and can more accurately control the execution of each task. This fine-grained scheduling method makes resource utilization higher and ensures that each kernel task is executed at the appropriate time.

[0024] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0026] Figure 1 This is a flow chart of a method according to an embodiment of the present invention.

[0027] Figure 2 This is a scheduling framework diagram of Embodiment 1 of the present invention.

[0028] Figure 3 This is a flowchart of scheduling applications of different priorities according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0031] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0032] Embodiment 1

[0033] SM (Streaming Multiprocessor) is a key component in the NVIDIA GPU architecture and is the basic computing unit of the GPU. Each SM contains multiple CUDA cores, texture units, TensorCores, stream controllers, and memory and other auxiliary units, which can execute multiple computing tasks in parallel. The SMs described below in this embodiment all refer to streaming multiprocessors.

[0034] This embodiment discloses a fine-grained, interference-aware GPU scheduling method for deep learning, which performs scheduling at the CUDA kernel level, prioritizes CUDA kernel tasks in high-priority task queues, and then schedules kernel tasks in low-priority task queues based on interference scores and kernel task resource requirements, builds an interference score model by analyzing GPU resource usage and kernel task resource requirements, and implements co-location of kernel tasks on SM. This method can reduce resource conflicts and interference between tasks, implement GPU resource sharing, and improve GPU resource utilization.

[0035] like Figure 1 As shown, the fine-grained, interference-aware GPU scheduling method for deep learning includes the following steps:

[0036] Collecting workload resource requirement information during the offline phase;

[0037] Generate CUDA kernel tasks for each application to be executed on the GPU based on the application's computing requirements;

[0038] Intercept the launch request of each application's CUDA kernel task, and cache the CUDA kernel task corresponding to the launch request in the task queue corresponding to each application;

[0039] Determine whether each application has a high priority or a low priority;

[0040] Poll the task queue of each application. For high-priority applications, directly submit the CUDA kernel tasks in the corresponding task queue to the GPU for scheduling. For low-priority applications, determine whether to submit the CUDA kernel tasks to the GPU for scheduling based on the interference score and resource requirements.

[0041] In some embodiments, the resource requirement information includes the number of thread blocks, the number of threads in each thread block, the utilization of computing units and memory bandwidth, and the estimated execution time.

[0042] In some embodiments, according to the computing requirements of the application, a CUDA kernel task corresponding to each application is generated and executed on the GPU, specifically:

[0043] The deep learning framework is used to compile the operations of the application into CUDA computing kernels. The CUDA computing kernel is the basic unit of GPU execution. Each CUDA computing kernel represents a specific computing task, forming a CUDA kernel task executed on the GPU.

[0044] In some embodiments, the launch request from each application CUDA kernel task is intercepted by means of a dynamic link library.

[0045] In some embodiments, the priority of an application is determined to be high priority or low priority based on multiple factors. If one of the factors meets the high priority requirement, the current application is high priority, otherwise it is low priority; the multiple factors include task type, business importance and real-time performance requirements.

[0046] In some embodiments, when a task queue of a high priority application is polled:

[0047] If there is a CUDA kernel task to be executed, the CUDA kernel task to be executed is immediately submitted to the GPU hardware for execution;

[0048] If there is no CUDA kernel task to be executed, continue to traverse the task queue of the next application.

[0049] In some embodiments, for low-priority applications, whether to submit CUDA kernel tasks to the GPU for scheduling is determined based on the interference score and resource requirements, specifically:

[0050] Set the SM requirement threshold SM_TH to determine whether the SM requirement of the low-priority CUDA kernel task is lower than SM_TH;

[0051] Set the execution time threshold TI_TH to determine whether the execution time of the low-priority CUDA kernel task is less than TI_TH;

[0052] Set the interference score threshold SCORE_TH to determine whether the interference score of the low-priority CUDA kernel task is lower than SCORE_TH;

[0053] When all the above conditions are met, the CUDA kernel task of the current low-priority application is submitted to the GPU for scheduling.

[0054] Next, we will combine Figure 2 and Figure 3 , the technical solution of the embodiment of the present invention is elaborated in detail.

[0055] The fine-grained, interference-aware GPU scheduling method for deep learning in this embodiment may specifically include:

[0056] S1: Collects and analyzes workload demand information in the offline phase and stores the collected data in a configuration file;

[0057] Furthermore, in the offline phase, the resource requirement information of each operation is analyzed and collected, including the number of thread blocks, the number of threads in each thread block, the utilization of computing units and memory bandwidth, and the estimated execution time.

[0058] Specifically, we use NVIDIA's Nsight Compute tool to collect resource requirement information for each kernel, including the number of thread blocks and the number of threads in each thread block. We use the throughput of the SM computing unit and the throughput of the memory bandwidth in the Nsight Compute tool to measure the computing throughput and memory bandwidth utilization of each kernel. We also use Nsight Systems to record the start and end time of the kernel to calculate the actual execution time of each kernel.

[0059] These analysis results are stored in a configuration file for the scheduler to use when actually scheduling the kernel.

[0060] It can be understood that the NVIDIA mentioned above refers to a company that focuses on artificial intelligence and deep learning, providing various solutions and platforms; Nsight Compute is an interactive kernel analyzer designed specifically for CUDA applications, which provides detailed performance indicators and API debugging capabilities, provides services through a graphical user interface and command line tools, and allows users to compare different performance analysis results in the tool; Nsight Systems is a system-level performance analysis tool for analyzing and optimizing the performance of CUDA applications. It provides visual performance analysis tools that support viewing CUDA core activity, kernel execution time, memory operations, call stacks and other information.

[0061] Furthermore, for each low-priority application's core , determine the number of thread blocks each SM can support , It may be limited by the number of threads, the number of registers, or the amount of shared memory available on the SM. To calculate the number of SMs required for a kernel, the formula is as follows:

[0062]

[0063] in, Representation kernel The total number of blocks required, Indicates the number of thread blocks that each SM can support; rounded up , we can get the kernel The number of SMs required .

[0064] S2: Generates a series of CUDA kernel tasks executed on the GPU based on the computing requirements of the application;

[0065] Furthermore, the deep learning framework is used to compile the operations of the application into CUDA computing kernels. These CUDA kernels are the basic units of GPU execution, and each kernel represents a specific computing task.

[0066] In the deep learning framework PyTorch, high-level APIs are used to define neural network models and operations, and the application framework compiles these operations into CUDA computing kernels. These CUDA kernels are the basic units of GPU execution, and each kernel represents a specific computing task.

[0067] S3: intercepts the launch request of the CUDA kernel of each application and caches the tasks corresponding to the requests in the task queue;

[0068] Furthermore, when the kernel issues a scheduling request, the scheduler intercepts the kernel request from each client application by means of a dynamic link library and caches these kernels in a dedicated software queue of each client application.

[0069] The scheduler intercepts requests through dynamic link libraries to achieve dynamic management of CUDA kernels. When applications issue GPU operations, these requests are not submitted directly to the GPU hardware, but are managed by the scheduler. The system caches these kernel tasks in the dedicated task queue of each application. The task queue is used to store the corresponding CUDA kernel tasks, which ensures that the CUDA kernels of different applications reduce mutual interference during the scheduling process.

[0070] S4: Determine whether the priority of the application is high priority or low priority based on multiple factors. If one of the factors meets the high priority requirement, the current application is high priority, otherwise it is low priority; the multiple factors include task type, business importance and real-time performance requirements.

[0071] The specific judgment method is:

[0072] Judge based on task type: If the application is a real-time reasoning task or an application that requires a quick response, it is considered a high priority;

[0073] Set a business importance threshold to determine whether the business importance value of the application is greater than the business importance threshold. If so, the current application is a high priority, otherwise it is a low priority;

[0074] A real-time performance requirement threshold is set to determine whether the real-time performance requirement value of the application is greater than the real-time performance requirement threshold. If so, the current application is a high priority, otherwise it is a low priority.

[0075] Furthermore, this embodiment evaluates the priority of applications based on multiple factors, including task type, business importance, and real-time performance requirements. Specifically, real-time reasoning tasks or applications that require fast response are usually considered high priority. If the success of an application has a significant impact on the overall business results, it should be given a higher priority. Real-time performance requirements are also an important consideration. Applications involving user experience, such as interactive programs, must prioritize their performance to improve user satisfaction. Applications are analyzed before scheduling, and the above multiple dimensions are combined to determine the priority of applications in advance. Based on the resource demand information of the application, high- and low-priority applications can share the GPU. During specific execution, you only need to view the relevant information of the application to derive its priority classification (high or low).

[0076] S5: Poll the task queue of each application. For high-priority applications, directly submit the corresponding CUDA kernel task to GPU scheduling; for low-priority applications, determine whether to submit the kernel task based on the interference score and resource requirements.

[0077] Furthermore, if Figure 3 As shown in the figure, when the scheduler polls the task queue of the high-priority application:

[0078] If there are kernels to be executed in the queue, the kernels will be submitted to a dedicated GPU stream, which will immediately submit these kernel tasks to the GPU hardware for execution;

[0079] If there is no CUDA kernel task to be executed in the queue, continue to traverse the task queue of the next application.

[0080] Furthermore, for the kernels of low-priority applications, the number of low-priority kernel SMs, running time, and interference scores are comprehensively considered to determine whether the kernel can be scheduled. If it is suitable for execution, it is submitted to the corresponding GPU stream and waits to be scheduled.

[0081] Furthermore, the task queues of high-priority applications are given priority. When the scheduler polls the task queues of high-priority applications, if there are CUDA kernel tasks to be executed in the queue, these kernel tasks will be submitted to the GPU hardware for execution immediately. These CUDA kernel tasks will be submitted to a dedicated GPU stream to ensure that the kernels in the same stream are executed sequentially.

[0082] Furthermore, for the kernels of low-priority applications, the number of SMs, running time, and interference score required by the low-priority kernels are comprehensively considered to determine whether the kernels can be scheduled. If suitable for execution, they are submitted to the corresponding GPU stream; specifically, the following are included:

[0083] Two parameters are set to limit the operation of low-priority kernel tasks, SM demand threshold (SM_TH) and execution time threshold (TI_TH). Only when the SM demand of the low-priority kernel is lower than the preset SM_TH and the execution time is less than TI_TH, the low-priority kernel is suitable for scheduling. By strictly limiting the resource usage and execution time of the low-priority kernel, it is ensured that the high-priority kernel can obtain sufficient resources and respond in time when sharing GPU resources.

[0084] Specifically, in this embodiment, SM_TH is set based on the number of stream processors of the GPU device and the resource requirements of the high-priority kernel to ensure that the high-priority kernel can obtain sufficient stream processor resources. SM_TH defaults to the total number of SMs on the GPU. For TI_TH, the time sensitivity of the high-priority kernel should be analyzed first, and the maximum delay it can tolerate should be calculated. TI_TH is set to 10% of the maximum request delay of the high-priority kernel. The two parameters are dynamically adjusted by monitoring the actual performance of the high-priority kernel.

[0085] Furthermore, the interference score of the low-priority kernel is calculated according to the interference score formula, thereby measuring the degree of interference caused when the low-priority kernel is scheduled to the GPU:

[0086]

[0087] in, is the computational unit utilization of the kernel, is the kernel's memory bandwidth utilization, , are the coefficients of the kernel's compute unit utilization and memory bandwidth utilization, respectively. and The value of is determined by the GPU resource usage:

[0088]

[0089] in, , They are the overall GPU computing unit utilization and memory bandwidth utilization. In order to prevent the denominator from being 0, a small positive number is introduced , which can be neglected in other calculations.

[0090] When the GPU computing unit utilization is high, the computing unit is the main interference factor, otherwise the memory bandwidth is the main interference factor. When calculating the interference score, the coefficients of each item are determined according to the GPU resource usage. According to the above formula, the interference score of the low-priority kernel is calculated and compared with the set interference score threshold SCORE_TH. When the interference score is lower than SCORE_TH, the low-priority kernel meets the scheduling conditions.

[0091] After the above conditions are met, the low-priority kernel tasks are submitted to the corresponding GPU stream, and the GPU will execute these kernels in sequence according to the order in the GPU stream. After the software queues of all applications have been checked, the system will continue to poll the software queues in sequence, evaluate the new kernel tasks in the software queues, and submit them to the appropriate GPU stream, so that the system can quickly respond to the needs of new kernels, avoid tasks waiting in the queue for a long time, and ensure that they enter the execution phase as soon as possible.

[0092] Specifically, the system is integrated into the PyTorch framework through model testing for verification, and GPU scheduling is implemented by overwriting the CUDA kernel launch function and common library functions. For the convenience of description, in this embodiment, the workload uses three inference models, Model 1, Model 2 and Model 3 are ResNet50, ResNet101 and MobileNetV2 respectively. In this embodiment, ResNet50 is a real-time inference task, which is set to high priority, and the other two models are low priority. Table 1 shows the batch size used for each workload and the resource utilization when a single model monopolizes the GPU. It can be seen from the table that the resource utilization of a single model is low when it monopolizes the GPU, and the sum of the resource occupancy of the three models is still lower than the total resources of the GPU. Therefore, these three models can be co-located on the same GPU to achieve GPU resource sharing.

[0093] Table 1 GPU utilization of deep learning models

[0094]

[0095] like Figure 2As shown in the figure, workload analysis is performed on the three models ResNet50, ResNet101 and MobileNetV2 mentioned above. According to the model requirements, a series of kernel tasks that need to be executed on the GPU are generated. The kernel tasks are analyzed and their demand information is collected. The system intercepts the scheduling requests of these kernel tasks and caches these kernel tasks in the corresponding task queues.

[0096] When polling the task queue of model 1, its kernel is directly submitted to the GPU for execution;

[0097] When the system scheduler polls the task queue of model 2 or model 3, the scheduler will first evaluate the kernel task based on its interference score and resource requirements. If the evaluation result shows that the kernel task is suitable for execution (that is, the interference score is lower than the threshold, and the resource requirements and execution time are within the preset acceptable range), the scheduler will submit these low-priority kernel tasks to the corresponding GPU stream for scheduling. Otherwise, the kernel task will not be scheduled temporarily and will wait for the next round of polling for evaluation.

[0098] Embodiment 2

[0099] This embodiment discloses a fine-grained, interference-aware GPU scheduling system for deep learning.

[0100] Fine-grained, interference-aware GPU scheduling system for deep learning, including:

[0101] The load analysis module is configured to: collect resource requirement information of the workload during the offline phase;

[0102] The task generation module is configured to: generate a CUDA kernel task corresponding to each application program to be executed on the GPU according to the computing requirements of the application program;

[0103] The kernel interception and management module is configured to: intercept the start request of the CUDA kernel task of each application, and cache the CUDA kernel task corresponding to the start request in the task queue corresponding to each application;

[0104] The priority determination module is configured to: determine whether the priority of each application is a high priority or a low priority;

[0105] The kernel task scheduling module is configured to: poll the task queue of each application, and for high-priority applications, directly submit the CUDA kernel tasks in the corresponding task queue to the GPU for scheduling; for low-priority applications, determine whether to submit the CUDA kernel tasks to the GPU for scheduling based on the interference score and resource requirements.

[0106] It can be understood that in the above load analysis module, resource demand information of the workload is collected in the offline stage, and the collected data is stored in the configuration file.

[0107] It should be noted here that the above-mentioned load analysis module, task generation module, kernel interception and management module, judgment module and kernel task scheduling module correspond to steps S101 to S105 in the first embodiment. The examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the first embodiment. It should be noted that the above-mentioned modules as part of the system can be executed in a computer system such as a set of computer executable instructions. The description of each embodiment in the above-mentioned embodiments has different emphases. For the parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0108] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0109] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A fine-grained, interference-aware GPU scheduling method for deep learning, characterized in that: The following steps are involved: Collecting workload resource requirement information during the offline phase; Generate a CUDA kernel task for each application to be executed on the GPU according to the computing requirements of the application; specifically: use the deep learning framework to compile the application's operations into CUDA computing kernels, which are the basic units of GPU execution. Each CUDA computing kernel represents a specific computing task, forming a CUDA kernel task executed on the GPU; Intercept the launch request of each application's CUDA kernel task, and cache the CUDA kernel task corresponding to the launch request in the task queue corresponding to each application; Determine whether each application has a high priority or a low priority; Poll the task queue of each application. For high-priority applications, directly submit the CUDA kernel tasks in the corresponding task queue to the GPU for scheduling. For low-priority applications, determine whether to submit the CUDA kernel tasks to the GPU for scheduling based on the interference score and resource requirements. The interference score is used to measure the interference level when scheduling low-priority CUDA kernel tasks to the GPU: ； in, is the computational unit utilization of the kernel, is the kernel's memory bandwidth utilization, , are the coefficients of the core’s compute unit utilization and memory bandwidth utilization, respectively; Score the interference.

2. The fine-grained, interference-aware GPU scheduling method for deep learning as claimed in claim 1, characterized in that: The resource requirement information includes the number of thread blocks, the number of threads in each thread block, the utilization of computing units and memory bandwidth, and the estimated execution time.

3. The fine-grained, interference-aware GPU scheduling method for deep learning as claimed in claim 1, characterized in that: The launch request of the CUDA kernel task from each application is intercepted by means of a dynamic link library.

4. The fine-grained, interference-aware GPU scheduling method for deep learning as claimed in claim 1, characterized in that: The priority of the application is determined to be high priority or low priority based on multiple factors. If one of the factors meets the high priority requirement, the current application is high priority, otherwise it is low priority; the multiple factors include task type, business importance and real-time performance requirements.

5. The fine-grained, interference-aware GPU scheduling method for deep learning as claimed in claim 1, characterized in that: When polling the task queue of a high priority application: If there is a CUDA kernel task to be executed, the CUDA kernel task to be executed is immediately submitted to the GPU hardware for execution; If there is no CUDA kernel task to be executed, continue to traverse the task queue of the next application.

6. The fine-grained, interference-aware GPU scheduling method for deep learning as claimed in claim 1, characterized in that: For low-priority applications, whether to submit CUDA kernel tasks to the GPU for scheduling is determined based on the interference score and resource requirements. Specifically: Setting a SM requirement threshold SM_TH to determine whether the SM requirement of a low-priority CUDA kernel task is lower than SM_TH; the SM is a streaming multiprocessor; Set the execution time threshold TI_TH to determine whether the execution time of the low-priority CUDA kernel task is less than TI_TH; Set the interference score threshold SCORE_TH to determine whether the interference score of the low-priority CUDA kernel task is lower than SCORE_TH; When all the above conditions are met, the CUDA kernel task of the current low-priority application is submitted to the GPU for scheduling.

7. The fine-grained, interference-aware GPU scheduling method for deep learning as claimed in claim 6, characterized in that: It also includes the calculation of the SM requirements for low-priority CUDA kernel tasks, specifically: ； in, Representation kernel The total number of blocks required, Indicates the number of thread blocks that each SM can support; Indicates rounding up; Indicates the number of SMs required for a CUDA kernel task.

8. Fine-grained, interference-aware GPU scheduling system for deep learning, featuring: include: The load analysis module is configured to: collect resource requirement information of the workload during the offline phase; The task generation module is configured to: generate a CUDA kernel task corresponding to each application and executed on the GPU according to the computing requirements of the application; specifically: use the deep learning framework to compile the operation of the application into a CUDA computing kernel, the CUDA computing kernel is the basic unit of GPU execution, and each CUDA computing kernel represents a specific computing task, forming a CUDA kernel task executed on the GPU; The kernel interception and management module is configured to: intercept the start request of the CUDA kernel task of each application, and cache the CUDA kernel task corresponding to the start request in the task queue corresponding to each application; The determination module is configured to: determine whether the priority of each application is a high priority or a low priority; The kernel task scheduling module is configured to: poll the task queue of each application, and directly submit the CUDA kernel task in the corresponding task queue to the GPU for scheduling for high-priority applications; for low-priority applications, determine whether to submit the CUDA kernel task to the GPU for scheduling based on the interference score and resource requirements; The interference score is used to measure the interference level when scheduling low-priority CUDA kernel tasks to the GPU: ； in, is the computational unit utilization of the kernel, is the kernel's memory bandwidth utilization, , are the coefficients of the core’s compute unit utilization and memory bandwidth utilization, respectively; Score the interference.

Images