GPU resource scheduling method and device, electronic equipment and readable medium

Abstract

The invention provides a GPU resource scheduling method and device, electronic equipment and a readable medium, and the method comprises the steps: writing a GPU resource allocation rule corresponding to an AI application into a rule library through a northbound interface of a GPU resource allocation controller, and monitoring and updating the allocated resource information of the AI application to an allocated resource library by the GPU resource allocation controller, when it is monitored that the process of the AI application is started, an uprobe probe calling the GPU drive API is inserted into the process, when the AI application calls the GPU drive API, the uprobe probe captures calling of the AI application to the GPU drive API and triggers running of a GPU resource allocation eBPF program, and when the GPU resource allocation eBPF program runs in an eBPF user mode, GPU resource allocation rules and allocated resource information corresponding to the AI application are read, and the GPU resource allocation rule and the allocated resource information corresponding to the AI application are allocated according to the GPU resource allocation rule and the allocated resource information. And carrying out GPU resource allocation on the AI application. According to the method, interception of calling a user state GPU driving API by an AI application is achieved through an uprobe mechanism of an eBPF, and dynamic allocation and isolation of GPU computing power and video memory resources are achieved through an eBPF program according to a GPU resource allocation rule.

Description

Technical Field

[0001] The present invention relates to the field of GPU virtualization technology, and in particular to a GPU resource scheduling method, a GPU resource scheduling device, an electronic device and a computer-readable medium. Background Art

[0002] With the rapid development of artificial intelligence technology, especially in the field of deep learning, the demand for computing resources in AI applications has increased dramatically. As a core component of high-performance computing, GPU can significantly improve AI computing efficiency. However, the resource requirements of AI applications usually do not require the capacity of the entire GPU. Therefore, how to make full use of existing GPU resources and maximize their efficiency has become an urgent problem to be solved. Virtualizing and splitting the physical GPU and allocating its resources to different tenants is an important means to achieve optimal resource utilization.

[0003] Currently, software-based GPU virtualization technology is limited by the closed source of GPU drivers and the differences in different GPU architectures and implementations. It is mainly implemented by intercepting API calls, such as Figure 1 As shown in the figure, there are mainly three levels of implementation:

[0004] 1. Modify the AI ​​training framework layer

[0005] By specifying or limiting GPU resource allocation at the AI ​​training framework layer, for example, by modifying training frameworks such as PyTorch or Tensorflow, shared GPUs for training tasks can be implemented at the framework layer, and GPU computing power and video memory for different training tasks can be allocated and limited. This method restricts user training scenarios, can only be applied to limited scenarios, and is highly bound to the modified training framework, so it has poor universality.

[0006] 2. Intercept user-mode GPU driver API calls:

[0007] The current mainstream method is to intercept GPU driver API calls, such as computing power and video memory API calls, through the LD_PRELOAD mechanism of the Linux operating system to allocate and restrict resources. The advantage of this method is that it is independent of application software such as frameworks, is universal, and meets the requirements of GPU virtualization in multi-tenant scenarios. The operating mechanism of the LD_PRELOAD mechanism determines that only the functions of the dynamic link library can be replaced and intercepted, so the disadvantage of this method is that it can only intercept dynamic GPU driver API link libraries, which is invalid for statically compiled AI applications. In addition, dynamic GPU resource allocation is not considered. After the AI ​​application is started, the allocated GPU resources can no longer be dynamically adjusted, which makes it difficult to meet the scenarios that require dynamic expansion and contraction of GPU resources.

[0008] 3. Intercept kernel-mode system calls:

[0009] This approach often requires adding an interception module to the kernel to implement GPU virtualization by intercepting kernel-user interfaces such as ioctl, mmap, read, and write in kernel state. Since GPU vendors close the source of driver code, kernel-state interception is often complex and usually requires reverse engineering and other technologies, which is also difficult to meet GPU heterogeneous scenarios. Summary of the invention

[0010] In view of the above problems, the present invention is proposed to provide a GPU resource scheduling method and a corresponding GPU resource scheduling device, an electronic device and a computer-readable medium that overcome the above problems or at least partially solve the above problems.

[0011] The present invention discloses a GPU resource scheduling method. The GPU resource management platform includes a GPU resource allocation controller, a GPU resource allocation eBPF program and a GPU resource allocation rule library. The GPU resource allocation rule library includes a GPU allocation rule library corresponding to an AI application and an AI application allocated resource library. The method includes:

[0012] The GPU resource allocation rules corresponding to the AI ​​application are written into the GPU allocation rule library corresponding to the AI ​​application through the northbound interface of the GPU resource allocation controller;

[0013] The GPU resource allocation controller monitors and updates the allocated resource information of the AI ​​application to the AI ​​application allocated resource library, monitors the process startup of the AI ​​application, and inserts an uprobe probe that calls the GPU driver API into the process when the process of the AI ​​application is started;

[0014] When the AI ​​application calls the GPU driver API to allocate computing power and video memory, the uprobe probe captures the AI ​​application's call to the GPU driver API and triggers the running of the GPU resource allocation eBPF program;

[0015] The GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and allocates GPU resources to the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application; the eBPF user state runtime is a sandbox that runs eBPF programs and eBPF maps in user state space.

[0016] Optionally, the GPU driver API includes a GPU driver computing power allocation API, the GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and performs GPU resource allocation for the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application, including:

[0017] The GPU resource allocation eBPF program triggered by calling the GPU driver computing power allocation API runs in the eBPF user state runtime, and reads the GPU resource computing power quota and the allocated computing power size corresponding to the AI ​​application from the GPU resource allocation rule library;

[0018] Determine whether the allocated computing power exceeds the GPU resource computing power quota;

[0019] If yes, hibernate the computing task to be executed until the computing task in execution releases computing power within the computing power quota of the GPU resource, and call the GPU driver computing power allocation API to execute the computing task to be executed;

[0020] If not, determine whether the amount of the allocated computing power plus the computing power required for the computing task to be executed exceeds the GPU resource computing power quota;

[0021] If not, the GPU driver computing power allocation API is called to execute the computing task to be executed.

[0022] Optionally, the GPU driver API includes a GPU driver memory allocation API, the GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rule and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and performs GPU resource allocation for the AI ​​application according to the GPU resource allocation rule and allocated resource information corresponding to the AI ​​application, including:

[0023] The GPU resource allocation eBPF program triggered by calling the GPU driver memory allocation API runs in the eBPF user mode, and reads the GPU resource memory quota and the allocated memory size corresponding to the AI ​​application from the GPU resource allocation rule library;

[0024] Determine whether the allocated video memory size exceeds the GPU resource video memory quota;

[0025] If so, an error indicating that the video memory allocation has exceeded the limit is returned to the AI ​​application;

[0026] If not, determining whether the allocated video memory size plus the currently applied video memory size exceeds the GPU resource video memory quota;

[0027] If not, the GPU driver video memory allocation API is called to allocate video memory space corresponding to the currently requested video memory size.

[0028] Optionally, the method further comprises:

[0029] During the operation of the AI ​​application, the GPU resource allocation rule of the AI ​​application is dynamically modified through the northbound interface of the GPU resource allocation controller, and the new GPU resource allocation rule is applied to the AI ​​application in real time.

[0030] Optionally, the method further comprises:

[0031] After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource computing power quota is greater than the original GPU resource computing power quota, determine whether the amount of the allocated computing power plus the computing power required by the dormant computing task exceeds the new GPU resource computing power quota;

[0032] If not, the GPU driver computing power allocation API is called to execute the dormant computing task.

[0033] Optionally, the method further comprises:

[0034] After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource computing power quota is less than the original GPU resource computing power quota, determining whether the allocated computing power exceeds the new GPU resource computing power quota;

[0035] If so, the computing task to be executed is dormant until the computing task being executed releases computing power to the new GPU resource computing power quota range, and the GPU driver computing power allocation API is called to execute the computing task to be executed.

[0036] Optionally, the method further comprises:

[0037] After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource memory quota is greater than or less than the original GPU resource memory quota, determining whether the allocated memory size exceeds the new GPU resource memory quota;

[0038] If so, an error indicating that the video memory allocation has exceeded the limit is returned to the AI ​​application;

[0039] If not, determining whether the allocated video memory size plus the currently applied video memory size exceeds the new GPU resource video memory quota;

[0040] If not, the GPU driver video memory allocation API is called to allocate video memory space corresponding to the video memory size currently requested.

[0041] The present invention also discloses a GPU resource scheduling device, wherein the GPU resource management platform includes a GPU resource allocation controller, a GPU resource allocation eBPF program, and a GPU resource allocation rule library, wherein the GPU resource allocation rule library includes a GPU allocation rule library corresponding to an AI application and an AI application allocated resource library, and the device includes:

[0042] A rule writing module is used to write the GPU resource allocation rules corresponding to the AI ​​application into the GPU allocation rule library corresponding to the AI ​​application through the northbound interface of the GPU resource allocation controller;

[0043] An allocated resource update and probe insertion module, which is used by the GPU resource allocation controller to monitor and update the allocated resource information of the AI ​​application to the AI ​​application allocated resource library, and to monitor the process startup of the AI ​​application, and when the process of the AI ​​application is started, insert an uprobe probe that calls the GPU driver API into the process;

[0044] A GPU resource allocation eBPF program triggering module, used for the uprobe probe to capture the AI ​​application's call to the GPU driver API and trigger the running of the GPU resource allocation eBPF program when the AI ​​application calls the GPU driver API to allocate computing power and video memory;

[0045] A GPU resource allocation module is used for the GPU resource allocation eBPF program to run in the eBPF user state runtime, and read the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and allocate GPU resources to the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application; the eBPF user state runtime is a sandbox for running eBPF programs and eBPF maps in user state space.

[0046] Optionally, the GPU driver API includes a GPU driver computing power allocation API, and the GPU resource allocation module includes:

[0047] The computing power quota and allocated computing power reading submodule is used to call the GPU resource allocation eBPF program triggered by the GPU driver computing power allocation API to run in the eBPF user state, and read the GPU resource computing power quota and the allocated computing power size corresponding to the AI ​​application from the GPU resource allocation rule library;

[0048] A first computing power excess judgment submodule is used to judge whether the allocated computing power exceeds the GPU resource computing power quota;

[0049] A dormancy submodule, for, if yes, dormantly placing the computing task to be executed until the computing task being executed releases computing power within the computing power quota of the GPU resource, and calling the GPU driver computing power allocation API to execute the computing task to be executed;

[0050] A second computing power excess judgment submodule is used to judge whether the amount of the allocated computing power plus the computing power required for the computing task to be executed exceeds the GPU resource computing power quota;

[0051] The computing task execution submodule to be executed is used to call the GPU driver computing power allocation API to execute the computing task to be executed if not.

[0052] Optionally, the GPU driver API includes a GPU driver video memory allocation API, and the GPU resource allocation module includes:

[0053] The memory quota and allocated memory reading submodule is used to call the GPU resource allocation eBPF program triggered by the GPU driver memory allocation API to run in the eBPF user mode, and read the GPU resource memory quota and allocated memory size corresponding to the AI ​​application from the GPU resource allocation rule library;

[0054] A first video memory excess judgment submodule is used to judge whether the allocated video memory size exceeds the GPU resource video memory quota;

[0055] an excess feedback submodule, for returning an error that the video memory allocation has exceeded the limit to the AI ​​application if so;

[0056] The second video memory excess judgment submodule is used to judge whether the allocated video memory size plus the currently applied video memory size exceeds the GPU resource video memory quota;

[0057] The video memory allocation submodule is used to call the GPU driver video memory allocation API to allocate video memory space corresponding to the video memory size currently applied for if no.

[0058] Optionally, the device further comprises:

[0059] The rule modification application module is used to dynamically modify the GPU resource allocation rules of the AI ​​application through the northbound interface of the GPU resource allocation controller during the operation of the AI ​​application, and apply the new GPU resource allocation rules to the AI ​​application in real time.

[0060] Optionally, the device further comprises:

[0061] A computing power quota expansion module is used to determine whether the amount of the allocated computing power plus the computing power required by the dormant computing task exceeds the new GPU resource computing power quota if the new GPU resource computing power quota is greater than the original GPU resource computing power quota after the new GPU resource allocation rule is applied to the AI ​​application in real time;

[0062] The dormant computing task execution module is used to call the GPU driver computing power allocation API to execute the dormant computing task if not.

[0063] Optionally, the device further comprises:

[0064] A computing power quota reduction module is used to determine whether the allocated computing power exceeds the new GPU resource computing power quota if the new GPU resource computing power quota is less than the original GPU resource computing power quota after the new GPU resource allocation rule is applied to the AI ​​application in real time;

[0065] The scaling down hibernation module is used to hibernate the computing tasks to be executed until the computing tasks being executed release computing power to the new GPU resource computing power quota range, and call the GPU driver computing power allocation API to execute the computing tasks to be executed.

[0066] Optionally, the device further comprises:

[0067] A video memory expansion and contraction module, configured to determine whether the allocated video memory size exceeds the new GPU resource video memory quota if the new GPU resource video memory quota is greater than or less than the original GPU resource video memory quota after the new GPU resource allocation rule is applied to the AI ​​application in real time;

[0068] an over-capacity expansion and contraction feedback module, configured to return an error that the video memory allocation has exceeded the limit to the AI ​​application if so;

[0069] The expansion / contraction capacity excess judgment module is used to judge whether the allocated video memory size plus the currently applied video memory size exceeds the new GPU resource video memory quota;

[0070] The expansion and contraction video memory allocation module is used to call the GPU driver video memory allocation API to allocate video memory space corresponding to the video memory size currently applied for if no.

[0071] The present invention also discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0072] The memory is used to store computer programs;

[0073] The processor is used to implement the GPU resource scheduling method of the present invention when executing the program stored in the memory.

[0074] The present invention also discloses one or more computer-readable media, on which instructions are stored, and when executed by one or more processors, the processors execute the GPU resource scheduling method as described in the present invention.

[0075] The present invention includes the following advantages:

[0076] The GPU resource scheduling method of the present invention writes the GPU resource allocation rules corresponding to the AI ​​application into the GPU allocation rule library corresponding to the AI ​​application through the northbound interface of the GPU resource allocation controller. The GPU resource allocation controller monitors and updates the allocated resource information of the AI ​​application to the AI ​​application allocated resource library, monitors the process startup of the AI ​​application, and inserts an uprobe probe that calls the GPU driver API into the process when the AI ​​application starts. When the AI ​​application calls the GPU driver API to allocate computing power and video memory, the uprobe probe captures the AI ​​application's call to the GPU driver API and triggers the running of the GPU resource allocation eBPF program. The GPU resource allocation eBPF program runs in the eBPF user mode, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and performs GPU resource allocation for the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application. The method of the present invention deploys a GPU resource allocation controller, a GPU resource allocation eBPF program and a GPU resource allocation rule base on a GPU node, intercepts AI applications calling a user-mode GPU driver API using the uprobe mechanism of eBPF, and dynamically allocates and isolates GPU computing power and video memory resources according to GPU resource allocation rules through the GPU resource allocation eBPF program. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a schematic diagram of GPU resource calls;

[0078] Figure 2 is a flowchart of a GPU resource scheduling method provided by an embodiment of the present invention;

[0079] Figure 3 is a schematic diagram of component deployment provided by an embodiment of the present invention;

[0080] Figure 4 is a flow chart of dynamically allocating and isolating GPU resources provided by an embodiment of the present invention;

[0081] Figure 5It is a structural block diagram of a GPU resource scheduling device provided by an embodiment of the present invention;

[0082] Figure 6 is a block diagram of an electronic device provided by an embodiment of the present invention;

[0083] Figure 7 It is a schematic diagram of a computer-readable medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0084] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0085] Reference Figure 2 , shows a flowchart of a GPU resource scheduling method provided in an embodiment of the present invention, which may specifically include the following steps:

[0086] Step 201, writing the GPU resource allocation rules corresponding to the AI ​​application into the GPU allocation rule library corresponding to the AI ​​application through the northbound interface of the GPU resource allocation controller;

[0087] Step 202, the GPU resource allocation controller monitors and updates the allocated resource information of the AI ​​application to the AI ​​application allocated resource library, monitors the process startup of the AI ​​application, and inserts an uprobe probe that calls the GPU driver API into the process when the AI ​​application process starts;

[0088] Step 203, when the AI ​​application calls the GPU driver API to allocate computing power and video memory, the uprobe probe captures the AI ​​application's call to the GPU driver API and triggers the running of the GPU resource allocation eBPF program;

[0089] Step 204, the GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and allocates GPU resources to the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application; the eBPF user state runtime is a sandbox that runs eBPF programs and eBPF maps in user state space.

[0090] Reference Figure 3The embodiment of the present invention deploys components such as a GPU resource allocation controller, a GPU resource allocation eBPF program, and a GPU resource allocation rule library on a GPU server node, and uses the uprobe mechanism of eBPF to intercept AI applications calling user-mode GPU driver APIs. A method is proposed in which the eBPF program dynamically allocates and isolates GPU computing power and video memory resources according to GPU resource allocation rules, so as to solve the problems existing in the current software-based GPU virtualization technology.

[0091] Among them, the functions of components such as GPU resource allocation controller, GPU resource allocation eBPF program and GPU resource allocation rule library are as follows:

[0092] GPU resource allocation controller: runs on the GPU server as a resident process. It provides a northbound Http Restful interface, which is responsible for adding, deleting, and modifying GPU allocation rules corresponding to tenant AI applications, and writing the rules to the GPU resource allocation rule library. It is also responsible for monitoring the process startup of the AI ​​application, and inserting uprobe probes that call the computing power allocation API and the video memory allocation API when the process starts. The uprobe probe will trigger the running of the GPU resource allocation eBPF program when the AI ​​application calls the computing power allocation API and the video memory allocation API. The GPU resource allocation controller will use a monitoring method to call the GPU manufacturer's monitoring interface to continuously update the GPU usage of the AI ​​application to the computing power allocation status of the AI ​​application's allocated resource library.

[0093] GPU resource allocation eBPF program: When the uprobe probe triggers the GPU resource allocation program to run, it will run in the eBPF user-mode runtime. The eBPF user-mode runtime is a sandbox that runs eBPF programs and eBPF maps in user space. It provides the same security checks and execution capabilities of eBPF programs as the kernel-mode eBPF runtime.

[0094] GPU resource allocation rule library: runs in eBPF user mode as BPF_MAP_TYPE_HASH type of eBPF maps. It is divided into two libraries: GPU allocation rule library corresponding to AI applications and resource library allocated to AI applications.

[0095] Reference Figure 4 The process of GPU resource management platform dynamically allocating and isolating GPU resources for a tenant's AI application is as follows:

[0096] 1. Writing GPU resource allocation rules and updating allocated resource information: The GPU resource management platform allocates GPU resources to a tenant's AI application, calls the northbound interface of the GPU resource allocation controller, and writes the allocation rules of the GPU resources corresponding to the AI ​​application into the GPU resource allocation rule library. The rules are based on computing power and video memory quotas. For example, the rule format is: "AI application 1 <-> Nvidia A100 / computing power 15% / video memory 2GB", which means that the GPU resource quota corresponding to AI application 1 is 15% of the computing power and 2GB of video memory of a NividaA100 GPU. The GPU resource allocation controller will use a monitoring method to call the GPU manufacturer's monitoring interface to continuously update the GPU usage of the AI ​​application to the computing power allocation status of the AI ​​application's allocated resource library.

[0097] 2. Insert probes: When the tenant's AI application process starts, the GPU resource allocation controller will insert uprobe probes that call the computing power allocation API and the video memory allocation API in the process. These APIs can include user-mode driver layer APIs of different GPU manufacturers such as Cuda, OpenCL, and HIP. For example: cuda's computing power APIs: cuLaunch, cuLaunchKernel, cuLaunchGrid, etc., and video memory allocation APIs: cuMemAlloc, cuMemAllocPitch, cuFree, etc.

[0098] 3. Probe capture and processing:

[0099] When the tenant's AI application process starts and calls the GPU driver API to allocate computing power and video memory, the inserted uprobe probe will capture these calls. After the probe captures, it triggers the GPU resource allocation eBPF program to run. When the program runs in eBPF user mode, it reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application in the GPU resource allocation rule library, and allocates GPU computing power and video memory to the AI ​​application based on these read GPU resource allocation rules and allocated resource information.

[0100] In the embodiment of the present invention, GPU resource allocation eBPF program and GPU resource allocation rule base are run in eBPF user mode, and the computing power and memory resources of different AI applications of different tenants are restricted to achieve GPU resource isolation between different tenants:

[0101] 1) Computing power isolation: Each tenant can only use the computing power share allocated to him / her. The excess will be restricted or delayed to ensure that the resources of other tenants are not occupied.

[0102] 2) Video memory isolation: Each tenant can only allocate and use a preset amount of video memory. Any excess will be rejected to ensure that video memory resources are not occupied by other tenants.

[0103] The user-mode eBPF runtime provides an isolated execution environment for GPU resource allocation eBPF programs and GPU resource allocation rule bases. In this environment, different resource limits can be set for different tenants, ensuring GPU resource isolation between tenants. Within the resource limit, the GPU computing power and video memory resource allocation can be dynamically adjusted to meet the needs of different AI applications.

[0104] In addition, the allocation and isolation process of the present invention is applicable to both homogeneous GPU scenarios and heterogeneous GPU scenarios, and can achieve unified management and control of different GPU manufacturers and models.

[0105] In one embodiment of the present invention, the GPU driver API includes a GPU driver computing power allocation API, the GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and performs GPU resource allocation for the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application, including:

[0106] The GPU resource allocation eBPF program triggered by calling the GPU driver computing power allocation API runs in the eBPF user state runtime, and reads the GPU resource computing power quota and the allocated computing power size corresponding to the AI ​​application from the GPU resource allocation rule library;

[0107] Determine whether the allocated computing power exceeds the GPU resource computing power quota;

[0108] If yes, hibernate the computing task to be executed until the computing task in execution releases computing power within the computing power quota of the GPU resource, and call the GPU driver computing power allocation API to execute the computing task to be executed;

[0109] If not, determine whether the amount of the allocated computing power plus the computing power required for the computing task to be executed exceeds the GPU resource computing power quota;

[0110] If not, the GPU driver computing power allocation API is called to execute the computing task to be executed.

[0111] In one embodiment of the present invention, the GPU driver API includes a GPU driver memory allocation API, the GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and performs GPU resource allocation for the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application, including:

[0112] The GPU resource allocation eBPF program triggered by calling the GPU driver memory allocation API runs in the eBPF user mode, and reads the GPU resource memory quota and the allocated memory size corresponding to the AI ​​application from the GPU resource allocation rule library;

[0113] Determine whether the allocated video memory size exceeds the GPU resource video memory quota;

[0114] If so, an error indicating that the video memory allocation has exceeded the limit is returned to the AI ​​application;

[0115] If not, determining whether the allocated video memory size plus the currently applied video memory size exceeds the GPU resource video memory quota;

[0116] If not, the GPU driver video memory allocation API is called to allocate video memory space corresponding to the currently requested video memory size.

[0117] The GPU resource allocation eBPF program processes the GPU computing power and video memory quota of the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application read as follows:

[0118] 1) GPU computing power limit: The GPU resource allocation eBPF program triggered by calling the GPU driver computing power allocation API will temporarily hibernate the current computing task of the AI ​​application until the existing computing task releases computing power within the quota range, and then call the original GPU driver computing power allocation API. If it does not exceed the quota, the computing power required for the current computing task to be executed is accumulated. If the accumulated computing power is still within the quota range, the original GPU driver computing power allocation API will be called to execute the computing task.

[0119] 2) GPU memory limit: The GPU resource allocation eBPF program is triggered by calling the GPU driver memory allocation API. If the memory size allocated to the AI ​​application exceeds the GPU memory quota, the eBPF program returns an error indicating that the memory allocation has exceeded the limit to the AI ​​application. If the current memory size applied for plus the memory size allocated to the AI ​​application does not exceed the GPU memory quota, the eBPF program calls the original GPU driver memory allocation API.

[0120] The GPU resource allocation controller monitors the usage of GPU resources. The GPU resource allocation eBPF program can dynamically adjust the GPU computing power and memory resource allocation for AI applications based on the usage of GPU resources and the GPU resource allocation rules corresponding to the AI ​​application, significantly improving the GPU resource utilization and maximizing its performance.

[0121] In one embodiment of the present invention, the method further comprises:

[0122] During the operation of the AI ​​application, the GPU resource allocation rule of the AI ​​application is dynamically modified through the northbound interface of the GPU resource allocation controller, and the new GPU resource allocation rule is applied to the AI ​​application in real time.

[0123] In one embodiment of the present invention, the method further comprises:

[0124] After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource computing power quota is greater than the original GPU resource computing power quota, determine whether the amount of the allocated computing power plus the computing power required by the dormant computing task exceeds the new GPU resource computing power quota;

[0125] If not, the GPU driver computing power allocation API is called to execute the dormant computing task.

[0126] In one embodiment of the present invention, the method further comprises:

[0127] After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource computing power quota is less than the original GPU resource computing power quota, determining whether the allocated computing power exceeds the new GPU resource computing power quota;

[0128] If so, the computing task to be executed is dormant until the computing task being executed releases computing power to the new GPU resource computing power quota range, and the GPU driver computing power allocation API is called to execute the computing task to be executed.

[0129] In one embodiment of the present invention, the method further comprises:

[0130] After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource memory quota is greater than or less than the original GPU resource memory quota, determining whether the allocated memory size exceeds the new GPU resource memory quota;

[0131] If so, an error indicating that the video memory allocation has exceeded the limit is returned to the AI ​​application;

[0132] If not, determining whether the allocated video memory size plus the currently applied video memory size exceeds the new GPU resource video memory quota;

[0133] If not, the GPU driver video memory allocation API is called to allocate video memory space corresponding to the video memory size currently requested.

[0134] In the GPU resource management platform, in order to adapt to the changes in resource requirements of different AI applications, the GPU resource allocation rules of AI applications can be dynamically modified by calling the northbound interface of the GPU resource allocation controller. The modified allocation rules are automatically applied to the currently running AI applications, thereby realizing the dynamic expansion or reduction of GPU resources. After receiving the new resource allocation rules, the GPU resource allocation controller writes them into the GPU resource allocation rule library. The new rules replace the old rules to ensure that subsequent resource allocation and use conform to the new configuration.

[0135] 1) If the computing power quota is expanded, the originally restricted computing tasks can be released and run immediately to make full use of the new computing power resource quota. If the computing power quota is reduced, when the GPU resource allocation eBPF program triggered by the GPU driver computing power allocation API is called, the current computing tasks to be executed by the AI ​​application will be temporarily dormant until the existing computing tasks release computing power within the quota range, and then the GPU driver computing power allocation API will be called to execute the computing tasks to be executed.

[0136] 2) If the memory quota is expanded or reduced, the memory limit requirements can be met according to the new quota when the GPU driver memory allocation API is called to trigger the GPU resource allocation eBPF program.

[0137] By dynamically adjusting the GPU resource allocation rules during the operation of the AI ​​application and applying new GPU resource allocation rules in real time, the allocated GPU resources can be dynamically adjusted after the AI ​​application is started to meet the scenarios that require dynamic expansion and contraction of GPU resources.

[0138] The present invention realizes fine-grained dynamic allocation and isolation of GPU resources for multiple tenants through eBPF, improves GPU resource utilization, and has high universality, compatibility and scalability, is suitable for a variety of complex and GPU heterogeneous application scenarios, and solves the problem of GPU virtualization implemented by current mainstream software.

[0139] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0140] Reference Figure 5 , shows a structural block diagram of a GPU resource scheduling device provided in an embodiment of the present invention, wherein the GPU resource management platform includes a GPU resource allocation controller, a GPU resource allocation eBPF program, and a GPU resource allocation rule library, wherein the GPU resource allocation rule library includes a GPU allocation rule library corresponding to an AI application and an AI application allocated resource library, and specifically may include the following modules:

[0141] A rule writing module 501 is used to write the GPU resource allocation rules corresponding to the AI ​​application into the GPU allocation rule library corresponding to the AI ​​application through the northbound interface of the GPU resource allocation controller;

[0142] An allocated resource update and probe insertion module 502 is used for the GPU resource allocation controller to monitor and update the allocated resource information of the AI ​​application to the AI ​​application allocated resource library, and to monitor the process startup of the AI ​​application, and when the process of the AI ​​application is started, to insert an uprobe probe that calls the GPU driver API into the process;

[0143] A GPU resource allocation eBPF program triggering module 503, which is used for the uprobe probe to capture the call of the AI ​​application to the GPU driver API and trigger the running of the GPU resource allocation eBPF program when the AI ​​application calls the GPU driver API to allocate computing power and video memory;

[0144] The GPU resource allocation module 504 is used for the GPU resource allocation eBPF program to run in the eBPF user state runtime, and read the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and allocate GPU resources to the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application; the eBPF user state runtime is a sandbox for running eBPF programs and eBPF maps in user state space.

[0145] Optionally, the GPU driver API includes a GPU driver computing power allocation API, and the GPU resource allocation module includes:

[0146] The computing power quota and allocated computing power reading submodule is used to call the GPU resource allocation eBPF program triggered by the GPU driver computing power allocation API to run in the eBPF user state, and read the GPU resource computing power quota and the allocated computing power size corresponding to the AI ​​application from the GPU resource allocation rule library;

[0147] A first computing power excess judgment submodule is used to judge whether the allocated computing power exceeds the GPU resource computing power quota;

[0148] A dormancy submodule, for, if yes, dormantly placing the computing task to be executed until the computing task being executed releases computing power within the computing power quota of the GPU resource, and calling the GPU driver computing power allocation API to execute the computing task to be executed;

[0149] A second computing power excess judgment submodule is used to judge whether the amount of the allocated computing power plus the computing power required for the computing task to be executed exceeds the GPU resource computing power quota;

[0150] The computing task execution submodule to be executed is used to call the GPU driver computing power allocation API to execute the computing task to be executed if not.

[0151] Optionally, the GPU driver API includes a GPU driver video memory allocation API, and the GPU resource allocation module includes:

[0152] The memory quota and allocated memory reading submodule is used to call the GPU resource allocation eBPF program triggered by the GPU driver memory allocation API to run in the eBPF user mode, and read the GPU resource memory quota and allocated memory size corresponding to the AI ​​application from the GPU resource allocation rule library;

[0153] A first video memory excess judgment submodule is used to judge whether the allocated video memory size exceeds the GPU resource video memory quota;

[0154] an excess feedback submodule, for returning an error that the video memory allocation has exceeded the limit to the AI ​​application if so;

[0155] The second video memory excess judgment submodule is used to judge whether the allocated video memory size plus the currently applied video memory size exceeds the GPU resource video memory quota;

[0156] The video memory allocation submodule is used to call the GPU driver video memory allocation API to allocate video memory space corresponding to the video memory size currently applied for if no.

[0157] Optionally, the device further comprises:

[0158] The rule modification application module is used to dynamically modify the GPU resource allocation rules of the AI ​​application through the northbound interface of the GPU resource allocation controller during the operation of the AI ​​application, and apply the new GPU resource allocation rules to the AI ​​application in real time.

[0159] Optionally, the device further comprises:

[0160] A computing power quota expansion module is used to determine whether the amount of the allocated computing power plus the computing power required by the dormant computing task exceeds the new GPU resource computing power quota if the new GPU resource computing power quota is greater than the original GPU resource computing power quota after the new GPU resource allocation rule is applied to the AI ​​application in real time;

[0161] The dormant computing task execution module is used to call the GPU driver computing power allocation API to execute the dormant computing task if not.

[0162] Optionally, the device further comprises:

[0163] A computing power quota reduction module is used to determine whether the allocated computing power exceeds the new GPU resource computing power quota if the new GPU resource computing power quota is less than the original GPU resource computing power quota after the new GPU resource allocation rule is applied to the AI ​​application in real time;

[0164] The scaling down hibernation module is used to hibernate the computing tasks to be executed until the computing tasks being executed release computing power to the new GPU resource computing power quota range, and call the GPU driver computing power allocation API to execute the computing tasks to be executed.

[0165] Optionally, the device further comprises:

[0166] A video memory expansion and contraction module, configured to determine whether the allocated video memory size exceeds the new GPU resource video memory quota if the new GPU resource video memory quota is greater than or less than the original GPU resource video memory quota after the new GPU resource allocation rule is applied to the AI ​​application in real time;

[0167] an over-capacity expansion and contraction feedback module, configured to return an error that the video memory allocation has exceeded the limit to the AI ​​application if so;

[0168] The expansion / contraction capacity excess judgment module is used to judge whether the allocated video memory size plus the currently applied video memory size exceeds the new GPU resource video memory quota;

[0169] The expansion and contraction video memory allocation module is used to call the GPU driver video memory allocation API to allocate video memory space corresponding to the video memory size currently applied for if no.

[0170] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0172] Memory 603, used for storing computer programs;

[0173] The processor 601 is used to implement the GPU resource scheduling method described in the above embodiment when executing the program stored in the memory 603.

[0174] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0175] The communication interface is used for communication between the above terminal and other devices.

[0176] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0177] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0178] like Figure 7 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 701 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the GPU resource scheduling method described in the above embodiment.

[0179] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer executes the GPU resource scheduling method described in the above embodiment.

[0180] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0181] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0182] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A GPU resource scheduling method, characterized in that: The GPU resource management platform includes a GPU resource allocation controller, a GPU resource allocation eBPF program, and a GPU resource allocation rule library. The GPU resource allocation rule library includes a GPU allocation rule library corresponding to an AI application and an AI application allocated resource library. The method includes: The GPU resource allocation rules corresponding to the AI ​​application are written into the GPU allocation rule library corresponding to the AI ​​application through the northbound interface of the GPU resource allocation controller; The GPU resource allocation controller monitors and updates the allocated resource information of the AI ​​application to the AI ​​application allocated resource library, monitors the process startup of the AI ​​application, and inserts an uprobe probe that calls the GPU driver API into the process when the process of the AI ​​application is started; When the AI ​​application calls the GPU driver API to allocate computing power and video memory, the uprobe probe captures the AI ​​application's call to the GPU driver API and triggers the running of the GPU resource allocation eBPF program; The GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and allocates GPU resources to the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application; the eBPF user state runtime is a sandbox that runs eBPF programs and eBPF maps in user state space.

2. The method according to claim 1, characterized in that The GPU driver API includes a GPU driver computing power allocation API, wherein the GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and performs GPU resource allocation for the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application, including: The GPU resource allocation eBPF program triggered by calling the GPU driver computing power allocation API runs in the eBPF user state runtime, and reads the GPU resource computing power quota and the allocated computing power size corresponding to the AI ​​application from the GPU resource allocation rule library; Determine whether the allocated computing power exceeds the GPU resource computing power quota; If yes, hibernate the computing task to be executed until the computing task in execution releases computing power within the computing power quota of the GPU resource, and call the GPU driver computing power allocation API to execute the computing task to be executed; If not, determine whether the amount of the allocated computing power plus the computing power required for the computing task to be executed exceeds the GPU resource computing power quota; If not, the GPU driver computing power allocation API is called to execute the computing task to be executed.

3. The method according to claim 1, characterized in that The GPU driver API includes a GPU driver memory allocation API, wherein the GPU resource allocation eBPF program runs in the eBPF user state runtime, and reads the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and performs GPU resource allocation for the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application, including: The GPU resource allocation eBPF program triggered by calling the GPU driver memory allocation API runs in the eBPF user mode, and reads the GPU resource memory quota and the allocated memory size corresponding to the AI ​​application from the GPU resource allocation rule library; Determine whether the allocated video memory size exceeds the GPU resource video memory quota; If so, an error indicating that the video memory allocation has exceeded the limit is returned to the AI ​​application; If not, determining whether the allocated video memory size plus the currently applied video memory size exceeds the GPU resource video memory quota; If not, the GPU driver video memory allocation API is called to allocate video memory space corresponding to the currently requested video memory size.

4. The method according to claim 1, characterized in that The method further comprises: During the operation of the AI ​​application, the GPU resource allocation rule of the AI ​​application is dynamically modified through the northbound interface of the GPU resource allocation controller, and the new GPU resource allocation rule is applied to the AI ​​application in real time.

5. The method according to claim 4, characterized in that The method further comprises: After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource computing power quota is greater than the original GPU resource computing power quota, determine whether the amount of the allocated computing power plus the computing power required by the dormant computing task exceeds the new GPU resource computing power quota; If not, the GPU driver computing power allocation API is called to execute the dormant computing task.

6. The method according to claim 4, characterized in that The method further comprises: After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource computing power quota is less than the original GPU resource computing power quota, determining whether the allocated computing power exceeds the new GPU resource computing power quota; If so, the computing task to be executed is dormant until the computing task being executed releases computing power to the new GPU resource computing power quota range, and the GPU driver computing power allocation API is called to execute the computing task to be executed.

7. The method according to claim 4, characterized in that The method further comprises: After the new GPU resource allocation rule is applied to the AI ​​application in real time, if the new GPU resource memory quota is greater than or less than the original GPU resource memory quota, determining whether the allocated memory size exceeds the new GPU resource memory quota; If so, an error indicating that the video memory allocation has exceeded the limit is returned to the AI ​​application; If not, determining whether the allocated video memory size plus the currently applied video memory size exceeds the new GPU resource video memory quota; If not, the GPU driver video memory allocation API is called to allocate video memory space corresponding to the video memory size currently requested.

8. A GPU resource scheduling device, characterized in that: The GPU resource management platform includes a GPU resource allocation controller, a GPU resource allocation eBPF program, and a GPU resource allocation rule library. The GPU resource allocation rule library includes a GPU allocation rule library corresponding to AI applications and an AI application allocated resource library. The device includes: A rule writing module is used to write the GPU resource allocation rules corresponding to the AI ​​application into the GPU allocation rule library corresponding to the AI ​​application through the northbound interface of the GPU resource allocation controller; An allocated resource update and probe insertion module, which is used by the GPU resource allocation controller to monitor and update the allocated resource information of the AI ​​application to the AI ​​application allocated resource library, and to monitor the process startup of the AI ​​application, and when the process of the AI ​​application is started, insert an uprobe probe that calls the GPU driver API into the process; A GPU resource allocation eBPF program triggering module, used for the uprobe probe to capture the AI ​​application's call to the GPU driver API and trigger the running of the GPU resource allocation eBPF program when the AI ​​application calls the GPU driver API to allocate computing power and video memory; A GPU resource allocation module is used for the GPU resource allocation eBPF program to run in the eBPF user state runtime, and read the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application from the GPU resource allocation rule library, and allocate GPU resources to the AI ​​application according to the GPU resource allocation rules and allocated resource information corresponding to the AI ​​application; the eBPF user state runtime is a sandbox for running eBPF programs and eBPF maps in user state space.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the GPU resource scheduling method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. One or more computer-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the GPU resource scheduling method according to any one of claims 1 to 7.

Images