User mode CUDA program execution method and system, storage medium and equipment

Abstract

The invention discloses a user mode CUDA program execution method and system, a storage medium and equipment, and belongs to the technical field of computers. Loading a hijacking layer when a CUDA program is started based on a dynamic hijacking technology; the hijacking layer registers a collaborative agent to the central scheduler according to the vGPU instance of the CUDA program; the central scheduler periodically issues vGPU instance future resource configuration information to the collaborative agent based on a collaborative super-division technology, and the resource configuration information comprises a computing power time window and a resource quota; the collaborative agent stores the resource configuration information to a resource configuration table; and when a process in the CUDA program initiates kernel calling, the hijacking layer queries the resource configuration table by using the collaborative agent, judges whether the kernel calling is matched with the computing power time window and the resource quota or not, and controls the kernel calling according to a judgment result. According to the invention, fine-grained resource control can be realized, the resource utilization rate is improved, resources cannot be maliciously seized, and the isolation during sharing operation is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, system, storage medium, and device for executing user-mode CUDA programs. Background Technology

[0002] In the fields of cloud computing and artificial intelligence computing, to support concurrent execution of multiple users or tasks on a single physical graphics processing unit (GPU) and improve hardware resource utilization, the industry has proposed various GPU virtualization and resource management technologies. These technologies aim to resolve the contradiction between the exclusive access requirements of Compute Unified Device Architecture (CUDA) programs to GPU resources and efficient resource sharing. However, existing mainstream technologies have significant shortcomings in terms of flexibility, isolation, and utilization when implementing CUDA program execution and GPU resource scheduling, specifically in the following three aspects: First, there are hardware-level virtualization technologies, such as NVIDIA's Multi-Instance GPU (MIG) or Virtual GPU (vGPU). This technology provides strict spatial isolation and reliable Quality of Service (QoS) guarantees for each vGPU instance by physically partitioning the GPU's hardware units, such as the Streaming Multiprocessor (SM) and memory controller. In this model, each CUDA program is bound to a fixed vGPU instance to run, and its execution environment is isolated at the hardware level. However, the fundamental drawback of this approach is that it allocates fixed and exclusive hardware resources to each instance. This leads to severe resource fragmentation; when the resources of one instance are not fully utilized, its idle resources cannot be reclaimed by other instances, resulting in low overall GPU resource utilization and hindering dynamic super-division and elastic scaling of resources.

[0003] Secondly, there are shared execution technologies, such as NVIDIA's Multi-Process Service (MPS). Unlike hardware virtualization, MPS allows multiple CUDA processes to share the same GPU context, improving resource utilization through time-multiplexed SM units. In this mode, the computational tasks of multiple CUDA programs can be executed alternately on the GPU. However, in the MPS architecture, all processes share the same memory space and error states. This means that any abnormal behavior of a CUDA program (such as memory access out of bounds or kernel execution failure) will pollute the shared context. An anomaly in one process may affect the entire service, causing all processes sharing that context to be terminated or exhibit unpredictable behavior. Its poor isolation makes it difficult to apply in multi-tenant production environments that require stability.

[0004] Secondly, there are coarse-grained resource limiting techniques based on the operating system, such as control groups (Cgroups) or container resource requests / limits. These techniques are commonly used in containerized environments and can set limits on CPU and memory resources. However, their ability to manage GPU resources is extremely limited. While they can limit container access to the GPU, they cannot precisely control fine-grained resources within the GPU, such as the utilization of SM units and memory access bandwidth. Therefore, they cannot prevent CUDA programs within a single container from exhausting GPU computing power by launching a large number of compute kernels or initiating high-bandwidth memory operations, thereby affecting the performance of CUDA programs in other containers on the same GPU, and cannot achieve true, fine-grained QoS guarantees. Summary of the Invention

[0005] This application provides a method, system, storage medium, and device for executing user-mode CUDA programs, addressing the problems of resource fragmentation and low utilization, poor isolation during shared runtime, and the inability to perform fine-grained resource control. The technical solution is as follows: According to a first aspect of this application, a method for executing a user-mode CUDA program is provided, the method comprising: When a CUDA program starts, the loader loads a hijacking layer into the CUDA program based on dynamic hijacking technology. The hijacking layer registers a collaborative agent with the central scheduler based on the virtual graphics processor (vGPU) instance to which the CUDA program belongs; The central scheduler, based on cooperative super-resolution technology, periodically sends future resource configuration information of the vGPU instance to the cooperative agent. The resource configuration information includes at least a computing power time window and resource quota. The collaborative agent stores the resource configuration information in a resource configuration table; When a process in the CUDA program initiates a kernel call, the hijacking layer uses the cooperative agent to query the resource configuration table, determines whether the kernel call matches the computing power time window and the resource quota based on the resource configuration table, and controls the kernel call based on the determination result.

[0006] In one possible implementation, determining whether the current kernel call matches the computing power time window and the resource quota based on the resource configuration table includes: The collaborative agent obtains the computing power time window and resource quota from the resource configuration table; The collaborative agent detects whether the kernel call can be completed within the computing power time window and whether the resources required for the kernel call exceed the resource quota. If the kernel call can be completed within the computing power time window and the required resources do not exceed the resource quota, then the collaborative agent determines that the kernel call matches the computing power time window and the resource quota. If the kernel call cannot be completed within the computing power time window, the collaborative agent determines that the kernel call does not match the computing power time window; If the required resources exceed the resource quota, the collaborative agent determines that the current kernel call does not match the resource quota.

[0007] In one possible implementation, controlling the kernel call based on the judgment result includes: If the current kernel call matches the computing power time window and the resource quota, the collaborative agent predicts the call duration of the current kernel call based on the exponentially weighted moving average (EWMA) algorithm. The collaborative agent adjusts the startup interval based on the call duration and the resource quota, and sends the startup interval as a judgment result to the hijacking layer. The hijacking layer performs this kernel call according to the startup interval to ensure that the resource utilization rate meets the resource quota.

[0008] In one possible implementation, controlling the kernel call based on the judgment result includes: If the current kernel call does not match the computing power time window, the collaborative agent will send the delayed execution as the judgment result to the hijacking layer; The hijacking layer enters a spin state based on the judgment result; The hijacking layer initiates another kernel call in the next computing power time window.

[0009] In one possible implementation, controlling the kernel call based on the judgment result includes: If the current kernel call does not match the resource quota, the collaborative agent determines the priority of the process; If the process has a high priority, the collaborative agent borrows existing resources from other low-priority processes within the vGPU instance and updates the resource configuration table based on the borrowed resources. The collaborative agent then determines, based on the resource configuration table, whether the current kernel call matches the computing power time window and the resource quota, and controls the current kernel call according to the determination result.

[0010] In one possible implementation, the central scheduler, based on cooperative super-resolution technology, periodically sends future resource configuration information of the vGPU instance to the cooperative agent, including: The central scheduler maintains a global vGPU instance status table through shared memory. The vGPU instance status table records the resource quotas and historical resource usage of multiple vGPU instances. The central scheduler negotiates with each cooperating agent based on the resources of the basic guarantee layer and the public competition layer, the vGPU instance status table, and the QoS policy to generate future resource configuration information for each vGPU instance. The basic guarantee layer is a resource layer that ensures each vGPU instance can obtain the minimum promised resources, and the public competition layer is a resource layer that multiple vGPU instances compete for. The central scheduler sends the future resource configuration information of the vGPU instance to the collaborative agent.

[0011] In one possible implementation, the method further includes: A hierarchical video memory architecture is constructed, which includes GPU physical video memory, system main memory and high-speed storage devices; Before the CUDA program initiates a kernel call, the hijacking layer allocates physical space for the vGPU instance from the GPU physical memory according to the resource quota. The physical space is used to store the data that needs to be processed during the kernel call. If the physical space of the GPU physical memory is insufficient, the hijacking layer uses the enhanced Least Recently Used (LRU) algorithm to identify cold pages in the GPU physical memory and asynchronously swaps the cold pages out to the system main memory or the high-speed storage device. The collaborative agent allocates physical space for the vGPU instance from the GPU physical memory again based on the resource quota.

[0012] According to a second aspect of this application, a system for executing user-mode CUDA programs is provided, the system comprising: A loader is used to load a hijacking layer into a CUDA program when the program starts, based on dynamic hijacking technology. The hijacking layer is used to register a cooperative agent with the central scheduler based on the virtual graphics processor (vGPU) instance to which the CUDA program belongs; The central scheduler is used to periodically send future resource configuration information of the vGPU instance to the collaborative agent based on cooperative super-resolution technology. The resource configuration information includes at least a computing power time window and a resource quota. The collaborative agent is used to store the resource configuration information in a resource configuration table; When a process in the CUDA program initiates a kernel call, the hijacking layer is also used to query the resource configuration table using the cooperative agent, determine whether the current kernel call matches the computing power time window and the resource quota based on the resource configuration table, and control the current kernel call based on the determination result.

[0013] According to a third aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the execution method of the user-mode CUDA program as described above.

[0014] According to a fourth aspect of this application, a computer device is provided, the computer device including the above-described user-mode CUDA program execution system.

[0015] The beneficial effects of the technical solution provided in this application include at least the following: The central scheduler, based on cooperative super-resolution technology, periodically sends future resource configuration information for the vGPU instances to which CUDA programs belong to the cooperative agents. This enables fine-grained resource control and significantly improves resource utilization. By configuring computing time windows and resource quotas for vGPU instances, kernel calls are strictly controlled by the cooperative agents, preventing CUDA programs from occupying SM units through multiple kernel calls and thus preventing malicious resource preemption. Each CUDA program can only access the memory allocated to its vGPU instance. Even if a kernel call encounters a problem, it will only cause that vGPU instance to be marked as faulty and will not affect other vGPU instances running on the GPU, improving the isolation of shared runtime. The super-resolution function can be used by loading the hijacking layer without modifying the code.

[0016] By adjusting the startup interval based on the call duration and resource quota, and performing kernel calls based on this startup detection, the resource utilization rate can be made to meet the resource quota, ensuring that the resources of the vGPU instance are not reduced.

[0017] If the remaining time in the computing time window is insufficient to complete the kernel call, it enters a spin state to smoothly consume the remaining time slice and avoid brute-force interruption.

[0018] When the resources required by a kernel call exceed the resource quota, if the process initiating the kernel call has a high priority, the cooperating agent can borrow existing resources from other low-priority processes within the vGPU instance, thereby ensuring that the high-priority process can execute first.

[0019] The constructed hierarchical video memory architecture includes GPU physical video memory, system main memory, and high-speed storage devices. When the physical space of GPU physical video memory is insufficient, the enhanced LRU algorithm can be used to identify cold pages in GPU physical video memory and asynchronously swap them out to system main memory or high-speed storage devices. Then, physical space is allocated for vGPU instances again from GPU physical video memory, which improves both the success rate of resource allocation and resource utilization. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a user-mode CUDA program execution method provided in one embodiment of this application; Figure 2 This is a flowchart of a user-mode CUDA program execution method provided in one embodiment of this application; Figure 3 This is a structural block diagram of a user-mode CUDA program execution system provided in one embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0023] like Figure 1 The diagram illustrates a flowchart of a method for executing a user-mode CUDA program according to an embodiment of this application. This method can be applied to a computer device. The method for executing a user-mode CUDA program may include: Step 101: When the CUDA program starts, the loader loads the hijacking layer in the CUDA program based on dynamic hijacking technology.

[0024] In this embodiment, the CUDA program runs in user mode, which is a non-privileged execution state of a computer system. User mode cannot directly access hardware devices, can only access limited memory areas, and uses a limited instruction set.

[0025] CUDA programs load the CUDA Driver Application Programming Interface (API), which includes, but is not limited to, cuLaunchKernel, cuMemAlloc, and cuMemcpyHtoD. Typically, cuLaunchKernel uses a token bucket mechanism to limit SM utilization, ensuring it does not exceed the configured computing power limit. cuMemAlloc checks if memory allocation exceeds the quota; if so, it rejects allocation. In practical applications, cuLaunchKernel, cuMemAlloc, and cuMemcpyHtoD can be wrapped and combined for calls to control computing power and GPU memory.

[0026] The hijacking layer is a separate dynamic link library (.so file) that is preferentially loaded into the process address space when a CUDA program starts, via the LD_PRELOAD mechanism. This dynamic link library contains functions with the same names as the original CUDA Driver API (such as cuLaunchKernel, cuMemAlloc, and cuMemcpyHtoD). When a CUDA program calls the CUDA API, it is actually calling the function with the same name in the hijacking layer, not the original function in the original CUDA Driver API.

[0027] Step 102: The hijacking layer registers a cooperative agent with the central scheduler based on the vGPU instance to which the CUDA program belongs.

[0028] The hijacking layer can obtain the identifier and QoS level of the vGPU instance to which the CUDA program belongs, and send a registration request carrying the identifier and QoS level to the central scheduler. The central scheduler registers the cooperative agent based on the registration request. Optionally, when the CUDA program does not declare resource requirements, the registration request can also carry resource requirements.

[0029] Step 103: Based on cooperative super-resolution technology, the central scheduler periodically sends future resource configuration information of vGPU instances to the cooperative agent. The resource configuration information includes at least the computing power time window and resource quota.

[0030] Super-resolution is a resource allocation strategy that refers to allocating more resources to users than the actual total physical resources. Super-resolution types include computing power super-resolution and memory super-resolution. Computing power super-resolution refers to the super-resolution of the computing power of SM units, achieved through time-based reuse; memory super-resolution refers to the super-resolution of memory capacity, achieved through spatial-based reuse.

[0031] In this embodiment, each CUDA program corresponds to a collaborative agent. All collaborative agents and the central scheduler maintain a global vGPU instance status table through shared memory, recording the historical resource usage of each vGPU instance, such as computing power quota, memory quota, memory usage, and historical load. The central scheduler can periodically negotiate the resource configuration information of each vGPU instance for a future time period with each collaborative agent based on information such as the vGPU instance status table, resource requirements, QoS level, and QoS policy, and then send the resource configuration information to the corresponding collaborative agent.

[0032] The computing power time window must include at least the start time and end time. Resource quotas must include at least the SM computing power quota (e.g., 20%) and the video memory quota. For example, if a physical GPU has 100 SM units, vGPU instance 1 and vGPU instance 2 each use 50 SM units (50%) within a 0-50ms time window; vGPU instance 3 and vGPU instance 4 each use 50 SM units (50%) within a 50-100ms time window. If a physical GPU has 16GB of video memory, it is virtualized into a larger virtual address space through tiered storage (building a tiered video memory architecture that includes GPU physical video memory, system main memory, and high-speed storage devices). For example, if vGPU instance 1 is allocated 16GB of virtual video memory, 8GB is actually in GPU physical video memory and 8GB in system main memory; if vGPU instance 2 is allocated 16GB of virtual video memory, 8GB is actually in GPU physical video memory and 8GB in system main memory; if vGPU instance 3 is allocated 16GB of virtual video memory, 0GB is actually in GPU physical video memory and 16GB in system main memory.

[0033] Step 104: The collaborative agent stores the resource configuration information in the resource configuration table.

[0034] Step 105: When a process in the CUDA program initiates a kernel call, the hijacking layer uses the cooperative agent to query the resource configuration table, determines whether the kernel call matches the computing power time window and resource quota based on the resource configuration table, and controls the kernel call based on the determination result.

[0035] When a process in a CUDA program initiates at least one kernel call, the hijacking layer uses a cooperative agent to determine whether the kernel call is allowed. The cooperative agent sends the determination result to the hijacking layer. If the memory call is allowed, the hijacking layer calls cuLaunchKernel in the original CUDA Driver API to implement the kernel call.

[0036] In summary, the user-space CUDA program execution method provided in this application embodiment uses a central scheduler based on cooperative super-resolution technology. This scheduler periodically sends future resource configuration information for the vGPU instance to which the CUDA program belongs to the cooperative agent, enabling fine-grained resource control and significantly improving resource utilization. By configuring computing power time windows and resource quotas for vGPU instances, kernel calls are strictly controlled by the cooperative agent, preventing CUDA programs from occupying SM units through multiple kernel calls and thus preventing malicious resource preemption. Each CUDA program can only access the memory allocated to its vGPU instance. Even if a kernel call encounters a problem, it only results in the vGPU instance being marked as faulty, without affecting other vGPU instances running on the GPU, thus improving the isolation of shared runtime. The super-resolution function can be used simply by loading the hijacking layer, without modifying the code.

[0037] like Figure 2 The diagram illustrates a flowchart of a user-mode CUDA program execution method according to an embodiment of this application. This user-mode CUDA program execution method can be applied to a computer device. The user-mode CUDA program execution method may include: Step 201: When the CUDA program starts, the loader loads the hijacking layer in the CUDA program based on dynamic hijacking technology.

[0038] Step 202: The hijacking layer registers a cooperative agent with the central scheduler based on the vGPU instance to which the CUDA program belongs.

[0039] The implementation process of steps 201-202 is the same as that of steps 101-102, and will not be repeated here.

[0040] Step 203: The central scheduler maintains a global vGPU instance status table through shared memory. The vGPU instance status table records the resource quotas and historical resource usage of multiple vGPU instances.

[0041] Each CUDA program corresponds to a cooperating agent. All cooperating agents and the central scheduler maintain a global vGPU instance status table through shared memory, recording the historical resource usage of each vGPU instance, such as computing power quota, memory quota, memory usage, and historical load.

[0042] Step 204: The central scheduler negotiates super-resolution with each cooperating agent based on the resources of the basic guarantee layer and the public competition layer, the vGPU instance status table and QoS policy, and generates future resource configuration information for each vGPU instance. The resource configuration information includes at least the computing power time window and resource quota. The basic guarantee layer is the resource layer that ensures that each vGPU instance can obtain the minimum promised resources, and the public competition layer is the resource layer that multiple vGPU instances compete for.

[0043] This embodiment defines a three-layer computing power allocation model, including a basic guarantee layer, a public competition layer, and a burst expansion layer. The basic guarantee layer ensures that each vGPU instance can obtain the minimum promised resources. The public competition layer is a public resource pool composed of unoccupied resources, which can be used by each vGPU instance according to its weight and real-time competition needs. The burst expansion layer allows a high-priority process in a vGPU instance to temporarily borrow resources from other low-priority processes within the same vGPU instance when it has a peak load.

[0044] The central scheduler can coordinate via semaphores to periodically (e.g., every 10ms) negotiate resource configuration information for each vGPU instance for a future time period with each cooperating agent, based on information such as resources in the basic guarantee layer and public contention layer, vGPU instance status tables, resource requirements, QoS levels, and QoS policies. The resource configuration information includes at least a computing power time window (e.g., 50ms) and resource quotas; details of the computing power time window and resource quotas are described above. The central scheduler can update the resource configuration information to the vGPU instance status table.

[0045] During resource scheduling, QoS policies can dynamically adjust the quota weights of each vGPU instance based on service levels to adjust the resource quotas of vGPU instances. The central scheduler can identify whether a task is compute-intensive or memory-intensive and allocate resources accordingly. For example, if a task is compute-intensive, it means that more computing power is required and less video memory is required, so the compute power quota can be increased and the video memory quota can be decreased. Conversely, if a task is memory-intensive, it means that less computing power is required and more video memory is required, so the compute power quota can be decreased and the video memory quota can be increased.

[0046] Optionally, when the task type is memory intensive, the computing power quota and video memory quota can be adjusted in conjunction to ensure that the balance and optimization can be achieved between the two resource dimensions of "computing" and "memory", thereby significantly improving the overall resource utilization and task performance while ensuring fairness.

[0047] In one example, the resource scheduling period can be set in the range of 10ms-200ms to balance overhead and accuracy.

[0048] Step 205: The central scheduler sends the future resource configuration information of the vGPU instance to the collaborative agent.

[0049] Step 206: The collaborative agent stores the resource configuration information in the resource configuration table.

[0050] Step 207: When a process in the CUDA program initiates a kernel call, the hijacking layer uses the cooperative agent to query the resource configuration table, determines whether the kernel call matches the computing power time window and resource quota based on the resource configuration table, and controls the kernel call based on the determination result.

[0051] When a process in a CUDA program initiates at least one kernel call, the hijacking layer uses a cooperative agent to determine whether the kernel call is allowed. The cooperative agent sends the determination result to the hijacking layer. If the memory call is allowed, the hijacking layer calls cuLaunchKernel in the original CUDA Driver API to implement the kernel call.

[0052] Specifically, determining whether the current kernel call matches the computing power time window and resource quota based on the resource configuration table can include: (1) The collaborative agent obtains the computing power time window and resource quota from the resource allocation table.

[0053] The collaborative agent reads the latest resource configuration information from the resource configuration table and obtains the computing power time window and resource quota from the resource configuration information.

[0054] (2) The collaborative agent detects whether the kernel call can be completed within the computing power time window and detects whether the resources required for the kernel call exceed the resource quota.

[0055] When determining whether a kernel call can be completed within the computing power time window, the collaborative agent checks if the current time is between the start and end times of the computing power time window. If the current time is not between the start and end times, it is determined that the kernel call cannot be completed within the computing power time window. If the current time is between the start and end times, the agent predicts the call duration of the kernel call based on the Exponentially Weighted Moving-Average (EWMA) algorithm and checks if the sum of the current time and the call duration is less than the end time. If the sum of the current time and the call duration is less than or equal to the end time, it is determined that the kernel call can be completed within the computing power time window; if the sum of the current time and the call duration is greater than the end time, it is determined that the kernel call cannot be completed within the computing power time window.

[0056] When checking whether the resources required for this kernel call exceed the resource quota, the collaborative agent checks whether the required computing power exceeds the computing power quota and whether the required video memory exceeds the video memory quota. If the required computing power does not exceed the computing power quota and the required video memory does not exceed the video memory quota, it is determined that the resources required for this kernel call do not exceed the resource quota. If the required computing power exceeds the computing power quota and / or the required video memory exceeds the video memory quota, it is determined that the resources required for this kernel call exceed the resource quota.

[0057] (3) If the kernel call can be completed within the computing power time window and the required resources do not exceed the resource quota, the collaborative agent determines that the kernel call matches the computing power time window and resource quota.

[0058] In this embodiment, if the current kernel call matches the computing power time window and resource quota, the collaborative agent predicts the call duration of the current kernel call according to the EWMA algorithm; the collaborative agent adjusts the startup interval according to the call duration and resource quota, and sends the startup interval as the judgment result to the hijacking layer; the hijacking layer performs the current kernel call according to the startup interval so that the resource utilization rate meets the resource quota.

[0059] Taking SM utilization as an example, SM utilization ≈ (kernel execution time × number of kernels) / total time, while the number of kernels ≈ total time / startup interval. Therefore, SM utilization ≈ kernel execution time / startup interval. It can be seen that SM utilization is inversely proportional to startup interval. Therefore, it is necessary to adjust the startup interval to make the resource utilization meet the resource quota.

[0060] (4) If the kernel call cannot be completed within the computing power time window, the collaborative agent determines that the kernel call does not match the computing power time window.

[0061] In this embodiment, if the current kernel call does not match the computing power time window, the collaborative agent will send the delayed execution as the judgment result to the hijacking layer; the hijacking layer enters a spin state according to the judgment result; the hijacking layer initiates the kernel call again in the next computing power time window.

[0062] That is, when the computing power time window is about to run out, the collaborative agent actively delays the kernel startup to smoothly consume the remaining time slice and avoid violent interruption.

[0063] (5) If the required resources exceed the resource quota, the collaborative agent determines that the kernel call does not match the resource quota.

[0064] In this embodiment, if the current kernel call does not match the resource quota, the collaborative agent determines the priority of the process; if the process has a high priority, the collaborative agent borrows existing resources from other low-priority processes within the vGPU instance and updates the resource configuration table based on the borrowed resources; the collaborative agent then determines whether the current kernel call matches the computing power time window and resource quota based on the resource configuration table, and controls the current kernel call based on the determination result.

[0065] If a higher-priority process borrows resources that other processes already have, those other processes will be suspended and wait until the higher-priority process finishes or other free resources become available before they can continue execution.

[0066] In this embodiment, a hierarchical video memory architecture can also be constructed, which includes GPU physical video memory, system main memory, and high-speed storage devices. Before a process in the CUDA program initiates a kernel call, the hijacking layer allocates physical space for the vGPU instance from the GPU physical video memory according to the resource quota. The physical space is used to store the data that needs to be processed during the kernel call. If the physical space of the GPU physical video memory is insufficient, the hijacking layer uses the enhanced Least Recently Used (LRU) algorithm to identify cold pages in the GPU physical video memory and asynchronously swaps the cold pages out to the system main memory or high-speed storage devices. The cooperative agent allocates physical space for the vGPU instance from the GPU physical video memory again according to the resource quota.

[0067] In the tiered memory architecture, GPU physical memory is layer L0, system main memory is layer L1, and high-speed storage devices are layer L3. These three layers of memory implement a unified memory architecture (UMA) abstraction, presenting a massive virtual memory pool to CUDA programs. If the GPU physical memory has sufficient physical space, physical space is directly allocated to the vGPU instance from it. If the GPU physical memory also has sufficient physical space, cold pages are asynchronously swapped out to system main memory or high-speed storage devices, and physical space is then allocated to the vGPU instance again from the GPU physical memory.

[0068] When data is accessed in a swapped-out cold page, on-demand loading is triggered, and the data migration is completed at the next cuStreamSynchronize call point. The data migration out and migration back is transparent to the CUDA program.

[0069] After allocating the physical space, the hijacking layer calls cuLaunchKernel in the original CUDA Driver API to copy the data required for this kernel call to the physical space and process the data.

[0070] In summary, the user-space CUDA program execution method provided in this application embodiment uses a central scheduler based on cooperative super-resolution technology. This scheduler periodically sends future resource configuration information for the vGPU instance to which the CUDA program belongs to the cooperative agent, enabling fine-grained resource control and significantly improving resource utilization. By configuring computing power time windows and resource quotas for vGPU instances, kernel calls are strictly controlled by the cooperative agent, preventing CUDA programs from occupying SM units through multiple kernel calls and thus preventing malicious resource preemption. Each CUDA program can only access the memory allocated to its vGPU instance. Even if a kernel call encounters a problem, it only results in the vGPU instance being marked as faulty, without affecting other vGPU instances running on the GPU, thus improving the isolation of shared runtime. The super-resolution function can be used simply by loading the hijacking layer, without modifying the code.

[0071] By adjusting the startup interval based on the call duration and resource quota, and performing kernel calls based on this startup detection, the resource utilization rate can be made to meet the resource quota, ensuring that the resources of the vGPU instance are not reduced.

[0072] If the remaining time in the computing time window is insufficient to complete the kernel call, it enters a spin state to smoothly consume the remaining time slice and avoid brute-force interruption.

[0073] When the resources required by a kernel call exceed the resource quota, if the process initiating the kernel call has a high priority, the cooperating agent can borrow existing resources from other low-priority processes within the vGPU instance, thereby ensuring that the high-priority process can execute first.

[0074] The constructed hierarchical video memory architecture includes GPU physical video memory, system main memory, and high-speed storage devices. When the physical space of GPU physical video memory is insufficient, the enhanced LRU algorithm can be used to identify cold pages in GPU physical video memory and asynchronously swap them out to system main memory or high-speed storage devices. Then, physical space is allocated for vGPU instances again from GPU physical video memory, which improves both the success rate of resource allocation and resource utilization.

[0075] The following section explains the application scenarios of the execution method for user-mode CUDA programs.

[0076] 1. Public cloud and private cloud scenarios Current situation: A single GPU typically supports 4 users, with each user having exclusive access to 25% of the computing power and video memory.

[0077] Excess value: (1) Super-resolution computing power: Through time slice rotation and idle SM unit reuse, a single GPU can support 16 concurrent vGPU instances.

[0078] (2) Memory over-expansion: 16GB of video memory can be virtually expanded to 128GB, allowing more users to run deep learning tasks at the same time.

[0079] (3) Resource utilization: The overall GPU utilization rate is increased to 70-90%, significantly reducing the waste of idle computing power.

[0080] (4) Cost advantage: The cost of a single user GPU is reduced by about 75%, and the efficiency of hardware investment is significantly improved.

[0081] 2. AI Training Platform Scenarios Current situation: Training large models is limited by GPU memory capacity, and a single GPU usually processes a single training task.

[0082] Excess value: (1) Memory virtualization: 32GB of memory can be expanded to 256GB to support larger models or simultaneous multi-task training.

[0083] (2) Dynamic allocation of computing power: During the training interval, the resources of the SM unit can be lent to other inference tasks to realize computing power reuse.

[0084] (3) Flexible expansion and contraction: Resources are dynamically increased during peak training tasks and automatically released to other tasks after training is completed.

[0085] (4) Results: Overall training throughput increased by 2-3 times, and hardware investment utilization increased to 80-90%.

[0086] 3. AI Inference Platform Scenarios Current situation: Inference model deployment is limited by GPU memory and SM units, and the number of models supported at the same time is limited.

[0087] Excess value: (1) Super-resolution of video memory: 32GB video memory is virtualized to 256GB, supporting the simultaneous deployment of more model instances.

[0088] (2) Computing time slice reuse: Different model inference requests are executed in staggered shifts to achieve high concurrency processing.

[0089] (3) Resource isolation guarantee: Each model instance does not interfere with each other, ensuring that the response delay is controllable.

[0090] (4) Effect: The number of supported concurrent inference tasks is increased by 4-6 times, while latency and throughput remain stable.

[0091] 4. Edge computing scenarios Current situation: Edge devices have limited GPU computing power, making it difficult to run multiple AI applications simultaneously.

[0092] Excess value: (1) Dynamic reuse of computing power and video memory: A single card supports 10-15 lightweight AI applications running simultaneously.

[0093] (2) Intelligent scheduling: Automatically allocate computing power and video memory according to real-time load to handle peak loads.

[0094] (3) Cost optimization: Compared with deploying multiple GPU nodes, hardware investment is reduced by about 70%.

[0095] (4) Effect: Edge AI applications respond quickly, the system is highly flexible, and hardware resources are fully utilized.

[0096] like Figure 3 The diagram illustrates a structural block diagram of a user-mode CUDA program execution system according to an embodiment of this application. This user-mode CUDA program execution system can be applied to a computer device. The system includes: Loader 310 is used to load a hijacking layer in a CUDA program based on dynamic hijacking technology when the CUDA program starts. Hijacking layer 320 is used to register cooperative agent 340 with central scheduler 330 based on the vGPU instance to which the CUDA program belongs; The central scheduler 330 is used to periodically send future resource configuration information of vGPU instances to the collaborative agent 340 based on collaborative super-resolution technology. The resource configuration information includes at least the computing power time window and resource quota. Collaborative Agent 340 is used to store resource configuration information in the resource configuration table; When a process in a CUDA program initiates a kernel call, the hijacking layer 320 is also used to query the resource configuration table using the cooperative agent 340. Based on the resource configuration table, it determines whether the kernel call matches the computing power time window and resource quota, and controls the kernel call based on the judgment result.

[0097] In an optional embodiment, the cooperative agent 340 is further configured to: Obtain the computing power time window and resource quota from the resource allocation table; Check whether the kernel call can be completed within the computing power time window, and check whether the resources required for the kernel call exceed the resource quota. If the kernel call can be completed within the computing power time window and the required resources do not exceed the resource quota, then the kernel call is determined to be compatible with the computing power time window and resource quota. If the kernel call cannot be completed within the computing power time window, it is determined that the kernel call does not match the computing power time window. If the required resources exceed the resource quota, then the kernel call is determined to be mismatched with the resource quota.

[0098] In an optional embodiment, if the kernel call matches the computing time window and resource quota, the collaborative agent 340 is also used to predict the call duration of the kernel call based on the exponentially weighted moving average (EWMA) algorithm. The collaborative agent 340 is also used to adjust the startup interval based on the call duration and resource quota, and sends the startup interval as the judgment result to the hijacking layer 320. Hijacking layer 320 is also used to make this kernel call according to the startup interval so that the resource utilization rate meets the resource quota.

[0099] In an optional embodiment, if the current kernel call does not match the computing power time window, the collaborative agent 340 is also used to send the delayed execution as a judgment result to the hijacking layer 320. Hijacking layer 320 is also used to enter a spin state based on the judgment result; Hijacking layer 320 is also used to initiate kernel calls again in the next computing time window.

[0100] In an optional embodiment, if the current kernel call does not match the resource quota, then the cooperative agent 340 is further configured to: Determine the priority of processes; If the process has a high priority, it borrows existing resources from other low-priority processes within the vGPU instance and updates the resource configuration table based on the borrowed resources. The system then checks the resource configuration table again to determine whether the kernel call matches the computing power time window and resource quota, and controls the kernel call based on the result.

[0101] In an optional embodiment, the central scheduler 330 is further configured to: A global vGPU instance status table is maintained through shared memory. The vGPU instance status table records the resource quotas and historical resource usage of multiple vGPU instances. Based on the resources of the basic guarantee layer and the public competition layer, the vGPU instance status table and the QoS policy, super-resolution negotiation is conducted with each cooperating agent 340 to generate future resource configuration information for each vGPU instance. The basic guarantee layer is the resource layer that ensures that each vGPU instance can obtain the minimum promised resources, and the public competition layer is the resource layer that multiple vGPU instances compete for. Send future resource configuration information for the vGPU instance to the collaborative agent 340.

[0102] In an optional embodiment, a memory virtualization engine is used to build a hierarchical memory architecture, which includes GPU physical memory, system main memory, and high-speed storage devices. Before the CUDA program initiates a kernel call, hijacking layer 320 is also used to allocate physical space for vGPU instances from the GPU physical memory according to resource quotas. The physical space is used to store the data that needs to be processed during the kernel call. If the physical space of the GPU physical memory is insufficient, the hijacking layer 320 is also used to identify cold pages in the GPU physical memory using the enhanced LRU algorithm and asynchronously swap the cold pages out to the system main memory or high-speed storage device. Collaborative Agent 340 is also used to reallocate physical space for vGPU instances from GPU physical memory again based on resource quotas.

[0103] In summary, the user-space CUDA program execution system provided in this application embodiment uses a central scheduler based on cooperative super-resolution technology. This scheduler periodically sends future resource configuration information for the vGPU instance to which the CUDA program belongs to the cooperative agent, enabling fine-grained resource control and significantly improving resource utilization. By configuring computing time windows and resource quotas for vGPU instances, kernel calls are strictly controlled by the cooperative agent, preventing CUDA programs from occupying SM units through multiple kernel calls and thus preventing malicious resource preemption. Each CUDA program can only access the memory allocated to its vGPU instance. Even if a kernel call encounters a problem, it only results in the vGPU instance being marked as faulty, without affecting other vGPU instances running on the GPU, thus improving the isolation of shared runtime. The super-resolution function can be used simply by loading the hijacking layer without modifying the code.

[0104] By adjusting the startup interval based on the call duration and resource quota, and performing kernel calls based on this startup detection, the resource utilization rate can be made to meet the resource quota, ensuring that the resources of the vGPU instance are not reduced.

[0105] If the remaining time in the computing time window is insufficient to complete the kernel call, it enters a spin state to smoothly consume the remaining time slice and avoid brute-force interruption.

[0106] When the resources required by a kernel call exceed the resource quota, if the process initiating the kernel call has a high priority, the cooperating agent can borrow existing resources from other low-priority processes within the vGPU instance, thereby ensuring that the high-priority process can execute first.

[0107] The constructed hierarchical video memory architecture includes GPU physical video memory, system main memory, and high-speed storage devices. When the physical space of GPU physical video memory is insufficient, the enhanced LRU algorithm can be used to identify cold pages in GPU physical video memory and asynchronously swap them out to system main memory or high-speed storage devices. Then, physical space is allocated for vGPU instances again from GPU physical video memory, which improves both the success rate of resource allocation and resource utilization.

[0108] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the execution method of a user-mode CUDA program as described above.

[0109] One embodiment of this application provides a computer device, which includes an execution system for any of the user-mode CUDA programs described above.

[0110] It should be noted that the user-mode CUDA program execution system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the user-mode CUDA program execution system can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the user-mode CUDA program execution system and the user-mode CUDA program execution method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0111] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0112] The above description is not intended to limit the embodiments of this application. Any adjustments, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A method for executing a user-mode CUDA program, characterized in that, The method includes: When a CUDA program starts, the loader loads a hijacking layer into the CUDA program based on dynamic hijacking technology. The hijacking layer registers a collaborative agent with the central scheduler based on the virtual graphics processor (vGPU) instance to which the CUDA program belongs; The central scheduler, based on cooperative super-resolution technology, periodically sends future resource configuration information of the vGPU instance to the cooperative agent. The resource configuration information includes at least a computing power time window and resource quota. The collaborative agent stores the resource configuration information in a resource configuration table; When a process in the CUDA program initiates a kernel call, the hijacking layer uses the cooperative agent to query the resource configuration table, determines whether the kernel call matches the computing power time window and the resource quota based on the resource configuration table, and controls the kernel call based on the determination result.

2. The method for executing a user-mode CUDA program according to claim 1, characterized in that, The step of determining whether the current kernel call matches the computing power time window and the resource quota based on the resource configuration table includes: The collaborative agent obtains the computing power time window and resource quota from the resource configuration table; The collaborative agent detects whether the kernel call can be completed within the computing power time window and whether the resources required for the kernel call exceed the resource quota. If the kernel call can be completed within the computing power time window and the required resources do not exceed the resource quota, then the collaborative agent determines that the kernel call matches the computing power time window and the resource quota. If the kernel call cannot be completed within the computing power time window, the collaborative agent determines that the kernel call does not match the computing power time window; If the required resources exceed the resource quota, the collaborative agent determines that the current kernel call does not match the resource quota.

3. The method for executing a user-mode CUDA program according to claim 2, characterized in that, The process of controlling this kernel call based on the judgment result includes: If the current kernel call matches the computing power time window and the resource quota, the collaborative agent predicts the call duration of the current kernel call based on the exponentially weighted moving average (EWMA) algorithm. The collaborative agent adjusts the startup interval based on the call duration and the resource quota, and sends the startup interval as a judgment result to the hijacking layer. The hijacking layer performs this kernel call according to the startup interval to ensure that the resource utilization rate meets the resource quota.

4. The method for executing a user-mode CUDA program according to claim 2, characterized in that, The process of controlling this kernel call based on the judgment result includes: If the current kernel call does not match the computing power time window, the collaborative agent will send the delayed execution as the judgment result to the hijacking layer; The hijacking layer enters a spin state based on the judgment result; The hijacking layer initiates another kernel call in the next computing power time window.

5. The method for executing a user-mode CUDA program according to claim 2, characterized in that, The process of controlling this kernel call based on the judgment result includes: If the current kernel call does not match the resource quota, the collaborative agent determines the priority of the process; If the process has a high priority, the collaborative agent borrows existing resources from other low-priority processes within the vGPU instance and updates the resource configuration table based on the borrowed resources. The collaborative agent then determines, based on the resource configuration table, whether the current kernel call matches the computing power time window and the resource quota, and controls the current kernel call according to the determination result.

6. The method for executing a user-mode CUDA program according to claim 1, characterized in that, The central scheduler, based on cooperative super-resolution technology, periodically sends future resource configuration information for the vGPU instance to the cooperative agent, including: The central scheduler maintains a global vGPU instance status table through shared memory. The vGPU instance status table records the resource quotas and historical resource usage of multiple vGPU instances. The central scheduler negotiates with each cooperating agent based on the resources of the basic guarantee layer and the public competition layer, the vGPU instance status table, and the QoS policy to generate future resource configuration information for each vGPU instance. The basic guarantee layer is a resource layer that ensures each vGPU instance can obtain the minimum promised resources, and the public competition layer is a resource layer that multiple vGPU instances compete for. The central scheduler sends the future resource configuration information of the vGPU instance to the collaborative agent.

7. The method for executing a user-mode CUDA program according to any one of claims 1 to 6, characterized in that, The method further includes: A hierarchical video memory architecture is constructed, which includes GPU physical video memory, system main memory and high-speed storage devices; Before the CUDA program initiates a kernel call, the hijacking layer allocates physical space for the vGPU instance from the GPU physical memory according to the resource quota. The physical space is used to store the data that needs to be processed during the kernel call. If the physical space of the GPU physical memory is insufficient, the hijacking layer uses the enhanced Least Recently Used (LRU) algorithm to identify cold pages in the GPU physical memory and asynchronously swaps the cold pages out to the system main memory or the high-speed storage device. The collaborative agent allocates physical space for the vGPU instance from the GPU physical memory again based on the resource quota.

8. A user-mode CUDA program execution system, characterized in that, The system includes: A loader is used to load a hijacking layer into a CUDA program when the program starts, based on dynamic hijacking technology. The hijacking layer is used to register a cooperative agent with the central scheduler based on the virtual graphics processor (vGPU) instance to which the CUDA program belongs; The central scheduler is used to periodically send future resource configuration information of the vGPU instance to the collaborative agent based on cooperative super-resolution technology. The resource configuration information includes at least a computing power time window and a resource quota. The collaborative agent is used to store the resource configuration information in a resource configuration table; When a process in the CUDA program initiates a kernel call, the hijacking layer is also used to query the resource configuration table using the cooperative agent, determine whether the current kernel call matches the computing power time window and the resource quota based on the resource configuration table, and control the current kernel call based on the determination result.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method for executing a user-mode CUDA program as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The computer device includes: the user-mode CUDA program execution system as described in claim 8.

Images