GPU video memory isolation and scheduling method between containers

Abstract

The invention relates to the technical field of GPU distribution, in particular to a method for isolating and scheduling GPU video memories between containers. The method is realized based on a management system, and the management system comprises a host machine and a plurality of containers. Each container corresponds to an independent video memory pool on the host machine; the container comprises an RPC client module, an interceptor and a calculation task module; the host machine comprises a calling interface, an RPC service module, a video memory management module and GPU equipment; the RPC service module communicates with the video memory management module through a calling interface; the GPU device communicates with the GPU device through the driving interface; and the video memory management module creates a corresponding video memory pool on the GPU equipment for each container. The method comprises a video memory pool management process and a task scheduling process. And the host machine uniformly manages video memory resources and task scheduling of the GPU equipment, communicates with a plurality of containers through an RPC interface, and performs centralized control on video memory allocation and calculation task execution of each container.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of GPU allocation, and in particular to a GPU memory isolation and scheduling method between containers. BACKGROUND

[0002] With the wide application of GPUs in high-performance computing fields such as deep learning and graphics rendering, GPU resources (especially memory) have become a bottleneck restricting the performance of computing tasks. In traditional GPU sharing or virtualization schemes, different tasks or containers often share the memory of a same physical GPU (GPU device). This sharing mode is prone to cause uneven memory allocation and resource contention, thereby affecting the stability and running performance of each task. In a multi-task or multi-tenant environment, how to efficiently and flexibly isolate and manage GPU memory resources between different containers has become a technical problem to be solved.

[0003] Existing GPU memory management schemes mostly rely on hardware virtualization (for example, the MIG technology of NVIDIA) or adopt a static memory pre-allocation mode. However, these schemes have obvious deficiencies in actual application: the hardware isolation means is high in cost and lacks flexibility, and the static pre-allocation is often difficult to adapt to the dynamic changes of task loads in time, which may lead to low memory utilization or resource waste. Therefore, there is an urgent need for a new method at the software level, which can realize memory soft isolation and dynamic scheduling between multiple containers, improve the utilization efficiency of GPU memory resources and guarantee the performance isolation between containers. SUMMARY

[0004] To solve the above problems, the purpose of the present application is to provide a GPU memory isolation and scheduling method between containers, the host computer uniformly manages the memory resources and task scheduling of the GPU device, and communicates with multiple containers through an RPC interface to centrally control the memory allocation and computing task execution of each container.

[0005] The GPU memory isolation and scheduling method between containers of the present application is realized based on a management system, which includes a host computer and multiple containers; each container corresponds to an independent memory pool on the host computer; the method includes a memory pool management process; The specific steps of the memory pool management process are as follows: Step S1: The container issues a memory allocation call, which is encapsulated as an RPC call request and sent to the host computer; Step S2: The host computer locates the memory pool corresponding to the container based on the RPC call request; Step S3: The host computer judges whether the memory pool corresponding to the container has a free memory block that meets the RPC call request to allocate to the container; Step S4: If the memory pool has a free memory block satisfying the RPC call request, go to step S6; if the memory pool does not have a free memory block satisfying the RPC call request, go to step S5; Step S5: If the memory pool does not have a free memory block satisfying the RPC call request, merge adjacent free memory blocks in the memory pool, and determine whether the merged free memory block satisfies the RPC call request; if the merged free memory block satisfies the RPC call request, go to step S6; if the merged free memory block still does not satisfy the RPC call request, the host machine fails to allocate a free memory block, and the host machine returns a second allocation result to the container; Step S6: The host machine allocates a free memory block satisfying the RPC call request from the memory pool to the corresponding container, and obtains a first allocation result, which includes allocation success information and a memory handle or a corresponding Tensor ID; Step S7: The host machine returns the first allocation result to the container.

[0006] Further, the container includes an RPC client module, an interceptor, and a computing task module connected in sequence; the host machine includes a calling interface, an RPC service module, a memory management module, and a GPU device; the RPC service module communicates with the memory management module through the calling interface; the memory management module communicates with the GPU device through a driver interface; the memory management module creates a corresponding memory pool for each container on the GPU device; and the RPC client module is communicatively connected to the RPC service module.

[0007] Further, step S1 includes the following steps: Step S11: When the computing task module of the container needs the host machine to allocate memory, an allocation call of the memory is issued; Step S12: After the interceptor intercepts the allocation call of the memory, the allocation call of the memory is forwarded to the RPC client module; Step S13: The RPC client module packs the allocation request of the memory into an RPC allocation request, and sends the RPC allocation request to the host machine.

[0008] Further, step S2 includes the following steps: Step S21: The RPC service module of the host machine receives the RPC allocation request and passes the RPC allocation request to the calling interface, Step S22: The calling interface translates the RPC allocation request into underlying call information and passes the underlying call information to the memory management module; Step S23: The memory management module locates the memory pool corresponding to the container based on the underlying call information.

[0009] Further, step S7 includes the following steps: Step S71: The video memory management module of the host computer returns the first allocation result to the RPC service module through the RPC communication module; Step S72: The RPC service module returns the first allocation result to the RPC client module of the container; Step S73: The RPC client module returns the first allocation result to the computing task module of the container through the interceptor; the computing task module uses the idle video memory block allocated by the video memory pool based on the first allocation result.

[0010] Further, the video memory pool management process further includes step S8; Step S8: When the computing task module of the container finishes using the allocated idle video memory block, the computing task module of the container performs a video memory release call; after the interceptor captures the video memory release call, the video memory release call is forwarded to the RPC client module; the RPC client module packages the video memory release call into an RPC release request and sends it to the RPC service module of the host computer; the RPC service module translates the RPC release request into underlying release information through the calling interface, and then passes the underlying release information to the video memory management module; the video memory management module releases the corresponding video memory block based on the underlying release information.

[0011] Further, the host computer further includes a task scheduling module, which is in communication connection with the calling interface and the GPU device, and is used for receiving the computing task of the container and completing the task scheduling of the computing task; The task scheduling process of the computing task includes: Step A1: The container submits a first computing task, encapsulates it into an RPC task request, and sends it to the host computer; Step A2: After the host computer receives the RPC task request, the RPC task request is translated into task information of the first computing task through the calling interface; the task information of the first computing task is forwarded to the task scheduling module; Step A3: The task scheduling module creates a task unit corresponding to the first computing task based on the task information of the first computing task, and adds the first computing task to the GPU task queue; Step A4: The task scheduling module makes scheduling decisions for the GPU task queue based on a task scheduling strategy; when the task scheduling module schedules the first computing task, the task scheduling module submits the task unit corresponding to the first computing task to the GPU device; Step A5: The GPU device executes the first computing task based on the task unit corresponding to the first computing task, and the task scheduling module monitors the task state of the first computing task; Step A6: When the GPU device finishes executing the first computing task, the GPU device obtains the task result of the first computing task; the task scheduling module reads the task result of the first computing task from the GPU device and sends the task result of the first computing task back to the container after encapsulating the task result of the first computing task through the RPC service module; the task result includes the execution state of the first computing task and output data.

[0012] Further, step A1 includes the following steps: Step A11: The computing task module of the container constructs the first computing task; Step A12: After the interceptor intercepts the first computing task, the first computing task is forwarded to the RPC client module; Step A13: The RPC client module encapsulates the first computing task into an RPC task request and sends the RPC task request to the RPC service module of the host computer.

[0013] Further, the task scheduling process further includes step A7; Step A7: After the RPC client module of the container receives the encapsulated task result of the first computing task, the encapsulated task result of the first computing task is restored and handed over to the interceptor, and the interceptor sends the task result of the first computing task to the computing task module that initiates the first computing task.

[0014] Further, the calling interface is a kompute calling interface, and the driving interface is a Vulkan driving interface.

[0015] Advantages of the present application: 1. Efficient use of video memory resources: The video memory pool management and on-demand dynamic allocation mechanism are adopted, so that the video memory resources of the GPU device are fully utilized, and the idle and waste are reduced. Compared with the fixed division or static pre-allocation mode, the present application can adjust the video memory occupancy of each container in real time according to the load, improve the video memory utilization rate, and relieve the video memory bottleneck.

[0016] 2. Strong container isolation: Each container corresponds to an independent video memory pool, and each container can only access the video memory in its own video memory pool, realizing complete isolation of video memory use, avoiding the problem that excessive occupation of video memory by a container affects other containers, and greatly improving the reliability and fairness of the system in a multi-tenant environment.

[0017] 3. Flexible video memory scheduling: The size and allocation of the video memory pool can be dynamically adjusted according to the running situation, and the present application can flexibly provide or recycle video memory resources according to the real-time needs of different tasks. When a container task ends and releases video memory, other containers can quickly utilize this part of idle resources, thereby optimizing the overall performance. The system is no longer limited to static configuration and has elastic expansion capability.

[0018] 4. Simplified development and integration: By defining a unified RPC interface, the memory allocation and task scheduling between the container and the host are completely transparent to the developer. The present solution only needs to load the interceptor plug-in and the RPC client module on the container side, without any modification to the application code. Therefore, existing applications can have the ability of GPU memory soft isolation and dynamic scheduling with almost zero modification, greatly reducing the deployment cost.

[0019] 5. No need for hardware virtualization: The present solution completely implements the isolation and sharing management of GPU memory through software, without relying on SR-IOV, vGPU, and other hardware virtualization technologies, nor requiring the support of proprietary drivers from specific manufacturers. This reduces the cost and complexity of system deployment, making the solution more applicable in more general environments and having wider applicability.

[0020] 6. Fine-grained soft isolation: The present solution intercepts the memory allocation calls of Tensor or Vulkan / OpenCL at runtime, and implements fine-grained memory isolation control in a software manner. Compared with the coarse-grained solution relying on hardware strong isolation, soft isolation allows more flexible sharing of GPU resources while ensuring safety, improving resource utilization efficiency and avoiding direct competition. This reflects the advantages of software-defined isolation.

[0021] 7. Dynamic scheduling and fragmentation management: The memory pool supports real-time fragmentation merging and memory recycling strategies, ensuring that fragmented memory can be reused in a timely manner and avoiding allocation failures due to fragmentation. At the same time, the host can perform load balancing or task migration between multiple GPU nodes according to the overall load, achieving more advanced dynamic resource scheduling capabilities. The present solution excels in resource elasticity and fragmentation processing.

[0022] 8. Cross-platform compatibility: Based on open standards such as Vulkan / OpenCL, the present solution can adapt to various GPU hardware environments. It can be deployed and used on GPUs from different manufacturers such as NVIDIA, AMD, and Intel, with good cross-platform compatibility. Users are no longer locked into a specific manufacturer's ecosystem, protecting existing software and hardware investments.

[0023] 9. Transparent integration of containers: Containers only need to load the interceptor plug-in and RPC client module provided by the present solution, without modifying existing application code or interfering with application logic, to automatically obtain GPU memory soft isolation and dynamic scheduling capabilities. This transparent integration of container applications facilitates the upgrade of existing systems, and the behavior and performance of applications running in an isolated environment are almost identical to those in a non-isolated environment. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the specific embodiments or the prior art of the present application, the drawings required to be used in the specific embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 A schematic diagram of the management system of the present application; Figure 2 A schematic diagram of the management flow of the video memory pool of the present application; Figure 3 A schematic diagram of the task scheduling flow of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below. In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0027] The present application provides a container-to-container GPU video memory isolation and scheduling method. The host machine uniformly manages the video memory resources and task scheduling of the GPU device, communicates with multiple containers through an RPC interface, and centrally controls the video memory allocation and computing task execution of each container.

[0028] Embodiment 1:

[0029] The present embodiment discloses a management system, which includes a host machine and multiple containers. The host machine runs various core management services, and the containers run applications and lightweight proxy components, thereby achieving controlled use of the resources of the host machine.

[0030] As shown in Figure 1 , the host machine includes a calling interface, an RPC service module, a video memory management module, a task scheduling module, and a GPU device. The containers include an RPC client module, an interceptor, and a computing task module. The video memory management module and the task scheduling module communicate with the GPU device through a Vulkan driver interface. The RPC service module communicates with the video memory management module and the task scheduling module through the calling interface. The video memory management module creates a corresponding video memory pool for each container on the GPU device; the host machine dynamically allocates video memory according to the real-time needs of the containers, and timely recycles and releases the video memory after the task is completed; at the same time, the host machine is responsible for the global task scheduling of the GPU device, and tries to improve the utilization rate of the video memory and the computing unit.

[0031] The memory management module is responsible for creating and maintaining the memory pool, providing dynamic allocation and reclamation of memory. It initializes the GPU device through the Vulkan driver interface and creates a memory pool manager. Based on system configuration or policy, the host machine allocates an independent memory pool for each container to isolate memory usage across containers. The memory management module maintains a linked list for each memory pool, recording information on allocated and free memory blocks to efficiently execute subsequent memory allocation and deallocation.

[0032] The task scheduling module is responsible for the global scheduling and management of computing tasks. It maintains the GPU task queue on the host machine and allocates GPU computing time slices and schedules the execution of each computing task based on factors such as the memory pool usage status of each container, container priority policies, and QoS requirements. The task scheduling module submits computing tasks from different containers through the Vulkan driver interface's command buffer, enabling the parallel or asynchronous execution of multiple computing tasks. The goal of the task scheduling module is to fully utilize the GPU's computing cores and bandwidth, preventing any single container from monopolizing GPU resources for extended periods.

[0033] API Call Interface: In this embodiment, the API call interface is the kompute API. The kompute API encapsulates the functionalities of the kompute framework for Vulkan computation and management. It receives high-level computation requests from containers (such as creating tensors or executing computation graphs), translates them into low-level Vulkan API calls, and thus triggers actual memory allocation and computation execution on the host machine. The kompute API simplifies the container's calls to GPU devices, enabling the task scheduling module and memory management module to easily manage computation tasks and call requests from containers.

[0034] RPC service module ( Figure 1 The "RPC communication module" in the context of the host machine serves as the RPC server, providing a unified entry point for communication between containers and the host. This module listens for and receives RPC requests from containers, including requests for memory allocation, memory release, and computational tasks. It then forwards these requests to the appropriate modules within the host machine for processing and encapsulates the results before returning them to the requesting container. This RPC service module ensures that container access to host resources is a controlled remote procedure, preventing containers from directly manipulating GPU devices and thus guaranteeing secure isolation.

[0035] GPU devices ( Figure 1The "physical GPU" in this invention refers to the actual physical GPU connected to the host machine, including physical resources such as GPU memory and computing cores. The host machine's memory management module and task scheduling module communicate with the GPU device through the Vulkan driver interface to execute specific memory allocation, data transfer, and computing tasks. In this invention, the GPU device does not require special hardware partitioning support; the logical partitioning and scheduling of its resources are implemented by software.

[0036] RPC client module ( Figure 1 The RPC client in the container refers to a lightweight agent process or thread running inside the container, responsible for communicating with the RPC service module on the host machine. The RPC client module intercepts calls to GPU resources from applications (computation task modules) within the container and packages operations such as tensor creation, memory allocation, and computation graph execution into RPC requests. Figure 1 The RPC client module sends remote call requests to the host machine and waits for the host machine to return the result. The RPC client module is like a "proxy" within the container, enabling the container to use the host machine's GPU capabilities via remote calls.

[0037] Interceptors: Special components deployed within containers to intercept function calls related to GPU memory allocation from applications within the container. For example, when an application in the container uses the kompute API, the interceptor plugin intercepts memory allocation methods such as kompute::Tensor::allocate(); if the application directly uses the Vulkan driver API, it intercepts low-level memory allocation interfaces such as vkAllocateMemory(). After intercepting these calls, the interceptor does not execute them directly within the container. Instead, the RPC client module converts them into a recognizable request format and sends them to the host machine for processing, thus preventing the container from directly manipulating physical GPU memory. This mechanism achieves transparent forwarding of memory requests and non-intrusive integration of applications within the container.

[0038] Computation task module ( Figure 1“compute task” in the container): refers to an application (such as a deep learning training script or a graphics rendering process) running in the container, which can build a compute task or GPU (for example, create a tensor, define a compute shader, etc.) through the high-level API provided by the kompute invocation interface. When the application needs to perform GPU computation, it calls the kompute invocation interface or the Vulkan driver interface to trigger the application for memory allocation and compute task submission, which is captured by the interceptor and processed by the RPC client module. Finally, the host machine completes the actual memory allocation and GPU computation and returns the result to the application in the container through the RPC client module. With the help of the compute task module, the application in the container does not need to be aware of how the underlying memory is managed, and its use of the GPU device is intercepted by the host machine for execution, ensuring the transparent operation of the application in the isolated environment.

[0039] Embodiment 2:

[0040] In this embodiment, a container GPU memory isolation and scheduling method is disclosed, which is implemented based on the management system of embodiment 1. The method includes a memory pool management process.

[0041] As shown in Figure 2 The memory pool management process includes the following steps: Step S1: The container issues a memory allocation call and sends it to the host after being encapsulated as an RPC call request.

[0042] Specifically, when the application (compute task module) in the container needs to allocate memory, the compute task module issues a call (tensor creation or memory allocation call); the interceptor intercepts the memory allocation call issued by the compute task module and hands it over to the RPC client module of the container. The RPC client module encapsulates it as an RPC allocation request, and then initiates an RPC allocation request (for example, calls AllocateMemory(taskID, size)) to the RPC service module of the host, where taskID is used to identify the current request (task or tensor), and size is the size of the allocated memory. This request is equivalent to the container “applying” for a memory of a specified size to the host.

[0043] Step S2: The host locates the memory pool corresponding to the container based on the RPC call request; Step S2 includes the following steps: Step S21: The RPC service module of the host receives the RPC allocation request and passes it to the invocation interface; Step S22: The invocation interface translates the RPC allocation request into underlying call information and passes it to the memory management module; Step S23: The video memory management module locates the video memory pool corresponding to the container based on the underlying call information, and then performs step S3 in the linked list corresponding to the video memory pool.

[0044] Step S3: The host machine's video memory management module uses a linked list to determine whether there are any free video memory blocks in the video memory pool corresponding to the container that meet the RPC call request, and then allocates them to the container. Step S4: If there is a free memory block in the memory pool that satisfies the RPC call request, proceed to step S6; if there is no free memory block in the memory pool that satisfies the RPC call request, proceed to step S5. Step S5: If there is no free memory block in the memory pool that satisfies the RPC call request, then merge adjacent free memory blocks in the memory pool (fragmentation merging) to free up a larger contiguous free memory block, and determine whether the merged free memory block satisfies the RPC call request; if the merged free memory block still does not meet the requested size, then the host machine's memory management module fails to allocate a free memory block; if the merged free memory block satisfies the RPC call request, then proceed to step S6.

[0045] If the video memory management module fails to allocate a free video memory block, it can choose to refuse the allocation or add it to the waiting queue, suspend the container's RPC allocation request, wait for video memory to be released, and then retry. The second allocation result is returned to the container, which is either a failure message or a waiting signal.

[0046] Step S6: The host machine's video memory management module allocates free video memory blocks that meet the RPC call request from the video memory pool to the corresponding container and obtains the first allocation result, which includes allocation success information and video memory handle or corresponding Tensor ID.

[0047] Specifically, the host machine allocates a free block of video memory of the requested size from the video memory pool, calls vkAllocateMemory() on the GPU device to actually allocate the video memory, and associates this video memory block with a newly created tensor object in the kompute framework. In this way, the video memory required by the container is successfully allocated and occupied.

[0048] Step S7: The host machine returns the first allocation result to the container.

[0049] Step S7 includes the following steps: Step S71: The host machine's video memory management module encapsulates the first allocation result into an RPC allocation result through the RPC communication module and returns it to the RPC service module; Step S72: The RPC service module returns the RPC allocation result to the container's RPC client module; Step S73: After the RPC client module reinterprets it as the first allocation result, the first allocation result is returned to the compute task module of the container through the interceptor; the compute task module uses the idle GPU memory block allocated by the GPU memory pool based on the first allocation result. The application (compute task module) obtains the GPU memory address or reference allocated by the host based on the first allocation result, and can continue to perform subsequent operations.

[0050] Step S8: When the compute task module in the container finishes using the allocated idle GPU memory block, the compute task module of the container performs a GPU memory release call; after the interceptor captures the GPU memory release call, the GPU memory release call is forwarded to the RPC client module; the RPC client module packages the GPU memory release call into an RPC release request and sends it to the RPC service module of the host; the RPC service module translates the RPC release request into underlying release information through a calling interface, and then delivers the underlying release information to the GPU memory management module; the GPU memory management module releases the corresponding GPU memory block based on the underlying release information.

[0051] Specifically, when the application in the container finishes using the GPU memory or a certain segment of the GPU memory is long-term idle and invalid, the application needs to return the GPU memory to the host for use. At this time, the application performs a GPU memory release call (calling the GPU memory release interface of kompute or Vulkan, such as Tensor.release()), which is also captured by the interceptor and notifies the host to perform GPU memory recovery through the RPC client module. After receiving the RPC release request (FreeMemory(handle)), the host GPU memory management module marks the corresponding GPU memory block as available and inserts it into the linked list of the GPU memory pool corresponding to the container. If the released idle GPU memory block is connected to an adjacent region in the GPU memory pool (i.e., the adjacent GPU memory block is also idle), the adjacent idle blocks are merged to form a larger continuous idle GPU memory block. By timely recovering and merging the fragments, the application can reduce GPU memory fragmentation and improve the success rate and efficiency of subsequent GPU memory allocation.

[0052] In the GPU memory management process of the application, each container can only use the GPU memory pool authorized for it. As shown in FIG. 6, when the container initiates a GPU memory application, the host only searches for available space in the GPU memory pool corresponding to the container, ensuring that the GPU memory of the containers does not be occupied by each other; when the GPU memory is insufficient, the service continuity is maintained through the fragment consolidation or queue waiting mechanism; when the request is completed and the GPU memory is released, the freed space is timely recovered into the GPU memory pool, preparing for subsequent applications. Figure 2

[0053] Embodiment 3:

[0054] In this embodiment, the method includes a task scheduling process.

[0055] ​The specific steps of the task scheduling process are as follows: Step A1: The container submits a first computing task, and sends the first computing task to the host computer after encapsulating the first computing task into an RPC task request; Step A1 includes the following steps: Step A11: The computing task module of the container constructs the first computing task by using an API provided by kompute; Step A12: After the interceptor intercepts the first computing task, the first computing task is forwarded to the RPC client module; Step A13: After the RPC client module encapsulates the first computing task into an RPC task request (such as ExecuteTask(taskID, params...)), the RPC task request is sent to the RPC service module of the host computer. The RPC task request includes the task identifier and required parameters of the first computing task, triggering the host computer to execute the first computing task for the container.

[0056] Step A2: After the RPC service module of the host computer receives the RPC task request, the RPC task request is translated into task information of the first computing task by using a kompute calling interface; the task information of the first computing task is transferred to the task scheduling module; Step A3: The task scheduling module creates an executable task unit corresponding to the first computing task (the task unit includes a call to a specific GPU Kernel / Shaders and a required resource handle) based on the task information of the first computing task, and adds the first computing task to a GPU task queue for waiting for scheduling; at this time, the first computing task is in a queued waiting state; Step A4: The task scheduling module makes scheduling decisions for tasks in the GPU task queue based on a pre-set task scheduling strategy; factors considered by the task scheduling strategy include the priority of a container to which the task belongs, quality of service (QoS) requirements, the current memory pool occupancy rate of each container, and the estimated execution time of the task. The task scheduling module sorts the tasks in the GPU task queue according to these strategies, selects a batch of tasks that can be immediately run, and submits the batch of tasks to the GPU device for execution. If the GPU device resources are sufficient (there are more idle computing units), the task scheduling module can select multiple tasks at a time and send the multiple tasks to the GPU device in parallel, so as to fully utilize the parallel computing capability; if the GPU device load is high, the tasks are arranged for execution one by one in order of priority, and other tasks wait in the GPU task queue for GPU to be idle. This centralized intelligent scheduling ensures that high-priority tasks are serviced in a timely manner, and prevents a container from monopolizing the GPU device for a long time.

[0057] When the task scheduling module schedules to the first computing task, the task scheduling module submits the task unit corresponding to the first computing task to the GPU device through a Vulkan driver interface; the task scheduling module allocates or reuses a Vulkan command buffer instruction for the first computing task, records the GPU execution instruction stream of the first computing task, and then submits the command buffer instruction to the driver of the GPU device. The GPU device will strictly execute each container task according to the command buffer instruction, and still maintains the memory isolation between each other, only realizing shared utilization on the computing unit. For a GPU device supporting parallelism (having multiple hardware queues), the task scheduling module can allocate different tasks to multiple independent command queues, so that they are executed simultaneously to realize real parallel computing.

[0058] Step A5: The GPU device executes the first computing task based on the task unit corresponding to the first computing task, and the synchronization mechanism provided by the Vulkan of the task scheduling module monitors the task state of the first computing task.

[0059] Step A6: When the GPU device completes the first computing task, the GPU device obtains the task result of the first computing task; the task scheduling module reads the task result of the first computing task from the GPU device and encapsulates and sends back the task result of the first computing task to the container through an RPC service module; the task result includes an execution state and output data.

[0060] Step A7: After the RPC client module of the container receives the encapsulated task result of the first computing task, the encapsulated task result of the first computing task is restored and handed over to the interceptor, and the interceptor sends the task result of the first computing task to the computing task module initiating the first computing task.

[0061] For an application (computing task module), the application calls kompute to execute a GPU computing task and obtains a returned result, and the entire remote execution process is transparent to the application and is no different from direct execution on a local GPU.

[0062] In terms of memory pool management: for a containerized environment, the application proposes a soft isolation management method based on a memory pool. The host machine divides an independent memory pool for each container, ensures that the memory resources of each container are isolated from each other and do not encroach on each other. This pool management avoids resource contention caused by traditional shared memory.

[0063] In terms of memory soft isolation: the application forwards the memory application in the form of an RPC request to the host machine by intercepting the memory allocation call of the kompute framework or Vulkan in the container, and performs memory allocation in the memory pool corresponding to the container by the host machine. Without relying on hardware support, the application can realize memory isolation between containers and prevent any container from accessing unauthorized memory regions.

[0064] In terms of dynamic memory allocation: the host machine adjusts the memory allocation strategy in real time according to the current task requirements of each container and the remaining available resources of its memory pool. Memory applications are provided by the host machine on demand and by size, and the memory is immediately recycled and released after the task is completed, thereby achieving dynamic and flexible management of memory resources. Compared with the fixed pre-allocation method, this mechanism significantly improves the memory utilization rate.

[0065] In terms of memory pool fragmentation consolidation mechanism: the host machine maintains a linked list of free memory blocks for each memory pool. When memory fragmentation occurs, the host machine can automatically merge adjacent free blocks and recycle the fragmented space to provide larger contiguous memory blocks to new requests. This mechanism allows the memory to maintain a high availability rate even after multiple rounds of allocation / release, avoiding memory waste caused by fragmentation.

[0066] In terms of RPC communication mechanism: an RPC communication bridge is built between the container and the host machine, and all memory allocation / release requests and GPU task scheduling instructions are transmitted back and forth through RPC. This ensures that every operation of the container on the GPU is executed within the controllable range of the host machine, achieving remote unified management and control. RPC communication also reduces the coupling between the container and the GPU device, improving system security.

[0067] In terms of RPC memory allocation protocol: a standardized RPC interface is defined for memory management, such as memory allocation (AllocateMemory) and release (FreeMemory) operations. By encapsulating the details of kompute and Vulkan underlying calls through these interfaces, the container does not need to understand the specific implementation of memory allocation, but only needs to request the required resources through the RPC interface. This achieves interface abstraction and protocol standardization of memory management operations.

[0068] In terms of task scheduling scheme: the host machine is responsible for the GPU task scheduling of multiple containers. An intelligent scheduling algorithm is used to allocate GPU computing resources based on the priority, memory usage, and QoS requirements of each container. This ensures timely execution of high-priority tasks while maximizing GPU utilization and reducing resource idle time and conflicts between different tasks.

[0069] In terms of dynamic task scheduling strategy: a dynamic task scheduling based on the state of the memory pool, the priority of the container, and the QoS strategy is implemented in the host scheduling module. The scheduling strategy supports asynchronous parallel scheduling of multiple container tasks and can adjust the task execution order or concurrency level based on real-time load changes. By dynamically adjusting task scheduling, the invention can adapt to load fluctuations and ensure that the established performance indicators and isolation requirements are met.

[0070] Abbreviations and key terms definition: GPU(Graphics Processing Unit): A graphics processing unit responsible for performing high-performance computing tasks, especially in the fields of graphics rendering, machine learning, and scientific computing.

[0071] MIG(Multi-Instance GPU): A multi-instance GPU is a hardware virtualization technology provided by NVIDIA that can divide a single GPU into multiple isolated small GPU instances, achieving memory and computing unit isolation.

[0072] Vulkan: A low-overhead, high-performance graphics and computing API that provides efficient low-level control over GPU hardware acceleration.

[0073] OpenCL(Open Computing Language): An open computing language is a framework and standard for cross-platform parallel computing, which can execute general-purpose computing on heterogeneous devices such as GPUs from different manufacturers.

[0074] kompute: A Vulkan-based GPU computing framework that provides efficient computation graph execution and memory management functions.

[0075] NVML(NVIDIA Management Library): NVIDIA management library provides API interfaces for monitoring and managing NVIDIA GPU devices, such as querying memory usage, etc.

[0076] Tensor: A tensor is a multidimensional data array, a basic data structure in deep learning and high-performance computing, representing input data, weights, or computing results, etc.

[0077] Memory pool: Logical division of GPU memory. Each container has an independent memory pool and can only access the memory space within its own pool to avoid resource conflicts.

[0078] RPC (Remote Procedure Call): A remote procedure call is an inter-process communication mechanism. Containers request the host to perform certain operations (such as memory allocation or task scheduling) through RPC, and the host returns the results after processing, realizing cross-process function calls.

[0079] Container: A lightweight virtualized application environment that can encapsulate applications and their dependencies, supporting the isolated running of multiple instances on a shared operating system kernel.

[0080] QoS(Quality of Service): Service quality is a set of predefined policies or parameters used to ensure the priority and performance requirements of different tasks or containers in resource allocation and task scheduling.

[0081] API (Application Programming Interface): Application Programming Interface.

[0082] SR-IOV (Single Root I / O Virtualization): Single Root I / O Virtualization.

[0083] The above merely describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical scheme falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A method for GPU memory isolation and scheduling between containers, characterized in that, The method is implemented based on a management system, which includes a host machine and multiple containers; each container corresponds to an independent video memory pool on the host machine; the method includes a video memory pool management process. The specific steps of the video memory pool management process are as follows: Step S1: The container issues a video memory allocation call, which is then encapsulated as an RPC call request and sent to the host machine; Step S2: The host machine locates the corresponding video memory pool of the container based on the RPC call request; Step S3: The host machine determines whether there is a free memory block in the memory pool corresponding to the container that satisfies the RPC call request and allocates it to the container; Step S4: If there is a free memory block in the memory pool that satisfies the RPC call request, proceed to step S6; if there is no free memory block in the memory pool that satisfies the RPC call request, proceed to step S5. Step S5: If there is no free memory block in the memory pool that satisfies the RPC call request, then merge adjacent free memory blocks in the memory pool and determine whether the merged free memory block satisfies the RPC call request; if the merged free memory block satisfies the RPC call request, then proceed to step S6; if the merged free memory block still does not satisfy the RPC call request, then the host fails to allocate a free memory block and the host returns the second allocation result to the container. Step S6: The host machine allocates a free memory block from the memory pool that satisfies the RPC call request to the corresponding container and obtains the first allocation result, which includes allocation success information and memory handle or corresponding Tensor ID. Step S7: The host machine returns the first allocation result to the container.

2. The inter-container GPU memory isolation and scheduling method according to claim 1, characterized in that, The container includes an RPC client module, an interceptor, and a computing task module that are connected in sequence; the host machine includes a call interface, an RPC service module, a video memory management module, and a GPU device. The RPC service module communicates with the memory management module through API calls; the memory management module communicates with the GPU device through the driver interface; the memory management module creates a corresponding memory pool for each container on the GPU device. The RPC client module communicates with the RPC service module.

3. The inter-container GPU memory isolation and scheduling method according to claim 2, characterized in that, Step S1 includes the following steps: Step S11: When the container's computing task module needs the host machine to allocate video memory, it issues a video memory allocation call; Step S12: After intercepting the video memory allocation call, the interceptor forwards the video memory allocation call to the RPC client module; Step S13: The RPC client module packages the video memory allocation request call into an RPC allocation request and sends it to the host machine.

4. The inter-container GPU memory isolation and scheduling method according to claim 2, characterized in that, Step S2 includes the following steps: Step S21: The host machine's RPC service module receives the RPC allocation request and passes it to the calling interface. Step S22: The API call translates the RPC allocation request into underlying call information and passes it to the video memory management module; Step S23: The video memory management module locates the video memory pool corresponding to the container based on the underlying call information.

5. The inter-container GPU memory isolation and scheduling method according to claim 3, characterized in that, Step S7 includes the following steps: Step S71: The host machine's video memory management module returns the first allocation result to the RPC service module through the RPC communication module; Step S72: The RPC service module returns the first allocation result to the container's RPC client module; Step S73: The RPC client module returns the first allocation result to the container's computing task module through the interceptor; the computing task module uses the free memory block allocated by the memory pool based on the first allocation result.

6. The inter-container GPU memory isolation and scheduling method according to claim 5, characterized in that, The memory pool management process also includes step S8; Step S8: After the container's computing task module finishes using the allocated free video memory block, the container's computing task module performs a video memory release call. After the interceptor captures the memory release call, it forwards the memory release call to the RPC client module; The RPC client module packages the video memory release call into an RPC release request and sends it to the host machine's RPC service module; The RPC service module translates the RPC release request into underlying release information by calling the interface, and then passes the underlying release information to the video memory management module. The video memory management module releases the corresponding video memory block based on the underlying release information.

7. The inter-container GPU memory isolation and scheduling method according to claim 2, characterized in that, The host machine also includes a task scheduling module, which is connected to the calling interface and GPU device to receive the container's computing tasks and complete the task scheduling of the computing tasks. The task scheduling process for computation tasks includes: Step A1: The container submits the first computing task, which is encapsulated as an RPC task request and sent to the host machine; Step A2: After receiving the RPC task request, the host machine translates the RPC task request into the task information of the first computing task by calling the interface; the task information of the first computing task is forwarded to the task scheduling module. Step A3: Based on the task information of the first computing task, the task scheduling module creates a task unit corresponding to the first computing task and adds the first computing task to the GPU task queue; Step A4: The task scheduling module makes scheduling decisions on the GPU task queue based on the task scheduling policy; when the task scheduling module schedules the first computing task, the task scheduling module submits the task unit corresponding to the first computing task to the GPU device; Step A5: The GPU device executes the first computing task based on the task unit corresponding to the first computing task, and the task scheduling module monitors the task status of the first computing task; Step A6: When the GPU device completes the first computing task, the GPU device obtains the task result of the first computing task; the task scheduling module reads the task result of the first computing task from the GPU device and encapsulates the task result of the first computing task through the RPC service module and sends it back to the container; the task result includes the execution status of the first computing task and the output data.

8. The inter-container GPU memory isolation and scheduling method according to claim 5, characterized in that, Step A1 includes the following steps: Step A11: The container's computation task module constructs the first computation task; Step A12: After the interceptor intercepts the first computing task, it forwards the first computing task to the RPC client module; Step A13: The RPC client module encapsulates the first computation task into an RPC task request and sends it to the RPC service module on the host machine.

9. The inter-container GPU memory isolation and scheduling method according to claim 8, characterized in that, The task scheduling process also includes step A7; Step A7: After receiving the encapsulated task result of the first computation task, the container's RPC client module restores the encapsulated task result and passes it to the interceptor. The interceptor then sends the task result of the first computation task to the computation task module that initiated the first computation task.

10. The inter-container GPU memory isolation and scheduling method according to claim 2, characterized in that, The API call is the kompute API call, and the driver API is the Vulkan driver API.

Images