Multi-tenant GPU fine-grained access control method and system and medium

Abstract

The invention discloses a multi-tenant GPU fine-grained access control method and system and a medium, mainly relates to the technical field of access control, and is used for solving the problems that the control granularity of a K8s native scheme is relatively coarse, hardware binding limitation exists in equipment kits, drivers, operators and the like of GPU manufacturers and the like. Comprising the following steps: screening working nodes conforming to GPU Filter logic through a preset scheduler containing the GPU Filter logic; the corresponding working node returns GPU information, meeting the Allocation request, of the current working node to the Kubelet through Device Plug, wherein the GPU information meets the Allocation request of the current working node; the Kubelet injects GPU information returned by the Device Plug into container configuration, and a container is started when the container is called to run; and when the life cycle of the target Pod is ended, the GPUClaim controller deletes the associated GPUClaim instance.

Description

Technical Field

[0001] This application relates to the field of GPU access control technology, and in particular to a method, system and medium for fine-grained access control of multi-tenant GPUs. Background Technology

[0002] In the context of the rapid development of large-scale AI models, the training and inference of large models heavily rely on high-performance computing resources such as GPUs. It has become common for multiple teams and users within enterprises to share a single GPU-deployed environment. To address the resource management needs in such shared scenarios, existing technologies mainly include two types of solutions: First, native Kubernetes solutions, which specify the number of GPUs through Resources, limit the total amount of resources in the Namespace nvidia.com / gpu using ResourceQuota, and report the number of GPU resources using DevicePlugins (such as NVIDIA / k8s-device-plugin). They can also specify the GPU device number or name used by a Pod through environment variables combined with worker node tags. Second, vendor-specific solutions such as NVIDIA GPUOperator, which can automatically deploy GPU drivers, device plugins, and DCGM monitoring, and supports MIG (Multi-Instance GPU) partitioning.

[0003] Existing technologies have significant shortcomings: The native Kubernetes solution offers coarse-grained control, only limiting the number of GPUs and failing to differentiate between GPU models, computing power, or memory. It also lacks a policy engine, making it difficult to implement tenant-based GPU usage rules. Furthermore, it suffers from security isolation vulnerabilities, allowing malicious Pods to manipulate other tenants' GPU tasks via nvidia-smi. Additionally, specifying GPUs via environment variables cannot prevent Pods from using GPUs; departments authorized to deploy Pods can use all GPU resources without restriction if they don't set the relevant environment variables. NVIDIA GPUOperator, on the other hand, has hardware-binding limitations, supporting only enterprise-grade GPUs (such as A100 / A30). MIG configuration is complex and requires manual partitioning of computing units, resulting in poor flexibility. It also lacks tenant-level control policies, making fine-grained management, such as "tenant A can only use 2 MIG instances," impossible. Summary of the Invention

[0004] This application provides a multi-tenant GPU fine-grained access control method, system, and medium to solve the problems of the coarse control granularity of the native K8s solution. In addition, the method of specifying the GPU through environment variables cannot restrict Pods from using the GPU, and NVIDIA GPUOperator has hardware binding restrictions, poor flexibility, and lack of fine-grained management.

[0005] Firstly, this application provides a fine-grained access control method for multi-tenant GPUs, the method comprising: When the DevicePlugin on the worker node starts, a GPU instance is created; the GPU instance contains GPU information. Create a ClusterRole containing GPUAccessPolicy creation permissions; create a GPUAccessPolicy instance that defines the GPU instance access policy through the ClusterRole; Define the permission rules and scope of the GPUClaim; create a GPUClaim instance that defines the GPU instance that the Pod will request; bind the GPUClaim instance to the target Pod; The system verifies the target Pod creation request by using a pre-defined admission controller; it also verifies the existence of the GPUClaim instance associated with the creation request, and verifies the target Pod's access rights to the corresponding GPU instance based on the target Pod, GPUClaim instance, and GPUAccessPolicy. When a GPUClaim instance exists and has the corresponding GPU instance access permission, a preset scheduler containing GPU Filter logic is used to filter worker nodes that match the GPU Filter logic. Eligible worker nodes send an Allocate request to the Device Plugin via Kubelet; the Device Plugin returns GPU information that satisfies the Allocate request to the Kubelet; the Kubelet injects the GPU information returned by the Device Plugin into the container configuration and calls the container runtime to start the container; When the lifecycle of the target Pod ends, the GPUClaim controller deletes the associated GPUClaim instance and notifies the Device Plugin to release the corresponding GPU instance resources.

[0006] In one implementation of this application, when the DevicePlugin on the worker node starts, a GPU instance is created, specifically including: When the Kubelet on the worker node starts, it provides an interface for registering the DevicePlugin for the GPU instance; When DevicePlugin starts, it registers a port with kubelet to receive GPRC requests, calls the GPU driver interface of the current worker node to obtain the GPU card information of the current worker node, and sends a request to Kubelet to create a GPU instance.

[0007] In one implementation of this application, a preset admission controller is used to verify the target Pod creation request, specifically including: By using a pre-defined admission controller, the creation request of the target Pod is checked, the annotation field is filtered, and the existence of the specified GPUClaim instance is verified.

[0008] In one implementation of this application, the target Pod's access rights to the corresponding GPU instance are verified based on the target Pod, the GPUClaim instance, and the GPUAccessPolicy. Specifically, this includes: When the target Pod specifies a GPUclaim instance, and the target Pod is within the defined permission rules and scope of GPUAccessPolicy, the target Pod has the right to access the GPU instance, and the default admission controller returns a success message. When the target Pod specifies a GPUclaim instance, but the target Pod is not within the defined permission rules and scope of GPUAccessPolicy, the target Pod does not have permission to access the GPU instance, and the default admission controller returns a failure message. When a target Pod specifies the number of GPUs in the resource field but does not specify the GPUClaim in the annotation field, the default admission controller assumes that the target Pod does not have permission to access the GPU instance and returns an error message.

[0009] In one implementation of this application, before filtering worker nodes that match the GPU Filter logic using a preset scheduler that includes GPU Filter logic, the method further includes: Configure the GPU Filter logic corresponding to the preset scheduler.

[0010] In one implementation of this application, after returning GPU information indicating that the current worker node satisfies the Allocate request to the Kubelet via the Device Plugin, the method further includes: The Device Plugin synchronizes the remaining available GPU information of the node with the Kubelet.

[0011] Secondly, this application provides a multi-tenant GPU fine-grained access control system, the system comprising: The GPU instance creation module is used to create a GPU instance when the DevicePlugin on the worker node starts; the GPU instance contains GPU information. The access policy creation module is used to create a ClusterRole containing GPUAccessPolicy creation permissions; and to create a GPUAccessPolicy instance that defines the GPU instance access policy through the ClusterRole. The access request creation module is used to define the permission rules and scope of the GPUClaim; create a GPUClaim instance that defines the GPU instance that the Pod will request; and bind the GPUClaim instance to the target Pod. The admission controller module is used to verify the target Pod creation request through a preset admission controller; verify the existence of the GPUClaim instance associated with the creation request; and verify the target Pod's usage rights to the corresponding GPU instance based on the target Pod, GPUClaim instance, and GPUAccessPolicy. The filtering node module is used to filter worker nodes that match the GPU Filter logic when a GPUClaim instance exists and the user has the corresponding GPU instance access rights, using a preset scheduler that includes GPU Filter logic. Allocate a specified GPU module for eligible worker nodes to send an Allocate request to the Device Plugin via Kubelet; the Device Plugin returns GPU information that satisfies the Allocate request to the Kubelet; the Kubelet injects the GPU information returned by the Device Plugin into the container configuration and calls the container runtime to start the container; The resource release module is used to notify the Device Plugin to release the corresponding GPU instance resources when the lifecycle of the target Pod ends, by deleting the associated GPUClaim instance from the GPUClaim controller.

[0012] In one implementation of this application, the GPU instance creation module includes a GPU instance creation unit. This interface is used to provide the registration of the DevicePlugin for GPU instances when the Kubelet on the worker node starts. When DevicePlugin starts, it registers a port with kubelet to receive GPRC requests, calls the GPU driver interface of the current worker node to obtain the GPU card information of the current worker node, and sends a request to Kubelet to create a GPU instance.

[0013] In one implementation of this application, the admission controller module includes a verification unit. When the target Pod specifies a GPUclaim instance and the target Pod is within the defined permission rules and scope of GPUAccessPolicy, the target Pod is authorized to access the GPU instance. The default access controller returns a success message. When the target Pod specifies a GPUclaim instance, but the target Pod is not within the defined permission rules and scope of GPUAccessPolicy, the target Pod does not have permission to access the GPU instance, and the default admission controller returns a failure message. When a target Pod specifies the number of GPUs in the resource field but does not specify the GPUClaim in the annotation field, the default admission controller assumes that the target Pod does not have permission to access the GPU instance and returns an error message.

[0014] Thirdly, this application provides a non-volatile computer storage medium storing computer instructions thereon, which, when executed, implement a multi-tenant GPU fine-grained access control method as described above.

[0015] As can be seen from the above technical solutions, this application has the following advantages: 1. Improved fine-grained access control capabilities By defining access policies using GPUAccessPolicy, this application enables fine-grained permission management of GPU resources. Compared to native Kubernetes solutions, this application allows for flexible configuration of GPU usage permissions for different tenants or Pods, preventing resource abuse. For example, when verifying Pod creation requests through a preset admission controller, it can precisely control the Pod's access permissions to specific GPU instances, solving the problem that traditional methods of specifying GPUs using environment variables cannot prevent Pods from using GPUs.

[0016] 2. Enhanced hardware decoupling and scheduling flexibility: A pre-defined scheduler using GPUFilter logic filters worker nodes that meet certain criteria, and in conjunction with DevicePlugin, dynamically allocates GPU resources, breaking the hardware binding limitations of NVIDIA GPUOperator. Worker nodes do not need to be pre-bound to specific GPU devices; Kubelet dynamically obtains available GPU information and injects it into container configurations via Allocate requests, improving the flexibility of resource scheduling and cluster utilization.

[0017] 3. Optimize full lifecycle resource management: By binding GPUClaim instances to Pods, the entire process of GPU resource allocation, usage, and release is managed. When a Pod's lifecycle ends, the GPUClaim controller automatically deletes the associated instance and notifies the DevicePlugin to release resources, preventing resource leaks. This mechanism ensures efficient GPU resource reclamation and reallocation, making it particularly suitable for dynamic resource scheduling needs in multi-tenant scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a multi-tenant GPU fine-grained access control method provided in an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the internal structure of a multi-tenant GPU fine-grained access control system provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0023] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0024] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0025] The embodiment provides a fine-grained access control method for multi-tenant GPUs, such as... Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps: Step 110: When the DevicePlugin on the worker node starts, create a GPU instance.

[0026] The GPU instance contains GPU information.

[0027] In some embodiments, when the DevicePlugin on the worker node starts, a GPU instance is created, specifically including: When the Kubelet on the worker node starts, it provides an interface for registering the DevicePlugin for the GPU instance; When DevicePlugin starts, it registers a port with kubelet to receive GPRC requests, calls the GPU driver interface of the current worker node to obtain the GPU card information of the current worker node, and sends a request to Kubelet to create a GPU instance.

[0028] It should be noted that the GPU information for a GPU instance includes: manufacturer, model, name, device ID, and video memory size. A corresponding Resource Controller is added to perform CRUD operations on this resource. The instance of this resource is created when the Device Plugin on the worker node starts.

[0029] Step 120: Create a ClusterRole containing GPUAccessPolicy creation permissions; create a GPUAccessPolicy instance that defines the GPU instance access policy through the ClusterRole.

[0030] It should be noted that GPUAccessPolicy is a custom resource GPU access policy that defines the access permissions of selected Kubernetes resources to GPU devices. The creation of this resource instance is performed by an authorized administrator and can be defined in conjunction with Role / ClusterRole and Rolebinding / ClusterRolebinding. Key attributes include: An array of GPU instance names, consisting of one or more GPU resource instance names. Resource filters for accessing the GPU, such as Namespace Selector, Pod Selector, and other filter fields that can select Kubernetes resources. Permission-related attributes, such as whether access is allowed, usage mode, memory size, and usage time.

[0031] Specifically, this step involves: creating a ClusterRole that can create GPUAccessPolicies, such as defining it so that only a specific user has permission to set GPU access policies, and other users do not have permission to define them. Then, creating a GPUAccessPolicy instance, defining which GPU instances can be accessed by which namespaces or Pods. The GPU instances are defined in step 110.

[0032] Step 130: Define the permission rules and scope of the GPUClaim; create a GPUClaim instance that defines the GPU instance that the Pod will request; bind the GPUClaim instance to the target Pod.

[0033] It's important to note that a GPUClaim is a custom GPU resource request that defines the GPU resources a Pod will request. This includes, for example, an array of GPU instance names, the requested GPU size, the namespace, and the Pod Selector. This resource is used to bind to the target Pod. The creation of this resource instance is handled by personnel authorized to create GPUClaims.

[0034] Specifically, this step involves: creating the definition permission rules (ClusterRole) and scope (Rolebinding) for the GPUClaim, specifying which namespaces have permission to request GPU resources. For example, in a multi-tenant system, some departments may have permission to use GPUs, so the Rolebinding for the namespace of that tenant can be configured. Next, create a GPUClaim instance, specifying the GPU instance name, GPU usage method, GPU usage amount, and other data. Finally, bind the GPUClaim instance to the target pod using annotations.

[0035] Step 140: Verify the target Pod creation request through the preset access controller; verify the existence of the GPUClaim instance associated with the creation request; and verify the target Pod's usage rights to the corresponding GPU instance based on the target Pod, GPUClaim instance, and GPUAccessPolicy.

[0036] Specifically, the target Pod creation request is verified through a pre-defined admission controller, including: By using a pre-defined admission controller, the creation request of the target Pod is checked, the annotation field is filtered, and the existence of the specified GPUClaim instance is verified.

[0037] Specifically, based on the target Pod, the GPUClaim instance, and the GPUAccessPolicy, the verification of the target Pod's access rights to the corresponding GPU instance includes: When the target Pod specifies a GPUclaim instance, and the target Pod is within the defined permission rules and scope of GPUAccessPolicy, the target Pod has the right to access the GPU instance, and the default admission controller returns a success message. When the target Pod specifies a GPUclaim instance, but the target Pod is not within the defined permission rules and scope of GPUAccessPolicy, the target Pod does not have permission to access the GPU instance, and the default admission controller returns a failure message. When a target Pod specifies the number of GPUs in the resource field but does not specify the GPUClaim in the annotation field, the default admission controller assumes that the target Pod does not have permission to access the GPU instance and returns an error message.

[0038] Step 150: When a GPUClaim instance exists and has the corresponding GPU instance usage permission, a preset scheduler containing GPUFilter logic is used to filter worker nodes that match the GPUFilter logic.

[0039] Before filtering worker nodes that match the GPU Filter logic using a preset scheduler that includes GPU Filter logic, the method also includes: Configure the GPU Filter logic corresponding to the preset scheduler.

[0040] It should be noted that this step is as follows: If the target Pod has the authority to request the corresponding GPU, it enters the container filtering and scheduling phase. The scheduler's filtering phase is extended by adding a GPU Filter. Based on the target Pod's request, the scheduler selects the worker nodes that meet the conditions, scores them, and outputs the optimal worker node that meets the conditions.

[0041] Step 160: The eligible worker node sends an Allocate request to the Device Plugin through the Kubelet; the Device Plugin returns GPU information that the current worker node meets the Allocate request to the Kubelet; the Kubelet injects the GPU information returned by the Device Plugin into the container configuration and calls the container runtime to start the container.

[0042] After returning GPU information to Kubelet via the Device Plugin indicating that the current worker node satisfies the Allocate request, the method also includes: The Device Plugin synchronizes the remaining available GPU information of the node with the Kubelet.

[0043] It should be noted that this step is as follows: The Kubelet on the worker node receives the instruction, calls the runtime to start the container, the Kubelet sends an Allocate request to the Device Plugin, the extended Device Plugin processes the specified GPU information in the Allocate request, such as the specified GPU name, video memory size, etc., and returns the GPU information on the current worker node to the Kubelet, the Kubelet calls the container runtime to create the container, and the Device Plugin updates the node's available GPU information to the Kubelet.

[0044] Step 170: When the lifecycle of the target Pod ends, the GPUClaim controller deletes the associated GPUClaim instance and notifies the Device Plugin to release the corresponding GPU instance resources.

[0045] It should be noted that this step can be specifically as follows: The extended Pod controller listens for the Pod's delete event. When the Pod's lifecycle ends, the controller triggers the GPUClaim resource's delete event. The GPUClaim controller monitors the creation, deletion, modification, and query of GPUClaim resources, and notifies the Device Plugin to mark the corresponding GPU resources as available, thus completing the release of the GPU resources.

[0046] As described above, this embodiment achieves hardware resource abstraction and permission isolation through the introduction of GPU instantiation (step 110) and GPUAccessPolicy (step 120). DevicePlugin encapsulates parameters such as the physical GPU manufacturer and memory into standardized resources. Combined with the dual filtering mechanism of Namespace Selector and Pod Selector, it can precisely control the access permissions of different tenants or teams. For example, the R&D team can only apply for a specific model of GPU through a predefined GPUClaim (step 130), avoiding system instability caused by resource contention or misoperation. The real-time verification mechanism of the access controller (step 140) further strengthens the permission boundary. When a Pod is not bound through GPUClaim or exceeds the policy scope, the system directly blocks the request, avoiding the risk of unauthorized access from the source.

[0047] The extended scheduler's GPU Filter logic (step 150) enables intelligent matching of resource requirements with node capabilities. Compared to traditional static allocation methods, this solution can dynamically filter nodes based on the GPU memory size and GPU model requested by the Pod, reducing the overhead of cross-node scheduling. For example, when multiple worker nodes have heterogeneous GPUs, the scheduler prioritizes nodes with sufficient GPU memory and matching GPU models, avoiding waste caused by resource fragmentation. The DevicePlugin's real-time synchronization mechanism (step 160) keeps the cluster resource view up-to-date. Combined with resource release triggered by the Pod's lifecycle (step 170), the GPU memory reclamation latency is reduced from minutes to seconds, significantly improving resource turnover in high-concurrency scenarios.

[0048] By decoupling custom resources (GPUClaim, GPUAccessPolicy) from standard Kubernetes components, the solution reduces the management burden on operations personnel. Administrators do not need to directly manipulate node-level configurations; they only need to define policies through CRDs (CustomResourceDefinitions) to complete permission allocation. For example, when adding a new GPU model, only the driver interface in DevicePlugin needs to be extended (step 110), while the scheduling policy (step 150) and access control (step 120) do not need to be modified. This modular design facilitates future support for more heterogeneous computing resources. At the same time, the loose coupling between the GRPC interface (step 110) and Kubelet ensures that plugin upgrades or replacements do not affect the core functionality of the cluster.

[0049] In addition, this application Figure 2 This application provides a multi-tenant GPU fine-grained access control system as an embodiment. For example... Figure 2 As shown in the embodiments of this application, the system mainly includes: GPU instance creation module 210 is used to create a GPU instance when the DevicePlugin on the worker node starts; the GPU instance contains GPU information.

[0050] The GPU instance creation module 210 includes a GPU instance creation unit. This interface is used to provide the registration of the DevicePlugin for GPU instances when the Kubelet on the worker node starts. When DevicePlugin starts, it registers a port with kubelet to receive GPRC requests, calls the GPU driver interface of the current worker node to obtain the GPU card information of the current worker node, and sends a request to Kubelet to create a GPU instance.

[0051] Access policy creation module 220 is used to create a ClusterRole containing GPUAccessPolicy creation permissions; and to create a GPUAccessPolicy instance that defines the GPU instance access policy through the ClusterRole.

[0052] Access request creation module 230 is used to create the definition permission rules and scope of GPUClaim; create a GPUClaim instance that defines the GPU instance that the Pod will request; and bind the GPUClaim instance to the target Pod.

[0053] The admission controller module 240 is used to verify the target Pod creation request through a preset admission controller; verify the existence of the GPUClaim instance associated with the creation request; and verify the target Pod's usage rights to the corresponding GPU instance based on the target Pod, GPUClaim instance, and GPUAccessPolicy.

[0054] The access controller module 240 includes a verification unit. When the target Pod specifies a GPUclaim instance and the target Pod is within the defined permission rules and scope of GPUAccessPolicy, the target Pod is authorized to access the GPU instance. The default access controller returns a success message. When the target Pod specifies a GPUclaim instance, but the target Pod is not within the defined permission rules and scope of GPUAccessPolicy, the target Pod does not have permission to access the GPU instance, and the default admission controller returns a failure message. When a target Pod specifies the number of GPUs in the resource field but does not specify the GPUClaim in the annotation field, the default admission controller assumes that the target Pod does not have permission to access the GPU instance and returns an error message.

[0055] The filtering node module 250 is used to filter worker nodes that match the GPU Filter logic by using a preset scheduler that includes GPU Filter logic when a GPUClaim instance exists and the user has the corresponding GPU instance access rights.

[0056] Allocate a specified GPU module 260, which is used by eligible worker nodes to send an Allocate request to the Device Plugin via Kubelet; the Device Plugin returns GPU information that satisfies the Allocate request to the Kubelet; the Kubelet injects the GPU information returned by the Device Plugin into the container configuration and calls the container runtime to start the container.

[0057] The resource release module 270 is used to notify the Device Plugin to release the corresponding GPU instance resources when the lifecycle of the target Pod ends, by deleting the associated GPUClaim instance from the GPUClaim controller.

[0058] In addition, embodiments of this application also provide a non-volatile computer storage medium storing executable instructions, which, when executed, implement a multi-tenant GPU fine-grained access control method as described above.

[0059] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fine-grained access control method for multi-tenant GPUs, characterized in that, The method includes: When the DevicePlugin on the worker node starts, a GPU instance is created; the GPU instance contains GPU information. Create a ClusterRole containing GPUAccessPolicy creation permissions; create a GPUAccessPolicy instance that defines the GPU instance access policy through the ClusterRole; Define the permission rules and scope of the GPUClaim; create a GPUClaim instance that defines the GPU instance that the Pod will request; bind the GPUClaim instance to the target Pod; The system verifies the target Pod creation request by using a pre-defined admission controller; it also verifies the existence of the GPUClaim instance associated with the creation request, and verifies the target Pod's access rights to the corresponding GPU instance based on the target Pod, GPUClaim instance, and GPUAccessPolicy. When a GPUClaim instance exists and has the corresponding GPU instance access permission, a preset scheduler containing GPU Filter logic is used to filter worker nodes that match the GPU Filter logic. Eligible worker nodes send an Allocate request to the Device Plugin via Kubelet; the Device Plugin returns GPU information that satisfies the Allocate request to the Kubelet; the Kubelet injects the GPU information returned by the Device Plugin into the container configuration and calls the container runtime to start the container; When the lifecycle of the target Pod ends, the GPUClaim controller deletes the associated GPUClaim instance and notifies the DevicePlugin to release the corresponding GPU instance resources.

2. The multi-tenant GPU fine-grained access control method according to claim 1, characterized in that, When the DevicePlugin starts on the worker node, it creates a GPU instance, specifically including: When the Kubelet on the worker node starts, it provides an interface for registering the DevicePlugin for the GPU instance; When DevicePlugin starts, it registers a port with kubelet to receive GPRC requests, calls the GPU driver interface of the current worker node to obtain the GPU card information of the current worker node, and sends a request to Kubelet to create a GPU instance.

3. The multi-tenant GPU fine-grained access control method according to claim 1, characterized in that, Verify the target Pod creation request through a pre-defined admission controller, specifically including: By using a pre-defined admission controller, the creation request of the target Pod is checked, the annotation field is filtered, and the existence of the specified GPUClaim instance is verified.

4. The multi-tenant GPU fine-grained access control method according to claim 1, characterized in that, Based on the target Pod, GPUClaim instance, and GPUAccessPolicy, verify the target Pod's usage permissions to the corresponding GPU instance, specifically including: When the target Pod specifies a GPUclaim instance, and the target Pod is within the defined permission rules and scope of GPUAccessPolicy, the target Pod has the right to access the GPU instance, and the default admission controller returns a success message. When the target Pod specifies a GPUclaim instance, but the target Pod is not within the defined permission rules and scope of GPUAccessPolicy, the target Pod does not have permission to access the GPU instance, and the default admission controller returns a failure message. When a target Pod specifies the number of GPUs in the resource field but does not specify the GPUClaim in the annotation field, the default admission controller assumes that the target Pod does not have permission to access the GPU instance and returns an error message.

5. The multi-tenant GPU fine-grained access control method according to claim 1, characterized in that, Before filtering worker nodes that match the GPU Filter logic using a preset scheduler that includes GPU Filter logic, the method further includes: Configure the GPU Filter logic corresponding to the preset scheduler.

6. The multi-tenant GPU fine-grained access control method according to claim 1, characterized in that, After returning GPU information to Kubelet via DevicePlugin indicating that the current worker node satisfies the Allocate request, the method further includes: The Device Plugin synchronizes the remaining available GPU information of the node with the Kubelet.

7. A multi-tenant GPU fine-grained access control system, characterized in that, The system includes: The GPU instance creation module is used to create a GPU instance when the DevicePlugin on the worker node starts; the GPU instance contains GPU information. The access policy creation module is used to create a ClusterRole containing GPUAccessPolicy creation permissions; and to create a GPUAccessPolicy instance that defines the GPU instance access policy through the ClusterRole. The access request creation module is used to define the permission rules and scope of the GPUClaim; create a GPUClaim instance that defines the GPU instance that the Pod will request; and bind the GPUClaim instance to the target Pod. The admission controller module is used to verify the target Pod creation request through a preset admission controller; verify the existence of the GPUClaim instance associated with the creation request; and verify the target Pod's usage rights to the corresponding GPU instance based on the target Pod, GPUClaim instance, and GPUAccessPolicy. The filtering node module is used to filter worker nodes that match the GPU Filter logic when a GPUClaim instance exists and the user has the corresponding GPU instance access rights, using a preset scheduler that includes GPU Filter logic. Allocate a specified GPU module for eligible worker nodes to send an Allocate request to the Device Plugin via Kubelet; the Device Plugin returns GPU information that satisfies the Allocate request to the Kubelet; the Kubelet injects the GPU information returned by the Device Plugin into the container configuration and calls the container runtime to start the container; The resource release module is used to notify the Device Plugin to release the corresponding GPU instance resources when the lifecycle of the target Pod ends, by deleting the associated GPUClaim instance from the GPUClaim controller.

8. The multi-tenant GPU fine-grained access control system according to claim 7, characterized in that, The GPU instance creation module contains GPU instance creation units. This interface is used to provide the registration of the DevicePlugin for GPU instances when the Kubelet on the worker node starts. When DevicePlugin starts, it registers a port with kubelet to receive GPRC requests, calls the GPU driver interface of the current worker node to obtain the GPU card information of the current worker node, and sends a request to Kubelet to create a GPU instance.

9. The multi-tenant GPU fine-grained access control system according to claim 7, characterized in that, The access control module includes a verification unit. When the target Pod specifies a GPUclaim instance and the target Pod is within the defined permission rules and scope of GPUAccessPolicy, the target Pod is authorized to access the GPU instance. The default access controller returns a success message. When the target Pod specifies a GPUclaim instance, but the target Pod is not within the defined permission rules and scope of GPUAccessPolicy, the target Pod does not have permission to access the GPU instance, and the default admission controller returns a failure message. When a target Pod specifies the number of GPUs in the resource field but does not specify the GPUClaim in the annotation field, the default admission controller assumes that the target Pod does not have permission to access the GPU instance and returns an error message.

10. A non-volatile computer storage medium, characterized in that, It stores computer instructions that, when executed, implement a multi-tenant GPU fine-grained access control method as described in any one of claims 1-6.

Images