A GPU time-sharing sharing method and system based on Kubernetes cluster

Abstract

The present invention relates to a GPU time-sharing sharing method and system based on a Kubernetes cluster. The method comprises: setting a front-end module in a working node of the Kubernetes cluster, wherein the front-end module is a GPU device library installed inside a container, and the GPU device library schedules the use of the GPU in tasks by intercepting all memory and computing-related APIs in the CUDA library; setting a back-end module in the working node of the Kubernetes cluster, wherein the back-end module manages tokens and time quotas between containers, and implements GPU time-sharing through tokens and time quotas. The present invention proposes a rule for GPU time-sharing sharing, adds a GPU device library as a front-end in the image, and adds a back-end module to the node for control, which can ensure the security and fairness of task execution between multiple containers in the node.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for implementing time-sharing sharing of GPU resources based on a Kubernetes cluster. Background Art

[0002] In recent years, information technology has advanced rapidly, and internet-related technologies have matured at an unprecedented pace, making our daily lives and work increasingly dependent on computers. Simultaneously, the rapid development of these technologies has led to an increasing demand for both the quality and quantity of computing services. However, traditional computing models are no longer able to meet these demands, leading to the emergence of new models to replace them. Cloud computing is a representative example. Emerging as a new, internet-based computing model, it represents a significant revolution in modern science and technology. Cloud computing is considered the next generation of the internet and has become a core pillar of future IT research.

[0003] Since the application of cloud computing technology, hardware performance optimization and network-based software computing have led to its rapid growth. Cloud computing technology has also attracted increasing attention from major large and medium-sized Internet companies.

[0004] With the development of virtualization and container technologies, using cloud services has become much simpler and more convenient. Containers, with their lightweight nature, have gradually replaced traditional virtual machine technologies. While there are many different container technologies, Docker is currently the dominant technology. Kubernetes, a container orchestration tool developed by Google, evolved from the Borg system, which Google had used internally for many years. Therefore, Kubernetes's architecture and technology were already relatively mature when it was first released, attracting widespread attention within the cloud computing industry and quickly becoming a mainstream orchestration tool. The use of a distributed cloud computing cluster platform based on Kubernetes accelerates data processing. Compared to using personal computers or traditional clusters, cloud clusters significantly improve the efficiency of task deployment and execution.

[0005] Cloud computing technology continues to advance, expanding its scale and user base, leading to an increasing number of tasks requiring processing. The efficient allocation of these tasks to appropriate computing resources impacts the overall efficiency and service quality of cloud computing. When executing tasks on a shared GPU, once a task is executed on the GPU, all subsequent tasks must wait for it to complete before executing, which can compromise task execution fairness. As a core cluster resource, GPU resources are becoming increasingly expensive with each version update. Therefore, how to ensure secure and reasonable resource sharing between tasks while simultaneously improving cluster resource utilization has become a hot topic.

[0006] Patent publication number CN111506404A, titled "A Shared GPU Scheduling Method Based on Kubernetes," proposes a shared GPU scheduling method comprising the following steps: Step 1: GPU information reporting and statistics; Step 2: Sharing the GPU for use by multiple Pods. Compared to Kubernetes' support for exclusive GPU scheduling, this Kubernetes-based shared GPU scheduling method enhances shared scheduling capabilities and supports multiple scheduling strategies to meet GPU usage requirements in different scenarios. This method combines Kubernetes and GPUs to fully and rationally utilize GPU resources. Kubernetes can be composed of multiple nodes, each with or without a graphics card. Resources are created and GPU memory resource requests are added using the Kubernetes API. Upon receiving the GPU memory resource request, Kubernetes sends the request to the GPU scheduling plugin, which schedules the Pod to a node with sufficient GPU memory quota and labels the GPU ID. Finally, multiple Pods run on a single GPU, achieving GPU sharing. The invention does not specify the type of GPU physical resources, making this method potentially inapplicable in practice. Furthermore, the method still uses virtual machine technology, which consumes significant system resources.

[0007] Patent publication number CN111223036A, titled "A GPU Virtualization Sharing Method, Apparatus, Electronic Device, and Storage Medium," proposes a GPU virtualization sharing method, apparatus, electronic device, and computer-readable storage medium. The method comprises: determining a target physical GPU and dividing the target physical GPU into multiple virtual GPUs; upon receiving a target task, selecting a target virtual GPU from the target physical GPU so that the target task can be executed using the target virtual GPU; during the execution of the target task, upon receiving a call request for a CUDA function, redirecting the call request to a hijacking library; wherein the hijacking library includes multiple CUDA functions corresponding to duplicated names, and the return value of each duplicated name function is determined based on information about the target virtual GPU. The GPU virtualization sharing method provided in this patent application uses hijacking and isolation technologies to separate a physical GPU resource into multiple virtual GPU resources, thereby enabling multi-user, multi-task sharing of the GPU. However, this method uses multiple virtual GPUs for sharing, which is very unstable in large production clusters and can easily cause cluster or node crashes.

[0008] Most current GPU sharing methods schedule GPU resources spatially, but there are still more or less problems with time-sharing sharing. For example, each task requires GPU memory, and the amount of GPU memory resources requested is estimated, making it difficult to ensure that the maximum amount of resources requested will not be exceeded during operation. Overuse or excessive use of resources often leads to performance interference issues. More seriously, this may cause the application to fail or crash. Therefore, in addition to enabling GPU sharing between containers, we must also ensure that the container's GPU resource usage can be limited based on the container's requested resource requirements. How to further reduce GPU resource waste and ensure that shared tasks are executed reasonably and efficiently on the GPU has become a top priority.

[0009] The emergence of cloud computing has reduced infrastructure investment for businesses and users. Leveraging cloud computing's resource-sharing model, products can be developed and deployed more quickly, eliminating the need to wait until infrastructure and programming environments are fully built. This improves development efficiency and product manageability while also reducing the need for routine infrastructure maintenance. Furthermore, in the current era of big data, utilizing cloud machines to process data has significantly improved production efficiency. With the continuous development of cloud computing technology, cloud computing is becoming increasingly large, with an increasing number of users using it and a growing number of tasks requiring processing. The rational and efficient use of cluster resources, especially expensive GPU resources, impacts the overall efficiency and service quality of cloud computing. Therefore, GPU sharing in cloud computing has become a key issue in cloud computing research. Summary of the Invention

[0010] In response to the above problems, the present invention provides a method and system for realizing time-sharing sharing of GPU resources based on Kubernetes cluster.

[0011] The technical solution adopted in the present invention is as follows:

[0012] A GPU time-sharing method based on a Kubernetes cluster includes the following steps:

[0013] Set up a front-end module in the worker nodes of the Kubernetes cluster. The front-end module is a GPU device library installed inside the container. The GPU device library schedules the use of the GPU in tasks by intercepting all memory and computing-related APIs in the CUDA library.

[0014] A backend module is set in the working node of the Kubernetes cluster. The backend module manages tokens and time quotas between containers, and realizes time-sharing sharing of GPUs through tokens and their time quotas.

[0015] Furthermore, a container can execute its code on the GPU only while holding a valid token; a token is associated with a time quota, and when the time quota exceeds the specified period, the container must reacquire the token to execute.

[0016] Furthermore, the backend module tracks the GPU usage time of each container and determines the time quota of the next token to achieve reasonable scheduling of tokens between containers; the GPU usage rate of the container is calculated by the time quota of the valid tokens held within the time range of the sliding window.

[0017] Furthermore, all token requests from the front-end module are queued in the back-end module for processing; when the token itself expires and becomes invalid, the back-end module selects a request from the queue and issues the token to the container of the request.

[0018] Furthermore, the backend module selects a request from the queue, including: first, filtering requests from containers whose GPU usage has exceeded their maximum usage requirements, and then selecting a request from a container whose GPU usage is most different from the minimum usage requirement.

[0019] Furthermore, if all containers requesting tokens have reached the minimum usage requirement, the backend module transfers the token to the container with the lowest current GPU usage.

[0020] Furthermore, the following steps are used to schedule cluster tasks:

[0021] Step 1: Build a GPU sharing cluster using an appropriate number of nodes and install the backend module on the nodes.

[0022] Step 2: The user submits Pod information to the cluster using the kubectl application using a YAML file. The image used in the Pod should contain the GPU device library, i.e., the front-end module. During submission, the cluster determines whether the format of the submitted content meets the cluster characteristics. If not, the user is given feedback. If it does, the cluster proceeds to the next step.

[0023] Step 3: After receiving the Pod information, the API Server stores it in Etcd and prepares to schedule the nodes for these Pods.

[0024] Step 4: After filtering eligible nodes based on the other scheduling resources required by the pod, the scheduler uses the scheduling plug-in in the cluster that manages and schedules GPU memory resources to plan the pod's scheduling plan according to its rules and form a pre-allocation plan.

[0025] Step 5: Execute according to the pre-allocation plan, bind the Pod to be executed to the node through the scheduler, and write the binding information to Etcd;

[0026] Step 6: If the Pod does not run successfully on the node, re-enter step 4;

[0027] Step 7: Repeat steps 1 to 6 until the list of tasks to be scheduled is empty.

[0028] Step 8: The backend module on the node is started, and the Pod running on the GPU node is scheduled using tokens based on the GPU usage of each container;

[0029] Step 9: Repeat steps 2 to 8 until the Pod finishes running.

[0030] A Kubernetes cluster-based GPU time-sharing system adopting the above method includes a front-end module and a back-end module arranged in the working nodes of the Kubernetes cluster; the front-end module is a GPU device library installed in the container, which schedules the use of GPUs in tasks by intercepting all memory and computing-related APIs in the CUDA library; the back-end module manages tokens and token time quotas between containers, and realizes GPU time-sharing through tokens and their time quotas.

[0031] The beneficial effects of the present invention are as follows:

[0032] This paper addresses the current unfair use of shared GPU resources in data processing clusters by proposing a GPU time-sharing rule. To effectively schedule GPU resources and prevent resource over-allocation and competition among jobs, this paper incorporates a GPU device library as a front-end within the image and a back-end control module within the node. This ensures secure and fair execution of tasks across multiple containers within the node. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the Kubernetes cluster thumbnail.

[0034] Figure 2 This is a diagram of Kubernetes cluster improvements. DETAILED DESCRIPTION

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to specific embodiments and accompanying drawings.

[0036] The specific implementation of this patent is based on the Kubernetes cloud cluster. First, a brief introduction to the cluster architecture is given:

[0037] The microservice design concept followed by the Kubernetes system architecture is "everything is a service, everything revolves around the service." It adopts a distributed architecture of the control panel (Master) and worker nodes (Node), with one master node managing multiple slave nodes. Figure 1 shown.

[0038] The Master (control panel) is the cluster control node that manages and controls the entire cluster. Its key processes are as follows:

[0039] (1) API server: This is the entry point for cluster control and the entry point for adding, deleting, modifying, and querying all resources. It also provides the HTTP REST API service.

[0040] (2) Controller Manager: The management and control center of the cluster, responsible for the management of working nodes, Pod replicas, namespaces, etc. within the cluster

[0041] (3) Scheduler: Responsible for scheduling application processes, equivalent to the "dispatching room" of the railway bureau's transportation dispatch.

[0042] (4) Etcd: Based on the Go language, it uses the raft protocol as the consistency algorithm to achieve the availability and consistency of distributed system data.

[0043] A node (worker) is a machine that runs containerized applications. It can be a physical host or a virtual machine. A node is a worker node in a Kubernetes cluster, running Docker container applications. When a node fails, the Docker containers on it are automatically transferred by the master to other healthy nodes to continue running. Its core components are as follows:

[0044] (1) Kubelet: Communicates with the API server and is responsible for all content running on worker nodes. It first registers the node with the API server, then continuously monitors whether the API server has assigned a pod to the node, then starts the pod container, and finally continuously monitors the running containers and reports their status, events, and resource consumption to the API server.

[0045] (2) Kube-proxy: Communicates with the API server and ensures that clients can connect to the application pods. External clients connect to the kube-proxy through the external service IP and port to reach the application pods. If multiple pods correspond to a service, kube-proxy will perform load balancing.

[0046] The connection between kubu-proxy and the service IP and port will eventually reach a Pod supporting the service. If multiple Pods support a service, the proxy will play a load balancing role for the Pods.

[0047] Pod is the most basic unit of Kubernetes. Each Pod has a special Pause container, called the "root container." In addition to the Pause container, each Pod also contains one or more containers closely related to the business.

[0048] In 2006, NVIDIA released CUDA, a general-purpose parallel computing platform and programming model built on NVIDIA CPUs. CUDA-based programming can leverage the parallel computing engine of GPUs to more efficiently solve complex computational problems. However, the GPU is not a standalone computing platform; it works in conjunction with the CPU and can be considered a coprocessor. Therefore, when we talk about GPU parallel computing, we are actually referring to a heterogeneous computing architecture based on the CPU + GPU. In this heterogeneous computing architecture, the GPU and CPU are connected via the PCIe bus to work together. The CPU is called the host, while the GPU is called the device. Because GPUs have more computing cores, they are particularly well-suited for data-parallel, compute-intensive tasks, such as large matrix operations. CPUs, on the other hand, have fewer computing cores but can perform complex logical operations, making them suitable for control-intensive tasks. Furthermore, CPU threads are heavyweight, resulting in high context switching overhead, whereas GPU threads are lightweight due to their large number of cores. Therefore, heterogeneous computing platforms based on CPU+GPU can complement each other's strengths. The CPU is responsible for processing logically complex serial programs, while the GPU focuses on processing data-intensive parallel computing programs, thereby maximizing its effectiveness. CUDA is a GPU programming model developed by NVIDIA. It provides a simple interface for GPU programming. GPU computing-based applications can be built based on CUDA programming. CUDA provides support for other programming languages, such as C / C++, Python, Fortran and other languages. The CUDA programming model is a heterogeneous model that requires the CPU and GPU to work together. In CUDA, host refers to the CPU and its memory, while device refers to the GPU and its memory. The CUDA program contains both host programs and device programs, which run on the CPU and GPU respectively. At the same time, the host and device can communicate, so that data can be copied between them. The execution process of a typical CUDA program is as follows:

[0049] (1) Allocate host memory and initialize data;

[0050] (2) Allocate device memory and copy data from the host to the device;

[0051] (3) Call the CUDA kernel function to complete the specified operation on the device;

[0052] (4) Copy the calculation results on the device to the host;

[0053] (5) Release the memory allocated on the device and host.

[0054] The most important step in the above process is calling the CUDA kernel function to perform parallel computing. Kernel is an important concept in CUDA. A kernel is a function that is sent from the CPU to the GPU and executed in parallel in threads on the device.

[0055] As described above, this patent implements GPU time-sharing for CUDA. First, this patent implements a GPU device library, installed in each container. This library schedules GPU usage for tasks by intercepting all memory-related and compute-related APIs in the CUDA library. This patent uses tokens to implement time-sharing. A container can only execute its code on the GPU if it holds a valid token. This token is associated with a time quota. When the time quota exceeds the specified period, the container must reacquire the token before execution can begin. The time used by tasks in each container can be conveniently controlled by setting a time quota for its token.

[0056] In order to implement the above solution, this patent designs a scheduling system, which is divided into a front-end module (front-end) and a back-end module (back-end), such as Figure 2 As shown in the figure, the front-end module is a GPU device library within the container. It intercepts all memory- and compute-related CUDA library APIs through the Linux LD_PRELOAD mechanism. This forces applications in the container to load the patented GPU device library before executing standard GPU CUDA kernel functions. If a container does not have a token, its program execution is blocked until it obtains a valid token.

[0057] The backend module is a standalone daemon running on the host, responsible for managing tokens between containers. It tracks the GPU usage time of each container and determines the time quota for the next token, ensuring optimal token scheduling between containers. A container's GPU usage is calculated based on the time quota of valid tokens held within a sliding window. The backend module tracks this information from the timestamp when the token is assigned to the container. All token requests from the frontend module are queued for processing within the backend module. When a token expires, the backend module selects a request from the queue and issues the token to the container requesting it. How is a request selected from the queue? First, requests are filtered from containers whose GPU usage exceeds their maximum demand. Then, a request is selected from the container whose GPU usage is the largest difference from its minimum demand. It is important to note that the sum of the minimum demands of all pods sharing a GPU must be less than or equal to the GPU's computing capacity. If all containers requesting a token have met their minimum demand, the token is passed to the container with the lowest current GPU utilization.

[0058] It's important to note that this system protects against GPU memory over-allocation to a certain extent. Simply put, the total memory requirements of containers sharing the same GPU must be less than or equal to the total available memory. If a container attempts to request more space than it requested, the front-end module blocks its execution and only allocates a token when there's enough remaining memory to satisfy its needs.

[0059] Based on the above description, this patent proposes a cluster task scheduling method. The steps are as follows:

[0060] Step 1: Initialization: Use an appropriate number of nodes to build a GPU sharing cluster and install the backend module on the nodes.

[0061] Step 2: Submit the Pod. Users submit the Pod information to the cluster using the kubectl application using a YAML file. The image used in the Pod should include the patented GPU device library, i.e., the front-end module. Upon submission, the cluster will determine whether the format of the submitted content complies with cluster characteristics, such as Pod version information. If not, feedback will be provided to the user; if so, the process proceeds to the next step.

[0062] Step 3: After receiving the Pod information, the API Server stores it in Etcd. The client can query the Pod information using kubectl and prepare to schedule these Pods on nodes.

[0063] Step 4: After filtering eligible nodes based on the other scheduling resources required by the Pod, the default scheduler uses the scheduling plug-in in the cluster that manages and schedules GPU memory resources to plan the Pod's scheduling plan according to its rules and form a pre-allocation plan.

[0064] Step 5: Follow the pre-allocation plan and bind the Pod to be executed to the node through the scheduler. At this time, the binding method in the cluster is to update the Pod and node information cached in the cluster and write the binding information to Etcd. Finally, the node will verify whether the bound Pod can run on the node.

[0065] Step 6: If the Pod does not run successfully on the node, then re-enter step 4.

[0066] Step 7: Repeat the above steps (steps 1 to 6) until the list of tasks to be scheduled (i.e., Pods waiting to be scheduled) is empty.

[0067] Step 8: The backend module on the node is turned on, and the Pod running on the GPU node is scheduled using tokens based on the GPU usage of each container.

[0068] Step 9: Repeat the above steps (steps 2 to 8) until the Pod finishes running.

[0069] Based on the same inventive concept, another embodiment of the present invention provides a GPU time-sharing sharing system based on a Kubernetes cluster using the above method, including a front-end module and a back-end module set in the working node of the Kubernetes cluster; the front-end module is a GPU device library installed inside the container, and the GPU device library schedules the use of the GPU in tasks by intercepting all memory and computing-related APIs in the CUDA library; the back-end module manages tokens and time quotas of tokens between containers, and realizes GPU time-sharing through tokens and their time quotas.

[0070] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0071] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0072] The specific embodiments of the present invention disclosed above are intended to facilitate understanding and implementation of the present invention. Those skilled in the art will appreciate that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the embodiments disclosed in this specification; the scope of protection of the present invention shall be determined by the scope defined in the claims.

Claims

1. A GPU time-sharing sharing method based on Kubernetes cluster, characterized in that: The following steps are involved: Set up a front-end module in the worker node of the Kubernetes cluster. The front-end module is a GPU device library installed inside the container. The GPU device library schedules the use of the GPU in tasks by intercepting all memory and computing-related APIs in the CUDA library. A backend module is set up in the working nodes of the Kubernetes cluster. The backend module manages tokens and time quotas between containers, and implements time-sharing sharing of GPUs through tokens and time quotas. A container can execute its code on the GPU only if it holds a valid token; The token is associated with a time quota. When the time quota exceeds the specified period, the container must re-acquire the token to execute; The backend module tracks the GPU usage time of each container and determines the time quota of the next token to achieve reasonable scheduling of tokens between containers; the GPU usage rate of the container is calculated by the time quota of the valid token held within the time range of the sliding window.

2. The method according to claim 1, characterized in that All token requests from the front-end module are queued in the back-end module for processing; when the token itself expires and becomes invalid, the back-end module selects a request from the queue and issues the token to the container of the request.

3. The method according to claim 2, characterized in that The backend module selects a request from the queue, including: First, we filter requests from containers whose GPU usage has exceeded their maximum usage requirements, and then select a request from the container whose GPU usage is the farthest from its minimum usage requirements.

4. The method according to claim 2, characterized in that If all containers requesting tokens have reached the minimum usage requirement, the backend module transfers the token to the container with the lowest current GPU usage.

5. The method according to claim 1, wherein Use the following steps to schedule cluster tasks: Step 1: Build a GPU sharing cluster using an appropriate number of nodes and install the backend module on the nodes. Step 2: The user submits Pod information to the cluster using the kubectl application using a YAML file. The image used in the Pod should contain the GPU device library, i.e., the front-end module. During submission, the cluster determines whether the format of the submitted content meets the cluster characteristics. If not, the user is given feedback. If it does, the cluster proceeds to the next step. Step 3: After receiving the Pod information, the API Server stores it in Etcd and prepares to schedule the nodes for these Pods. Step 4: After filtering eligible nodes based on the other scheduling resources required by the pod, the scheduler uses the scheduling plug-in in the cluster that manages and schedules GPU memory resources to plan the pod's scheduling plan according to its rules and form a pre-allocation plan. Step 5: Execute according to the pre-allocation plan, bind the Pod to be executed to the node through the scheduler, and write the binding information to Etcd; Step 6: If the Pod does not run successfully on the node, re-enter step 4; Step 7: Repeat steps 1 to 6 until the list of tasks to be scheduled is empty. Step 8: The backend module on the node is started, and the Pod running on the GPU node is scheduled using tokens based on the GPU usage of each container; Step 9: Repeat steps 2 to 8 until the Pod finishes running.

6. A GPU time-sharing sharing system based on a Kubernetes cluster using the method according to any one of claims 1 to 5, characterized in that: It includes a front-end module and a back-end module set in the working node of the Kubernetes cluster; the front-end module is a GPU device library installed inside the container, which schedules the use of GPU in tasks by intercepting all memory and computing-related APIs in the CUDA library; the back-end module manages tokens and token time quotas between containers, and realizes time-sharing sharing of GPUs through tokens and their time quotas.

7. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 5 is implemented.

Images