GPU hot plug method based on Kubernetes architecture and readable storage medium

Abstract

The invention discloses a GPU (Graphics Processing Unit) hot plug method based on a Kubernetes architecture and a readable storage medium. The method comprises the following steps: a user selects to start a GPU hot plug option when a service is created, NVIDIA GPU resources are changed into self-defined GPU resources adaptive to GPU hot plug, and a condition that the service capacity is reduced to 0 is configured; when the GPU HotPlug Controller detects the newly built Pod using the hot plug, the GPU equipment is dynamically inserted into the Pod; when it is detected that the service Pod has no request and the duration time reaches a GPU release condition configured by a user, executing GPU dynamic release operation; when it is detected that the service Pod has a request, the operation of dynamic allocation of the GPU is executed immediately, and the Knative initiates a GPU hot plug request to the GPU HotPlug Controller; and after the GPU hot plug succeeds, the Knative sends the originally accumulated request to the service Pod, and at the same time, the route is updated: the subsequent flow is directly transmitted to the Pod without being forwarded by the Knative. According to the invention, the problem of resource waste caused by incapability of hot plugging of the GPU in the prior art can be solved.

Description

Technical Field

[0001] The present invention belongs to the field of cloud native, and in particular relates to a GPU hot-plug method based on Kubernetes architecture and a readable storage medium. Background Art

[0002] Kubernetes (often shortened to K8s) is an open source container orchestration platform that allows users to automate the deployment, scaling, and management of containerized applications.

[0003] When using GPU resources in a K8s environment, in order to avoid wasting GPU resources and improve the execution efficiency of GPU tasks, many methods for sharing GPU resources have been studied in the prior art. For example, Chinese patent document with publication number CN114675976A discloses a GPU sharing method based on kubernetes, and Chinese patent document with publication number CN114706690A discloses a Kubernetes container GPU sharing method and system.

[0004] However, in addition to the above problems, there are other challenges when using GPU resources in a K8s environment, especially the hot-plugging problem of GPU devices. The main reasons why GPUs cannot be hot-plugged include the following aspects.

[0005] Hardware limitations: Most server-class GPUs are not designed to support hot-swap. Physically inserting or removing a running GPU may cause system instability or even hardware damage. Therefore, in most cases, adding or removing a GPU requires downtime.

[0006] Driver and kernel integration: When a GPU is installed in a server and booted for the first time, the operating system loads the appropriate drivers to manage the hardware. If you try to remove a GPU without rebooting, you may encounter driver errors or more serious kernel crashes. Similarly, a newly added GPU also requires a reboot so that the operating system can correctly identify and initialize it.

[0007] Limitations of the Kubernetes scheduler: Kubernetes is currently designed to not support dynamic perception of physical hardware changes, including dedicated computing resources such as GPUs. This means that once the cluster is configured, changing the number of available GPUs on a node usually requires reconfiguring the entire cluster, which often involves stopping related services and updating node information. Summary of the invention

[0008] The present invention provides a GPU hot-swap method based on Kubernetes architecture and a computer-readable storage medium, aiming to solve the problem of resource waste caused by the inability to hot-swap GPUs in the prior art. A GPU hot-swap method based on Kubernetes architecture comprises the following steps: When creating a service, the user chooses to enable the GPU hot-plug option, replaces NVIDIA's GPU resources with custom GPU resources that are compatible with GPU hot-plug, and configures the conditions for scaling the service down to 0. When the GPU HotPlug Controller detects a newly created Pod that uses hot plugging, it dynamically inserts the GPU device into the Pod; When it is detected that the business Pod has no request and the duration reaches the GPU release condition configured by the user, the GPU dynamic release operation is performed; the Knative component modifies the route, and all business Pod requests are intercepted by Knative; Knative initiates a GPU removal request to the GPUHotPlug Controller; When a request is detected from the business Pod, the GPU dynamic allocation operation is immediately executed, and Knative initiates a GPU hot plug request to the GPUHotPlug Controller. After the GPU hot plug is successful, Knative sends the previously accumulated requests to the business Pod and updates the route at the same time: subsequent traffic goes directly to the Pod without being forwarded by Knative.

[0009] Furthermore, Knative initiates a GPU removal request to the GPU HotPlug Controller, which includes: The GPU HotPlug Controller requests the API Server on Kubernetes to query the node where the business Pod is located and the GPU device information corresponding to the Pod; The GPU HotPlug Controller requests the GPU Mounter on the node to uninstall the GPU device; the GPU Mounter performs the release operation of the GPU device; The GPU HotPlug Controller requests the Kubernetes API Server to delete the Slave Pod, reallocate the GPU device, and hot-plug it into other business Pods.

[0010] Furthermore, the GPU Mounter performs the release operation of the GPU device, specifically: Determine whether there is a process using the GPU device, send a sigterm signal, and the business process releases the GPU device; when it is detected that no process is using the GPU, modify the device cgroup and mountnamespace configurations to release the GPU device.

[0011] Furthermore, Knative initiates a GPU hot-plug request to the GPU HotPlug Controller, specifically: GPU HotPlug Controller requests the API Server to query the node where the business Pod is located; The GPU HotPlug Controller requests the Kubernetes API Server to create a Slave Pod corresponding to the business Pod. After the Slave Pod is successfully started, the Controller requests the GPU Mounter on the corresponding node to hot-plug the GPU device corresponding to the Slave Pod into the business Pod.

[0012] Furthermore, after the Slave Pod is successfully started, the Controller also obtains the device ID of the GPU, modifies the devices cgroup corresponding to the business Pod, allows the business process to access the corresponding gpu device, and mounts the GPU device to the mount namespace where the business Pod is located.

[0013] In the present invention, by customizing the Device Plugin method, the number of requests that can be initiated by the Pod is dynamically increased or decreased according to the actual number of available GPUs, thereby controlling the scope of the request.

[0014] The Device plugin is implemented and finally runs in the Kubernetes cluster as a pod and is deployed as a DaemonSet. After the device plugin is started, the first step is to register with kubelet to let kubelet know that a new device is connected. In order to call the Register interface of kubelet, the device plugin pod mounts the kubelet.sock file on the host into the container. After initiating a call through the kubelet.sock file to register the cluster deployment, kubelet is started.

[0015] Based on the same inventive principle, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned GPU hot-plugging method is implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. Improved resource utilization: By supporting GPU hot plugging, the present invention can flexibly adjust GPU resource allocation according to actual needs, avoiding the waste problem caused by fixed configuration.

[0017] 2. Enhanced system flexibility and response speed: Combined with Knative's traffic management and intelligent scheduling mechanism, it can quickly respond to changing workloads in a short period of time, improving the agility and efficiency of the overall system.

[0018] 3. Simplified operation and maintenance: Automated management of GPU resources reduces the need for manual intervention, reduces the complexity of operation and maintenance, and also reduces the risks caused by human operational errors.

[0019] 4. Promoted the development of cloud-native applications: It provided strong support for building more efficient, reliable and scalable applications, and promoted the advancement and development of cloud computing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a GPU hot-swap method based on Kubernetes architecture according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0022] like Figure 1 As shown, a GPU hot-swap method based on Kubernetes architecture includes the following steps: Step 1. Create a service, enable the hot-swap option, replace nvidia.com / gpu with custom.com / gpu that is suitable for GPU hot-swap in the background, and configure the conditions for shrinking the service to 0.

[0023] Step 2: GPU HotPlug Controller detects the newly created Pod that uses hot plugging and dynamically inserts the GPU device into the Pod. When it is detected that the business Pod has no request and it lasts for a period of time, and the GPU release condition configured by the user is met, the GPU dynamic release operation is performed. The Knative component modifies the route, and all requests from the business Pod are intercepted by Knative. Knative initiates a request to the GPU HotPlug Controller to remove the GPU. The GPU HotPlug Controller requests the API Server on Kubernetes to query the GPU device information corresponding to the node where the business Pod is located and the Pod. The GPUHotPlug Controller requests the GPU Mounter on the node to uninstall the GPU device. The GPU Mounter performs the release operation of the GPU device: determines whether there is a process using the GPU device, sends a sigterm signal, and the business process releases the GPU device. When it is detected that no process is using the GPU, the device cgroup and mountnamespace configurations are modified to release the GPU device. The GPU HotPlug Controller requests the Kubernetes APIServer to delete the Slave Pod, and the GPU device can be reallocated and hot-plugged into other business Pods.

[0024] Step 3. When a request is detected from the business Pod, the GPU dynamic allocation operation is performed immediately. Knative initiates a GPU hot-plug request to the GPUHotPlug Controller. The GPU HotPlug Controller requests the API Server to query the node where the business Pod is located. The GPU HotPlug Controller requests the Kubernetes API Server to create a Slave Pod corresponding to the business Pod. After the Slave Pod is successfully started, the Controller requests the GPUMounter on the corresponding node to hot-plug the GPU device corresponding to the Slave Pod into the business Pod. At the same time, obtain the device ID of the GPU, modify the devices cgroup corresponding to the business Pod, and allow the business process to access the corresponding gpu device. Mount the GPU device to the mount namespace where the business Pod is located. After the GPU hot-plug is successful, Knative sends the previously accumulated requests to the business Pod and updates the route at the same time: subsequent traffic goes directly to the Pod without being forwarded by Knative.

[0025] As the business load changes, when a new request arrives at the previously suspended service, a new round of GPU hot plugging is immediately triggered. This process follows the steps mentioned above to quickly allocate the required GPU resources to the recovering service to ensure that service continuity is not affected.

[0026] Monitor business status, perform capacity expansion and lock operations, and initiate hot swap processes. During cold start, when the number of replicas increases from 0 to 1, intercept business requests and wait for the Pod's GPU to be ready before forwarding the request to the business Pod. Knative is built on the Kubernetes and Istio platforms, using container management components (deployment, replicaset, and pod, etc.) provided by Kubernetes, and network management components (ingress, LB, dynamic route, etc.) provided by Istio. There are two important components in Knative, namely the Serving component that provides traffic for it and the Event component that ensures that applications can easily produce and consume events. Among them, the Serving component automatically scales based on load, including reducing to zero when there is no load, allowing users to create traffic policies for multiple revisions of applications, so that they can easily route to the target application through the URL; and the role of the Event component is to make it easy to produce and consume events, allowing operators to use the messaging layer of their choice. In addition to the Serving and Event components, Build is also one of the components of Kantive. It provides a "run to completion" display function, which is very useful for creating CI / CD workflows. It builds user source code into containers through a flexible plug-in build system. At present, it already supports multiple build systems, such as Google's Kaniko, which can build container images on Kubernetes clusters without running Docker Daemon. Serving uses it to convert source repositories into container images containing applications. Among many Serverless open source projects, Knative's advantages are also more obvious. On the one hand, Knative uses Kubernetes as the underlying framework and is more closely integrated with the Kubernetes ecosystem. Whether it is a cloud Kubernetes service or a self-built Kubernetes cluster, you can quickly build a Serverless platform by installing the Knative plug-in. On the other hand, Knative, in conjunction with CNCF, standardizes all events as CloudEvent, provides cross-platform operation of events, and decouples functions from specific calling methods. At the elasticity level, Knative can monitor application requests and automatically scale up and down. With the help of Istio (Ambassador, Gloo, etc.), it supports blue-green release and rollback functions to facilitate application release. At the same time, Knative supports log collection, search, and analysis, as well as VAmetrics data display and call relationship tracking.

[0027] Generally, all device plugin implementations will eventually run in the k8s cluster as pods. Because all nodes need to be managed, they are all deployed in DaemonSet mode. After the device plugin is started, the first step is to register with kubelet to let kubelet know that a new device is connected. In order to call the Register interface of kubelet, the device plugin pod will mount the kubelet.sock file (unix sock) on the host into the container and initiate a call through the kubelet.sock file to achieve registration. The difference between the custom Device Plugin and the NVIDIA Device Plugin is that it dynamically increases and decreases the number of requests that can be initiated by the Pod according to the actual number of available GPUs, thereby controlling the scope of the request.

[0028] The custom Device plugin part in the embodiment of the present invention may include the following steps: Generally, all device plugin implementations will eventually run in the k8s cluster in the form of pods. Because all nodes need to be managed, they will be deployed in DaemonSet mode. After the device plugin is started, the first step is to register with kubelet to let kubelet know that a new device is connected. In order to call the Register interface of kubelet, the device plugin pod will mount the kubelet.sock file (unix sock) on the host into the container. After initiating a call through the kubelet.sock file to register the cluster deployment, kubelet will start. The steps are as follows: 1. Kubelet starts the Registration gRPC service (kubelet.sock) and provides the Register interface.

[0029] 2. After device-plugin is started, the Register interface is called through kubelet.sock to register with kubelet. The registration information includes the Unix socket, API Version, and ResourceName of the device plugin.

[0030] 3. After successful registration, kubelet calls ListtAndWatch to the device plugin through the unix socket of device-plugin to obtain the resources on the current node.

[0031] 4. Kubelet updates the node status to the api-server to record the resources found in the previous step.

[0032] 5. The user creates a pod and applies for the resource. After scheduling is completed, the kubelet on the corresponding node calls the Allocate interface of deviceplugin to allocate resources.

[0033] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A GPU hot-swap method based on Kubernetes architecture, characterized in that: The following steps are involved: When creating a service, the user chooses to enable the GPU hot-plug option, replaces NVIDIA's GPU resources with custom GPU resources that are compatible with GPU hot-plug, and configures the conditions for scaling the service down to 0. When the GPU HotPlug Controller detects a newly created Pod that uses hot plugging, it dynamically inserts the GPU device into the Pod; When it is detected that the business Pod has no request and the duration reaches the GPU release condition configured by the user, the GPU dynamic release operation is performed; the Knative component modifies the route, and all business Pod requests are intercepted by Knative; Knative initiates a GPU removal request to the GPUHotPlug Controller; When a request is detected from the business Pod, the GPU dynamic allocation operation is immediately executed, and Knative initiates a GPU hot plug request to the GPU HotPlugController. After the GPU hot plug is successful, Knative sends the previously accumulated requests to the business Pod and updates the route at the same time: subsequent traffic goes directly to the Pod without being forwarded by Knative.

2. The GPU hot-swap method based on Kubernetes architecture according to claim 1, characterized in that: Knative initiates a request to the GPU HotPlug Controller to remove the GPU, including: The GPU HotPlug Controller requests the API Server on Kubernetes to query the node where the business Pod is located and the GPU device information corresponding to the Pod; The GPU HotPlug Controller requests the GPU Mounter on the node to uninstall the GPU device; the GPU Mounter performs the release operation of the GPU device; The GPU HotPlug Controller requests the Kubernetes API Server to delete the Slave Pod, reallocate the GPU device, and hot-plug it into other business Pods.

3. The GPU hot-swap method based on Kubernetes architecture according to claim 2, characterized in that: GPUMounter performs the release operation of the GPU device, specifically: Determine whether there is a process using the GPU device, send a sigterm signal, and the business process releases the GPU device; when it is detected that no process is using the GPU, modify the device cgroup and mountnamespace configurations to release the GPU device.

4. The GPU hot-swap method based on Kubernetes architecture according to claim 1, characterized in that: Knative initiates a GPU hot-plug request to the GPU HotPlug Controller. Specifically: GPU HotPlug Controller requests the API Server to query the node where the business Pod is located; The GPU HotPlug Controller requests the Kubernetes API Server to create a SlavePod corresponding to the business Pod. After the Slave Pod is successfully started, the Controller requests the GPU Mounter on the corresponding node to hot-plug the GPU device corresponding to the Slave Pod into the business Pod.

5. The GPU hot-swap method based on Kubernetes architecture according to claim 4, characterized in that: After the SlavePod is successfully started, the Controller also obtains the GPU device ID, modifies the devices cgroup corresponding to the business Pod, allows the business process to access the corresponding gpu device, and mounts the GPU device to the mount namespace where the business Pod is located.

6. The GPU hot-swap method based on Kubernetes architecture according to claim 1, characterized in that: By customizing the Device Plugin method, the number of requests that a Pod can initiate can be dynamically increased or decreased based on the actual number of available GPUs, thereby controlling the scope of the requests.

7. The GPU hot-swap method based on Kubernetes architecture according to claim 6, characterized in that: The Deviceplugin implementation is finally run in the Kubernetes cluster in the form of a pod and deployed in DaemonSet mode; After the device plugin is started, the first step is to register with kubelet to let kubelet know that a new device is connected. In order to call the Register interface of kubelet, the device plugin pod mounts the kubelet.sock file on the host into the container. After initiating a call through the kubelet.sock file to register the cluster deployment, kubelet is started.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the GPU hot-plug method according to any one of claims 1 to 7 is implemented.

Images