Fine-grained monitoring method and system of shared GPU container based on k8s

Abstract

The invention relates to a k8s-based fine-grained monitoring method and system for a shared GPU container, and belongs to the technical field of computers.The method comprises the steps that configuration information of a monitoring module is set and transmitted into a service container in an environment variable form; the method comprises the following steps: deploying a monitoring module on each k8s cluster machine to obtain GPU process information; acquiring local process information of a machine by using a monitoring module to obtain a system-level index; obtaining a k8s container UID through a process identifier PID in the GPU process information, associating the k8s container UID to a k8s container name, calculating occupation information of a plurality of GPU containers, and obtaining a process-level index; and exposing the system-level index and the process-level index to a prometheus system alarm tool and carrying out verification on the system-level index and the process-level index. According to the method and the system provided by the invention, the granularity of GPU resource collection can be improved through two modes of obtaining the GPU resource information in the machine and obtaining the GPU information in the container.

Description

Technical Field

[0001] This invention belongs to the fields of communication technology and computer technology, and particularly relates to a fine-grained monitoring method and system for shared GPU containers based on k8s. Background Technology

[0002] Kubernetes (k8s) is an open-source container orchestration platform used to automate the deployment, scaling, management and operation of containers, enabling users to efficiently manage large-scale containerized applications.

[0003] Traditional dedicated GPU usage leads to wasted computing resources in scenarios such as model prediction. Shared GPUs can solve this problem to some extent.

[0004] Shared GPU mode refers to a Kubernetes cluster where multiple containers can share the same GPU card by tagging nodes with GPU devices and installing relevant components. For example, by modifying the application's YAML file and adjusting the replica count and resource dimensions, multiple Pods can run on the same GPU device, with each Pod requesting GPU memory on demand.

[0005] However, the shared GPU mode requires effective monitoring to ensure the reasonable allocation and use of resources, and to prevent some tasks from excessively consuming resources and affecting other tasks, so as to achieve the optimal balance between cost and performance.

[0006] Most current GPU monitoring only focuses on system-level monitoring of the entire GPU. In shared GPU mode, it is necessary to monitor the GPU usage ratio of different containers to ensure that each task can obtain the GPU resources it needs, avoid performance degradation or task failure caused by resource contention, and ensure business stability and service quality.

[0007] Specifically, most current open-source GPU monitoring software only monitors system data for the entire GPU card. However, in a Kubernetes environment where multiple containers share the same GPU card, the system monitoring data for the entire GPU card cannot reflect the usage of the GPU card by each individual pod container. For example, situations where a pod task consumes excessive resources or has unreasonable resource allocation require more granular monitoring data to reflect this.

[0008] Currently, most GPU monitoring software in Kubernetes clusters can only obtain coarse-grained metrics such as GPU utilization and GPU memory, and cannot obtain fine-grained metrics used in business applications. Summary of the Invention

[0009] In view of the shortcomings of the prior art, the purpose of this invention is to provide a fine-grained monitoring method and system for shared GPU containers based on Kubernetes, which improves the granularity of GPU resource acquisition by obtaining GPU resource information in both the machine and the container.

[0010] A first aspect of the present invention proposes a fine-grained monitoring method for shared GPU containers based on Kubernetes, comprising:

[0011] S1, configure the monitoring module's configuration information and pass it to the service container via environment variables; deploy the daemonset, service, and serviceMonitor services to the Kubernetes cluster;

[0012] S2, the monitoring module is deployed on each k8s cluster machine in the form of daemonset, and the monitoring module is deployed on each k8s cluster machine to detect whether there is a GPU container online. If so, the monitoring module is deployed to detect whether the GPU container is in normal status. When the GPU container is in normal status, the corresponding GPU process information is obtained.

[0013] S3. Use the monitoring module to obtain local machine process information and obtain system-level metrics; obtain the k8s container UID through the process identifier PID in the GPU process information, associate it with the k8s container name, calculate the usage information of multiple GPU containers, and obtain process-level metrics.

[0014] S4. The system-level and process-level metrics are exposed to the Prometheus system alerting tool using the data reporting module. The Prometheus system alerting tool is then used to verify the system-level and process-level metrics, and the verification results are sent to the user.

[0015] Furthermore, in S1, the configuration information includes: k8s cluster authentication information, collected metrics, cgroup file configuration information, and proc file configuration information.

[0016] Furthermore, in S2, the GPU process information includes: process-level video memory usage, the number of SM cores used by the process in real time, and the bandwidth used by the process.

[0017] Furthermore, in S2, the daemonset service is mounted to the GPU on the k8s cluster machine, the GPU driver is initialized, and the GPU process information is obtained. Here, a GPU process can support mounting one or more GPUs.

[0018] Furthermore, in S3, the daemonset service is mounted to the / proc folder on the Kubernetes cluster machine. By reading the / proc / pid / status file, it is determined whether the GPU process is associated with the Kubernetes container UID. If they are associated, the GPU process is exposed with metrics.

[0019] Furthermore, following S4, the following steps are also included:

[0020] S51, Start the business container and the sidecar container, wherein the sidecar container is located in the user pod; while using the GPU, the business container obtains fine-grained GPU utilization metrics.

[0021] S52, the business container writes the fine-grained GPU utilization rate indicator into the data log, the sidecar container reads the data log, and sends the fine-grained GPU utilization rate indicator back to the data reporting module via webhook.

[0022] S53, the data reporting module performs anomaly detection on the fine-grained GPU utilization index. If no anomaly is found, the fine-grained GPU utilization index is exposed to the Prometheus system alarm tool.

[0023] Furthermore, the fine-grained GPU utilization metrics include: operator time spent on CPU, operator time spent on GPU, operator memory and video memory usage, and operator floating-point operations.

[0024] A second aspect of the present invention proposes a fine-grained monitoring system for shared GPU containers based on Kubernetes, comprising: a monitoring module, a Kubernetes client, and a data reporting module, wherein,

[0025] The monitoring module is deployed on each Kubernetes cluster machine in the form of daemonset. It is used to detect whether there are GPU containers online on each Kubernetes cluster machine. If so, it further checks whether the GPU containers are in normal status. When the GPU containers are in normal status, it obtains the corresponding GPU process information; it obtains the machine's local process information to get system-level metrics; it obtains the Kubernetes container UID through the process identifier PID in the GPU process information and associates it with the Kubernetes container name to calculate the usage information of multiple GPU containers and obtain process-level metrics.

[0026] The Kubernetes client is used to read the Kubernetes namespace and container information of the Kubernetes cluster.

[0027] The data reporting module is used to integrate system-level and process-level metrics from the monitoring module and k8s namespace and container information from the k8s client, expose the system-level and process-level metrics to the Prometheus system alerting tool, use the Prometheus system alerting tool to verify the system-level and process-level metrics, and send the verification results to the user.

[0028] Furthermore, the monitoring module obtains the machine's local process information by reading the / proc / pid / mountinfo file of each Kubernetes cluster, and obtains the GPU process information of each Kubernetes cluster by accessing the GPU container of each Kubernetes cluster.

[0029] Furthermore, it also includes: business containers and sidecar containers, among which,

[0030] The business container is used to obtain fine-grained GPU utilization metrics while using the GPU, and write the fine-grained GPU utilization metrics into the data log.

[0031] The sidecar container is located in the user's pod and is used to read the data logs. It also uses a webhook to send the fine-grained GPU utilization metric back to the data reporting module. The data reporting module performs anomaly detection on the fine-grained GPU utilization metric. If no anomaly is found, the fine-grained GPU utilization metric is exposed to the Prometheus system alerting tool.

[0032] The beneficial effects of this invention are as follows:

[0033] The method and system described in this invention combine the high availability and ease of deployment of Kubernetes, making it convenient to deploy various modules in a cluster of machines. Furthermore, it adopts a scheme that allows optional services to proactively expose indicators, thereby improving the granularity of resource monitoring.

[0034] 1. This invention improves the granularity of GPU resource acquisition by using two methods: acquiring GPU resource information in the machine and acquiring GPU information in the container.

[0035] 2. This invention improves data reliability by collecting process-level and system-level metrics and verifying both types of metrics. It can also identify problems such as video memory overflow, GPU core utilization, video memory usage, and bandwidth usage.

[0036] 3. In addition to passively acquiring GPU resource information, this invention also provides a method for actively pushing GPU resources in a sidecar mode to expose GPU metrics, achieving more granular GPU resource data monitoring. The utilization rate of each operator is statistically analyzed in the business and reported through a sidecar container, improving the reliability and convenience of data processing.

[0037] 4. In the shared GPU mode, this invention monitors the GPU usage ratio of different containers to ensure that each task can obtain the GPU resources it needs, avoiding performance degradation or task failure caused by resource contention, and ensuring business stability and service quality. Attached Figure Description

[0038] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0039] Figure 1 This is a flowchart of a fine-grained monitoring method for shared GPU containers based on Kubernetes, according to an embodiment of the present invention.

[0040] Figure 2 This is a flowchart illustrating the acquisition of GPU information according to an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram illustrating GPU process resource usage in various scenarios according to embodiments of the present invention;

[0042] Figure 4 This is a schematic diagram illustrating data reporting via a sidecar container according to an embodiment of the present invention;

[0043] Figure 5 This is a structural diagram of a fine-grained monitoring system for a shared GPU container based on Kubernetes, according to an embodiment of the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0045] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0046] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0048] like Figure 1 As shown, this invention proposes a fine-grained monitoring method for shared GPU containers based on Kubernetes, including:

[0049] S1 sets the configuration information for the monitoring module, passing it to the service container as an environment variable. Users then deploy the daemonset, service, and serviceMonitor services to the Kubernetes cluster.

[0050] In this invention, the configuration information includes: Kubernetes cluster authentication information, collected metrics, cgroup file configuration information, and proc file configuration information. The collected metrics include: GPU memory usage, GPU computing power utilization, and GPU frequency.

[0051] After completing the above configuration, start the monitoring module to periodically monitor changes in GPU processes and container information.

[0052] S2 deploys the monitoring module on each Kubernetes cluster machine via daemonset to detect whether there are GPU containers online on each Kubernetes cluster machine. If so, it further checks whether the GPU containers are in normal status. When the GPU containers are in normal status, it obtains the corresponding GPU process information.

[0053] refer to Figure 2 As shown, the process first checks if any GPU containers are online on the Kubernetes cluster machines. If they are online, it further checks if the GPU containers are in a normal state. If the GPU containers are offline, the process ends. If the GPU containers are in a normal state, the GPU process information is obtained; if the GPU containers are in an abnormal state, no metrics are exposed.

[0054] In this invention, the method for determining whether the GPU container is in normal condition is as follows: the GPU container’s container health check interface checks its own status and then reports it to the k8s cluster.

[0055] Specifically, the daemonset service is mounted to the GPU on the Kubernetes cluster machine, the GPU driver is initialized, and GPU process information is obtained.

[0056] In this invention, the monitoring module obtains the GPU process information of each Kubernetes cluster by accessing the GPU container of each Kubernetes cluster.

[0057] In this invention, a GPU process can support mounting one or more GPU containers.

[0058] refer to Figure 3 As shown, GPU processes and GPU containers include the following types:

[0059] (1) A GPU process can mount multiple GPU containers

[0060] For example, process 1 can simultaneously mount GPU0, GPU1, and GPU2.

[0061] (2) Multiple GPU processes are mounted on the same GPU container

[0062] For example, process 1 and process 2 can both mount GPU0.

[0063] (3) A GPU process mounts a GPU container.

[0064] For example, process 2 only mounts one GPU0.

[0065] In this invention, GPU process information includes: process-level video memory usage, the number of SM cores used by the process in real time, and process bandwidth usage information, etc.

[0066] In addition, the monitoring module reads cgroup files to obtain container-related information.

[0067] S3 uses the monitoring module to obtain local process information of the machine and obtain system-level metrics; it obtains the k8s container UID through the process identifier PID in the GPU process information, associates it with the k8s container name, calculates the usage information of multiple GPU containers, and obtains process-level metrics.

[0068] The monitoring module obtains local process information (i.e., local process information of the machine) by reading the / proc / pid / mountinfo file of each Kubernetes cluster.

[0069] Specifically, the daemonset service is mounted to the / proc folder on the Kubernetes cluster machine. The / proc / pid / status file is read to determine whether the GPU process is associated with the Kubernetes container UID. If they are associated, the GPU process is added to the data reporting module for metric exposure.

[0070] Specifically, first, the PID of the target GPU process is obtained using NVIDIA tools; then, the pod ID and container ID of the Kubernetes container are obtained using the PID; finally, the containers and GPU processes of the Kubernetes cluster are associated, and metrics are exposed through the sidecar container.

[0071] S4 uses the data reporting module to expose system-level and process-level metrics to the Prometheus system alerting tool, uses the Prometheus system alerting tool to verify the system-level and process-level metrics, and sends the verification results to the user.

[0072] In this invention, the verification results are communicated to the user via email and SMS.

[0073] The following is an optional solution: use a sidecar container in the user's pod to collect GPU metrics directly within the container and send them back to the data reporting module via a webhook. The data reporting module then verifies the data and reports it to the Prometheus system alerting tool.

[0074] refer to Figure 4 Following S4, the following steps are also included:

[0075] S51, start the business container and the sidecar container. The sidecar container is located in the user's pod.

[0076] While the business container is using the GPU, it obtains fine-grained GPU utilization metrics.

[0077] In this invention, fine-grained GPU utilization metrics include: operator time spent on CPU, operator time spent on GPU, operator memory and video memory usage, and operator floating-point operations, etc.

[0078] In S52, the business container writes fine-grained GPU utilization metrics to the data log, the sidecar container reads the data log, and sends the fine-grained GPU utilization metrics back to the data reporting module via a webhook.

[0079] In this invention, metric data is recorded through business code and then sent to the sidecar container.

[0080] S53, the data reporting module performs anomaly detection on the fine-grained GPU utilization metric. If no anomalies are found, the fine-grained GPU utilization metric is exposed to the Prometheus system alarm tool.

[0081] like Figure 5 As shown, the present invention also proposes a fine-grained monitoring system for shared GPU containers based on k8s, including: a monitoring module 100, a k8s client 200, and a data reporting module 300.

[0082] Specifically, the monitoring module 100 is deployed on each Kubernetes cluster machine in the form of a daemonset to detect whether there are online GPU containers on each Kubernetes cluster machine. If so, it further checks whether the GPU containers are in normal status. When the GPU containers are in normal status, it obtains the corresponding GPU process information and obtains local machine process information to get system-level metrics.

[0083] The process identifier (PID) in the GPU process information is used to obtain the Kubernetes container UID, which is then associated with the Kubernetes container name. This allows for the calculation of the occupancy information of multiple GPU containers, resulting in process-level metrics.

[0084] It should be noted that before starting the monitoring module 100, configuration information needs to be set and passed to the service via environment variables. This configuration information includes: Kubernetes cluster authentication information, collected metrics, cgroup file configuration information, and proc file configuration information. The collected metrics include: GPU memory usage, GPU computing power utilization, and GPU frequency. Users deploy the daemonset, service, and serviceMonitor services to the Kubernetes cluster.

[0085] In this invention, the monitoring module 100 obtains local machine process information by reading the / proc / pid / mountinfo file of each Kubernetes cluster, and obtains GPU process information of the Kubernetes cluster by accessing the GPU container of each Kubernetes cluster.

[0086] The k8s client 200 is used to read the k8s namespace and container information of the k8s cluster.

[0087] The data reporting module 300 integrates system-level and process-level metrics from the monitoring module 100 and k8s namespace and container information from the k8s client 200. It exposes the system-level and process-level metrics to the Prometheus system alerting tool, verifies the system-level and process-level metrics using the Prometheus system alerting tool, and sends the verification results to the user.

[0088] The specific role of Kubernetes namespace and container information in data integration and verification: Synchronizing namespace and container information.

[0089] In this invention, the data reporting module 300 integrates the data information obtained by the monitoring module 100 and the k8s client 200, and provides an interface to report the data to the database.

[0090] In addition, the fine-grained monitoring system for shared GPU containers based on Kubernetes of the present invention also includes: business containers and sidecar containers.

[0091] Specifically, the business container is used to obtain fine-grained GPU utilization metrics while using GPUs, and write the fine-grained GPU utilization metrics to the data log.

[0092] The sidecar container resides in the user's pod and is used to read data logs. It then uses a webhook to send fine-grained GPU utilization metrics back to the data reporting module. The data reporting module performs anomaly detection on the fine-grained GPU utilization metrics, and if no anomalies are found, it exposes the fine-grained GPU utilization metrics to the Prometheus system alerting tool.

[0093] The fine-grained monitoring system and method for shared GPU containers based on Kubernetes of the present invention can be applied to the following scenarios: monitoring the resource usage ratio of multiple containers sharing a single GPU, and services requiring fine-grained collection of GPU resource metrics. This invention can improve resource control of shared GPU services, increase resource utilization for services, and fill the gap in fine-grained monitoring of shared GPUs.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A fine-grained monitoring method for shared GPU containers based on Kubernetes, characterized in that, include: S1, configure the monitoring module's configuration information and pass it to the service container via environment variables; deploy the daemonset, service, and serviceMonitor services to the Kubernetes cluster; S2, the monitoring module is deployed on each k8s cluster machine in the form of daemonset, and the monitoring module is deployed on each k8s cluster machine to detect whether there is a GPU container online. If so, the monitoring module is deployed to detect whether the GPU container is in normal status. When the GPU container is in normal status, the corresponding GPU process information is obtained. S3. Use the monitoring module to obtain local machine process information and obtain system-level metrics; obtain the k8s container UID through the process identifier PID in the GPU process information, associate it with the k8s container name, calculate the usage information of multiple GPU containers, and obtain process-level metrics. S4. The system-level and process-level metrics are exposed to the Prometheus system alerting tool using the data reporting module. The Prometheus system alerting tool is then used to verify the system-level and process-level metrics, and the verification results are sent to the user.

2. The fine-grained monitoring method for shared GPU containers based on Kubernetes according to claim 1, characterized in that, In S1, the configuration information includes: k8s cluster authentication information, collected metrics, cgroup file configuration information, and proc file configuration information.

3. The fine-grained monitoring method for shared GPU containers based on Kubernetes according to claim 1, characterized in that, In S2, the GPU process information includes: process-level video memory usage, the number of SM cores used by the process in real time, and the bandwidth used by the process.

4. The fine-grained monitoring method for shared GPU containers based on Kubernetes according to claim 1, characterized in that, In S2, the daemonset service is mounted to the GPU on the k8s cluster machine, the GPU driver is initialized, and the GPU process information is obtained. One GPU process can support mounting one or more GPUs.

5. A fine-grained monitoring method for shared GPU containers based on Kubernetes according to claim 1, characterized in that, In S3, the daemonset service is mounted to the / proc folder on the Kubernetes cluster machine. The / proc / pid / status file is read to determine whether the GPU process is associated with the Kubernetes container UID. If they are associated, the GPU process is exposed with metrics.

6. A fine-grained monitoring method for shared GPU containers based on Kubernetes according to claim 1, characterized in that, Following S4, the following steps are also included: S51, Start the business container and the sidecar container, wherein the sidecar container is located in the user pod; while using the GPU, the business container obtains fine-grained GPU utilization metrics. S52, the business container writes the fine-grained GPU utilization rate indicator into the data log, the sidecar container reads the data log, and sends the fine-grained GPU utilization rate indicator back to the data reporting module via webhook. S53, the data reporting module performs anomaly detection on the fine-grained GPU utilization index. If no anomaly is found, the fine-grained GPU utilization index is exposed to the Prometheus system alarm tool.

7. A fine-grained monitoring method for a shared GPU container based on Kubernetes according to claim 6, characterized in that, The fine-grained GPU utilization metrics include: operator time spent on CPU, operator time spent on GPU, operator memory and video memory usage, and operator floating-point operations.

8. A fine-grained monitoring system for shared GPU containers based on Kubernetes, characterized in that, include: The monitoring module, Kubernetes client, and data reporting module are included. The monitoring module is deployed on each Kubernetes cluster machine in the form of daemonset. It is used to detect whether there are GPU containers online on each Kubernetes cluster machine. If so, it further checks whether the GPU containers are in normal status. When the GPU containers are in normal status, it obtains the corresponding GPU process information; it obtains the machine's local process information to get system-level metrics; it obtains the Kubernetes container UID through the process identifier PID in the GPU process information and associates it with the Kubernetes container name to calculate the usage information of multiple GPU containers and obtain process-level metrics. The Kubernetes client is used to read the Kubernetes namespace and container information of the Kubernetes cluster. The data reporting module is used to integrate system-level and process-level metrics from the monitoring module and k8s namespace and container information from the k8s client, expose the system-level and process-level metrics to the Prometheus system alerting tool, use the Prometheus system alerting tool to verify the system-level and process-level metrics, and send the verification results to the user.

9. A fine-grained monitoring system for shared GPU containers based on Kubernetes according to claim 8, characterized in that, The monitoring module obtains local process information of the machine by reading the / proc / pid / mountinfo file of each Kubernetes cluster, and obtains GPU process information of the Kubernetes cluster by accessing the GPU container of each Kubernetes cluster.

10. A fine-grained monitoring system for shared GPU containers based on Kubernetes according to claim 8, characterized in that, Also includes: Business container and sidecar container, among which, The business container is used to obtain fine-grained GPU utilization metrics while using the GPU, and write the fine-grained GPU utilization metrics into the data log. The sidecar container is located in the user's pod and is used to read the data logs. It also uses a webhook to send the fine-grained GPU utilization metric back to the data reporting module. The data reporting module performs anomaly detection on the fine-grained GPU utilization metric. If no anomaly is found, the fine-grained GPU utilization metric is exposed to the Prometheus system alerting tool.

Images