Automatic Pod migration method after Kubernetes large model service GPU (Graphics Processing Unit) exception based on rolling update mechanism

Abstract

The invention provides an automatic Pod migration method after a Kubernetes large model service GPU (Graphics Processing Unit) is abnormal based on a rolling update mechanism. According to the method for automatically migrating the Pod after the GPU of the Kubernetes large model service is abnormal based on the rolling update mechanism, automatic migration of the Pod when the GPU is abnormal can be achieved, the high availability of the large model service is improved, and service interruption and resource waste caused by GPU faults are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Pod automatic migration in container orchestration systems, and particularly relates to a Pod automatic migration method after GPU exception of a Kubernetes large model service based on a rolling update mechanism. BACKGROUND

[0002] With the rapid development of containerization technology, Kubernetes (k8s for short) as the core container orchestration system in cloud native architecture is widely used in the deployment and management of large-scale distributed services. In the field of artificial intelligence, large model services are usually deployed in the form of Deployment workloads on the k8s platform, and service instances are run and scheduled through Pods. Since large model training and inference are highly dependent on GPU resources, the high availability of the service directly depends on the stability of the GPU device. In related technologies, the health status of the Pod is detected through the LivenessProbe and ReadinessProbe mechanisms, and the kubelet and apiserver are cooperated to build a basic fault recovery system. Specifically, the system covers the whole process from health check configuration, event listening, container restart to state recovery, including key links such as Pod life cycle management, resource allocation strategy, and node state monitoring.

[0003] However, in the existing k8s health check mechanism, the probe detection is directly performed at the Pod level, and there is no identification and response capability for GPU hardware exceptions, which may cause the Pod to be marked as abnormal but unable to escape from the faulty GPU, or only recovered through local restart, thereby causing service interruption or resource shock. Specifically, when the GPU has problems such as memory error and driver exception, the health check of the Pod can trigger container restart, but it is still bound to the faulty GPU and cannot realize resource migration across GPUs. In addition, frequent restart operations will increase the scheduling burden of the cluster and affect the overall resource utilization and service stability. Based on this, the service recovery capability of the existing technology in the GPU exception scenario has obvious deficiencies, and it is urgent to build a cooperative mechanism that can accurately identify the GPU state and realize Pod automatic migration. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0005] To this end, the first object of the present application is to provide a Pod automatic migration method after GPU exception of a Kubernetes large model service based on a rolling update mechanism.

[0006] The second object of the present application is to provide a Kubernetes large model service GPU abnormality post-Pod automatic migration device based on a rolling update mechanism.

[0007] The third object of the present application is to provide an electronic device.

[0008] The fourth object of the present application is to provide a computer-readable storage medium.

[0009] The fifth object of the present application is to provide a computer program product.

[0010] To achieve the above object, the first aspect of the present application provides a Kubernetes large model service GPU abnormality post-Pod automatic migration method based on a rolling update mechanism, comprising: S1, through a special monitoring component deployed on each node of a Kubernetes cluster, a List / Watch mechanism is used to listen to a health check failure event of a Pod in real time, and identification information of the Pod in the event is extracted; S2, according to the identification information of the Pod, a GPU device number mounted by the Pod is queried, and a node agent is called to execute a GPU state detection command to obtain a health state index of the GPU; S3, if the health state index of the GPU is detected to be out of a preset threshold range, it is determined that the GPU is abnormal, and Deployment configuration information to which the Pod belongs is obtained; S4, a kube-apiserver interface is called to perform an increment operation on an environment variable UPDATE_CNT of the Deployment to trigger a rolling update mechanism, and a new Pod is created on a normal GPU card to replace an abnormal Pod.

[0011] In an embodiment of the present application, the special monitoring component deployed on each node of the Kubernetes cluster listens to a health check failure event of a Pod in real time based on a List / Watch mechanism, and extracts identification information of the Pod in the event, which further comprises: S11, the special monitoring component listens to an / events interface through a 10255 port of a node kubelet to obtain a real-time event stream; S12, a filtering condition of the health check failure event comprises that an event reason reason is "Unhealthy" and a message message is "Liveness probe failed".

[0012] In an embodiment of the present application, the step of querying the GPU device number mounted by the Pod according to the identification information of the Pod, and invoking a node agent to execute a GPU state detection command to obtain the health status indicators of the GPU further comprises: S21, the GPU device number is extracted through the environment variable NVIDIABLE_VISIBLE_DEVICES of the Pod; and S22, the GPU state detection command is nvidia-smi, which is used to check the temperature, ECC error count, driver state and other indicators of the GPU, and compare them with preset thresholds to determine whether they are abnormal.

[0013] In an embodiment of the present application, the step of triggering the rolling update mechanism by performing an increment operation on the environment variable UPDATE_CNT of the Deployment through the kube-apiserver interface to create a new Pod on a normal GPU card and replace the abnormal Pod further comprises: S41, the initial value of the environment variable UPDATE_CNT is 0, and the value is incremented by 1 each time the rolling update is triggered; and S42, the increment operation is updated through the container environment variable field of the Deployment, and the rolling update strategy is automatically executed by the Deployment controller of Kubernetes.

[0014] In an embodiment of the present application, the method further comprises: S5, matching the node information of the abnormal Pod with the normal GPU node information, and selecting a target node based on a node affinity strategy to create a new Pod.

[0015] To achieve the above purpose, a second aspect embodiment of the present application provides a Kubernetes large model service GPU abnormal Pod automatic migration device based on a rolling update mechanism, comprising: an event listening module, configured to listen to a health check failure event of a Pod in real time based on a List / Watch mechanism through a special monitoring component deployed on each node of a Kubernetes cluster, and extract identification information of the Pod in the event; a GPU state query module, configured to query a GPU device number mounted by the Pod according to the identification information of the Pod, and invoke a node agent to execute a GPU state detection command to obtain health status indicators of the GPU; an abnormality determination module, configured to determine that the GPU is abnormal if the health status indicators of the GPU are detected to be out of a preset threshold range, and obtain Deployment configuration information to which the Pod belongs; and a rolling update triggering module, configured to trigger a rolling update mechanism by performing an increment operation on an environment variable UPDATE_CNT of the Deployment through a kube-apiserver interface to create a new Pod on a normal GPU card and replace the abnormal Pod.

[0016] In an embodiment of the present application, the event listening module is further configured to listen to a 10255 port of a node kubelet through an / events interface to obtain a real-time event stream, and filter events in the event stream, wherein the filtering condition comprises that an event reason is "Unhealthy" and an event message is "Liveness probe failed".

[0017] To achieve the above object, the third aspect of the present application provides an electronic device, comprising: a processor, and a memory connected with the processor in communication;

[0018] The memory stores computer execution instructions.

[0019] The processor executes the computer execution instructions stored in the memory to implement the method according to any one of the first aspect.

[0020] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method according to any one of the first aspect.

[0021] To achieve the above object, the fifth aspect of the present application provides a computer program product, wherein the computer program is executed by a processor to implement the method according to any one of the first aspect.

[0022] The embodiments of the present application provide the technical solutions at least with the following beneficial effects: the automatic migration of the Pod when the GPU is abnormal is realized, the high availability of the large model service is improved, and the service interruption and resource waste caused by the GPU failure are avoided.

[0023] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and the accompanying drawings or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 is a flowchart of a Kubernetes large model service GPU abnormality Pod automatic migration method based on a rolling update mechanism according to an embodiment of the present application;

[0026] Figure 2FIG. 1 is a structural schematic diagram of a Kubernetes large model service GPU abnormality post-Pod automatic migration device based on a rolling update mechanism according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0028] Figure 1 FIG. 2 is a flowchart of a Kubernetes large model service GPU abnormality post-Pod automatic migration method based on a rolling update mechanism according to an embodiment of the present application.

[0029] As shown in FIG. 2, the Kubernetes large model service GPU abnormality post-Pod automatic migration method based on a rolling update mechanism includes the following steps: Figure 1 S1, through a special monitoring component deployed on each node of the Kubernetes cluster, real-time monitoring of the health check failure event of the Pod based on the List / Watch mechanism, and extracting the identification information of the Pod in the event.

[0030] Specifically, in the present application, through a special monitoring component deployed on each node of the Kubernetes cluster, real-time monitoring of the health check failure event of the Pod based on the List / Watch mechanism, and extracting the identification information of the Pod in the event, is a core pre-step to realize the GPU abnormality post-Pod automatic migration. The technical implementation principle of this step is based on the event-driven architecture of Kubernetes, by listening to the Pod state event reported by Kubelet, combined with specific filtering conditions, to accurately identify the Pod instance that fails the health check due to GPU abnormality.

[0031] In the specific implementation mode, the monitoring component is deployed in the form of DaemonSet to ensure that an instance is running on each node. The component accesses the / events interface through the 10255 port of Kubelet, and continuously monitors the event stream of the Pod in the cluster using the List / Watch mechanism. When the event type is ProbeFailed and the event reason is Unhealthy, and the message content is Liveness probe failed, the component will trigger the event processing process. At this time, the Ready state of the Pod has changed to 0, indicating that its service is unavailable.

[0032]

[0033] ​Further, the component obtains the complete metadata of the Pod, including the information of the mounted GPU device, by calling the / api / v1 / namespaces / {namespace} / pods / {podName} interface of the Kubernetes API Server. Specifically, the GPU device ID (e.g., nvidia0, nvidia1, etc.) bound to the Pod can be extracted by reading the environment variable NVIDIA_VISIBLE_DEVICES of the Pod. If the environment variable does not exist, it indicates that the Pod is not mounted with a GPU device, and no subsequent processing is needed.

[0034] The key technical indicators involved in this step include event listening frequency, health check failure determination threshold, and GPU device identification resolution method, etc. For example, the default health check failure determination threshold of Kubelet is 3 consecutive failures (failureThreshold = 3), and the detection interval is 10 seconds (initialDelaySeconds = 10, periodSeconds = 10), which can be adjusted through Deployment configuration. In addition, the extraction of GPU device ID needs to follow the standard environment variable format of NVIDIA container runtime to ensure compatibility with the DevicePlugin mechanism.

[0035] In actual application scenarios, this step is suitable for Kubernetes clusters that deploy large model services at a large scale, especially in GPU resource-intensive tasks such as deep learning inference and training services. By real-time listening to Pod health events, the system can quickly identify the affected Pods when GPU memory errors, driver abnormalities, or temperature exceeding the standard occur, providing a data basis for subsequent GPU state checking and Pod migration decisions.

[0036] The technical value of this step lies in the combination of event-driven mechanism and environment variable resolution, which realizes the accurate identification and positioning of abnormal GPU Pods, and provides a trigger basis for the subsequent rolling update strategy. Compared with the original health check mechanism of Kubernetes, this scheme not only improves the real-time performance of fault response, but also avoids the resource waste and service interruption caused by Pod restarting on abnormal GPU, thereby significantly enhancing the high availability and fault tolerance of large model services.

[0037] S2, according to the identification information of the Pod, queries the GPU device number mounted by the Pod, and calls the node agent to execute the GPU state detection command to obtain the health status indicators of the GPU.

[0038] Specifically, in some implementations, according to the identification information of the Pod, the number of GPU devices mounted by the Pod is queried, and a node agent is called to execute a GPU state detection command to obtain the health status indicators of the GPU, which is one of the core steps of the present application to realize automatic migration of the Pod. This step interacts with the environment variables of the Pod through the DevicePlugin mechanism of Kubernetes, thereby accurately identifying the GPU device number bound by the Pod. Specifically, after receiving the Pod event of health check failure, the Pod listening module first obtains the metadata information of the Pod through the Kubernetes API (such as / api / v1 / namespaces / {namespace} / pods / {podName}), and focuses on parsing the environment variable NVIDIA_VISIBLE_DEVICES. This variable is injected by NVIDIA's DevicePlugin when the Pod is scheduled, and is used to identify the GPU device ID (such as gpu-0000:01:00.0) mounted by the current Pod.

[0039] Further, the GPU state checking module executes the nvidia-smi command on the host through the node agent (such as the gpu-fault-handler component) to obtain the real-time state information of the GPU device. This command can output key health indicators including GPU temperature, video memory usage, ECC error count, GPU utilization, driver status, etc. In the present application, a plurality of health thresholds are set as the basis for judgment, for example: GPU temperature exceeding 85℃, ECC error count exceeding 100 times, video memory error rate exceeding 0.01%, etc. If any indicator exceeds the threshold range, it is determined that the GPU is in an abnormal state.

[0040] Optionally, this step can also combine NVIDIA's DCGM monitoring tool to realize more fine-grained GPU state collection and analysis, meeting the high-precision detection needs of GPU health status in large-scale clusters. Through the above technical means, the system can accurately identify whether the GPU device causing the Pod health check failure is the abnormal source, thereby providing a reliable basis for subsequent triggering of rolling update and realization of Pod migration.

[0041] This step is deployed on the Kubernetes node in actual application, and runs in the form of DaemonSet to ensure that each node has local GPU state detection capability. Its technical value lies in breaking through the limitation of the Kubernetes native health check mechanism that can only trigger container restart, realizing Pod resource-level migration based on GPU health status, and significantly improving the availability and fault tolerance capability of large model services in GPU abnormal scenarios.

[0042] S3, if it is detected that the GPU health status indicator exceeds the preset threshold range, it is determined that the GPU is abnormal, and the Deployment configuration information to which the Pod belongs is obtained.

[0043] Specifically, in some implementations, when it is detected that the GPU health status indicator exceeds the preset threshold range, the system will determine that the GPU is in an abnormal state, and further obtain the Deployment configuration information to which the Pod belongs to trigger the automatic migration process of the Pod. This step is a key link in the entire GPU abnormal handling mechanism, and its implementation relies on the cooperative work of the event listening mechanism of Kubernetes and the GPU state detection tool.

[0044] From the technical implementation aspect, the determination process is completed by the gpu-fault-handler component deployed on each node of the cluster. The component is deployed in the form of DaemonSet, and listens to the Pod events reported by Kubelet on the node in real time through the List / Watch mechanism of Kubernetes. When the LivenessProbe detection of the Pod fails, Kubelet will generate an Event event of type ProbeFailed, and set the Ready state of the Pod to 0. The gpu-fault-handler accurately identifies the Pod whose health check fails due to GPU abnormality by listening to the / events interface (the default port is 10255) and setting the filtering conditions as reason="Unhealthy" and message="Liveness probe failed".

[0045] Further, the component obtains the bound GPU device ID by querying the environment variable NVIDIA_VISIBLE_DEVICES of the Pod. Then, it calls the nvidia-smi command provided by NVIDIA to perform state detection on the specified GPU device. The detection indicators include but are not limited to GPU temperature (the default threshold is 85℃), ECC error count (exceeding the preset threshold such as 1000 times is determined to be abnormal), GPU utilization, and memory usage state. If any indicator exceeds the set threshold, it is determined that the GPU is abnormal.

[0046] At the parameter indicator level, the determination of the GPU health status relies on the preset health threshold configuration, which can be dynamically adjusted according to the GPU model (such as Tesla V100, A100, etc.) and the actual application scenario. For example, for high-load inference services, the temperature threshold can be set to 80℃, and the ECC error threshold can be set to 500 times, to improve the sensitivity of fault response.

[0047] In application scenarios, this step is suitable for large-scale Kubernetes clusters running large model services, especially in distributed inference scenarios with multiple nodes and multiple GPU cards. When a node GPU fails, this mechanism can quickly identify and trigger Pod migration, avoiding service interruption and improving overall system availability.

[0048] In terms of technical effects, this step decouples GPU exceptions from Pod health status, enabling Pod automatic migration based on hardware status and breaking through the limitation of Kubernetes native health check mechanism that can only trigger container restart. By updating the Deployment environment variable UPDATE_CNT and incrementing its value, the rolling update strategy can be effectively triggered to ensure that new Pods are scheduled to nodes with normal GPU resources, thereby achieving seamless service switching and high availability guarantee.

[0049] S4, by calling the kube-apiserver interface, increments the environment variable UPDATE_CNT of the Deployment to trigger the rolling update mechanism, creating a new Pod on a normal GPU card and replacing the abnormal Pod.

[0050] Specifically, in the present application, the environment variable UPDATE_CNT of the Deployment is incremented by calling the kube-apiserver interface to trigger the rolling update mechanism, which is the core control logic for Pod automatic migration after GPU exception. Based on the declarative configuration model and rolling update strategy of Kubernetes, combined with GPU device state detection results, the abnormal Pod is actively replaced.

[0051] Technical implementation level: In Kubernetes, the Deployment controller manages the life cycle of Pod replicas through ReplicaSet. When a Pod fails health check due to GPU exception, the gpu-fault-handler component in the present application locates the Deployment object to which the Pod belongs through the REST API interface of kube-apiserver, and performs atomic increment operation on the environment variable UPDATE_CNT of its container. This operation can be performed through kubectl patch or directly calling / apis / apps / v1 / namespaces / <namespace> / deployments / <deployment-name>The interface is implemented by updating the spec.template.spec.containers.env field of Deployment using the strategic merge patch method. Since the Pod template of Deployment changes once, the controller will automatically trigger a rolling update strategy, which creates new Pods that meet the new template first, and then gradually terminates old Pods, thereby achieving Pod migration.

[0052] Parameter indicator level: In the specific implementation, the initial value of the UPDATE_CNT environment variable is 0, which is incremented by 1 each time the update is triggered, such as from 0 to 1, 1 to 2, etc. The update of this variable needs to meet the following parameter constraints:

[0053] Update method: Use PATCH request, follow the strategic merge policy of Kubernetes API;

[0054] Update frequency: It is recommended to set the minimum update interval to 30 seconds to prevent resource competition or scheduling conflicts caused by frequent updates;

[0055] Environment variable format: name: UPDATE_CNT, value: "1", which needs to be passed in string form to ensure that the Deployment template change is correctly identified;

[0056] Rolling update strategy parameters: maxSurge is recommended to be set to 1 and maxUnavailable is set to 0 to ensure that there are always available Pods providing services during the update process, achieving zero-downtime migration.

[0057] Application scenario level: This step is suitable for large model services deployed in Kubernetes clusters, especially deep learning inference or training tasks that rely on GPU resources. When the GPU on a node has a memory error, driver crash, or temperature exceeds the standard, etc. After the Pod listening module detects a health check failure event, the GPU state checking module confirms the abnormality, and the environment variable updating module immediately performs the UPDATE_CNT self-increment operation on the Deployment, thereby rescheduling and creating new Pods on nodes with healthy GPU resources, ensuring service continuity.

[0058] Technical effect level: Through this step, the system can bypass the limitation of Kubernetes native health check mechanism supporting only Pod restart, realize the migration of Pod from abnormal GPU node to normal node. Its technical value lies in: realizing the automatic migration of Pod in GPU exception, improving the high availability of large model service; using environment variable change as the trigger condition, without modifying the image or service configuration, strong compatibility; through the rolling update strategy, guaranteeing the continuity and stability of the service in the migration process, avoiding service interruption or jitter.

[0059] This method has significant practicality and innovation in containerized AI service operation and maintenance, and provides an effective solution for service self-healing in GPU exception scenarios.

[0060] The Kubernetes large model service GPU exception Pod automatic migration method based on the rolling update mechanism of the embodiment of the application realizes the automatic migration of Pod in GPU exception, improves the high availability and resource utilization of large model service on the Kubernetes platform.

[0061] S5, match the node information of the abnormal Pod with the normal GPU node information, and select a target node based on the node affinity strategy to create a new Pod.

[0062] Specifically, matching the node information of the abnormal Pod with the normal GPU node information and selecting a target node based on the node affinity strategy to create a new Pod is one of the key steps to realize the high availability of large model service in the application. This step combines the rolling update mechanism of Kubernetes and the node affinity scheduling strategy to ensure that when the GPU hardware is abnormal, the Pod can be rescheduled to a node with normal GPU resources, thereby restoring the service running.

[0063] In terms of technical implementation, this step first relies on the abnormal Pod information collected by the Pod listening module, including the node it is located in, the bound GPU device ID, etc. Then, the system confirms whether the GPU device used by the Pod is in an abnormal state, such as memory error, ECC error count exceeding the standard, temperature anomaly, etc. Once the GPU is confirmed to be abnormal, the system will trigger the migration process. At this time, the environment variable update module will update the environment variable UPDATE_CNT of the Deployment to which the Pod belongs, and the value will start from 0 and increment each time the rolling update (RollingUpdate) operation is triggered. The rolling update mechanism will first select a node that meets the resource requirements (such as GPU model, number, driver version, etc.) and has a normal GPU state in the cluster, create a new Pod instance, and delete the original abnormal Pod after the new Pod is ready.

[0064] In the node matching process, the system schedules based on the node affinity policy. Node affinity specifies that a Pod should run on a node with specific labels by configuring the nodeSelector or affinity fields in the Pod template of Deployment. For example, nvidia.com / gpu.present:"true" can be set to ensure that the Pod is only scheduled to a node with GPU. Further, the Taints and Tolerations mechanism can be combined to exclude nodes that have been marked as unavailable or have abnormal GPU, thereby improving the accuracy and safety of scheduling.

[0065] At the parameter index level, the node matching process involves multiple key parameters, including but not limited to: GPU device ID, GPU model (such as Tesla V100, A100), GPU driver version (such as NVIDIA 470.57.02), GPU availability status (obtained through the nvidia-smi command), node labels (such as gpu-health:"healthy"), Pod resource request (such as nvidia.com / gpu:"1"), etc. These parameters together constitute the constraints of node selection, ensuring that new Pods are only deployed on nodes with normal GPU resources.

[0066] In application scenarios, this step is widely applicable to large-scale Kubernetes clusters deploying large model inference or training services. For example, in AI inference services, if the GPU of a node is unavailable due to hardware failure, the system can automatically migrate the service Pod to other nodes with healthy GPUs, avoiding service interruption and improving the overall fault tolerance and availability of the system.

[0067] The technical effect of this step is that through precise node matching and scheduling strategies, it realizes the automatic migration of Pods after GPU abnormalities, solving the problem that Kubernetes native mechanisms cannot migrate GPU-bound Pods across nodes. At the same time, through the rolling update mechanism, it ensures the continuity and stability of the service migration process, avoiding service interruption and resource shock, significantly improving the high availability and operation and maintenance automation level of large model services.

[0068] To achieve the above embodiments, the present application also proposes a Kubernetes large model service Pod automatic migration device after GPU abnormality based on a rolling update mechanism. Figure 2 A structure diagram of a Kubernetes large model service Pod automatic migration device after GPU abnormality based on a rolling update mechanism is provided for the embodiments of the present application. As shown in Figure 2 The device includes:

[0069] The event listening module 100 is configured to listen to a health check failure event of a pod in real time based on a List / Watch mechanism through a special monitoring component deployed on each node of a Kubernetes cluster, and extract identification information of the pod in the event;

[0070] The GPU state query module 200 is configured to query a GPU device number mounted by the pod according to the identification information of the pod, and call a node agent to execute a GPU state detection command to obtain a health state index of the GPU;

[0071] The abnormality determination module 300 is configured to determine that the GPU is abnormal if the health state index of the GPU is detected to be out of a preset threshold range, and obtain Deployment configuration information to which the pod belongs;

[0072] The rolling update triggering module 400 is configured to trigger a rolling update mechanism by performing an increment operation on an environment variable UPDATE_CNT of the Deployment through a kube-apiserver interface, to create a new pod on a normal GPU card and replace the abnormal pod.

[0073] As to the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be described in detail here.

[0074] In order to achieve the above-mentioned embodiments, the present application further provides an electronic device, comprising a processor and a memory connected with the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to realize the method provided by the foregoing embodiments.

[0075] In order to achieve the above-mentioned embodiments, the present application further provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to realize the method provided by the foregoing embodiments.

[0076] In order to achieve the above-mentioned embodiments, the present application further provides a computer program product, comprising a computer program, the computer program is executed by the processor to realize the method provided by the foregoing embodiments.

[0077] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present application comply with relevant laws and regulations, and do not violate public order and good customs.

[0078] It is important to note that user's personal information should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection / sharing should occur after the user is informed of and individually agrees to the purposes for which it is gathered and shared, in compliance with applicable privacy laws and regulations. Additionally, measures should be taken to safeguard and secure access to such collected personal information data and ensure that other users with access to the personal information data adhere to the privacy policies and procedures.

[0079] The present application contemplates providing an implementation to users to selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates providing a user with controls to enable or disable the collection of personal information data. In one embodiment, a user, with such controls, can opt in to or opt out of the collection of personal information data. In another embodiment, the user can be required to opt in to the collection of personal information data. In yet another embodiment, the user can be required to opt in to the collection of personal information data to access or use the service.

[0080] In the preceding embodiments descriptions, reference has been made to descriptive terms such as "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. It is intended that individual features, structures, materials or characteristics described in connection with such embodiments or examples can be included in at least one embodiment or example of the application. Descriptive terms of the above type are not necessarily referring to the same embodiment or example. Also, the particular features, structures, materials or characteristics described in connection with any embodiments or examples can be combined in any suitable manner in one or more embodiments or examples. Furthermore, embodiments or examples described in this specification can be combined with one another in any suitable manner without departing from the scope of the application.

[0081] In addition, the terms "first", "second", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of indicated technical features. Thus, the features defined with "first", "second" can include at least one of the features explicitly or implicitly. In the description of the application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0082] Any processes or methods described in the flowcharts or elsewhere in this specification can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and / or that the various processes can operate successfully with others being performed simultaneously or in another order. The various embodiments described herein can be further understood in relation to the following clauses:

[0083] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other) a machine-readable storage diskette (e.g., floppy, flexible or other), a machine-readable storage card (e.g., ROM, EEPROM, flash memory or other), a machine- readable storage tape (e.g., magnetic, optical or other), a machine-readable storage medium (e.g., a portable electronic device, a computer diskette, a computer memory, a broadcast transmission, or the like), or a machine-readable interface device (e.g., a wireless link, optical link or other). The computer-readable medium can also be, or be included in, a computer program product apparatus that tangibly embodies the programming of instructions. The instructions can be executable by a processor of the instruction execution system, apparatus, or device. In another embodiment, the logic and / or steps represented in flow diagrams or otherwise described herein can be considered as a sequence of logic elements, such as steps, functions, operations, or the like, that can be embodied in any computer-readable medium for execution by an instruction execution system, apparatus, or device.

[0084] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As such, in some embodiments, the memory can be a computer-readable medium. If implemented in hardware, as in another embodiment, the hardware can include any or a combination of the following: discrete logic circuits having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and / or the like.

[0085] Those of skill in the art would understand that information and signals can be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0086] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0087] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

[0088] It should be understood that various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0089] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and replacements can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements within the spirit and principles of the present application should be included in the scope of protection of the present application. < / namespace>

Claims

1. A Kubernetes large model service GPU exception post-pod automatic migration method based on a rolling update mechanism, characterized in that, The method comprises the following steps: S1, through a special monitoring component deployed on each node of the Kubernetes cluster, a List / Watch mechanism is used to listen to the health check failure event of a Pod in real time, and the identification information of the Pod in the event is extracted; S2, according to the identification information of the Pod, the number of the GPU device mounted by the Pod is queried, and a node agent is called to execute a GPU state detection command to obtain the health status index of the GPU; S3, if the health status index of the GPU is detected to be out of the preset threshold range, it is determined that the GPU is abnormal, and the Deployment configuration information to which the Pod belongs is obtained; S4, by calling a kube-apiserver interface, a self-increment operation is performed on the environment variable UPDATE_CNT of the Deployment to trigger a rolling update mechanism, and a new Pod is created on a normal GPU card to replace the abnormal Pod.

2. The method of claim 1, wherein, The special monitoring component deployed on each node of the Kubernetes cluster listens to the health check failure event of a Pod in real time based on the List / Watch mechanism, and extracts the identification information of the Pod in the event, which further comprises the following steps: S11, the special monitoring component listens to the / events interface through the 10255 port of the node kubelet to obtain a real-time event stream; S12, the filtering condition of the health check failure event comprises that the event reason reason is "Unhealthy" and the message message is "Liveness probe failed".

3. The method of claim 1, wherein, According to the identification information of the Pod, the number of the GPU device mounted by the Pod is queried, and a node agent is called to execute a GPU state detection command to obtain the health status index of the GPU, which further comprises the following steps: S21, the number of the GPU device is extracted through the environment variable NVIDIA_VISIBLE_DEVICES of the Pod; S22, the GPU state detection command is nvidia-smi, which is used to check the temperature, ECC error count and driver state of the GPU, and compare with the preset threshold to determine whether it is abnormal.

4. The method of claim 1, wherein, The UPDATE_CNT environment variable of the Deployment is incremented by calling the kube-apiserver interface, which further comprises the following steps: S41, the initial value of the UPDATE_CNT environment variable is 0, and the value is incremented by 1 each time the rolling update is triggered; S42, the self-increment operation is updated through the container environment variable field of the Deployment, and the rolling update strategy is automatically executed through the Deployment controller of Kubernetes.

5. The method of claim 1, wherein, Further comprising: S5, the node information of the abnormal Pod is matched with the normal GPU node information, and a target node is selected based on a node affinity strategy to create a new Pod.

6. An apparatus for automatically migrating a pod after a GPU exception of a Kubernetes large model service based on a rolling update mechanism, characterized in that, The method comprises the following steps: An event listening module is configured to listen to a health check failure event of a pod in real time based on a List / Watch mechanism through a special monitoring component deployed on each node of a Kubernetes cluster, and extract identification information of the pod in the event; A GPU state querying module is configured to query a GPU device number mounted by the pod according to the identification information of the pod, and call a node agent to execute a GPU state detection command to obtain a health status index of the GPU; An abnormality determining module is configured to determine that the GPU is abnormal if the health status index of the GPU is detected to be out of a preset threshold range, and obtain Deployment configuration information to which the pod belongs; A rolling update triggering module is configured to perform an increment operation on an environment variable UPDATE_CNT of the Deployment by calling a kube-apiserver interface, so as to trigger a rolling update mechanism to create a new pod on a normal GPU card and replace an abnormal pod.

7. The apparatus of claim 6, wherein, The event listening module is further configured to: Listen to an / events interface through a 10255 port of a node kubelet to obtain a real-time event stream; Filter events in the event stream, and the filtering condition includes that an event reason reason is "Unhealthy" and a message message is "Liveness probe failed".

8. An electronic device, comprising: comprise: a processor, and a memory connected to the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-5.

10. A computer program product, characterised in that, comprise a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-5.

Images