A fault isolation method and system based on containerized application management system

Abstract

The present invention discloses a fault isolation method and system based on a containerized application management system, which belongs to the technical field of containerized application management. The fault isolation method includes a container isolation method: performing health monitoring on the container to obtain one or more fault container groups; obtaining a Deployment that manages the fault container group; determining whether the Deployment has multiple fault container groups; if so, isolating one or more first fault container groups of the Deployment, and terminating the second fault container group. During the container isolation process, the fault application POD is retained to facilitate troubleshooting of the cause of the fault; for multiple fault container groups under the same Deployment, only some of the fault container groups are isolated, and the remaining fault container groups are terminated or killed to reduce resource consumption caused by isolation.

Description

Technical Field

[0001] The present invention relates to the technical field of containerized application management, and in particular to a fault isolation method based on a containerized application management system. Background Art

[0002] With the trend of containerized applications moving to the cloud, more and more companies are choosing containerized application management systems (Kubernetes) to deploy business applications in cluster environments.

[0003] Based on the cluster container environment characteristics of Kubernetes, after the application container group (pod) is scheduled to a node, if the application pod cannot run successfully due to the host itself or other reasons, Kubelet will continuously restart the container but will not restart the application pod, resulting in the application pod being unable to be rescheduled to a suitable node, and the business pod will only be continuously restarted on an inappropriate node.

[0004] When an application fails, Kubelet restarts the container and deletes the failed container, resulting in the failure to retain the "scene" (the application container at the time of the failure), making it inconvenient to trace back and troubleshoot the cause of the application error.

[0005] When a host in a Kubernetes cluster fails, it will affect the business pods running on the failed host. The only way to remove it is through manual intervention. The application failure caused by the failed host is slow to recover, and it is impossible to isolate the failed host in real time and remove the business in real time to reduce the impact on the business.

[0006] Therefore, in order to better utilize the Kubernetes cluster, it is necessary to break through the fault isolation mechanism. Summary of the invention

[0007] In view of the above technical problems existing in the prior art, the present invention provides a fault isolation method and system based on a containerized application management system, which isolates or terminates a faulty container or container group to facilitate troubleshooting of the cause of the fault.

[0008] The present invention discloses a fault isolation method based on a containerized application management system, the fault isolation method comprising a container isolation method: performing health monitoring on containers to obtain one or more fault container groups; obtaining a Deployment for managing the fault container groups; determining whether the Deployment has multiple fault container groups; if so, isolating one or more first fault container groups of the Deployment and terminating a second fault container group.

[0009] Preferably, the isolated first fault container group is used for troubleshooting:

[0010] Use the troubleshooting script to troubleshoot the first fault container group, and save the troubleshooting results.

[0011] Preferably, the method for monitoring container health comprises:

[0012] Detection of faulty containers by probes;

[0013] Or a faulty container exists when any of the following conditions or a combination thereof is detected: the CPU or memory usage of the container continues to exceed the first threshold; the thread pool of the container is full; an error keyword appears in the container.

[0014] Preferably, the method of isolating by the first tag is:

[0015] Setting a first label for the fault container group;

[0016] The container group with the first label is removed from the Deployment, the IP and port of the container group in the Endpoint list are removed, and the container group is deregistered.

[0017] Preferably, the scheduling method of the container group is:

[0018] Determine whether the third container group meets the first condition: it is a faulty container group and the startup time exceeds the second threshold;

[0019] If satisfied, delete the third container group and record the node information where it is located;

[0020] Get the scheduling list of the newly created fourth container group;

[0021] Determining whether the scheduling list includes the node;

[0022] If the node is included, deleting the node from the scheduling list;

[0023] If not included, scheduling is performed according to the scheduling list;

[0024] Determine whether the scheduling list is empty;

[0025] If it is empty, clear the recorded node information.

[0026] Preferably, the fault isolation method also includes a node isolation method:

[0027] Check the health status of the node,

[0028] The inspection includes any one of the following items or a combination thereof: kubelet health check, disk pressure, memory pressure, PID pressure, network, docker component, containerd component and Calico component;

[0029] If there is a failure in the health of a node, isolate the node.

[0030] Preferably, if there is a health failure of a node, check whether the global lock of the node is valid;

[0031] If invalid, isolate the node, deregister the corresponding container group, and add a second label to the node.

[0032] Preferably, the node monitoring module deployed in the node periodically reports its own heartbeat to the node object of the global lock;

[0033] The node controller monitors the node object and determines whether the heartbeat data of the node object is abnormal;

[0034] If there is an abnormality, a second label is added to the node through the node controller, and the node is isolated.

[0035] Preferably, the detection indicators of the docker component include: the Dockerd process exists and is not a zombie process, and is detected once every second. If it fails three times in a row, the node is considered to be faulty;

[0036] The detection indicators of the Calico component include: the Kubelet process exists and is not a zombie process. It is detected once every second. If it fails three times in a row, the node is considered to be faulty.

[0037] The detection indicators of the containerd component include: the docker-containerd process exists and is not a zombie process, the Confd / Felix / Bird process exists and is not a zombie process, and the detection is performed once every second. If it fails three times in a row, the node is considered to be faulty.

[0038] The present invention also provides a system for implementing the above-mentioned fault isolation method, including a container monitoring module and a container isolation module, wherein the container monitoring module is used to perform health monitoring on the container and obtain one or more faulty container groups; the container isolation module is used to obtain a Deployment that manages the faulty container group, and if the Deployment has multiple faulty container groups, isolate one or more first faulty container groups of the Deployment and terminate the second faulty container group.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows: during the container isolation process, the faulty application POD is retained to facilitate troubleshooting of the fault cause; among multiple faulty container groups under the same Deployment, only some of the faulty container groups are isolated, and the remaining faulty container groups are terminated or killed, thereby reducing resource consumption caused by isolation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1is a flow chart of a fault isolation method based on a containerized application management system of the present invention;

[0041] Figure 2 It is a flow chart of the method of container isolation;

[0042] Figure 3 This is the architecture diagram of container isolation;

[0043] Figure 4 It is a flow chart of the scheduling method for creating a new container group;

[0044] Figure 5 This is the architecture diagram of node isolation;

[0045] Figure 6 It is a system logic block diagram of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0048] The present invention provides a fault isolation method based on a containerized application management system, such as Figure 1 As shown, including:

[0049] Step S1: Container isolation.

[0050] By performing health checks on containers, faulty container groups can be isolated or terminated in a timely manner to facilitate troubleshooting and prevent continuous restarts.

[0051] Step S2: Scheduling of container groups.

[0052] After terminating the container group, schedule the new container group.

[0053] Step S3: Node isolation: Perform health checks on nodes / hosts and isolate faulty nodes in a timely manner.

[0054] like Figure 2 As shown, the container isolation method is:

[0055] Step 101: Perform health monitoring on the container to obtain one or more faulty container groups (Pods).

[0056] Among them, the health monitoring or isolation triggering mechanism includes: a probe mechanism and a monitoring mechanism.

[0057] Probe mechanism: When the Kubelet fails to dial the readiness probe, there is a faulty container group. The readiness probe can sense container crashes, OOM, business dialing, etc. However, it is not limited to readiness probes. Online probes can also be used, but the overhead will increase.

[0058] Monitoring mechanism: Based on the existing container monitoring platform (Prometheus), monitoring alarms, operation and maintenance monitoring, and decision-making systems, faulty container groups can be discovered and related APIs can be called to trigger isolation. For example, a faulty container group exists when any of the following conditions or a combination of them is detected: the CPU or memory usage of the container is continuously high, for example, exceeding the first threshold; the thread pool of the container is full; or an error keyword appears in the container.

[0059] Step 102: Obtain the Deployment that manages the faulty container group.

[0060] Step 103: Determine whether the Deployment has multiple fault container groups.

[0061] If yes, execute step 104: isolate one or more first faulty container groups of the Deployment, and terminate the second faulty container group.

[0062] The second fault container group and the first fault container group are fault container groups managed by the same Deployment. Deployment is a controller used to deploy and manage multiple POD instances, which can achieve seamless migration, automatic expansion and reduction, automatic disaster recovery, one-click rollback and other functions.

[0063] If not, execute step 105: isolate the faulty container group.

[0064] Step 106: Use the isolated fault container group or the first fault container group to perform fault troubleshooting: Use the troubleshooting script to perform fault troubleshooting on the first fault container group, and save the troubleshooting result.

[0065] During container isolation, the faulty application POD is retained to facilitate troubleshooting of the fault cause. For multiple faulty container groups under the same Deployment, only some of them are isolated, while the rest are terminated or killed to reduce resource consumption caused by isolation.

[0066] Example 1

[0067] like Figure 3 As shown, isolation is performed by a first tag.

[0068] Step 111: Monitor the health of the container group through the container monitoring platform (Prometheus) and the readiness control needle. The container group includes one or more containers. The process management tool can be deployed in the container to manage the application subprocess to prevent the main process of the application from accidentally exiting and causing the container to be destroyed. The failure scene cannot be retained. It can be deployed through the basic image of the container.

[0069] Step 112: When the readiness control fails, it is determined to be a faulty container group, and a first label is added to the faulty container group, for example, the Pod status is: NoReady; or the container monitoring platform finds a fault in the operation and maintenance monitoring and decision-making system, and calls the API to add the first label to the faulty container.

[0070] Step 113: Perform isolation through a fault isolation controller (Controller), move the container group with the first label out of the deployment, remove the IP and port of the container group in the endpoint list, and deregister the container group in the registration center.

[0071] In a specific embodiment, the Controller monitors the status changes of all Pods (with isolation requirements) in the cluster in the form of List / Watch. Once it finds that the Pod status changes from Ready to Not Ready, it executes the action of isolating the Pod: modify the Label of the Pod (for example, delete the label Labels: app: nginx), and add a special label to the Label of the pod to identify the faulty Pod (for example, status: problem). The faulty Pod after the Label is modified will be removed from the Deploy, the pod IP will be removed from the original Endpoint list, and the traffic of the faulty Pod will be cut off.

[0072] While the Controller isolates the faulty pod, for pods that do not use the service discovery mechanism in the Kubenetes cluster, such as microservices, other registration centers are used to implement service registration and discovery. However, if the Pod application process is destroyed and not deregistered from the registration center, the traffic of the Pod is still there. The Controller will obtain the information of the Pod in the corresponding registration center and remove the pod information (pod IP, etc.) from the registration center. For example, the Provider leaf node Path of the Zookeeper registration center and delete the Provider leaf node Path corresponding to the Pod IP from the Zookeeper registration center.

[0073] When the Controller performs Pod isolation, for multiple faulty Pods belonging to the same Deployment, only one or more of the first faulty Pods are isolated and retained (the number of retained Pods can be passed to the Controller through configuration), and the remaining second faulty Pods are directly terminated and restarted.

[0074] Step 114: At the same time, Kubelet will re-pull a new pod for the deployment to maintain the existing traffic. The time taken to re-pull the new pod is similar to the time taken to restart a regular pod container, which preserves the fault site without affecting the existing traffic.

[0075] Step 115: When the Controller isolates the faulty Pod, you can use the JVM troubleshooting script to generate jmap files, jstack files, etc., and keep them in the container to facilitate subsequent troubleshooting of the Pod failure. After troubleshooting the cause of the isolated faulty application Pod, you can manually delete it.

[0076] Example 2

[0077] like Figure 4 In step S2, the scheduling method of the container group includes:

[0078] Step 201: Determine whether the third container group meets the first condition: it is a faulty container group, and the startup time exceeds a second threshold, such as 3 minutes, but not limited thereto.

[0079] In a specific embodiment, a first label is established for the faulty container group: the Pod status is: NoReady; the startup time can be calculated based on the creation time (time field, metadata.creationTimestamp) of the container group and the current time.

[0080] If satisfied, execute step 202: delete the third container group and record the node information. The faulty Pod can be deleted by calling the apiserver interface. The node information of the deleted pod can be written into the annotation of the deployment corresponding to the pod for recording.

[0081] If not met, continue monitoring the third container group.

[0082] Step 203: Get the scheduling list of the newly created fourth container group. Create a new fourth container group for Deployment through kubelet. The time taken to restart the new pod is similar to the time taken to restart a regular pod container.

[0083] Step 204: Determine whether the scheduling list includes the node.

[0084] If the node is included, step 205: delete the node from the scheduling list and execute step 206 or 208.

[0085] If not included, execute step 206: perform scheduling according to the scheduling list.

[0086] Step 207: Determine whether the scheduling list is empty.

[0087] If it is empty, execute step 208: clear the recorded node information and perform scheduling according to the original scheduling list.

[0088] If it is not empty, execute step 206.

[0089] You can perform steps 201-202 by deploying the pod-problem-controller component and perform steps 203-207 by using a custom extended scheduler.

[0090] The container group scheduling method of this embodiment can effectively avoid the problem that the application pod cannot be successfully run on the current node due to the node / host itself or other reasons, and can promptly discover pods that have not been successfully run for a long time.

[0091] Example 3

[0092] In step S3, the node isolation method includes:

[0093] Step 301: Check the health status of the node, wherein the check includes any one of the following items or a combination thereof: kubelet health check, disk pressure, memory pressure, PID pressure, network, docker component, containerd component and Calico component.

[0094] The host can be detected through the node-agent+node-controller components. The detection indicators are shown in Table 1:

[0095]

[0096] Step 302: If there is a health failure in the node, isolate the node.

[0097] When a host in a Kubernetes cluster fails, it will affect the pods running on the failed host. By isolating the failed host and expelling services in real time, the impact on the services can be reduced.

[0098] In a specific embodiment, the Figure 5 In the architecture shown, a detection module (node-agent) is deployed in the node.

[0099] like Figure 5 As shown by the dotted line, in a specific isolation mechanism, step 302 includes:

[0100] Step 312: If there is a health problem with the node, check whether the global lock (etcd) of the node is valid.

[0101] If invalid, execute step 312: isolate the node, deregister the container group deployed in the node, and add a second label to the node. For example, the faulty node may be tainted with NoExecute by calling the kube-apiserver interface.

[0102] If valid, the node is continuously monitored without isolation. The global lock is used for node protection, for example, node isolation is not performed within 30-60 minutes.

[0103] Figure 5 The solid line shows another specific isolation mechanism, step 302 includes:

[0104] Step 321: The node monitoring module (node-agent) deployed in the node periodically reports its heartbeat to the node object of the global lock (etcd). The node object can be a defined lightweight CRD object, including the node health status of the node where the node-agent is located.

[0105] Step 323: The node controller monitors the node object and determines whether there is an abnormality in the heartbeat data of the node object, for example, no changes in the node object can be obtained after connecting three times.

[0106] If there is an abnormality, execute step 324: perform isolation through the node controller: isolate the node, deregister the corresponding container group, and add a second label to the node.

[0107] The node controller monitors the heartbeat data to prevent node abnormalities from being discovered in time when the node-agent fails or the node fails, such as downtime caused by node hardware failure, node network card failure, node system crash, node system hang (poor network), etc.

[0108] Example 4

[0109] like Figure 6 As shown, this embodiment provides a system for implementing the above fault isolation method, including a container monitoring module 1 and a container isolation module 2.

[0110] The container monitoring module 1 is used to perform health monitoring on the container and obtain one or more faulty container groups;

[0111] The container isolation module 2 is used to obtain a Deployment that manages a faulty container group, and if the Deployment has multiple faulty container groups, isolate one or more first faulty container groups of the Deployment and terminate the second faulty container group.

[0112] The system of the present invention further includes a container scheduling module 3 , a node monitoring module 4 and a node controller 5 .

[0113] The container scheduling module 3 is used for scheduling container groups; the node monitoring module 4 is used to detect the health status of the node and node isolation, and can also be used to upload its own heartbeat; the node controller 5 is used to perform node isolation based on the heartbeat data in the node object.

[0114] The present invention completely preserves the faulty application pod container site through container isolation, which is convenient for subsequent investigation of the cause of the application error; through the debugging of the container group, the application pod is scheduled to avoid the influence of host factors on the application pod, and the business pod that has not run successfully for a long time is discovered in time; in node isolation, a variety of host fault isolation detection and triggering mechanisms are added, and the timeliness of health detection is optimized.

[0115] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fault isolation method based on a containerized application management system, characterized in that: The fault isolation method includes a container isolation method: Perform health monitoring on the container to obtain one or more faulty container groups; Get the Deployment that manages the faulty container group; Determine whether the Deployment has multiple fault container groups; If yes, isolate one or more first faulty container groups of the Deployment and terminate the second faulty container group of the Deployment; It also includes the scheduling method of the container group: Determine whether the third container group meets the first condition: it is a faulty container group and the startup time exceeds the second threshold; If satisfied, delete the third container group and record the node information where it is located; Get the scheduling list of the newly created fourth container group; Determining whether the scheduling list includes the node; If the node is included, deleting the node from the scheduling list; If not included, scheduling is performed according to the scheduling list; Determine whether the scheduling list is empty; If it is empty, clear the recorded node information.

2. The isolation method according to claim 1, characterized in that: Troubleshooting using isolated first-failure container groups: Use the troubleshooting script to troubleshoot the first fault container group, and save the troubleshooting results.

3. The isolation method according to claim 1, characterized in that: Methods for monitoring container health include: Detection of faulty containers by probes; Or a faulty container exists when any of the following conditions or a combination thereof is detected: the CPU or memory usage of the container continues to exceed the first threshold; the thread pool of the container is full; an error keyword appears in the container.

4. The isolation method according to claim 1, characterized in that: Also included is a method for isolation via the first tag: Setting a first label for the fault container group; The container group with the first label is removed from the Deployment, the IP and port of the container group in the Endpoint list are removed, and the container group is deregistered.

5. The isolation method according to any one of claims 1 to 4, characterized in that: The fault isolation method also includes a node isolation method: Check the health status of the node, The inspection includes any one of the following items or a combination thereof: kubelet health check, disk pressure, memory pressure, PID pressure, network, docker component, containerd component and Calico component; If there is a failure in the health of a node, isolate the node.

6. The isolation method according to claim 5, characterized in that: If there is a health problem with the node, check whether the global lock of the node is valid; If invalid, isolate the node, deregister the corresponding container group, and add a second label to the node.

7. The isolation method according to claim 6, characterized in that: The node monitoring module deployed in the node periodically reports its own heartbeat to the node object of the global lock; The node controller monitors the node object and determines whether the heartbeat data of the node object is abnormal; If there is an abnormality, a second label is added to the node through the node controller, and the node is isolated.

8. The isolation method according to claim 7, characterized in that: The detection indicators of the docker component include: the Dockerd process exists and is not a zombie process. It is detected once every second. If it fails three times in a row, the node is considered to be faulty. The detection indicators of the Calico component include: the Kubelet process exists and is not a zombie process. It is detected once every second. If it fails three times in a row, the node is considered to be faulty. The detection indicators of the containerd component include: the docker-containerd process exists and is not a zombie process, the Confd / Felix / Bird process exists and is not a zombie process, and the detection is performed once every second. If it fails three times in a row, the node is considered to be faulty.

9. A system for implementing the isolation method according to any one of claims 1 to 8, characterized in that: Including container monitoring module and container isolation module, The container monitoring module is used to perform health monitoring on the container and obtain one or more fault container groups; The container isolation module is used to obtain the Deployment that manages the faulty container group, and if the Deployment has multiple faulty container groups, isolate one or more first faulty container groups of the Deployment and terminate the second faulty container group.

Images