A method, apparatus and storage medium for protecting container clusters

Abstract

This invention relates to a container cluster protection method, apparatus, and storage medium. When a Node in a Kubernetes cluster crashes, all container instance names on the crashed Node are recorded in a container instance-crash node mapping table. When a Node running a container instance corresponding to a given container instance name crashes, the Node is recorded under the corresponding container instance name in the container instance-crash node mapping table. The currently crashed Node and the previously crashed Nodes are arranged chronologically to form a crashed Node list. If the length of the crashed Node list corresponding to a container instance name in the container instance-crash node mapping table exceeds a set length threshold, the container instance corresponding to that container instance name is determined to be abnormal. The abnormal container instance is then circuit-broken, preventing it from being rebuilt on the Node and avoiding further crashes of Nodes in the Kubernetes cluster caused by the reconstruction of the abnormal container instance.

Description

Technical Field

[0001] This invention relates to the field of cluster protection technology, and in particular to a container cluster protection method, apparatus and storage medium. Background Technology

[0002] Container technology demonstrates significant advantages in terms of lightweightness and elasticity, but it suffers from weak isolation. This weak isolation introduces hidden risks, such as improper configuration of container resource usage limits, abnormalities in the internal application itself, or virus vulnerabilities causing a single container to malfunction. This malfunctioning container may then exhaust the resources of the entire node, leading to the malfunction of the entire node and consequently causing all containers on that node to fail.

[0003] Kubernetes is the real-time standard for container orchestration engines. It enables the construction of container clusters, with a core feature being replica persistence—meaning that if a node fails, containers on that node will be recreated on other nodes. A Kubernetes cluster consists of two types of nodes: Master and Node. The Master runs four components: etcd, API Server, Controller Manager, and Scheduler. The API Server, Controller Manager, and Scheduler form the central control unit of Kubernetes, responsible for managing and scheduling all resources in the cluster. Each Node runs three components: Kubelet, Proxy, and Docker Daemon, responsible for managing the lifecycle of Pods on that node and providing service proxy functionality. The Kubectl command-line tool can be run on both Master and Node nodes, providing a Kubernetes cluster management toolkit. etcd is a highly available key / value store used for persistent storage of all resource objects in the cluster. The API Server provides a wrapper API for manipulating etcd, offering services via REST for creating, deleting, modifying, and querying cluster resource objects, as well as monitoring resource changes. The Controller Manager, as the internal management and control center of the cluster, is responsible for managing nodes, pod replicas, service endpoints, namespaces, service accounts, resource quotas, and more, ensuring the cluster is in its expected working state. For example, if a node unexpectedly crashes, the Controller Manager will automatically replenish pod replicas on other nodes in the cluster. The Controller Manager includes a Replication Controller, whose core function is to ensure that at any given time, the pods associated with a resource object in the cluster maintain a certain number of pod replicas. This mechanism improves disaster recovery capabilities and reduces losses caused by node crashes, but it also introduces a problem: if a node crashes due to a faulty container, when that container is rebuilt on other nodes, it will continue to cause other nodes to become faulty or crash. If the faulty container is repeatedly rebuilt across different nodes, the entire cluster will be paralyzed. Traditional operation and maintenance models rely on manual discovery and log analysis to identify which container instance is problematic. This process, from discovering the phenomenon to resolving it, is generally very time-consuming, and the cluster may already be paralyzed and unable to continue operating before the problem is resolved. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this invention provides a container cluster protection method, apparatus, and storage medium. It can automatically detect and shut down abnormal containers before the cluster becomes paralyzed due to their abnormality, thus preventing further cluster failure.

[0005] In a first aspect, the present invention provides a container cluster protection method, comprising: when a Node node in a Kubernetes cluster crashes, recording all container instance names on the crashed Node node in a container instance-crash node correspondence table; and when a Node node running a container instance corresponding to a container instance name crashes, recording the Node node under the corresponding container instance name in the container instance-crash node correspondence table, and the currently crashed Node node and the previously crashed Node nodes are arranged in chronological order to form a crashed Node node list;

[0006] If the length of the list of downed Node nodes corresponding to a container instance name in the container instance and downed node correspondence table exceeds the set length threshold, then the container instance corresponding to that container instance name is determined to be abnormal, and the abnormal container instance is circuit-broken, so that the abnormal container instance cannot be rebuilt on the Node node.

[0007] Furthermore, within a preset time window, for any instance container name corresponding to the list of failed nodes, if the list of failed nodes changes continuously within the preset time window, the list of failed nodes for that instance container is retained; if there are no continuous changes in the list of failed nodes within the preset time window, the contents of the list of failed nodes are cleared.

[0008] Furthermore, the preset time window is configured with the sum of the maximum time to detect a single node failure and the reconstruction time for the maximum number of container instances in the node to be rebuilt.

[0009] Furthermore, when rebuilding container instances in a failed Node, if a specified node name label is provided and affinity is forced, the corresponding Node will be matched based on the node name label, and the container instances in the failed Node will be rebuilt in the matched Node. For abnormal container instances, the abnormal container instances are configured with node name labels that do not exist in the Kubernetes cluster, so that the abnormal instance container scheduling and rebuilding based on the node name label fails in the corresponding Node, thus realizing the circuit breaker for abnormal container instances.

[0010] Furthermore, when an abnormal container instance is triggered, the corresponding abnormal container instance name and its list of failed nodes are deleted from the container instance and failed node correspondence table; the deleted abnormal container instance name and failed node list are added to the abnormal container instance list; based on each abnormal container instance name and failed node list in the abnormal container instance list, logs are generated, or failed node analysis or failed node recovery and container instance recovery are performed.

[0011] Furthermore, the recovery of crashed Node nodes and container instances includes: periodically creating corresponding replicas as recovery replicas for container instances; if the abnormal container instance corresponding to the container instance name causes the Node node to leave a crash mark before crashing, the Node node is restarted; if the Node node detects the crash mark before starting the container instance, it obtains the recovery replica of the abnormal container instance to replace the original abnormal container instance.

[0012] Furthermore, a set number of recovery copies are retained. Starting from the recovery copy closest to the time when the abnormal container instance caused the downtime, all recovery copies are traversed in reverse chronological order to attempt to recover the downed Node. If the traversed recovery copy still causes the Node to crash, the attempt is continued to try to recover with a more distant recovery copy. If all recovery copies fail to recover, corresponding logs are generated and corresponding prompts are given.

[0013] Furthermore, when performing Kubernetes cluster operations and maintenance, input the operation and maintenance start information into the Kubernetes cluster. The Kubernetes cluster will automatically disable container cluster protection based on the operation and maintenance start information. When the operation and maintenance ends, input the operation and maintenance end information into the Kubernetes cluster. The container cluster protection will automatically be enabled.

[0014] Secondly, the present invention provides a container cluster protection device, comprising: a processing unit, a bus unit, and a storage unit, wherein the bus unit is connected to the storage unit and the processing unit, the storage unit stores a computer program, and the computer program is executed by the processing unit to implement the container cluster protection method.

[0015] Thirdly, the present invention provides a storage medium for implementing a container cluster protection method, wherein the storage medium stores a computer program, and the computer program implements the container cluster protection method when executed by a processor.

[0016] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art:

[0017] When a Node in a Kubernetes cluster crashes, this invention records the names of all container instances on the crashed Node in a table mapping container instances to crashed nodes. When a Node running a container instance corresponding to a given container instance name crashes, the Node is recorded under the corresponding container instance name in the table. The currently crashed Node and previously crashed Nodes are arranged chronologically to form a list of crashed Nodes. If the length of the list of crashed Nodes corresponding to a container instance name in the table exceeds a set threshold, the container instance corresponding to that name is considered abnormal. The abnormal container instance is then circuit-broken, preventing its reconstruction on the Node and avoiding further crashes of Nodes in the Kubernetes cluster caused by its reconstruction.

[0018] This invention utilizes a preset time window to retain changes in the list of downed nodes when changes occur within that time window, and otherwise clears the list. This leverages the continuous changes in the list of downed nodes caused by abnormal container instances to delete discontinuous changes, thus avoiding the accumulation of downed node list length due to non-abnormal container instance factors and preventing false alarms caused by such accumulation. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a container cluster protection method provided in an embodiment of the present invention;

[0022] Figure 2 This is a flowchart provided by an embodiment of the present invention to determine whether the container instance corresponding to the list of failed Nodes is an abnormal container instance by comparing the length of the list of failed Nodes with a set length threshold.

[0023] Figure 3 The flowcharts for recovering a failed Node and a container instance are provided in this embodiment of the invention.

[0024] Figure 4This is a schematic diagram of a container cluster protection device provided for an embodiment of the invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0027] Example 1

[0028] See Figure 1 As shown, the present invention provides a container cluster protection method, including:

[0029] S100: When a Node in a Kubernetes cluster crashes, the names of all container instances on the crashed Node are recorded in the container instance and crashed node mapping table.

[0030] In practice, when a Node in the Kubernetes cluster fails, the Controller Manager on the Master node identifies the failed Node. A Kubernetes cluster consists of two types of nodes: Master and Nodes. The Master runs four components: etcd, API Server, Controller Manager, and Scheduler. The API Server, Controller Manager, and Scheduler form the central control unit of Kubernetes, responsible for managing and scheduling all resources in the cluster. Each Node runs three components: Kubelet, Proxy, and Docker Daemon, responsible for managing the lifecycle of Pods on that node and providing service proxy functionality. The Kubectl command-line tool can be run on both the Master and Nodes, providing a Kubernetes cluster management toolset. etcd is a highly available key / value store used for persistent storage of all resource objects in the cluster. The API Server provides a wrapper API for manipulating etcd, offering services via REST for creating, deleting, modifying, and querying cluster resource objects, as well as monitoring resource changes. The Controller Manager acts as the cluster's internal management and control center, responsible for managing nodes, pod replicas, service endpoints, namespaces, service accounts, resource quotas, and other components within the cluster. It also executes automated repair processes to ensure the cluster remains operational. For example, if a node unexpectedly fails, the Controller Manager will automatically replenish pod replicas on other nodes in the cluster. Internally, the Controller Manager includes Node Controllers, which are responsible for discovering, managing, and monitoring each node in the cluster. Kubelet registers node information with the API Server upon startup and periodically sends node information to the API Server. Upon receiving the node information, the API Server writes it to etcd. The node information stored in etcd includes node health status, node resources, node name, node address information, operating system version, Docker version, and Kubelet version. Node health status is categorized into three levels: "Ready" (True), "Not Ready" (False), and "Unknown."If the `--cluster-cidr` parameter is set during Controller Manager startup, a CIDR address is generated for each Node that does not have a `Spec.PodCIDR` set, and this CIDR address is used to set the Node's `Spec.PodCIDR` to prevent conflicts between different Nodes' CIDR addresses. Node information is read one by one, and the node status information in `nodeStatusMap` is modified multiple times. This node information is compared with the node status information stored in `nodeStatusMap`. If it is determined that no node information was received from the Kubelet, the node information was received from the Kubelet for the first time, or the node's health status changed to unhealthy during the process, the node's status information is saved in `nodeStatusMap`, and the system time of the Node Controller at that time is used as the detection time and the node's health status change time. If it is determined that new node information was received within a specified time and the node's health status changed, the system time of the Node Controller at that time is used as the detection time and the node's health status change time. If it is determined that new node information is received within the specified time, but the node's health status has not changed, then the system time of the node where the Node Controller is located at that time is used as the probe time, and the node health status change time in the last node information is used as the node's health status change time. If it is determined that no node information is received within the specified time, then the node's health status is set to unknown, and the node status is saved through the APIServer.

[0031] For example, if three container instances C1, C2, and C3 are running in Node1, when Node1 crashes, a table is formed to map the container instances to the crashed nodes in the format C1:[Node1,], C2:[Node1,], and C3:[Node1,].

[0032] S200: When a Node node running the container instance corresponding to the container instance name crashes, the Node node is recorded under the corresponding container instance name in the container instance and crash node mapping table. The currently crashed Node node and the previously crashed Node nodes are arranged in chronological order to form a crash Node node list. Specifically, if Node2 subsequently crashes, and Node2 is running container instances C1, C2, C4, and C5, the resulting container instance and crash node mapping table is as follows: C1: [Node1, Node2,], C2: [Node1, Node2,], C3: [Node1,], C4: [Node2,], C5: [Node2,].

[0033] S300 determines whether the container instance corresponding to the list of failed Nodes is an abnormal container instance by comparing its length with a set length threshold. For specific implementation details, please refer to [link / reference needed]. Figure 2 As shown, determining whether a container instance corresponding to a list of failed nodes is an abnormal container instance involves comparing the length of the failed node list with a set length threshold.

[0034] S301, preset length threshold and preset time window; wherein, the preset time window is configured with reference to the maximum time for single node failure to be detected and the reconstruction time for the most container instances in the node to be rebuilt.

[0035] S302, check if the time interval between adjacent changes in the list of failed Nodes is greater than the preset time window. If yes, execute S303; otherwise, execute S304.

[0036] S303, clear the contents of the list of failed Nodes.

[0037] S304, retain the contents of the list of failed Nodes in this instance container.

[0038] S305, iterate through the container instance and crash node mapping table to find the length of the list of crash nodes corresponding to all container instance names. The length of the crash node list refers to the number of crash nodes running the corresponding container instance stored in the crash node list.

[0039] S306. Compare whether the length of the list of crashed Nodes corresponding to the container instance name exceeds the set length threshold. If yes, execute S307; otherwise, execute S308.

[0040] S307, Determine that the container instance corresponding to the corresponding container instance name has an exception.

[0041] S308, determine if there are any anomalies in the container instance corresponding to the corresponding container instance name.

[0042] For any instance container name corresponding to a list of failed Nodes, if the list of failed Nodes changes consecutively within a preset time window, the list of failed Nodes for that instance container is retained. If there are no consecutive changes within the preset time window, the list of failed Nodes is cleared. When a failed container instance causes a Node to fail, and then the failed container instance is rebuilt and runs on other Nodes, causing those Nodes to fail, this often results in a continuous chain of failures. Node failures caused by non-failed container instances are usually not continuous chains. Filtering the failed Nodes recorded in the instance container's failed Node list through a preset time window can effectively avoid the impact of accumulated failures caused by non-failed container instances on the analysis results.

[0043] S400 implements a circuit breaker mechanism to prevent abnormal container instances from being rebuilt on Node nodes. Specifically, when rebuilding a container instance on a failed Node node, if a node name label is specified and affinity is set to mandatory, the system will match the corresponding Node node based on the label and rebuild the container instance on the failed Node node within the matched Node. For abnormal container instances, a node name label that does not exist in the Kubernetes cluster is configured for the abnormal container instance, causing the scheduling and rebuilding of the abnormal instance container on the corresponding Node node based on the node name label to fail, thus implementing a circuit breaker for abnormal container instances. When an abnormal container instance is circuit-broken, the corresponding abnormal container instance name and its associated list of failed Node nodes are deleted from the container instance and failed Node mapping table. The deleted abnormal container instance name and its associated list of failed Node nodes are then added to the abnormal container instance list. Log generation, failed Node analysis, or failed Node recovery and container instance recovery are then performed based on each abnormal container instance name and associated list in the abnormal container instance list.

[0044] In the specific implementation process, please refer to Figure 3 As shown, a feasible method for recovering a failed Node and container instance includes: periodically creating corresponding replicas as recovery replicas for container instances within the Node, and retaining a set number of recovery replicas.

[0045] An abnormal container instance corresponding to the container instance name caused the Node node to leave a crash mark before it crashed.

[0046] When restarting a Node, if a crash is detected before starting the container instance, the Node obtains a recovery copy of the failed container instance to replace the original failed container instance. If no crash is detected, the Node runs the original container instance. Specifically, starting with the recovery copy closest to the time the failed container instance caused the downtime, all recovery copies are traversed in reverse chronological order. Each recovery copy attempts to recover the failed Node. Specifically, the affinity of the recovery copies is changed to make them compatible with the failed Node, allowing the recovery copy to attempt recovery on the failed Node. If the traversed recovery copy still causes the Node to crash, attempts are made with older recovery copies. If all recovery copies fail to recover the Node, corresponding logs are generated and appropriate prompts are given. This recovery attempt mechanism effectively avoids serious losses caused by misjudgments.

[0047] In practice, to avoid the impact of continuous downtime caused by operations and maintenance on the results, which could lead to normal container instances being identified as abnormal, the container cluster protection method is terminated during operations and maintenance. Specifically, when performing Kubernetes cluster operations and maintenance, the operation and maintenance start information is entered into the Master node of the Kubernetes cluster. Based on the operation and maintenance start information, the Kubernetes cluster automatically disables container cluster protection. When the operation and maintenance ends, the operation and maintenance end information is entered into the Kubernetes cluster, and the Kubernetes cluster automatically enables container cluster protection based on the operation and maintenance end information.

[0048] Example 2

[0049] See Figure 4 As shown, this embodiment of the invention provides a container cluster protection device, including: a processing unit, a bus unit, and a storage unit, wherein the bus unit is connected to the storage unit and the processing unit, the storage unit stores a computer program, and the computer program is executed by the processing unit to implement the container cluster protection method.

[0050] Example 3

[0051] This invention provides a storage medium for implementing a container cluster protection method. The storage medium stores a computer program, which, when executed by a processor, implements the container cluster protection method.

[0052] When a Node in a Kubernetes cluster crashes, this invention records the names of all container instances on the crashed Node in a table mapping container instances to crashed nodes. When a Node running a container instance corresponding to a given container instance name crashes, the Node is recorded under the corresponding container instance name in the table. The currently crashed Node and previously crashed Nodes are arranged chronologically to form a list of crashed Nodes. If the length of the list of crashed Nodes corresponding to a container instance name in the table exceeds a set threshold, the container instance corresponding to that name is considered abnormal. The abnormal container instance is then circuit-broken, preventing its reconstruction on the Node and avoiding further crashes of Nodes in the Kubernetes cluster caused by its reconstruction.

[0053] This invention utilizes a preset time window to retain changes in the list of downed nodes when changes occur within that time window, and otherwise clears the list. This leverages the continuous changes in the list of downed nodes caused by abnormal container instances to delete discontinuous changes, thus avoiding the accumulation of downed node list length due to non-abnormal container instance factors and preventing false alarms caused by such accumulation.

[0054] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0055] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0056] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0057] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method of protecting a cluster of containers, characterized by, include: When a Node in a Kubernetes cluster crashes, all container instance names on the crashed Node are recorded in the container instance-crash node mapping table. When a Node running a container instance corresponding to a container instance name crashes, the Node is recorded under the corresponding container instance name in the container instance-crash node mapping table. The currently crashed Node and the previously crashed Nodes are arranged in chronological order to form a list of crashed Nodes. A preset time window is set up. For any instance container name, if the list of crashed nodes changes continuously within the preset time window, the list of crashed nodes for that instance container is retained. If there are no continuous changes in the list of crashed nodes within the preset time window, the contents of the list of crashed nodes are cleared. If the length of the list of crashed Node nodes corresponding to a container instance name in the container instance and crashed node correspondence table exceeds a set length threshold, then the container instance corresponding to that container instance name is determined to be abnormal, and the abnormal container instance is circuit-broken, so that the abnormal container instance cannot be rebuilt on the Node node; wherein, the length of the list of crashed Node nodes refers to the number of crashed Node nodes running the corresponding container instance stored in the list of crashed Node nodes.

2. The container cluster protection method according to claim 1, wherein, The preset time window is configured by referencing the maximum time to detect a single Node failure and the reconstruction time to complete the reconstruction of the maximum number of container instances in the Node.

3. The container cluster protection method according to claim 1, wherein, When rebuilding container instances in a failed Node, if a node name label is specified and affinity is forced, the corresponding Node will be matched based on the node name label, and the container instances in the failed Node will be rebuilt in the matched Node. For abnormal container instances, the abnormal container instance is configured with a node name label that does not exist in the Kubernetes cluster, so that the abnormal instance container scheduling and reconstruction on the corresponding Node fails based on the node name label, thus realizing the circuit breaker for abnormal container instances.

4. The container cluster protection method according to claim 1, wherein, When an abnormal container instance is tripped, the corresponding abnormal container instance name and its list of crashed nodes are deleted from the container instance and crashed node correspondence table; the deleted abnormal container instance name and crashed node list are added to the abnormal container instance list; based on each abnormal container instance name and crashed node list in the abnormal container instance list, logs are generated, or crashed node analysis or crashed node recovery and container instance recovery are performed.

5. The container cluster protection method according to claim 4, wherein, Recovery of crashed Node nodes and container instances includes: periodically creating corresponding replicas as recovery replicas for container instances; if a Node node crashes due to an abnormal container instance corresponding to the container instance name, leaving a crash mark before the Node node crashes, the Node node is restarted; if the Node node detects a crash mark before starting the container instance, it obtains a recovery replica of the abnormal container instance to replace the original abnormal container instance.

6. The container cluster protection method according to claim 5, wherein, Retain a set number of recovery copies. Starting with the recovery copy closest to the time when the abnormal container instance caused the downtime, traverse all recovery copies in reverse chronological order to attempt to recover the downed Node. If the traversed recovery copy still causes the Node to crash, continue to try to recover with a recovery copy that is older. If all recovery copies cannot be recovered, generate the corresponding log and give the corresponding prompt.

7. The container cluster protection method according to claim 1, wherein When performing Kubernetes cluster maintenance, input the maintenance start information into the Kubernetes cluster. The Kubernetes cluster will automatically disable container cluster protection based on the maintenance start information. When the maintenance ends, input the maintenance end information into the Kubernetes cluster. The container cluster protection will automatically be enabled.

8. A container cluster protection apparatus characterized by comprising: include: The system includes a processing unit, a bus unit, and a storage unit, wherein the bus unit connects the storage unit and the processing unit, the storage unit stores a computer program, and the computer program, when executed by the processing unit, implements the container cluster protection method as described in any one of claims 1-7.

9. A storage medium for implementing a container cluster protection method, the storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the container cluster protection method as described in any one of claims 1-7.

Images