Method, device and computing equipment for detecting the operational state of a container cluster

Abstract

The application provides a container cluster running state detection method and device and a computing device, and belongs to the technical field of computers. The method acquires running information of nodes in a container cluster, and detects the running state of a container unit according to the running information and a drift condition of an application running on the container unit on the node. The running information indicates information related to the running of the container unit, and the drift condition indicates a condition for triggering drift of the container unit where the application is located. Obviously, the detection process is performed in the granularity of a single container unit on the node to ensure normal running of the application on the single container unit. Compared with the granularity of the node, the detection granularity is refined, the sensitivity of detection is improved, the situation that the container cluster cannot timely discover running abnormalities of a single container unit is avoided, the container cluster can timely drift the container unit, the application on the container unit can be restored to running as soon as possible, and thus the business downtime caused by the running abnormal container unit is shortened.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, and computing device for detecting the running status of a container cluster. Background Technology

[0002] Containerization is an application virtualization technology that packages an application and all its dependencies, configuration files, and runtime environment into containers, enabling the application to run on different operating systems and hardware platforms without being affected by environmental differences. Containerized applications can be deployed in container clusters, which consist of multiple nodes: master nodes and worker nodes. Master nodes manage and schedule the container cluster, while worker nodes run the application. The smallest deployable unit in a container cluster is a container pod. A worker node in a container cluster deploys one or more container pods, and a container pod contains one or more containers. A container pod runs an application or a portion of an application's functionality through its constituent containers.

[0003] Currently, container clusters use the following method to detect their own operational status: the management node in the container cluster maintains heartbeat information for the worker nodes in the cluster. Worker nodes send new heartbeat information to the management node every preset time interval, allowing the management node to update the heartbeat information for that worker node. If the management node detects that the heartbeat information for a worker node has not been updated within a timeout period, the management node determines that the worker node in the container cluster has malfunctioned and performs a migration operation on all container units deployed on that worker node.

[0004] In the above-mentioned container cluster operation status detection method, the operation status is detected at the worker node level. The sensitivity of container cluster detection is low, and the time consumed from the occurrence of an anomaly to the detection of the anomaly is long. This makes the container cluster less efficient at migrating container units on the abnormal worker nodes, resulting in a longer service interruption time caused by the operation anomaly. Summary of the Invention

[0005] This application provides a method, apparatus, and computing device for detecting the operational status of a container cluster, which can shorten the service interruption time caused by operational anomalies. The technical solution is as follows.

[0006] Firstly, a method for detecting the running status of a container cluster is provided. The method includes: obtaining running information of nodes in the container cluster, wherein at least one container unit runs on the node, the container unit is used to run an application, and the running information indicates information related to the running of at least one container unit on the node; and detecting the running status of at least one container unit based on the running information of the node and the drift conditions of the application running by at least one container unit, wherein the drift conditions indicate the conditions that trigger the drift of the container unit where the application is located.

[0007] The container cluster can be a Kubernetes cluster, etc., and a container unit includes one or more containers. Nodes provide background services for corresponding applications by running these containers. This container cluster runtime status detection method, after obtaining the runtime information of the nodes in the container cluster, detects the runtime status of the container units based on the node's runtime information and the drift conditions of the applications running on the container units. Since the runtime information indicates information related to the container unit's operation, and the drift conditions indicate the conditions that trigger the drift of the container unit containing the application, the detection process is clearly performed at the granularity of a single container unit on a node. This ensures the normal operation of the application on that single container unit. Compared to detection at the node level, this refines the granularity of detection, thereby improving the sensitivity of container cluster runtime status detection. It avoids situations where the container cluster cannot detect abnormalities in a single container unit in a timely manner, enabling the container cluster to drift the container unit promptly and allowing the application on that container unit to resume operation as soon as possible, thus shortening the business downtime caused by abnormal container units.

[0008] Optionally, the above method further includes: for each container unit running on a node, if the node's operating information matches the drift conditions of the application running the container unit, triggering a drift operation for that container unit. This process, by comparing the drift conditions with the node's corresponding operating information, determines that an operational anomaly has occurred and immediately drifts the container unit, thus shortening the anomaly switching time and consequently reducing service downtime.

[0009] Optionally, the node's operational information includes at least one of the following: the node's local storage space usage, the node's memory consumption, the node's computing resource consumption, the application's running status, and the connection status between the application and storage resources. This node operational information covers various resources required for container unit operation, enabling multi-faceted monitoring of the container unit's operational status on the node, timely detection of anomalies, and improved sensitivity to anomaly detection.

[0010] Optionally, obtaining the running information of nodes in the container cluster includes: for each node in the container cluster, obtaining the periodically updated running information of the node, which can ensure that the running status of the container units on the node is detected based on the latest running information of the node, thereby improving the accuracy of detection.

[0011] Optionally, the aforementioned drift conditions include a tolerance count. Triggering drift for the container unit if the node's operational information matches the drift conditions of the application running the container unit includes: if the node's operational information indicates that the node's failure count has reached the tolerance count in the drift conditions of the application running the container unit, then triggering drift for the container unit. The node's failure count indicates the number of times the node has failed. By drifting the container unit only after the failure count has reached the tolerance count, erroneous drifting can be avoided, and the accuracy of drifting can be improved.

[0012] Optionally, the above method further includes: if the target timestamp has not been updated within a preset duration, updating the number of node failures, wherein the target timestamp indicates the time when any item in the node's operational information was last updated. By measuring the duration of the target timestamp's lack of update, node failures can be detected in a timely manner, resulting in high sensitivity in fault detection.

[0013] Optionally, the above method further includes: in response to the number of consecutive drifts of a container unit being greater than or equal to a preset number, sending fault information to the client device connected to the container cluster, the fault information indicating that the container cluster has failed. This process enables timely detection and reporting of faults when the container cluster fails and the container units cannot run, thus improving the reliability of the container cluster.

[0014] Optionally, the process of obtaining the connection status between the application and the storage resource includes: the node where the application resides sending a status detection request to the storage resource through the connection between the application and the storage resource; and determining the connection status between the application and the storage resource based on the storage resource's response to the status detection request. This process, by sending a detection request to the storage resource to determine the connection status between the application and the storage resource, can promptly detect whether the connection is faulty, thus improving the sensitivity of fault detection.

[0015] Secondly, a container cluster runtime status detection device is provided, which is used to execute the aforementioned container cluster runtime status detection method. Specifically, the device includes a functional module for executing the container cluster runtime status detection method provided in the first aspect or any optional embodiment of the first aspect.

[0016] Thirdly, a computing device is provided, comprising a processor and a memory, wherein the processor is configured to execute at least one piece of program code stored in the memory to cause the computing device to perform operations as described in the above-described method for detecting the running status of a container cluster.

[0017] Fourthly, a computing device cluster is provided, comprising multiple computing devices, each including a processor and memory. The processor of the multiple computing devices executes instructions stored in the memory of the multiple computing devices, causing the computing device cluster to perform operations as described in the container cluster runtime status detection method. For each container unit running on any computing device in the computing device cluster, if the runtime information of the computing device matches the drift conditions of the application running on that container unit, the container unit is drifted from that computing device to another computing device in the computing device cluster. Drifting the container unit from one computing device to another computing device in the computing device cluster means deleting the container unit from the first computing device and recreating it on the other computing device in the computing device cluster.

[0018] Fifthly, a computer-readable storage medium is provided, which stores at least one piece of program code that is read by a processor to cause a computing device to perform operations as described in the above-described method for detecting the running status of a container cluster.

[0019] In a sixth aspect, a computer program product or computer program is provided, the computer program product or computer program including program code stored in a computer-readable storage medium, a processor of a computing device reading the program code from the computer-readable storage medium, the processor executing the program code, causing the computing device to execute the container cluster running status detection method provided in the first aspect or various optional implementations of the first aspect.

[0020] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the implementation environment of a container cluster operation status detection method provided in this application embodiment;

[0022] Figure 2 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the structure of a container cluster provided in an embodiment of this application;

[0024] Figure 4 This is a flowchart illustrating a node operation information update method provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of another container cluster structure provided in an embodiment of this application;

[0026] Figure 6 This is a flowchart of a method for detecting the running status of a container cluster provided in an embodiment of this application;

[0027] Figure 7 This is an architecture diagram of a Kubernetes cluster provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of a container cluster operation status detection device provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0031] The implementation environment of the container cluster running status detection method provided in the embodiments of this application is described below.

[0032] Figure 1 This is a schematic diagram illustrating the implementation environment of a container cluster runtime status detection method provided in this application embodiment, such as... Figure 1 As shown, the implementation environment includes a client device 101, a container cluster 102, and a storage device 103. The client device 101 and the container cluster 102 are connected via a wired or wireless network, and the container cluster 102 and the storage device 103 are connected via a wired or wireless network.

[0033] The client device 101 can be a portable mobile terminal, such as a smartphone, tablet, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, laptop, or desktop computer. The client device 101 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names. The client device 101 can also be a standalone physical server, or implemented as... Figure 2The computing device cluster shown refers to a server cluster composed of multiple physical servers or a distributed file system, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Taking client device 101 as an example of a cloud server, client device 101 can also be referred to as a cloud platform (short for cloud computing platform), which refers to a service based on hardware and software resources, providing computing, network, and storage capabilities. Through the network "cloud," massive amounts of data are processed and analyzed remotely before being returned to the user, featuring large-scale, distributed, virtualized, highly available, scalable, on-demand service, and security characteristics. Cloud platforms can achieve rapid deployment and release of configurable computing resources with relatively low management costs or low interaction complexity between users and service providers.

[0034] A client application is installed on the client device 101, which enables the client device 101 to send requests to the container cluster 102.

[0035] The container cluster 102 described above includes multiple nodes, each of which can be implemented as a physical server or a virtual machine; this embodiment does not limit this. These multiple nodes include management nodes and worker nodes. Worker nodes are used to run applications, and each worker node runs one or more container units (pods). A container unit includes one or more containers, and each container unit runs an application or a portion of an application's business logic (such as a microservice application) based on its included containers. The management node is used to manage the worker nodes in the container cluster, such as scheduling container units in the container cluster, deploying new container units in the container cluster, deleting container units in the container cluster, monitoring the security of the container cluster, and monitoring the failure status of worker nodes in the container cluster. The management node includes an application programming interface (API) server and a storage system. The storage system in the management node is used to store information about the container cluster, such as the nodes in the container cluster, the running information of the nodes in the container cluster, and the correspondence between container units and the nodes where the container units reside. The API server is the external interface of the management node. Client device 101, nodes in container cluster 102, and storage device 103 can access the management node through the API server to query information such as resources in the container cluster, and monitor changes in resources in the container cluster, thereby realizing corresponding business functions, such as the running status detection of the container cluster and the drift of container units.

[0036] The container cluster 102 is used to provide background services for corresponding applications through container units running on the nodes. Accordingly, the application running on the container unit in the container cluster 102 receives the request sent by the client device 101 through the corresponding client application and executes the request.

[0037] In some embodiments, such as Figure 3 As shown, the container cluster 102 described above is implemented as a Kubernetes cluster (hereinafter referred to as a k8s cluster). The k8s cluster includes multiple master nodes and multiple worker nodes. The master nodes and worker nodes communicate with each other through the internal network of the container cluster (management plane network). The master nodes include kube-api-server, kube-etcd, kube-scheduler, and kube-controller manager. kube-api-server is the external interface of the master nodes, kube-etcd is the storage system of the master nodes, kube-scheduler is used to schedule container units in the container cluster, and kube-controller manager is used to monitor the security of the container cluster and the failure status of worker nodes in the container cluster through the controller in the master nodes, and manage the resources in the container cluster according to the failure status. The controller can be a deployment or a statefulSet, etc., and this embodiment does not limit it. Figure 3 As shown, worker nodes in a Kubernetes cluster run kubelet and container units. kubelet manages the container units running on worker nodes, such as managing the creation, deletion, starting, stopping, and monitoring of container units. kubelet also monitors the status of worker nodes, periodically reporting the status of worker nodes to the management node via the kube-api-server.

[0038] The aforementioned storage device 103 is implemented as block storage, a storage area network (SAN), network attached storage (NAS), or a network file system (NFS) to provide data storage services for the aforementioned applications. Storage device 103 includes multiple persistent volumes (PVs). Container units on worker nodes mount PVs via persistent volume claims (PVCs). A PV represents a portion of the storage space in storage device 103. A PVC includes information such as the size and access mode of the storage space requested by the container unit. Each PV corresponds to an Internet Protocol (IP) address. Container units on worker nodes access the mounted PV based on its IP address to execute requests received by the applications running on that container unit.

[0039] In some embodiments, such as Figure 3 As shown, the storage device 103 and the container cluster 102 are connected via a business plane input / output (IO) network. The storage device 103 is a network file system server (NFS server), etc., but this embodiment does not limit it.

[0040] In related technologies, the management node in a container cluster maintains heartbeat information for the worker nodes. Worker nodes send new heartbeat information to the management node every preset time interval, allowing the management node to update the corresponding heartbeat information. If the management node detects that the heartbeat information for a worker node has not been updated within a timeout period, it determines that the worker node has malfunctioned, marks the worker node as tainted, and waits for a preset time interval. After the preset time interval expires, it deletes the container unit deployed on that worker node and redeploys the container unit on other nodes in the container cluster. This related technology detects operational status at the worker node level, resulting in low sensitivity and a long time consuming from the occurrence of an operational anomaly to its detection. Furthermore, after marking a worker node as tainted, the management node waits for a preset time interval after marking it as tainted before migrating the container units deployed on that worker node. The efficiency of migrating container units on malfunctioning worker nodes is low, leading to prolonged service interruptions caused by operational anomalies.

[0041] To address the issue of prolonged service interruptions caused by operational anomalies in container clusters, this application provides a method for detecting the operational status of container clusters. This method is applied to any node (e.g., any worker node) in the container cluster, referred to as the first node. After acquiring the operational information of nodes in the container cluster, the first node detects the operational status of container units based on the node's operational information and the drift conditions of the applications running on those container units. Since the operational information indicates information related to the container unit's operation, and the drift conditions indicate the conditions that trigger the drift of the container unit containing the application, the detection process is clearly performed at the granularity of a single container unit on the node. This ensures the normal operation of the application on that single container unit. Compared to detection at the node level, this refines the granularity of detection, thereby improving the sensitivity of container cluster operational status detection. This avoids situations where the container cluster cannot detect operational anomalies in a single container unit in a timely manner, enabling the container cluster to drift the container unit promptly and allowing the application on that container unit to resume operation as quickly as possible, thus shortening the service interruption time caused by the abnormal operation of the container unit.

[0042] The first node described above performs runtime status detection based on the runtime information corresponding to each node in the container cluster. This runtime information is stored in the container cluster's storage system. Each node periodically updates its runtime information in the storage system, while the first node periodically retrieves the runtime information of each node from the storage system and performs runtime status detection based on this information. Alternatively, the first node locally stores its own runtime information, periodically updates its own runtime information, and also locally stores the runtime information of each node in the container cluster. Each node periodically updates its own runtime information in the first node, and the first node periodically retrieves the runtime information of each node locally and performs runtime status detection based on this information. This embodiment of the application does not limit this approach.

[0043] In some embodiments, the above method is implemented as an application running on a node in a container cluster, for example, Figure 3 As shown, a `drift-controller` application runs on a worker node in the container cluster. This `drift-controller` is used to execute the container cluster runtime status detection method provided in this embodiment. The method by which the nodes in the container cluster periodically update the runtime information corresponding to that node in the storage system is also implemented as an application running on the corresponding node in the container cluster, for example, such as... Figure 3As shown, each worker node in the container cluster runs the application dri ft-node, which is used to detect the node and periodically update the corresponding running information of the node in the storage system based on the detection results.

[0044] The following example illustrates how the runtime information of each node in a container cluster is updated, using the storage system of the container cluster as an example. Figure 4 This is a flowchart of a node operation information update method provided in an embodiment of this application, such as... Figure 4 As shown, the method includes the following steps.

[0045] 401. Each node in the container cluster periodically checks the node according to the corresponding drift monitoring policy and obtains the corresponding detection results.

[0046] The drift monitoring strategy includes at least one monitored object and a corresponding detection period (e.g., 3 seconds) for each monitored object. For example, if the monitored object in the drift monitoring strategy is the running status of each application on the node, and the detection period for this monitored object is 3 seconds, then the drift monitoring strategy instructs to check whether each application on the node is running normally every 3 seconds. The drift monitoring strategy operates on the node to display real-time network and storage status information corresponding to the node to the container cluster. This allows the first node to compare this information with the drift conditions corresponding to the applications running on the node, thereby triggering the drift of the container unit where the application resides. The monitored objects that the node can detect include various types, such as the connection status of the network within the container cluster (network monitor), the node's local storage space usage (disk pressure), the node's local storage space usage ratio, the node's memory consumption (memory pressure), the node's memory usage ratio, the node's computing resource consumption (CPU pressure), the running status of applications on the node, and the connection status between applications on the node and storage resources. The node's computing resource consumption is also known as process pressure. Each node has a pre-configured drift monitoring policy. The container cluster's storage system stores the mapping between each node and its corresponding drift monitoring policy. Each node in the container cluster retrieves relevant data, such as drift monitoring policies, from the storage system by calling its corresponding API (such as the API server included in the management node mentioned above). Conversely, when the drift monitoring policy corresponding to that node in the storage system is updated, each node retrieves the updated drift monitoring policy by calling the corresponding API. By associating drift monitoring policies with nodes in the storage system, the container cluster enables nodes to generate detection tasks based on the corresponding drift monitoring policies to monitor the status of various monitored objects.

[0047] In this embodiment of the application, each node in the container cluster performs a detection on each monitored object indicated by the drift monitoring strategy corresponding to that node, according to the detection period corresponding to that monitored object in the drift monitoring strategy, and obtains the detection result corresponding to that monitored object every corresponding detection period.

[0048] The process of detecting the various monitoring objects mentioned above will be explained below.

[0049] When monitoring the connectivity status of the container cluster's internal network, nodes send heartbeat messages to the container cluster's storage system or other nodes via the internal network at corresponding monitoring intervals. By setting this monitoring interval, the container cluster can reduce the time spent monitoring its internal network, improving monitoring efficiency. For example, ... Figure 3 As shown, the drift-node or ku belet in the worker node sends heartbeat information to the internal network of the container cluster.

[0050] When the monitored object is the usage of the node's local storage space, the node obtains the usage of the node's local storage space every corresponding detection period and uses this usage as the detection result of the monitored object. Alternatively, if the usage of the node's local storage space reaches the preset usage limit, the detection result of the monitored object is determined to be a fault. If the usage of the node's local storage space does not reach the preset usage limit, the detection result of the monitored object is determined to be normal.

[0051] When the monitored object is the occupancy rate of the node's local storage space, the consumption of the node's memory, or the consumption of the node's computing resources, the process of detecting the monitored object is the same as the process of detecting the occupancy rate of the node's local storage space, and will not be described again in the embodiments of this application.

[0052] When the monitored object is the running status of an application on a node, the node checks the application's running status every corresponding detection period using a pre-defined detection method (custom rule set) and obtains the detection result. One detection method is that the node calls the application's status query interface, sending a status query request to the application. The application returns the running status based on the status query request. This running status is either normal (true), abnormal (false), or unknown. If the running status is normal, the detection result is normal; if the running status is abnormal, the detection result is faulty; if the running status is unknown, the detection result is related to the preset rules. Accordingly, if the preset rule indicates that the unknown status is a fault, the detection result is a fault; if the preset rule indicates that the unknown status is normal, the detection result is normal. For example, as follows... Figure 3 As shown, the drift-node or kubelet in the worker node detects the application's running status by calling the application's status query interface. For example, ... Figure 3 As shown, the drift-node in the worker node detects the running status of kubelet by calling the kubelet's status query interface. The above application is an application such as MySQL, and this embodiment of the application does not limit this to such applications.

[0053] When the monitoring target is the connection status between an application and storage resources on a node, the node sends a status detection request to the storage resource at corresponding detection cycles, based on the connection between the application and the storage resource. The node determines the connection status between the application and the storage resource based on the storage resource's response to this status detection request. For example, Figure 3 As shown, the drift-node in the worker node sends status detection requests (such as Transmission Control Protocol (TCP) packets) to the storage device based on the connection status between the application running on the worker node and the storage device, in order to detect the connection status between the application and the storage device.

[0054] In some embodiments, applications on nodes in a container cluster mount PVs via PVCs, enabling the applications to access the PVs. When the storage resource is implemented as a network file system, the node sends a first status check request to the PV based on its IP address. This first status check request may be a data read / write request, etc. If the node does not receive a response from the PV within a timeout, the node determines that the connection between the application and the storage resource is faulty. If the node receives a response from the PV within a preset time, the node determines that the connection between the application and the storage resource is normal. In this process, the network file system is mounted to the application as shared storage, and the process of detecting the connection status between the application and the storage resource uses shared storage to implement I / O plane connectivity checks.

[0055] When the storage resource is implemented as block storage, the node sends a second status check request to the PV (PV) that the application is mounted to. This second status check request is used to query the status of the initiator corresponding to the PV. The initiator is used to establish a connection between the node and the PV so that the node can access the PV. The initiator's status includes online and offline. If the initiator is online, the connection between the node and the PV is normal; if the initiator is offline, the connection between the node and the PV has failed. Based on the second status check request, the PV returns the status information of its corresponding initiator to the node. If a node does not receive status information from the PV within a timeout period, the node determines that the connection between the node and the PV is broken, and correspondingly, the connection between the application and the storage resource is faulty. If the node receives status information from the PV, and the status information indicates that the status of all initiators corresponding to the PV is offline, the node determines that the connection between the node and the PV is faulty, and correspondingly, the connection between the application and the storage resource is faulty. If the node receives status information from the PV, and the status information indicates that the status of any initiator corresponding to the PV is online, the node determines that the connection between the node and the PV is normal, and correspondingly, the connection between the application and the storage resource is normal. The above process of detecting the connection status between the application and the storage resource is a method of determining the connectivity between the node and the storage resource by checking the initiator status.

[0056] In some embodiments, nodes in a container cluster generate at least one monitor on each node according to a drift monitoring strategy; that is, at least one periodically executed task is established on each node. Each monitor corresponds to at least one monitored object in the drift monitoring strategy and is used to periodically detect that monitored object. For example, the monitored objects in the drift monitoring strategy include the usage of local storage space on the node, the consumption of memory on the node, the consumption of computing resources on the node, the running status of applications on the node, and the connection status between applications and storage resources on the node, such as... Figure 5 As shown, based on this drift monitoring strategy, three monitors are generated on the nodes: a first monitor (node ​​monitor), a second monitor (pod monitor), and a third monitor (storage monitor). The first monitor is used to detect the connection status of the network within the container cluster, the usage of the node's local storage space, the consumption of the node's memory, and the consumption of the node's computing resources. The second monitor is used to detect the running status of applications on the node, and the third monitor is used to detect the connection status between applications and storage resources on the node.

[0057] The above process describes how nodes within a container cluster perform tasks such as internal heartbeat detection based on a drift monitoring strategy to obtain the corresponding detection results. The drift monitoring strategy used in this process can be customized by the user based on different faults, thereby improving monitoring flexibility and meeting the user's flexible monitoring needs. In some embodiments, this drift monitoring strategy is carried out through a user-defined resource (CRD) or a static configuration file; this application does not limit this approach.

[0058] 402. Each node in the container cluster updates the corresponding runtime information of that node in the container cluster's storage system based on the detection results of that node.

[0059] The operational information corresponding to a node can include different content. For example, the operational information corresponding to a node may include the detection results of multiple monitored objects corresponding to that node. Accordingly, the operational information of a node includes at least one of the following: the node's local storage space usage, the node's memory consumption, the node's computing resource consumption, the running status of the application on the node, and the connection status between the application and the storage resources. As another example, the operational information corresponding to a node may include a timestamp corresponding to each of the aforementioned monitored objects. This timestamp indicates the time when the storage system last received a normal operation message corresponding to that monitored object, and this normal operation message indicates that the detection result of that monitored object is normal. Accordingly, for the different content included in the above operational information, each node in the container cluster updates the operational information corresponding to that node in the container cluster's storage system in different ways based on the detection results of that node. These methods are illustrated below.

[0060] In some embodiments, the runtime information corresponding to a node includes the detection results for each of the multiple monitored objects. Accordingly, each node in the container cluster, for each monitored object indicated by its corresponding drift monitoring strategy, performs detection on the monitored object every corresponding detection period according to the drift monitoring strategy, obtains the detection result for that monitored object, and sends the detection result to the storage system in the container cluster. After receiving the detection result for that monitored object, the storage system of the container cluster updates the runtime information of that node with the detection result for that monitored object.

[0061] In some embodiments, the operational information corresponding to a node includes a timestamp for each of the multiple monitored objects, indicating the time when the normal operation message corresponding to that monitored object was last received. Accordingly, for each monitored object in the corresponding drift monitoring policy, if the detection result for that monitored object is normal, the node sends a normal operation message to the storage system of the container cluster. This normal operation message carries the identifier of the monitored object and the identifier of the node, so that the storage system of the container cluster can update the timestamp corresponding to the monitored object after receiving the normal operation message, that is, update the timestamp to the time when the normal operation message was received. If the detection result for that monitored object is a fault, the node does not send the aforementioned normal operation message to the storage system of the container cluster.

[0062] In some embodiments, multiple monitors run on a node. One monitor on the node is used to detect multiple monitored objects. The node's operational information includes a timestamp corresponding to each of the multiple monitors, indicating the time when the normal operation message corresponding to that monitor was last received. If the detection results of the multiple monitored objects are all normal, the node sends a normal operation message to the container cluster's storage system. This normal operation message carries the identifier of the monitor and the identifier of the node, so that the container cluster's storage system can update the timestamp corresponding to the monitor after receiving the normal operation message. If the detection result of any of the multiple monitored objects is a fault, the node does not send the aforementioned normal operation message to the container cluster's storage system.

[0063] In some embodiments, the monitored object is the connection status between the node and the storage system of the container cluster (i.e., the connection status of the internal network of the container cluster). The node sends heartbeat information to the storage system of the container cluster every corresponding detection period. Each time the storage system of the container cluster receives a heartbeat message, it updates the timestamp corresponding to the monitored object to the time the heartbeat message was received.

[0064] Related technologies generally formulate drift monitoring strategies for the entire container cluster, while this application formulates drift monitoring strategies for a single node in the container cluster, which refines the granularity of detection and thus improves the sensitivity and flexibility of container cluster operation status detection.

[0065] The following is combined with Figure 6 The method for detecting the running status of a container cluster provided in the embodiments of this application will be described. Figure 6 This is a flowchart illustrating a method for detecting the running status of a container cluster, provided in an embodiment of this application. This method is applied to nodes within a container cluster. Figure 6 As shown, the method includes the following steps.

[0066] 601. For each drift condition in the container cluster, the first node in the container cluster determines the container unit corresponding to the drift condition at preset intervals. Each node in the container cluster runs at least one container unit, which is used to run applications. The drift condition indicates the condition that triggers the drift of the container unit where the application is located.

[0067] The drift condition includes the number of tolerances for each monitored object among multiple monitored objects. Each drift condition corresponds to one or more container units. When a drift condition corresponds to multiple container units, these multiple container units run on the same or different nodes in the container cluster; this embodiment does not limit this. The first node mentioned above can be any node in the container cluster; this embodiment does not limit this. Data such as the drift conditions of the container cluster, the correspondence between drift conditions and container units, the correspondence between container units and nodes, and the running information of nodes in the container cluster are stored in the container cluster's storage system. Each node in the container cluster retrieves data such as drift conditions from the storage system by calling the corresponding API. The drift condition corresponding to each container unit is pre-set. By associating the drift condition with the container unit (the application running on the container unit) in the container cluster's storage system, the first node can retrieve the container unit corresponding to each drift condition from the storage system. This association between the drift condition and the container unit can be achieved through a label, i.e., setting a label for the container unit, which indicates the drift condition corresponding to that container unit. The drift conditions of the container cluster can be set by the user, and the tolerance count can be flexibly set for different monitoring objects corresponding to the application, so as to flexibly set different drift conditions for different applications and improve the flexibility of runtime status detection. In some embodiments, the drift conditions are carried out through CRD or static configuration files, which is not limited in this application. The application running on any container unit is a complete application or a part of a complete application, such as a microservice application, which is not limited in this application.

[0068] In this embodiment of the application, for each drift condition of the container cluster, the first node in the container cluster obtains the drift condition from the storage system of the container cluster by calling the API corresponding to the storage system of the container cluster every preset time interval, and determines the container unit corresponding to the drift condition in the container cluster according to the correspondence between the drift condition and the container unit in the storage system.

[0069] 602. For each identified container unit, the first node in the container cluster determines the node where the container unit is located, i.e., the second node.

[0070] The node where the aforementioned container unit is located can be any node in the container cluster. That is, the node can be the first node in the container cluster or any other node in the container cluster other than the first node. This application embodiment does not limit this.

[0071] In this embodiment of the application, for each determined container unit, the first node in the container cluster obtains the correspondence between the container unit and the node in the container cluster from the storage system of the container cluster by calling the API corresponding to the container cluster storage system, and obtains the node where the container unit is located, i.e., the second node.

[0072] 603. The first node in the container cluster obtains the running information corresponding to the second node. This running information indicates information related to the operation of the container unit on the second node, including the running information corresponding to each monitored object in the drift condition.

[0073] In this embodiment, the first node in the container cluster obtains the running information of the second node from the container cluster's storage system by calling the API corresponding to the container cluster's storage system.

[0074] The above description uses the example of storing data such as the drift conditions of the container cluster, the correspondence between drift conditions and container units, the correspondence between container units and nodes, and the operational information of each node in the container cluster in the container cluster's storage system. In some embodiments, the above data is stored in the first node. Accordingly, for each drift condition of the container cluster, the first node in the container cluster determines the container unit corresponding to the drift condition from the locally stored data at preset intervals. For each determined container unit, the first node obtains the correspondence between the container unit and the nodes in the container cluster from the locally stored data to obtain the node where the container unit is located, i.e., the second node, and then obtains the operational information corresponding to the second node from the locally stored data.

[0075] Steps 601 to 603 described above represent one possible implementation of the first node in a container cluster obtaining runtime information of nodes within the cluster. In this process, the first node polls the container cluster for drift conditions. For each drift condition, it retrieves the container unit associated with that condition and sequentially obtains the node (i.e., the second node) containing each container unit and its corresponding runtime information. This allows the first node to determine whether to drift the container unit based on the runtime information of the second node. Furthermore, this process obtains the periodically updated runtime information of each node in the container cluster, ensuring that the runtime status of container units on the node is detected based on the latest runtime information, thus improving the accuracy of the detection.

[0076] 604. For each monitored object in the drift condition, if the running information corresponding to the monitored object does not match the tolerance condition corresponding to the monitored object, the first node in the container cluster increments the failure count corresponding to the monitored object by 1. The failure count indicates the number of times the second node has failed.

[0077] The tolerance conditions corresponding to the monitored object are related to the operational information of the monitored object. As explained above, in some embodiments, the operational information corresponding to the monitored object is the detection result of the monitored object, while in other embodiments, the operational information corresponding to the monitored object is the timestamp of the monitored object. Different operational information corresponds to different tolerance conditions. The above steps will be explained below for different types of operational information.

[0078] In some embodiments, when the operational information corresponding to a monitored object is the detection result corresponding to that monitored object, the tolerance condition corresponding to that monitored object is the ideal result corresponding to that monitored object. For each monitored object in the drift condition, if the operational information corresponding to that monitored object does not match the ideal result corresponding to that monitored object, the first node in the container cluster increments the failure count corresponding to that monitored object by 1. Taking the monitored object as the local storage space usage of a node as an example, the operational information corresponding to that monitored object is the detected local storage space usage of the node, and the tolerance condition corresponding to that monitored object is the maximum value of the local storage space usage of the node. If the usage indicated by the operational information corresponding to that monitored object is greater than the maximum usage indicated by the tolerance condition corresponding to that monitored object, then the operational information corresponding to that monitored object does not match the tolerance condition corresponding to that monitored object, and the first node in the container cluster increments the failure count corresponding to that monitored object by 1. For example, in the operational information of the second node, the monitored object is the local storage space usage of the node, and the detection result corresponding to that monitored object is 11 terabytes (TB). The tolerance condition corresponding to that monitored object is less than 10 TB, then the first node increments the failure count corresponding to that monitored object by 1.

[0079] When the monitored object is the percentage of local storage space occupied by a node, the amount of memory consumed by a node, or the amount of computing resources consumed by a node, the way the first node updates the number of faults corresponding to the monitored object is the same as the way the first node updates the number of faults corresponding to the local storage space occupied by the node, and will not be described again in this embodiment.

[0080] Taking the monitoring object as an example of the connection status of the internal network of the container cluster, the corresponding operational information is the detected connection status of the internal network of the container cluster, and the tolerance condition for the monitoring object is that the connection status of the internal network of the container cluster is normal. If the connection status indicated by the operational information corresponding to the monitoring object is faulty, then the operational information corresponding to the monitoring object does not meet the tolerance condition corresponding to the monitoring object, and the first node in the container cluster increments the fault count corresponding to the monitoring object by 1. When the monitoring object is the operational status of the application on the node or the connection status between the application on the node and the storage resources, the way the first node updates the fault count corresponding to the monitoring object is the same as the way the first node updates the fault count corresponding to the connection status of the internal network of the container cluster, and will not be described again in this embodiment.

[0081] In some embodiments, when the runtime information corresponding to a monitored object is a timestamp corresponding to that monitored object, the tolerance condition corresponding to that monitored object is a tolerance duration. For each monitored object in the drift condition, the first node in the container cluster obtains the unupdated duration of the timestamp corresponding to that monitored object. If the unupdated duration is greater than or equal to the tolerance duration corresponding to that monitored object, the failure count corresponding to that monitored object is incremented by 1. Accordingly, for each monitored object in the drift condition, the first node in the container cluster obtains the timestamp corresponding to that monitored object, subtracts the current time from the timestamp, and obtains the unupdated duration of the timestamp. The first node compares the unupdated duration of the timestamp corresponding to that monitored object with the tolerance duration corresponding to that monitored object. If the unupdated duration is greater than or equal to the tolerance duration corresponding to that monitored object, the failure count corresponding to that monitored object is incremented by 1.

[0082] In this implementation, the timestamp corresponding to the monitored object is the target timestamp. The above process is a possible way for the first node to update the fault count of the monitored object if the target timestamp has not been updated within a preset time period. This implementation can promptly detect the fault status of the monitored object by observing the unupdated target timestamp, resulting in high sensitivity in fault detection. The target timestamp indicates the last update time of any item in the node's operational information. In the above possible implementation, any item in the node's operational information is the timestamp corresponding to the monitored object, and this timestamp itself records the last update time of that information. In some embodiments, the node's operational information includes the detection results corresponding to each of multiple monitored objects. Each item in the node's operational information is the detection result corresponding to the monitored object, and this detection result has a timestamp indicating the last update time of the detection result. The tolerance condition corresponding to the monitored object is a tolerance duration. Accordingly, for each monitored object in the drift condition, the first node in the container cluster obtains the unupdated duration of the timestamp corresponding to the detection result of the monitored object. If the unupdated duration is greater than or equal to the tolerance duration corresponding to the monitored object, the failure count corresponding to the monitored object is incremented by 1.

[0083] In some embodiments, the tolerance conditions corresponding to the monitored object are included in the drift conditions, and this application embodiment does not limit this.

[0084] The above process is a possible implementation for detecting the operating status of at least one container unit based on the node's operating information and the drift conditions of the application running at least one container unit. In this possible implementation, the node's operating information and the drift conditions of the application running the container unit can be flexibly set, which improves the flexibility of detecting the container unit.

[0085] 605. If the number of failures corresponding to the monitored object reaches the tolerance limit for the monitored object in the drift condition, the first node in the container cluster will trigger the drift of the corresponding container unit running on the second node.

[0086] In this embodiment, each time the first node in the container cluster updates the number of failures corresponding to the monitored object, it compares the number of failures corresponding to the monitored object with the tolerance count corresponding to the monitored object in the drift conditions. If the number of failures corresponding to the monitored object is greater than or equal to the tolerance count corresponding to the monitored object in the drift monitoring information, it triggers drifting of the corresponding container unit running on the second node. Alternatively, the first node in the container cluster polls the corresponding monitored object of the second node every preset time interval, based on multiple monitored objects in the drift conditions. If the polled number of failures is greater than or equal to the tolerance count corresponding to the drift conditions, it triggers drifting of the corresponding container unit running on the second node.

[0087] The above process describes one possible implementation for drifting a container unit running on a node if the node's operational information matches the drift conditions of the application it runs. Specifically, if the node's operational information indicates that the number of failures has reached the tolerance limit specified in the drift conditions of the application, drifting is triggered. This process, by comparing the drift conditions with the corresponding node's operational information, determines that an anomaly has occurred and immediately drifts the container unit, thus shortening the time consumed by anomaly handling.

[0088] In some embodiments, when the number of failures corresponding to a monitored object reaches the tolerance limit for that monitored object under the drift condition, the first node in the container cluster establishes a drift task during the process of triggering the drift of the corresponding container unit running on the second node, and then polls and executes the established drift tasks. This drift task is used to drift the corresponding container unit running on the second node, that is, to delete the container unit on the node where the container unit is currently located and rebuild the container unit on another node in the container cluster. In some embodiments, the first node establishes a drift task in the storage system by calling the API corresponding to the storage system in the container cluster. The third node in the container cluster polls the multiple drift tasks already established in the storage system, and when a drift task is polled, it executes the drift task to drift the corresponding container unit running on the second node. The third node can be any node in the container cluster, such as the management node, the first node, or the second node, etc., and this embodiment does not limit this.

[0089] In some embodiments, the drift task includes the identifier of the container unit to be drifted, the identifier of the node where the container unit is located, and the reason for drifting.

[0090] In some embodiments, during the migration of a corresponding container unit running on the second node, taking the third node in the container cluster as an example, the third node determines the VA corresponding to the container unit based on the container unit's identifier, the identifier of the PV bound to the container unit, and the PVC corresponding to the container unit. It then deletes the correspondence between the container unit and the VA in the container cluster's storage system, deletes the container unit and related data on the second node, and reconstructs the container unit on nodes other than the second node in the container cluster. Here, VA stands for volumeattachment, used to describe the relationship between nodes and persistent volumes in the container cluster.

[0091] In some embodiments, where the node's operational information includes the detection results corresponding to each of multiple monitored objects, during the reconstruction of the container unit, for each candidate node in the container cluster, a third node in the container cluster obtains the operational information corresponding to that candidate node. If the operational information matches the operational requirements of the container unit, the third node reconstructs the container unit on that candidate node. If the operational information does not match the operational requirements of the container unit, the third node obtains the operational information of the next candidate node. Here, "the operational information of the candidate node matches the operational requirements of the container unit" means that for each monitored object in the operational information, the detection result matches the ideal result required by the container unit. For example, if the monitored object in the candidate node's operational information is the node's local storage space usage, and the detection result corresponding to that monitored object is 9TB, and the ideal result required by the container unit is that the node's local storage space usage is less than 10TB, then the candidate node's operational information matches the operational requirements of the container unit.

[0092] In some embodiments, a third node in the container cluster includes a fail-over executor that performs the operation of drifting the aforementioned container units.

[0093] In some embodiments, in the event of a second node failure or a connection failure between the second node and the first node, the second node, after recovering from the failure, cleans up the aforementioned container units and related data on the second node. The container unit-related data includes the mapping relationship between the container units and storage resources, as well as the drive letter corresponding to the storage resource on the node. For example, the second node includes a node cleaner, which is used to delete any remaining data related to the aforementioned container units on the node after the node failure is recovered.

[0094] In some embodiments, in the event of a second node failure or a connection failure between the second and first nodes, there may be a situation where the application corresponding to the container unit running on the second node remains connected to the storage resource. That is, the application on the failed second node may still be able to access the storage resource through the connection between the application and the storage resource, leading to data corruption in the storage resource. To address this issue, the first node in the container cluster, in response to the failure count of the monitored object reaching the tolerance count corresponding to the monitored object in the drift condition, sends an input / output (IO) isolation request to the storage resource. Upon receiving the IO isolation request, the storage resource stops executing access requests from the aforementioned application.

[0095] In some embodiments, a third node in the container cluster records the number of consecutive drifts of the container unit. In response to the number of consecutive drifts of the container unit being greater than or equal to a preset number, the third node sends a fault message to the client device connected to the container cluster. The fault message indicates that the container cluster has failed and cannot run the container unit.

[0096] The container cluster operation status detection method provided in this application, after obtaining the operation information of nodes in the container cluster, detects the operation status of container units based on the node operation information and the drift conditions of the applications running on the container units on the nodes. Since the operation information indicates information related to the operation of the container unit, and the drift conditions indicate the conditions that trigger the drift of the container unit containing the application, the detection process is clearly performed at the granularity of a single container unit on a node. This ensures the normal operation of the application on that single container unit. Compared with detection at the node level, this refines the granularity of detection, thereby improving the sensitivity of container cluster operation status detection. It avoids situations where the container cluster cannot detect abnormal operation of a single container unit in a timely manner, enabling the container cluster to drift the container unit promptly and allowing the application on that container unit to resume operation as soon as possible, thus shortening the business downtime caused by the abnormal operation of the container unit. Furthermore, this method does not require modification of the application and container load, has low cost, and does not require creating a large number of replicas for container units in the container cluster, reducing the occupation of container cluster storage resources and reducing data consumption costs.

[0097] In related technologies, when a Kubernetes cluster accesses storage resources in read-write-execute (RWX) mode, the time from the occurrence of an exception on a node in the container cluster to the node being marked as tainted is approximately 40 seconds. The default tolerance time (default wait time) for the container unit is approximately 300 seconds, and the application running in the container unit also takes some time to restart, resulting in an overall exception recovery time greater than 340 seconds. When a Kubernetes cluster accesses storage resources in read-write-once (RWO) mode, the time from the occurrence of an exception on a node in the container cluster to the node being marked as tainted is approximately 40 seconds. The default tolerance time (default wait time) for the container unit is approximately 300 seconds, the time for the PVC to be unloaded is 300 seconds, and the application running in the container unit also takes some time to restart, resulting in an overall exception recovery time greater than 640 seconds.

[0098] In addition to the technologies mentioned above, other technologies employ a multi-replica mechanism to ensure the reliability of container clusters. This multi-replica mechanism involves creating multiple container units for an application on different nodes of the container cluster. These multiple container units include a master container unit and multiple slave container units. The master container unit provides background services for the application. Simultaneously, the master container unit sends heartbeat information to the multiple slave container units on other nodes, allowing the slave container units to determine that the master container unit is functioning normally based on the received heartbeat information. If the multiple slave container units do not receive a heartbeat information within a timeout period, they determine that the master container unit has malfunctioned. The multiple slave container units then elect a new master container unit through an election mechanism, which then provides background services for the application. This technology requires deploying multiple container unit replicas on different nodes, consuming significant storage resources of the container cluster. Furthermore, this technology requires modifications to the application to enable it to perform the heartbeat information sending and election mechanism operations, resulting in high production costs.

[0099] The container cluster runtime status detection method provided in this application uses drift fault tolerance instead of the aforementioned replica switching technology. When a node malfunctions, it improves drift efficiency and performance through rapid detection and application-based rapid drift, reducing drift time from more than 340 seconds to 30 seconds. This achieves platform-based application-level rapid drift capability, reducing business downtime and further reducing the number of replicas without compromising reliability, thereby reducing production costs. Furthermore, this method can be applied to a wide range of fault scenarios, such as internal network failures within the container cluster (management network failures), server power outages, server restarts, kubelet failures, connection interruptions between applications and storage resources (e.g., switch failures), and application service failures. This method targets applications by detecting the internal network of the management cluster, the application's runtime status, and the connection between the application and storage resources to determine if the application meets the drift conditions, thereby quickly restarting the application on other nodes in the container cluster, improving the reliability of applications within the container cluster.

[0100] The following example illustrates the application of the container cluster runtime status detection method provided in this application to a Kubernetes cluster. Figure 7 This is an architecture diagram of a Kubernetes cluster provided in an embodiment of this application, such as... Figure 7 As shown, the container cluster includes a management node and multiple nodes. The management node includes a storage system (etcd) that stores data such as the drift policy, the mapping between drift policies and container units (pods), the mapping between pods and nodes, the drift monitoring policy for each node, the runtime information of each node in the container cluster (driftmonitor), and established drift jobs. The management node also includes a kube-api-server, which nodes in the container cluster access data in the storage system by calling the kube-api-server.

[0101] The container cluster runtime status detection method provided in this application embodiment is deployed in the form of containers on the nodes of the container cluster. Each node of the container cluster has a drift-node deployed, and any node of the container cluster has a drift-controller deployed. Figure 7As shown, the `drift-node` is an application that provides node detection services. It detects nodes according to the drift monitoring policy corresponding to the node where the `drift-node` resides, and periodically calls the `kube-api-server` to update the running information of the node in the storage system based on the detection results, thus updating the node's running status in a timely manner. The `drift-node` supports detecting at least one of the following: the container cluster's internal network (container cluster management network), the kubelet service, the connection status between the application and storage resources (storage service I / O), and the running status of the application running in the pod. In some embodiments, such as... Figure 7 As shown, this drift-node uses the above... Figure 5 The multiple monitors shown (node ​​monitor, pod monitor, and storage monitor) periodically detect the internal network of the container cluster corresponding to the node, the running status of kubelet, and the connection status between the application and storage resources. This application embodiment does not limit this.

[0102] The aforementioned drift-controller is an application that provides pod runtime status monitoring. It detects the runtime status of pods based on node runtime information and the drift conditions corresponding to the pods on those nodes. If an anomaly is detected, the pod is deleted from its current node and rebuilt on another node in the container cluster. During this process, the drift-controller polls the container cluster for drift conditions at preset intervals by calling the kube-api-server, retrieving the pod, the node containing the pod, and the runtime information of that node from the storage system for each drift condition. The drift-controller compares the runtime information of the node with the drift conditions. If the runtime information of the node meets the drift conditions, the drift-controller creates a drift task, which includes the pod's identifier, the identifier of the node containing the pod, and the reason for the drift. For each drift task, the drift controller determines the PVC and PV corresponding to the pod based on the pod identifier in the drift task, and then determines the VA corresponding to the pod based on the pod identifier, the PVC and PV corresponding to the pod. The mapping relationship between the pod and the VA is deleted in the storage system, the pod is deleted on the node, and the pod is rebuilt on another node in the container cluster.

[0103] In some embodiments, drift-node is deployed as a resource object (daemonset) on each node in the container cluster, and drift-controller is deployed as a daemonset on any node in the container cluster. This application embodiment does not limit this.

[0104] The methods of the embodiments of this application have been described above; the apparatus of the embodiments of this application will be described below. It should be understood that the apparatus described below has any of the functions of the computing device in the above methods. (The above is in conjunction with...) Figures 1 to 7 The method for detecting the running status of a container cluster according to embodiments of this application is described in detail. Based on the same inventive concept, the following will be combined with... Figure 8 This application describes a container cluster operation status detection apparatus according to embodiments thereof. It should be understood that the technical features described in the method embodiments are also applicable to the following apparatus embodiments.

[0105] See Figure 8 This application provides a container cluster runtime status detection device, which includes:

[0106] The acquisition module 801 is used to acquire the running information of nodes in the container cluster. At least one container unit runs on the node. The container unit is used to run applications. The running information indicates information related to the running of at least one container unit on the node.

[0107] The detection module 802 is used to detect the running status of at least one container unit based on the running information of the node and the drift conditions of the application running in at least one container unit, wherein the drift conditions indicate the conditions that trigger the drift of the container unit where the application is located.

[0108] In some embodiments, the above-described apparatus further includes:

[0109] The drift module is used to trigger the drift of each container unit running on a node if the node's running information matches the drift conditions of the application running the container unit.

[0110] In some embodiments, the node's operational information includes at least one of the following: the node's local storage space usage, the node's memory consumption, the node's computing resource consumption, the application's running status, and the connection status between the application and storage resources.

[0111] In some embodiments, the acquisition module 801 is used for:

[0112] For each node in the container cluster, obtain the periodically updated runtime information of that node.

[0113] In some embodiments, the drift conditions include the number of tolerances, and the drift module is used to:

[0114] If the node's operational information indicates that the number of node failures has reached the tolerance limit in the drift conditions of the application running the container cell, it will trigger a drift operation on the container cell. The node's failure count indicates the number of times the node has failed.

[0115] In some embodiments, the above-described apparatus further includes:

[0116] The update module is used to update the number of node failures if the target timestamp has not been updated within a preset time period. The target timestamp indicates the time when any item in the node's operation information was last updated.

[0117] In some embodiments, the above-described apparatus further includes:

[0118] The sending module is used to send fault information to the client device connected to the container cluster in response to the number of consecutive drifts of the container unit being greater than or equal to a preset number. The fault information indicates that the container cluster has failed.

[0119] In some embodiments, the process of obtaining the connection status between the application and the storage resource includes:

[0120] The node where the application resides sends a status check request to the storage resource through the connection between the application and the storage resource;

[0121] The connection status between the application and the storage resource is determined based on the storage resource's response to the status detection request.

[0122] It should be understood that the container cluster running status detection device corresponds to the node in the above method embodiment. The modules in the device and the other operations and / or functions described above are for implementing various steps and methods implemented by the node in the method embodiment. For specific details, please refer to the above method embodiment. For the sake of brevity, they will not be repeated here.

[0123] The nodes in the aforementioned container cluster can be implemented as follows: Figure 9 The computing device shown Figure 9 This is a schematic diagram of a computing device provided in an embodiment of this application. It should be understood that the computing device described below can implement any function of any of the methods described below. Typically, the computing device 900 includes a processor 901 and a memory 902.

[0124] Processor 901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from digital signal processing (DSP), field-programmable gate array (FPG A), and programmable logic array (PLA). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as the central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an artificial intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0125] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one program segment, which is executed by the processor 901 to implement the container cluster runtime status detection method provided in the method embodiments of this application.

[0126] In some embodiments, the computing device 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal lines, or a circuit board.

[0127] This application also provides a computing device cluster, which includes multiple computing devices, each of which includes a processor and a memory. The processor of the multiple computing devices is used to execute instructions stored in the memory of the multiple computing devices, so that the computing device cluster executes to implement the container cluster running status detection method provided in the method embodiment of this application. This allows the computing device cluster to migrate each container unit running on any computing device in the computing device cluster from that computing device to another computing device in the computing device cluster if the running information of the computing device matches the drift conditions of the application running the container unit.

[0128] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program code that can be executed by a processor in a computing device to complete the container cluster operation status detection method in the above embodiments. For example, the computer-readable storage medium is a non-transitory computer-readable storage medium, such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0129] This application also provides a computer program product or computer program, which includes program code. The computer instructions are stored in a computer-readable storage medium. A processor in a computing device reads the program code from the computer-readable storage medium and executes the program code, causing the computing device to execute the above-described container cluster operation status detection method.

[0130] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the container cluster running status detection method in the above method embodiments.

[0131] In this embodiment, the apparatus, device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0132] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the container cluster operation status detection method embodiment provided in the above embodiments belongs to the same concept, and its specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0137] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0138] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0139] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sensitive words involved in this application were obtained with full authorization.

[0140] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0141] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting the running status of a container cluster, characterized in that, The method includes: Obtain the running information of the nodes in the container cluster, wherein at least one container unit runs on the node and the container unit is used to run an application, and the running information indicates information related to the running of the at least one container unit on the node; Based on the node's operational information and the drift conditions of the application running in the at least one container unit, the operational status of the at least one container unit is detected, wherein the drift conditions indicate the conditions that trigger the drift of the container unit where the application resides.

2. The method according to claim 1, characterized in that, The method further includes: For each container unit running on the node, if the node's running information matches the drift conditions of the application running the container unit, drifting of the container unit is triggered.

3. The method according to claim 1, characterized in that, The node's operational information includes at least one of the following: the node's local storage space usage, the node's memory consumption, the node's computing resource consumption, the application's operational status, and the connection status between the application and storage resources.

4. The method according to claim 1, characterized in that, The process of obtaining the runtime information of the nodes in the container cluster includes: For each node in the container cluster, obtain the periodically updated runtime information of that node.

5. The method according to claim 2, characterized in that, The drift conditions include the number of tolerances, and triggering the drift of the container unit if the node's operating information matches the drift conditions of the application running the container unit includes: If the node's operational information indicates that the number of failures of the node has reached the tolerance limit in the drift conditions of the application running the container unit, drifting of the container unit is triggered. The number of failures of the node indicates the number of times the node has failed.

6. The method according to claim 5, characterized in that, The method further includes: If the target timestamp is not updated within a preset time period, the failure count of the node is updated, wherein the target timestamp indicates the time when any item in the node's operation information was last updated.

7. The method according to claim 1, characterized in that, The method further includes: In response to the number of consecutive drifts of the container unit being greater than or equal to a preset number, a fault message is sent to the client device connected to the container cluster, the fault message indicating that the container cluster has failed.

8. The method according to claim 2, characterized in that, The process of obtaining the connection status between the application and the storage resources includes: The node where the application resides sends a status detection request to the storage resource through the connection between the application and the storage resource; The connection status between the application and the storage resource is determined based on the storage resource's response to the status detection request.

9. A device for detecting the operational status of a container cluster, characterized in that, The device includes: The acquisition module is used to acquire the running information of nodes in the container cluster, wherein at least one container unit is running on the node, the container unit is used to run an application, and the running information indicates information related to the running of the at least one container unit on the node; The detection module is used to detect the operating status of the at least one container unit based on the node's operating information and the drift conditions of the application running in the at least one container unit, wherein the drift conditions indicate the conditions that trigger the drift of the container unit where the application is located.

10. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute at least one piece of program code stored in the memory to enable the computing device to perform the method as described in any one of claims 1 to 8.

11. A computing device cluster, characterized in that, It includes multiple computing devices, each of which includes a processor and memory; The processors of the plurality of computing devices are configured to execute instructions stored in the memory of the plurality of computing devices to cause the computing device cluster to perform the method as described in any one of claims 1 to 8, such that for each container unit running on any computing device in the computing device cluster, the computing device cluster migrates the container unit from the computing device to another computing device in the computing device cluster if the operating information of the computing device matches the drift conditions of the application running the container unit.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one piece of program code, which, when executed by a computing device, causes the computing device to perform the method as described in any one of claims 1 to 8.

13. A computer program product, characterized in that, When the computer program product is run on a computing device, the computing device performs the method as described in any one of claims 1 to 8.

Images