Cluster fault detection method and device, computer device and readable storage medium

Abstract

The application relates to a cluster fault detection method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: when a creation event of a self-defined resource for detecting a cluster fault is monitored, reading configuration information carried by the self-defined resource; determining a to-be-detected node and a to-be-detected container according to the configuration information, and creating a fault detection container; for each to-be-detected node, taking a node namespace of the to-be-detected node as an access object, controlling the fault detection container distributed to the to-be-detected node to access the access object, and controlling a fault detection script to execute in the node namespace to obtain a node fault detection result corresponding to the to-be-detected node, then taking a container namespace of a to-be-detected container deployed in the to-be-detected node as an access object, and controlling the fault detection script to execute in the container namespace to obtain a container fault detection result corresponding to the to-be-detected node. The method can efficiently detect a cluster fault.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cluster fault detection, and in particular to a cluster fault detection method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of computer technology, clusters are currently commonly used to improve the performance, flexibility and scalability of systems, such as Kubernetes clusters. Kubernetes clusters are open source systems for automatically deploying, expanding and managing "containerized applications", and can provide a platform that can automatically deploy, expand and run application containers across a host cluster, thereby improving the performance, flexibility and scalability of systems.

[0003] In the conventional technology, in the case of a failure of a Kubernetes cluster, since the Kubernetes cluster abstracts application running environments and computing resources into a large number of resources such as Nodes (i.e., nodes responsible for executing tasks allocated by the cluster) and Pods (i.e., container sets deployed to a single node and containing one or more containers), and containers are isolated in multiple process spaces, and users generally cannot directly access Nodes or Pods, it is difficult to quickly detect whether each Node or Pod in the Kubernetes cluster has failed, resulting in low efficiency of Kubernetes cluster fault detection. SUMMARY

[0004] Therefore, it is necessary to provide a cluster fault detection method, device, computer equipment, computer readable storage medium and computer program product capable of efficiently detecting cluster faults in view of the above technical problems.

[0005] In a first aspect, the present application provides a cluster fault detection method, comprising:

[0006] monitoring events in the cluster, reading configuration information carried by a custom resource for detecting cluster faults when a creation event of the custom resource is monitored;

[0007] determining to-be-detected nodes and to-be-detected containers in the cluster according to the configuration information, and creating a fault detection container according to the configuration information; the fault detection container has been configured with a fault detection script and has been allocated a to-be-detected node belonging to it;

[0008] for each to-be-detected node, taking a node namespace of the to-be-detected node as an access object, controlling a fault detection container allocated to the to-be-detected node to access the access object, and controlling the fault detection script to execute in the node namespace to obtain a node fault detection result corresponding to the to-be-detected node;

[0009] In the case of obtaining the node fault detection result, the container namespace of the to-be-detected container deployed in the to-be-detected node is taken as an access object, and the fault detection script is controlled to execute in the container namespace, so as to obtain the container fault detection result corresponding to the to-be-detected node.

[0010] In one of the embodiments, the fault detection container is created according to the configuration information, including:

[0011] From the configuration information, the target configuration information used to control the fault detection process is determined, and the workload resource list is assembled according to the target configuration information.

[0012] Based on the workload resource list, the fault detection container is created.

[0013] In one of the embodiments, the workload resource list includes the following target configuration information: quantity configuration information, permission configuration information, node affinity configuration information, and fault detection script.

[0014] Based on the workload resource list, the fault detection container is created, including:

[0015] According to the quantity configuration information, a plurality of initial containers are created.

[0016] For each initial container, the namespace access permission of the initial container is configured according to the permission configuration information, the initial container is assigned to the to-be-detected node according to the node affinity configuration information, and the fault detection script is mounted under the container directory of the initial container, so as to obtain a plurality of fault detection containers.

[0017] In one of the embodiments, the workload resource list further includes the following target configuration information: container lifecycle configuration information.

[0018] The fault detection script is mounted under the container directory of the initial container to obtain a plurality of fault detection containers, including:

[0019] For each initial container, the initial container is configured with a cleaning delay time length according to the container lifecycle configuration information, so as to obtain a plurality of fault detection containers.

[0020] In one of the embodiments, at least one to-be-detected container is deployed in each to-be-detected node; the method further includes:

[0021] For each to-be-detected node, when the container fault detection result of each to-be-detected container in the to-be-detected node has been obtained, the node fault detection result corresponding to the to-be-detected node and the at least one container fault detection result are written into the state information associated with the custom resource.

[0022] determining a cleaning delay duration of the fault detection container allocated to the to-be-detected node, and cleaning the fault detection container allocated to the to-be-detected node after waiting for the cleaning delay duration.

[0023] In one of the embodiments, the cluster fault detection method further comprises:

[0024] receiving a cluster access authentication request triggered by the requestor, and verifying authentication information carried in the cluster access authentication request;

[0025] when the authentication information passes the verification, receiving and executing a cluster detection result reading instruction triggered by the requestor for the state information, and feeding back node fault detection results and container fault detection results stored in the state information.

[0026] In a second aspect, the present application further provides a cluster fault detection device, comprising:

[0027] an event monitoring module, configured to monitor events in the cluster, read configuration information carried by a self-defined resource for detecting cluster faults when a creation event of the self-defined resource is monitored;

[0028] a detection container creation module, configured to determine to-be-detected nodes and to-be-detected containers in the cluster according to the configuration information, and create a fault detection container according to the configuration information; the fault detection container is configured with a fault detection script and is allocated with a to-be-detected node belonging to the fault detection container;

[0029] a node detection module, configured to, for each to-be-detected node, take a node namespace of the to-be-detected node as an access object, control the fault detection container allocated to the to-be-detected node to access the access object, and control the fault detection script to execute in the node namespace, so as to obtain a node fault detection result corresponding to the to-be-detected node;

[0030] a container detection module, configured to, in a case where the node fault detection result is obtained, take a container namespace of a to-be-detected container deployed in the to-be-detected node as an access object, and control the fault detection script to execute in the container namespace, so as to obtain a container fault detection result corresponding to the to-be-detected node.

[0031] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above fault detection method when executing the computer program.

[0032] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above fault detection method.

[0033] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the above fault detection method.

[0034] The cluster fault detection method, device, computer device, computer readable storage medium and computer program product can first monitor events in the cluster, read configuration information carried by the custom resource when a creation event of the custom resource for detecting cluster faults is monitored, determine the to-be-detected nodes and the to-be-detected containers in the cluster according to the configuration information, and create a fault detection container according to the configuration information. The fault detection container is configured with a fault detection script and is assigned to the to-be-detected nodes, so that subsequent detection of the to-be-detected nodes and the to-be-detected containers in the cluster can be automatically performed through the fault detection container, and the cluster fault detection efficiency is improved. Specifically, for each to-be-detected node, the node namespace of the to-be-detected node can be taken as an access object, the fault detection container assigned to the to-be-detected node is controlled to access the access object, and the fault detection script is controlled to execute in the node namespace to obtain a node fault detection result corresponding to the to-be-detected node. Further, in the case where the node fault detection result is obtained, the container namespace of the to-be-detected container deployed in the to-be-detected node can be taken as an access object, and the fault detection script is controlled to execute in the container namespace to obtain a container fault detection result corresponding to the to-be-detected node. Based on this, by using the above cluster fault detection method, the user does not need to perform fault detection on a large number of nodes or containers in the cluster one by one, but can automatically perform detection on the corresponding to-be-detected nodes and the to-be-detected containers deployed on the to-be-detected nodes one by one through the fault detection container assigned to each to-be-detected node, thereby achieving efficient detection of cluster faults. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0036] Figure 1 An application environment diagram of the cluster fault detection method in an embodiment;

[0037] Figure 2 A flowchart of the cluster fault detection method in an embodiment;

[0038] Figure 3 A flowchart of the Kubernetes cluster fault detection method in an embodiment;

[0039] Figure 4 This is a structural block diagram of a cluster fault detection device in one embodiment;

[0040] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] The cluster fault detection method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the user can use terminal 102 to communicate with server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. A cluster, such as a Kubernetes cluster, can be deployed on server 104.

[0043] Taking the cluster fault detection method provided in this application embodiment as an example of applying it to a Kubernetes cluster, the Kubernetes cluster may specifically include: nodes, pods deployed on the nodes, services, and custom resource controllers (CR controllers).

[0044] A node is a machine that hosts a Kubernetes cluster (which can be considered a host). All business applications in a Kubernetes cluster run on nodes.

[0045] A container set is the smallest scheduling and management unit in a Kubernetes cluster. Each container set contains one or more containers. Containers can decouple applications from the underlying host facilities through the process-level isolation capabilities provided by the operating system, enabling the running of multiple independent applications on the host. This has the advantages of faster application startup and less resource usage.

[0046] Services, which can expose applications running on a container set as network services. Taking the API Server in the service contained in the Kubernetes cluster as an example, a user can communicate with the API Server and the Kubernetes cluster through the network, but it is difficult for the user to directly log in to a node or a container to execute a command to troubleshoot a fault. The API Server can provide interfaces for adding, deleting, modifying, querying, and monitoring various resource objects (including nodes, containers, custom resources, services, and the like) in the Kubernetes cluster.

[0047] Custom resource controllers, which can be used to monitor the creation events of custom resources (CRs) in a Kubernetes cluster, and monitor and control the entire life cycle of the custom resources in the Kubernetes cluster. Custom resources are an extension of applications in the Kubernetes cluster, and the custom resources can appear or disappear in the cluster through dynamic registration.

[0048] Specifically, a user can use the terminal 102 to send an instruction to the server 104 through the API Server to create a custom resource CR in the Kubernetes cluster, where the custom resource CR is used to detect the failure of the Kubernetes cluster. Further, the custom resource controller CR Controller in the Kubernetes cluster can monitor events in the Kubernetes cluster through the API Server. When the creation event of the custom resource CR is monitored, the custom resource controller CR Controller can read the configuration information carried by the custom resource CR, and determine the to-be-detected node and the to-be-detected container in the Kubernetes cluster according to the configuration information, so as to create a failure detection container according to the configuration information. Wherein, the failure detection container has been configured with a failure detection script, and has been allocated with the to-be-detected node. Further, for each to-be-detected node, the custom resource controller CR Controller can first take the node namespace of the to-be-detected node as an access object, control the failure detection container allocated to the to-be-detected node to access the access object, and control the failure detection script to execute in the node namespace to obtain the node failure detection result corresponding to the to-be-detected node. In the case where the node failure detection result is obtained, the custom resource controller CR Controller can take the container namespace of the to-be-detected container deployed in the to-be-detected node as an access object, and control the failure detection script to execute in the container namespace to obtain the container failure detection result corresponding to the to-be-detected node. Finally, when the custom resource controller CR Controller monitors that the failure detection container on a to-be-detected node has worked, the custom resource controller CR Controller can store the node failure detection result and the container failure detection result corresponding to the to-be-detected node to a specified location for subsequent feedback to the user.

[0049] In one exemplary embodiment, as shown in Figure 2 A cluster failure detection method is provided, which is applied to the custom resource controller CR Controller in the Kubernetes cluster as an example for illustration, including the following steps 202 to 208. Wherein:

[0050] Step 202, monitor events in the cluster, and when the creation event of the custom resource for detecting the failure of the cluster is monitored, read the configuration information carried by the custom resource.

[0051] Optionally, during the running of the Kubernetes cluster, the API Server in the Kubernetes cluster is a tool for users to interact with the Kubernetes cluster, and therefore, the custom resource controller CR Controller can monitor events in the Kubernetes cluster in real time through the APIServer to monitor whether the user creates a custom resource in the cluster. When the creation event of the custom resource for detecting cluster failure is monitored, the custom resource controller CR Controller can automatically read the configuration information carried by the custom resource CR.

[0052] In an embodiment, the custom resource controller CR Controller is deployed into the Kubernetes cluster in the form of Deployment. Specifically, the form of Deployment refers to a stateless load form to ensure high availability, flexibility and operation and maintenance simplification of the custom resource controller CR Controller.

[0053] For example, the user can package the custom resource into a container image and interact with the Kubernetes cluster through the API Server in the Kubernetes cluster to issue the custom resource to the Kubernetes cluster to create the custom resource in the Kubernetes cluster. The file system used to provide an isolated execution environment for the container process is the container image. In this embodiment, the configuration information of the custom resource can be configured with the following contents: a detection script to be executed, nodes and containers to be detected, and a failure detection container creation rule, thereby providing an execution environment for cluster fault diagnosis, so that the created failure detection container can be used to complete the fault detection of multiple containers existing in isolation one by one and automatically.

[0054] In step 204, the nodes to be detected and the containers to be detected in the cluster are determined according to the configuration information, and a failure detection container is created according to the configuration information. The failure detection container has been configured with a fault detection script and has been allocated with the node to be detected.

[0055] Each of the nodes to be detected is allocated with at least one failure detection container.

[0056] Optionally, the custom resource controller CR Controller can determine the nodes to be detected and the containers to be detected in the Kubernetes cluster according to the read configuration information, and create a failure detection container according to the failure detection container creation rule carried in the configuration information.

[0057] Exemplarily, the custom resource controller CR Controller can create a fault detection Job containing a fault detection container creation rule according to the read configuration information, so as to create a certain number of fault detection containers according to the fault detection Job. Wherein, the Job is one of the resource objects of the cluster and can be used to execute a one-time task, and the number of fault detection containers to be created can be specified in the Job, and the created fault detection containers are stopped after completing the task.

[0058] In step 206, for each to-be-detected node, the node namespace of the to-be-detected node is taken as an access object, the fault detection container allocated to the to-be-detected node is controlled to access the access object, and the fault detection script is controlled to execute in the node namespace, so as to obtain the node fault detection result corresponding to the to-be-detected node.

[0059] Wherein, the namespace is a mechanism for organizing and isolating codes, which allows different objects with the same name in the same environment to not conflict. Different resource objects (nodes or containers) in the Kubernetes cluster can correspond to different namespaces, and the namespaces can realize the isolation between different resource objects in the Kubernetes cluster. The node namespace can realize the resource isolation between different nodes in the Kubernetes cluster, and the container namespace can realize the resource isolation between different containers in the Kubernetes cluster. When the fault detection container successfully accesses the namespace of any resource object in the Kubernetes cluster, the fault detection container can execute a process for the resource object, for example, by running the fault detection script to detect the fault of the resource object.

[0060] Based on this, for each to-be-detected node, the custom resource controller CR Controller can first take the node namespace of the to-be-detected node as an access object, and control the fault detection container allocated to the to-be-detected node to access the access object, that is, to access the node namespace of the to-be-detected node. Further, in the case where the node namespace of the to-be-detected node has been accessed, the custom resource controller CR Controller can control the fault detection script to execute in the node namespace of the to-be-detected node, that is, to detect the fault of the to-be-detected node, so as to obtain the node fault detection result corresponding to the to-be-detected node.

[0061] In step 208, in the case where the node fault detection result is obtained, the container namespace of the to-be-detected container deployed in the to-be-detected node is taken as an access object, and the fault detection script is controlled to execute in the container namespace, so as to obtain the container fault detection result corresponding to the to-be-detected node.

[0062] Each to-be-detected node can correspond to at least one to-be-detected container, and the related information of the to-be-detected node and the to-be-detected container can be preconfigured in the custom resource.

[0063] Optionally, for each to-be-detected node, in the case where the node fault detection result is obtained, the custom resource controller CR Controller can sequentially take the container namespace of the to-be-detected container deployed in the to-be-detected node as a new access object, and control the fault detection container allocated to the to-be-detected node to sequentially access the new access object, that is, sequentially access the container namespace of the to-be-detected container on the to-be-detected node. Further, in the case where the container namespace of a to-be-detected container on the to-be-detected node has been accessed, the custom resource controller CR Controller can control the fault detection script to execute in the container namespace, that is, perform fault detection on the to-be-detected container, so as to obtain the container fault detection result of the to-be-detected container. Then, repeat the above steps to obtain the container fault detection results of all to-be-detected containers on the to-be-detected node.

[0064] It should be noted that for each to-be-detected node, when the node fault detection result corresponding to the to-be-detected node and the container fault detection results of all to-be-detected containers on the to-be-detected node are obtained, the custom resource controller CR Controller can determine that the fault detection for the to-be-detected node has been completed, that is, the fault detection container on the to-be-detected node has ended the fault detection work.

[0065] In an optional embodiment, the custom resource controller CR Controller can specifically enter any node namespace or any container namespace through the nsenter tool, and after executing the fault detection script, query the node log or the container log through the logs interface in the Kubernetes cluster to obtain the execution result of the fault detection script. The nsenter is a Linux tool for entering a namespace. The nsenter supports multiple types of isolation such as network and PID based on the namespace function of the Linux kernel. It allows debugging and operation without stopping the container.

[0066] The cluster fault detection device can first monitor events in the cluster. When a creation event of a custom resource for detecting cluster faults is monitored, configuration information carried by the custom resource is read, so as to determine to-be-detected nodes and to-be-detected containers in the cluster according to the configuration information, and create a fault detection container according to the configuration information. The fault detection container is configured with a fault detection script and is assigned with a to-be-detected node, so that subsequent detection of the to-be-detected nodes and the to-be-detected containers in the cluster can be automatically performed through the fault detection container, and the cluster fault detection efficiency is improved. Specifically, for each to-be-detected node, a node namespace of the to-be-detected node is taken as an access object, the fault detection container assigned to the to-be-detected node is controlled to access the access object, and the fault detection script is controlled to execute in the node namespace, so as to obtain a node fault detection result corresponding to the to-be-detected node. Further, in the case where the node fault detection result is obtained, a container namespace of a to-be-detected container deployed in the to-be-detected node is taken as an access object, and the fault detection script is controlled to execute in the container namespace, so as to obtain a container fault detection result corresponding to the to-be-detected node. Based on this, by using the cluster fault detection method, a user does not need to perform fault detection on a large number of nodes or containers in the cluster one by one, but can automatically perform detection on the corresponding to-be-detected nodes and the to-be-detected containers deployed on the to-be-detected nodes one by one through the fault detection container assigned to each to-be-detected node, so as to realize efficient detection of cluster faults.

[0067] In an exemplary embodiment, creating the fault detection container according to the configuration information comprises:

[0068] From the configuration information, target configuration information for controlling the fault detection process is determined, and a workload resource list is assembled according to the target configuration information;

[0069] Based on the workload resource list, the fault detection container is created.

[0070] Specifically, the custom resource controller CR Controller can determine target configuration information for controlling the fault detection process according to the read configuration information, and the target configuration information is also a fault detection container creation rule. Based on this, the custom resource controller CR Controller can assemble a workload resource manifest according to the target configuration information, and the workload resource manifest is a manifest of a fault detection Job and can represent the fault detection Job. Further, the custom resource controller CR Controller can submit the assembled workload resource manifest to an API Server in the Kubernetes cluster, so that the API Server creates a fault diagnosis container according to the workload resource manifest. In the workload resource manifest, it can be specified which fault detection container needs to be created and how many fault detection containers need to be created.

[0071] For example, if a user needs to perform fault detection on a Kubernetes cluster, the user can pre-configure a custom resource and interact with the Kubernetes cluster through an API Server in the Kubernetes cluster to create the custom resource in the Kubernetes cluster. Based on this, the custom resource controller CR Controller can assemble a workload resource manifest according to configuration information in the custom resource, and then submit the workload resource manifest to the API Server to make the API Server create a specified number of fault detection containers.

[0072] In this embodiment, the corresponding fault detection container can be automatically created according to the configuration information of the custom resource issued by the user to the cluster, so that the fault detection container can be used to automatically and sequentially perform fault detection on the to-be-detected nodes and the to-be-detected containers in the Kubernetes cluster, without the need for the user to manually access each to-be-detected node and to-be-detected container, thereby improving the fault detection efficiency of the Kubernetes cluster.

[0073] In one of the embodiments, the workload resource manifest includes the following target configuration information: quantity configuration information, permission configuration information, node affinity configuration information, and fault detection script.

[0074] Based on the workload resource manifest, the fault detection container is created, including:

[0075] According to the quantity configuration information, a plurality of initial containers are created.

[0076] For each initial container, the namespace access right of the initial container is configured according to the permission configuration information, the to-be-detected node of the initial container is allocated according to the node affinity configuration information, and the fault detection script is mounted under the container directory of the initial container, to obtain a plurality of fault detection containers.

[0077] The quantity configuration information specifically refers to: specifying how many fault detection containers are to be created, and ensuring that a fault detection container can be allocated to each to-be-detected node. The permission configuration information: configures the access rights of the fault detection container to the node namespace and the container namespace. The node affinity configuration information can be used to: allocate a to-be-detected node to each fault detection container. The fault detection script is: an automated program for fault detection of the to-be-detected node or the to-be-detected container, which can be set in advance according to detection requirements.

[0078] Specifically, the API Server can create a plurality of initial containers according to the quantity configuration information. Further, for each initial container, the API Server can configure the namespace access right of the initial container according to the permission configuration information, so that each initial container becomes a privileged container and has the right to access any node namespace and any container namespace, ensuring that the fault detection container on each to-be-detected node can subsequently implement fault detection on the to-be-detected node and all to-be-detected containers on the to-be-detected node by switching the accessed namespace. In addition, the API Server can allocate a to-be-detected node to each initial container according to the node affinity configuration information, so as to subsequently schedule each initial container to the specified node. Furthermore, the API Server can mount the fault detection script under the container directory of the initial container, so that the fault detection script to be executed can be read from the container directory when executed. Based on this, a plurality of fault detection containers can be obtained.

[0079] In one possible implementation, regarding setting each initial container as a privileged container, it can specifically include: setting the privileged, hostNetwork, and hostPID of the initial container to true. Wherein, privileged=true makes the initial container can switch the accessed namespace, hostNetwork=true makes the initial container can use the network space of the allocated to-be-detected node, and hostPID=true makes the initial container can use the PID space of the allocated to-be-detected node, so that the initial container can use the network space and the PID space of the allocated to-be-detected node, and has the privilege of switching the namespace, so as to implement fault detection on the to-be-detected node and all to-be-detected containers on the to-be-detected node by switching the namespace on the to-be-detected node.

[0080] Exemplarily, taking a to-be-detected node as an example, if a process space PID of the to-be-detected node is 1, a state of a fault detection container on the to-be-detected node is running, and the fault detection needs to be performed on the to-be-detected node, the fault detection container can automatically read out a fault detection script from the container directory, and call the nsenter tool to switch the namespace to the process space PID=1, so as to access the node namespace of the to-be-detected node. The node fault detection result is obtained by executing the fault detection script. Further, the fault detection container can execute a docker inspect command in the node namespace of the to-be-detected node to obtain the PID value of each to-be-detected container on the to-be-detected node, and then enter the container namespace of the corresponding to-be-detected container by using the nsenter tool, so as to obtain the node fault detection result by executing the script.

[0081] In the embodiment, the privileged container, that is, the fault detection container, can be automatically generated according to the configuration information carried by the custom resource, so that the subsequent fault detection of the to-be-detected nodes and the to-be-detected containers in the Kubernetes cluster can be automatically and sequentially performed through the fault detection container, and the fault detection efficiency of the Kubernetes cluster is improved.

[0082] In one of the embodiments, the workload resource manifest further includes the following target configuration information: container lifecycle configuration information. The fault detection script is mounted in the container directory of the initial container to obtain a plurality of fault detection containers, including:

[0083] For each initial container, the initial container is configured with a cleaning delay duration according to the container lifecycle configuration information, to obtain a plurality of fault detection containers.

[0084] The lifecycle configuration information is used to indicate that the fault detection container needs to be destroyed after completing the fault task for a certain period of time, so as to avoid that the fault detection container that has completed the fault detection task occupies the cluster resources. The cleaning delay duration can be flexibly configured according to the requirements of the actual application scenario, and the embodiment does not limit this.

[0085] Optionally, for each initial container, the API Server can configure the initial container with a cleaning delay duration according to the container lifecycle configuration information in the workload resource manifest, to obtain a plurality of fault detection containers that can be automatically destroyed after completing the task.

[0086] In the embodiment, the cleaning delay duration of the fault detection container can be configured to avoid that the fault detection container that has completed the task occupies the Kubernetes cluster resources.

[0087] In one of the embodiments, at least one to-be-detected container is deployed in each to-be-detected node; and the method further includes:

[0088] When the container fault detection result of at least one of the to-be-detected containers in the to-be-detected node is obtained, the node fault detection result corresponding to the to-be-detected node and the at least one container fault detection result are written into the status information associated with the custom resource;

[0089] A cleaning delay duration of the fault detection container allocated to the to-be-detected node is determined, and after waiting for the cleaning delay duration, the fault detection container allocated to the to-be-detected node is cleaned up.

[0090] The cleaning delay duration can be any duration within 100 seconds to 400 seconds, such as 300 seconds.

[0091] Specifically, in actual application, for each to-be-detected node, under the control of the custom resource controller CRController, the fault detection container on the to-be-detected node can perform fault detection on the to-be-detected node by switching the namespace, and then the custom resource controller CRController can read the node log through the logs interface in the Kubernetes cluster to obtain the node fault detection result of the to-be-detected node. Further, under the control of the custom resource controller CRController, the fault detection container on the to-be-detected node can perform fault detection on all to-be-detected containers on the to-be-detected node in turn, and the custom resource controller CRController can read the container log through the logs interface to obtain the container fault detection result of each to-be-detected container on the to-be-detected node in turn.

[0092] Based on this, for each to-be-detected node, when the container fault detection result of at least one of the to-be-detected containers in the to-be-detected node is obtained, it indicates that the detection on the to-be-detected node and all to-be-detected containers on the to-be-detected node has been completed, and the results have been obtained. The custom resource controller CRController can write the node fault detection result corresponding to the to-be-detected node and the at least one container fault detection result into the status information (status field) associated with the custom resource CR. At the same time, a cleaning delay duration of the fault detection container allocated to the to-be-detected node can be determined, and after waiting for the cleaning delay duration, the fault detection container allocated to the to-be-detected node is cleaned up.

[0093] In this embodiment, the fault detection container can be destroyed in time after completing the detection task on the allocated to-be-detected node, so as to avoid occupying cluster resources.

[0094] In one of the embodiments, the cluster fault detection method further includes:

[0095] receive a cluster access authentication request triggered by a requester, verify authentication information carried in the cluster access authentication request;

[0096] When the authentication information is verified, receive and execute a cluster detection result reading instruction triggered by the requester for the status information, and feed back node fault detection results and container fault detection results stored in the status information.

[0097] The authentication information can be specifically an account password for user access to the cluster.

[0098] Optionally, the requester (i.e., the user) can trigger a cluster access authentication request through an API Server in the Kubernetes cluster. The API Server can receive the cluster access authentication request triggered by the requester and verify authentication information carried in the cluster access authentication request. When the authentication information is verified, the API Server can receive and execute a cluster detection result reading instruction triggered by the requester for the status information, and feed back node fault detection results and container fault detection results stored in the status information to the user.

[0099] That is, the user can view node fault detection results and container fault detection results stored in the status information (status) of the custom resource CR through the API Server in the Kubernetes cluster.

[0100] In this embodiment, the user can view the node fault detection results and the container fault detection results at any time.

[0101] In one detailed embodiment, based on the above embodiments, as shown in Figure 3 A Kubernetes cluster fault detection method is provided, which can be applied to a Kubernetes cluster or a cloud platform built based on the Kubernetes cluster. The Kubernetes cluster fault detection method mainly includes the following steps:

[0102] Step 1: The user triggers an instruction to issue a custom resource CR for cluster fault detection through an API Server in the Kubernetes cluster, so as to create the custom resource CR in the Kubernetes cluster.

[0103] Step two: When the custom resource controller CR Controller in the Kubernetes cluster monitors the creation event of the custom resource CR, the custom resource controller CR Controller can read the configuration information in the custom resource CR and create a fault diagnosis container according to the configuration information. Each to-be-detected node can be assigned a fault detection container, and the fault detection container can switch the accessed namespace through the nsenter tool to sequentially complete the fault detection of the to-be-detected node and the to-be-detected container on the to-be-detected node.

[0104] Step three: For each to-be-detected node, the custom resource controller CR Controller can read the node detection result and the container detection result output by the fault detection container.

[0105] Step four: The custom resource controller CR Controller writes the read node detection result and container detection result into the status information (status field) of the custom resource CR.

[0106] Step five: The user reads the node detection result and the container detection result from the status information (status field) of the custom resource CR through the API Server.

[0107] The above Kubernetes cluster fault detection method includes the whole process of custom resource issuing, node fault detection, container fault detection, detection result storage, and detection result feedback, and the whole process is managed by the Kubernetes cluster. The user only needs the authentication information of the Kubernetes cluster, and does not need to care about the underlying resources in the Kubernetes cluster, which is more in line with the design of cloud native and the use habit of the Kubernetes cluster user. By using the above Kubernetes cluster fault detection method, the user can perform comprehensive detection on the to-be-detected node and the to-be-detected container in the Kubernetes cluster through the API Server in the Kubernetes cluster, quickly troubleshoot faults, and improve the fault detection efficiency of the node and the container in the Kubernetes cluster.

[0108] In one embodiment, the Kubernetes cluster to which the above Kubernetes cluster fault detection method is applied can include the following resources: 4 virtual machines configured with 16G memory and 8-core CPU; 1 virtual machine configured with 8G memory and 4-core CPU; 1 load balancer; Linxu operating system kernel of 4.19 and above, and 3 Master nodes (all 8-core 16G) and 2 Worker nodes (one node 4-core 8G, one node 8-core 16G) are deployed, and the node names are named as master1, master2, master3, worker1, worker2.

[0109] Based on this, after the CR Controller is deployed in the Kubernetes cluster in the form of deployment, a certain number of fault detection containers can be created by the user issuing the custom resource CR to the Kubernetes cluster, and the fault detection containers are respectively scheduled to master1, master2, master3, worker1, and worker2. If the state of the fault detection container on a certain node becomes "Complete", it indicates that the detection on the to-be-detected node and the to-be-detected container on the to-be-detected node is completed, the detection result can be read out, and the detection result is written into the status field of the custom resource CR for subsequent feedback to the user.

[0110] It should be understood that although each step in the flowchart involved in each embodiment described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be alternately executed with other steps or steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiments of the present application also provide a cluster fault detection device for implementing the above-mentioned cluster fault detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more cluster fault detection device embodiments provided below can refer to the limitations of the cluster fault detection method described above, which will not be repeated here.

[0112] In one exemplary embodiment, asFigure 4 As shown, a cluster fault detection apparatus is provided, comprising an event monitoring module 402, a detection container creation module 404, a node detection module 406 and a container detection module 408, wherein:

[0113] The event monitoring module is configured to monitor events in the cluster, and when a creation event of a custom resource for detecting cluster faults is monitored, read configuration information carried by the custom resource;

[0114] The detection container creation module is configured to determine to-be-detected nodes and to-be-detected containers in the cluster according to the configuration information, and create a fault detection container according to the configuration information; the fault detection container is configured with a fault detection script and is assigned with a to-be-detected node belonging to the fault detection container;

[0115] The node detection module is configured to, for each to-be-detected node, take a node namespace of the to-be-detected node as an access object, control the fault detection container assigned to the to-be-detected node to access the access object, and control the fault detection script to execute in the node namespace to obtain a node fault detection result corresponding to the to-be-detected node;

[0116] The container detection module is configured to, in a case where the node fault detection result is obtained, take a container namespace of a to-be-detected container deployed in the to-be-detected node as an access object, and control the fault detection script to execute in the container namespace to obtain a container fault detection result corresponding to the to-be-detected node.

[0117] The cluster fault detection apparatus can first monitor events in the cluster, read configuration information carried by a self-defined resource for detecting cluster fault when a creation event of the self-defined resource is monitored, determine to-be-detected nodes and to-be-detected containers in the cluster according to the configuration information, and create a fault detection container according to the configuration information. The fault detection container is configured with a fault detection script and is assigned with the to-be-detected nodes, so that the to-be-detected nodes and the to-be-detected containers in the cluster can be automatically detected by the fault detection container subsequently, and the cluster fault detection efficiency is improved. Specifically, for each to-be-detected node, a node namespace of the to-be-detected node is taken as an access object, the fault detection container assigned to the to-be-detected node is controlled to access the access object, and the fault detection script is controlled to execute in the node namespace, so as to obtain a node fault detection result corresponding to the to-be-detected node. Further, in the case where the node fault detection result is obtained, a container namespace of a to-be-detected container deployed in the to-be-detected node is taken as an access object, and the fault detection script is controlled to execute in the container namespace, so as to obtain a container fault detection result corresponding to the to-be-detected node. Based on this, by using the cluster fault detection method, the user does not need to detect faults of a large number of nodes or containers in the cluster one by one, but can automatically detect each to-be-detected node and each to-be-detected container deployed on the to-be-detected node by using the fault detection container assigned to each to-be-detected node, so as to realize efficient detection of cluster faults.

[0118] In one of the embodiments, the event monitoring module further includes: a manifest assembling unit, configured to determine target configuration information for controlling the fault detection process from the configuration information, and assemble a workload resource manifest according to the target configuration information; and a detection container creating unit, configured to create the fault detection container based on the workload resource manifest.

[0119] In one of the embodiments, the workload resource manifest includes the following target configuration information: quantity configuration information, permission configuration information, node affinity configuration information, and fault detection script. Based on this, the detection container creating unit is further configured to: create a plurality of initial containers according to the quantity configuration information; for each initial container, configure namespace access permission for the initial container according to the permission configuration information, assign the to-be-detected node to the initial container according to the node affinity configuration information, and mount the fault detection script under a container directory of the initial container, to obtain the plurality of fault detection containers.

[0120] In one of the embodiments, the workload resource manifest further includes the following target configuration information: container lifecycle configuration information. Based on this, the detection container creating unit is further configured to: for each initial container, configure a clean-up delay time length for the initial container according to the container lifecycle configuration information, to obtain the plurality of fault detection containers.

[0121] In one of the embodiments, at least one to-be-detected container is deployed in each to-be-detected node. Based on this, the cluster fault detection method further includes a fault detection container cleaning module, which is configured to: for each to-be-detected node, when the container fault detection result of each to-be-detected container in the to-be-detected node is obtained, write the node fault detection result corresponding to the to-be-detected node and the at least one container fault detection result into the state information associated with the custom resource; determine a cleaning delay duration of the fault detection container allocated to the to-be-detected node, and clean the fault detection container allocated to the to-be-detected node after waiting for the cleaning delay duration.

[0122] In one of the embodiments, the cluster fault detection apparatus further includes a detection result feedback module, which is configured to: receive a cluster access authentication request triggered by a request party, and verify authentication information carried in the cluster access authentication request; when the authentication information passes the verification, receive and execute a cluster detection result reading instruction triggered by the request party for the state information, and feed back the node fault detection result and the container fault detection result stored in the state information.

[0123] The modules in the cluster fault detection apparatus can be implemented by software, hardware, or a combination thereof, in whole or in part. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to the modules.

[0124] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 5 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store cluster fault detection data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a cluster fault detection method.

[0125] Those skilled in the art can understand that Figure 5The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In an exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0127] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0128] In an embodiment, a computer program product is provided, including a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0131] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0132] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A cluster failure detection method, characterized by, The method comprises: monitoring events in a cluster, reading configuration information carried by a creation event of a custom resource for detecting cluster failure when the creation event is monitored; determining to-be-detected nodes and to-be-detected containers in the cluster according to the configuration information, and determining target configuration information for controlling a failure detection process from the configuration information, and assembling a workload resource manifest according to the target configuration information; the workload resource manifest comprises the following target configuration information: quantity configuration information, permission configuration information, node affinity configuration information, and failure detection script; creating a plurality of initial containers according to the quantity configuration information; for each of the initial containers, configuring namespace access permission for the initial container according to the permission configuration information to set the initial container as a privileged container, the privileged container having the permission to access any node namespace and any container namespace, allocating the to-be-detected nodes to the initial container according to the node affinity configuration information, and mounting the failure detection script under a container directory of the initial container to obtain a plurality of failure detection containers; the failure detection containers achieve failure detection on the to-be-detected nodes and all to-be-detected containers on the to-be-detected nodes by switching the accessed namespaces; for each of the to-be-detected nodes, taking a node namespace of the to-be-detected node as an access object, controlling the failure detection containers allocated to the to-be-detected node to access the access object, and controlling the failure detection script to execute in the node namespace to obtain a node failure detection result corresponding to the to-be-detected node; in a case where the node failure detection result is obtained, taking a container namespace of a to-be-detected container deployed in the to-be-detected node as the access object, and controlling the failure detection script to execute in the container namespace to obtain a container failure detection result corresponding to the to-be-detected node.

2. The method of claim 1, wherein, The workload resource manifest further comprises the following target configuration information: container lifecycle configuration information; The mounting of the failure detection script under the container directory of the initial container to obtain the plurality of failure detection containers comprises: for each of the initial containers, configuring a clean-up delay duration for the initial container according to the container lifecycle configuration information to obtain the plurality of failure detection containers.

3. The method of claim 1, wherein, Each of the to-be-detected nodes has at least one to-be-detected container deployed therein; the method further comprises: for each of the to-be-detected nodes, when the container failure detection results of the at least one to-be-detected container in the to-be-detected node have been obtained, writing the node failure detection result corresponding to the to-be-detected node and the at least one container failure detection result into state information associated with the custom resource; determining a clean-up delay duration of the failure detection containers allocated to the to-be-detected node, and cleaning up the failure detection containers allocated to the to-be-detected node after the clean-up delay duration.

4. The method of claim 3, wherein, The method further comprises: receiving a cluster access authentication request triggered by a requestor, and verifying authentication information carried in the cluster access authentication request; When the authentication information is verified, a cluster detection result reading instruction triggered by the request party for the state information is received and executed, and a node fault detection result and a container fault detection result stored in the state information are fed back.

5. A cluster failure detection apparatus characterized by comprising: The device comprises: An event monitoring module is configured to monitor events in a cluster, read configuration information carried by a self-defined resource for detecting cluster faults when a creation event of the self-defined resource is monitored, and determine to-be-detected nodes and to-be-detected containers in the cluster according to the configuration information. A detection container creation module is configured to determine to-be-detected nodes and to-be-detected containers in the cluster according to the configuration information. The event monitoring module further comprises a manifest assembling unit and a detection container creation unit; the manifest assembling unit is configured to determine target configuration information for controlling a fault detection process from the configuration information, and assemble a workload resource manifest according to the target configuration information; the workload resource manifest comprises the following target configuration information: quantity configuration information, permission configuration information, node affinity configuration information, and fault detection scripts. The detection container creation unit is configured to create a plurality of initial containers according to the quantity configuration information, configure namespace access permissions for each initial container according to the permission configuration information to set the initial container as a privileged container, the privileged container has the permission to access any node namespace and any container namespace, assign to-be-detected nodes to the initial container according to the node affinity configuration information, and mount the fault detection scripts under a container directory of the initial container to obtain a plurality of fault detection containers; the fault detection containers achieve fault detection of to-be-detected nodes and all to-be-detected containers on the to-be-detected nodes by switching the accessed namespaces. A node detection module is configured to, for each to-be-detected node, take a node namespace of the to-be-detected node as an access object, control a fault detection container assigned to the to-be-detected node to access the access object, and control the fault detection scripts to execute in the node namespace to obtain a node fault detection result corresponding to the to-be-detected node. A container detection module is configured to, in a case where the node fault detection result is obtained, take a container namespace of a to-be-detected container deployed in the to-be-detected node as the access object, and control the fault detection scripts to execute in the container namespace to obtain a container fault detection result corresponding to the to-be-detected node.

6. The apparatus of claim 5, wherein, The workload resource manifest further comprises the following target configuration information: container lifecycle configuration information; and the detection container creation unit is further configured to, for each initial container, configure a clean-up delay duration for the initial container according to the container lifecycle configuration information to obtain a plurality of fault detection containers.

7. The apparatus of claim 5, wherein, At least one to-be-detected container is deployed in each of the to-be-detected nodes; the device further comprises a fault detection container cleaning module, which is configured to, for each of the to-be-detected nodes, when the container fault detection result of each of the at least one to-be-detected container in the to-be-detected node is obtained, write the node fault detection result corresponding to the to-be-detected node and the at least one container fault detection result into the state information associated with the self-defined resource, determine a cleaning delay duration of the fault detection container allocated to the to-be-detected node, wait for the cleaning delay duration, and clean the fault detection container allocated to the to-be-detected node.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4.

Images