Cluster fault detection method and device, computer equipment and readable storage medium

Abstract

The invention relates to a cluster fault detection method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of reading configuration information carried by a custom resource when a creation event of the custom resource for detecting a cluster fault is monitored; determining a to-be-detected node and a to-be-detected container according to the configuration information, and creating a fault detection container; for each to-be-detected node, taking the node namespace of the to-be-detected node as an access object, controlling a fault detection container distributed to the to-be-detected node to access the access object, and controlling a fault detection script to be executed in the node namespace to obtain a node fault detection result corresponding to the to-be-detected node; and taking the container namespace of the to-be-detected container deployed in the to-be-detected node as an access object, and controlling the fault detection script to be executed in the container namespace to obtain a container fault detection result corresponding to the to-be-detected node. By adopting the method, cluster faults can be efficiently detected.

Description

Technical Field

[0001] The present application relates to the technical field of cluster fault detection, and in particular to a cluster fault detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of computer technology, clusters are often used to improve system performance, flexibility, and scalability, such as Kubernetes clusters. Kubernetes clusters are open source systems for automatically deploying, scaling, and managing "containerized applications". They can provide a platform that can automatically deploy, scale, and run application containers across host clusters, thereby improving system performance, flexibility, and scalability.

[0003] In traditional technologies, when a Kubernetes cluster fails, since the Kubernetes cluster abstracts the application operating environment and computing resources into a large number of resources such as Node (also known as a node, responsible for executing tasks assigned by the cluster) and Pod (also known as a container set, deployed on a single node and containing one or more containers), and the containers are isolated in multiple process spaces, and users usually cannot directly access the Node or Pod, it is difficult to quickly detect whether the Node or Pod in the Kubernetes cluster has failed one by one, resulting in low fault detection efficiency in the Kubernetes cluster. Summary of the invention

[0004] Based on this, it is necessary to provide a cluster fault detection method, apparatus, computer equipment, computer-readable storage medium and computer program product that can efficiently detect cluster faults in response to the above technical problems.

[0005] In a first aspect, the present application provides a cluster fault detection method, comprising:

[0006] Monitor events in the cluster. When a creation event of a custom resource used to detect cluster failures is detected, read the configuration information carried by the custom resource.

[0007] Determine the nodes and containers to be detected in the cluster according to the configuration information, and create a fault detection container according to the configuration information; the fault detection container has been configured with a fault detection script and has been assigned the nodes to be detected;

[0008] For each node to be detected, the node namespace of the node to be detected is used as the access object, the fault detection container allocated to the node to be detected is controlled to access the access object, and the fault detection script is controlled to be executed in the node namespace to obtain the node fault detection result corresponding to the node to be detected;

[0009] When the node fault detection result is obtained, the container namespace of the container to be detected deployed in the node to be detected is used as the access object, and the fault detection script is controlled to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected.

[0010] In one embodiment, creating a fault detection container according to configuration information includes:

[0011] Determine target configuration information for controlling the fault detection process from the configuration information, and assemble a workload resource list according to the target configuration information;

[0012] Create a failure detection container based on the workload resource list.

[0013] In one of the embodiments, the workload resource list includes the following target configuration information: quantity configuration information, permission configuration information, node affinity configuration information, and a fault detection script;

[0014] Create a fault detection container based on the workload resource list, including:

[0015] Create multiple initial containers according to the quantity configuration information;

[0016] For each initial container, namespace access permissions are configured for the initial container according to the permission configuration information, nodes to be detected are allocated to the initial container according to the node affinity configuration information, and the fault detection script is mounted in the container directory of the initial container to obtain multiple fault detection containers.

[0017] In one of the embodiments, the workload resource list also includes the following target configuration information: container lifecycle configuration information;

[0018] Mount the fault detection script in the container directory of the initial container to obtain multiple fault detection containers, including:

[0019] For each initial container, a cleanup delay time is configured for the initial container according to the container lifecycle configuration information, thereby obtaining multiple fault detection containers.

[0020] In one of the embodiments, at least one container to be detected is deployed in each node to be detected; the method further includes:

[0021] For each node to be detected, when the container fault detection result of at least one container to be detected in the node to be detected has been obtained, the node fault detection result and at least one container fault detection result corresponding to the node to be detected are written into the status information associated with the custom resource;

[0022] Determine a cleaning delay time of the fault detection container allocated to the node to be detected, wait for the cleaning delay time, and then clean up the fault detection container allocated to the node to be detected.

[0023] In one of the embodiments, the cluster failure detection method further includes:

[0024] Receive a cluster access authentication request triggered by a requester, and verify the authentication information carried in the cluster access authentication request;

[0025] When the authentication information is verified, the cluster detection result reading instruction triggered by the requester for the status information is received and executed, and the node fault detection results and container fault detection results stored in the status information are fed back.

[0026] In a second aspect, the present application further provides a cluster fault detection device, comprising:

[0027] The event monitoring module is used to monitor events in the cluster. When the creation event of a custom resource used to detect cluster failures is detected, the configuration information carried by the custom resource is read.

[0028] A detection container creation module is used to determine the nodes and containers to be detected in the cluster according to the configuration information, and to create a fault detection container according to the configuration information; the fault detection container has been configured with a fault detection script and has been assigned the nodes to be detected;

[0029] The node detection module is used to control the fault detection container allocated to the node to be detected to access the access object for each node to be detected, and to control the execution of the fault detection script in the node namespace to obtain the node fault detection result corresponding to the node to be detected;

[0030] The container detection module is used to, when a node fault detection result is obtained, use the container namespace of the container to be detected deployed in the node to be detected as an access object, and control the fault detection script to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected.

[0031] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned fault detection method when executing the computer program.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned fault detection method when executed by a processor.

[0033] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the above-mentioned fault detection method when executed by a processor.

[0034] The above-mentioned cluster fault detection method, device, computer equipment, computer-readable storage medium and computer program product can first monitor the events in the cluster, and when the creation event of the custom resource used to detect the cluster fault is monitored, the configuration information carried by the custom resource is read, so as to determine the nodes to be detected and the containers to be detected in the cluster according to the configuration information, and create a fault detection container according to the configuration information. Among them, the fault detection container has been configured with a fault detection script, and the nodes to be detected have been assigned to it, so that the detection of the nodes to be detected and the containers to be detected in the cluster can be automatically performed through the fault detection container in the future, thereby improving the efficiency of cluster fault detection. Specifically, for each node to be detected, the node namespace of the node to be detected can be used as an access object, the fault detection container assigned to the node to be detected can be controlled to access the access object, and the fault detection script can be controlled to be executed in the node namespace to obtain the node fault detection result corresponding to the node to be detected. Further, in the case of obtaining the node fault detection result, the container namespace of the container to be detected deployed in the node to be detected can be used as an access object, and the fault detection script can be controlled to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected. Based on this, by adopting the above-mentioned cluster fault detection method, the user does not need to perform fault detection on a large number of nodes or containers in the cluster one by one. Instead, the corresponding nodes to be detected and the containers to be detected deployed on the nodes to be detected can be automatically detected one by one through the fault detection container allocated to each node to be detected, thereby realizing efficient detection of cluster faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 An application environment diagram of a cluster fault detection method in an embodiment;

[0037] Figure 2 A schematic diagram of a flow chart of a cluster failure detection method in an embodiment;

[0038] Figure 3 A schematic diagram of a process of a Kubernetes cluster fault detection method in one embodiment;

[0039] Figure 4 is a structural block diagram of a cluster fault detection device in one embodiment;

[0040] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The cluster fault detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. A user can use a terminal 102 to communicate with a server 104 through a network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. A cluster, such as a Kubernetes cluster, can be deployed on the server 104.

[0043] Taking the application of the cluster fault detection method provided in the embodiment of the present application to a Kubernetes cluster as an example, the Kubernetes cluster may specifically include: a node (Node), a container set (Pod) deployed on the node, a service (Service), and a custom resource controller (Custom Resource Controller, referred to as CR Controller). Among them:

[0044] A node is a machine that hosts a Kubernetes cluster (can be regarded as a host). All business applications of the Kubernetes cluster run on the node.

[0045] Container set is the smallest scheduling management unit of Kubernetes cluster. Each container set contains one or more containers. Containers can decouple applications from the underlying host facilities through the process-level isolation capability provided by the operating system, and realize the ability to run multiple independent applications on the host. It has the advantages of faster application startup and less resource usage.

[0046] Services can open applications running on container sets as network services. Taking the API Server in the services included in the Kubernetes cluster as an example, users can communicate with the Kubernetes cluster through the API Server, but it is difficult for users to directly log in to the node or container to execute commands to troubleshoot. The API Server can provide interfaces for adding, deleting, modifying, checking, and monitoring various resource objects (including nodes, containers, custom resources, services, etc.) in the Kubernetes cluster.

[0047] The custom resource controller can be used to monitor the creation events of custom resources (CR) in the Kubernetes cluster, and monitor and control the entire life cycle of custom resources in the Kubernetes cluster. Custom resources are extensions of applications in the Kubernetes cluster, and custom resources can appear or disappear in the cluster through dynamic registration.

[0048] Specifically, the user can use the terminal 102 to send instructions to the server 104 through the API Server to create a custom resource CR in the Kubernetes cluster, wherein the custom resource CR is used to detect the fault of the Kubernetes cluster. Further, the custom resource controller CR Controller in the Kubernetes cluster can monitor the events in the Kubernetes cluster through the API Server. When the creation event of the custom resource CR is monitored, the custom resource controller CR Controller can read the configuration information carried by the custom resource CR, and determine the nodes to be detected and the containers to be detected in the Kubernetes cluster according to the configuration information, thereby creating a fault detection container according to the configuration information. Among them, the fault detection container has been configured with a fault detection script and has been assigned to the node to be detected. Furthermore, for each node to be detected, the custom resource controller CR Controller can first use the node namespace of the node to be detected as the access object, control the fault detection container assigned to the node to be detected to access the access object, and control the fault detection script to be executed in the node namespace to obtain the node fault detection result corresponding to the node to be detected. When the node fault detection result is obtained, the custom resource controller CR Controller can use the container namespace of the container to be detected deployed in the node to be detected as the access object, and control the fault detection script to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected. Finally, when the custom resource controller CR Controller detects that the fault detection container on a node to be detected has completed its work, the custom resource controller CR Controller can store the node fault detection result and container fault detection result corresponding to the node to be detected in a specified location for subsequent feedback to the user.

[0049] In an exemplary embodiment, Figure 2 As shown, a cluster fault detection method is provided, which is described by taking the method applied to a custom resource controller CR Controller in a Kubernetes cluster as an example, including the following steps 202 to 208. Among them:

[0050] Step 202 : monitor events in the cluster, and when a creation event of a custom resource for detecting cluster failure is detected, read configuration information carried by the custom resource.

[0051] Optionally, during the operation of the Kubernetes cluster, the API Server in the Kubernetes cluster is a tool for users to interact with the Kubernetes cluster. Therefore, the custom resource controller CR Controller can monitor events in the Kubernetes cluster in real time through the API Server to monitor whether users create custom resources in the cluster. When the creation event of a custom resource used to detect cluster failures is detected, the custom resource controller CRController can automatically read the configuration information carried by the custom resource CR.

[0052] In one embodiment, the custom resource controller CR Controller is deployed to the Kubernetes cluster in a Deployment manner, wherein the Deployment manner specifically refers to: a stateless load form, so as to ensure high availability, flexibility, and simplified operation and maintenance of the custom resource controller CR Controller.

[0053] Exemplarily, users can package custom resources into container images, and interact with the Kubernetes cluster through the API Serve in the Kubernetes cluster, and send the custom resources to the Kubernetes cluster to create custom resources in the Kubernetes cluster. Among them, the file system used to provide an isolated execution environment for the container process is the container image. In this embodiment, the following contents can be configured in the configuration information of the custom resource: the detection script to be executed, the nodes and containers to be detected, and the fault detection container creation rules, so as to provide an execution environment for cluster fault diagnosis, so that the created fault detection container can be used later to complete the fault detection of multiple isolated containers one by one and automatically.

[0054] Step 204, determine the nodes and containers to be detected in the cluster according to the configuration information, and create a fault detection container according to the configuration information; the fault detection container has been configured with a fault detection script and has been assigned the nodes to be detected.

[0055] Wherein, each node to be detected is respectively allocated with at least one fault detection container.

[0056] Optionally, the custom resource controller CR Controller may determine the nodes to be detected and the containers to be detected in the Kubernetes cluster according to the read configuration information, and create the fault detection container according to the fault detection container creation rule carried in the configuration information.

[0057] For example, the custom resource controller CR Controller can create a fault detection job including a fault detection container creation rule according to the read configuration information, so as to create a certain number of fault detection containers according to the fault detection job. Wherein, the job is one of the resource objects of the cluster, which can be used to perform a one-time task. The number of fault detection containers to be created can be specified in the job, and it is ensured that the created fault detection container stops working after completing the task.

[0058] Step 206, for each node to be detected, the node namespace of the node to be detected is used as the access object, the fault detection container allocated to the node to be detected is controlled to access the access object, and the fault detection script is controlled to be executed in the node namespace to obtain the node fault detection result corresponding to the node to be detected.

[0059] Among them, namespace is a mechanism for organizing and isolating code, which allows different objects with the same name in the same environment to not conflict. Different resource objects (nodes or containers) in a Kubernetes cluster can correspond to different namespaces, and namespaces can achieve isolation between different resource objects in a Kubernetes cluster. Node namespaces can achieve resource isolation between different nodes in a Kubernetes cluster, and container namespaces can achieve resource isolation between different containers in a Kubernetes cluster. When the fault detection container successfully accesses the namespace of any resource object in the Kubernetes cluster, the fault detection container can execute a process for the resource object, for example, by running a fault detection script to perform fault detection on the resource object.

[0060] Based on this, for each node to be detected, the customized resource controller CR Controller can first use the node namespace of the node to be detected as an access object, and control the fault detection container allocated to the node to be detected to access the access object, that is, access the node namespace of the node to be detected. Further, in the case where the node namespace of the node to be detected has been accessed, the customized resource controller CR Controller can control the fault detection script to be executed in the node namespace of the node to be detected, that is, perform fault detection on the node to be detected, thereby obtaining the node fault detection result corresponding to the node to be detected.

[0061] Step 208, when the node fault detection result is obtained, the container namespace of the container to be detected deployed in the node to be detected is used as the access object, and the fault detection script is controlled to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected.

[0062] Each node to be detected may correspond to at least one container to be detected, and the relevant information of the node to be detected and the container to be detected may be pre-configured in the custom resource.

[0063] Optionally, for each node to be detected, when the node fault detection result is obtained, the customized resource controller CR Controller can use the container namespace of the container to be detected deployed in the node to be detected as a new access object in turn, and control the fault detection container allocated to the node to be detected to access the new access object in turn, that is, to access the container namespace of the container to be detected on the node to be detected in turn. Further, when the container namespace of a container to be detected on the node to be detected has been accessed, the customized resource controller CR Controller can control the fault detection script to be executed in the container namespace, that is, to perform fault detection on the container to be detected, thereby obtaining the container fault detection result of the container to be detected. Then, repeat the above steps to obtain the container fault detection results of all containers to be detected on the node to be detected.

[0064] It should be noted that, for each node to be detected, after obtaining the node fault detection result corresponding to the node to be detected and the container fault detection results of all containers to be detected on the node to be detected, the customized resource controller CRController can determine that the fault detection for the node to be detected has been completed, that is, the fault detection container on the node to be detected has completed the fault detection work.

[0065] In an optional embodiment, the custom resource controller CR Controller can specifically enter any node namespace or any container namespace through the nsenter tool, and after executing the fault detection script, query the node log or container log through the logs interface in the Kubernetes cluster to obtain the execution result of the fault detection script. Among them, nsenter is a Linux tool for entering the namespace. nsenter is based on the namespace function of the Linux kernel and supports multiple types of isolation such as network and PID. Debugging and operation are allowed without stopping the container.

[0066] The above-mentioned cluster fault detection device can first monitor the events in the cluster, and when the creation event of the custom resource used to detect the cluster fault is monitored, the configuration information carried by the custom resource is read, so as to determine the nodes to be detected and the containers to be detected in the cluster according to the configuration information, and create a fault detection container according to the configuration information. Among them, the fault detection container has been configured with a fault detection script, and the nodes to be detected have been assigned to it, so that the detection of the nodes to be detected and the containers to be detected in the cluster can be automatically performed through the fault detection container in the future, thereby improving the efficiency of cluster fault detection. Specifically, for each node to be detected, the node namespace of the node to be detected can be used as an access object, the fault detection container assigned to the node to be detected can be controlled to access the access object, and the fault detection script can be controlled to be executed in the node namespace to obtain the node fault detection result corresponding to the node to be detected. Further, in the case of obtaining the node fault detection result, the container namespace of the container to be detected deployed in the node to be detected can be used as an access object, and the fault detection script can be controlled to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected. Based on this, by adopting the above-mentioned cluster fault detection method, the user does not need to perform fault detection on a large number of nodes or containers in the cluster one by one. Instead, the corresponding nodes to be detected and the containers to be detected deployed on the nodes to be detected can be automatically detected one by one through the fault detection container allocated to each node to be detected, thereby realizing efficient detection of cluster faults.

[0067] In an exemplary embodiment, creating a fault detection container according to configuration information includes:

[0068] Determine target configuration information for controlling the fault detection process from the configuration information, and assemble a workload resource list according to the target configuration information;

[0069] Create a failure detection container based on the workload resource list.

[0070] Specifically, the custom resource controller CR Controller can determine the target configuration information used to control the fault detection process based on the read configuration information, and the target configuration information is also the fault detection container creation rule. Based on this, the custom resource controller CR Controller can assemble a workload resource list based on the target configuration information. This workload resource list is the manifest of the fault detection job, which can represent the fault detection job. Furthermore, the custom resource controller CR Controller can submit the assembled workload resource list to the API Server in the Kubernetes cluster, so that the API Server can create a fault diagnosis container based on the workload resource list. Among them, the workload resource list can specify: what kind of fault detection container to create and how many.

[0071] For example, if a user needs to perform fault detection on a Kubernetes cluster, the user can pre-configure custom resources and interact with the Kubernetes cluster through the API Server in the Kubernetes cluster to create custom resources in the Kubernetes cluster. Based on this, the custom resource controller CR Controller can assemble a workload resource list according to the configuration information in the custom resource, and then submit the workload resource list to the API Server so that the API Server creates a specified number of fault detection containers.

[0072] In this embodiment, a corresponding fault detection container can be automatically created according to the configuration information of the custom resource sent by the user to the cluster, so that the fault detection container can be used to automatically and one by one perform fault detection on the nodes to be detected and the containers to be detected in the Kubernetes cluster, without the need for the user to personally visit each node to be detected and the containers to be detected one by one, thereby improving the fault detection efficiency of the Kubernetes cluster.

[0073] In one of the embodiments, the workload resource list includes the following target configuration information: quantity configuration information, permission configuration information, node affinity configuration information, and a fault detection script;

[0074] Create a fault detection container based on the workload resource list, including:

[0075] Create multiple initial containers according to the quantity configuration information;

[0076] For each initial container, namespace access permissions are configured for the initial container according to the permission configuration information, nodes to be detected are allocated to the initial container according to the node affinity configuration information, and the fault detection script is mounted in the container directory of the initial container to obtain multiple fault detection containers.

[0077] The quantity configuration information specifically refers to: specifying the number of fault detection containers to be created and ensuring that each node to be detected can be assigned a fault detection container. Permission configuration information: configuring the fault detection container with permissions to access the node namespace and container namespace. Node affinity configuration information can be used to: assign a node to be detected to each fault detection container. The fault detection script is: an automated program for fault detection of the node to be detected or the container to be detected, which can be set in advance according to the detection requirements.

[0078] Specifically, the API Server can create a certain number of multiple initial containers according to the quantity configuration information. Furthermore, for each initial container, the API Server can configure the namespace access rights for the initial container according to the permission configuration information, so that each initial container becomes a privileged container with the permission to access any node namespace and any container namespace, ensuring that the fault detection container on each node to be detected can subsequently switch the namespace accessed to achieve fault detection of the node to be detected and all containers to be detected on the node to be detected. In addition, the API Server can assign a node to be detected to each initial container according to the node affinity configuration information, so that each initial container can be scheduled to the specified node separately in the future. Furthermore, the API Server can mount the fault detection script under the container directory of the initial container, so that the fault detection script to be executed can be read from the container directory when it is executed. Based on this, multiple fault detection containers can be obtained.

[0079] In a possible implementation, setting each initial container as a privileged container may specifically include: setting privileged, hostNetwork, and hostPID of the initial container to true. Among them, privileged=true enables the initial container to switch the namespace to be accessed, hostNetwork=true enables the initial container to use the network space of the node to be detected, and hostPID=true enables the initial container to use the PID space of the node to be detected, so that the initial container can use the network space and PID space of the node to be detected and has the privilege of switching namespaces, so that the node to be detected and all containers to be detected on the node to be detected can be detected by switching the namespace on the node to be detected.

[0080] For example, taking a node to be detected as an example, if the process space PID of the node to be detected is 1, the state of the fault detection container on the node to be detected is running, and the node to be detected needs to be fault-detected, the fault detection container can automatically read the fault detection script from the container directory, and call the nsenter tool to switch the namespace to the process space of PID=1, thereby accessing the node namespace of the node to be detected, and obtaining the node fault detection result by executing the fault detection script. Furthermore, the fault detection container can execute the docker inspect command in the node namespace of the node to be detected to obtain the PID value of each container to be detected on the node to be detected, and then enter the container namespace of the corresponding container to be detected in turn through nsenter, and obtain the node fault detection result by executing the script.

[0081] In this embodiment, a privileged container, that is, a fault detection container, can be automatically generated according to the configuration information carried by the custom resource, so that the fault detection container can be used to automatically and one by one perform fault detection on the nodes and containers to be detected in the Kubernetes cluster, thereby improving the fault detection efficiency of the Kubernetes cluster.

[0082] In one embodiment, the workload resource list also includes the following target configuration information: container lifecycle configuration information. The fault detection script is mounted in the container directory of the initial container to obtain multiple fault detection containers, including:

[0083] For each initial container, a cleanup delay time is configured for the initial container according to the container lifecycle configuration information, thereby obtaining multiple fault detection containers.

[0084] The lifecycle configuration information is used to indicate how long after the fault detection container completes the fault task before it needs to be destroyed, so as to prevent the fault detection container that has completed the fault detection task from occupying cluster resources. The cleaning delay time can be flexibly configured according to the needs of the actual application scenario, and this embodiment does not limit this.

[0085] Optionally, for each initial container, the API Server can configure a cleanup delay time for the initial container according to the container lifecycle configuration information in the workload resource list, and obtain multiple fault detection containers that can be self-destructed after completing the task.

[0086] In this embodiment, the cleaning delay time of the fault detection container can be configured to prevent the fault detection container that has completed the task from occupying Kubernetes cluster resources.

[0087] In one of the embodiments, at least one container to be detected is deployed in each node to be detected; the method further includes:

[0088] For each node to be detected, when the container fault detection result of at least one container to be detected in the node to be detected has been obtained, the node fault detection result and at least one container fault detection result corresponding to the node to be detected are written into the status information associated with the custom resource;

[0089] Determine a cleaning delay time of the fault detection container allocated to the node to be detected, wait for the cleaning delay time, and then clean up the fault detection container allocated to the node to be detected.

[0090] The cleaning delay time may be any time between 100 seconds and 400 seconds, such as 300 seconds.

[0091] Specifically, in the actual application process, for each node to be detected, under the control of the customized resource controller CRController, the fault detection container on the node to be detected can first perform fault detection on the node to be detected by switching the namespace, and then the customized resource controller CR Controller can read the node log through the logs interface in the Kubernetes cluster to obtain the node fault detection result of the node to be detected. Further, under the control of the customized resource controller CR Controller, the fault detection container on the node to be detected can perform fault detection on all containers to be detected on the node to be detected in turn, and the customized resource controller CR Controller can read the container log through the logs interface to obtain the container fault detection results of each container to be detected on the node to be detected in turn.

[0092] Based on this, for each node to be detected, when the container fault detection result of at least one container to be detected in the node to be detected has been obtained, it means that the detection of the node to be detected and all containers to be detected on the node to be detected has been completed, and the results have been obtained. The custom resource controller CR Controller can write the node fault detection result and at least one container fault detection result corresponding to the node to be detected into the status information (status field) associated with the custom resource CR. At the same time, the cleaning delay time of the fault detection container assigned to the node to be detected can be determined, and after waiting for the cleaning delay time, the fault detection container assigned to the node to be detected is cleaned up.

[0093] In this embodiment, after the fault detection container completes the detection task on the assigned node to be detected, the fault detection container can be destroyed in time to avoid occupying cluster resources.

[0094] In one of the embodiments, the cluster failure detection method further includes:

[0095] Receive a cluster access authentication request triggered by a requester, and verify the authentication information carried in the cluster access authentication request;

[0096] When the authentication information is verified, the cluster detection result reading instruction triggered by the requester for the status information is received and executed, and the node fault detection results and container fault detection results stored in the status information are fed back.

[0097] The authentication information may specifically include: the account and password of the user to access the cluster.

[0098] Optionally, the requester (i.e., user) can trigger a cluster access authentication request through the API Server in the Kubernetes cluster. The API Server can receive the cluster access authentication request triggered by the requester and verify the authentication information carried in the cluster access authentication request. When the authentication information passes the verification, the API Server can receive and execute the cluster detection result reading instruction triggered by the requester for the status information, and feedback the node fault detection results and container fault detection results stored in the status information to the user.

[0099] That is, users can view the node fault detection results and container fault detection results stored in the status information (status) of the custom resource CR through the API Server in the Kubernetes cluster.

[0100] In this embodiment, users are supported to view node fault detection results and container fault detection results at any time.

[0101] In a detailed embodiment, based on the above embodiments, Figure 3 As shown, a Kubernetes cluster fault detection method is provided, which can be applied to a Kubernetes cluster or a cloud platform built based on a Kubernetes cluster. The Kubernetes cluster fault detection method mainly includes the following steps:

[0102] Step 1: The user triggers an instruction to issue a custom resource CR for cluster fault detection through the API Server in the Kubernetes cluster to create a custom resource CR in the Kubernetes cluster.

[0103] Step 2: When the custom resource controller CR Controller in the Kubernetes cluster detects the creation event of the custom resource CR, the custom resource controller CR Controller can read the configuration information in the custom resource CR and create a fault diagnosis container based on the configuration information. Among them, each node to be detected can be assigned a fault detection container, and the fault detection container can switch the namespace accessed by the nsenter tool to complete the fault detection of the node to be detected and the container to be detected on the node to be detected in turn.

[0104] Step 3: For each node to be detected, the customized resource controller CR Controller can read the node detection result and container detection result output by the fault detection container.

[0105] Step 4: The custom resource controller CR Controller writes the read node detection results and container detection results into the status information (status field) of the custom resource CR.

[0106] Step 5: The user reads the node detection results and container detection results from the status information (status field) of the custom resource CR through the API Server.

[0107] The above-mentioned Kubernetes cluster fault detection method includes the whole process of custom resource delivery, node fault detection, container fault detection, detection result storage, and detection result feedback. The whole process is managed by the Kubernetes cluster. Users only need the authentication information of the Kubernetes cluster, and do not need to care about the underlying resources in the Kubernetes cluster. It is more in line with the cloud-native design and the usage habits of Kubernetes cluster users. Using the above-mentioned Kubernetes cluster fault detection method, users only need to use the API Server in the Kubernetes cluster to conduct a comprehensive detection of the nodes and containers to be detected in the Kubernetes cluster, quickly troubleshoot faults, and improve the efficiency of fault detection of nodes and containers in the Kubernetes cluster.

[0108] In one embodiment, the Kubernetes cluster to which the above-mentioned Kubernetes cluster fault detection method is applied may include the following resources: 4 virtual machines configured with 16G memory and 8-core CPU; 1 virtual machine configured with 8G memory and 4-core CPU; a load balancer; the Linux operating system kernel is 4.19 and above, and 3 Master nodes (all 8-core 16G) and 2 Worker nodes (one node is 4-core 8G and the other node is 8-core 16G) are deployed, and the node names are named master1, master2, master3, worker1, and worker2 respectively.

[0109] Based on this, after deploying CR Controller in the Kubernetes cluster in the form of deployment, a certain number of fault detection containers can be created through the custom resource CR sent by the user to the Kubernetes cluster, and the fault detection containers can be scheduled to master1, master2, master3, worker1, and worker2 respectively. If the status of the fault detection container on a node changes to "Complete", it means that the detection of the node to be detected and the container to be detected on the node to be detected is completed, and the user detection result can be read and written into the status field of the custom resource CR for subsequent feedback to the user.

[0110] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiment of the present application also provides a cluster fault detection device for implementing the cluster fault detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more cluster fault detection device embodiments provided below can refer to the above limitations on the cluster fault detection method, and will not be repeated here.

[0112] In an exemplary embodiment, Figure 4 As shown, a cluster fault detection device is provided, including: an event monitoring module 402, a detection container creation module 404, a node detection module 406 and a container detection module 408, wherein:

[0113] The event monitoring module is used to monitor events in the cluster. When the creation event of a custom resource used to detect cluster failures is detected, the configuration information carried by the custom resource is read.

[0114] A detection container creation module is used to determine the nodes and containers to be detected in the cluster according to the configuration information, and to create a fault detection container according to the configuration information; the fault detection container has been configured with a fault detection script and has been assigned the nodes to be detected;

[0115] The node detection module is used to control the fault detection container allocated to the node to be detected to access the access object for each node to be detected, and to control the execution of the fault detection script in the node namespace to obtain the node fault detection result corresponding to the node to be detected;

[0116] The container detection module is used to, when a node fault detection result is obtained, use the container namespace of the container to be detected deployed in the node to be detected as an access object, and control the fault detection script to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected.

[0117] The above-mentioned cluster fault detection device can first monitor the events in the cluster, and when the creation event of the custom resource used to detect the cluster fault is monitored, the configuration information carried by the custom resource is read, so as to determine the nodes to be detected and the containers to be detected in the cluster according to the configuration information, and create a fault detection container according to the configuration information. Among them, the fault detection container has been configured with a fault detection script, and the nodes to be detected have been assigned to it, so that the detection of the nodes to be detected and the containers to be detected in the cluster can be automatically performed through the fault detection container in the future, thereby improving the efficiency of cluster fault detection. Specifically, for each node to be detected, the node namespace of the node to be detected can be used as an access object, the fault detection container assigned to the node to be detected can be controlled to access the access object, and the fault detection script can be controlled to be executed in the node namespace to obtain the node fault detection result corresponding to the node to be detected. Further, in the case of obtaining the node fault detection result, the container namespace of the container to be detected deployed in the node to be detected can be used as an access object, and the fault detection script can be controlled to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected. Based on this, by adopting the above-mentioned cluster fault detection method, the user does not need to perform fault detection on a large number of nodes or containers in the cluster one by one. Instead, the corresponding nodes to be detected and the containers to be detected deployed on the nodes to be detected can be automatically detected one by one through the fault detection container allocated to each node to be detected, thereby realizing efficient detection of cluster faults.

[0118] In one of the embodiments, the event monitoring module also includes: a list assembly unit, which is used to determine the target configuration information used to control the fault detection process from the configuration information, and assemble a workload resource list according to the target configuration information; and a detection container creation unit, which is used to create a fault detection container based on the workload resource list.

[0119] In one embodiment, the workload resource list includes the following target configuration information: quantity configuration information, permission configuration information, node affinity configuration information, and fault detection script. Based on this, the detection container creation unit is also used to: create multiple initial containers according to the quantity configuration information; for each initial container, configure the namespace access rights for the initial container according to the permission configuration information, allocate the node to be detected for the initial container according to the node affinity configuration information, and mount the fault detection script under the container directory of the initial container to obtain multiple fault detection containers.

[0120] In one embodiment, the workload resource list also includes the following target configuration information: container lifecycle configuration information. Based on this, the detection container creation unit is further used to: for each initial container, configure a cleanup delay time for the initial container according to the container lifecycle configuration information to obtain multiple fault detection containers.

[0121] In one of the embodiments, at least one container to be detected is deployed in each node to be detected. Based on this, the above cluster fault detection method also includes a fault detection container cleaning module, which is used to: for each node to be detected, when the container fault detection result of at least one container to be detected in the node to be detected has been obtained, write the node fault detection result and at least one container fault detection result corresponding to the node to be detected into the state information associated with the custom resource; determine the cleaning delay time of the fault detection container assigned to the node to be detected, and clean the fault detection container assigned to the node to be detected after waiting for the cleaning delay time.

[0122] In one of the embodiments, the cluster fault detection device also includes a detection result feedback module, which is used to: receive a cluster access authentication request triggered by a requester, and verify the authentication information carried in the cluster access authentication request; when the authentication information passes the verification, receive and execute a cluster detection result reading instruction triggered by the requester for the status information, and feedback the node fault detection results and container fault detection results stored in the status information.

[0123] Each module in the above cluster fault detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0124] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store cluster fault detection data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a cluster fault detection method is implemented.

[0125] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0127] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0130] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0132] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A cluster fault detection method, characterized in that: The method comprises: Monitor events in the cluster, and when a creation event of a custom resource for detecting cluster failures is detected, read configuration information carried by the custom resource; Determine the nodes to be detected and the containers to be detected in the cluster according to the configuration information, and create a fault detection container according to the configuration information; the fault detection container has been configured with a fault detection script and has been assigned the nodes to be detected; For each of the nodes to be detected, the node namespace of the node to be detected is used as an access object, a fault detection container allocated to the node to be detected is controlled to access the access object, and the fault detection script is controlled to be executed in the node namespace to obtain a node fault detection result corresponding to the node to be detected; When the node fault detection result is obtained, the container namespace of the container to be detected deployed in the node to be detected is used as the access object, and the fault detection script is controlled to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected.

2. The method according to claim 1, characterized in that The creating a fault detection container according to the configuration information includes: Determine target configuration information for controlling the fault detection process from the configuration information, and assemble a workload resource list according to the target configuration information; Based on the workload resource list, a fault detection container is created.

3. The method according to claim 2, characterized in that The workload resource list includes the following target configuration information: quantity configuration information, authority configuration information, node affinity configuration information, and fault detection scripts; The creating a fault detection container based on the workload resource list includes: Creating multiple initial containers according to the quantity configuration information; For each of the initial containers, namespace access permissions are configured for the initial container according to the permission configuration information, nodes to be detected are allocated to the initial container according to the node affinity configuration information, and the fault detection script is mounted under the container directory of the initial container to obtain multiple fault detection containers.

4. The method according to claim 3, characterized in that The workload resource list also includes the following target configuration information: container lifecycle configuration information; The step of mounting the fault detection script in the container directory of the initial container to obtain multiple fault detection containers includes: For each of the initial containers, a cleaning delay duration is configured for the initial container according to the container lifecycle configuration information to obtain multiple fault detection containers.

5. The method according to claim 1, characterized in that At least one container to be detected is deployed in each of the nodes to be detected; the method further includes: For each of the nodes to be detected, when a container fault detection result of at least one container to be detected in the node to be detected has been obtained, a node fault detection result and at least one container fault detection result corresponding to the node to be detected are written into the status information associated with the custom resource; Determine a cleaning delay time of the fault detection container allocated to the node to be detected, wait for the cleaning delay time, and then clean up the fault detection container allocated to the node to be detected.

6. The method according to claim 5, characterized in that The method further comprises: Receiving a cluster access authentication request triggered by a requester, and verifying the authentication information carried in the cluster access authentication request; When the authentication information passes the verification, the cluster detection result reading instruction triggered by the requesting party for the status information is received and executed, and the node fault detection result and the container fault detection result stored in the status information are fed back.

7. A cluster fault detection device, characterized in that: The device comprises: An event monitoring module is used to monitor events in the cluster, and when a creation event of a custom resource used to detect cluster failures is detected, the configuration information carried by the custom resource is read; A detection container creation module, used to determine the nodes to be detected and the containers to be detected in the cluster according to the configuration information, and to create a fault detection container according to the configuration information; the fault detection container has been configured with a fault detection script and has been assigned the nodes to be detected; A node detection module is used to, for each of the nodes to be detected, use the node namespace of the node to be detected as an access object, control the fault detection container allocated to the node to be detected to access the access object, and control the fault detection script to be executed in the node namespace to obtain a node fault detection result corresponding to the node to be detected; The container detection module is used to, when the node fault detection result is obtained, use the container namespace of the container to be detected deployed in the node to be detected as the access object, and control the fault detection script to be executed in the container namespace to obtain the container fault detection result corresponding to the node to be detected.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images