Fault handling method, apparatus, and computer program product

Abstract

The present disclosure provides a fault processing method and device, electronic equipment, storage medium and computer program product, relates to the technical field of computers, in particular to the technical field of Kubernetes, containerization, cloud platform and cloud computing, and can be used in the fault processing scenario of Kubernetes. The specific implementation scheme is as follows: based on the fault detection rule corresponding to the to-be-detected object in the preset container orchestration engine, it is determined whether the target fault is included in the to-be-detected object; in response to determining that the target fault is included in the to-be-detected object, based on the fault self-healing rule corresponding to the target fault, it is determined whether to repair the target fault; in response to determining to repair the target fault, the target fault is repaired through the self-healing operation executor corresponding to the to-be-detected object. The present disclosure improves the processing efficiency and automation degree of kubernetes faults.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the fields of Kubernetes, containerization, cloud platforms, and cloud computing, and particularly to fault handling methods, apparatus, electronic devices, storage media, and computer program products that can be used in fault handling scenarios of Kubernetes. Background Technology

[0002] Kubernetes is a widely used container orchestration and management tool in the industry. With the increasing popularity of Kubernetes, the industry's demand for its use is growing. A single cluster often struggles to handle the increasing workload, and high availability within a single cluster is difficult to guarantee. Disaster recovery requirements often necessitate a two-site, three-center architecture, gradually driving the need to deploy clusters across multiple regions and availability zones. Therefore, the number of Kubernetes clusters is gradually increasing (reaching hundreds), and the number of nodes managed under multiple clusters is also increasing (reaching hundreds of thousands). Consequently, intelligent operation and maintenance management of multiple clusters is crucial.

[0003] Currently, manual operation and maintenance management of Kubernetes clusters is often required. Summary of the Invention

[0004] This disclosure provides a fault handling method, apparatus, electronic device, storage medium, and computer program product.

[0005] According to the first aspect, a fault handling method is provided, comprising: determining whether the object to be detected includes a target fault based on the fault detection rules corresponding to the object to be detected in a preset container orchestration engine; in response to determining that the object to be detected includes a target fault, determining whether to repair the target fault based on the fault self-healing rules corresponding to the target fault; and in response to determining that the target fault needs to be repaired, repairing the target fault through the self-healing operation executor corresponding to the object to be detected.

[0006] According to a second aspect, a fault handling apparatus is provided, comprising: a detection unit configured to determine whether a target fault is included in the target object based on a fault detection rule corresponding to the target object in a preset container orchestration engine; a self-healing unit configured to, in response to determining that the target fault is included in the target object, determine whether to repair the target fault based on a fault self-healing rule corresponding to the target fault; and a repair unit configured to, in response to determining that the target fault needs to be repaired, repair the target fault through a self-healing operation executor corresponding to the target object.

[0007] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0008] According to a fourth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first aspect.

[0009] According to a fifth aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0010] According to the technology disclosed herein, a fault handling method for Kubernetes is provided. The method determines the target fault based on the fault detection rules corresponding to the object to be detected, determines whether to repair the fault based on the fault self-healing rules corresponding to the target fault, and then repairs the target fault through the self-healing operation executor corresponding to the object to be detected, thereby improving the efficiency and automation of Kubernetes fault handling.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure;

[0014] Figure 2 This is a flowchart of one embodiment of the fault handling method according to the present disclosure;

[0015] Figure 3 This is a schematic diagram of the structure of the preset container orchestration engine according to this embodiment;

[0016] Figure 4 This is a schematic diagram illustrating an application scenario of the fault handling method according to this embodiment;

[0017] Figure 5 This is a flowchart of yet another embodiment of the fault handling method according to the present disclosure;

[0018] Figure 6This is a structural diagram of one embodiment of the fault handling apparatus according to the present disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0022] Figure 1 An exemplary architecture 100 is shown that can be applied to the fault handling methods and apparatus of this disclosure.

[0023] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0024] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices supporting network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to image acquisition devices, voice acquisition devices, smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0025] Server 105 can be a server providing various services, such as a backend processing server providing container orchestration engine services to terminal devices 101, 102, and 103. The server can determine the target fault based on the fault detection rules corresponding to the object to be detected, determine whether to repair the fault based on the fault self-healing rules corresponding to the target fault, and then repair the target fault through the self-healing operation executor corresponding to the object to be detected. As an example, server 105 can be a cloud server.

[0026] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0027] It should also be noted that the fault handling method provided in the embodiments of this disclosure can be executed by a server, or it can be executed by the server and the terminal device in cooperation with each other. Accordingly, the various parts (e.g., various units) included in the fault handling device can all be set in the server, or they can be set in the server and the terminal device respectively.

[0028] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. When the electronic devices on which the fault handling method runs do not need to transmit data with other electronic devices, the system architecture may only include the electronic devices on which the fault handling method runs (e.g., servers or terminal devices).

[0029] Please refer to Figure 2 , Figure 2 A flowchart of a fault handling method provided in this disclosure embodiment, wherein process 200 includes the following steps:

[0030] Step 201: Based on the fault detection rules corresponding to the object to be detected in the preset container orchestration engine, determine whether the object to be detected includes the target fault.

[0031] In this embodiment, the entity executing the fault handling method (e.g., Figure 1 The terminal device or server in the container can determine whether the target fault is included in the object to be detected based on the fault detection rules corresponding to the object to be detected in the preset container orchestration engine.

[0032] Pre-defined container orchestration engines include, but are not limited to, Kubernetes, Docker Swarm, and Apache Mesos. Taking Kubernetes as an example, a single cluster size is often limited to under 5000 nodes, making it difficult to support gradually increasing business demands. Furthermore, high availability within a single cluster cannot be guaranteed, often necessitating disaster recovery requirements across two sites and three data centers. This has gradually spurred the need to deploy clusters across multiple regions and availability zones. Currently, the number of Kubernetes clusters is steadily increasing (reaching hundreds), and the number of nodes managed across multiple clusters is also growing (reaching hundreds of thousands).

[0033] In this embodiment, the object to be detected can be either a cluster within a multi-cluster environment or a node within a cluster. Corresponding fault detection rules can be pre-set for each object to be detected. Specifically, different fault detection rules can be set for different objects to be detected, or different fault detection rules can be set. These rules can be specifically set according to actual conditions (e.g., the running tasks and performance indicators of each object to be detected), and are not limited here.

[0034] Fault detection rules allow for real-time or periodic fault detection of the target object, determining whether it exhibits the target fault. Each target object can contain multiple rules, each capable of identifying one or more faults. Faults detected by a fault detection rule are considered the target faults corresponding to that rule.

[0035] As an example, a fault detection rule could be a script that extracts relevant parameters from logs generated during the operation of the object under test, or a script used to detect whether a certain running process of the object under test is running normally.

[0036] In some optional implementations of this embodiment, before performing step 201, the execution entity can determine the fault detection rule corresponding to the object to be detected based on the received detection rule registration operation.

[0037] Taking Kubernetes as an example, fault detection rules for the object to be detected can be registered through the detection rule manager in Kubernetes based on the received detection rule registration operation. It should be noted that this implementation is not limited to detection rule registration; the detection rule manager can also be used to modify, delete, and perform other editing operations on fault detection rules.

[0038] In this implementation, the aforementioned execution entity can flexibly register fault detection rules based on the received detection rule registration operation, thereby improving the flexibility and practicality of fault detection based on fault detection rules.

[0039] In some optional implementations of this embodiment, the aforementioned execution entity can register fault detection rules in the following manner:

[0040] Method 1: First, determine the fault detection rules based on the registration operation of the detection rules received by the management cluster in the preset container orchestration engine; then, synchronize the fault detection rules to the business cluster in the preset container orchestration engine.

[0041] Method 2: Register the fault detection rules based on the detection rules received by the business cluster to determine the fault detection rules.

[0042] like Figure 3 The diagram illustrates the structure of the preset container orchestration engine 300 in this implementation. The preset container orchestration engine 300 includes a management cluster 301 and multiple service clusters 302. The management cluster 301 includes a detection rule manager 3011 and a rule synchronizer 3012, and the service clusters 302 include a detection rule manager 3021. Corresponding to the first method described above, the detection rule manager 3011 can register fault detection rules based on a detection rule registration operation, and then synchronize the registered fault detection rules to the corresponding service clusters 302 through the rule synchronizer 3012. The rule synchronization operation can be performed on multiple service clusters or a single service cluster, depending on the instructions of the detection rule registration operation.

[0043] As an example, the detection rule registration operation can include the identification information of the target business cluster, and the rule synchronizer 3012 can synchronize the fault detection rule to the target business cluster corresponding to the identification information.

[0044] Corresponding to the second method described above, the detection rule manager 3021 in the service cluster 302 can determine the fault detection rule corresponding to the service cluster based on the detection rule registration operation.

[0045] In this embodiment, fault detection rules can be registered in multiple ways, further improving the flexibility of rule registration.

[0046] Step 202: In response to determining that the object to be detected includes the target fault, determine whether to repair the target fault based on the fault self-healing rule corresponding to the target fault.

[0047] In this embodiment, the execution entity can determine whether to repair the target fault based on the fault self-healing rule corresponding to the target fault, in response to determining that the object to be detected includes the target fault.

[0048] The fault self-healing rule is used to determine whether the corresponding target fault needs to be repaired. When it is determined that the target fault needs to be repaired, the repair operation for the target fault is triggered; when it is determined that the target fault does not need to be repaired, the target fault is ignored.

[0049] In this embodiment, the objects targeted by the fault self-healing rules can be further subdivided. For example, corresponding fault self-healing rules are set for target faults in different objects to be detected. As an example, depending on the actual situation, different fault self-healing rules can be set for target faults in different objects to be detected.

[0050] In some optional implementations of this embodiment, before performing step 202, the execution entity can determine the fault self-healing rule corresponding to the target fault based on the received self-healing rule registration operation.

[0051] Taking Kubernetes as an example, the self-healing rule manager of the object to be inspected in Kubernetes can register the corresponding fault self-healing rules for the object to be inspected based on the received self-healing rule registration operation. It should be noted that this implementation is not limited to the self-healing rule registration operation. Through the self-healing rule manager, it is also possible to perform editing operations such as modification and deletion of fault self-healing rules.

[0052] In this implementation, the aforementioned execution entity can flexibly register fault self-healing rules based on the received self-healing rule registration operation, thereby improving the flexibility and practicality of determining whether to repair the target fault based on the fault self-healing rules.

[0053] In some optional implementations of this embodiment, the aforementioned execution entity can register the fault self-healing rules in the following manner:

[0054] Method 1: First, determine the fault self-healing rules based on the self-healing rule registration operation received by the management cluster in the preset container orchestration engine; then, synchronize the fault self-healing rules to the business cluster in the preset container orchestration engine.

[0055] Method 2: Determine the fault self-healing rules based on the self-healing rule registration operation received by the business cluster.

[0056] Continue to refer to Figure 3The control cluster 301 also includes a self-healing rule manager 3013, and the business cluster 302 includes a self-healing rule manager 3022. Corresponding to Method 1 above, the self-healing rule manager 3013 can register fault self-healing rules based on the self-healing rule registration operation, and then synchronize the registered fault self-healing rules to the corresponding business cluster 302 through the rule synchronizer 3012. Corresponding to Method 2 above, the self-healing rule manager 3022 in the business cluster 302 can determine the fault self-healing rules based on the self-healing rule registration operation.

[0057] In this embodiment, the registration of fault self-healing rules can be achieved in multiple ways, further improving the flexibility of rule registration.

[0058] Fault detection and self-healing operations are divided into cluster level and node level. At the cluster level, new pods can be started in the cluster to perform detection and self-healing operations. At the node level, detection and self-healing operations can be performed through fixed pods on each node, thus making the coverage of fault repair wider.

[0059] Step 203: In response to determining the target fault to be repaired, the target fault is repaired by the self-healing actuator corresponding to the object to be tested.

[0060] In this embodiment, the aforementioned execution entity can, in response to determining the target fault to be repaired, repair the target fault through the self-healing operation actuator corresponding to the object to be tested.

[0061] In this embodiment, each object to be tested includes a corresponding self-healing executor. The self-healing executor is used to handle the identified target fault and can be deployed in each object to be tested as a component or plug-in to flexibly expand the self-healing executor.

[0062] Based on cloud-native design, fault detection, self-healing, and repair operations can all be performed through Kubernetes. Managing Kubernetes through Kubernetes results in lower operational costs.

[0063] In some optional implementations of this embodiment, the execution entity can perform step 203 as follows:

[0064] First, in response to identifying the target fault for repair, self-healing metadata is generated based on the fault self-healing rules.

[0065] As an example, the self-healing metadata includes information such as the self-healing operation executor that performs fault repair and information about the target fault to be repaired.

[0066] Second, based on the self-healing metadata, the target fault is repaired by the self-healing operation executor corresponding to the object to be detected.

[0067] According to the instructions of the self-healing metadata, the aforementioned execution entity can repair the target fault through the self-healing operation executor corresponding to the indicated object to be detected.

[0068] This implementation provides a specific method for repairing the target fault, thereby improving the efficiency of handling the target fault.

[0069] In some optional implementations of this embodiment, the execution entity can perform the second step as follows:

[0070] In response to the determination that the object to be detected is a business cluster in the preset container orchestration engine, the target fault is repaired based on the self-healing metadata through the self-healing operation executor corresponding to the business cluster.

[0071] In response to the determination that the object to be detected is a node in the preset container orchestration engine, the target fault is repaired based on the self-healing metadata through the self-healing operation executor corresponding to the node.

[0072] Continue to refer to Figure 3 Each business cluster 302 includes a cluster-level self-healing rule executor 3023, multiple nodes, and a self-healing rule executor included in each node.

[0073] In this implementation, the object to be detected can be a business cluster within a multi-cluster system or a node within a business cluster. Regardless of whether it's a cluster or a node, a corresponding self-healing executor is configured. This implementation allows the aforementioned executor to perform cluster-level and node-level fault repair, improving the practicality of fault repair.

[0074] See also Figure 4 , Figure 4 This is a schematic diagram 400 illustrating an application scenario of the fault handling method according to this embodiment. Figure 4 In the application scenario, based on the fault detection rule 403 corresponding to the object to be detected 402 in the preset container orchestration engine 401, it is determined whether the object to be detected 402 includes the target fault 404; in response to determining that the object to be detected includes the target fault 404, based on the fault self-healing rule 405 corresponding to the target fault, it is determined whether to repair the target fault 404; in response to determining to repair the target fault 404, the target fault 404 is repaired through the self-healing operation executor 406 corresponding to the object to be detected 402.

[0075] This embodiment provides a fault handling method for Kubernetes. The method determines the target fault based on the fault detection rules corresponding to the object to be detected, determines whether to repair the fault based on the fault self-healing rules corresponding to the target fault, and then repairs the target fault through the self-healing operation executor corresponding to the object to be detected, thereby improving the efficiency and automation of Kubernetes fault handling.

[0076] Continue to refer to Figure 5 The illustration shows a schematic flow 500 of an embodiment of a fault handling method according to the present disclosure, including the following steps:

[0077] Step 501: Based on the received detection rule registration operation, determine the fault detection rule corresponding to the object to be detected.

[0078] Step 502: Based on the received self-healing rule registration operation, determine the fault self-healing rule corresponding to the target fault.

[0079] Step 503: Based on the fault detection rules corresponding to the object to be detected in the preset container orchestration engine, determine whether the object to be detected includes the target fault.

[0080] Step 504: In response to determining that the object to be detected includes the target fault, determine whether to repair the target fault based on the fault self-healing rule corresponding to the target fault.

[0081] Step 505: In response to identifying the target fault for repair, generate self-healing metadata based on the fault self-healing rules.

[0082] Step 506: Based on the self-healing metadata, repair the target fault through the self-healing operation executor corresponding to the object to be detected.

[0083] As can be seen from this embodiment, with Figure 2 Compared with the corresponding embodiments, the fault handling method in this embodiment specifically illustrates the registration process of fault detection rules and fault self-healing rules, as well as the repair process of the target fault, which further improves the flexibility and practicality of rule registration and the efficiency of fault repair.

[0084] Continue to refer to Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a fault handling device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0085] like Figure 6As shown, the fault handling device includes: a detection unit 601, configured to determine whether the object to be detected contains a target fault based on the fault detection rules corresponding to the object to be detected in a preset container orchestration engine; a self-healing unit 602, configured to determine whether to repair the target fault based on the fault self-healing rules corresponding to the target fault in response to determining that the object to be detected contains a target fault; and a repair unit 603, configured to repair the target fault through the self-healing operation executor corresponding to the object to be detected in response to determining that the target fault needs to be repaired.

[0086] In some optional implementations of this embodiment, the repair unit 603 is further configured to: in response to determining the target fault to be repaired, generate self-healing metadata according to the fault self-healing rules; and repair the target fault through the self-healing operation executor corresponding to the object to be detected according to the self-healing metadata.

[0087] In some optional implementations of this embodiment, the repair unit 603 is further configured to: in response to determining that the object to be detected is a business cluster in a preset container orchestration engine, repair the target fault according to the self-healing metadata through the self-healing operation executor corresponding to the business cluster; in response to determining that the object to be detected is a node in a preset container orchestration engine, repair the target fault according to the self-healing metadata through the self-healing operation executor corresponding to the node.

[0088] In some optional implementations of this embodiment, the above-mentioned device further includes: a detection rule registration unit (not shown in the figure), configured to: determine the fault detection rule corresponding to the object to be detected according to the received detection rule registration operation.

[0089] In some optional implementations of this embodiment, the detection rule registration unit (not shown in the figure) is further configured to: determine fault detection rules according to the detection rule registration operation received by the management cluster in the preset container orchestration engine; synchronize the fault detection rules to the service cluster in the preset container orchestration engine; or determine fault detection rules according to the detection rule registration operation received by the service cluster.

[0090] In some optional implementations of this embodiment, the above apparatus further includes: a self-healing rule registration unit (not shown in the figure), configured to: determine the fault self-healing rule corresponding to the target fault based on the received self-healing rule registration operation.

[0091] In some optional implementations of this embodiment, the self-healing rule registration unit (not shown in the figure) is further configured to: determine fault self-healing rules according to the self-healing rule registration operation received by the management cluster in the preset container orchestration engine; synchronize the fault self-healing rules to the service cluster in the preset container orchestration engine; or determine fault self-healing rules according to the self-healing rule registration operation received by the service cluster.

[0092] In this embodiment, a fault handling device for Kubernetes is provided. The device determines the target fault based on the fault detection rules corresponding to the object to be detected, determines whether to repair the fault based on the fault self-healing rules corresponding to the target fault, and then repairs the target fault through the self-healing operation executor corresponding to the object to be detected, thereby improving the efficiency and automation of Kubernetes fault handling.

[0093] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the fault handling method described in any of the above embodiments when executed.

[0094] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the fault handling method described in any of the above embodiments when executed.

[0095] This disclosure provides a computer program product that, when executed by a processor, can implement the fault handling method described in any of the above embodiments.

[0096] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0097] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0098] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0099] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as fault handling methods. For example, in some embodiments, the fault handling method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the fault handling method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform fault handling methods by any other suitable means (e.g., by means of firmware).

[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0105] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers integrated with blockchain technology.

[0106] According to the technical solution of the embodiments of this disclosure, a fault handling method for Kubernetes is provided. The method determines the target fault according to the fault detection rules corresponding to the object to be detected, determines whether to repair the fault according to the fault self-healing rules corresponding to the target fault, and then repairs the target fault through the self-healing operation executor corresponding to the object to be detected, thereby improving the efficiency and automation of Kubernetes fault handling.

[0107] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0108] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A fault processing method, comprising: determining whether a target fault is included in a to-be-detected object based on a fault detection rule corresponding to the to-be-detected object in a preset container orchestration engine; in response to determining that the target fault is included in the to-be-detected object, determining whether to repair the target fault based on a fault self-recovery rule corresponding to the target fault; in response to determining to repair the target fault, repairing the target fault by a self-recovery operation executor corresponding to the to-be-detected object; wherein the management and control cluster of the preset container orchestration engine is provided with a detection rule manager for managing fault detection rules corresponding to a plurality of business clusters in the preset container orchestration engine, a self-recovery rule manager for managing fault self-recovery rules of the plurality of business clusters, and a rule synchronizer for synchronizing fault detection rules and fault self-recovery rules to the business clusters; the business cluster includes a detection rule manager for managing fault detection rules corresponding to the business cluster, a detection operation executor for detecting target faults at a cluster level, a self-recovery rule manager for managing fault self-recovery rules corresponding to the business cluster, and a self-recovery operation executor for repairing target faults at a cluster level; the nodes of the business cluster include a detection operation executor for detecting target faults at a node level and a self-recovery operation executor for repairing target faults at a node level.

2. The method of claim 1, wherein, in response to determining to repair the target fault, generating self-recovery metadata according to the fault self-recovery rule; repairing the target fault by the self-recovery operation executor corresponding to the to-be-detected object according to the self-recovery metadata. in response to determining that the to-be-detected object is a business cluster in the preset container orchestration engine, repairing the target fault according to the self-recovery metadata by the self-recovery operation executor corresponding to the business cluster; 3. The method of claim 2, wherein, in response to determining that the to-be-detected object is a node in the preset container orchestration engine, repairing the target fault according to the self-recovery metadata by the self-recovery operation executor corresponding to the node. before the determination of whether the target fault is included in the to-be-detected object based on the fault detection rule corresponding to the to-be-detected object in the preset container orchestration engine, further comprising: determining the fault detection rule corresponding to the to-be-detected object according to a received detection rule registration operation.

4. The method of any one of claims 1-3, wherein, determining the fault detection rule corresponding to the to-be-detected object according to a received detection rule registration operation, comprising: determining the fault detection rule according to a detection rule registration operation received by the management and control cluster in the preset container orchestration engine; 5. The method of claim 4, wherein, synchronizing the fault detection rule to a business cluster in the preset container orchestration engine; or determining the fault detection rule according to a detection rule registration operation received by the business cluster. ​ ​ 6. The method of any one of claims 1-3, wherein, Before the determining whether to repair the target fault based on the fault self-recovery rule corresponding to the target fault, further comprising: According to the received self-recovery rule registration operation, determining the fault self-recovery rule corresponding to the target fault.

7. The method of claim 6, wherein, The determining the fault self-recovery rule corresponding to the target fault according to the received self-recovery rule registration operation comprises: According to the self-recovery rule registration operation received by the management and control cluster in the preset container orchestration engine, determining the fault self-recovery rule; Synchronizing the fault self-recovery rule to the business cluster in the preset container orchestration engine; or According to the self-recovery rule registration operation received by the business cluster, determining the fault self-recovery rule.

8. A fault processing apparatus, comprising: A detection unit configured to determine whether a target fault is included in a to-be-detected object based on a fault detection rule corresponding to the to-be-detected object in a preset container orchestration engine; A self-recovery unit configured to, in response to determining that a target fault is included in the to-be-detected object, determine whether to repair the target fault based on a fault self-recovery rule corresponding to the target fault; A repair unit configured to, in response to determining to repair the target fault, repair the target fault by a self-recovery operation executor corresponding to the to-be-detected object; The management and control cluster of the preset container orchestration engine is provided with a detection rule manager for managing fault detection rules corresponding to a plurality of business clusters in the preset container orchestration engine, a self-recovery rule manager for managing fault self-recovery rules of the plurality of business clusters, and a rule synchronizer for synchronizing fault detection rules and fault self-recovery rules to the business clusters; the business cluster includes a detection rule manager for managing fault detection rules corresponding to the business cluster, a detection operation executor for detecting target faults at the cluster level, a self-recovery rule manager for managing fault self-recovery rules corresponding to the business cluster, and a self-recovery operation executor for repairing target faults at the cluster level; the nodes of the business cluster include a detection operation executor for detecting target faults at the node level and a self-recovery operation executor for repairing target faults at the node level.

9. The apparatus of claim 8, wherein, The repair unit is further configured to: In response to determining to repair the target fault, generate self-recovery metadata according to the fault self-recovery rule; and repair the target fault by the self-recovery operation executor corresponding to the to-be-detected object according to the self-recovery metadata.

10. The apparatus of claim 9, wherein, The repair unit is further configured to: In response to determining that the to-be-detected object is a business cluster in the preset container orchestration engine, repair the target fault by the self-recovery operation executor corresponding to the business cluster according to the self-recovery metadata; In response to determining that the to-be-detected object is a node in the preset container orchestration engine, repair the target fault by the self-recovery operation executor corresponding to the node according to the self-recovery metadata.

11. The apparatus of any one of claims 8-10, wherein, Further comprising: A detection rule registration unit configured to: According to the received detection rule registration operation, determine the fault detection rule corresponding to the to-be-detected object.

12. The apparatus of claim 11, wherein, The detection rule registration unit is further configured to: According to a detection rule registration operation received by the management cluster in the preset container orchestration engine, the fault detection rule is determined; The fault detection rule is synchronized to a business cluster in the preset container orchestration engine; Or According to a detection rule registration operation received by the business cluster, the fault detection rule is determined.

13. The apparatus of any one of claims 8-10, wherein, Further comprising: A self-recovery rule registration unit configured to: According to a received self-recovery rule registration operation, the fault self-recovery rule corresponding to the target fault is determined.

14. The apparatus of claim 13, wherein, The self-recovery rule registration unit is further configured to: According to a self-recovery rule registration operation received by the management cluster in the preset container orchestration engine, the fault self-recovery rule is determined; and the fault self-recovery rule is synchronized to a business cluster in the preset container orchestration engine; Or According to a self-recovery rule registration operation received by the business cluster, the fault self-recovery rule is determined.

15. An electronic device, comprising: Comprise: At least one processor; And The memory is in communication connection with the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to execute the method of any one of claims 1-7.

17. A computer program product, comprising: A computer program, when executed by a processor, implements the method according to any one of claims 1-7.

Images