Cluster fault processing method and system, electronic device and storage medium

Abstract

The invention relates to a cluster fault processing method and system, an electronic device and a storage medium, and the method comprises the steps: determining a fault propagation link of an alarm after the alarm of target equipment in a target cluster is monitored, and determining all related nodes of the alarm; if there is a node corresponding to the node fault resource, adding the alarm information to an associated alarm list of the corresponding node fault resource; determining the sum of comprehensive influence factors of all the associated alarms on the nodes without corresponding node fault resources; and when the sum of the comprehensive influence factors is greater than a node fault threshold value, creating corresponding node fault resources, adding alarm information to an associated alarm list of the corresponding node fault resources, and performing fault processing on nodes without the corresponding node fault resources according to the workflow of the target cluster. According to the invention, the problem of high operation and maintenance cost is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, container orchestration and operation and maintenance automation, and particularly relates to a cluster fault processing method and system, an electronic device and a storage medium. BACKGROUND

[0002] In a Kubernetes (K8s for short) platform, a bottom problem such as excessive node CPU pressure, network partition, kernel deadlock, etc. can easily trigger an "avalanche effect", resulting in a large number of associated alarms in a very short time.

[0003] The prior art solution needs to manually locate the root cause, assess the business impact and manually repair the fault, which is highly dependent on expert experience and has the problem of high operation and maintenance cost.

[0004] Therefore, an effective solution has not yet been proposed for the problem of high operation and maintenance cost. SUMMARY

[0005] A cluster fault processing method, system, electronic device and storage medium are provided in the embodiment to solve the problem of high operation and maintenance cost in the related art.

[0006] In a first aspect, a cluster fault processing method is provided in the embodiment, and the method comprises:

[0007] After an alarm of a target device in a target cluster is monitored, a fault propagation link of the alarm of the target device is determined; the alarm is a K8s custom resource;

[0008] According to the fault propagation link, all associated nodes of the target device alarm are determined;

[0009] It is determined whether each associated node has a corresponding node fault resource; the associated node having the node fault resource is identified as a first associated node; the associated node not having the node fault resource is identified as a second associated node; the node fault resource is a K8s custom resource;

[0010] For the first associated node, the alarm information of the target device is added to the associated alarm list of the node fault resource of the first associated node; the alarm information includes an alarm name, an alarm state and an alarm impact factor;

[0011] For the second associated node, according to all alarms associated with the second associated node, a total sum of comprehensive impact factors of the all alarms on the second associated node is determined;

[0012] When the comprehensive influence factor sum is greater than a preset node fault threshold, a node fault resource corresponding to the second associated node is created, and the alarm information of the target device is added to an associated alarm list of the node fault resource of the second associated node, and the second associated node is processed according to a preset workflow template.

[0013] In some embodiments, determining the fault propagation link of the alarm of the target device comprises:

[0014] Obtaining a propagation link resource of the target cluster; the propagation link resource comprises a propagation path of the target cluster and a propagation strategy of the target cluster; the propagation link resource is a K8s custom resource.

[0015] Traversing the propagation path of the target cluster, and determining the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster.

[0016] In some embodiments, traversing the propagation path of the target cluster, and determining the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster comprises:

[0017] Traversing the propagation path of the target cluster, and determining the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster based on a breadth-first search algorithm and a graph traversal algorithm and by using a device topology relationship graph of the target cluster.

[0018] In some embodiments, determining all associated nodes of the target device alarm according to the fault propagation link comprises:

[0019] Determining all associated nodes of the target device alarm according to a device type in the fault propagation link of the alarm of the target device.

[0020] In some embodiments, determining a comprehensive influence factor sum of all alarms on the second associated node according to all alarms associated with the second associated node comprises:

[0021] According to all alarms associated with the second associated node, obtaining a comprehensive influence factor of each of the all alarms on the second associated node by using the propagation link resource of the target cluster.

[0022] Adding the comprehensive influence factors of the all alarms on the second associated node to obtain the comprehensive influence factor sum of the all alarms on the second associated node.

[0023] In some embodiments, the method further comprises:

[0024] After monitoring that the alarm is deleted, all associated nodes corresponding to the alarm are determined; the nodes associated with the alarm are identified as third associated nodes;

[0025] The state corresponding to the associated alarm in the node fault resource corresponding to the third associated node is modified to be recovered, and the sum of the comprehensive influence factors of the associated alarms with the state of unrecovered in the node fault resource corresponding to the third associated node on the third associated node is determined;

[0026] When the sum of the comprehensive influence factors is less than the node fault threshold, the third associated node is subjected to recovery verification;

[0027] After the third associated node passes the recovery verification, the alarm corresponding to the associated alarm with the state of unrecovered in the node fault resource corresponding to the third associated node is deleted, the node fault resource corresponding to the third associated node is deleted, and a message of the third associated node fault recovery is sent to the associated business system.

[0028] In some embodiments, the recovery verification of the third associated node comprises:

[0029] The associated alarm verification of the third associated node is performed; the associated alarm verification comprises judging whether the sum of the comprehensive influence factors is less than the node fault threshold;

[0030] If the associated alarm verification passes, the health check verification of the third associated node is performed; the health check verification comprises performing health check on the third associated node.

[0031] In a second aspect, a cluster fault processing system is provided in the present embodiment, comprising an alarm processing module and a node fault processing module:

[0032] The alarm processing module is configured to, after monitoring an alarm of a target device in a target cluster, determine a fault propagation link of the alarm of the target device, determine all associated nodes of the target device according to the fault propagation link, determine whether a corresponding node fault resource exists in each associated node, identify an associated node with the node fault resource as a first associated node, identify an associated node without the node fault resource as a second associated node, add the alarm information of the target device to an associated alarm list of the node fault resource of the first associated node, for the first associated node, determine a total sum of comprehensive influence factors of all alarms associated with the second associated node on the second associated node, and when the total sum of the comprehensive influence factors is greater than a preset node fault threshold, create a node fault resource corresponding to the second associated node, and add the alarm information of the target device to an associated alarm list of the node fault resource of the second associated node.

[0033] The node fault processing module is configured to perform fault processing on the second associated node according to a preset workflow template.

[0034] In a third aspect, an electronic device is provided in the embodiment, which includes a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the cluster fault processing method in the first aspect.

[0035] In a fourth aspect, a computer readable storage medium is provided in the embodiment, which stores a computer program, and the computer program is executed by a processor to implement the cluster fault processing method in the first aspect.

[0036] Compared with the related art, the cluster fault processing method, system, electronic device and storage medium are provided in the embodiment. The cluster fault processing method comprises: after monitoring an alarm of a target device in a target cluster, determining a fault propagation link of the alarm of the target device; determining all associated nodes of the alarm of the target device according to the fault propagation link; determining whether a corresponding node fault resource exists in each associated node; identifying the associated node with the node fault resource as a first associated node; identifying the associated node without the node fault resource as a second associated node; for the first associated node, adding the alarm information of the target device to an associated alarm list of the node fault resource of the first associated node; for the second associated node, determining a comprehensive influence factor sum of all alarms associated with the second associated node on the second associated node according to the all alarms; when the comprehensive influence factor sum is greater than a preset node fault threshold, creating a node fault resource corresponding to the second associated node, and adding the alarm information of the target device to an associated alarm list of the node fault resource of the second associated node, and performing fault processing on the second associated node according to a workflow of the target cluster. The cluster fault processing method can analyze the fault influence range of the alarm, combine all alarms on the influence node, and process the fault node in combination with the workflow, thereby solving the problem of high operation and maintenance cost.

[0037] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate exemplary embodiments of the present application and its description, which serve to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 is a terminal hardware structure block diagram of the cluster fault processing method of the embodiment;

[0040] Figure 2 is a flowchart of the cluster fault processing method of the embodiment;

[0041] Figure 3 is a flowchart of the cluster fault processing method of some of the embodiments;

[0042] Figure 4 is a flowchart of the recovery check in the cluster fault processing method of the embodiment;

[0043] Figure 5 is a structure schematic diagram of the cluster fault processing system of the embodiment. DETAILED DESCRIPTION

[0044] For more clearly understanding the purpose, technical solutions and advantages of the present application, the present application is described and explained below in connection with the accompanying drawings and embodiments.

[0045] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the common meaning understood by one of ordinary skill in the art to which the present application pertains. In the present application, the terms "one", "a", "an", "the", "these", and similar words do not indicate quantity, and they can be singular or plural. In the present application, the terms "include", "contain", "have", and any variants thereof are intended to cover non-exclusive inclusion; for example, a process, method, and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and similar words do not limit to physical or mechanical connection, but can include electrical connection, whether direct or indirect. In the present application, "multiple" means two or more. The term "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. Generally, the character " / " represents an "or" relationship between the associated objects. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0046] The method embodiments provided in the present embodiment can be executed in a terminal, a computer or similar electronic device with certain computing power. For example, the method embodiments are executed on a terminal, Figure 1 is a hardware structure diagram of the terminal of the cluster fault processing method of the present embodiment. As shown in Figure 1 , the terminal can include one or more (only one is shown in Figure 1 ) processor 102 and memory 104 for storing data, wherein the processor 102 can include but not limited to processing devices such as microprocessor MCU or programmable logic device FPGA. The above terminal can also include transmission device 106 for communication function and input / output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above terminal. For example, the terminal can also include more or less components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0047] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as the computer program corresponding to the cluster fault processing method in the embodiment. The processor 102 performs various functional applications and data processing, i.e., implements the method described above, by running the computer programs stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include memories remotely arranged with respect to the processor 102, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0048] The transmission device 106 is used to receive or send data via a network. The network described above includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network interface controller (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0049] In the embodiment, a cluster fault processing method is provided, Figure 2 is a flowchart of the cluster fault processing method of the embodiment, as shown in Figure 2 The flowchart includes the following steps:

[0050] In step S210, after monitoring the alarm of the target device in the target cluster, the fault propagation link of the alarm of the target device is determined.

[0051] The target device can be a node, i.e., a server, a GPU, a container, a switch, a CPU, a disk, or the like physical hardware. The alarm is a self-defined K8s resource, and the definition can include fields such as alarm basic information, cluster name, state, fault link, and associated node, wherein the alarm basic information includes identification information of the device, type of the device, and alarm level. The alarm of the target device in the target cluster can be created by a business system according to the alarm notification of the target device in the target cluster reported by a monitoring component, and the alarm of the alarm notification of the target device in the target cluster is created.

[0052] Specifically, after monitoring the alarm of the target device in the target cluster created by the business system, according to the alarm basic information in the alarm, the corresponding alarm is generated, and the state of the alarm is set to initialization, and the fault propagation link of the alarm of the target device is determined. The alarm basic information includes the identification information of the target device, the type of the target device, and the alarm level. The alarm includes the identification information of the target device, the type of the target device, and the alarm level. The fault propagation link includes the propagation events of all associated devices of the alarm of the target device.

[0053] Step S220, according to the fault propagation link, determine all associated nodes of the target device alarm.

[0054] The fault propagation link includes the propagation events of all associated devices of the alarm of the target device, wherein the type of the associated device includes the node, i.e. server, GPU, container, switch, CPU, disk and other physical hardware, and the propagation event includes device identification information, device type and comprehensive influence factor.

[0055] Specifically, the fault propagation link of the alarm of the target device is determined, the propagation events of all associated devices of the alarm of the target device are determined according to the fault propagation link, the type of the associated device in the propagation event is the node, and all the propagation events of the associated device with the type of the node are added to the associated node list of the alarm, and the state of the alarm is set to running.

[0056] Step S230, determine whether each associated node has a corresponding node fault resource; identify the associated node with the node fault resource as the first associated node; identify the associated node without the node fault resource as the second associated node.

[0057] The node fault resource is a self-defined K8s resource, which can include node identification information, state, associated alarm and other fields in the definition, wherein the associated alarm includes associated alarm information and associated alarm state. Specifically, the associated nodes of the alarm are traversed, whether each associated node has a corresponding node fault resource is determined according to the identification information of the associated node, the associated node with the node fault resource is identified as the first associated node, and the associated node without the node fault resource is identified as the second associated node.

[0058] Step S240, for the first associated node, the alarm information of the target device is added to the associated alarm list of the node fault resource of the first associated node; for the second associated node, according to all alarms associated with the second associated node, the comprehensive influence factor sum of all alarms on the second associated node is determined.

[0059] Specifically, for the first associated node, the alarm information of the target device is added to the associated alarm list of the node fault resource of the first associated node. For the second associated node, all alarms associated with the second associated node are obtained through the identification information of the second associated node, and then the total sum of the comprehensive influence factors of all alarms on the second associated node is determined.

[0060] In step S250, when the total sum of the comprehensive influence factors is greater than the preset node fault threshold, a node fault resource corresponding to the second associated node is created, and the alarm information of the target device is added to the associated alarm list of the node fault resource of the second associated node. According to the preset workflow template, the second associated node is processed for fault.

[0061] The fault processing can be to give the node a stain label, to backup and migrate the service on the node, etc. Specifically, when the total sum of the comprehensive influence factors is greater than the preset node fault threshold, a node fault resource corresponding to the second associated node is created, the state of the node fault resource is set to initialization, and the alarm information of the target device is added to the associated alarm list of the node fault resource of the second associated node. According to the preset workflow template, the second associated node is processed for fault, and the state of the node fault resource is set to running.

[0062] Through the above steps S210 to S250, compared with the related art, the embodiment determines the fault propagation link of the alarm of the target device after monitoring the alarm of the target device in the target cluster; determines all associated nodes of the target device alarm according to the fault propagation link; determines whether each associated node has a corresponding node fault resource; identifies the associated node with the node fault resource as the first associated node; identifies the associated node without the node fault resource as the second associated node; for the first associated node, the alarm information of the target device is added to the associated alarm list of the node fault resource of the first associated node; for the second associated node, the total sum of the comprehensive influence factors of all alarms on the second associated node is determined according to all alarms associated with the second associated node; when the total sum of the comprehensive influence factors is greater than the preset node fault threshold, a node fault resource corresponding to the second associated node is created, and the alarm information of the target device is added to the associated alarm list of the node fault resource of the second associated node. According to the workflow of the target cluster, the second associated node is processed for fault. It can analyze the fault influence range of the alarm, combine all alarms on the influence node, and process the fault node combined with the workflow, solving the problem of high operation and maintenance cost.

[0063] In one embodiment, determining the fault propagation link of the alarm of the target device comprises:

[0064] obtaining a propagation link resource of the target cluster; the propagation link resource comprises a propagation path of the target cluster and a propagation strategy of the target cluster; and traversing the propagation path of the target cluster, and determining a fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster.

[0065] The propagation link resource is a self-defined K8s resource, and the definition can include fields such as cluster name, propagation path, and propagation strategy. The propagation path includes a starting device type, an ending device type, and a propagation factor, and the propagation factor represents a probability of fault propagation along the path, ranging from 0 to 1. The propagation strategy can be a dynamic propagation factor strategy or a static propagation factor strategy, wherein the dynamic propagation factor strategy calculates the propagation factor using the performance indicators of the current device, and the static propagation factor strategy uses the propagation factor of the propagation path. The propagation link resource can be created by a user, and the user can create different propagation link resources according to different clusters. Different propagation link resources are stored in the propagation link cache by the propagation link controller, and have the advantages of convenient expansion, no need for training data, and immediate effect.

[0066] Specifically, according to the name of the target cluster, the propagation link resource of the target cluster pre-stored in the propagation link cache is obtained. The propagation link resource comprises a propagation path of the target cluster and a propagation strategy of the target cluster. The propagation path of the target cluster can include different starting device types, ending device types, and propagation factors. For example, the starting device type is a node, the ending device type is a graphics processor, and the propagation factor is 1.0; the starting device type is a graphics processor, the ending device type is a container, and the propagation factor is 0.8; and the starting device type is a container, the ending device type is a controller for managing the container, and the propagation factor is 0.6. According to the propagation path of the target cluster, the corresponding ending device type and the corresponding propagation factor of the target device can be obtained. According to the propagation strategy of the target cluster, a comprehensive influence factor of the alarm of the target device on the ending device can be obtained. The propagation path of the target cluster is traversed, and a set of propagation events of all associated devices of the alarm of the target device is obtained according to the propagation strategy of the target cluster.

[0067] In one embodiment, traversing the propagation path of the target cluster, and determining the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster comprises:

[0068] Traversing the propagation path of the target cluster, and determining the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster based on a breadth-first search algorithm and a graph traversal algorithm and using a device topology relationship graph of the target cluster.

[0069] Specifically, according to the propagation path of the target cluster, the end device type corresponding to the current device type and the corresponding propagation factor can be obtained. Based on the breadth-first search algorithm and the graph traversal algorithm, the identification information of the end device can be obtained by using the device topology relationship graph of the target cluster. According to the propagation strategy of the target cluster, the comprehensive influence factor of the current device on the end device can be obtained, so as to obtain the propagation event of the end device. The end device is identified as the current device, the propagation event of the end device is obtained, and the recursive call is performed in this way until the propagation path of the target cluster is traversed. Thus, the fault propagation link of the alarm of the target device is determined.

[0070] For example, the alarm of the target device: the target device identification information is 001, the target device type is node, and the alarm level is serious; the propagation path of the target cluster: the start device type is node, the end device type is graphics processor, and the propagation factor is 1.0; the start device type is graphics processor, the end device type is container, and the propagation factor is 0.8; the start device type is container, the end device type is the controller of the management container, and the propagation factor is 0.6; the propagation strategy of the target cluster: static propagation factor strategy. The alarm influence factor is obtained by quantifying the alarm level, which is 80. According to the propagation path of the target cluster, the end device type corresponding to the target device is graphics processor, and the corresponding propagation factor is 1.0. Based on the breadth-first search algorithm and the graph traversal algorithm, the identification information of all end devices corresponding to the target device is obtained by using the device topology relationship graph of the target cluster. According to the static propagation factor strategy, the comprehensive influence factor of the alarm of the target device on the end device is obtained, which is the alarm influence factor 80 multiplied by the corresponding propagation factor 1.0. Thus, the propagation event of all end devices corresponding to the target device is obtained. The propagation path of the target cluster is traversed, the end device type corresponding to all end devices is container, and the corresponding propagation factor is 0.8. Based on the breadth-first search algorithm and the graph traversal algorithm, the identification information of all end devices corresponding to all end devices is obtained by using the device topology relationship graph of the target cluster. According to the static propagation factor strategy, the comprehensive influence factor of the propagation event of the end device on the corresponding end device is obtained, which is the comprehensive influence factor of the alarm of the target device on the end device 80 multiplied by the corresponding propagation factor 0.8. Thus, the propagation event of all end devices corresponding to all end devices is obtained. In this way, the propagation event of all associated devices of the alarm of the target device, i.e., the fault propagation link of the alarm, is obtained by traversing the propagation path of the target cluster.

[0071] In one embodiment, according to all alarms associated with the second associated node, the sum of the comprehensive influence factors of all alarms on the second associated node is determined, including:

[0072] Based on all alarms associated with the second associated node, and utilizing the propagation link resources of the target cluster, the comprehensive impact factor of each alarm on the second associated node is obtained.

[0073] Add up the comprehensive impact factors of all alarms on the second associated node to obtain the total comprehensive impact factors of all alarms on the second associated node.

[0074] Specifically, based on all alarms associated with the second associated node, the comprehensive impact factor of each alarm on the second associated node is obtained by utilizing the propagation link resources and propagation strategy of the target cluster; the comprehensive impact factors of each alarm on the second associated node are added together to obtain the total comprehensive impact factor of each alarm on the second associated node.

[0075] In one embodiment, Figure 3 These are flowcharts of cluster fault handling methods in some embodiments, such as... Figure 3 As shown, the process includes the following steps:

[0076] Step S310: After detecting that the alarm has been deleted, the node associated with the alarm is identified as the third associated node.

[0077] Specifically, after detecting an alarm from the business system that the target device has been deleted from the target cluster, the node associated with the alarm is identified as the third associated node based on the node identifier associated with the alarm.

[0078] Step S320: Modify the status of the associated alarm in the node fault resource corresponding to the third associated node to "recovered", and determine the status of the associated alarm in the node fault resource corresponding to the third associated node as the sum of the comprehensive impact factors of the unrecovered associated alarms on the third associated node.

[0079] Specifically, based on the identification information of the third associated node, the node fault resource corresponding to the third associated node is determined. Based on the device identification information of the alarm, the status of the associated alarm corresponding to the alarm in the node fault resource is modified to "recovered". Based on the status of the associated alarms, the associated alarm information with a status of "unrecovered" is obtained, and the comprehensive impact factor of all associated alarms with a status of "unrecovered" on the third associated node is obtained. The comprehensive impact factors of all associated alarms with a status of "unrecovered" on the third associated node are summed to obtain the sum of the comprehensive impact factors of all associated alarms with a status of "unrecovered" on the third associated node.

[0080] Step S330: When the sum of the comprehensive influence factors is less than the node failure threshold, perform recovery verification on the third associated node.

[0081] The node failure threshold can be set by the user, and can be set to 100. Specifically, when the sum of the comprehensive influence factors is less than the preset node failure threshold, the third associated node is restored and verified.

[0082] Specifically, the recovery verification includes associated alarm verification and node health check verification. The associated alarm verification includes determining whether the sum of the comprehensive impact factors is less than the node failure threshold;

[0083] If the associated alarm verification passes, a health check verification is performed on the third associated node; the health check verification includes performing a health check on the third associated node.

[0084] Step S340: After the third associated node passes the recovery verification, delete the alarms corresponding to the associated alarms whose status is not recovered in the node fault resources corresponding to the third associated node, delete the node fault resources corresponding to the third associated node, and send the fault recovery message of the third associated node to the associated business system.

[0085] Specifically, after the third associated node passes the recovery verification, the node taint label is removed according to the pre-bound workflow template, and node scheduling is restored. Based on the device identification information of the associated alarms with an unrecovered status, the alarms corresponding to the unrecovered associated alarms in the node fault resources corresponding to the third associated node are deleted. Based on the identification information of the third associated node, the node fault resources corresponding to the third associated node are deleted, and a message indicating that the third associated node has recovered is sent to the associated business system. The associated business system then restores the scheduling of the third associated node.

[0086] In one embodiment, Figure 4 This is a flowchart of the recovery verification process in the cluster fault handling method of this embodiment, as follows: Figure 4 As shown, the process includes the following steps:

[0087] Step S410: Perform association alarm verification on the third associated node.

[0088] The associated alarm verification involves determining whether the sum of the comprehensive impact factors of unrecovered associated alarms on the third associated node is less than the node failure threshold. Specifically, based on the unrecovered associated alarms associated with the third associated node, the comprehensive impact factors of each unrecovered associated alarm on the third associated node are obtained using the propagation link resources of the target cluster. These comprehensive impact factors are then summed to obtain the total comprehensive impact factors of the unrecovered associated alarms on the third associated node. Based on this sum of comprehensive impact factors, the associated alarm verification is performed on the third associated node. If the sum of these comprehensive impact factors is less than the node failure threshold, the associated alarm verification passes.

[0089] Step S420: If the associated alarm verification passes, then perform a health check verification on the third associated node.

[0090] Specifically, after the associated alarm verification passes, a health check is performed on the third associated node. This health check includes a health check of the third associated node. This health check can be a load test, stress test, etc.

[0091] This embodiment also provides a cluster fault handling system. Figure 5 This is a schematic diagram of the cluster fault handling system in this embodiment, as shown below. Figure 5 As shown, the cluster fault handling system 50 includes an alarm handling module 502 and a node fault handling module 504.

[0092] The alarm processing module 502 is used to, after detecting an alarm of a target device in the target cluster, determine the fault propagation link of the alarm of the target device, and based on the fault propagation link, determine all associated nodes of the alarm of the target device; determine whether each associated node has a corresponding node fault resource; identify associated nodes with node fault resources as first associated nodes; identify associated nodes without node fault resources as second associated nodes; for the first associated node, add the alarm information of the target device to the associated alarm list of the node fault resource of the first associated node; for the second associated node, determine the sum of the comprehensive impact factors of all alarms on the second associated node based on all alarms associated with the second associated node; when the sum of the comprehensive impact factors is greater than a preset node fault threshold, create the node fault resource corresponding to the second associated node, and add the alarm information of the target device to the associated alarm list of the node fault resource of the second associated node;

[0093] The node fault handling module 504 is used to handle faults in the second associated node according to the preset workflow template.

[0094] The cluster fault handling system 50 can analyze the fault impact of alarms, merge all alarms on affected nodes, and process faulty nodes in conjunction with workflow, thus solving the problem of high operation and maintenance costs.

[0095] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0096] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above method embodiments.

[0097] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0099] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0100] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0101] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A cluster failure handling method, characterized by, The method comprises the following steps: After monitoring the alarm of the target device in the target cluster, a fault propagation link of the alarm of the target device is determined; The alarm is a K8s custom resource; According to the fault propagation link, all associated nodes of the alarm of the target device are determined; It is determined whether there is a corresponding node fault resource in each associated node; the associated node with the node fault resource is identified as a first associated node; the associated node without the node fault resource is identified as a second associated node; the node fault resource is a K8s custom resource; For the first associated node, the alarm information of the target device is added to the associated alarm list of the node fault resource of the first associated node; the alarm information includes alarm name, alarm state and alarm impact factor; For the second associated node, according to all alarms associated with the second associated node, the total sum of the comprehensive impact factors of all alarms on the second associated node is determined; When the total sum of the comprehensive impact factors is greater than a preset node fault threshold, a node fault resource corresponding to the second associated node is created, and the alarm information of the target device is added to the associated alarm list of the node fault resource of the second associated node; according to a preset workflow template, the second associated node is processed for fault.

2. The cluster failure handling method of claim 1, wherein, The method for determining the fault propagation link of the alarm of the target device comprises the following steps: Obtain the propagation link resource of the target cluster; the propagation link resource includes the propagation path of the target cluster and the propagation strategy of the target cluster; the propagation link resource is a K8s custom resource; Traverse the propagation path of the target cluster, and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster.

3. The cluster failure handling method of claim 2, wherein, Traverse the propagation path of the target cluster, and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster, which comprises the following steps: Traverse the propagation path of the target cluster, and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster based on the breadth-first search algorithm and the graph traversal algorithm and by using the device topology relationship graph of the target cluster.

4. The cluster failure handling method of claim 1, wherein, According to the fault propagation link, all associated nodes of the alarm of the target device are determined, which comprises the following steps: According to the type of the device in the fault propagation link of the alarm of the target device, all associated nodes of the alarm of the target device are determined.

5. The cluster failure handling method of claim 2, wherein, According to all alarms associated with the second associated node, the total sum of the comprehensive impact factors of all alarms on the second associated node is determined, which comprises the following steps: According to all alarms associated with the second associated node, the total sum of the comprehensive impact factors of all alarms on the second associated node is determined, which comprises the following steps: According to all alarms associated with the second associated node, the total sum of the comprehensive impact factors of all alarms on the second associated node is determined, which comprises the following steps:

6. The cluster failure handling method of claim 1, wherein, Further comprising: After monitoring that the alarm is deleted, all associated nodes of the alarm are determined; The nodes associated with the alarm are identified as third associated nodes; modify a state of a corresponding associated alarm in the node fault resource corresponding to the third associated node to recovered, and determine a sum of comprehensive influence factors of associated alarms in the node fault resource corresponding to the third associated node whose state is unrecovered on the third associated node; when the sum of the comprehensive influence factors is less than the node fault threshold, perform recovery verification on the third associated node; after the third associated node passes the recovery verification, delete alarms corresponding to associated alarms in the node fault resource corresponding to the third associated node whose state is unrecovered, delete the node fault resource corresponding to the third associated node, and send a message of recovery of the third associated node fault to an associated service system.

7. The cluster failure handling method of claim 6, wherein, The recovery verification on the third associated node includes: performing associated alarm verification on the third associated node; the associated alarm verification includes judging whether the sum of the comprehensive influence factors is less than the node fault threshold; if the associated alarm verification passes, performing health check verification on the third associated node; the health check verification includes performing health check on the third associated node.

8. A cluster failure handling system, characterized by, The alarm processing module and the node fault processing module are included: The alarm processing module is configured to, after monitoring an alarm of a target device in a target cluster, determine a fault propagation link of the alarm of the target device, determine all associated nodes of the alarm of the target device according to the fault propagation link, determine whether each associated node has a corresponding node fault resource, identify an associated node having the node fault resource as a first associated node, identify an associated node not having the node fault resource as a second associated node, and add alarm information of the target device to an associated alarm list of a node fault resource of the first associated node. For the second associated node, determine a sum of comprehensive influence factors of all alarms associated with the second associated node on the second associated node. When the sum of the comprehensive influence factors is greater than a preset node fault threshold, create a node fault resource corresponding to the second associated node, and add the alarm information of the target device to an associated alarm list of the node fault resource of the second associated node. The node fault processing module is configured to perform fault processing on the second associated node according to a preset workflow template. The memory stores a computer program, and the processor is configured to run the computer program to perform the cluster fault processing method in any one of claims 1 to 7. The computer program is executed by the processor to implement the cluster fault processing method in any one of claims 1 to 7. 9.An electronic device comprising a memory and a processor, the electronic device characterized by, ​ 10. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images