Cluster fault processing method, system, electronic device and storage medium

Abstract

The application relates to a cluster fault processing method, a system, an electronic device and a storage medium, wherein the cluster fault processing method comprises the following steps: after monitoring an alarm of a target device in a target cluster, determining a fault propagation link of the alarm, and then determining all associated nodes of the alarm; for a node with a corresponding node fault resource, adding alarm information to an associated alarm list of the corresponding node fault resource; for a node without a corresponding node fault resource, determining a total sum of comprehensive influence factors of all associated alarms on the node; when the total sum of the comprehensive influence factors is greater than a node fault threshold, creating a corresponding node fault resource, adding alarm information to an associated alarm list of the corresponding node fault resource, and performing fault processing on the node without the corresponding node fault resource according to a work flow of the target cluster. Through the application, the problem of high operation and maintenance cost is solved.

Description

Technical Field

[0001] This application relates to the fields of cloud computing, container orchestration and operation and maintenance automation, and in particular to cluster fault handling methods, systems, electronic devices and storage media. Background Technology

[0002] In the Kubernetes (K8s) platform, a low-level problem, such as excessive CPU pressure on a node, network partitioning, or kernel deadlock, can easily trigger an "avalanche effect," causing a large number of related alerts to be generated in a very short period of time.

[0003] Existing technical solutions require manual identification of root causes, assessment of business impact, and manual fault repair, which heavily rely on expert experience and result in high operation and maintenance costs.

[0004] Therefore, no effective solution has yet been proposed to address the issue of high operation and maintenance costs. Summary of the Invention

[0005] This embodiment provides a cluster fault handling method, system, electronic device, and storage medium to address the problem of high operation and maintenance costs in related technologies.

[0006] Firstly, this embodiment provides a cluster fault handling method, the method comprising:

[0007] After detecting an alarm on a target device in the target cluster, the fault propagation path of the alarm on the target device is determined; the alarm is a K8s custom resource.

[0008] Based on the fault propagation chain, determine all associated nodes of the target device alarm;

[0009] Determine whether each of the associated nodes has a corresponding faulty node resource; identify the associated nodes with the faulty node resource as the first associated node; identify the associated nodes without the faulty node resource as the second associated node; the faulty node resource is a K8s custom resource;

[0010] For the first associated node, the alarm information of the target device is added to the associated alarm list of the node fault resources of the first associated node; the alarm information includes alarm name, alarm status and alarm impact factor;

[0011] For the second associated node, based on all alarms associated with the second associated node, determine the sum of the comprehensive impact factors of all alarms on the second associated node;

[0012] When the sum of the comprehensive impact factors is greater than the preset node fault threshold, a node fault resource corresponding to the second associated node is created, and the alarm information of the target device is added to the associated alarm list of the node fault resource of the second associated node. The second associated node is then processed according to the preset workflow template.

[0013] In some embodiments, determining the fault propagation path of the alarm of the target device includes:

[0014] Obtain the propagation link resources of the target cluster; the propagation link resources include the propagation path and propagation strategy of the target cluster; the propagation link resources are K8s custom resources;

[0015] Traverse the propagation path of the target cluster and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster.

[0016] In some embodiments, the propagation path of the target cluster is traversed, and the fault propagation link of the alarm of the target device is determined according to the propagation strategy of the target cluster, including:

[0017] Traverse the propagation path of the target cluster, and based on the breadth-first search algorithm and graph traversal algorithm, utilize the device topology graph of the target cluster, and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster.

[0018] In some embodiments, based on the fault propagation chain, all associated nodes of the target device alarm are determined, including:

[0019] Based on the device type in the fault propagation link of the target device's alarm, determine all associated nodes of the target device's alarm.

[0020] In some embodiments, the sum of the comprehensive impact factors of all alarms associated with the second associated node is determined, including:

[0021] Based on all the alarms associated with the second associated node, and using the propagation link resources of the target cluster, the comprehensive impact factor of each alarm on the second associated node is obtained.

[0022] The comprehensive impact factors of all alarms on the second associated node are added together to obtain the total comprehensive impact factor of all alarms on the second associated node.

[0023] In some embodiments, the method further includes:

[0024] After detecting that the alarm has been deleted, identify all associated nodes corresponding to the alarm; identify the nodes associated with the alarm as third associated nodes;

[0025] Modify the status of the associated alarm in the node fault resource corresponding to the third associated node to "recovered", and determine the status of the node fault resource corresponding to the third associated node as the sum of the comprehensive impact factors of the unrecovered associated alarms on the third associated node;

[0026] When the sum of the comprehensive impact factors is less than the node failure threshold, the third associated node is subjected to recovery verification.

[0027] After the third associated node passes the recovery verification, the alarms corresponding to the associated alarms with the status of "not recovered" in the node fault resources corresponding to the third associated node are deleted, the node fault resources corresponding to the third associated node are deleted, and the fault recovery message of the third associated node is sent to the associated business system.

[0028] In some embodiments, the recovery verification of the third associated node includes:

[0029] The third associated node is subjected to associated alarm verification; the associated alarm verification includes determining whether the sum of the comprehensive influence factors is less than the node fault threshold;

[0030] If the associated alarm verification passes, a health check verification is performed on the third associated node; the health check verification includes performing a health check on the third associated node.

[0031] Secondly, this embodiment provides a cluster fault handling system, including an alarm handling module and a node fault handling module:

[0032] The alarm processing module is used to, after detecting an alarm of a target device in the target cluster, determine the fault propagation link of the alarm of the target device, and based on the fault propagation link, determine all associated nodes of the target device; determine whether each associated node has a corresponding node fault resource; identify associated nodes with node fault resources as first associated nodes; identify associated nodes without node fault resources as second associated nodes; for the first associated node, add the alarm information of the target device to the associated alarm list of the node fault resources of the first associated node; for the second associated node, determine the sum of the comprehensive impact factors of all alarms associated with the second associated node based on all alarms associated with the second associated node; when the sum of the comprehensive impact factors is greater than a preset node fault threshold, create the node fault resource corresponding to the second associated node, and add the alarm information of the target device to the associated alarm list of the node fault resources of the second associated node;

[0033] The node fault handling module is used to handle faults in the second associated node according to a preset workflow template.

[0034] Thirdly, this embodiment provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the cluster fault handling method described in the first aspect above.

[0035] Fourthly, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cluster fault handling method described in the first aspect above.

[0036] Compared with related technologies, this embodiment provides a cluster fault handling method, system, electronic device, and storage medium. The cluster fault handling method includes: after detecting an alarm from a target device in a target cluster, determining the fault propagation path of the alarm from the target device; based on the fault propagation path, determining all associated nodes of the alarm from the target device; determining whether each associated node has a corresponding node fault resource; identifying associated nodes with node fault resources as first associated nodes; identifying associated nodes without node fault resources as second associated nodes; for the first associated node, adding the alarm information of the target device to the associated alarm list of the node fault resources of the first associated node; for the second associated node, determining the sum of the comprehensive impact factors of all alarms associated with the second associated node; when the sum of the comprehensive impact factors is greater than a preset node fault threshold, creating a node fault resource corresponding to the second associated node, adding the alarm information of the target device to the associated alarm list of the node fault resources of the second associated node, and performing fault handling on the second associated node according to the workflow of the target cluster. It can analyze the fault impact of alarms, merge all alarms on affected nodes, and process faulty nodes in conjunction with workflow, thus solving the problem of high operation and maintenance costs.

[0037] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1 This is a block diagram of the terminal hardware structure of the cluster fault handling method in this embodiment;

[0040] Figure 2 This is a flowchart of the cluster fault handling method in this embodiment;

[0041] Figure 3 This is a flowchart of a cluster fault handling method in some embodiments;

[0042] Figure 4 This is a flowchart of the recovery verification process in the cluster fault handling method of this embodiment;

[0043] Figure 5 This is a schematic diagram of the cluster fault handling system in this embodiment. Detailed Implementation

[0044] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0045] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0046] The method embodiments provided in this example can be executed on a terminal, computer, or similar electronic device with a certain computing power. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal in the cluster fault handling method of this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0047] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the cluster fault handling method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0049] This embodiment provides a cluster fault handling method. Figure 2 This is a flowchart of the cluster fault handling method in this embodiment, such as... Figure 2 As shown, the process includes the following steps:

[0050] Step S210: After detecting an alarm from a target device in the target cluster, determine the fault propagation link of the alarm from the target device.

[0051] The target device can be a node, i.e., physical hardware such as a server, GPU, container, switch, CPU, or disk. The alarm is a custom Kubernetes resource, and its definition can include fields such as basic alarm information, cluster name, status, fault link, and associated nodes. The basic alarm information includes the device's identification information, device type, and alarm level. Alarms for the target devices in this target cluster can be created by the business system. The business system creates alarms for the target devices in the target cluster based on alarm notifications reported by the monitoring component.

[0052] Specifically, upon detecting an alarm notification from a target device in the target cluster created by the business system, a corresponding alarm is generated based on the basic alarm information in the alarm. Simultaneously, the alarm's status is set to initialization, and the fault propagation path of the target device's alarm is determined. The basic alarm information includes the target device's identification information, target device type, and alarm level. The fault propagation path includes the propagation events of all associated devices related to the target device's alarm.

[0053] Step S220: Based on the fault propagation link, determine all associated nodes of the target device's alarm.

[0054] The fault propagation chain includes the propagation events of all associated devices of the target device's alarm. The types of associated devices include physical hardware such as nodes, servers, GPUs, containers, switches, CPUs, and disks. The propagation events include device identification information, device type, and comprehensive impact factor.

[0055] Specifically, the fault propagation chain of the alarm of the target device is determined. Based on the fault propagation chain, the propagation events of the alarm of the target device with the type of node in the propagation events of all related devices are determined. All propagation events of the type of node in the related devices are added to the list of related nodes of the alarm, and the status of the alarm is set to running.

[0056] Step S230: Determine whether each associated node has a corresponding faulty resource; identify the associated node with the faulty resource as the first associated node; identify the associated node without the faulty resource as the second associated node.

[0057] The node failure resource is a custom Kubernetes resource, which can include fields such as node identifier information, status, and associated alarms in its definition. Associated alarms include associated alarm information and associated alarm status. Specifically, the associated nodes of the alarm are traversed, and based on the identifier information of the associated nodes, it is determined whether each associated node has a corresponding node failure resource. Associated nodes with node failure resources are identified as first associated nodes, and associated nodes without node failure resources are identified as second associated nodes.

[0058] Step S240: For the first associated node, add the alarm information of the target device to the associated alarm list of the node fault resources of the first associated node; for the second associated node, determine the total comprehensive impact factor of all alarms on the second associated node based on all alarms associated with the second associated node.

[0059] Specifically, for the first associated node, the alarm information of the target device is added to the associated alarm list of the node fault resources of the first associated node. For the second associated node, all alarms associated with the second associated node are obtained through the identification information of the second associated node, and then the total comprehensive impact factor of all alarms on the second associated node is determined.

[0060] Step S250: When the sum of comprehensive influence factors is greater than the preset node fault threshold, create the node fault resource corresponding to the second associated node, add the alarm information of the target device to the associated alarm list of the node fault resource of the second associated node, and perform fault handling on the second associated node according to the preset workflow template.

[0061] This fault handling can include operations such as tainting nodes and backing up and migrating services on nodes. Specifically, when the sum of comprehensive impact factors exceeds a preset node fault threshold, a node fault resource corresponding to the second associated node is created, and the status of this node fault resource is set to initialized. The alarm information of the target device is added to the associated alarm list of the node fault resource of the second associated node. According to the preset workflow template, the fault handling of the second associated node is performed, and the status of this node fault resource is set to running.

[0062] By using steps S210 to S250 above, compared with related technologies, this embodiment, after detecting an alarm from a target device in the target cluster, determines the fault propagation link of the target device's alarm; based on the fault propagation link, it determines all associated nodes of the target device's alarm; it determines whether each associated node has a corresponding node fault resource; it identifies associated nodes with node fault resources as first associated nodes; and it identifies associated nodes without node fault resources as second associated nodes; for the first associated node, it adds the target device's alarm information to the associated alarm list of the node fault resource of the first associated node; for the second associated node, it determines the sum of the comprehensive impact factors of all alarms on the second associated node based on all alarms associated with the second associated node; when the sum of the comprehensive impact factors is greater than a preset node fault threshold, it creates the node fault resource corresponding to the second associated node, adds the target device's alarm information to the associated alarm list of the node fault resource of the second associated node, and performs fault handling on the second associated node according to the workflow of the target cluster. This approach can analyze the fault impact surface of alarms, merge all alarms on affected nodes, and handle faulty nodes in conjunction with the workflow, thus solving the problem of high operation and maintenance costs.

[0063] In one embodiment, determining the fault propagation path of the alarm of the target device includes:

[0064] Obtain the propagation link resources of the target cluster; the propagation link resources include the propagation path and propagation strategy of the target cluster; traverse the propagation path of the target cluster, and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster.

[0065] This propagation link resource is a custom Kubernetes resource. Its definition can include fields such as cluster name, propagation path, and propagation strategy. The propagation path includes the starting device type, ending device type, and propagation factor. The propagation factor represents the probability of a fault propagating along this path, ranging from 0 to 1. The propagation strategy can be a dynamic propagation factor strategy, a static propagation factor strategy, etc. The dynamic propagation factor strategy calculates the propagation factor using the performance metrics of the current device, while the static propagation factor strategy uses the propagation factor of the propagation path. This propagation link resource can be created by the user, allowing different users to create different propagation link resources for different clusters. Different propagation link resources are stored in the propagation link cache by the propagation link controller, offering advantages such as easy expansion, no need for training data, and immediate effectiveness.

[0066] Specifically, based on the name of the target cluster, the propagation link resources of the target cluster, pre-stored in the propagation link cache, are retrieved. These resources include the propagation path and propagation strategy of the target cluster. The propagation path can contain different starting device types, ending device types, and propagation factors. For example: starting device type is a node, ending device type is a graphics processor, and propagation factor is 1.0; starting device type is a graphics processor, ending device type is a container, and propagation factor is 0.8; starting device type is a container, ending device type is a controller managing the container, and propagation factor is 0.6. Based on the propagation path of the target cluster, the ending device type and corresponding propagation factor of the target device can be obtained. Based on the propagation strategy of the target cluster, the comprehensive impact factor of the target device's alarms on the ending devices can be obtained. The propagation path of the target cluster is traversed, and based on the propagation strategy, the set of propagation events of all associated devices for the target device's alarms is obtained.

[0067] In one embodiment, the propagation path of the target cluster is traversed, and the fault propagation link of the alarm of the target device is determined according to the propagation strategy of the target cluster, including:

[0068] Traverse the propagation path of the target cluster, and based on the breadth-first search algorithm and graph traversal algorithm, use the device topology graph of the target cluster and the propagation strategy of the target cluster to determine the fault propagation link of the alarm of the target device.

[0069] Specifically, based on the propagation path of the target cluster, the endpoint device type and corresponding propagation factor corresponding to the current device type can be obtained. Using a breadth-first search algorithm and a graph traversal algorithm, and leveraging the device topology graph of the target cluster, the identification information of the endpoint device can be obtained. Based on the propagation strategy of the target cluster, the comprehensive influence factor of the current device on the endpoint device can be obtained, thus yielding the propagation event of the endpoint device. This endpoint device is identified as the current device, and its propagation event is obtained. This process is recursively repeated until the propagation path of the target cluster has been traversed. This determines the fault propagation link of the alarm for the target device.

[0070] For example: Target device alarm: Target device identification information is 001, target device type is node, alarm level is severe; Propagation path of the target cluster: Starting device type is node, ending device type is graphics processor, propagation factor is 1.0; Starting device type is graphics processor, ending device type is container, propagation factor is 0.8; Starting device type is container, ending device type is controller managing container, propagation factor is 0.6; Propagation strategy of the target cluster: static propagation factor strategy. Quantifying the alarm level, the alarm impact factor is 80. Based on the propagation path of the target cluster, the ending device type corresponding to the target device is graphics processor, with a corresponding propagation factor of 1.0. Based on the breadth-first search algorithm and graph traversal algorithm, using the device topology graph of the target cluster, the identification information of all ending devices corresponding to the target device is obtained. According to the static propagation factor strategy, the comprehensive impact factor of the target device's alarm on the ending devices is obtained, which is the alarm impact factor 80 multiplied by the corresponding propagation factor 1.0. Thus, the propagation events of all ending devices corresponding to the target device are obtained. Traversing the propagation path of the target cluster, we find that all endpoint devices are of container type, with a propagation factor of 0.8. Using a breadth-first search algorithm and a graph traversal algorithm, and leveraging the device topology graph of the target cluster, we obtain the identification information of all endpoint devices corresponding to these endpoint devices. According to the static propagation factor strategy, we obtain the comprehensive impact factor of the propagation events of the endpoint devices on their corresponding endpoint devices. This comprehensive impact factor is 80, the comprehensive impact factor of the target device's alarm on the endpoint devices, multiplied by the corresponding propagation factor of 0.8. Thus, we obtain the propagation events of all endpoint devices corresponding to these endpoint devices. By repeating this process, after traversing the propagation path of the target cluster, we obtain the set of propagation events of all associated devices for the target device's alarm, i.e., the fault propagation chain of the alarm.

[0071] In one embodiment, based on all alarms associated with the second associated node, the sum of the comprehensive impact factors of all alarms on the second associated node is determined, including:

[0072] Based on all alarms associated with the second associated node, and utilizing the propagation link resources of the target cluster, the comprehensive impact factor of each alarm on the second associated node is obtained.

[0073] Add up the comprehensive impact factors of all alarms on the second associated node to obtain the total comprehensive impact factor of all alarms on the second associated node.

[0074] Specifically, based on all alarms associated with the second associated node, the comprehensive impact factor of each alarm on the second associated node is obtained by utilizing the propagation link resources and propagation strategy of the target cluster; the comprehensive impact factors of each alarm on the second associated node are added together to obtain the total comprehensive impact factor of each alarm on the second associated node.

[0075] In one embodiment, Figure 3 These are flowcharts of cluster fault handling methods in some embodiments, such as... Figure 3 As shown, the process includes the following steps:

[0076] Step S310: After detecting that the alarm has been deleted, the node associated with the alarm is identified as the third associated node.

[0077] Specifically, after detecting an alarm from the business system that the target device in the target cluster has been deleted, the node associated with the alarm is identified as the third associated node based on the node identifier associated with the alarm.

[0078] Step S320: Modify the status of the associated alarm in the node fault resource corresponding to the third associated node to "recovered", and determine the status of the associated alarm in the node fault resource corresponding to the third associated node as the sum of the comprehensive impact factors of the unrecovered associated alarms on the third associated node.

[0079] Specifically, based on the identification information of the third associated node, the node fault resource corresponding to the third associated node is determined. Based on the device identification information of the alarm, the status of the associated alarm corresponding to the alarm in the node fault resource is modified to "recovered". Based on the status of the associated alarms, the associated alarm information with a status of "unrecovered" is obtained, and the comprehensive impact factor of all associated alarms with a status of "unrecovered" on the third associated node is obtained. The comprehensive impact factors of all associated alarms with a status of "unrecovered" on the third associated node are summed to obtain the sum of the comprehensive impact factors of all associated alarms with a status of "unrecovered" on the third associated node.

[0080] Step S330: When the sum of the comprehensive impact factors is less than the node failure threshold, perform recovery verification on the third associated node.

[0081] The node failure threshold can be set by the user, and can be set to 100. Specifically, when the sum of the comprehensive influence factors is less than the preset node failure threshold, the third associated node is restored and verified.

[0082] Specifically, the recovery verification includes associated alarm verification and node health check verification. The associated alarm verification includes determining whether the sum of the comprehensive impact factors is less than the node failure threshold;

[0083] If the associated alarm verification passes, a health check verification is performed on the third associated node; the health check verification includes performing a health check on the third associated node.

[0084] Step S340: After the third associated node passes the recovery verification, delete the alarms corresponding to the associated alarms whose status is not recovered in the node fault resources corresponding to the third associated node, delete the node fault resources corresponding to the third associated node, and send the fault recovery message of the third associated node to the associated business system.

[0085] Specifically, after the third associated node passes the recovery verification, the node taint label is removed according to the pre-bound workflow template, and node scheduling is restored. Based on the device identification information of the associated alarms with an unrecovered status, the alarms corresponding to the unrecovered associated alarms in the node fault resources corresponding to the third associated node are deleted. Based on the identification information of the third associated node, the node fault resources corresponding to the third associated node are deleted, and a message indicating that the third associated node has recovered is sent to the associated business system. The associated business system then restores the scheduling of the third associated node.

[0086] In one embodiment, Figure 4 This is a flowchart of the recovery verification process in the cluster fault handling method of this embodiment, as follows: Figure 4 As shown, the process includes the following steps:

[0087] Step S410: Perform association alarm verification on the third associated node.

[0088] The associated alarm verification involves determining whether the sum of the comprehensive impact factors of unrecovered associated alarms on the third associated node is less than the node failure threshold. Specifically, based on the unrecovered associated alarms associated with the third associated node, the comprehensive impact factors of each unrecovered associated alarm on the third associated node are obtained using the propagation link resources of the target cluster. These comprehensive impact factors are then summed to obtain the total comprehensive impact factors of the unrecovered associated alarms on the third associated node. Based on this sum of comprehensive impact factors, the associated alarm verification is performed on the third associated node. If the sum of these comprehensive impact factors is less than the node failure threshold, the associated alarm verification passes.

[0089] Step S420: If the associated alarm verification passes, then perform a health check verification on the third associated node.

[0090] Specifically, after the associated alarm verification passes, a health check is performed on the third associated node. This health check includes a health check of the third associated node. This health check can be a load test, stress test, etc.

[0091] This embodiment also provides a cluster fault handling system. Figure 5 This is a schematic diagram of the cluster fault handling system in this embodiment, as shown below. Figure 5 As shown, the cluster fault handling system 50 includes an alarm handling module 502 and a node fault handling module 504.

[0092] The alarm processing module 502 is used to, after detecting an alarm of a target device in the target cluster, determine the fault propagation link of the alarm of the target device, and based on the fault propagation link, determine all associated nodes of the alarm of the target device; determine whether each associated node has a corresponding node fault resource; identify associated nodes with node fault resources as first associated nodes; identify associated nodes without node fault resources as second associated nodes; for the first associated node, add the alarm information of the target device to the associated alarm list of the node fault resource of the first associated node; for the second associated node, determine the sum of the comprehensive impact factors of all alarms on the second associated node based on all alarms associated with the second associated node; when the sum of the comprehensive impact factors is greater than a preset node fault threshold, create the node fault resource corresponding to the second associated node, and add the alarm information of the target device to the associated alarm list of the node fault resource of the second associated node;

[0093] The node fault handling module 504 is used to handle faults in the second associated node according to the preset workflow template.

[0094] The cluster fault handling system 50 can analyze the fault impact of alarms, merge all alarms on affected nodes, and process faulty nodes in conjunction with workflow, thus solving the problem of high operation and maintenance costs.

[0095] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0096] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above method embodiments.

[0097] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0099] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0100] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0101] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A cluster fault handling method, characterized in that, include: After detecting alarms from target devices in the target cluster, determine the fault propagation path of the alarms from the target devices; The alarm is for a custom K8s resource. The step of determining the fault propagation link of the alarm of the target device includes: acquiring the propagation link resources of the target cluster; the propagation link resources include the propagation path of the target cluster and the propagation strategy of the target cluster; the propagation link resources are K8s custom resources; traversing the propagation path of the target cluster, and determining the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster; Based on the fault propagation chain, determine all associated nodes of the target device alarm; The step of determining all associated nodes of the target device alarm based on the fault propagation link includes: determining all associated nodes of the target device alarm based on the device type in the fault propagation link of the target device alarm; Determine whether each of the associated nodes has a corresponding faulty node resource; identify the associated nodes with the faulty node resource as the first associated node; identify the associated nodes without the faulty node resource as the second associated node; the faulty node resource is a K8s custom resource; For the first associated node, the alarm information of the target device is added to the associated alarm list of the node fault resources of the first associated node; the alarm information includes alarm name, alarm status and alarm impact factor; For the second associated node, based on all alarms associated with the second associated node, determine the sum of the comprehensive impact factors of all alarms on the second associated node; When the sum of the comprehensive impact factors is greater than the preset node fault threshold, a node fault resource corresponding to the second associated node is created, and the alarm information of the target device is added to the associated alarm list of the node fault resource of the second associated node. The second associated node is then processed according to the preset workflow template.

2. The cluster fault handling method according to claim 1, characterized in that, Traverse the propagation path of the target cluster, and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster, including: Traverse the propagation path of the target cluster, and based on the breadth-first search algorithm and graph traversal algorithm, utilize the device topology graph of the target cluster, and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster.

3. The cluster fault handling method according to claim 1, characterized in that, Based on all alarms associated with the second associated node, determine the sum of the comprehensive impact factors of all alarms on the second associated node, including: Based on all the alarms associated with the second associated node, and using the propagation link resources of the target cluster, the comprehensive impact factor of each alarm on the second associated node is obtained. The comprehensive impact factors of all alarms on the second associated node are added together to obtain the total comprehensive impact factor of all alarms on the second associated node.

4. The cluster fault handling method according to claim 1, characterized in that, Also includes: After detecting that the alarm has been deleted, identify all associated nodes corresponding to the alarm; The node associated with the alarm is identified as a third associated node; Modify the status of the associated alarm in the node fault resource corresponding to the third associated node to "recovered", and determine that the status of the node fault resource corresponding to the third associated node is the sum of the comprehensive impact factors of the unrecovered associated alarms on the third associated node. When the sum of the comprehensive impact factors is less than the node failure threshold, the third associated node is subjected to recovery verification. After the third associated node passes the recovery verification, the alarms corresponding to the associated alarms with the status of "not recovered" in the node fault resources corresponding to the third associated node are deleted, the node fault resources corresponding to the third associated node are deleted, and the fault recovery message of the third associated node is sent to the associated business system.

5. The cluster fault handling method according to claim 4, characterized in that, The recovery verification of the third associated node includes: The third associated node is subjected to associated alarm verification; the associated alarm verification includes determining whether the sum of the comprehensive influence factors is less than the node fault threshold; If the associated alarm verification passes, a health check verification is performed on the third associated node; the health check verification includes performing a health check on the third associated node.

6. A cluster fault handling system, characterized in that, Includes an alarm processing module and a node fault handling module: The alarm processing module is used to determine the fault propagation link of the alarm of the target device after detecting the alarm of the target device in the target cluster, and to determine all the associated nodes of the alarm of the target device based on the fault propagation link. Determine whether each of the associated nodes has a corresponding faulty node resource; identify the associated node with the faulty node resource as the first associated node; Identify the associated nodes that do not have the node fault resource as the second associated nodes; for the first associated node, add the alarm information of the target device to the associated alarm list of the node fault resource of the first associated node; For the second associated node, based on all alarms associated with the second associated node, determine the sum of the comprehensive impact factors of all alarms on the second associated node; When the sum of the comprehensive impact factors is greater than the preset node fault threshold, a node fault resource corresponding to the second associated node is created, and the alarm information of the target device is added to the associated alarm list of the node fault resource of the second associated node. The alarm processing module is also used to obtain the propagation link resources of the target cluster; the propagation link resources include the propagation path of the target cluster and the propagation strategy of the target cluster; The propagation link resources are K8s custom resources; Traverse the propagation path of the target cluster and determine the fault propagation link of the alarm of the target device according to the propagation strategy of the target cluster; The alarm processing module is also used to determine all associated nodes of the target device alarm based on the device type in the fault propagation link of the target device alarm; The node fault handling module is used to handle faults in the second associated node according to a preset workflow template.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the cluster fault handling method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cluster fault handling method according to any one of claims 1 to 5.

Images