Fault processing method and related device

Abstract

Disclosed in embodiments of the present application are a fault processing method and a related device, for use in improving the accuracy of device fault range prediction in clustered pooling environments, and improving the reliability of fault propagation blocking strategy selection. The method in the embodiments of the present application comprises: determining a node fault mode on the basis of node load data; inputting the node fault mode into a fault prediction model, so as to determine a first fault propagation chain; inputting the node fault mode and at least one fault propagation blocking strategy into the fault prediction model, so as to determine at least one second fault propagation chain, wherein the at least one fault propagation blocking strategy is determined on the basis of the node fault mode; and determining a target fault propagation blocking strategy on the basis of the first fault propagation chain and the at least one second fault propagation chain.

Description

Troubleshooting methods and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411458517.5, filed on October 17, 2024, entitled "Fault Handling Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer security technology, specifically to a fault handling method and related equipment. Background Technology

[0003] Cluster pooling refers to the reuse of resources or tasks in a cluster architecture. By pre-creating and managing pools of resources or tasks, it enables efficient sharing and utilization of resources, reducing the overhead of repeated creation and destruction, and improving system efficiency and performance.

[0004] As cluster size and complexity increase, failures in individual nodes can easily spread, impacting a wide area. Cluster reliability operation and maintenance management technologies ensure high system reliability and uninterrupted operation of critical services through fault detection and isolation, node backup and repair, and cluster monitoring and maintenance.

[0005] Existing cluster fault range detection technologies include reliability assessment schemes based on probability distributions. However, such reliability calculation frameworks are limited to fault range prediction under specific conditions, making it difficult to accurately predict and assess device fault conditions in cluster pooling environments, and they cannot select effective fault propagation prevention strategies. Summary of the Invention

[0006] This application provides a fault handling method and related equipment to improve the accuracy of device fault range prediction in a clustered pooling environment and improve the reliability of fault propagation blocking strategy selection.

[0007] The embodiments of this application provide the following technical solutions:

[0008] In a first aspect, embodiments of this application provide a fault handling method, the method comprising: determining a node fault mode based on node load data; inputting the node fault mode into a fault prediction model to determine a first fault propagation chain; inputting the node fault mode and at least one fault propagation blocking strategy into the fault prediction model to determine at least one second fault propagation chain; wherein the at least one fault propagation blocking strategy is determined based on the node fault mode; and determining a target fault propagation blocking strategy based on the first fault propagation chain and the at least one second fault propagation chain. In the above implementation, the fault prediction model can predict the impact of a single node fault in a cluster on the cluster network, that is, the impact of a single node fault within the cluster topology on the overall computing cluster. Based on the fault prediction model, fault propagation chains corresponding to different fault propagation blocking strategies are determined. The fault propagation chain indicates the nodes affected by the fault in the cluster and the direction of fault propagation. Therefore, the fault propagation chain can reflect the propagation range and impact degree of the fault in the cluster. The reliability of different fault propagation blocking strategies is evaluated through the fault propagation chain, thereby determining the final fault propagation blocking strategy and improving the reliability of fault propagation blocking strategy selection.

[0009] In this embodiment, the node load data can be the load data corresponding to any node in the cluster network. This embodiment does not limit the method of obtaining the node load data. For example, real-time system resource monitoring tools can be used to obtain the load data of each node in the cluster. These real-time system resource monitoring tools may include DStat, Atop, Nmon, Slabtop, Saidar, vmstat, iperf, etc.

[0010] In this context, any node in the cluster network refers to any device among the various devices included in the cluster network. In some implementations, the fault handling method provided in this application can be applied to various devices in the cluster network, and the fault handling method of this application can be executed by a certain device or some devices in the cluster network, or by other devices outside the cluster network. This application does not impose any restrictions on this.

[0011] It should be noted that the fault prediction model used in the embodiments of this application can be pre-built and trained. The fault prediction model is specifically a regression model of the fault time sequence state, such as a neural network model or a decision tree model. The embodiments of this application do not limit the specific type of fault prediction model.

[0012] In this embodiment, at least one fault propagation blocking strategy is determined based on the node fault mode, and the fault propagation blocking strategy is used to block the spread of faults in the cluster. In this embodiment, a fault propagation blocking strategy library can be pre-set, and there is a mapping relationship between node fault modes and fault propagation blocking strategies. Based on this mapping relationship, at least one fault propagation blocking strategy can be determined from the fault propagation blocking strategy library.

[0013] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: acquiring the node load data, wherein the node load data includes at least one of node CPU utilization, node memory utilization, and node network traffic. In the above implementation, acquiring the node load data facilitates the identification of the current node failure mode.

[0014] It should be noted that the node load data in this embodiment may also include other types of load data, such as: node storage resource utilization, node resource lifecycle information, node resource-borne service information, server architecture characteristics, server system characteristics, server interconnection characteristics, server cluster characteristics, etc. This embodiment does not impose any limitations, and those skilled in the art can make appropriate selections based on their evaluation needs.

[0015] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the node failure mode based on node load data includes: determining abnormal load data from the node load data by combining historical load data; and determining the node failure mode based on the abnormal load data. In the above implementation, abnormal load data is determined from the node load data based on historical load data, and the node failure mode is determined based on the abnormal load data.

[0016] This application does not limit the method of determining node failure modes based on abnormal load data. For example, determining node failure modes based on abnormal load data in this application can be implemented using machine learning algorithms. For instance, node failure modes can be identified using clustering analysis algorithms, pattern recognition algorithms, supervised learning algorithms, etc. The pattern recognition algorithm can include algorithms such as Bayesian classifiers, K-nearest neighbor analysis, principal component analysis, and non-negative matrix factorization. For example, determining node failure modes based on abnormal load data in this application can also be implemented using non-machine learning algorithms. For instance, node failure modes can be identified using rule-based logic judgment, lookup table methods, etc.

[0017] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: performing data cleaning and preprocessing on the node load data to obtain processed node load data; wherein, the data cleaning includes at least one of the following algorithms: deduplication algorithm, median and mean statistical analysis algorithm, and data format and logical analysis verification algorithm; the preprocessing includes a normalization algorithm or a standardization algorithm. In the above implementation scheme, data cleaning can remove invalid data, duplicate data, and inaccurate data, avoiding abnormal data from affecting the subsequent identification of fault modes. Standardizing the load data through normalization or standardization algorithms facilitates subsequent identification of node fault modes based on algorithms.

[0018] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the node failure mode based on node load data includes: combining historical load data to determine processed abnormal load data from the processed node load data; and determining the node failure mode based on the processed abnormal load data. In the above implementation, node failure mode identification is performed based on the processed node load data to ensure the accuracy of the identification.

[0019] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the target fault propagation blocking strategy based on the first fault propagation chain and the at least one second fault propagation chain includes: determining a fault propagation blocking score for the at least one fault propagation blocking strategy based on the first fault propagation chain and the at least one second fault propagation chain; and determining the target fault propagation blocking strategy based on the highest score among the at least one fault propagation blocking scores. In the above implementation, the first fault propagation chain is the fault propagation chain obtained without implementing any fault propagation blocking strategy, and the second fault propagation chain is the fault propagation chain obtained after implementing a fault propagation blocking strategy. Each fault propagation blocking strategy corresponds to a second fault propagation chain. By comparing the first fault propagation chain and the second fault propagation chain, the effectiveness and reliability of the fault blocking strategy in blocking fault chains in the cluster can be evaluated. Therefore, by evaluating the second fault propagation chains corresponding to each fault propagation blocking strategy, the fault propagation blocking strategy with the best effectiveness and reliability, i.e., the target fault propagation blocking strategy, can be determined from among the various fault propagation blocking strategies.

[0020] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: inputting at least one fault recovery strategy and a target fault propagation chain into a cluster availability model to determine a fault recovery score for the at least one fault recovery strategy, wherein the at least one fault recovery strategy is determined based on the node failure mode, and the target fault propagation chain includes either the first fault propagation chain or the second fault propagation chain; and determining a target fault recovery strategy based on the highest score among the fault recovery scores of the at least one fault recovery strategy. In the above implementation, a cluster availability model can be constructed based on the fault propagation chain, and the fault recovery scores corresponding to each fault recovery strategy can be evaluated based on the cluster availability model, with the fault recovery strategy with the highest score being determined as the target fault recovery strategy.

[0021] Cluster availability refers to the ability of a cluster network to provide services normally within a specific time period. It can be evaluated by calculating the impact of node failures in the cluster network on service availability. In this embodiment, a cluster availability model is constructed based on the fault propagation chain predicted by the fault prediction model. The effectiveness and reliability of fault recovery strategies can be evaluated through the cluster availability model.

[0022] In this embodiment of the application, the cluster availability model can combine the series and parallel relationships by analyzing the maintenance status of the overall system after the failure of different hardware components, so as to finally complete the evaluation of the mean time between failures (MTBF) of cluster reliability under different fault recovery strategies.

[0023] In this embodiment, at least one fault recovery strategy is determined based on the node fault mode, and the fault recovery strategy is used to restore the faulty node to a normal state. In this embodiment, a fault recovery strategy library can be pre-set, and there is a mapping relationship between node fault modes and fault recovery strategies. Based on this mapping relationship, at least one fault recovery strategy can be determined from the fault recovery strategy library.

[0024] In conjunction with the first aspect, in one possible implementation of the first aspect, the step of inputting at least one fault recovery strategy and the target fault propagation chain into the cluster availability model to determine the fault recovery score of the at least one fault recovery strategy includes: abstracting the software dynamic scheduling process corresponding to the at least one fault recovery strategy into an update process, and inputting the update process and the target fault propagation chain into the cluster availability model to determine the fault recovery score of the at least one fault recovery strategy. In the above implementation, the software dynamic scheduling process corresponding to the fault recovery strategy is first abstracted into an update process, which is a mapping of the fault recovery strategy at the software level. Then, the update process and the fault propagation chain are input into the cluster availability model to obtain the fault recovery score corresponding to the fault recovery strategy. The cluster availability model may include the overall topology of the cluster network. Outputting the fault propagation chain into the cluster availability model can determine the node topology information that needs to be used in the cluster availability model during this availability assessment, improving the efficiency and accuracy of the availability assessment.

[0025] In conjunction with the first aspect, in one possible implementation of the first aspect, the cluster availability model includes a Markov chain-based state transition model. In the above implementation, the cluster availability model includes a Markov chain-based state transition model, which constructs an availability calculation model by analyzing a Markov chain in the state space that undergoes a stochastic process of transitioning from one state, such as a normal state or a fault state, to another state, such as a fault state or a normal state.

[0026] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: configuring the fault prediction model and the cluster availability model based on cluster hardware parameter information and cluster topology. In the above implementation, different cluster networks may correspond to different cluster topologies and have different hardware parameter information. Therefore, for different cluster networks, the fault prediction model and cluster availability model can be pre-configured based on the cluster hardware parameter information and cluster topology corresponding to the cluster network to ensure the accuracy and reliability of the model output.

[0027] In conjunction with the first aspect, in one possible implementation of the first aspect, the method further includes: optimizing the target fault propagation blocking strategy and / or the target fault recovery strategy based on the cluster availability model using a sensitivity analysis method to determine the optimized fault propagation blocking strategy and / or the optimized fault recovery strategy. In this embodiment, the fault propagation blocking strategy / fault recovery strategy may have optional parameters, such as the appropriate buffer size. This embodiment uses a sensitivity analysis method based on the cluster availability model to optimize the optional parameters in the fault propagation blocking strategy / fault recovery strategy to determine the optimal fault propagation blocking strategy / fault recovery strategy.

[0028] In conjunction with the first aspect, in one possible implementation of the first aspect, the step of optimizing the target fault propagation blocking strategy and / or the target fault recovery strategy based on the cluster availability model using a sensitivity analysis method to determine the optimized fault propagation blocking strategy and / or the optimized fault recovery strategy includes: using a sensitivity analysis method to determine multiple values ​​of a first undetermined parameter and / or multiple values ​​of a second undetermined parameter; wherein the target fault propagation blocking strategy includes the first undetermined parameter, and the target fault recovery strategy includes the second undetermined parameter; and inputting the multiple values ​​of the first undetermined parameter into the cluster availability model. The system obtains availability scores corresponding to multiple values ​​of the first undetermined parameter, and / or inputs multiple values ​​of the second undetermined parameter into the cluster availability model to obtain availability scores corresponding to multiple values ​​of the second undetermined parameter; a first target parameter is determined based on the highest score among the availability scores corresponding to the multiple values ​​of the first undetermined parameter, and / or a second target parameter is determined based on the highest score among the availability scores corresponding to the multiple values ​​of the second undetermined parameter; the optimized fault propagation blocking strategy is determined based on the first target parameter, and / or the optimized fault recovery strategy is determined based on the second target parameter. In the above implementation scheme, multiple values ​​corresponding to the undetermined parameters included in the strategy are obtained through sensitivity analysis, and availability scores are evaluated for each of the multiple values ​​based on the cluster availability model. The target parameter is determined based on the highest score among the availability scores corresponding to the multiple values ​​of the undetermined parameter to determine the optimized strategy. Combined with the cluster availability model, the parameter tuning of the target strategy is achieved through sensitivity analysis, further improving the reliability and effectiveness of fault blocking and fault recovery in the cluster network.

[0029] Secondly, embodiments of this application also provide a fault handling apparatus, the apparatus comprising:

[0030] The pattern recognition module is used to determine the node failure mode based on the node load data;

[0031] The fault prediction module is used to input the node fault mode into the fault prediction model to determine the first fault propagation chain;

[0032] The fault prediction module is further configured to input the node fault mode and at least one fault propagation blocking strategy into the fault prediction model to determine at least one second fault propagation chain; wherein, the at least one fault propagation blocking strategy is determined based on the node fault mode.

[0033] The strategy determination module is used to determine a target fault propagation blocking strategy based on the first fault propagation chain and the at least one second fault propagation chain.

[0034] In conjunction with the second aspect, in one possible implementation of the second aspect, the apparatus further includes:

[0035] The data acquisition module is used to acquire the node load data, which includes at least one of node CPU utilization, node memory utilization, and node network traffic.

[0036] In conjunction with the second aspect, in one possible implementation of the second aspect, the pattern recognition module includes:

[0037] The abnormal data determination submodule is used to determine abnormal load data from the node load data by combining historical load data;

[0038] The fault mode identification submodule is used to determine the node fault mode based on the abnormal load data.

[0039] In conjunction with the second aspect, in one possible implementation of the second aspect, the strategy determination module includes:

[0040] The fault blocking scoring submodule is used to determine the fault propagation blocking score of the at least one fault propagation blocking strategy based on the first fault propagation chain and the at least one second fault propagation chain.

[0041] The blocking strategy determination submodule is used to determine the target fault propagation blocking strategy based on the highest score among the fault propagation blocking scores of the at least one fault propagation blocking strategy.

[0042] In conjunction with the second aspect, in one possible implementation of the second aspect, the apparatus further includes:

[0043] A fault recovery scoring module is used to input at least one fault recovery strategy and a target fault propagation chain into a cluster availability model to determine the fault recovery score of the at least one fault recovery strategy, wherein the at least one fault recovery strategy is determined based on the node fault mode, and the target fault propagation chain includes the first fault propagation chain or the second fault propagation chain.

[0044] The recovery strategy determination module is used to determine a target fault recovery strategy based on the highest score among the fault recovery scores of the at least one fault recovery strategy.

[0045] In conjunction with the second aspect, in one possible implementation of the second aspect, the fault recovery scoring module is specifically used to abstract the software dynamic scheduling process corresponding to the at least one fault recovery strategy into an update process, and input the update process and the target fault propagation chain into the cluster availability model to determine the fault recovery score of the at least one fault recovery strategy.

[0046] In conjunction with the second aspect, in one possible implementation of the second aspect, the apparatus further includes:

[0047] The model configuration module is used to configure the fault prediction model and the cluster availability model based on the cluster hardware parameter information and the cluster topology.

[0048] In conjunction with the second aspect, in one possible implementation of the second aspect, the apparatus further includes:

[0049] The strategy optimization module is used to optimize the target fault propagation blocking strategy and / or the target fault recovery strategy based on the cluster availability model and using sensitivity analysis methods, so as to determine the optimized fault propagation blocking strategy and / or optimized fault recovery strategy.

[0050] In conjunction with the second aspect, in one possible implementation of the second aspect, the strategy optimization module includes:

[0051] The parameter determination submodule is used to determine multiple values ​​of a first undetermined parameter and / or multiple values ​​of a second undetermined parameter using a sensitivity analysis method; wherein, the target fault propagation blocking strategy includes the first undetermined parameter, and the target fault recovery strategy includes the second undetermined parameter;

[0052] The parameter evaluation submodule is used to input multiple values ​​of the first undetermined parameter into the cluster availability model to obtain the availability score corresponding to the multiple values ​​of the first undetermined parameter, and / or to input multiple values ​​of the second undetermined parameter into the cluster availability model to obtain the availability score corresponding to the multiple values ​​of the second undetermined parameter.

[0053] The parameter determination submodule is further configured to determine a first target parameter based on the highest available score among the multiple values ​​of the first undetermined parameter, and / or to determine a second target parameter based on the highest available score among the multiple values ​​of the second undetermined parameter.

[0054] The strategy optimization submodule is used to determine the optimized fault propagation blocking strategy based on the first target parameter, and / or to determine the optimized fault recovery strategy based on the second target parameter.

[0055] In conjunction with the second aspect, in one possible implementation of the second aspect, the cluster availability model includes a Markov chain-based state transition model.

[0056] In the second aspect of this application, the constituent modules of the fault handling apparatus may also perform the steps described in the first aspect and various possible implementations, as detailed in the foregoing description of the first aspect and various possible implementations.

[0057] Thirdly, embodiments of this application also provide a fault handling device, which may include a memory and a processor, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the method as described in any one of the first aspects above.

[0058] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any one of the first aspects above.

[0059] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in any one of the first aspects above.

[0060] Sixthly, this application provides a chip system including a processor for implementing the functions involved in the methods described in any one of the first aspects, such as transmitting or processing data or information involved in the methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for executing or training the device. This chip system may be composed of chips or may include chips and other discrete devices.

[0061] In a seventh aspect, embodiments of this application provide a chip including one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to perform the method described in any one of the first aspects above.

[0062] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions of the first aspect and any implementation thereof, and will not be repeated here. Attached Figure Description

[0063] Figure 1 is a schematic diagram of a cluster network architecture provided in an embodiment of this application;

[0064] Figure 2 is a schematic diagram of another cluster network architecture provided in an embodiment of this application;

[0065] Figure 3 is a flowchart illustrating a fault handling method provided in an embodiment of this application;

[0066] Figure 4 is a schematic diagram of a first fault propagation chain provided in an embodiment of this application;

[0067] Figure 5 is a schematic diagram of another first fault propagation chain provided in an embodiment of this application;

[0068] Figure 6 is a flowchart illustrating another fault handling method provided in an embodiment of this application;

[0069] Figure 7 is a flowchart illustrating another fault handling method provided in an embodiment of this application;

[0070] Figure 8 is a schematic diagram of a fault handling device provided in an embodiment of this application;

[0071] Figure 9 is a structural schematic diagram of a fault handling device provided in an embodiment of this application;

[0072] Figure 10 is a schematic diagram of the structure of a device cluster provided in an embodiment of this application. Detailed Implementation

[0073] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0074] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of singular or plural items. The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to those processes, methods, products, or apparatus.

[0075] Cluster pooling is a technology that integrates multiple computing, storage, or network resources into a unified resource pool, reusing resources or tasks to reduce the overhead of repeated creation and destruction. It's a technique that pre-creates and manages pools of resources or tasks to more efficiently manage and use these resources. This technology is widely used in data centers, AI training, cloud computing, and large-scale network environments to improve resource utilization, reduce costs, and enhance system flexibility and scalability.

[0076] Currently, cluster fault range detection technologies include reliability assessment schemes based on probability distributions. However, such reliability calculation frameworks are limited to fault range prediction under specific conditions, making it difficult to accurately predict and assess device fault conditions in cluster pooling environments and select effective fault propagation prevention schemes.

[0077] To address this technical problem, this application provides a fault handling method that can be applied to various devices in a cluster network. The fault handling method of this application can be executed by one or more devices in the cluster network, or by other devices outside the cluster network. This application does not impose any restrictions on this.

[0078] Figure 1 illustrates a schematic diagram of a cluster network architecture. This cluster network includes network element devices 101, data acquisition devices 102, and network devices 103. Network element devices 101 refer to devices capable of independently performing a specific function or task, such as routers, switches, servers, etc. For example, network element devices 101 may include multiple server devices. Data acquisition devices 102 are used to monitor and manage the entire cluster network, collecting relevant data from each device in the cluster network. For example, data acquisition devices 102 can collect monitoring information of the cluster network, including load data and log data of network element 101, as well as various alarm information in the network. Network devices 103 are devices used to execute the fault handling methods described in the various implementations of this application, and are used to detect faults in the cluster network. In some implementations, network device 103 can be any one of the network element device 101 or data acquisition device 102 in the cluster network, or it can be another device independent of network element device 101 and data acquisition device 102 and capable of communicating with network element device 101 and data acquisition device 102, or another device outside the cluster network. This application does not impose any restrictions on this. In Figure 1, each device can be referred to as a node or node device in the cluster network topology.

[0079] For example, Figure 2 illustrates another cluster network architecture. This cluster network includes a server cluster, which comprises one or more application servers 110 (Figure 2 shows three servers 110, but is not limited to three servers 110), and the servers 110 can communicate with each other. A server 110 is a device with both computing and storage capabilities, such as a server or desktop computer. In the implementation of this application, the server 110 can be used to execute the fault handling methods described in various implementations of this application. The fault handling methods of this application can be executed by one or more servers 110 in the cluster network, or by other devices outside the cluster network; this application does not impose any restrictions on this.

[0080] In terms of software, each server 110 has an operating system. Virtual machines (VMs) 107 can be created on server 110. The computing resources required by VMs 107 come from the local processor 112 and memory 113 of server 110, while the storage resources required by VMs 107 can come from the local hard disk 105 of server 110 or from the hard disk 105 of other servers 110. In addition, various applications can run in VMs 107, and users can trigger read / write data requests through the applications in VMs 107.

[0081] In terms of hardware, server 110 includes at least a processor 112, memory 113, network interface card (NIC) 114, and hard disk 105. The processor 112, memory 113, NIC 114, and hard disk 105 are connected via a bus. The processor 112 and memory 113 provide computing resources. Specifically, the processor 112 is a central processing unit (CPU) used to process data access requests from outside the server 110 or requests generated internally by the server 110. For example, when the processor 112 receives a write data request from a user, it temporarily stores the data in the write data request in memory 113. When the total amount of data in memory 113 reaches a certain threshold, the processor 112 sends the data stored in memory 113 to hard disk 105 for persistent storage. In addition, the processor 112 is also used for data computation or processing, such as metadata management, deduplication, data compression, data verification, virtualization of storage space, and address translation. Figure 2 shows only one central processing unit 112. In practical applications, there are often multiple central processing units 112, and each central processing unit 112 has one or more CPU cores. This embodiment does not limit the number of CPUs or the number of CPU cores.

[0082] Memory 113 refers to internal memory that directly exchanges data with the processor. It can read and write data at any time and at high speed, serving as temporary data storage for the operating system or other running programs. Memory includes at least two types of memory, such as random access memory (RAM) or read-only memory (ROM). For example, RAM can be Dynamic Random Access Memory (DRAM) or Storage Class Memory (SCM). DRAM is a semiconductor memory and, like most RAM, is a volatile memory device. SCM is a hybrid storage technology that combines the characteristics of traditional storage devices and RAM. Storage Class Memory offers faster read and write speeds than hard drives but slower access speeds than DRAM and is cheaper. However, DRAM and SCM are only illustrative examples in this embodiment; memory can also include other types of RAM, such as Static Random Access Memory (SRAM). For read-only memory, examples include programmable read-only memory (PROM) and erasable programmable read-only memory (EPROM). Additionally, memory 113 can also be a dual in-line memory module (DIMM), i.e., a module composed of dynamic random access memory (DRAM), or a solid-state drive (SSD). In practical applications, server 110 can be configured with multiple memory modules 113, and different types of memory modules 113. This embodiment does not limit the number or type of memory modules 113. Furthermore, memory modules 113 can be configured to have a power-saving function. The power-saving function means that when the system experiences a power outage and then power is restored, the data stored in memory modules 113 will not be lost. Memory with a power-saving function is called non-volatile memory.

[0083] Hard disk 105 provides storage resources, such as data storage. It can be a disk or other types of storage media, such as a solid-state drive or a shingled magnetic recording hard disk. Network interface card 114 is used to communicate with other application servers 110.

[0084] This application provides a fault handling method that uses a fault prediction model to predict the impact of a single node failure on the cluster network, i.e., the impact of a single node failure within the cluster topology on the overall computing cluster. Based on the fault prediction model, the method determines the corresponding fault propagation chain under different fault propagation blocking strategies. The fault propagation chain indicates the nodes affected by the fault in the cluster and the direction of fault propagation. Therefore, the fault propagation chain can reflect the propagation range and impact of the fault in the cluster. The reliability of different fault propagation blocking strategies is evaluated through the fault propagation chain, thereby determining the final fault propagation blocking strategy and improving the reliability of fault propagation blocking strategy selection.

[0085] The fault handling method provided in this application will be described below with reference to a flowchart of a fault handling method shown in Figure 3. The fault handling method includes:

[0086] Step 301: Determine the node failure mode based on node load data.

[0087] The node load data refers to the load data corresponding to any node in the cluster network, where any node refers to any device among the various devices included in the cluster network. For example, any node in the cluster network can be network element device 101, data acquisition device 102, or network device 103 in Figure 1, or any server 110 in Figure 2. The node failure mode indicates the type and severity of the node failure.

[0088] It is understood that the failure modes mentioned in some embodiments of this application include link disconnection, equipment quality degradation, excessive network connectivity delay, etc., and this application does not limit these.

[0089] In an optional implementation, the fault handling method provided in this application further includes: acquiring node load data, wherein the node load data includes at least one of node CPU utilization, node memory utilization, and node network traffic.

[0090] In this embodiment, the method of obtaining node load data is not limited. For example, load data of each node in the cluster can be obtained using a real-time system resource monitoring tool. Such real-time system resource monitoring tools may include DStat, Atop, Nmon, Slabtop, Saidar, vmstat, iperf, etc.

[0091] In one optional implementation, step 301 specifically includes: combining historical load data to determine abnormal load data from node load data; and determining node failure modes based on the abnormal load data.

[0092] Specifically, in this embodiment, abnormal load data is determined from node load data by combining historical load data, and node failure modes corresponding to these abnormal load data are identified. Abnormal load data refers to data that is abnormal compared to normal load data, while normal load data refers to the load data corresponding to a node under normal operation without failure.

[0093] Historical load data includes historical normal data of nodes and historical fault data of nodes. Historical fault data can include fault data corresponding to each node in the cluster network when each node is the root cause of the fault and fault data corresponding to each node when each node is a node affected by the faulty node. Fault data is the relevant load data corresponding to the node when it generates a fault or is affected by the faulty node.

[0094] This application does not limit the method of determining node failure modes based on abnormal load data. For example, determining node failure modes based on abnormal load data in this application can be implemented using machine learning algorithms. For instance, node failure modes can be identified using clustering analysis algorithms, pattern recognition algorithms, supervised learning algorithms, etc. The pattern recognition algorithm can include algorithms such as Bayesian classifiers, K-nearest neighbor analysis, principal component analysis, and non-negative matrix factorization. For example, determining node failure modes based on abnormal load data in this application can also be implemented using non-machine learning algorithms. For instance, node failure modes can be identified using rule-based logic judgment, lookup table methods, etc.

[0095] It should be noted that the node load data in this embodiment may also include other types of load data, such as: node storage resource utilization, node resource lifecycle information, node resource-borne service information, server architecture characteristics, server system characteristics, server interconnection characteristics, server cluster characteristics, etc. This embodiment does not impose any limitations, and those skilled in the art can make appropriate selections based on their evaluation needs.

[0096] In an optional implementation, the fault handling method provided in this application further includes: performing data cleaning and preprocessing on the node load data to obtain processed node load data; wherein, data cleaning includes at least one of the following algorithms: deduplication algorithm, median and mean statistical analysis algorithm, data format and logical analysis verification algorithm; and preprocessing includes normalization algorithm or standardization algorithm.

[0097] In this embodiment, data cleaning removes invalid, duplicate, and inaccurate data, preventing abnormal data from affecting subsequent fault mode identification. Normalization or standardization algorithms are used to standardize the load data, making it easier to identify node fault modes.

[0098] In one optional implementation, step 301 specifically includes: combining historical load data to determine processed abnormal load data from the processed node load data; and determining node failure modes based on the processed abnormal load data. This embodiment of the application identifies node failure modes based on the processed node load data, ensuring the accuracy of the identification.

[0099] Step 302: Input the node failure mode into the failure prediction model to determine the first failure propagation chain.

[0100] Since there are topological links between nodes in a cluster, a failure in one node may affect other nodes, meaning there may be connections between failures of different nodes. Therefore, this application uses a fault prediction model to identify the chain of fault propagation in the cluster when a node failure mode occurs in a node. In other words, this application uses a fault propagation chain to represent the fault propagation situation of each node in the cluster network. For example, in one optional implementation, as shown in Figure 4, after node A experiences a certain node failure mode, the fault prediction model identifies the first fault propagation chain as: Node A → Node B → Node C. In another optional implementation, → indicates the direction of fault propagation. As shown in Figure 5, the fault prediction model may also identify the first fault propagation chain as: Node D ← Node A → Node B → Node C. This application does not limit the specific propagation method of the fault propagation chain.

[0101] It should be noted that the fault prediction model used in the embodiments of this application can be pre-built and trained. Specifically, the fault prediction model is a regression model of the fault's temporal state, such as a neural network model or a decision tree model. This application does not limit the specific type of fault prediction model. In some implementations of this application, the fault detection model can be a feedback neural network (FNN), a fully connected neural network (FCNN), a convolutional neural network (CNN), a recurrent neural network (RNN), or other neural network models. This application does not impose any restrictions on this. In some implementations of this application, the fault detection model can also be a random forest, a decision tree, or other similar models. This application does not impose any restrictions on this.

[0102] Step 303: Input the node failure mode and at least one failure propagation blocking strategy into the failure prediction model to determine at least one second failure propagation chain.

[0103] Specifically, in this application embodiment, the node failure mode and at least one fault propagation blocking strategy are input into the fault prediction model to obtain at least one second fault propagation chain generated under the action of at least one fault propagation blocking strategy. That is, the second fault propagation chain is a fault propagation chain obtained by superimposing the effect of the fault propagation blocking strategy on the basis of the first fault propagation chain.

[0104] In this embodiment, at least one fault propagation blocking strategy is determined based on node fault modes. This strategy is used to prevent faults from spreading within the cluster. In this application embodiment, a fault propagation blocking strategy library can be pre-set, and a mapping relationship exists between node fault modes and fault propagation blocking strategies. Based on this mapping relationship, at least one fault propagation blocking strategy can be determined from the library. It should be noted that this application embodiment does not limit the specific content of the fault propagation blocking strategy. For example, fault propagation blocking strategies include setting resource isolation, load balancing, and disabling nodes.

[0105] Step 304: Determine the target fault propagation blocking strategy based on the first fault propagation chain and at least one second fault propagation chain.

[0106] Specifically, in this embodiment, the first fault propagation chain is the fault propagation chain obtained without implementing any fault propagation blocking strategy, and the second fault propagation chain is the fault propagation chain obtained with the implementation of a fault propagation blocking strategy. Each fault propagation blocking strategy corresponds to a second fault propagation chain. By comparing the first and second fault propagation chains, the effectiveness and reliability of the fault propagation blocking strategy in blocking fault chains in the cluster can be evaluated, so as to determine the target fault propagation blocking strategy and improve the reliability of fault propagation blocking strategy selection. For example, in an optional implementation, the first fault propagation chain is node D←node A→node B→node C, the second fault propagation chain corresponding to the first fault propagation blocking strategy is node A→node B, and the second fault propagation chain corresponding to the second fault propagation blocking strategy is node D←node A→node B. In comparison, the first fault propagation blocking strategy has a better fault blocking effect than the second fault propagation blocking strategy. Therefore, the first fault propagation blocking strategy can be determined as the target fault propagation blocking strategy.

[0107] In an optional implementation, step 304 specifically includes: determining a fault propagation blocking score for at least one fault propagation blocking strategy based on a first fault propagation chain and at least one second fault propagation chain; and determining a target fault propagation blocking strategy based on the highest score among the fault propagation blocking scores of at least one fault propagation blocking strategy.

[0108] Specifically, in this application embodiment, each fault propagation blocking strategy corresponds to a second fault propagation chain. The second fault propagation chain can characterize the fault propagation blocking effect of the corresponding fault propagation blocking strategy. By comparing the first fault propagation chain and the second fault propagation chain, the effectiveness and reliability of the fault propagation blocking strategy in blocking the fault chain in the cluster can be evaluated. Therefore, by evaluating the second fault propagation chain corresponding to each fault propagation blocking strategy, the fault propagation blocking strategy with the best effectiveness and reliability, i.e., the target fault propagation blocking strategy, can be determined from each fault propagation blocking strategy.

[0109] In an optional implementation, the fault handling method provided in this application further includes: inputting at least one fault recovery strategy and a target fault propagation chain into a cluster availability model to determine a fault recovery score for at least one fault recovery strategy; wherein the target fault propagation chain includes a first fault propagation chain or a second fault propagation chain; and determining the target fault recovery strategy based on the highest score among the fault recovery scores of at least one fault recovery strategy.

[0110] Specifically, at least one fault recovery strategy is determined based on the node failure mode, and the fault recovery strategy is used to restore the failed node to a normal state. In this embodiment, a fault recovery strategy library can be pre-set, and there is a mapping relationship between node failure modes and fault recovery strategies. Based on this mapping relationship, at least one fault recovery strategy can be determined from the fault recovery strategy library. It should be noted that this embodiment does not limit the specific content of the fault recovery strategy. For example, fault recovery strategies include automatic restart, node replacement, and connection reconstruction. In the above implementation scheme, a cluster availability model can be constructed based on the fault propagation chain, and the fault recovery score corresponding to each fault recovery strategy can be evaluated based on the cluster availability model. The fault recovery strategy with the highest score is determined as the target fault recovery strategy.

[0111] In this embodiment, the cluster availability model combines serial and parallel relationships by analyzing the maintenance status of the overall system after different hardware component failures to ultimately evaluate the mean time between failures (MTBF) of cluster reliability under different fault recovery strategies. Cluster availability refers to the ability of a cluster network to provide services normally within a specific time. It can be evaluated by calculating the impact of node failures on service availability. In this embodiment, the cluster availability model is constructed based on the fault propagation chain predicted by the fault prediction model. The cluster availability model can be used to evaluate the effectiveness and reliability of fault recovery strategies.

[0112] In an optional implementation, the above method inputs at least one fault recovery strategy and the target fault propagation chain into the cluster availability model to determine the fault recovery score of at least one fault recovery strategy. Specifically, it includes: abstracting the software dynamic scheduling process corresponding to at least one fault recovery strategy into an update process, and inputting the update process and the target fault propagation chain into the cluster availability model to determine the fault recovery score of at least one fault recovery strategy.

[0113] Specifically, in this embodiment, the software dynamic scheduling process corresponding to the fault recovery strategy is first abstracted into an update process. This update process is a mapping of the fault recovery strategy at the software level. Then, the update process and the fault propagation chain are input into the cluster availability model to obtain the fault recovery score corresponding to the fault recovery strategy. The cluster availability model may include the overall topology of the cluster network. Outputting the fault propagation chain into the cluster availability model can determine the node topology information needed in the cluster availability model during this availability assessment, improving the targeting of the assessment object and thus improving the efficiency and accuracy of the availability assessment.

[0114] In one optional implementation, the cluster availability model provided in this application is a Markov chain-based state transition model. The availability calculation model is constructed by analyzing the stochastic process of Markov chains in the state space transitioning from one state, such as a normal state or a fault state, to another state, such as a fault state or a normal state.

[0115] In an optional implementation, the fault handling method provided in this application further includes: configuring a fault prediction model and a cluster availability model based on cluster hardware parameter information and cluster topology.

[0116] Specifically, different cluster networks may have different cluster topologies and hardware parameters. This application embodiment, for different cluster networks, can pre-configure the fault prediction model and cluster availability model based on the cluster hardware parameters and cluster topology to ensure the accuracy and reliability of the model output. Pre-configuring the fault prediction model and cluster availability model can involve adjusting and configuring the parameters in the model, or it can involve training and iteratively fitting the fault prediction model and cluster availability model based on the cluster hardware parameters and cluster topology until the model loss reaches convergence accuracy. This application embodiment does not limit the pre-configuration of the fault prediction model and cluster availability model.

[0117] In an optional implementation, the fault handling method provided in this application further includes: optimizing the target fault propagation blocking strategy and / or the target fault recovery strategy based on the cluster availability model using a sensitivity analysis method, so as to determine the optimized fault propagation blocking strategy and / or the optimized fault recovery strategy.

[0118] Specifically, in the embodiments of this application, there may be optional parameters in the fault propagation blocking strategy / fault recovery strategy, such as how large the buffer size should be. The embodiments of this application use a sensitivity analysis method, based on the cluster availability model, to optimize the optional parameters in the fault propagation blocking strategy / fault recovery strategy in order to determine the optimal fault propagation blocking strategy / fault recovery strategy.

[0119] Sensitivity analysis, in particular, refers to a method for analyzing how sensitive changes in the state or output of a model or system are to changes in system parameters or surrounding conditions. Sensitivity analysis typically involves changing the input parameter variables in the model, applying incremental gradients as a percentage of the baseline parameter values, and assessing the degree of change in the model's output. Through sensitivity analysis, the stability of the model can be tested, and the impact of parameter changes on the model or system can be determined. In this embodiment, by incorporating sensitivity analysis under the evaluation of a cluster availability model, the optimal parameters corresponding to the fault propagation prevention strategy / fault recovery strategy can be determined, thereby identifying the optimal fault propagation prevention strategy / fault recovery strategy.

[0120] In an optional implementation, the above method, based on a cluster availability model, employs sensitivity analysis to optimize the target fault propagation prevention strategy and / or the target fault recovery strategy to determine the optimized fault propagation prevention strategy and / or the optimized fault recovery strategy. This includes: using sensitivity analysis to determine multiple values ​​of a first undetermined parameter and / or multiple values ​​of a second undetermined parameter; wherein the target fault propagation prevention strategy includes the first undetermined parameter, and the target fault recovery strategy includes the second undetermined parameter; inputting multiple values ​​of the first undetermined parameter into the cluster availability model to obtain availability scores corresponding to the multiple values ​​of the first undetermined parameter, and / or inputting multiple values ​​of the second undetermined parameter into the cluster availability model to obtain availability scores corresponding to the multiple values ​​of the second undetermined parameter; determining a first target parameter based on the highest score among the availability scores corresponding to the multiple values ​​of the first undetermined parameter, and / or determining a second target parameter based on the highest score among the availability scores corresponding to the multiple values ​​of the second undetermined parameter; determining the optimized fault propagation prevention strategy based on the first target parameter, and / or determining the optimized fault recovery strategy based on the second target parameter.

[0121] Specifically, in this embodiment, a sensitivity analysis method is used to obtain multiple values ​​corresponding to the undetermined parameters included in the strategy. Based on a cluster availability model, availability scores are evaluated for each of these multiple values. The target parameter is determined based on the highest availability score among the multiple values ​​of the undetermined parameter, thus determining the optimized strategy. By combining the cluster availability model with the sensitivity analysis method, parameter tuning of the target strategy is achieved, further improving the reliability and effectiveness of fault blocking and fault recovery in the cluster network.

[0122] Figure 6 shows a flowchart of another fault handling method applied in this embodiment of the application. This method consists of three stages: cluster hardware load perception, fault propagation quantification analysis, and fault operation and maintenance benefit analysis.

[0123] The cluster hardware load awareness stage is used to acquire node load data and determine node failure modes. Specifically, this stage includes two sub-stages: cluster heterogeneous resource awareness and cluster physical server awareness. Cluster heterogeneous resource awareness is used to sense and collect information such as the CPU, memory, storage, network, resource lifecycle, services carried by resources, and heterogeneous resource pooling characteristics of cluster nodes. Cluster physical server awareness is used to sense and collect information such as server architecture characteristics, server system characteristics, server interconnection characteristics, and server cluster characteristics. This information can be used as node load data and for identifying node failure modes. It should be noted that this cluster hardware load awareness stage can be used to acquire the node load data and node failure modes involved in the above embodiments.

[0124] The fault propagation quantification analysis step involves identifying the fault propagation range, i.e., the fault propagation chain, which can include the first fault propagation chain and / or the second fault propagation chain in the above embodiments. Specifically, this step includes three sub-steps: pattern recognition, resource management modeling, and fault propagation modeling. Pattern recognition includes load pattern analysis and resource utilization identification; resource management modeling includes modeling of heterogeneous resource management and cluster state management; and fault propagation modeling includes modeling the fault propagation range and the degree of fault propagation. It should be noted that this fault propagation quantification analysis step can be used to determine the first and second fault propagation chains involved in the above embodiments.

[0125] The fault operation and maintenance benefit analysis step is crucial for determining the target fault propagation prevention strategy. Specifically, this step includes three sub-steps: availability modeling, adaptive scheduling, and cluster operation and maintenance benefit modeling. A cluster availability model is constructed based on the abstraction of fault types. Availability modeling includes a state transition model and an update process model. The state transition model is a Markov chain-based model, and the update process model abstracts the software dynamic scheduling process corresponding to the strategy into an update process model. Adding this update process model to the state transition model yields the cluster availability model. Adaptive scheduling includes adaptive scheduling and multi-granularity scheduling, used to adjust and optimize strategy parameters. Cluster operation and maintenance benefit modeling includes availability simulation, sensitivity analysis, and recovery time modeling. Availability assessment and recovery time assessment allow for the evaluation of the cluster operation and maintenance benefits of different strategies, thus determining the final strategy. It should be noted that this fault operation and maintenance benefit analysis step can be used to determine the target fault propagation prevention strategy and target fault recovery strategy involved in the above embodiments. Furthermore, this step can also be used to determine the optimized fault propagation prevention strategy and optimized fault recovery strategy involved in the above embodiments.

[0126] Figure 7 shows a flowchart of another fault handling method applied in this application embodiment. This method is based on six basic modules to implement the fault handling method provided in this application, including:

[0127] (1) Load awareness module. This module is used to collect node load data in the cluster, including resource usage and fault information under pooled management.

[0128] (2) Data processing module. This module is used to analyze cluster load data, analyze fault modes with the help of data tags, and identify node fault modes. The functions implemented by this module correspond to step 301 above.

[0129] (3) Pooling Fault Prediction Module. Based on the fault prediction model, the impact of faults caused by a single node is analyzed in the pooling scenario, the cluster performance degradation is analyzed, and the fault propagation range is obtained, i.e., the fault propagation chain. The fault propagation chain may include the first fault propagation chain and / or the second fault propagation chain in the above embodiments. The functions implemented by this module correspond to steps 302 and 303 above.

[0130] (4) Propagation blocking assessment module. Based on the fault prediction model, the blocking effect of the given fault propagation blocking strategy is evaluated, and the fault blocking strategy selection is completed. The function implemented by this module corresponds to step 304 above.

[0131] (5) Recovery Strategy Evaluation Module. For a given fault recovery strategy, a reliability assessment at the cluster level is completed based on the availability model, and the cluster recovery strategy is selected.

[0132] (6) Fault Operation and Maintenance Optimization Module. Based on the fault propagation blocking strategy and fault recovery strategy determined in the previous steps, the fault operation and maintenance scheduling scheme is selected. Sensitivity analysis is used to identify the optimal parameters configured by the strategy and to clarify the final direction of fault operation and maintenance optimization.

[0133] To facilitate better implementation of the above-described solutions in the embodiments of this application, related apparatus for implementing the above-described solutions is also provided below.

[0134] As shown in Figure 8, this application embodiment provides a fault handling device 800, which includes: a pattern recognition module 801, a fault prediction module 802, and a strategy determination module 803; wherein,

[0135] The pattern recognition module is used to determine the node failure mode based on node load data;

[0136] The fault prediction module is used to input node fault modes into the fault prediction model to determine the first fault propagation chain;

[0137] The fault prediction module is also used to input node fault modes and at least one fault propagation blocking strategy into the fault prediction model to determine at least one second fault propagation chain; wherein, at least one fault propagation blocking strategy is determined based on node fault modes.

[0138] The strategy determination module is used to determine the target fault propagation blocking strategy based on the first fault propagation chain and at least one second fault propagation chain.

[0139] In this embodiment, the fault handling device can be any node device, some node devices, or other devices set up outside the cluster network in a cluster pooling scenario. This application does not impose any limitations. In this embodiment, the module is an example of a software functional unit, and the fault handling device may include code running on a computing instance. The computing instance can be at least one of a physical host (computing device), a virtual machine, a container, etc. Further, the aforementioned computing device can be one or more. For example, the fault handling device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically proximate data centers. Typically, a region may include multiple AZs.

[0140] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0141] As an example of a hardware functional unit, a fault handling device may include at least one computing device, such as a server. Alternatively, the fault handling device may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex PLD (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0142] The fault handling unit comprises multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0143] In one alternative embodiment, the apparatus further includes:

[0144] The data acquisition module is used to acquire node load data, which includes at least one of the following: node CPU utilization, node memory utilization, and node network traffic.

[0145] In one alternative implementation, the pattern recognition module includes:

[0146] The abnormal data identification submodule is used to identify abnormal load data from node load data by combining historical load data.

[0147] The fault mode identification submodule is used to determine the node fault mode based on abnormal load data.

[0148] In one alternative implementation, the strategy determination module includes:

[0149] The fault blocking scoring submodule is used to determine the fault propagation blocking score of at least one fault propagation blocking strategy based on a first fault propagation chain and at least one second fault propagation chain.

[0150] The blocking strategy determination submodule is used to determine the target fault propagation blocking strategy based on the highest score among the fault propagation blocking scores of at least one fault propagation blocking strategy.

[0151] In one alternative embodiment, the apparatus further includes:

[0152] The fault recovery scoring module is used to input at least one fault recovery strategy and the target fault propagation chain into the cluster availability model to determine the fault recovery score of at least one fault recovery strategy. The at least one fault recovery strategy is determined based on the node failure mode, and the target fault propagation chain includes a first fault propagation chain or a second fault propagation chain.

[0153] The recovery strategy determination module is used to determine the target fault recovery strategy based on the highest score among the fault recovery scores of at least one fault recovery strategy.

[0154] In one optional implementation, the fault recovery scoring module is specifically used to abstract the software dynamic scheduling process corresponding to at least one fault recovery strategy into an update process, and input the update process and the target fault propagation chain into the cluster availability model to determine the fault recovery score of at least one fault recovery strategy.

[0155] In one alternative embodiment, the apparatus further includes:

[0156] The model configuration module is used to configure fault prediction models and cluster availability models based on cluster hardware parameter information and cluster topology.

[0157] In one alternative embodiment, the apparatus further includes:

[0158] The strategy optimization module is used to optimize the target fault propagation blocking strategy and / or target fault recovery strategy based on the cluster availability model and using sensitivity analysis methods, so as to determine the optimized fault propagation blocking strategy and / or optimized fault recovery strategy.

[0159] In one optional implementation, the strategy optimization module includes:

[0160] The parameter determination submodule is used to determine multiple values ​​of a first undetermined parameter and / or multiple values ​​of a second undetermined parameter using a sensitivity analysis method; wherein, the target fault propagation blocking strategy includes the first undetermined parameter, and the target fault recovery strategy includes the second undetermined parameter;

[0161] The parameter evaluation submodule is used to input multiple values ​​of the first undetermined parameter into the cluster availability model to obtain the availability score corresponding to the multiple values ​​of the first undetermined parameter, and / or to input multiple values ​​of the second undetermined parameter into the cluster availability model to obtain the availability score corresponding to the multiple values ​​of the second undetermined parameter.

[0162] The parameter determination submodule is also used to determine the first target parameter based on the highest score among the availability scores corresponding to multiple values ​​of the first undetermined parameter, and / or to determine the second target parameter based on the highest score among the availability scores corresponding to multiple values ​​of the second undetermined parameter;

[0163] The strategy optimization submodule is used to determine the optimized fault propagation blocking strategy based on the first target parameter, and / or to determine the optimized fault recovery strategy based on the second target parameter.

[0164] In one alternative implementation, the cluster availability model includes a Markov chain-based state transition model.

[0165] To facilitate better implementation of the above-described solutions in the embodiments of this application, this application also provides a fault handling device 90. As shown in FIG9, the fault handling device 90 includes: a bus 99, a processor 94, a memory 96, and a communication interface 98. The processor 94, the memory 96, and the communication interface 98 communicate with each other via the bus 99. The fault handling device 90 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the fault handling device 90.

[0166] Bus 99 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not imply that there is only one bus or one type of bus. Bus 99 can include pathways for transmitting information between various components of fault handling device 90 (e.g., memory 96, processor 94, communication interface 98).

[0167] Processor 94 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0168] The memory 96 may include volatile memory, such as random access memory (RAM). The processor 94 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0169] The memory 96 stores executable program code, and the processor 94 executes the executable program code to implement the functions of the aforementioned acquisition module and training module, thereby realizing the fault handling method applied to the fault handling device in the above embodiment. That is, the memory 96 stores instructions for executing the fault handling method in the above embodiment.

[0170] The communication interface 98 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the fault handling device 90 and other devices or communication networks.

[0171] This application also provides a device cluster. The device cluster includes at least one node device 100. The node device 100 can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the node device can also be a terminal device such as a desktop computer, laptop computer, or smartphone.

[0172] It should be noted that the node device 100 included in the device cluster can be the network element device 101, data acquisition device 102 or network device 103 in the cluster network shown in Figure 1, or the server 110 in the cluster network shown in Figure 2, or the fault handling device 90 shown in Figure 9.

[0173] As shown in Figure 10, the device cluster includes at least one node device 100. The node device 100 includes a bus 109, a processor 104, a memory 106, and a communication interface 108. The processor 104, memory 106, and communication interface 108 are connected via the bus 109. The memory 106 of one or more node devices 100 in the device cluster may store the same instructions for executing fault handling methods.

[0174] In some possible implementations, the memory 106 of one or more node devices 100 in the device cluster may also store partial instructions for executing the fault handling method. In other words, a combination of one or more node devices 100 can jointly execute the instructions for executing the fault handling method.

[0175] It should be noted that the memory 106 in different node devices 100 in the device cluster can store different instructions, which are used to execute part of the functions of the fault handling method.

[0176] In some possible implementations, one or more node devices in a device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar.

[0177] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computer or stored on any usable medium. When the computer program product is run on at least one computer, it causes the at least one computer to perform a fault-handling method.

[0178] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by node devices in a cluster network, or a data storage device such as a data center containing one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that instruct node devices to perform a fault handling method.

[0179] This application also provides a chip system including a processor for implementing the steps performed by the aforementioned fault handling device. In one possible design, the chip system may further include a memory for storing necessary program instructions and data. This chip system may be composed of chips or may include chips and other discrete devices.

[0180] This application also provides a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to perform the steps performed by the above-mentioned fault handling device.

[0181] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0183] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0185] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A failure handling method characterized by, The method includes: Determine node failure modes based on node load data; The node failure mode is input into the failure prediction model to determine the first failure propagation chain; The node failure mode and at least one failure propagation blocking strategy are input into the failure prediction model to determine at least one second failure propagation chain; wherein, the at least one failure propagation blocking strategy is determined based on the node failure mode. A target fault propagation blocking strategy is determined based on the first fault propagation chain and the at least one second fault propagation chain.

2. The method of claim 1, wherein, The method further includes: The node load data is obtained, which includes at least one of node CPU utilization, node memory utilization, and node network traffic.

3. The method according to claim 1 or 2, characterized in that, The determination of node failure modes based on node load data includes: By combining historical load data, abnormal load data is identified from the node load data; The node failure mode is determined based on the abnormal load data.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the target fault propagation blocking strategy based on the first fault propagation chain and the at least one second fault propagation chain includes: The fault propagation blocking score of the at least one fault propagation blocking strategy is determined based on the first fault propagation chain and the at least one second fault propagation chain. The target fault propagation blocking strategy is determined based on the highest score among the fault propagation blocking scores of the at least one fault propagation blocking strategy.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: At least one fault recovery strategy and a target fault propagation chain are input into the cluster availability model to determine the fault recovery score of the at least one fault recovery strategy, wherein the at least one fault recovery strategy is determined based on the node fault mode, and the target fault propagation chain includes the first fault propagation chain or the second fault propagation chain. The target fault recovery strategy is determined based on the highest score among the fault recovery scores of the at least one fault recovery strategy.

6. The method of claim 5, wherein, The step of inputting at least one fault recovery strategy and the target fault propagation chain into the cluster availability model to determine the fault recovery score of the at least one fault recovery strategy includes: The software dynamic scheduling process corresponding to the at least one fault recovery strategy is abstracted into an update process, and the update process and the target fault propagation chain are input into the cluster availability model to determine the fault recovery score of the at least one fault recovery strategy.

7. The method according to claim 5 or 6, characterized in that, The method further includes: The fault prediction model and the cluster availability model are configured based on cluster hardware parameter information and cluster topology.

8. The method according to any one of claims 5-7, characterized in that, The method further includes: Based on the cluster availability model, the target fault propagation blocking strategy and / or the target fault recovery strategy are optimized using sensitivity analysis to determine the optimized fault propagation blocking strategy and / or the optimized fault recovery strategy.

9. The method of claim 8, wherein, The optimization of the target fault propagation prevention strategy and / or the target fault recovery strategy based on the cluster availability model using sensitivity analysis to determine the optimized fault propagation prevention strategy and / or optimized fault recovery strategy includes: Sensitivity analysis is used to determine multiple values ​​of a first undetermined parameter and / or multiple values ​​of a second undetermined parameter; wherein, the target fault propagation blocking strategy includes the first undetermined parameter, and the target fault recovery strategy includes the second undetermined parameter; Multiple values ​​of the first undetermined parameter are input into the cluster availability model to obtain availability scores corresponding to the multiple values ​​of the first undetermined parameter, and / or multiple values ​​of the second undetermined parameter are input into the cluster availability model to obtain availability scores corresponding to the multiple values ​​of the second undetermined parameter. The first target parameter is determined based on the highest available score among the multiple values ​​of the first undetermined parameter, and / or the second target parameter is determined based on the highest available score among the multiple values ​​of the second undetermined parameter. The optimized fault propagation blocking strategy is determined based on the first target parameter, and / or the optimized fault recovery strategy is determined based on the second target parameter.

10. The method according to any one of claims 5-9, characterized in that, The cluster availability model includes a state transition model based on Markov chains.

11. A failure handling apparatus characterized by comprising: The device includes: The pattern recognition module is used to determine the node failure mode based on the node load data; The fault prediction module is used to input the node fault mode into the fault prediction model to determine the first fault propagation chain; The fault prediction module is further configured to input the node fault mode and at least one fault propagation blocking strategy into the fault prediction model to determine at least one second fault propagation chain; wherein, the at least one fault propagation blocking strategy is determined based on the node fault mode. The strategy determination module is used to determine a target fault propagation blocking strategy based on the first fault propagation chain and the at least one second fault propagation chain.

12. The apparatus of claim 11, wherein, The pattern recognition module includes: The abnormal data determination submodule is used to determine abnormal load data from the node load data by combining historical load data; The fault mode identification submodule is used to determine the node fault mode based on the abnormal load data.

13. The apparatus of claim 11, wherein, The strategy determination module includes: The fault blocking scoring submodule is used to determine the fault propagation blocking score of the at least one fault propagation blocking strategy based on the first fault propagation chain and the at least one second fault propagation chain. The blocking strategy determination submodule is used to determine the target fault propagation blocking strategy based on the highest score among the fault propagation blocking scores of the at least one fault propagation blocking strategy.

14. The apparatus of any one of claims 11-13, wherein, The device further includes: A fault recovery scoring module is used to input at least one fault recovery strategy and a target fault propagation chain into a cluster availability model to determine the fault recovery score of the at least one fault recovery strategy, wherein the at least one fault recovery strategy is determined based on the node fault mode, and the target fault propagation chain includes the first fault propagation chain or the second fault propagation chain. The recovery strategy determination module is used to determine a target fault recovery strategy based on the highest score among the fault recovery scores of the at least one fault recovery strategy.

15. The apparatus of claim 14, wherein, The device further includes: The strategy optimization module is used to optimize the target fault propagation blocking strategy and / or the target fault recovery strategy based on the cluster availability model and using sensitivity analysis methods, so as to determine the optimized fault propagation blocking strategy and / or optimized fault recovery strategy.

16. A fault handling device, characterized by The fault handling equipment includes: Memory is used to store computer programs or computer instructions; A processor for executing a computer program or computer instructions stored in the memory, causing the fault handling device to perform the method as described in any one of claims 1-10.

17. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 10.

18. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 10.

Images