Fault handling method and apparatus, electronic device, storage medium, and program product

Abstract

The present disclosure relates to the field of communication technology, and particularly relates to a fault processing method and device, electronic equipment, storage medium and program product. The fault processing method of the present disclosure comprises: obtaining a virtual link between a computing node and a storage node between a computing cluster to be investigated and a storage cluster; determining a node fault probability of each link node in the virtual link by using historical fault data; when the node fault probability of any one link node does not satisfy a hypothesis condition, verifying the hypothesis condition by using an overall distribution probability; when the hypothesis condition passes the verification, determining the link node as a fault node. In summary, by constructing a virtual link structure, the present disclosure can simplify the complex interaction relationship between the computing node and the storage node, that is, by constructing the virtual link structure, the complex interaction or nested relationship can be simplified as a connection between a series of link nodes, and when troubleshooting, each link node of each virtual link can be directly and efficiently investigated.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and particularly relates to a fault processing method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] At present, when performing fault processing in a resource pool, the state of all components such as virtual machines, physical machines, switches and storage devices needs to be checked, which causes the process of determining the fault cause to be time-consuming and laborious, and reduces the efficiency of the entire fault processing. Moreover, when the virtual machine where the service container is located appears "false death" (i.e. the virtual machine appears to be normally running but actually cannot respond to external requests), the existing fault processing method cannot accurately determine the fault cause, and the accuracy of fault processing is reduced. SUMMARY

[0003] The present disclosure provides a fault processing method, device, electronic equipment, storage medium and program product to solve the problem of low efficiency of the existing process of determining the fault cause to some extent.

[0004] According to one aspect of the present disclosure, a fault processing method is provided, comprising: obtaining a virtual link between a computing node and a storage node between a computing cluster to be checked and a storage cluster; determining a node fault probability of each link node in the virtual link by using historical fault data; when the node fault probability of any one link node does not satisfy a hypothesis condition, verifying the hypothesis condition by using an overall distribution probability; and when the hypothesis condition passes the verification, determining the link node as a fault node.

[0005] In addition, according to the method of one aspect of the present disclosure, the computing node includes at least one of the following: a physical machine, a virtual machine, and a container; when the computing node is a virtual machine, the link nodes included in the virtual link include: a virtual machine, a physical machine, a cluster link, and a storage node; wherein the cluster link is a link node corresponding to a communication path between the computing cluster and the storage cluster.

[0006] In addition, according to the method of one aspect of the present disclosure, the node fault probability of each link node in the virtual link is determined by using the historical fault data, comprising: obtaining historical fault data of each link node, the historical fault data including at least one of the following: an abnormal running state, a crash or a restart event; determining a link fault probability of the virtual link based on the historical fault data; and performing regression processing on any one link node in the virtual link to obtain the node fault probability of the link node.

[0007] In addition, according to the method of one aspect of the present disclosure, the hypothesis condition includes: the node fault probability of the virtual machine link node is equal to an average fault probability, and the average fault probability is an average value of the node fault probabilities of each link node in the virtual link.

[0008] Further, according to the method of one aspect of the present disclosure, the overall distribution probability is used to represent a probability that the overall sample data of the link node satisfies a preset distribution; the hypothesis condition is verified by using the overall distribution probability, including: obtaining historical sample data of the link node; obtaining the overall distribution probability of the link node based on the node failure probability and the historical sample data; and verifying whether the overall distribution probability satisfies a preset distribution corresponding to the hypothesis condition.

[0009] Further, according to the method of one aspect of the present disclosure, verifying whether the overall distribution probability satisfies the preset distribution corresponding to the hypothesis condition includes: when the overall distribution probability is greater than a preset first threshold value, determining that the hypothesis condition fails to pass the verification, and determining that the link node is normal; or when the overall distribution probability is less than or equal to the preset first threshold value, determining that the hypothesis condition passes the verification, and determining that the link node is abnormal, wherein the first threshold value is a probability threshold value of the overall distribution being the preset distribution.

[0010] Further, according to the method of one aspect of the present disclosure, the method further includes: obtaining index data of the link node, the index data including at least one of the following: central processing unit (CPU) utilization, memory, and disk input / output (I / O); and determining the failure cause based on the index data and the log data.

[0011] Further, according to the method of one aspect of the present disclosure, determining the failure cause based on the index data and the log data includes: for any one virtual machine, when at least one of the index data is abnormal and the preset keyword exists in the log data, determining that the failure cause of the virtual machine is memory leakage.

[0012] Further, according to the method of one aspect of the present disclosure, determining the failure cause based on the index data and the log data further includes: using a timing task to perform timing detection on the index data and the log data to determine the failure cause.

[0013] Further, according to the method of one aspect of the present disclosure, the method further includes: when the failure cause is determined, limiting a scheduling possibility of a container in the virtual machine; wherein limiting the scheduling possibility of the container in the virtual machine includes: marking a virtual machine where the container is located as non-schedulable.

[0014] According to another aspect of the present disclosure, a fault processing apparatus is provided, the apparatus including: an obtaining unit configured to obtain a virtual link between a computing node and a storage node between a computing cluster to be investigated and a storage cluster; a first determining unit configured to determine a node failure probability of each link node in the virtual link by using historical failure data; a verifying unit configured to verify a hypothesis condition by using an overall distribution probability when the node failure probability of any one link node does not satisfy the hypothesis condition; and a second determining unit configured to determine that the link node is a failure node when the hypothesis condition passes the verification.

[0015] According to still another aspect of the present disclosure, an electronic device is provided, comprising a memory for storing computer readable instructions, and a processor for running the computer readable instructions, so that the electronic device performs the method according to any one of the above embodiments.

[0016] According to yet another aspect of the present disclosure, a non-transitory computer readable storage medium is provided for storing computer readable instructions, which, when executed by a processor, cause the processor to perform the method according to any one of the above embodiments.

[0017] According to still another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any one of the above embodiments.

[0018] The present disclosure provides a fault processing method and device, electronic device, storage medium and program product. The present disclosure constructs a virtual link structure between the computing nodes and the storage nodes between the computing cluster and the storage cluster to be checked. The node failure probability of each link node of the virtual link is determined by using the historical fault data. When the node failure probability of any one link node does not meet the hypothesis condition, the hypothesis condition can be verified by using the overall distribution probability. When the hypothesis condition is determined, the link node is determined as a fault node. In summary, the present disclosure can simplify the complex interaction relationship between the computing nodes and the storage nodes by constructing the virtual link structure, i.e. by constructing the virtual link structure, the complex interaction or nested relationship can be simplified as a series of connections between the link nodes. When troubleshooting, each link node of the virtual link can be directly and efficiently checked. Meanwhile, the historical fault data of the link node and the overall distribution probability can be combined to directly point to the most possible fault node in the virtual link, so as to realize accurate positioning of the fault node. The fault reason in the virtual machine dead condition can also be effectively determined, and the accuracy of fault processing is improved.

[0019] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the subject technology. BRIEF DESCRIPTION OF DRAWINGS

[0020] The foregoing and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description, which proceeds with reference to the accompanying drawings. The accompanying drawings are provided to assist in understanding the present disclosure and constitute a part of the specification, which illustrate embodiments of the present disclosure, and together with the detailed description, serve to explain the present disclosure. The drawings do not limit the scope of the present disclosure, and serve only to illustrate the embodiments of the present disclosure. In the drawings, the same reference numbers generally refer to the same components or steps.

[0021] Figure 1A flowchart of a fault processing method provided by an embodiment of the present disclosure.

[0022] Figure 2 A network structure diagram of a virtual link provided by an embodiment of the present disclosure.

[0023] Figure 3 A structural block diagram of a fault processing device provided by an embodiment of the present disclosure.

[0024] Figure 4 A hardware block diagram of an electronic device provided by an embodiment of the present disclosure.

[0025] Figure 5 A schematic diagram of a computer-readable storage medium provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present disclosure more apparent, the following will describe example embodiments according to the present disclosure in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0027] At present, when performing fault processing in a resource pool, it is necessary to check the status of all components such as virtual machines, physical machines, switches and storage devices, which causes the process of determining the fault cause to be time-consuming and laborious, and reduces the efficiency of the entire fault processing. Moreover, in the case of a "false death" (i.e. the virtual machine appears to be normally running but actually cannot respond to external requests) of a virtual machine where a service container is located, the existing fault processing method cannot accurately determine the fault cause, and reduces the accuracy of fault processing.

[0028] Therefore, the present application proposes a fault processing method, which can realize efficient troubleshooting of faults by constructing a virtual link structure, and can accurately locate the fault node according to historical fault data and overall distribution probability, thereby improving the accuracy of fault processing. Please refer to Figure 1 , Figure 1 A flowchart of a fault processing method provided by an embodiment of the present disclosure. As shown in Figure 1 , the method comprises:

[0029] In step S101, a virtual link between a computing node and a storage node is obtained between a computing cluster to be checked and a storage cluster.

[0030] In the present disclosure, one or more computing clusters and one or more storage clusters can be provided. The computing cluster can be understood as a collection of multiple computing nodes connected to each other through a network, which work together to complete computing tasks. The computing nodes can include, but are not limited to, at least one of the following: physical machines, virtual machines, containers. Specifically, one or more virtual machines can be deployed on a physical machine, and one or more containers can be deployed on a virtual machine. Thus, a nested relationship of physical machines, virtual machines, and containers can be formed, and the specific deployment relationship can be adjusted according to the system or needs, which is not limited here.

[0031] The storage cluster can be understood as a system composed of at least one storage node, which is used to provide reliable and scalable data storage services. The storage node can be responsible for storing data and can also provide a data access interface to the computing cluster through the network. The storage node can include, but is not limited to, at least one of the following: storage pool, network attached storage.

[0032] In the present disclosure, in the process of constructing the virtual link structure, all entities (computing nodes or storage nodes) involved in the entire communication process can be understood as a link node in the virtual link. The connection between these nodes is virtual and does not directly correspond to their physical location or deployment method in the actual environment. The virtual link is a hypothetical link. By abstracting each entity in a complex system as a virtual link node, the interaction or nested relationship between these entities can be greatly simplified, making fault troubleshooting more efficient. At the same time, after constructing the virtual link structure, it is easier to identify nodes that are more likely to cause faults, which facilitates analysis and processing. Therefore, the constructed virtual link can timely and accurately determine the fault node.

[0033] Specifically, the network structure between the computing cluster and the storage cluster can be determined. Using the network structure, the computing nodes involved in data processing in the computing cluster can be identified. These nodes can be at least one of physical machines, virtual machines, or containers. Similarly, in the storage cluster, the storage nodes responsible for data storage and access can be identified. These nodes can be at least one of storage pools, network attached storages, etc. Based on the logical relationship between the computing nodes and the storage nodes, a virtual link can be constructed through at least one of a network management tool or an application programming interface.

[0034] In an exemplary embodiment, Figure 2 A network structure diagram of the virtual link provided by the embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the computing cluster and the storage cluster are connected through the network, and the computing nodes and the storage nodes are connected through the network. Figure 2As shown, the computing cluster and the storage cluster can be connected by using a non-blocking switch (Clos Network, CLOS) network. In the scenario of connecting the computing cluster and the storage cluster, the CLOS network can provide an efficient, scalable and low-latency data transmission channel. This network design can ensure that when a large amount of data is transmitted between the computing nodes and the storage nodes, there will be no performance bottleneck or data congestion due to internal resource competition.

[0035] Specifically, Figure 2 In the figure, there are 2 computing clusters and 2 storage clusters. The number of specific computing clusters and storage clusters can be flexibly adjusted according to actual needs, which is not limited here. Figure 2 In each computing cluster in the figure, there are 2 physical machines (Hosts), and each physical machine has 2 virtual machines (VMs), and each virtual machine has 2 containers (Pods). This deployment method and number can be adjusted according to the system or actual needs, which is not limited here. Each storage cluster has 2 storage pools (i.e. storage nodes). Figure 2 Each circle in the figure can represent a node (computing node or storage node). It should be noted that there can be no physical connection between the computing nodes and the storage nodes, and any computing node and any storage node can be connected. The CLOS network can be used to connect the computing cluster and the storage cluster to obtain a virtual link between the computing nodes and the storage nodes (such as Figure 2 The transmission path in the figure). It should be noted that, Figure 2 The example in the figure shows that when the computing node is a virtual machine, the link nodes of the entire virtual link include: physical machine Host, virtual machine VM, cluster link, storage node. These four nodes together constitute the virtual link of the virtual machine.

[0036] In step S102, the node failure probability of each link node in the virtual link is determined by using historical failure data.

[0037] In this disclosure, the historical failure data can be understood as the failure-related information of each node in the virtual link in the past period of time. The above-mentioned past period of time can be flexibly adjusted according to needs. The node failure probability can be understood as the possibility of failure of a node in the link within a certain period of time.

[0038] When determining the node failure probability by using the historical failure data, at least one method such as statistics or machine learning can be used to determine the failure probability. In this way, the failure probability determined by using the historical failure data is more accurate, which can provide a solid foundation for subsequent accurate determination of the failure node.

[0039] In step S103, when the node failure probability of any one link node does not satisfy the hypothesis condition, the hypothesis condition is tested by using the overall distribution probability.

[0040] In the present disclosure, the hypothesis condition can be understood as an expected target or a limit condition set when performing failure analysis to test whether the failure probability satisfies the condition. For example, the hypothesis condition can be that the failure probability of a node should follow a specific probability distribution (such as normal distribution, Poisson distribution, etc.), which is not exhaustively listed here.

[0041] In the present disclosure, the overall distribution probability can be understood as the overall distribution of the failure probability of all nodes in the link, which is a comprehensive description of the failure probability of all nodes in the link. The overall distribution probability can be used to test whether the preset distribution corresponding to the hypothesis condition is satisfied, so as to determine the difference between the overall distribution of the virtual link and the preset distribution, and further determine the failure node. The overall distribution probability of the present disclosure includes but is not limited to at least one of the following: normal distribution, Poisson distribution, binomial distribution, exponential distribution, uniform distribution.

[0042] In the present disclosure, when the node probability of any one link node does not satisfy the hypothesis condition, the overall distribution probability of the link can be further used to test whether the hypothesis condition is satisfied. In this way, the link state can be comprehensively analyzed, and the failure link node can be more accurately identified, thereby providing a more solid basis for subsequent failure nodes. This method can not only rely on the data of a single node, but also analyze the node in the environment of the entire link, which can improve the accuracy of determining the failure node and thereby reduce the possibility of misjudgment.

[0043] In step S104, when the hypothesis condition passes the test, the link node is determined to be a failure node.

[0044] In the present disclosure, when the overall distribution probability satisfies the preset threshold condition, it can be considered that the hypothesis condition passes the test, and the link node is determined to be a failure node when the overall distribution probability is used to test the hypothesis condition. The preset threshold condition can be flexibly adjusted according to different test methods. The test method can include but is not limited to at least one of the following: chi-square test, parameter test. The chi-square test can be understood as a method for comparing the difference between the actual observation frequency and the expected frequency. In the scenario of determining the failure of a link node, the failure state (such as failure / non-failure) of a node can be compared with the expected distribution under a certain hypothesis condition. The parameter test can be understood as a hypothesis test based on the parameters (such as at least one of mean, variance, etc.) of a known distribution.

[0045] In summary, the present disclosure can simplify the complex interaction between the computing nodes and the storage nodes by constructing a virtual link structure, and can efficiently troubleshoot failures. Meanwhile, the present disclosure can accurately locate the fault node according to historical fault data and overall distribution probability, and can effectively determine the failure cause in the virtual machine death condition, thereby improving the accuracy of failure processing.

[0046] As described above, the virtual link of the computing nodes and the storage nodes can be constructed between the computing cluster and the storage cluster. The following specifically describes the detailed virtual link when the computing node is a virtual machine.

[0047] In the present disclosure, when the computing node is a virtual machine, the link nodes included in the virtual link include: a virtual machine, a physical machine, a cluster link, and a storage node. The cluster link is a link node corresponding to a communication path between the computing cluster and the storage cluster.

[0048] In the present disclosure, when the computing node is a virtual machine, the complete path from the virtual machine to the storage node includes: a virtual machine, a physical machine, a cluster link (net), and a storage node. The four nodes collectively constitute a complete virtual link from the virtual machine to the storage node. On this link, data can be transmitted from the virtual machine, forwarded through the physical machine, transmitted to the storage node through the cluster link, and the data storage or retrieval process is completed. Such a virtual link architecture provides a more efficient and flexible way to determine the fault node.

[0049] In an exemplary embodiment, as shown above, when the computing node is a VM, the complete virtual link can be a VM, a Host, a net, and a Pod. Figure 2

[0050] As described above, the node failure probability of each link node in each virtual link can be determined according to historical obstacle data. In this way, potential fault nodes can be quickly identified, unnecessary troubleshooting steps can be reduced, and false positives can be reduced, thereby making failure processing more accurate. At this time, the method for determining the node failure probability includes:

[0051] Obtaining historical fault data of each link node, the historical fault data including at least one of the following: an abnormal running state, a crash or restart event;

[0052] Determining the link failure probability of the virtual link based on the historical fault data;

[0053] For any one link node in the virtual link, performing regression processing to obtain the node failure probability of the link node.

[0054] ​In the present disclosure, the historical failure data includes, but is not limited to, at least one of the following: running state abnormality, crash or restart event. Specifically, the running state abnormality can be understood as at least one of the following states: performance degradation, response time extension, throughput reduction, etc. The crash event can be understood as the complete stop of the node software or hardware, which needs manual intervention or automatic restart to recover. The restart event can be understood as the node restart caused by software errors, system updates or maintenance, etc.

[0055] In the present disclosure, the link failure probability of a virtual link can be understood as an index for measuring the possibility of the link failure within a certain period of time. It can be calculated based on the initial node failure probability of all nodes in the link and the mutual relationship (such as at least one of the following: series, parallel or mixed relationship, etc.) between these nodes. The high or low of the link failure probability directly affects the reliability and stability of the entire network or system.

[0056] In the present disclosure, the regression processing can be understood as a process for studying one or more variables to obtain a variable. In the context of the present disclosure, the regression processing is applied to the failure probability prediction of the link nodes, i.e. all candidate failure nodes of the link are subjected to the regression processing to obtain a failure node with the maximum probability. In the regression analysis, various types of regression models can be used, such as at least one of the following: linear regression, polynomial regression, etc. The preferred regression model of the present disclosure can be the Least Absolute Shrinkage and Selection Operator (LASSO) model. It should be noted that the specific selection of the regression model can also be flexibly adjusted according to the nature of the data, the distribution of the dependent variable, etc., which is not limited here.

[0057] In the present disclosure, the historical failure data of each link node can be obtained, and the data acquisition method includes, but is not limited to, at least one of the following: log analysis, monitoring system, manual report. Then, the initial node failure probability of the virtual link is determined based on the historical failure data. This probability can reflect the possibility of node failure in the future period of time. Further, the link failure probability of the virtual link is determined according to the initial node failure probability. This can be derived by considering the mutual relationship (such as at least one of the following: series, parallel or mixed relationship, etc.) of the nodes of the link and the initial failure probability of the nodes. Then, based on the link failure probability, the regression processing is performed for any one node of the virtual link to obtain the node failure probability. Through the regression processing, a more accurate and specific node failure probability prediction can be obtained.

[0058] In an exemplary embodiment, the following is a detailed calculation process for obtaining the node failure probability:

[0059] Firstly, assume that any computing cluster to storage pool connection link is pathi, and the total number of containers related to the link is x, wherein the number of failed containers is y, then the initial node failure rate of the calculation satisfies the following formula one:

[0060]

[0061] Wherein, P represents the initial node failure probability, i represents the serial number of the failed node, and j represents the serial number of the link.

[0062] For convenience of operation, the above formula one can be taken logarithm on both sides to obtain the following formula two:

[0063] log(1-y i / x i )=∑ j∈pathi logp j

[0064] Considering the entire measurement error or occasional situation, an error variable δ can be introduced, and t i = log(1-y i / x i ), p j = exp(β j ), then the left side of the above formula two is equal to t i , and the right side of the above formula two is equal to ∑ j∈pathi logexp(β j ) = ∑ j∈pathi β j ,

[0065] Because the link weighting coefficient W ij (indicating whether the failed container is on the link) and the error variable δ are added, the above formula two can be further transformed into the link failure probability, satisfying the following formula three:

[0066] t i = ∑ j∈pathi β j w ij + δ j

[0067] Wherein w represents the link weighting coefficient, and takes the value of 0 or 1, i.e. the failed node is on the link or the failed node is not on the link.

[0068] Based on the link failure probability formula three, further assume that there are 2 physical machines in the computing cluster 1, each physical machine has 2 virtual machines, each virtual machine has 2 containers, and the storage cluster 1 has 2 storage pools, then the link failure probability from the computing cluster 1 to the storage cluster 1 satisfies the following formula:

[0069] t1 = β V1 + βH1 + β net + β P1 + δ1

[0070] t2= β V1 + β H1 + β net + β P2 + δ2

[0071] t3= β V2 + β H1 + β net + β P1 + δ3

[0072] t4= β V2 + β H1 + β net + β P2 + δ4

[0073] t5= β V1 + β H2 + β net + β P1 + δ5

[0074] t6= β V1 + β H2 + β net + β P2 + δ6

[0075] t7= β V2 + β H2 + β net + β P1 + δ7

[0076] t8= β V2 + β H2 + β net + β P2 + δ8

[0077] Wherein, V1, V2 represent two virtual machines (VM) of the computer cluster, H1, H2 represent two physical machines (Host) of the computing cluster, net represents the cluster link of the virtual link, P represents the storage pool of the storage cluster, and δ represents an error variable.

[0078] The above formula can be understood as when the computing cluster has 2 physical machines, 2 virtual machines and 2 containers, and the storage cluster has 2 storage pools, there are a total of 8 virtual links from the computer cluster to the storage cluster, and the link failure probability of each virtual link satisfies the above t1-t8.

[0079] Finally, based on the above link failure probability, the node failure probability is obtained by regression processing. The regression processing satisfies the following formula:

[0080] According to its sparse matrix characteristics, LASSO regression is used to solve:

[0081]

[0082] where t represents the link failure probability described above. w represents the matrix of all variables, that is, the matrix of t1-t8 described above. β represents the final calculated node failure probability. θ is a weighting coefficient for controlling the model complexity of the regression model. ||β||1 represents the 1-norm of β, that is, the sum of the absolute values of all coefficients.

[0083] The feature of LASSO regression is that some unimportant β can be made to be 0 by adjusting θ through feature selection, and the target node failure probability is finally obtained.

[0084] That is, the link node failure probability t calculated above is subjected to regression processing, and the unimportant β is made to be 0 to obtain the target node failure probability.

[0085] As described above, after obtaining the node failure probability, hypothesis testing can be performed, which can further evaluate the reliability of the failure probability and be used for accurate positioning of the failure node. It should be noted that the hypothesis condition of the present disclosure can be that the node failure probability of the virtual machine link node is equal to the average failure probability, and the average failure probability is the average value of the node failure probabilities of the link nodes in the virtual link. In the present disclosure, the failure node of multiple nodes of the virtual link can be detected by comparing multiple samples (that is, the failure probabilities of multiple nodes) with a central tendency (that is, the average failure probability). The hypothesis condition set in this way can consider the overall environment and effectively evaluate the probability of node failure.

[0086] As described above, when the node failure probability of any one link node does not meet the hypothesis condition, the overall distribution probability can be used to test the hypothesis condition. The overall distribution probability of the present disclosure is used to represent the probability that the overall sample data of the link node meets the preset distribution. In this way, not only the failure probability of a single node is considered, but also the overall distribution of all node failure probabilities. This helps to evaluate the link node failure probability from a more macro perspective. At the same time, by considering the overall distribution, the influence of individual outliers or noise data on the hypothesis test result can be reduced. At this time, the method for testing the hypothesis condition by using the overall distribution probability includes:

[0087] Obtain historical sample data of the link node;

[0088] Based on the node failure probability and the historical sample data, obtain the overall distribution probability of the link node;

[0089] Test whether the overall distribution probability meets the preset distribution corresponding to the hypothesis condition.

[0090] In the present disclosure, the historical sample data can be understood as at least one of the following data of the link node in the past period of time: running state, fault condition, but not limited to. The sample data can be obtained in at least one of the following ways: network management system, monitoring tool, log record.

[0091] In the present disclosure, the node failure probability can be normalized according to the historical sample data to obtain the overall distribution probability of the link node. The purpose of normalization can be understood as converting the failure probability values under different time periods or conditions to a unified dimension for comparison and analysis. This can be achieved by calculating the mean and standard deviation of the data, and then subtracting the mean and dividing by the standard deviation.

[0092] In the present disclosure, the overall distribution probability can be used to test whether the above-mentioned assumption condition (i.e. the preset distribution) is met. This can be achieved by using at least one of the hypothesis tests (such as chi-square test, parameter test, etc.). In this way, the overall distribution probability can be compared with the preset distribution corresponding to the assumption condition to determine whether there is a significant difference between the overall distribution probability and the preset distribution, and further determine the accurate failure node.

[0093] In an exemplary embodiment, the specific process of obtaining the overall distribution probability of the link node is as follows:

[0094] The node failure probability is normalized by zero mean (Z-score normalization) using the historical sample data to obtain the overall distribution probability of the link. The overall distribution probability satisfies the following formula:

[0095]

[0096] Wherein represents the variance, N represents the number of β historical data, represents the historical data sample, represents the mean of the sample data. The overall distribution probability of the link can be obtained by using the above formula.

[0097] As described above, the overall distribution probability can be used to test whether the preset distribution corresponding to the assumption condition is met. At this time, the testing method includes:

[0098] When the overall distribution probability is greater than the preset first threshold, it is determined that the assumption condition fails to pass the test, and it is determined that the link node is normal;

[0099] Or,

[0100] When the overall distribution probability is less than or equal to the preset first threshold, it is determined that the assumption condition passes the test, and it is determined that the link node is abnormal;

[0101] The first threshold is the probability threshold that the overall distribution is a preset distribution.

[0102] In this disclosure, the preset first threshold can be understood as a pre-set significance level or probability threshold that conforms to the overall distribution probability. The significance level can be understood as the upper limit of the probability of accepting an error (i.e., rejecting the true hypothesis). The probability threshold can be understood as the probability that the overall distribution is a preset distribution. When using the overall distribution probability to test the hypothesis, the preset first threshold can be used for comparison and determination. When the overall distribution probability is greater than the preset first threshold, it can be proven that the significance level of the overall distribution probability is not met, further confirming that the hypothesis has failed the test, and the link node is considered a normal node. When the overall distribution probability is less than or equal to the preset first threshold, it can be proven that the significance level of the overall distribution probability is met, further confirming that the hypothesis has passed the test, and the link node is considered abnormal, a faulty node.

[0103] In one exemplary embodiment, the preset first threshold can be understood as the significance level α of the normal distribution, and can be multiple values ​​such as 0.01 and 0.05. When the overall distribution probability z obtained from the above description... j If the value is greater than α, it proves that the overall probability distribution of the link does not meet the significance level, the hypothesis condition has not passed the test, and the link node is considered a normal node. That is, if Then it is assumed that a component of the current link is functioning normally. When the overall probability distribution z obtained above... j If the probability distribution is less than or equal to α, it proves that the overall probability distribution of the link meets the significance level. The hypothesis condition passes the test, and the link node is considered an anomalous node. This indicates that a component in the current link is faulty, for example: Then it is assumed that virtual machine 2 is faulty.

[0104] As mentioned above, the methods described can be used to determine virtual machine failures. This disclosure also provides a method for determining the cause of virtual machine failures, which is detailed below:

[0105] Obtain the metric data of the link nodes, including at least one of the following: CPU utilization, memory, and disk I / O;

[0106] Based on metric data and log data, the cause of the failure was determined.

[0107] In the present disclosure, when a virtual fault is located, the index data and log data of the link node can be utilized to determine the fault cause of the virtual machine. The index data includes, but is not limited to, at least one of the following: central processing unit (CPU) utilization, memory, and disk input / output (I / O). In this way, by monitoring and analyzing the index database such as CPU utilization, memory usage, and disk I / O, the performance bottleneck or abnormal behavior of the virtual machine can be quickly identified. This helps to quickly narrow down the scope of troubleshooting and reduce the time required to locate the problem. At the same time, the index data and log data provide rich context information, making the determination of the fault cause more accurate. In this way, combined with the comprehensive analysis of multiple data sources, false positives and omissions can be reduced, and the reliability of fault diagnosis can be improved.

[0108] The present disclosure also provides a detailed method for determining the fault cause based on index data and log data, comprising:

[0109] For any virtual machine, when at least one exception in the index data and a preset keyword in the log data exist, the fault cause of the virtual machine is determined to be memory leakage.

[0110] In the present disclosure, the index data exception includes, but is not limited to, at least one of the following: CPU usage is less than a threshold value, memory usage is greater than a threshold value, disk read / write does not respond, and virtual machine does not respond. Specifically, CPU usage less than a threshold value can be understood as CPU usage less than a threshold value for T minutes (not necessarily 5 minutes, 10 minutes, etc.), which may indicate that the virtual machine is dead. However, it should be noted that sometimes the virtual machine may only be in an idle state. Memory usage greater than a threshold value can be understood as memory usage greater than a threshold value for T minutes (not necessarily 5 minutes, 10 minutes, etc.), and there is no obvious release indication, which may indicate that the virtual machine is dead, but there may also be a large memory-consuming process that lasts for a long time, resulting in no available memory allocation for business. Disk read / write does not respond can be understood as disk read / write operation time not responding for T minutes (not necessarily 5 minutes, 10 minutes, etc.), which may indicate that the virtual machine is dead. Virtual machine does not respond can be understood as using a heartbeat detection mechanism to periodically send requests to the virtual machine, and if the virtual machine does not respond for T minutes (not necessarily 5 minutes, 10 minutes, etc.), it may be that the virtual machine and the physical machine to which it belongs are in a non-continuous connection state.

[0111] In the present disclosure, the log data can be understood as a data collection of various events, errors, warnings, informational messages, etc. in the process of business operation. The log data of the present disclosure can have various forms, such as at least one of a log ( / var / log / message) for recording system messages, a log ( / var / log / secure) for recording security-related messages, and a business container log, without limitation herein. The preset keyword of the log data can be: cannot allocate memory.

[0112] In the present disclosure, when any one of the above index data of any one virtual machine is abnormal, and the preset keyword is determined to exist in the log data, it can be determined that the failure cause of the virtual machine is memory leak. For example, when more than one of the monitoring indexes CPU, memory, and disk I / O is abnormal, and the system log is combined to search for "cannot allocate memory", it is determined that the virtual machine has a memory leak phenomenon.

[0113] In the present disclosure, the timing task can also be used to detect the index data and the log data in a timing manner to determine the failure cause. In this way, the timing detection can discover potential problems or performance bottlenecks in time, so as to perform preventive maintenance before the failure occurs. At the same time, the timing detection can cover different time periods and load conditions of the system, so as to more comprehensively understand the running state of the system and accurately determine the failure node.

[0114] In an exemplary embodiment, the master node of the container orchestration platform (Kubernetes, k8s) cluster sets a timing task (for example, every 1 min, 5 min, etc.) to automatically detect the above index data and log data and determine the judgment result.

[0115] As described above, when the index data and the log data are combined to determine the failure cause, scheduling adjustment can also be performed. The method of scheduling adjustment is specifically described as follows:

[0116] When the failure cause is determined, the scheduling possibility of the container in the virtual machine is limited.

[0117] The limiting of the scheduling possibility of the container in the virtual machine includes marking the virtual machine where the container is located as non-schedulable.

[0118] In the present disclosure, when the fault cause is determined to be content leakage, the scheduling possibility of the container in the virtual machine can be limited, i.e., the virtual machine where the container is located is marked as non-schedulable. In a popular sense, it can be understood that the virtual machine of the fault node is marked as non-schedulable, so that the new service request or task will be automatically (or manually by the system administrator) rescheduled to other healthy virtual machines (i.e., "other nodes") for processing. The purpose of this measure is to facilitate the self-recovery of the system and minimize the impact of content leakage on the business.

[0119] In an exemplary embodiment, when the k8s cluster determines that the above-mentioned fault cause is memory leakage, i.e., the above-mentioned timing task detects that the judgment result is true, i.e., there is a problem with the node, it will execute the following command. The function of this command is to remove the specified node ( <node-name>) marked as Cordoned. Once a node is marked as Cordoned, the k8s Scheduler stops scheduling new Pods to this node and instead allocates them to other available nodes in the cluster.

[0120] The present disclosure also provides a fault processing apparatus. Figure 3 A structural block diagram of a fault processing apparatus provided for an embodiment of the present disclosure is shown in Figure 3 The fault processing apparatus 300 includes:

[0121] An acquisition unit 301 is configured to acquire a virtual link between a computing node and a storage node between a computing cluster and a storage cluster to be checked.

[0122] A first determination unit 302 is configured to determine a node failure probability of each link node in the virtual link by using historical failure data.

[0123] An inspection unit 303 is configured to inspect the hypothesis condition by using an overall distribution probability when the node failure probability of any one link node does not satisfy the hypothesis condition.

[0124] A second determination unit 304 is configured to determine the link node as a failure node when the hypothesis condition passes the inspection.

[0125] In an exemplary embodiment, the computing node includes at least one of a physical machine, a virtual machine, and a container; when the computing node is a virtual machine, the link nodes included in the virtual link include a virtual machine, a physical machine, a cluster link, and a storage node; the cluster link is a link node corresponding to a communication path between the computing cluster and the storage cluster.

[0126] In an exemplary embodiment, the determination of the node failure probability of each link node in the virtual link by using the historical failure data includes: acquiring historical failure data of each link node, the historical failure data including at least one of a running state exception, a crash or a restart event; determining a link failure probability of the virtual link based on the historical failure data; and performing regression processing on any one link node in the virtual link to obtain the node failure probability of the link node.

[0127] In an exemplary embodiment, the hypothesis condition includes that the node failure probability of the virtual machine link node is equal to an average failure probability, and the average failure probability is an average value of the node failure probabilities of each link node in the virtual link.

[0128] In an example embodiment, the overall distribution probability is used to represent a probability that overall sample data of the link node satisfies a preset distribution; and the overall distribution probability is used to test the hypothesis condition, including: obtaining historical sample data of the link node; obtaining the overall distribution probability of the link node based on the node failure probability and the historical sample data; and testing whether the overall distribution probability satisfies a preset distribution corresponding to the hypothesis condition.

[0129] In an example embodiment, testing whether the overall distribution probability satisfies the preset distribution corresponding to the hypothesis condition includes: when the overall distribution probability is greater than a preset first threshold value, determining that the hypothesis condition fails to pass the test, and determining that the link node is normal; or when the overall distribution probability is less than or equal to the preset first threshold value, determining that the hypothesis condition passes the test, and determining that the link node is abnormal; wherein the first threshold value is a probability threshold value of the overall distribution being the preset distribution.

[0130] In an example embodiment, the method further includes: obtaining index data of the link node, the index data including at least one of: central processing unit (CPU) utilization, memory, and disk input / output (I / O); and determining the failure cause based on the index data and the log data.

[0131] In an example embodiment, determining the failure cause based on the index data and the log data includes: for any one virtual machine, when at least one of the index data is abnormal and the preset keyword exists in the log data, determining that a failure cause of the virtual machine is memory leakage.

[0132] In an example embodiment, determining the failure cause based on the index data and the log data further includes: using a timing task to perform timing detection on the index data and the log data to determine the failure cause.

[0133] In an example embodiment, the method further includes: when the failure cause is determined, limiting scheduling possibility of a container in the virtual machine; wherein limiting the scheduling possibility of the container in the virtual machine includes: marking a virtual machine where the container is located as non-schedulable.

[0134] Figure 4 A hardware block diagram of an electronic device is provided for the embodiments of the present disclosure. The electronic device 400 according to the embodiments of the present disclosure at least includes a processor; and a memory configured to store computer readable instructions. When the computer readable instructions are loaded and run by the processor, the processor performs the failure processing method described in any one of the preceding embodiments of the present disclosure.

[0135] Figure 4 The illustrated electronic device 400 specifically includes a central processing unit (CPU) 401, a graphics processing unit (GPU) 402, and a memory 403. These units are interconnected via a bus 404. The CPU 401 and / or GPU 402 can function as the aforementioned processor, and the memory 403 can function as the aforementioned memory storing computer-readable instructions. Furthermore, the electronic device 400 may also include a communication unit 405, a storage unit 406, an output unit 407, an input unit 408, and an external device 409, all of which are also connected to the bus 404.

[0136] Figure 5 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this disclosure. (As shown...) Figure 5 As shown, a computer-readable storage medium 500 according to an embodiment of the present disclosure stores computer-readable instructions 501 thereon. When the computer-readable instructions 501 are executed by a processor, the fault handling method described with reference to the above figures according to any embodiment of the present disclosure is performed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0137] This disclosure further provides a computer program product, including a computer program that, when executed by a processor, implements the fault handling method described in any of the preceding embodiments of this disclosure.

[0138] In summary, the present disclosure can simplify the complex interaction relationship between the computing nodes and the storage nodes by constructing the virtual link structure, that is, by constructing the virtual link structure, the complex interaction or nested relationship can be simplified as a series of connections between the link nodes, and when troubleshooting, the link nodes of each virtual link can be directly targeted for efficient troubleshooting of the fault. At the same time, the historical fault data of the link nodes and the overall distribution probability can be combined to directly point to the most likely fault node in the virtual link, and the fault node can also be accurately located according to the historical fault data and the overall distribution probability, thereby achieving accurate positioning of the fault node. The fault reason in the virtual machine dead condition can also be effectively determined, and the accuracy of fault processing is improved.

[0139] The basic principles of the present disclosure are described above in combination with specific embodiments, but it should be noted that the advantages, advantages, effects and the like mentioned in the present disclosure are only examples and are not limiting, and these advantages, advantages, effects and the like cannot be considered as the various embodiments of the present disclosure must have. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present disclosure to the above specific details.

[0140] The block diagrams of the devices, apparatuses, equipment, systems involved in the present disclosure are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, mean "include but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0141] Also, as used in the description herein and throughout the claims that follow, the meaning of "or" includes "and / or" unless the context clearly dictates otherwise. For example, when used in the examination of "at least one of A or / and B" the phrase means A or B or both A and B.

[0142] It is also important to note that the systems and methods of the disclosure can be embodied in a variety of forms depending on the particular requirements of the applications. Some of these forms are described in more detail herein. However, it is not intended that the applications be limited to these forms but rather that the application be construed as including all possible forms.

[0143] Various changes, modifications, and improvements in the technologies described herein can be made without departing from the teachings of the technology defined by the appended claims. Moreover, the scope of the claims of the disclosure is not limited to the specific aspects described herein. Rather, the claims encompass all possible aspects falling within the scope of the claims. Accordingly, the disclosure is not to be restricted except in light of the attached claims and their equivalents.

[0144] The above description of the disclosed aspects is given for illustrative purposes only and is not intended to limit the scope of the disclosure. Although a few aspects have been described in detail above, various modifications, alterations, and changes to the described aspects can be made without departing from the scope of the disclosure. Accordingly, the disclosure is not limited to the described aspects, but intends to include all changes, equivalents, and substitutes that fall within the scope of the claims.

[0145] The above description has been presented for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of skill in the art will recognize that certain variations, modifications, alterations, additions, and substitutions are possible, given the benefit of this disclosure.

Claims

1. A failure handling method characterized by, The method comprises: between the computing cluster and the storage cluster to be checked, obtaining a virtual link between a computing node and a storage node; using historical fault data, determining a node failure probability of each link node in the virtual link; when the node failure probability of any one link node does not satisfy a hypothesis condition, using an overall distribution probability to test the hypothesis condition; when the hypothesis condition passes the test, determining the link node as a failure node; wherein the overall distribution probability is used to represent the probability that the overall sample data of the link node satisfies a preset distribution; the use of the overall distribution probability to test the hypothesis condition comprises: obtaining historical sample data of the link node; the historical sample data is used to represent the running state of the link node in the past period of time; based on the node failure probability and the historical sample data, obtaining the overall distribution probability of the link node; testing whether the overall distribution probability satisfies a preset distribution corresponding to the hypothesis condition; the testing whether the overall distribution probability satisfies the preset distribution corresponding to the hypothesis condition comprises: when the overall distribution probability is greater than a preset first threshold, determining that the hypothesis condition does not pass the test, and determining that the link node is normal; or, when the overall distribution probability is less than or equal to the preset first threshold, determining that the hypothesis condition passes the test, and determining that the link node is abnormal; wherein the first threshold is a probability threshold value of the overall distribution being the preset distribution.

2. The method of claim 1, wherein, The computing node comprises at least one of the following: a physical machine, a virtual machine, a container; when the computing node is a virtual machine, the link nodes included in the virtual link include: a virtual machine, a physical machine, a cluster link, and a storage node; wherein the cluster link is a link node corresponding to a communication path between the computing cluster and the storage cluster.

3. The method of claim 1, wherein, The use of historical fault data to determine the node failure probability of each link node in the virtual link comprises: obtaining historical fault data of each link node, the historical fault data comprising at least one of the following: running state abnormality, crash or restart event; based on the historical fault data, determining a link failure probability of the virtual link; for any one link node in the virtual link, performing regression processing to obtain the node failure probability of the link node.

4. The method of claim 1, wherein, The hypothesis condition comprises: the node failure probability of the virtual machine link node is equal to the average failure probability, and the average failure probability is the average of the node failure probabilities of each link node in the virtual link.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: obtaining index data of the link node, the index data comprising at least one of the following: central processing unit (CPU) utilization, memory, and disk input / output (I / O); based on the index data and log data, determining a failure cause.

6. The method of claim 5, wherein, The determination of the failure cause based on the index data and log data comprises: for any one virtual machine, when at least one of the index data is abnormal and there is a preset keyword in the log data, determining that the failure cause of the virtual machine is memory leakage.

7. The method of claim 5, wherein, The determining the fault cause based on the index data and the log data further includes: The index data and the log data are periodically detected by a periodic task to determine the fault cause.

8. The method of claim 5, wherein, The method further includes: When the fault cause is determined, scheduling possibility of a container in a virtual machine is limited. The limiting the scheduling possibility of the container in the virtual machine includes marking the virtual machine where the container is located as non-schedulable.

9. A failure handling apparatus characterized by comprising: The apparatus includes: An acquisition unit configured to acquire a virtual link between a computing node and a storage node between a computing cluster and a storage cluster to be investigated; A first determination unit configured to determine a node fault probability of each link node in the virtual link based on historical fault data; An inspection unit configured to inspect a hypothesis condition based on an overall distribution probability when the node fault probability of any one link node does not satisfy the hypothesis condition; A second determination unit configured to determine the link node as a fault node when the hypothesis condition passes the inspection. The overall distribution probability represents a probability that overall sample data of the link node satisfies a preset distribution; the inspection of the hypothesis condition based on the overall distribution probability includes: acquiring historical sample data of the link node; the historical sample data represents a running state of the link node in the past period of time; based on the node fault probability and the historical sample data, an overall distribution probability of the link node is acquired; it is inspected whether the overall distribution probability satisfies a preset distribution corresponding to the hypothesis condition. The inspection of whether the overall distribution probability satisfies the preset distribution corresponding to the hypothesis condition includes: when the overall distribution probability is greater than a preset first threshold, it is determined that the hypothesis condition does not pass the inspection, and it is determined that the link node is normal; or when the overall distribution probability is less than or equal to the preset first threshold, it is determined that the hypothesis condition passes the inspection, and it is determined that the link node is abnormal; the first threshold is a probability threshold of the overall distribution being the preset distribution.

10. An electronic device, comprising: The apparatus includes: A memory configured to store computer readable instructions; and A processor configured to run the computer readable instructions, so that the electronic device performs the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer-readable instructions, the computer-readable instructions comprising: When the computer readable instructions are executed by the processor, the processor performs the method of any one of claims 1-8.

12. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1-8.

Images