Fault detection method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a fault detection method and device, computer equipment, a storage medium and a computer program product, and the method comprises the steps: receiving interaction state information which is transmitted by a plurality of target interaction nodes in an information interaction cluster and carries out information interaction with a main node in a management cluster; and then performing fault detection on the main node in the management cluster according to the interaction state information sent by each target interaction node to obtain a detection result. Wherein the target interaction node comprises a proxy service node or an underlying service node. In the method, the external service of the management cluster is realized through interaction between the proxy service node and the underlying service node or direct interaction between the proxy service node and the underlying service node, so that whether the management cluster can normally serve the external service or not can be accurately identified by detecting the interaction state between the proxy service node or the underlying service node and the main node; and whether the main node has a false death fault or not can be accurately and quickly detected according to the external service state of the management cluster.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault detection method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] The distributed storage system consists of multiple servers forming a cluster. Each server is a node. The cluster selects a master node through election rules, and the remaining nodes are slave nodes. When the master node fails, a slave node will be re-elected as the new master node. If the master node fails but the slave node is not re-elected in time, the entire cluster will not work properly, resulting in serious consequences. Therefore, it is very necessary to accurately detect the failure of the master node.

[0003] At present, when fault detection is performed on the master node, the slave nodes in the cluster usually send heartbeat messages to the master node regularly to detect whether the master node can be pinged, and then determine whether the master node has a fault. However, the above fault detection method is difficult to accurately detect the pseudo-death fault of the master node. Summary of the invention

[0004] Based on this, it is necessary to provide a fault detection method, device, computer equipment, storage medium and computer program product that can accurately detect the pseudo-death fault of the main node in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a fault detection method, the method comprising:

[0006] Receiving interaction status information for information interaction with a management cluster sent by multiple target interaction nodes in an information interaction cluster; the target interaction nodes include proxy service nodes or bottom service nodes;

[0007] According to the interaction status information sent by each target interaction node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0008] The fault detection method provided in the embodiment of the present application receives the interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with the main node in the management cluster, and then performs fault detection on the main node in the management cluster according to the interaction status information sent by each target interaction node to obtain a detection result. Among them, the target interaction node includes a proxy service node or an underlying service node. In the above method, since the external service of the management cluster is realized by the interaction between the proxy service node and the underlying service node or directly interacting with the underlying service, by detecting the interaction status between the proxy service node or the underlying service node and the main node, it is possible to accurately identify whether the management cluster can provide normal external services, and then according to the status of the external service of the management cluster, it is possible to accurately and quickly detect whether the main node has a pseudo-death fault.

[0009] In one embodiment, fault detection is performed on a master node in a management cluster according to the interaction status information sent by each target interaction node to obtain a detection result, including:

[0010] Determine the interaction failure rate between each target interaction node and the master node according to the interaction status information sent by each target interaction node;

[0011] According to the interaction failure rate between each target interaction node and the master node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0012] In one embodiment, each target interaction node is a proxy service node or an underlying service node. According to the interaction failure rate between each target interaction node and the master node, a fault detection is performed on the master node in the management cluster to obtain a detection result, including:

[0013] Determine, according to the interaction failure rate between each target interaction node and the master node, a first number of target interaction nodes whose interaction failure rate exceeds a first failure rate threshold;

[0014] Fault detection is performed on the master nodes in the management cluster according to the first number to obtain a detection result.

[0015] In one embodiment, the multiple target interaction nodes include proxy service nodes and underlying service nodes. According to the interaction failure rate between each target interaction node and the master node, fault detection is performed on the master node in the management cluster to obtain detection results, including:

[0016] Determining, according to the interaction failure rate between each proxy service node and the master node, a second number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold;

[0017] Determining, according to the interaction failure rate between each underlying service node and the master node, a third number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold;

[0018] According to the second number and the third number, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0019] The method described in the embodiment of the present application, since the interaction status information between the target interaction node and the main node can accurately reflect the external service status of the main node, the external service status of the main node can be detected by detecting the number of interaction failure rates between the target interaction node and the main node reaching the threshold. The method is simple and convenient, and can accurately and efficiently detect the main node. Moreover, the above method has different detection methods corresponding to the node types contained in different target interaction nodes, so the method can perform detection considering different scenario factors, thereby improving the detection accuracy.

[0020] In one of the embodiments, fault detection is performed on the master node in the management cluster according to the second number and the third number to obtain a detection result, including:

[0021] According to the fault type corresponding to the fault detection, determine the degree of influence of the proxy service node and the underlying service node on the fault type;

[0022] According to the degree of influence of the proxy service node and the underlying service node on the fault type, corresponding weights are assigned to the proxy service node and the underlying service node;

[0023] According to the weight and the second number of the proxy service node and the weight and the third number of the underlying service node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0024] The method described in the embodiment of the present application combines the influence degree of different service nodes when performing fault detection, and can accurately reflect the role of each node in the fault, thereby improving the accuracy of fault detection.

[0025] In one embodiment, receiving interaction status information sent by multiple target interaction nodes in an information interaction cluster for information interaction with a master node in a management cluster includes:

[0026] When receiving an alarm message sent by any target interaction node in the information interaction cluster, an information acquisition request is sent to other target interaction nodes in the management cluster; the alarm message carries the interaction status information of any target interaction node;

[0027] Receive interaction status information returned by other target interaction nodes.

[0028] The method described in the embodiment of the present application, when an alarm message is received, performs fault detection on the main node based on the interaction information of all target interaction nodes. Compared with the method of always performing fault detection on the main node based on the interaction information of all target interaction nodes, the above method can reduce the waste of resources to a certain extent.

[0029] In one embodiment, the method further comprises:

[0030] When the detection result indicates that the master node is faulty, the detection result is sent to each slave node in the management cluster; the detection result is used to instruct each slave node to interact with each other to determine a new master node and isolate the master node.

[0031] The method described in the embodiment of the present application can ensure that when a master node fails, the cluster can quickly recover and continue to provide services, thereby ensuring the stability of the storage system by reselecting the master node from the slave nodes.

[0032] In a second aspect, the present application further provides a fault detection device, the device comprising:

[0033] A receiving module, used to receive interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with the master node in the management cluster; the target interaction node includes a proxy service node or an underlying service node;

[0034] The detection module is used to perform fault detection on the master node in the management cluster according to the interaction status information sent by each target interaction node to obtain a detection result.

[0035] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0036] Receiving interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with a master node in the management cluster; the target interaction nodes include proxy service nodes or bottom service nodes;

[0037] According to the interaction status information sent by each target interaction node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0039] Receiving interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with a master node in the management cluster; the target interaction nodes include proxy service nodes or bottom service nodes;

[0040] According to the interaction status information sent by each target interaction node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0041] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0042] Receiving interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with a master node in the management cluster; the target interaction nodes include proxy service nodes or bottom service nodes;

[0043] According to the interaction status information sent by each target interaction node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0044] The above-mentioned fault detection method, device, computer equipment, storage medium and computer program product, the method receives the interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with the main node in the management cluster, and then performs fault detection on the main node in the management cluster according to the interaction status information sent by each target interaction node to obtain the detection result. Among them, the target interaction node includes a proxy service node or an underlying service node. In the above-mentioned method, since the external service of the management cluster is realized by the interaction between the proxy service node and the underlying service node or directly interacting with the underlying service, by detecting the interaction status between the proxy service node or the underlying service node and the main node, it is possible to accurately identify whether the management cluster can provide normal external services, and then according to the status of the external service of the management cluster, it is possible to accurately and quickly detect whether the main node has a pseudo-death fault. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 An application environment diagram of a fault detection method in an embodiment;

[0046] Figure 2 is a flow chart of a fault detection method in one embodiment;

[0047] Figure 3 is a flow chart of a fault detection method in another embodiment;

[0048] Figure 4 is a flow chart of a fault detection method in another embodiment;

[0049] Figure 5A schematic diagram of information interaction of a management cluster in one embodiment;

[0050] Figure 6 is a flow chart of a fault detection method in another embodiment;

[0051] Figure 7 is a flow chart of a fault detection method in another embodiment;

[0052] Figure 8 is a flow chart of a fault detection method in another embodiment;

[0053] Fig. 9 is a flow chart of a fault detection method in another embodiment;

[0054] Fig.10 is a flow chart of a fault detection method in another embodiment;

[0055] Fig.11 is a structural block diagram of a fault detection device in one embodiment;

[0056] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] A distributed storage system is composed of a cluster of multiple servers, each server being a node. The cluster selects a master node through election rules, and the remaining nodes are slave nodes. When a master node fails, a slave node will be re-elected as the new master node. If the master node fails but the slave node is not re-elected in time, the entire cluster will not work properly, resulting in serious consequences. Therefore, it is very necessary to accurately detect the failure of the master node. At present, when performing fault detection on the master node, it is usually performed by the slave nodes in the cluster to periodically send heartbeat messages to the master node to detect whether the master node can be pinged, and then determine whether the master node fails. However, the above fault detection method is difficult to accurately detect the pseudo-death fault of the master node. The present application provides a fault detection method, which aims to solve the above technical problems. The following embodiments will specifically illustrate the fault detection method described in the present application.

[0059] The fault detection method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the application environment includes a management cluster 101, an information interaction cluster 102 and a monitoring node 103. The monitoring node 103 is connected to the management cluster 101 and the information interaction cluster 102 respectively, and the management cluster 101 and the information interaction cluster 102 are connected to each other. Among them, the management cluster 101 is used to provide services to the outside; the management cluster 101 includes multiple management nodes, specifically including a master node and multiple slave nodes, and the management cluster 101 can be a server cluster composed of multiple servers. The information interaction cluster 102 is used to interact with the management cluster 101; the information interaction cluster 102 includes multiple interaction nodes, each interaction node specifically includes a node that interacts with the management node and a node that does not interact with the management node; the information interaction cluster 102 can specifically be a management message proxy service cluster and / or an underlying service cluster; the information interaction cluster 102 can be a server cluster composed of multiple servers. The monitoring node 103 is used to monitor the interaction status of the management cluster 101 and the information interaction cluster 102; the monitoring node 103 can be implemented with an independent server or a server cluster composed of multiple servers.

[0060] Those skilled in the art will understand that Figure 1 The structure shown in the figure is merely a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the application environment to which the present application scheme is applied. The specific application environment may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0061] In one embodiment, Figure 2 As shown, a fault detection method is provided, which is applied to Figure 1 The monitoring node in the example is used as an example to illustrate the following steps:

[0062] S201, receiving interaction status information sent by multiple target interaction nodes in an information interaction cluster for information interaction with a master node in a management cluster.

[0063] The target interaction node is a node that interacts with the management node. The target interaction node includes a proxy service node or an underlying service node. The interaction status information is the status information of at least one information interaction between the target interaction node and the master node in the management cluster, specifically including the number of successful interactions and the number of failed interactions. The interaction status information can be the status of the message sent by the target interaction node to the master node, or the status of the message received by the target interaction node from the master node, or other information interaction methods, as long as it can reflect the status of the master node's external services.

[0064] In an embodiment of the present application, the monitoring node may first determine multiple target interaction nodes from each interaction node in the information interaction cluster, and then send an information acquisition request to each target interaction node in real time. After each target node receives the information acquisition request sent by the monitoring node, it can send the interaction status information of its own information interaction with the master node in the management cluster to the monitoring node according to the content indicated by the information acquisition request, and the monitoring node can receive the interaction status information. Optionally, the target interaction node can actively upload the interaction status information of its own information interaction with the master node in the management cluster to the monitoring node in real time, and the monitoring node can receive the interaction status information. Optionally, the monitoring node can obtain the interaction status information of all target interaction nodes at one time. Optionally, the monitoring node can obtain the interaction status information of some target interaction nodes at one time, first perform a preliminary detection on the interaction status information of some target interaction nodes, and if there is an abnormality in the preliminary detection, obtain the interaction status information of other target interaction nodes, and then perform a final detection based on the interaction status information of other target interaction nodes. Optionally, some target interaction nodes may first send interaction status information to the monitoring node, and the monitoring node may perform preliminary detection on the interaction status information of some target interaction nodes. If there is an abnormality in the preliminary detection, the interaction status information of other target interaction nodes is obtained, and then the final detection is performed based on the interaction status information of other target interaction nodes. It should be noted that in order to ensure that the monitoring node can accurately monitor, the monitoring node is set to a cluster of a non-master-slave architecture, because if the monitoring node is set to a cluster of a master-slave architecture, there may be a situation where the master node is suspended, resulting in the inability of the monitoring node to accurately monitor, and the embodiment of the present application sets the monitoring node to a cluster of a non-master-slave architecture, which can avoid the problem of suspended master node in the monitoring node. If the monitoring node is a monitoring cluster composed of a server cluster, then in the process of receiving the interaction status information sent by multiple target interaction nodes, the multiple target interaction nodes will send the interaction status information to each server node in the monitoring cluster. For any monitoring node, the information it receives is the interaction status information sent by each target interaction node, and then the master node in the management cluster is fault-detected according to the interaction status information sent by each target interaction node received by each monitoring node, and the detection result is obtained.

[0065] S202: Perform fault detection on the master node in the management cluster according to the interaction status information sent by each target interaction node to obtain a detection result.

[0066] The detection result includes whether the master node in the management cluster fails or not.

[0067] In an embodiment of the present application, after the monitoring node obtains the interaction status information of the information interaction between the multiple target interaction nodes and the master node in the management cluster based on the above steps, it can perform fault detection on the master node in the management cluster according to the number of successful interactions and the number of failed interactions in the interaction status information to obtain a detection result. Optionally, the interaction status information can be input into a pre-trained neural network, and fault identification can be performed through the neural network to perform fault detection on the master node in the management cluster to obtain a detection result. Optionally, the master node can also be fault-detected according to the interaction status information based on a preset fault identification algorithm to obtain a detection result.

[0068] The fault detection method provided in the embodiment of the present application receives the interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with the main node in the management cluster, and then performs fault detection on the main node in the management cluster according to the interaction status information sent by each target interaction node to obtain a detection result. Among them, the target interaction node includes a proxy service node or an underlying service node. In the above method, since the external service of the management cluster is realized by the interaction between the proxy service node and the underlying service node or directly interacting with the underlying service, by detecting the interaction status between the proxy service node or the underlying service node and the main node, it is possible to accurately identify whether the management cluster can provide normal external services, and then according to the status of the external service of the management cluster, it is possible to accurately and quickly detect whether the main node has a pseudo-death fault.

[0069] In one embodiment, a specific implementation method for performing fault detection on the master node in the management cluster according to the interaction status information sent by each target interaction node is also provided, such as Figure 3 As shown, the above step S202 of "performing fault detection on the master node in the management cluster according to the interaction status information sent by each target interaction node to obtain a detection result" includes:

[0070] S301, determining the interaction failure rate between each target interaction node and the master node according to the interaction state information sent by each target interaction node.

[0071] Among them, the interaction failure rate is the ratio of the number of interaction failures to the total number of interactions.

[0072] In an embodiment of the present application, after the monitoring node obtains the interaction status information of multiple target interaction nodes interacting with the main node in the management cluster, for each target interaction node, the monitoring node can extract at least two of the number of interaction failures, the number of interaction successes, and the total number of interactions from the interaction status information of the target interaction node, and then determine the interaction failure rate between the target interaction node and the main node based on at least two of the number of interaction failures, the number of interaction successes, and the total number of interactions, and so on, to obtain the interaction failure rate between each target interaction node and the main node.

[0073] S302, performing fault detection on the master node in the management cluster according to the interaction failure rate between each target interaction node and the master node, and obtaining a detection result.

[0074] In an embodiment of the present application, after the monitoring node obtains the interaction failure rate between each target interaction node and the main node based on the above steps, the interaction failure rate between each target interaction node and the main node can be compared with the failure rate threshold to obtain a comparison result, and then fault detection is performed on the main node in the management cluster based on the comparison result to obtain a detection result.

[0075] Optional, such as Figure 4 As shown, the above-mentioned target interaction nodes are all proxy service nodes or underlying service nodes. On this basis, the above-mentioned step S302 of "performing fault detection on the master node in the management cluster according to the interaction failure rate between each target interaction node and the master node to obtain the detection result" includes:

[0076] S401: Determine, according to the interaction failure rate between each target interaction node and the master node, a first number of target interaction nodes whose interaction failure rate exceeds a first failure rate threshold.

[0077] The first number is the number of target interaction nodes whose interaction failure rate exceeds the first failure rate threshold. Each target interaction node is a proxy service node or an underlying service node, that is, each target interaction node is a proxy service node, or each target interaction node is an underlying service node. The proxy service node is a node in the intermediate proxy service cluster between the management cluster and the external service, and the underlying service node is a node in the underlying service cluster of the external service that interacts with the management cluster. Figure 5 As shown, the management cluster can interact with the underlying service nodes directly or through proxy service nodes.

[0078] In an embodiment of the present application, after the monitoring node obtains the interaction failure rate between each target interaction node and the main node, for each target interaction node, it first determines whether the interaction failure rate of the target interaction node exceeds a first failure rate threshold, and then determines that the number of target interaction nodes whose interaction failure rate exceeds the first failure rate threshold among all target interaction nodes is the first number.

[0079] S402: Perform fault detection on the master nodes in the management cluster according to the first number to obtain a detection result.

[0080] In an embodiment of the present application, after obtaining the first number of target interaction nodes whose interaction failure rate exceeds the first failure rate threshold, the monitoring node can further determine whether the first number exceeds the first number threshold. If it exceeds, it indicates that the master node is faulty; if it does not exceed, it indicates that the master node is not faulty. It should be noted that if the monitoring node is a monitoring cluster composed of a server cluster, each server node in the monitoring cluster can first determine the preliminary detection result based on whether the first number of target interaction nodes whose interaction failure rate exceeds the first failure rate threshold reaches the first number threshold, obtain the preliminary detection result corresponding to each server node, and then determine whether the number of server nodes whose preliminary detection results indicate a preliminary failure of the master node exceeds a preset threshold. If it exceeds, it indicates that the master node is faulty; if it does not exceed, it indicates that the master node is not faulty.

[0081] Optionally, the above target interaction nodes include proxy service nodes and bottom service nodes. On this basis, another specific implementation method for performing fault detection on the main node according to the interaction failure rate between each target interaction node and the main node is provided, such as Figure 6 As shown, the above step S302 of "performing fault detection on the master node in the management cluster according to the interaction failure rate between each target interaction node and the master node to obtain a detection result" includes:

[0082] S501: Determine a second number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold according to the interaction failure rate between each proxy service node and the master node. The second failure rate threshold may be the same as or different from the first failure rate threshold.

[0083] The second number is the number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold.

[0084] In an embodiment of the present application, after the monitoring node obtains the interaction failure rate between each proxy service node and the master node, for each proxy service node, it first determines whether the interaction failure rate of the proxy service node exceeds a second failure rate threshold, and then determines that the number of proxy service nodes whose interaction failure rate exceeds the second failure rate threshold among all proxy service nodes is the second number.

[0085] S502: Determine a third number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold according to the interaction failure rate between each underlying service node and the master node. The third failure rate threshold, the first failure rate threshold, and the second failure rate threshold may be the same or different.

[0086] The third number is the number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold.

[0087] In an embodiment of the present application, after the monitoring node obtains the interaction failure rate between each underlying service node and the main node, for each underlying service node, it first determines whether the interaction failure rate of the underlying service node exceeds the third failure rate threshold, and then determines that the number of underlying service nodes whose interaction failure rate exceeds the third failure rate threshold among all underlying service nodes is the third number.

[0088] S503: Perform fault detection on the master node in the management cluster according to the second number and the third number to obtain a detection result.

[0089] In the embodiment of the present application, after the monitoring node obtains the second quantity and the third quantity based on the above steps, the second quantity and the third quantity can be summed to obtain the quantity sum, and then the quantity sum is determined to be greater than the quantity sum threshold. If it is greater, it indicates that the master node is faulty, and if it is not greater, it indicates that the master node is not faulty. It should be noted that if the monitoring node is a monitoring cluster composed of a server cluster, when any server node in the monitoring cluster determines that the quantity sum exceeds the quantity sum threshold, the master node is determined to be faulty.

[0090] The method described in the embodiment of the present application, since the interaction status information between the target interaction node and the main node can accurately reflect the external service status of the main node, the external service status of the main node can be detected by detecting the number of interaction failure rates between the target interaction node and the main node reaching the threshold. The method is simple and convenient, and can accurately and efficiently detect the main node. Moreover, the above method has different detection methods corresponding to the node types contained in different target interaction nodes, so the method can perform detection considering different scenario factors, thereby improving the detection accuracy.

[0091] In one embodiment, a specific implementation method for performing fault detection on the master node in the management cluster according to the second number and the third number is also provided, such as Figure 7 As shown, the above step S503 of "performing fault detection on the master node in the management cluster according to the second number and the third number to obtain a detection result" includes:

[0092] S601, determining the degree of influence of the proxy service node and the bottom service node on the fault type according to the fault type corresponding to the fault detection.

[0093] Among them, the fault types include master node pseudo-death faults and communication faults between nodes.

[0094] In an embodiment of the present application, a mapping relationship between fault types and different types of service nodes can be constructed in advance based on the fault type, the fault impact degree of the proxy service node, and the fault impact degree of the underlying service node. The mapping relationship includes the fault impact degree of the proxy service node and the fault impact degree of the underlying service node corresponding to each fault type, and the mapping relationship is stored under a preset path.

[0095] When the monitoring node needs to detect the main node pseudo-death fault, it can obtain the mapping relationship from the preset path, and then query the fault impact degree of the proxy service node and the fault impact degree of the underlying service node corresponding to the main node pseudo-death fault.

[0096] S602: assign corresponding weights to the proxy service node and the underlying service node according to the degree of influence of the proxy service node and the underlying service node on the fault type.

[0097] The weight is proportional to the impact of the fault.

[0098] In an embodiment of the present application, after the monitoring node obtains the fault impact degree of the proxy service node and the fault impact degree of the underlying service node based on the above steps, it can then use a weight allocation method to assign corresponding weights to the proxy service node and the underlying service node according to the fault impact degree of the proxy service node and the fault impact degree of the underlying service node, and obtain the proxy service node weight and the underlying service node weight.

[0099] S603: Perform fault detection on the master node in the management cluster according to the weight and the second number of the proxy service node and the weight and the third number of the bottom service node to obtain a detection result.

[0100] In an embodiment of the present application, after obtaining the second number, the third number, the proxy service node weight and the underlying service node weight, the computer device can perform a weighted sum calculation to obtain a calculation result. The calculation method of the calculation result can be to first determine the product of the weight of the proxy service node and the second number to obtain a first product, and determine the product of the weight of the underlying service node and the third number to obtain a second product, and then sum the first product and the second product to obtain the calculation result.

[0101] The method described in the embodiment of the present application combines the influence degree of different service nodes when performing fault detection, and can accurately reflect the role of each node in the fault, thereby improving the accuracy of fault detection.

[0102] In one embodiment, a specific implementation method for receiving interaction status information sent by multiple target interaction nodes in an information interaction cluster for information interaction with a master node in a management cluster is also provided, such as Figure 8 As shown, the "receiving interaction status information of information interaction with the master node in the management cluster sent by multiple target interaction nodes in the information interaction cluster" in the above step S201 includes:

[0103] S701, when receiving alarm information sent by any target interaction node in the information interaction cluster, sending an information acquisition request to other target interaction nodes in the information interaction cluster.

[0104] The alarm information carries the interaction status information of any target interaction node.

[0105] In the embodiment of the present application, for any target interaction node, the target interaction node can analyze the interaction status information between itself and the master node in the management cluster in real time, and determine the interaction failure rate. When the interaction failure rate reaches the failure rate threshold, an alarm message is generated and reported to the monitoring node. After receiving the alarm message reported by the target interaction node, the monitoring node can send an information acquisition request to other target interaction nodes in the information interaction cluster to obtain the interaction status information between other target interaction nodes and the master node.

[0106] S702: Receive interaction status information returned by other target interaction nodes.

[0107] In an embodiment of the present application, after receiving the interaction status information returned by other target interaction nodes, the monitoring node can perform fault detection on the master node in the management cluster according to the interaction status information sent by each target interaction node, obtain the detection result, and then return to execute step S202.

[0108] The method described in the embodiment of the present application, when an alarm message is received, performs fault detection on the main node based on the interaction information of all target interaction nodes. Compared with the method of always performing fault detection on the main node based on the interaction information of all target interaction nodes, the above method can reduce the waste of resources to a certain extent.

[0109] In one embodiment, Fig. 9 As shown, the above fault detection method further includes:

[0110] S203: When the detection result indicates that the master node is faulty, the detection result is sent to each slave node in the management cluster.

[0111] The detection result is used to instruct the slave nodes to interact with each other to determine a new master node and isolate the master node.

[0112] In an embodiment of the present application, when the monitoring node determines that the detection result indicates that the master node is faulty, it sends the detection result to each slave node in the management cluster to instruct each slave node to re-select a new master node and isolate the faulty master node.

[0113] The method described in the embodiment of the present application can ensure that when a master node fails, the cluster can quickly recover and continue to provide services, thereby ensuring the stability of the storage system by reselecting the master node from the slave nodes.

[0114] Based on all the above embodiments, a fault detection method is also provided. Fig.10 As shown, the method includes:

[0115] S801, when receiving alarm information sent by any target interaction node in the information interaction cluster, sending an information acquisition request to other target interaction nodes in the information interaction cluster. The target interaction node includes a proxy service node or a bottom service node.

[0116] S802: Receive interaction status information returned by other target interaction nodes.

[0117] S803, receiving interaction status information of information interaction with the management cluster sent by multiple target interaction nodes in the information interaction cluster; the target interaction nodes include proxy service nodes or bottom service nodes.

[0118] S804: Determine the interaction failure rate between each target interaction node and the master node according to the interaction status information sent by each target interaction node.

[0119] S805: Determine, according to the interaction failure rate between each target interaction node and the master node, a first number of target interaction nodes whose interaction failure rate exceeds a first failure rate threshold.

[0120] S806: Perform fault detection on the master nodes in the management cluster according to the first number to obtain a detection result.

[0121] S807: Determine, according to the interaction failure rate between each proxy service node and the master node, a second number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold.

[0122] S808: Determine, according to the interaction failure rate between each underlying service node and the master node, a third number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold.

[0123] S809: Determine the degree of influence of the proxy service node and the bottom service node on the fault type according to the fault type corresponding to the fault detection.

[0124] S810 , assigning corresponding weights to the proxy service node and the underlying service node according to the degree of influence of the proxy service node and the underlying service node on the fault type.

[0125] S811, performing fault detection on the master node in the management cluster according to the weight and the second number of the proxy service node and the weight and the third number of the bottom service node to obtain a detection result.

[0126] S812, when the detection result indicates that the master node is faulty, the detection result is sent to each slave node in the management cluster, wherein the detection result is used to instruct each slave node to interact with each other to determine a new master node and isolate the master node.

[0127] The methods described in the above steps are all described in the above embodiments. Please refer to the above description for details and will not be repeated here.

[0128] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a fault detection device for implementing the fault detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more fault detection device embodiments provided below can refer to the limitations of the fault detection method above, and will not be repeated here.

[0130] In one embodiment, Fig.11 As shown, a fault detection device is provided, comprising:

[0131] The receiving module 11 is used to receive interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with the management cluster; the target interaction nodes include proxy service nodes or bottom service nodes.

[0132] The detection module 12 is used to perform fault detection on the master node in the management cluster according to the interaction status information sent by each target interaction node to obtain a detection result.

[0133] In one embodiment, the detection module 12 includes:

[0134] The first determining unit is used to determine the interaction failure rate between each target interaction node and the main node according to the interaction state information sent by each target interaction node.

[0135] The first detection unit is used to perform fault detection on the master node in the management cluster according to the interaction failure rate between each target interaction node and the master node to obtain a detection result.

[0136] In one embodiment, the first detection unit comprises:

[0137] The first determining subunit is used to determine a first number of target interaction nodes whose interaction failure rate exceeds a first failure rate threshold according to the interaction failure rate between each target interaction node and the main node.

[0138] The first detection subunit is used to perform fault detection on the master nodes in the management cluster according to the first number to obtain a detection result.

[0139] In one embodiment, the detection module 12 includes:

[0140] The second determining unit is configured to determine, according to the interaction failure rate between each proxy service node and the master node, a second number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold.

[0141] The third determining unit is used to determine, according to the interaction failure rate between each underlying service node and the master node, a third number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold.

[0142] The second detection unit is used to perform fault detection on the master node in the management cluster according to the second number and the third number to obtain a detection result.

[0143] In one embodiment, the second detection unit comprises:

[0144] The second determining subunit is used to determine the degree of influence of the proxy service node and the bottom service node on the fault type according to the fault type corresponding to the fault detection.

[0145] The allocation subunit is used to allocate corresponding weights to the proxy service node and the underlying service node according to the degree of influence of the proxy service node and the underlying service node on the fault type.

[0146] The second detection subunit is used to perform fault detection on the master node in the management cluster according to the weight and the second number of the proxy service node and the weight and the third number of the underlying service node to obtain a detection result.

[0147] In one embodiment, the fault detection device further includes:

[0148] The sending module is used to send an information acquisition request to other target interaction nodes in the management cluster when receiving an alarm message sent by any target interaction node in the information interaction cluster; the alarm message carries the interaction status information of any target interaction node.

[0149] The return module is used to receive the interaction status information returned by other target interaction nodes.

[0150] In one embodiment, the fault detection device further includes:

[0151] The processing module is used to send the detection result to each slave node in the management cluster when the detection result indicates that the master node is in failure; the detection result is used to instruct each slave node to interact with each other to determine a new master node and isolate the master node.

[0152] Each module in the above fault detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0153] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. The internal structure diagram thereof may be as follows: Fig.12As shown, the computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a fault detection method is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0154] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0156] Receiving interaction status information for information interaction with a management cluster sent by multiple target interaction nodes in an information interaction cluster; the target interaction nodes include proxy service nodes or bottom service nodes;

[0157] According to the interaction status information sent by each target interaction node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0158] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0159] Determine the interaction failure rate between each target interaction node and the master node according to the interaction status information sent by each target interaction node;

[0160] According to the interaction failure rate between each target interaction node and the master node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] Determine, according to the interaction failure rate between each target interaction node and the master node, a first number of target interaction nodes whose interaction failure rate exceeds a first failure rate threshold;

[0163] Fault detection is performed on the master nodes in the management cluster according to the first number to obtain a detection result.

[0164] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0165] Determining, according to the interaction failure rate between each proxy service node and the master node, a second number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold;

[0166] Determining, according to the interaction failure rate between each underlying service node and the master node, a third number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold;

[0167] According to the second number and the third number, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0168] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0169] According to the fault type corresponding to the fault detection, determine the degree of influence of the proxy service node and the underlying service node on the fault type;

[0170] According to the degree of influence of the proxy service node and the underlying service node on the fault type, corresponding weights are assigned to the proxy service node and the underlying service node;

[0171] According to the weight and the second number of the proxy service node and the weight and the third number of the underlying service node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0172] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0173] When receiving an alarm message sent by any target interaction node in the information interaction cluster, an information acquisition request is sent to other target interaction nodes in the management cluster; the alarm message carries the interaction status information of any target interaction node;

[0174] Receive interaction status information returned by other target interaction nodes.

[0175] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0176] When the detection result indicates that the master node is faulty, the detection result is sent to each slave node in the management cluster; the detection result is used to instruct each slave node to interact with each other to determine a new master node and isolate the master node.

[0177] The above embodiment provides a computer device, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0178] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0179] Receiving interaction status information for information interaction with a management cluster sent by multiple target interaction nodes in an information interaction cluster; the target interaction nodes include proxy service nodes or bottom service nodes;

[0180] According to the interaction status information sent by each target interaction node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0182] Determine the interaction failure rate between each target interaction node and the master node according to the interaction status information sent by each target interaction node;

[0183] According to the interaction failure rate between each target interaction node and the master node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0185] Determine, according to the interaction failure rate between each target interaction node and the master node, a first number of target interaction nodes whose interaction failure rate exceeds a first failure rate threshold;

[0186] Fault detection is performed on the master nodes in the management cluster according to the first number to obtain a detection result.

[0187] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0188] Determining, according to the interaction failure rate between each proxy service node and the master node, a second number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold;

[0189] Determining, according to the interaction failure rate between each underlying service node and the master node, a third number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold;

[0190] According to the second number and the third number, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0191] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0192] According to the fault type corresponding to the fault detection, determine the degree of influence of the proxy service node and the underlying service node on the fault type;

[0193] According to the degree of influence of the proxy service node and the underlying service node on the fault type, corresponding weights are assigned to the proxy service node and the underlying service node;

[0194] According to the weight and the second number of the proxy service node and the weight and the third number of the underlying service node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] When receiving an alarm message sent by any target interaction node in the information interaction cluster, an information acquisition request is sent to other target interaction nodes in the management cluster; the alarm message carries the interaction status information of any target interaction node;

[0197] Receive interaction status information returned by other target interaction nodes.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] When the detection result indicates that the master node is faulty, the detection result is sent to each slave node in the management cluster; the detection result is used to instruct each slave node to interact with each other to determine a new master node and isolate the master node.

[0200] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0201] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0202] Receiving interaction status information for information interaction with a management cluster sent by multiple target interaction nodes in an information interaction cluster; the target interaction nodes include proxy service nodes or bottom service nodes;

[0203] According to the interaction status information sent by each target interaction node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0204] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0205] Determine the interaction failure rate between each target interaction node and the master node according to the interaction status information sent by each target interaction node;

[0206] According to the interaction failure rate between each target interaction node and the master node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0207] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0208] Determine, according to the interaction failure rate between each target interaction node and the master node, a first number of target interaction nodes whose interaction failure rate exceeds a first failure rate threshold;

[0209] Fault detection is performed on the master nodes in the management cluster according to the first number to obtain a detection result.

[0210] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0211] Determining, according to the interaction failure rate between each proxy service node and the master node, a second number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold;

[0212] Determining, according to the interaction failure rate between each underlying service node and the master node, a third number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold;

[0213] According to the second number and the third number, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0214] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0215] According to the fault type corresponding to the fault detection, determine the degree of influence of the proxy service node and the underlying service node on the fault type;

[0216] According to the degree of influence of the proxy service node and the underlying service node on the fault type, corresponding weights are assigned to the proxy service node and the underlying service node;

[0217] According to the weight and the second number of the proxy service node and the weight and the third number of the underlying service node, fault detection is performed on the master node in the management cluster to obtain a detection result.

[0218] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0219] When receiving an alarm message sent by any target interaction node in the information interaction cluster, an information acquisition request is sent to other target interaction nodes in the management cluster; the alarm message carries the interaction status information of any target interaction node;

[0220] Receive interaction status information returned by other target interaction nodes.

[0221] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0222] When the detection result indicates that the master node is faulty, the detection result is sent to each slave node in the management cluster; the detection result is used to instruct each slave node to interact with each other to determine a new master node and isolate the master node.

[0223] The above embodiment provides a computer program product, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0224] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0225] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0226] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A fault detection method, characterized in that: The method comprises: Receiving interaction status information for information interaction with a management cluster sent by a plurality of target interaction nodes in an information interaction cluster; the target interaction nodes include proxy service nodes or bottom service nodes; Fault detection is performed on the master node in the management cluster according to the interaction status information sent by each of the target interaction nodes to obtain a detection result.

2. The method according to claim 1, characterized in that The performing fault detection on the master node in the management cluster according to the interaction status information sent by each of the target interaction nodes to obtain a detection result includes: Determining the interaction failure rate between each of the target interaction nodes and the master node according to the interaction status information sent by each of the target interaction nodes; According to the interaction failure rate between each of the target interaction nodes and the master node, fault detection is performed on the master node in the management cluster to obtain a detection result.

3. The method according to claim 2, characterized in that Each of the target interaction nodes is the proxy service node or the underlying service node. According to the interaction failure rate between each of the target interaction nodes and the master node, fault detection is performed on the master node in the management cluster to obtain a detection result, including: Determine, according to the interaction failure rate between each of the target interaction nodes and the master node, a first number of target interaction nodes whose interaction failure rate exceeds a first failure rate threshold; Fault detection is performed on master nodes in the management cluster according to the first number to obtain a detection result.

4. The method according to claim 2, characterized in that The multiple target interaction nodes include the proxy service node and the underlying service node, and the master node in the management cluster is subjected to fault detection according to the interaction failure rate between each of the target interaction nodes and the master node to obtain a detection result, including: Determining, according to the interaction failure rate between each of the proxy service nodes and the master node, a second number of proxy service nodes whose interaction failure rate exceeds a second failure rate threshold; Determining, according to the interaction failure rate between each of the underlying service nodes and the master node, a third number of underlying service nodes whose interaction failure rate exceeds a third failure rate threshold; According to the second number and the third number, fault detection is performed on the master node in the management cluster to obtain a detection result.

5. The method according to claim 4, characterized in that The performing fault detection on the master node in the management cluster according to the second number and the third number to obtain a detection result includes: Determining, according to the fault type corresponding to the fault detection, the degree of influence of the proxy service node and the underlying service node on the fault type; According to the degree of influence of the proxy service node and the underlying service node on the fault type, corresponding weights are assigned to the proxy service node and the underlying service node; According to the weight and the second number of the proxy service node, and the weight and the third number of the underlying service node, fault detection is performed on the master node in the management cluster to obtain a detection result.

6. The method according to claim 1, characterized in that The receiving of interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with a master node in the management cluster includes: When receiving an alarm message sent by any target interaction node in the information interaction cluster, sending an information acquisition request to other target interaction nodes in the information interaction cluster; the alarm message carries interaction status information of any target interaction node; Receive the interaction status information returned by the other target interaction node.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When the detection result indicates that the master node is faulty, the detection result is sent to each slave node in the management cluster; the detection result is used to instruct each slave node to interact with each other to determine a new master node and isolate the master node.

8. A fault detection device, characterized in that: The device comprises: A receiving module, used to receive interaction status information sent by multiple target interaction nodes in the information interaction cluster for information interaction with the management cluster; the target interaction nodes include proxy service nodes or bottom service nodes; The detection module is used to perform fault detection on the master node in the management cluster according to the interaction status information sent by each target interaction node to obtain a detection result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images