Storage node fault detection method and device, computer equipment and storage medium

Abstract

The invention relates to a storage node fault detection method and device, computer equipment and a storage medium. The method comprises the following steps: respectively sending a heartbeat detection message to a management node, a storage node and a client node in the distributed system, so that the management node, the storage node and the client node carry out heartbeat detection on a to-be-detected storage node, and receiving a first heartbeat detection result fed back by each management node, a second heartbeat detection result fed back by each storage node and a third heartbeat detection result fed back by each client node, and sending the first heartbeat detection result, the second heartbeat detection result and the third heartbeat detection result to the server node based on the first heartbeat detection result, the second heartbeat detection result and the third heartbeat detection result. And determining a first fault probability, a second fault probability and a third fault probability of the to-be-detected storage node, and judging whether the to-be-detected storage node has a fault based on the first fault probability, the second fault probability and the third fault probability. By adopting the method, the accuracy of storage node fault detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of distributed technology, and in particular to a storage node fault detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the advent of the big data era, the amount of data is increasing rapidly. In order to meet the surging data storage needs, distributed storage systems are also becoming larger and larger. Distributed storage systems need to provide a large amount of data storage needs while maintaining system stability and having a certain degree of fault self-healing capabilities. This requires the system to be able to detect storage node failures and perform corresponding fault handling according to the failures.

[0003] At present, distributed storage systems mainly use the heartbeat mechanism to detect whether storage nodes are faulty. However, this detection scheme is prone to misjudgment and time-consuming problems, resulting in low detection accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a more accurate storage node fault detection method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical issues.

[0005] In a first aspect, the present application provides a storage node failure detection method, comprising:

[0006] Sending heartbeat detection messages to management nodes, storage nodes and client nodes in a distributed system respectively, so that the management nodes, storage nodes and client nodes perform heartbeat detection on the storage node to be detected, and the number of the management nodes, storage nodes and client nodes is multiple;

[0007] Receive the first heartbeat detection result fed back by each of the management nodes, the second heartbeat detection result fed back by each of the storage nodes, and the third heartbeat detection result fed back by each of the client nodes;

[0008] Based on the first heartbeat detection result fed back by each of the management nodes, determine the first failure probability of the storage node to be tested; based on the second heartbeat detection result fed back by each of the storage nodes, determine the second failure probability of the storage node to be tested; based on the third heartbeat detection result fed back by each of the client nodes, determine the third failure probability of the storage node to be tested;

[0009] Based on the first failure probability, the second failure probability and the third failure probability, it is determined whether the storage node to be tested fails.

[0010] In one embodiment, determining a first failure probability of the storage node to be tested based on the first heartbeat detection result fed back by each of the management nodes includes:

[0011] Based on the first heartbeat detection result fed back by each of the management nodes, the number of management nodes in which the storage node to be tested has a fault is statistically determined, a first proportion of the number of management nodes to the total number of management nodes is determined, and the first failure probability is obtained based on the first proportion and a preset first weight;

[0012] The determining of the first failure probability of the storage node to be tested, based on the second heartbeat detection result fed back by each of the storage nodes, comprises:

[0013] Based on the second heartbeat detection result fed back by each of the storage nodes, the number of storage nodes in which the storage nodes to be tested have faults is statistically determined, a second proportion of the number of the storage nodes relative to the total number of the storage nodes is determined, and the second failure probability is obtained based on the second proportion and a preset second weight;

[0014] The determining, based on the third heartbeat detection result fed back by each of the client nodes, a third failure probability of the storage node to be tested comprises:

[0015] Based on the third heartbeat detection result fed back by each of the client nodes, the number of client nodes at which the storage node to be tested has a fault is statistically determined, a third proportion of the number of the client nodes relative to the total number of the client nodes is determined, and the third failure probability is obtained based on the third proportion and a preset third weight;

[0016] The sum of the first weight, the second weight and the third weight is 1.

[0017] In one embodiment, sending a heartbeat detection message to the storage node includes:

[0018] Obtaining the number of storage nodes;

[0019] Determine the number of target storage nodes participating in heartbeat detection according to the number of storage nodes and a preset storage node number determination rule;

[0020] Send a heartbeat detection message to the target storage node.

[0021] In one embodiment, sending a heartbeat detection message to the client node includes:

[0022] Obtaining the number of client nodes;

[0023] Determine the number of target client nodes participating in heartbeat detection according to the number of the client nodes and a preset client node number determination rule;

[0024] Send a heartbeat detection message to the target client node.

[0025] In one embodiment, the determining whether the storage node to be tested fails based on the first failure probability, the second failure probability, and the third failure probability includes:

[0026] Summing the first failure probability, the second failure probability and the third failure probability to obtain the failure probability of the storage node to be tested;

[0027] The failure probability of the storage node to be tested is compared with a preset probability threshold to determine whether the storage node to be tested fails.

[0028] In one embodiment, the determining whether the storage node to be tested fails based on the first failure probability, the second failure probability, and the third failure probability includes:

[0029] Summing the first failure probability, the second failure probability and the third failure probability to obtain the failure probability of the storage node to be tested;

[0030] When the failure probability measured at least twice in succession is greater than a preset probability threshold, it is determined that the storage node to be tested fails.

[0031] In a second aspect, the present application also provides a storage node failure detection device. The device comprises:

[0032] A message sending module, used for sending heartbeat detection messages to a management node, a storage node and a client node in a distributed system respectively, so that the management node, the storage node and the client node perform heartbeat detection on the storage node to be detected, and the number of the management node, the storage node and the client node is multiple;

[0033] A data receiving module, used to receive the first heartbeat detection result fed back by each of the management nodes, the second heartbeat detection result fed back by each of the storage nodes, and the third heartbeat detection result fed back by each of the client nodes;

[0034] A failure probability determination module is used to determine a first failure probability of the storage node to be tested based on a first heartbeat detection result fed back by each of the management nodes, determine a second failure probability of the storage node to be tested based on a second heartbeat detection result fed back by each of the storage nodes, and determine a third failure probability of the storage node to be tested based on a third heartbeat detection result fed back by each of the client nodes;

[0035] A fault judgment module is used to judge whether the storage node to be tested fails based on the first fault probability, the second fault probability and the third fault probability.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in any one of the above-mentioned storage node fault detection method embodiments when executing the computer program.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any one of the above-mentioned storage node fault detection method embodiments are implemented.

[0038] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in any one of the above-mentioned storage node failure detection method embodiments.

[0039] The above storage node fault detection method, device, computer equipment, computer-readable storage medium and computer program product, by incorporating the management node, storage node and client node into the heartbeat detection, so that they perform heartbeat detection on the same node to be tested, can provide a more comprehensive health status assessment, reduce the possibility of misjudgment due to network delays, self-failures and other reasons, and by receiving the heartbeat detection results fed back by nodes in three different dimensions, and determining different fault probabilities based on these heartbeat detection results, and finally comprehensively judging whether the storage node to be tested has a fault, it can significantly improve the accuracy of fault judgment. The entire scheme, through the method of multi-point detection and comprehensive analysis, not only improves the efficiency and accuracy of fault detection, but also enhances the overall reliability and service quality of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A diagram of an application environment of a storage node failure detection method in an embodiment;

[0042] Figure 2 A schematic diagram of a storage node failure detection method according to an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the system architecture of a distributed storage system in one embodiment;

[0044] Figure 4 is a schematic flow chart of a storage node failure detection method in another embodiment;

[0045] Figure 5 A schematic diagram of a flow chart of a storage node failure detection method in another embodiment;

[0046] Figure 6 is a structural block diagram of a storage node failure detection device in one embodiment;

[0047] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The storage node fault detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in the figure, the terminal (not shown in the figure) communicates with the main management node 104 in the distributed storage system, the management node 106, the storage node 108, and the client node 110 in the distributed storage system through the network. The data storage system can store the data that the main management node 104 needs to process. The data storage system can be integrated on the main management node 104, or it can be placed on the cloud or other network servers.

[0050] Specifically, the operator may send a data analysis instruction to the main management node 104 through the terminal, and the main management node 104 sends heartbeat detection messages to the management node 106, the storage node 108 and the client node 110 in the distributed system respectively, so that the management node 106, the storage node 108 and the client node 110 perform heartbeat detection on the storage node to be tested (not shown in the figure). Subsequently, the main management node 104 receives the first heartbeat detection result fed back by the management node 106, the second heartbeat detection result fed back by the storage node 108, and the third heartbeat detection result fed back by the client node 110. Then, based on the first heartbeat detection result, the first failure probability of the storage node to be tested is determined, based on the second heartbeat detection result, the second failure probability of the storage node to be tested is determined, and based on the third heartbeat detection result, the third failure probability of the storage node to be tested is determined. Finally, based on the first failure probability, the second failure probability and the third failure probability, it is determined whether the storage node to be tested has a failure.

[0051] Among them, the terminal and the client node 110 can be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The main management node 104, the management node 106 and the storage node 108 can be independent physical servers, or server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide cloud computing services.

[0052] In an exemplary embodiment, Figure 2 As shown, a storage node failure detection method is provided, and the method is applied to Figure 1 The main management node 104 in FIG. 1 is used as an example to illustrate the method, which includes the following steps S200 to S800. Among them:

[0053] S200, sending heartbeat detection messages to management nodes, storage nodes and client nodes in the distributed system respectively, so that the management nodes, storage nodes and client nodes perform heartbeat detection on the storage node to be tested, and the number of management nodes, storage nodes and client nodes is multiple.

[0054] Heartbeat detection information is a message used to check whether a remote node is online and in good condition. It usually contains a simple request and requires the other party to reply within a specified time. The management node (master) is responsible for managing and coordinating the entire distributed system, including tasks such as monitoring and scheduling. The storage node (store) is responsible for data storage. There are usually multiple nodes in a distributed storage system for redundancy and performance improvement. The client node (client) is the end user device or application that initiates a request to the distributed system.

[0055] In distributed storage systems, a heartbeat detection mechanism is usually used to detect whether a storage node fails. Common architectures or deployment methods of distributed storage systems are as follows: Figure 3 As shown in the figure, store is a storage node, master is a management node (also called management control node), and client is a client node. The master has a view of the entire cluster. After the client obtains the view from the master, it knows which storage node the corresponding request should be sent to based on the view. Among them, the main management control node is responsible for managing and coordinating the key component of the entire system, responsible for monitoring and managing all nodes participating in the distributed storage system, including storage nodes, computing nodes, and client nodes.

[0056] In specific implementation, the client sends a fault detection message to the main management node, and the main management node responds to the fault detection message and sends heartbeat detection messages to multiple management nodes, storage nodes and client nodes in the distributed storage system respectively. The heartbeat detection message carries the identifier of the storage node to be tested, so that each node can clearly perform a heartbeat detection on the storage node to be tested corresponding to the storage node identifier to be tested. It can be understood that there can be one or more storage nodes to be tested. If there are multiple storage nodes, each management node, storage node and client node can perform a heartbeat detection on the storage node to be tested simultaneously or in sequence.

[0057] S400, receiving a first heartbeat detection result fed back by each management node, a second heartbeat detection result fed back by each storage node, and a third heartbeat detection result fed back by each client node.

[0058] The heartbeat detection result means that after receiving the heartbeat detection message, each node will try to contact the storage node to be tested and return a result based on its response. The heartbeat detection result specifically includes but is not limited to the response status of whether the storage node to be tested successfully responds to the heartbeat detection information, the response time, and the judgment result of whether the node to be tested has a fault.

[0059] Each management node, each storage node, and each client node that receives the heartbeat detection message performs heartbeat detection on the storage node to be tested, detects whether the storage node to be tested is online and its operating status, and returns a heartbeat detection result to the main management node according to the response of the storage node to be tested. The main management node can store the heartbeat detection results fed back by each node in the memory or database for subsequent processing. The heartbeat detection process can be a timeout mechanism. If the storage node to be tested does not respond within a predetermined time, it is considered that the node has not responded.

[0060] In this embodiment, for the sake of distinction, the heartbeat detection result fed back by the management node is referred to as the first heartbeat detection result, the heartbeat detection result fed back by the storage node is referred to as the second heartbeat detection result, and the heartbeat detection result fed back by the client node is referred to as the third heartbeat detection result. It is understandable that in actual applications, the heartbeat detection results fed back by each node include but are not limited to the response status of whether the storage node to be tested successfully responds to the heartbeat detection information, the response time, the judgment result of whether the node to be tested fails, etc.

[0061] S600, based on the first heartbeat detection result fed back by each management node, determines the first failure probability of the storage node to be tested, based on the second heartbeat detection result fed back by each storage node, determines the second failure probability of the storage node to be tested, based on the third heartbeat detection result fed back by each client node, determines the third failure probability of the storage node to be tested.

[0062] The failure probability refers to the probability that the storage node under test may fail, calculated based on the heartbeat detection result.

[0063] After the main management node receives the heartbeat detection results fed back by each node, it can use a predefined algorithm (such as a Bayesian classifier or other statistical methods) to calculate the first failure probability of the storage node to be tested based on the received first heartbeat detection result. Similarly, the above process is repeated for the second heartbeat detection result and the third heartbeat detection result to obtain the second failure probability and the third failure probability, respectively.

[0064] S800, judging whether a storage node to be tested fails based on the first failure probability, the second failure probability and the third failure probability.

[0065] In specific implementation, after obtaining the first failure probability, the second failure probability and the third failure probability, the main management node can summarize the first failure probability, the second failure probability and the third failure probability, and can adopt weighted average or other comprehensive evaluation methods to obtain a comprehensive failure probability. Then, the comprehensive probability threshold is compared with the set failure probability threshold, such as 50%. If the comprehensive failure probability exceeds this threshold, it is determined that the storage node to be tested has failed. Furthermore, the state of the storage node to be tested can be changed to a failure state. Further measures can also be taken, such as issuing an alarm, starting a backup node, redirecting traffic, etc., to ensure the stable operation of the system.

[0066] In some embodiments, determining a first failure probability of the storage node to be tested based on a first heartbeat detection result fed back by each management node includes: based on the first heartbeat detection result fed back by each management node, statistically determining the number of management nodes at which failures exist in the storage node to be tested, determining a first proportion of the number of management nodes to the total number of management nodes, and obtaining a first failure probability based on the first proportion and a preset first weight.

[0067] In this embodiment, the proportion refers to the ratio of the number of nodes in a certain type of node that determine that the node to be tested has a fault to the total number of nodes of that type. The weight is the importance ratio of the detection results of different types of nodes when calculating the comprehensive failure probability. In this embodiment, for the convenience of distinction, the weight representing the importance ratio of the detection results of the management node is called the first weight x, the weight representing the importance ratio of the detection results of the storage node is called the second weight y, and the weight representing the importance ratio of the detection results of the client node is called the third weight z.

[0068] In a typical distributed storage system, when judging whether a storage node is faulty, the weight of the master (management node) will be larger, that is, in the above formula, the first weight is greater than the second weight, and greater than the third weight, and the sum of the first weight, the second weight, and the third weight is 1. Usually, the first weight x can be set to 0.5, and the total weight of the storage node and the client (client node) is 0.5, that is, y+z=0.5. As for the weights occupied by the storage node and the client node, you can flexibly adjust them according to the scale of the system and the desired effect.

[0069] In a specific implementation, the number of management nodes participating in fault detection in the current distributed storage system may be recorded as MSIZE. The master management node may count the number of management nodes that determine that the storage node to be tested has faults based on the first heartbeat detection result fed back by each management node, and record it as MRF. Subsequently, the ratio of the number of management nodes MRF that determine that the storage node to be tested has faults to the total number of management nodes MSIZE is calculated to obtain a first proportion. The first proportion is then multiplied by a preset first weight x to obtain a first fault probability, that is, the first fault probability = .

[0070] Determine a first failure probability of the storage node to be tested, and determine a second failure probability of the storage node to be tested based on a second heartbeat detection result fed back by each storage node, including: based on the second heartbeat detection result fed back by each storage node, statistically determine the number of storage nodes that have faults in the storage node to be tested, determine a second proportion of the number of storage nodes relative to the total number of storage nodes, and obtain a second failure probability based on the second proportion and a preset second weight.

[0071] Similarly, the number of storage nodes participating in fault detection in the current distributed storage system may be recorded as SRF. The main management node may count the number of storage nodes that are determined to have faults in the storage nodes to be tested based on the second heartbeat detection results fed back by each storage node, and record it as SSIZE. Subsequently, the ratio of the number of storage nodes SRF that are determined to have faults in the storage nodes to be tested to the total number of storage nodes SSIZE is calculated to obtain the second proportion. The second proportion is then multiplied by the preset second weight y to obtain the second fault probability, that is, the second fault probability = .

[0072] Based on the third heartbeat detection result fed back by each client node, a third failure probability of the storage node to be tested is determined, including: based on the third heartbeat detection result fed back by each client node, statistically determining the number of client nodes at which failures exist in the storage node to be tested, determining a third proportion of the number of client nodes relative to the total number of client nodes, and obtaining a third failure probability based on the third proportion and a preset third weight.

[0073] Similarly, the number of client nodes participating in fault detection in the current distributed storage system may be recorded as CRF. The main management node may count the number of client nodes that determine that the storage node to be tested has faults based on the third heartbeat detection result fed back by each client node, and record it as CSIZE. Subsequently, the ratio of the number of clients CRF that determine that the storage node to be tested has faults to the total number of clients CSIZE is calculated to obtain the third proportion. The third proportion is then multiplied by the preset third weight y to obtain the third fault probability, that is, the third fault probability = .

[0074] Among them, the first weight x is 0.5, the second weight y is 0.3, and the third weight z is 0.2.

[0075] In this embodiment, the failure probability is determined by setting weights, and the weights can be flexibly adjusted according to the scale of the system and the desired effect, thereby determining the failure probability under different circumstances.

[0076] In some embodiments, sending a heartbeat detection message to a storage node includes: obtaining the number of storage nodes, determining rules based on the number of storage nodes and a preset number of storage nodes, determining the number of target storage nodes participating in the heartbeat detection, and sending a heartbeat detection message to the target storage node.

[0077] In a distributed storage cluster, there are usually 3, 5 or 7 master nodes (management nodes). Therefore, all management nodes can participate in the heartbeat detection mechanism.

[0078] As the number of storage nodes increases with the size of the cluster, when the cluster size reaches a certain level, if all storage nodes are still required to participate in heartbeat detection, it will inevitably increase the network burden. Therefore, when the cluster reaches a certain size, some storage nodes are randomly selected to perform heartbeat detection instead of all storage nodes participating.

[0079] In practical applications, the number of storage nodes can be determined by presetting the following rules:

[0080] (1) When the number of storage nodes is less than or equal to 10, let all storage nodes participate in heartbeat detection;

[0081] (2) When the number of storage nodes is greater than 10 and less than 200, 10 storage nodes are randomly selected to participate in heartbeat detection;

[0082] (3) When the number of storage nodes is greater than or equal to 200 and less than 1000, 20 storage nodes are randomly selected to participate in heartbeat detection;

[0083] (4) When the number of storage nodes is greater than 1000, 30 storage nodes are randomly selected to participate in heartbeat detection.

[0084] It is understandable that in other embodiments, the rules for determining the number of storage nodes may be adjusted according to actual conditions, and are not limited to this rule.

[0085] In specific implementation, the main management node can count the total number of storage nodes in the entire cluster, and then determine the number of target storage nodes participating in the heartbeat detection based on the total number of storage nodes and the above storage node number determination rule, and then send heartbeat detection messages to the selected target storage nodes respectively, so that the target storage nodes perform heartbeat detection on the designated storage nodes to be tested and feedback the heartbeat detection results. After receiving the responses from all the selected storage nodes, the main management node calculates the probability that the storage node believes that a certain storage node is faulty according to the formula.

[0086] In this embodiment, the scale of the entire cluster is taken into consideration to determine the number of storage nodes participating in heartbeat detection, which can effectively reduce network pressure and fault detection time.

[0087] In some embodiments, sending a heartbeat detection message to a client node includes: obtaining the number of client nodes, determining rules based on the number of client nodes and a preset number of client nodes, determining the number of target client nodes participating in the heartbeat detection, and sending a heartbeat detection message to the target client nodes.

[0088] Similarly, client nodes are similar to storage nodes. If there are many client nodes, some client nodes can also be randomly selected to perform heartbeat detection on the specified storage nodes to be tested.

[0089] In actual applications, developers can pre-set the selection rules for client nodes as follows:

[0090] (1) When the number of client nodes is less than 6, select all clients;

[0091] (2) When the number of client nodes is greater than or equal to 6 and less than 100, 6 client nodes are randomly selected;

[0092] (4) When the number of client nodes is greater than 100 and less than 1000, 10 client nodes are randomly selected;

[0093] (4) When the number of client nodes is greater than or equal to 1000, 16 client nodes are randomly selected.

[0094] It is understandable that in other embodiments, the rule for determining the number of client nodes may be adjusted according to actual conditions, and is not limited to this rule.

[0095] In specific implementation, the main management node can count the total number of client nodes in the entire cluster, and then determine the number of target client nodes participating in the heartbeat detection based on the total number of client nodes and the above-mentioned client node number determination rule, and then send heartbeat detection messages to the selected target client nodes respectively, so that the target client nodes perform heartbeat detection on the specified storage node to be tested and feedback the heartbeat detection results. After the main management node receives the heartbeat detection results fed back by all the selected client nodes, it calculates the probability that the client node believes that a certain client node is faulty according to the formula.

[0096] In this embodiment, the scale of the client nodes in the entire cluster is taken into consideration to determine the number of client nodes participating in the heartbeat detection, which can effectively reduce network pressure and detection time.

[0097] like Figure 4 As shown, in some embodiments, S800 includes: S820, summing the first failure probability, the second failure probability and the third failure probability to obtain the failure probability of the storage node to be tested, comparing the failure probability of the storage node to be tested with a preset probability threshold to determine whether the storage node to be tested fails.

[0098] In this embodiment, after the main management node determines the first failure probability, the second failure probability and the third failure probability of a storage node to be tested by the failure probability formula in the above embodiment, the first failure probability, the second failure probability and the third failure probability may be summed to obtain the comprehensive failure probability of the storage node to be tested. . Subsequently, the calculated comprehensive failure probability Pf is compared with a preset failure probability threshold value, such as 50%. If the failure probability Pf is greater than 50%, it can be determined that the storage node to be tested has a failure, and the state of the storage node to be tested is updated to a failure state. Otherwise, it is determined that there is no failure, and the fault detection process continues. It can be understood that the failure probability threshold value can be 60%, 80% or other values, which can be determined according to the actual situation and is not limited here.

[0099] It can be understood that in other embodiments, after the main management node determines the first failure probability, the second failure probability and the third failure probability of a storage node to be tested, it can also obtain a comprehensive failure probability Pf by performing weighted summation on the first failure probability, the second failure probability and the third failure probability, and compare the failure probability Pf with a preset failure probability threshold such as 50%. If the failure probability Pf is greater than 50%, it can be determined that the storage node to be tested has a fault, and the status of the storage node to be tested is updated to a faulty state.

[0100] In this embodiment, by summing the first failure probability, the second failure probability and the third failure probability and comparing the summation result with a preset failure probability threshold, it is possible to simply and efficiently determine whether the storage node to be tested has a fault.

[0101] In addition to the above fault detection methods, there are other detection methods. Figure 5 As shown, in some embodiments, S800 includes: S840, summing the first failure probability, the second failure probability and the third failure probability to obtain the failure probability of the storage node to be tested, and determining that the storage node to be tested has a failure when the failure probability measured at least twice in a row is greater than a preset probability threshold.

[0102] In this embodiment, after the main management node determines the first failure probability, the second failure probability and the third failure probability of a storage node to be tested by the failure probability formula in the above embodiment, the first failure probability, the second failure probability and the third failure probability may be summed to obtain the comprehensive failure probability of the storage node to be tested. . Subsequently, the calculated comprehensive failure probability Pf is compared with a preset failure probability threshold value, such as 50%. If the failure probability Pf measured based on this round of heartbeat detection process is greater than 50%, the heartbeat detection result and failure probability of this round are saved, and the next round of heartbeat detection and failure detection are continued. If the failure probability Pf determined by the next round of heartbeat detection results is still greater than 50%, that is, the failure probability Pf measured twice in a row is greater than 50%, then it can be determined that there is a failure in the storage node to be tested, and the state of the storage node to be tested is updated to a failure state. Otherwise, it is determined that there is no failure, and the fault detection process continues. It can be understood that the failure probability threshold value can be 60%, 80% or other values, which can be determined according to the actual situation and is not limited here.

[0103] In this embodiment, by comprehensively analyzing two consecutive failure probabilities to determine whether the node to be stored is faulty, the reliability of the fault detection result can be improved, and the problem of inaccurate fault detection results due to other influencing factors can be reduced.

[0104] In order to more clearly explain the storage node failure detection method provided by the present application, a specific embodiment is described below. The specific embodiment includes the following steps:

[0105] S1, obtaining the number of storage nodes, determining rules according to the number of storage nodes and a preset number of storage nodes, and determining the number of target storage nodes participating in heartbeat detection.

[0106] S2, obtaining the number of client nodes, determining a rule according to the number of client nodes and a preset number of client nodes, and determining the number of target client nodes participating in the heartbeat detection.

[0107] S3, respectively sends heartbeat detection messages to all management nodes in the distributed system, as well as the screened target storage nodes and target client nodes, so that each management node, each storage node and each client node performs heartbeat detection on the storage node to be tested, and feeds back the heartbeat detection results to the main management node.

[0108] S4, receiving the first heartbeat detection result fed back by each management node, the second heartbeat detection result fed back by each storage node, and the third heartbeat detection result fed back by each client node.

[0109] S5, based on the first heartbeat detection result fed back by each management node, statistically determine the number of management nodes with faults in the storage node to be tested, determine a first ratio of the number of management nodes to the total number of management nodes, and obtain a first failure probability based on the first ratio and a preset first weight.

[0110] S6, based on the second heartbeat detection result fed back by each storage node, statistically determine the number of storage nodes with faults in the storage nodes to be tested, determine a second ratio of the number of storage nodes to the total number of storage nodes, and obtain a second failure probability based on the second ratio and a preset second weight.

[0111] S7, based on the third heartbeat detection result fed back by each client node, statistically determine the number of client nodes that have faults in the storage node to be tested, determine the third ratio of the number of client nodes to the total number of client nodes, and obtain the third failure probability based on the third ratio and the preset third weight.

[0112] S8, summing the first failure probability, the second failure probability and the third failure probability to obtain the failure probability of the storage node to be tested.

[0113] S9: When the failure probability measured at least twice in a row is greater than a preset probability threshold, it is determined that the storage node to be tested fails.

[0114] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a storage node fault detection device for implementing the storage node fault detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more storage node fault detection device embodiments provided below can refer to the limitations of the storage node fault detection method above, and will not be repeated here.

[0116] In an exemplary embodiment, Figure 6 As shown, a storage node fault detection device 600 is provided, comprising: a message sending module 610, a data receiving module 620, a fault probability determination module 630 and a fault judgment module 640, wherein:

[0117] The message sending module 610 is used to send heartbeat detection messages to the management node, storage node and client node in the distributed system respectively, so that the management node, storage node and client node perform heartbeat detection on the storage node to be tested. The number of management nodes, storage nodes and client nodes is multiple.

[0118] The data receiving module 620 is used to receive the first heartbeat detection result fed back by each management node, the second heartbeat detection result fed back by each storage node, and the third heartbeat detection result fed back by each client node.

[0119] The failure probability determination module 630 is used to determine the first failure probability of the storage node to be tested based on the first heartbeat detection result fed back by each management node, determine the second failure probability of the storage node to be tested based on the second heartbeat detection result fed back by each storage node, and determine the third failure probability of the storage node to be tested based on the third heartbeat detection result fed back by each client node.

[0120] The fault judgment module 640 is used to judge whether a fault occurs in the storage node to be tested based on the first fault probability, the second fault probability and the third fault probability.

[0121] In some embodiments, the failure probability determination module 630 is further used to count the number of management nodes that determine that the storage node to be tested has a failure based on the first heartbeat detection result fed back by each management node, determine a first proportion of the number of management nodes to the total number of management nodes, and obtain a first failure probability based on the first proportion and a preset first weight;

[0122] And, based on the second heartbeat detection result fed back by each storage node, the number of storage nodes that are faulty in the storage node to be tested is statistically determined, a second proportion of the number of storage nodes relative to the total number of storage nodes is determined, and a second fault probability is obtained based on the second proportion and a preset second weight;

[0123] In addition, based on the third heartbeat detection result fed back by each client node, the number of client nodes that have faults in the storage node to be tested is statistically determined, and a third proportion of the number of client nodes relative to the total number of client nodes is determined. Based on the third proportion and a preset third weight, a third failure probability is obtained, wherein the sum of the first weight, the second weight and the third weight is 1.

[0124] In one embodiment, the message sending module 610 is also used to obtain the number of storage nodes, determine the number of target storage nodes participating in heartbeat detection according to the number of storage nodes and a preset number of storage nodes, and send a heartbeat detection message to the target storage nodes.

[0125] In one embodiment, the message sending module 610 is also used to obtain the number of client nodes, determine the number of target client nodes participating in the heartbeat detection according to the number of client nodes and a preset number of client nodes, and send a heartbeat detection message to the target client nodes.

[0126] In one of the embodiments, the fault probability determination module 630 is also used to sum the first fault probability, the second fault probability and the third fault probability to obtain the failure probability of the storage node to be tested, and compare the failure probability of the storage node to be tested with a preset probability threshold to determine whether the storage node to be tested fails.

[0127] In one of the embodiments, the fault judgment module 640 is also used to sum the first fault probability, the second fault probability and the third fault probability to obtain the fault probability of the storage node to be tested, and when the fault probability measured at least twice in a row is greater than a preset probability threshold, it is determined that the storage node to be tested has a fault.

[0128] Each module in the above storage node fault detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0129] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as heartbeat detection results and node selection rules. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a storage node fault detection method is implemented.

[0130] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in any one of the above storage node failure detection method embodiments are implemented.

[0132] In one embodiment, when the processor executes the computer program, it also implements the steps in any one of the above storage node failure detection method embodiments.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any one of the above storage node failure detection method embodiments are implemented.

[0134] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps in any one of the above storage node failure detection method embodiments.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0136] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0137] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0138] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A storage node failure detection method, characterized in that: The method comprises: Sending heartbeat detection messages to management nodes, storage nodes and client nodes in a distributed system respectively, so that the management nodes, storage nodes and client nodes perform heartbeat detection on the storage node to be detected, and the number of the management nodes, storage nodes and client nodes is multiple; Receive the first heartbeat detection result fed back by each of the management nodes, the second heartbeat detection result fed back by each of the storage nodes, and the third heartbeat detection result fed back by each of the client nodes; Based on the first heartbeat detection result fed back by each of the management nodes, determine the first failure probability of the storage node to be tested; based on the second heartbeat detection result fed back by each of the storage nodes, determine the second failure probability of the storage node to be tested; based on the third heartbeat detection result fed back by each of the client nodes, determine the third failure probability of the storage node to be tested; Based on the first failure probability, the second failure probability and the third failure probability, it is determined whether the storage node to be tested fails.

2. The method according to claim 1, characterized in that The determining, based on the first heartbeat detection result fed back by each of the management nodes, the first failure probability of the storage node to be tested comprises: Based on the first heartbeat detection result fed back by each of the management nodes, the number of management nodes in which the storage node to be tested has a fault is statistically determined, a first proportion of the number of management nodes to the total number of management nodes is determined, and the first failure probability is obtained based on the first proportion and a preset first weight; The determining of the first failure probability of the storage node to be tested, based on the second heartbeat detection result fed back by each of the storage nodes, comprises: Based on the second heartbeat detection result fed back by each of the storage nodes, the number of storage nodes in which the storage nodes to be tested have faults is statistically determined, a second proportion of the number of the storage nodes relative to the total number of the storage nodes is determined, and the second failure probability is obtained based on the second proportion and a preset second weight; The determining, based on the third heartbeat detection result fed back by each of the client nodes, a third failure probability of the storage node to be tested comprises: Based on the third heartbeat detection result fed back by each of the client nodes, the number of client nodes at which the storage node to be tested has a fault is statistically determined, a third proportion of the number of the client nodes relative to the total number of the client nodes is determined, and the third failure probability is obtained based on the third proportion and a preset third weight; The sum of the first weight, the second weight and the third weight is 1.

3. The method according to claim 1 or 2, characterized in that: Sending a heartbeat detection message to the storage node includes: Obtaining the number of storage nodes; Determine the number of target storage nodes participating in heartbeat detection according to the number of storage nodes and a preset storage node number determination rule; Send a heartbeat detection message to the target storage node.

4. The method according to claim 3, characterized in that Sending a heartbeat detection message to the client node includes: Obtaining the number of client nodes; Determine the number of target client nodes participating in heartbeat detection according to the number of the client nodes and a preset client node number determination rule; Send a heartbeat detection message to the target client node.

5. The method according to claim 4, characterized in that The determining whether the storage node to be tested fails based on the first failure probability, the second failure probability, and the third failure probability includes: Summing the first failure probability, the second failure probability and the third failure probability to obtain the failure probability of the storage node to be tested; The failure probability of the storage node to be tested is compared with a preset probability threshold to determine whether the storage node to be tested fails.

6. The method according to claim 4, characterized in that The determining whether the storage node to be tested fails based on the first failure probability, the second failure probability, and the third failure probability includes: Summing the first failure probability, the second failure probability and the third failure probability to obtain the failure probability of the storage node to be tested; When the failure probability measured at least twice in succession is greater than a preset probability threshold, it is determined that the storage node to be tested fails.

7. A storage node fault detection device, characterized in that: The device comprises: A message sending module, used for sending heartbeat detection messages to a management node, a storage node and a client node in a distributed system respectively, so that the management node, the storage node and the client node perform heartbeat detection on the storage node to be detected, and the number of the management node, the storage node and the client node is multiple; A data receiving module, used to receive the first heartbeat detection result fed back by each of the management nodes, the second heartbeat detection result fed back by each of the storage nodes, and the third heartbeat detection result fed back by each of the client nodes; A failure probability determination module is used to determine a first failure probability of the storage node to be tested based on a first heartbeat detection result fed back by each of the management nodes, determine a second failure probability of the storage node to be tested based on a second heartbeat detection result fed back by each of the storage nodes, and determine a third failure probability of the storage node to be tested based on a third heartbeat detection result fed back by each of the client nodes; A fault judgment module is used to judge whether the storage node to be tested fails based on the first fault probability, the second fault probability and the third fault probability.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images