Heartbeat interaction method and device

Abstract

This specification provides a heartbeat interaction method and device, which relate to the field of communication technology. A heartbeat interaction method, applied to a slave node, includes: establishing a connection with a master node, obtaining a networked node list and an upper limit value of faulty nodes recorded in the master node; determining the number of heartbeat messages that it needs to send based on the upper limit value of the faulty nodes, and determining a detection object based on the number of heartbeat messages and the node list; sending a first heartbeat message to the detection object; if no heartbeat response from the detection object is received within the detection period, sending a fault notification to the master node, so that the master node determines the slave node that needs to be isolated based on the configured fault isolation period and the received fault notification. Through the above method, the efficiency of heartbeat interaction in networking can be improved.

Description

Technical Field

[0001] This specification relates to the field of communication technology, and in particular to a heartbeat interaction method and device. Background Art

[0002] With the development of network technology, users' requirements for networks and services are becoming increasingly stringent. Therefore, reliable service provision by various nodes is a key performance indicator in network technology. To monitor the operating status of associated nodes, nodes can exchange heartbeat messages. In a master-slave network deployment, there is a master node and several slave nodes. Slave nodes exchange heartbeat messages with each other. If a slave node determines that the heartbeat has timed out, it reports a failure to the master node.

[0003] In the above process, since heartbeat messages need to be sent between nodes, there are a large number of heartbeat messages in the network, which brings a heavy burden to network resources. Therefore, how to reduce the burden of heartbeat messages on the network is one of the problems that technicians in this field need to solve urgently. Summary of the Invention

[0004] To overcome the problems existing in the related art, this specification provides a heartbeat interaction method and device.

[0005] According to a first aspect of an embodiment of this specification, a heartbeat interaction method is provided, which is applied to a slave node, including:

[0006] Establish a connection with the master node and obtain the network node list and fault node upper limit recorded in the master node;

[0007] Determine the number of heartbeat messages that the fault node needs to send according to the upper limit value, and determine the detection object according to the number of heartbeat messages and the node list;

[0008] Sending a first heartbeat message to the detection object;

[0009] If no heartbeat response is received from the detection object within the detection period, a fault notification is sent to the master node, so that the master node determines the slave node that needs to be isolated based on the configured fault isolation period and the received fault notification.

[0010] Optionally, after sending the fault notification to the master node, the method further includes:

[0011] Receive an updated node list and an upper limit of faulty nodes sent by the master node;

[0012] The number of heartbeat messages that the node needs to send is determined according to the updated upper limit of the faulty node, and the detection object is determined according to the number of heartbeat messages and the updated node list.

[0013] Optionally, the number of heartbeat messages satisfies X+1≤Y<Z-1, the number of heartbeat messages is Y, the upper limit of faulty nodes is X, and the total number of nodes recorded in the node list is Z.

[0014] Optionally, determining the detection object according to the number of heartbeat messages and the node list includes:

[0015] Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer downward are determined as detection objects; or,

[0016] Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer upward are determined as detection objects; or,

[0017] Based on the node list, starting from the own node pointer and sequentially spaced by a preset number of slave nodes equal to the number of heartbeat messages are determined as detection objects.

[0018] Optionally, the master node is a control node in a distributed storage cluster, and the slave node is a storage node in the distributed storage cluster;

[0019] If the distributed storage cluster adopts the replica mode, the upper limit of the fault node is the number of stored replicas;

[0020] If the distributed storage cluster adopts the EC mode, the upper limit of the faulty nodes is the number of parity slices in the EC mode.

[0021] According to a second aspect of an embodiment of this specification, a heartbeat interaction method is provided, which is applied to a master node and includes:

[0022] Establish a connection with a slave node, and send a node list and a fault node upper limit of the network configured by itself to the slave node, so that the slave node determines the number of heartbeat messages of the first heartbeat message that the slave node needs to send according to the node list and the fault node upper limit;

[0023] Receive fault notifications sent from nodes;

[0024] If the number of fault notifications received for a slave node during the fault isolation period exceeds the isolation determination number, the slave nodes exceeding the isolation determination number are isolated;

[0025] If the fault notifications received for a slave node within the fault isolation period do not exceed the isolation determination quantity, a second heartbeat message is sent to the slave node, and the slave node is isolated when the second heartbeat message times out.

[0026] Optionally, the sending period of the second heartbeat message is determined based on the total number of nodes determined in the node list, wherein the relationship between the sending period T and the total number of nodes Z satisfies T=Q×Z+N, the adjustment parameter N is a positive integer not greater than 20, the weight value Q is a positive number not greater than 1, and the weight value Q decreases as the total number of nodes Z increases.

[0027] According to a third aspect of an embodiment of this specification, a heartbeat interaction device is provided, which is applied to a slave node and includes:

[0028] An acquisition unit, configured to establish a connection with a master node and acquire a networked node list and an upper limit of faulty nodes recorded in the master node;

[0029] A determining unit, configured to determine the number of heartbeat messages that the unit needs to send according to the upper limit value of the faulty node, and determine a detection object according to the number of heartbeat messages and the node list;

[0030] A sending unit, configured to send a first heartbeat message to the detection object;

[0031] The notification unit is used to send a fault notification to the master node if no heartbeat response is received from the detection object within the detection period, so that the master node determines the slave node that needs to be isolated based on the configured fault isolation period and the received fault notification.

[0032] Optionally, the device further includes:

[0033] An updating unit, configured to receive an updated node list and an upper limit of faulty nodes sent by the master node;

[0034] The determining unit is further configured to determine the number of heartbeat messages that it needs to send according to the updated upper limit of the faulty nodes, and to determine the detection object according to the number of heartbeat messages and the updated node list.

[0035] Optionally, the number of heartbeat messages satisfies X+1≤Y<Z-1, the number of heartbeat messages is Y, the upper limit of faulty nodes is X, and the total number of nodes recorded in the node list is Z.

[0036] Optionally, the determining unit is specifically configured to determine, based on the node list, slave nodes with the number of heartbeat messages starting from its own node pointer downward as detection objects; or,

[0037] Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer upward are determined as detection objects; or,

[0038] Based on the node list, starting from the own node pointer and sequentially spaced by a preset number of slave nodes equal to the number of heartbeat messages are determined as detection objects.

[0039] Optionally, the master node is a control node in a distributed storage cluster, and the slave node is a storage node in the distributed storage cluster;

[0040] If the distributed storage cluster adopts the replica mode, the upper limit of the fault node is the number of stored replicas;

[0041] If the distributed storage cluster adopts the EC mode, the upper limit of the faulty nodes is the number of parity slices in the EC mode.

[0042] According to a fourth aspect of an embodiment of this specification, a heartbeat interaction device is provided, which is applied to a master node and includes:

[0043] A sending unit is used to establish a connection with a slave node, and send a node list and a fault node upper limit of the network configured by itself to the slave node, so that the slave node determines the number of heartbeat messages of the first heartbeat message that the slave node needs to send according to the node list, the fault node upper limit and the total number of nodes;

[0044] A receiving unit, configured to receive a fault notification sent from a node;

[0045] An isolation unit is configured to isolate the slave nodes that exceed the isolation determination number if the fault notifications received for a slave node within a fault isolation period exceed the isolation determination number; and to send a second heartbeat message to the slave node if the fault notifications received for a slave node within the fault isolation period do not exceed the isolation determination number, and to isolate the slave node if the second heartbeat message times out.

[0046] Optionally, the sending period is obtained based on the total number of nodes determined in the node list, wherein the relationship between the sending period T and the total number of nodes Z satisfies T=Q×Z+N, the adjustment parameter N is a positive integer not greater than 20, the weight value Q is a positive number not greater than 1, and the weight value Q decreases as the total number of nodes Z increases.

[0047] According to the fifth aspect of the embodiments of this specification, an electronic device is provided, including a transceiver, a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to: implement the method steps executed by any of the above-mentioned master nodes or any of the above-mentioned slave nodes.

[0048] According to the sixth aspect of the embodiments of this specification, a machine-readable storage medium is provided, which stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions prompt the processor to: implement the method steps executed by any of the above-mentioned master nodes or any of the above-mentioned slave nodes.

[0049] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0050] In an embodiment of the present specification, the slave node determines the number of heartbeat messages and detection objects based on the networked node list and the upper limit of the faulty nodes obtained from the master node. The slave node determines whether it is necessary to report a fault notification based on the heartbeat message. When the master node receives the fault notification, it determines to isolate the determined faulty node based on the fault isolation period and the received fault notification, thereby reducing the number of heartbeat messages that the slave node needs to send and improving the efficiency of heartbeat interaction in the network.

[0051] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0053] Figure 1 This is a flowchart of a heartbeat interaction method involved in this application, which is applied to a slave node in a network;

[0054] Figure 2 This is a network structure diagram applicable to a heartbeat interaction method involved in this application;

[0055] Figure 3 This is a schematic diagram of a node list maintained by a master node in a heartbeat interaction method involved in this application;

[0056] Figure 4 This is a flowchart of a heartbeat interaction method involved in this application, which is applied to the master node of the network;

[0057] Figure 5 This is a network structure diagram applicable to a heartbeat interaction method involved in an embodiment of the present application;

[0058] Figure 6 This is a network structure diagram applicable to a heartbeat interaction method involved in an embodiment of the present application, wherein storage node 4 is confirmed to be faulty and storage node 1 is confirmed to be a false positive;

[0059] Figure 7This is a structural diagram of a heartbeat interaction device involved in this application, which is applied to a slave node in a network;

[0060] Figure 8 This is a structural diagram of a heartbeat interaction device involved in this application, which is applied to the master node of a network;

[0061] Figure 9 This is a structural diagram of an electronic device involved in this application. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0063] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0064] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0065] This application provides a heartbeat interaction method, such as Figure 1 As shown, it is applied to the slave node and includes:

[0066] S100: Establish a connection with a master node, and obtain a networked node list and an upper limit of faulty nodes recorded in the master node.

[0067] like Figure 2Taking the network shown as an example, it includes several nodes, each of which has a communication connection established between them. Among them, at least one node serves as a master node, and the other nodes serve as slave nodes. The slave nodes are used to provide forwarding, processing, or storage services to the outside world in the network, and the master node is used to manage the slave nodes in the network. The master node can pre-configure or form a list of nodes in the network and an upper limit of faulty nodes based on interaction. The node list can record information such as the pointer (index), identification, status, and number of slave nodes connected to the master node. The identification can include information such as the address information and name of the slave node. Optionally, the upper limit of faulty nodes is used to indicate the maximum number of slave nodes that the network can withstand. It can be configured based on the total number of slave nodes in the network or recorded in a pre-set manner. It can be set according to actual needs and is not restricted.

[0068] After the slave node establishes a connection with the master node, the slave node can obtain configuration information from the master node (ie, a list of connected nodes and an upper limit of faulty nodes maintained by the master node) and record it.

[0069] Furthermore, it should be noted that in some possible implementation scenarios, the nodes may be nodes deployed in a communication network, that is, the nodes may be network devices. For example, the master node may be a node in the communication network with control functions, such as an SDN controller in an SDN (Software Defined Network), and the slave node may be a node in the communication network with data processing and forwarding functions, such as an SDN switch in an SDN network.

[0070] In some possible implementation scenarios, nodes can be nodes deployed in a storage network, that is, nodes can be servers that carry storage, computing, or services. For example, the storage network can be a storage network in a distributed storage cluster, the master node can be a control node in the storage network that performs network configuration and node management functions, and the slave node can be a storage node in the storage network that carries data storage functions.

[0071] Of course, the specific implementation is not limited to the above two scenarios. This solution can be applied in any situation where at least one master node can manage several slave nodes in the network, and heartbeat messages can be exchanged between the master node and the slave nodes, as well as between the slave nodes.

[0072] S101. Determine the number of heartbeat messages that the node needs to send according to the upper limit of the faulty node, and determine the detection object according to the number of heartbeat messages and the node list.

[0073] To reduce the number of heartbeat messages exchanged within the network, a correlation can be established between the number of heartbeat messages and an upper limit on the number of faulty nodes. Using the determined number of heartbeat messages, each node selects a detection target from the slave nodes in the node list it obtains. The detection target refers to the detected slave node. In other words, the number of heartbeat messages refers to the number of detected slave nodes determined by the detecting slave node.

[0074] In a possible implementation, optionally, the number of heartbeat messages may satisfy X+1≤Y<Z-1, wherein the number of heartbeat messages is Y, the upper limit of faulty nodes is X, and the total number of nodes recorded in the node list is Z.

[0075] The above configuration is designed so that the detection of a slave node can at least cover the upper limit of faulty nodes set in the network, and so that the number of heartbeat messages sent by a slave node does not exceed the number of heartbeat messages sent to each slave node. Preferably, X+1 is less than (Z / 2)-1.

[0076] In another possible implementation, optionally, determining the detection object according to the number of heartbeat messages and the node list includes:

[0077] Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer downward are determined as detection objects; or,

[0078] Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer upward are determined as detection objects; or,

[0079] Based on the node list, starting from the own node pointer and sequentially spaced by a preset number of slave nodes equal to the number of heartbeat messages are determined as detection objects.

[0080] Node list, such as Figure 3 As shown, including Figure 2 In the five slave nodes in the network shown, from the perspective of one slave node, one of the slave nodes is itself, and the other four slave nodes are other slave nodes in the network.

[0081] Based on the node list, the node pointer determines its own position in the node list, and determines the detection object in a preset manner starting from this position.

[0082] For example, the slave node may start from its own position in the node list (node ​​pointer=1) and select two slave nodes upward as detection objects, namely, the two slave nodes with node pointers 2 and 3.

[0083] For another example, the slave node can start from its own position in the node list (node ​​pointer = 1) and select two slave nodes downward as detection objects. Since the slave node itself is the first slave node in the node list, it jumps to the end of the node list and selects in reverse order, that is, the two slave nodes with node pointers 5 and 4.

[0084] For example, a slave node can start from its own position in the node list (node ​​pointer = 1) and select two slave nodes at intervals of a certain number of nodes as detection targets. The selection direction can also be upward or downward according to the above selection direction. For example, if the selection direction is downward, the slave nodes with node pointers 3 and 5 can be selected as detection targets.

[0085] In another possible factual manner, in a storage network such as a data center to which the present application is applicable, optionally, the master node is a control node in a distributed storage cluster, and the slave node is a storage node in the distributed storage cluster.

[0086] In a distributed storage cluster, storage nodes handle file writing and reading, while control nodes manage the storage nodes within the cluster. Storage nodes can exchange heartbeat messages, which can be considered the aforementioned first heartbeat messages. The number of first heartbeat messages can be configured based on the storage mode used by the selected distributed file storage cluster.

[0087] If the distributed storage cluster adopts a replica mode, the upper limit of the faulty node is the number of stored replicas.

[0088] When deploying a distributed storage cluster, staff can configure the storage mode based on their needs. Replication mode involves storing N copies of a file in the distributed storage cluster: one file is called the original, and the other N-1 copies are called replicas. In replication mode, if both the original and replicas are damaged, the file will be lost. To prevent this, the distributed storage cluster must ensure that a maximum of N-1 slave node failures can occur, meaning the maximum number of failed nodes is N-1.

[0089] If the distributed storage cluster adopts the EC (Erasure Coding) mode, the upper limit of the faulty nodes is the number of parity slices in the EC mode.

[0090] When deploying a distributed storage cluster, if the storage mode is set to EC, a file is stored discretely across different storage nodes, forming M data shards, and L parity shards are set to verify the data shards. In EC mode, the sum of the number of valid data shards and parity shards in the distributed storage cluster must be no less than the number of data shards. If this number is less than the number of data shards, the file will become unreadable. Therefore, the upper limit of the number of failed nodes should be limited to no more than the number of parity shards.

[0091] Of course, the above selection method is only described by way of example and can be set according to actual needs without limitation. By the above method, when the slave node sends the first heartbeat message, the slave nodes in the network can be covered as evenly as possible, thereby improving the reliability of the heartbeat interaction.

[0092] S102: Send a first heartbeat message to the detection object.

[0093] After a slave node determines its own detection object, it can send a first heartbeat message to other slave nodes according to the identifier (address information) determined in the node list, and start a timer to wait for the heartbeat response of other slave nodes to the first heartbeat message. Wherein, the timer can also be set with a detection period, which is the period for waiting for the heartbeat response. When the detection period is exceeded and no heartbeat response is received, it can be considered that the heartbeat interaction has failed and the detected slave node may have a fault. When the detection period is not exceeded and a heartbeat response is received, it is considered that the detected slave node is in a normal state.

[0094] It should be noted that within a detection cycle, the slave node acting as the detecting party can send several first heartbeat messages to the slave node acting as the detected party. Only when no heartbeat response is received can it be determined that the detected slave node is faulty. The above method can avoid false detection caused by network delays and other reasons.

[0095] S103: If no heartbeat response is received from the detection object within the detection period, a fault notification is sent to the master node, so that the master node determines the slave node that needs to be isolated according to the configured fault isolation period and the received fault notification.

[0096] exist Figure 2 The solid arrows show the paths of the heartbeat messages exchanged between the slave nodes, and the dotted arrows show the paths of the fault notifications reported from the slave nodes to the master node. Figure 2 The example shows the path of a slave node reporting to the master node. This does not mean that other slave nodes will not report fault notifications to the master node, nor does it mean that only one slave node confirms that the slave node it detects has a fault.

[0097] When the heartbeat response of the detection object is not received within the detection cycle, the detected slave node is confirmed to be faulty. The slave node serving as the detection party sends a fault notification to the master node, which carries the identifier of the slave node that confirmed the fault. The master node can determine the slave node that needs subsequent processing based on the identifier.

[0098] After receiving the fault notification, the master node can determine the slave node that may have a fault based on the fault notification. The specific confirmation method is described on the master node side. When the master node determines that a slave node has a fault, it can isolate the slave node, that is, mark the status of the faulty slave node as isolated in the node list, and re-send the updated node list to the slave node. In addition, if the upper limit value of the faulty node is related to the number of nodes of the slave node, the upper limit value of the faulty node will be updated and sent based on the number of nodes determined in the updated node list. If the upper limit value of the faulty node is a preset value, it can be not sent, or the original preset value can be sent. You can choose according to actual needs without any restrictions.

[0099] Optionally, after step S103, sending the fault notification to the master node, the method further includes:

[0100] S104: Receive the updated node list and the upper limit of faulty nodes sent by the master node.

[0101] S105: Determine the number of heartbeat messages that the node needs to send according to the updated upper limit of the faulty node, and determine the detection object according to the number of heartbeat messages and the updated node list.

[0102] After receiving the updated node list and / or the upper limit of faulty nodes from the node, the slave node acting as the detection party can redetermine the number of heartbeat messages based on the updated node list and the upper limit of faulty nodes, and determine the detection object. The specific process is similar to the above step S101 and will not be described in detail.

[0103] Correspondingly, this application also provides a heartbeat interaction method, such as Figure 4 As shown, it is applied to the master node and includes:

[0104] S400, establish a connection with the slave node, and send the node list and fault node upper limit of the network configured by itself to the slave node, so that the slave node determines the number of heartbeat messages of the first heartbeat message that the slave node needs to send according to the node list and the fault node upper limit.

[0105] S401: Receive a fault notification sent by a slave node.

[0106] Step S400 and step S401 correspond to the above description on the slave node side and will not be described again.

[0107] S402: If the number of fault notifications received for a slave node during the fault isolation period exceeds the isolation determination number, isolate the slave nodes that exceed the isolation determination number.

[0108] A fault isolation period can be set on the master node. This period refers to the period for receiving fault notifications reported by slave nodes. The master node determines whether to isolate a slave node based on the currently received fault notification. This fault isolation period needs to be set to be greater than the detection period set on the slave node to avoid the master node making an incorrect judgment when receiving a small number of fault notifications reported by slave nodes. Preferably, the fault isolation period can be set to be greater than one detection period and less than two detection periods.

[0109] When performing isolation determination, the master node can confirm based on the set isolation determination number. When the number of fault notifications received is greater than the isolation determination number, the status of the slave node is marked as faulty and an isolation operation is performed on it.

[0110] S403: If the fault notifications received for a slave node within the fault isolation period do not exceed the isolation determination quantity, send a second heartbeat message to the slave node, and isolate the slave node when the second heartbeat message times out.

[0111] If, during the fault isolation period, the number of fault notifications received for a slave node does not exceed the isolation determination limit, the fault notification for that slave node may be a false alarm. In this case, the master node temporarily stops waiting to confirm that the slave node is not faulty and sends a second heartbeat message to the slave node, starting a timer with a set confirmation period. If, after the confirmation period expires, no heartbeat response is received from the slave node for the second heartbeat message, the master node is confirmed to be faulty and isolating the slave node.

[0112] Optionally, the sending period of the second heartbeat message is determined based on the total number of nodes determined in the node list, wherein the relationship between the sending period T and the total number of nodes Z satisfies T=Q×Z+N, the adjustment parameter N is a positive integer not greater than 20, the weight value Q is a positive number not greater than 1 and the weight value Q decreases as the total number of nodes Z increases.

[0113] That is to say, by setting the above weight values, the sending period will not be excessively extended due to the expansion of the scale of the distributed storage cluster, and the sending period is limited to less than one minute.

[0114] In addition, you can also set a longer active sending cycle (less than an hour). During the operation of the master node, periodically send a third heartbeat message corresponding to the active sending cycle, thereby supplementing and further improving the reliability of heartbeat interaction in the network.

[0115] A heartbeat interaction method of the present application is described below in conjunction with a specific implementation method, taking a distributed storage cluster as the networking, a copy mode as the storage mode, and storing files in three storage nodes as an example.

[0116] like Figure 5 As shown in FIG, 6 storage nodes and 1 control node are set in the distributed storage cluster. The upper limit of the fault nodes pre-configured on the control node is 3, and the isolation determination number is 2.

[0117] A heartbeat interaction method, comprising:

[0118] S1. Connections are established between the control node and the storage nodes, and between the storage nodes. The control node obtains node information of the storage nodes to form a node list.

[0119] The node information may include the address information and names of storage nodes 1 to 6, and the node list formed may include the address information, names, states, and node pointers of storage nodes 1 to 6. The node pointers are represented by numbers 1 to 6. In the initial stage, the state of each storage node is running.

[0120] S2. Storage nodes 1 to 6 interact with the control node to obtain and record the node list formed on the control node and the upper limit of fault nodes configured on the control node.

[0121] S3. The upper limit value of the faulty node of storage node 1 to storage node 6 determines the number of heartbeat messages that it needs to send, and selects several slave nodes from the node list as detection objects according to the number of heartbeat messages.

[0122] Storage node 1 selects three storage nodes from its own position downward in the node list as detection objects based on the upper limit of faulty nodes and the configured detection object selection strategy. That is, storage node 1 detects storage nodes 2-4, storage node 2 detects storage nodes 3-5, storage node 3 detects storage nodes 4-6, storage node 4 detects storage nodes 5, 6, and 1, storage node 5 detects storage nodes 6, 1, and 2, and storage node 6 detects storage nodes 1-3.

[0123] S4, storage node 1-storage node 6 sends a heartbeat message in the storage network of the distributed storage cluster. If a heartbeat response from the detection object is received within the detection period, step S4 is executed in a loop. If no heartbeat response from the detection object is received within the detection period, a fault notification is reported to the control node.

[0124] S5. The control node receives fault notifications in a cyclical fault isolation cycle and records the number of fault notifications reported by each storage node during each fault isolation cycle. If the number of recorded fault notifications exceeds the isolation determination number, the process proceeds to step S6. If the number of recorded fault notifications does not exceed the isolation determination number, the process proceeds to step S7. After the fault isolation cycle, the control node clears the recorded number of fault notifications and records them again, repeating the cycle.

[0125] like Figure 6 In the scenario shown, storage node 1, storage node 2, and storage node 3 all report a fault notification of storage node 4, and storage node 6 reports a fault notification of storage node 1.

[0126] S6: The control node updates the status of storage node 4 to isolated in the node list and sends the updated node list to other storage nodes to notify them of the failure of storage node 4. After receiving the updated node list, each storage node determines that storage node 4 is in a failed state and disconnects from storage node 4. The updated node list then overwrites the previous node list.

[0127] S7. During the fault isolation period, the control node received 1 fault notification for storage node 1, which does not exceed the isolation determination limit. Therefore, the control node cannot confirm that storage node 1 is faulty. The control node constructs its own heartbeat message, sends it to storage node 1, and waits for a heartbeat response. If a heartbeat response is received, storage node 1 is confirmed to be faulty, and the process ends. If no heartbeat response is received, storage node 1 is determined to be faulty, marked as faulty in the node list, and the updated node list is distributed to other storage nodes.

[0128] Afterwards, the control node receives the heartbeat response from storage node 1 and ends the heartbeat detection process for storage node 1.

[0129] In addition, since the storage nodes are different, step S6 and step S7 may be respectively executed in the same time period. Therefore, the execution process of step S6 and step S7 is not exclusive.

[0130] S8. After the control node sends the updated node list, the storage node adjusts the detection object according to the updated node list and jumps to step S4 to loop.

[0131] Through the above method, it is possible to reduce network resource usage caused by excessive heartbeat messages while maintaining the heartbeat detection effect.

[0132] Correspondingly, a heartbeat interaction device, such as Figure 7 As shown, it is applied to the slave node and includes:

[0133] An acquisition unit, configured to establish a connection with a master node and acquire a networked node list and an upper limit of faulty nodes recorded in the master node;

[0134] A determining unit, configured to determine the number of heartbeat messages that the unit needs to send according to the upper limit value of the faulty node, and determine a detection object according to the number of heartbeat messages and the node list;

[0135] A sending unit, configured to send a first heartbeat message to the detection object;

[0136] The notification unit is used to send a fault notification to the master node if no heartbeat response is received from the detection object within the detection period, so that the master node determines the slave node that needs to be isolated based on the configured fault isolation period and the received fault notification.

[0137] Optionally, the device further includes:

[0138] An updating unit, configured to receive an updated node list and an upper limit of faulty nodes sent by the master node;

[0139] The determining unit is further configured to determine the number of heartbeat messages that it needs to send according to the updated upper limit of the faulty nodes, and to determine the detection object according to the number of heartbeat messages and the updated node list.

[0140] Optionally, the number of heartbeat messages satisfies X+1≤Y<Z-1, the number of heartbeat messages is Y, the upper limit of faulty nodes is X, and the total number of nodes recorded in the node list is Z.

[0141] Optionally, the determining unit is specifically configured to determine, based on the node list, slave nodes with the number of heartbeat messages starting from its own node pointer downward as detection objects; or,

[0142] Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer upward are determined as detection objects; or,

[0143] Based on the node list, starting from the own node pointer and sequentially spaced by a preset number of slave nodes equal to the number of heartbeat messages are determined as detection objects.

[0144] Optionally, the master node is a control node in a distributed storage cluster, and the slave node is a storage node in the distributed storage cluster;

[0145] If the distributed storage cluster adopts the replica mode, the upper limit of the fault node is the number of stored replicas;

[0146] If the distributed storage cluster adopts the EC mode, the upper limit of the faulty nodes is the number of parity slices in the EC mode.

[0147] Correspondingly, the present application also provides a heartbeat interaction device, such as Figure 8 As shown, it is applied to the master node and includes:

[0148] A sending unit is used to establish a connection with a slave node, and send a node list and a fault node upper limit of the network configured by itself to the slave node, so that the slave node determines the number of heartbeat messages of the first heartbeat message that the slave node needs to send according to the node list, the fault node upper limit and the total number of nodes;

[0149] A receiving unit, configured to receive a fault notification sent from a node;

[0150] An isolation unit is configured to isolate the slave nodes that exceed the isolation determination number if the fault notifications received for a slave node within a fault isolation period exceed the isolation determination number; and to send a second heartbeat message to the slave node if the fault notifications received for a slave node within the fault isolation period do not exceed the isolation determination number, and to isolate the slave node if the second heartbeat message times out, wherein the sending period of the second heartbeat message is greater than the detection period of the slave node sending the first heartbeat message.

[0151] Optionally, the sending period is obtained based on the total number of nodes determined in the node list, wherein the relationship between the sending period T and the total number of nodes Z satisfies T=Q×Z+N, the adjustment parameter N is a positive integer not greater than 20, the weight value Q is a positive number not greater than 1 and the weight value Q decreases as the total number of nodes Z increases.

[0152] Correspondingly, the present application also provides an electronic device, such as Figure 9 As shown, it includes a transceiver, a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to: implement the method steps executed by any of the above-mentioned master nodes or any of the above-mentioned slave nodes.

[0153] Correspondingly, the present application also provides a machine-readable storage medium storing machine-executable instructions. When called and executed by a processor, the machine-executable instructions prompt the processor to: implement the method steps executed by any of the above-mentioned master nodes or any of the above-mentioned slave nodes.

[0154] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0155] In an embodiment of the present specification, the slave node determines the number of heartbeat messages and detection objects based on the networked node list and the upper limit of the faulty nodes obtained from the master node. The slave node determines whether it is necessary to report a fault notification based on the heartbeat message. When the master node receives the fault notification, it determines to isolate the determined faulty node based on the fault isolation period and the received fault notification, thereby reducing the number of heartbeat messages that the slave node needs to send and improving the efficiency of heartbeat interaction in the network.

[0156] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0157] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0158] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0159] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0160] Other embodiments of the present invention will readily occur to those skilled in the art upon consideration of the present invention and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present invention being indicated by the following claims.

[0161] It should be understood that the present description is not limited to the exact structure that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0162] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A heartbeat interaction method, characterized in that: Applied to slave nodes, including: Establish a connection with the master node and obtain the network node list and fault node upper limit recorded in the master node; Determine the number of heartbeat messages that the fault node needs to send according to the upper limit value, and determine the detection object according to the number of heartbeat messages and the node list; Sending a first heartbeat message to the detection object; If no heartbeat response is received from the detection object within the detection period, a fault notification is sent to the master node, so that the master node determines the slave node that needs to be isolated based on the configured fault isolation period and the received fault notification.

2. The method according to claim 1, characterized in that After sending the fault notification to the master node, the method further includes: Receive an updated node list and an upper limit of faulty nodes sent by the master node; The number of heartbeat messages that the node needs to send is determined according to the updated upper limit of the faulty node, and the detection object is determined according to the number of heartbeat messages and the updated node list.

3. The method according to claim 1 or 2, characterized in that The number of heartbeat messages satisfies X+1≤Y<Z-1, the number of heartbeat messages is Y, the upper limit of faulty nodes is X, and the total number of nodes recorded in the node list is Z.

4. The method according to claim 1 or 2, characterized in that The determining the detection object according to the number of the heartbeat messages and the node list includes: Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer downward are determined as detection objects; or, Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer upward are determined as detection objects; or, Based on the node list, starting from the own node pointer and sequentially spaced by a preset number of slave nodes equal to the number of heartbeat messages are determined as detection objects.

5. The method according to claim 1 or 2, characterized in that The master node is a control node in a distributed storage cluster, and the slave node is a storage node in a distributed storage cluster; If the distributed storage cluster adopts the replica mode, the upper limit of the fault node is the number of stored replicas; If the distributed storage cluster adopts the error correction code EC mode, the upper limit of the faulty nodes is the number of parity slices in the EC mode.

6. A heartbeat interaction method, characterized in that: Applied to the master node, including: Establish a connection with a slave node, and send a node list and a fault node upper limit of the network configured by itself to the slave node, so that the slave node determines the number of heartbeat messages of the first heartbeat message that the slave node needs to send according to the node list and the fault node upper limit; Receive fault notifications sent from nodes; If the number of fault notifications received for a slave node during the fault isolation period exceeds the isolation determination number, the slave nodes exceeding the isolation determination number are isolated; If the fault notifications received for a slave node within the fault isolation period do not exceed the isolation determination quantity, a second heartbeat message is sent to the slave node, and the slave node is isolated when the second heartbeat message times out.

7. The method according to claim 6, characterized in that The sending period of the second heartbeat message is determined based on the total number of nodes determined in the node list, wherein the relationship between the sending period T and the total number of nodes Z satisfies T=Q×Z+N, the adjustment parameter N is a positive integer not greater than 20, the weight value Q is a positive number not greater than 1, and the weight value Q decreases as the total number of nodes Z increases.

8. A heartbeat interactive device, characterized in that: Applied to slave nodes, including: An acquisition unit, configured to establish a connection with a master node and acquire a networked node list and an upper limit of faulty nodes recorded in the master node; A determining unit, configured to determine the number of heartbeat messages that the unit needs to send according to the upper limit value of the faulty node, and determine a detection object according to the number of heartbeat messages and the node list; A sending unit, configured to send a first heartbeat message to the detection object; The notification unit is used to send a fault notification to the master node if no heartbeat response is received from the detection object within the detection period, so that the master node determines the slave node that needs to be isolated based on the configured fault isolation period and the received fault notification.

9. The device according to claim 8, characterized in that Also includes: An updating unit, configured to receive an updated node list and an upper limit of faulty nodes sent by the master node; The determining unit is further configured to determine the number of heartbeat messages that it needs to send according to the updated upper limit of the faulty nodes, and to determine the detection object according to the number of heartbeat messages and the updated node list.

10. The device according to claim 8 or 9, characterized in that The number of heartbeat messages satisfies X+1≤Y<Z-1, the number of heartbeat messages is Y, the upper limit of faulty nodes is X, and the total number of nodes recorded in the node list is Z.

11. The device according to claim 8 or 9, characterized in that The determining unit is specifically configured to determine, based on the node list, from its own node pointer downwards, the number of slave nodes with the heartbeat messages as detection objects; or, Based on the node list, the slave nodes with the number of heartbeat messages starting from the own node pointer upward are determined as detection objects; or, Based on the node list, starting from the own node pointer and sequentially spaced by a preset number of slave nodes equal to the number of heartbeat messages are determined as detection objects.

12. The device according to claim 8 or 9, characterized in that The master node is a control node in a distributed storage cluster, and the slave node is a storage node in a distributed storage cluster; If the distributed storage cluster adopts the replica mode, the upper limit of the fault node is the number of stored replicas; If the distributed storage cluster adopts the EC mode, the upper limit of the faulty nodes is the number of parity slices in the EC mode.

13. A heartbeat interactive device, characterized in that: Applied to the master node, including: A sending unit is used to establish a connection with a slave node, and send a node list and a fault node upper limit of the network configured by itself to the slave node, so that the slave node determines the number of heartbeat messages of the first heartbeat message that the slave node needs to send according to the node list, the fault node upper limit and the total number of nodes; A receiving unit, configured to receive a fault notification sent from a node; An isolation unit is configured to isolate the slave nodes that exceed the isolation determination number if the fault notifications received for a slave node within a fault isolation period exceed the isolation determination number; and to send a second heartbeat message to the slave node if the fault notifications received for a slave node within the fault isolation period do not exceed the isolation determination number, and to isolate the slave node if the second heartbeat message times out.

14. The device according to claim 13, characterized in that The sending period of the second heartbeat message is obtained based on the total number of nodes determined in the node list, wherein the relationship between the sending period T and the total number of nodes Z satisfies T=Q×Z+N, the adjustment parameter N is a positive integer not greater than 20, the weight value Q is a positive number not greater than 1, and the weight value Q decreases as the total number of nodes Z increases.

15. An electronic device, characterized in that: The invention comprises a transceiver, a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to implement the method steps described in any one of claims 1 to 5 or any one of claims 6 to 7.

16. A machine-readable storage medium, characterized in that Machine executable instructions are stored, and when called and executed by a processor, the machine executable instructions prompt the processor to implement the method steps described in any one of claims 1 to 5 or any one of claims 6 to 7.

Images