Methods for restarting nodes in a centralized storage cluster

Abstract

This application discloses a node restart method for a centralized storage cluster, relating to the field of centralized storage cluster technology. The method includes: during node restart, when the node status indicates the existence of a restored first communication link and an unrestored second communication link, a timer is started. Before the timer expires, if any second communication link remains unrestored, a first message is sent through the restored communication link. If all second communication links have been restored, a second message is sent through the currently restored communication link. After the timer expires, if any second communication link remains unrestored, a second message is sent through the currently restored communication link. Finally, the node restart result is determined based on the second message. This method solves the problem of node mis-kickout caused by differences in the recovery times of different communication links, avoids cluster view oscillations, and significantly improves cluster stability and reliability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of centralized storage cluster, and particularly relates to a node restart method of centralized storage cluster. BACKGROUND

[0002] At present, the centralized storage cluster is composed of multiple nodes, and full connection state needs to be maintained between the nodes, that is, a communication link exists between any two nodes.

[0003] In the related art, in order to balance the cost, distance and bandwidth, multiple types of communication links are used in the cluster. When a node is restarted, some communication links of the node have been restored, and other communication links have not been restored. In order to meet the full connection requirement of the centralized storage cluster, the centralized storage cluster will kick out the node, which leads to poor stability of the centralized storage cluster. SUMMARY

[0004] The present application provides a node restart method of centralized storage cluster, which at least solves the problem of poor stability of the centralized storage cluster in the related art.

[0005] The present application provides a node restart method of centralized storage cluster, which comprises the following steps:

[0006] In the process of node restart, the communication link corresponding to the node is acquired;

[0007] When the node state indicates that the node has a restored first communication link and a non-restored second communication link, a timer corresponding to the node is started;

[0008] Before the timer expires, if there is a second communication link that has not been restored, a first message is sent based on the first communication link and the currently restored second communication link, the first message indicating that the node is not ready to join the cluster; if all second communication links have been restored, a second message is sent based on the first communication link and the second communication link, and the timer is canceled, the second message indicating that the node is ready to join the cluster;

[0009] After the timer expires, if there is a second communication link that has not been restored, the second message is sent based on the first communication link and the second communication link that has been restored in the timer counting process;

[0010] Based on the sent second message, the restart result of the node is determined.

[0011] The present application also provides a node restart device of centralized storage cluster, which comprises:

[0012] The acquisition module is configured to acquire the communication link corresponding to the node in the process of node restart.

[0013] The first determining module is configured to determine a node state of the node based on the communication link corresponding to the node, the node state being used to indicate a recovery situation of the communication link of the node.

[0014] The processing module is configured to start a timer corresponding to the node when the node state indicates that there is a first communication link that has been recovered and a second communication link that has not been recovered.

[0015] The first sending module is configured to send a first message based on the first communication link and the second communication link that has been recovered before the timer expires, the first message indicating that the node is not ready to join the cluster if there is a second communication link that has not been recovered, and to send a second message based on the first communication link and the second communication link and cancel the timer if all the second communication links have been recovered, the second message indicating that the node is ready to join the cluster.

[0016] The second sending module is configured to send the second message based on the first communication link and the second communication link that has been recovered during the counting process of the timer if there is a second communication link that has not been recovered after the timer expires.

[0017] The second determining module is configured to determine a restart result of the node based on the second message that has been sent.

[0018] The application further provides an electronic device, including a memory configured to store a computer program and a processor configured to execute the computer program to implement the steps of the node restart method of the centralized storage cluster.

[0019] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of the node restart method of the centralized storage cluster.

[0020] The application further provides a computer program product, including a computer program, and the computer program is executed by a processor to implement the steps of the node restart method of the centralized storage cluster.

[0021] The application intelligently manages the states of the communication links in different recovery stages during the node restart process and introduces a timer to coordinate the waiting and state synchronization mechanism, so that the problem of frequent oscillation of the cluster member view caused by the recovery time difference of different communication links can be solved, and the stability and reliability of the cluster are improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0023] Figure 1 A structural schematic diagram of any node of a centralized storage cluster provided by the embodiments of the present application is shown in FIG. 1.

[0024] Figure 2 A flowchart of a node restart method of a centralized storage cluster provided by the embodiments of the present application is shown in FIG. 2.

[0025] Figure 3 A structural schematic diagram of each node in a full connection state provided by the embodiments of the present application is shown in FIG. 3.

[0026] Figure 4 A flowchart of a timer starting method provided by the embodiments of the present application is shown in FIG. 4.

[0027] Figure 5 A flowchart of a node restart result determination method provided by the embodiments of the present application is shown in FIG. 5.

[0028] Figure 6 A connection topology view of a restarted cluster provided by the embodiments of the present application is shown in FIG. 6.

[0029] Figure 7 A structural schematic diagram of a node restart device of a centralized storage cluster provided by the embodiments of the present application is shown in FIG. 7.

[0030] Figure 8 A structural schematic diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present application.

[0032] It should be noted that in the description of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or inherent to such process, method, article or apparatus. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0033] Currently, a centralized storage cluster is composed of multiple nodes, and full connection state needs to be maintained between nodes, that is, a communication link exists between any two nodes.

[0034] In the related art, in order to balance the cost, distance and bandwidth, multiple types of communication links are used in the cluster. When a node is restarted, some communication links of the node have been restored, and other communication links have not been restored. In order to meet the full connection requirement of the centralized storage cluster, the centralized storage cluster will kick out the node from the cluster, resulting in poor stability of the centralized storage cluster.

[0035] In view of this, the embodiments of the present application provide a node restart method of a centralized storage cluster. In the process of node restart, the communication links corresponding to the node are obtained. Based on the communication links corresponding to the node, the current node state of the node is determined, and the node state is used to indicate the restoration of the communication links of the node. When the node state indicates that there are restored first communication links and un-restored second communication links, the timer corresponding to the node is started. Before the timer expires, if there is a second communication link that has not been restored, a first message is sent based on the first communication link and the currently restored second communication link, and the first message indicates that the node is not ready to join the cluster. If all the second communication links have been restored, a second message is sent based on the first communication link and the second communication link, and the timer is canceled, and the second message indicates that the node is ready to join the cluster. After the timer expires, if there is a second communication link that has not been restored, a second message is sent based on the first communication link and the second communication link that has been restored during the timer counting process. Based on the sent second message, the restart result of the node is determined. Through the above method, the cluster view shock problem caused by the difference in restoration time of different communication links in the node restart process of the centralized storage cluster can be effectively avoided, and the stability and reliability of the cluster are significantly improved.

[0036] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] The specific application environment architecture or the specific hardware architecture on which the execution of the node restart method of the centralized storage cluster depends is described herein. Reference is made to Figure 1 , Figure 1 A structure diagram of any node of a centralized storage cluster is provided in an embodiment of the present application, and the node comprises a cluster management module, a node information storage module, a power backup battery, a link recovery module, and a restart recovery management module, wherein the restart recovery management module comprises a heartbeat module and a timer.

[0038] The cluster management module is configured to manage nodes that have a communication link with the node.

[0039] The node information storage module is configured to store the communication link corresponding to the node and the type of the communication link.

[0040] The power backup battery is configured to supply power to the node information storage module in the node when the node is abnormally powered off, so that the communication link corresponding to the node can be successfully stored.

[0041] The link recovery module is configured to recover the original network configuration of the node during the node restart process.

[0042] The heartbeat module is configured to send a first message or a second message during the node restart process.

[0043] The timer is configured to start timing during the node restart process, and provide a protected waiting time window for the recovery of the unrecovered communication link.

[0044] Figure 2 A flowchart of a node restart method of a centralized storage cluster is provided in an embodiment of the present application, as shown in Figure 2 The embodiments of the present application provide a node restart method of a centralized storage cluster, and the method is described in detail as follows:

[0045] S201, during the node restart process, the communication link corresponding to the node is obtained.

[0046] In the embodiments of the present application, the execution subject is taken as an example of any node in the centralized storage cluster. The nodes in the centralized storage cluster are equal in function and processing logic. Therefore, the node restart method and the flow described in the embodiments are also applicable to any other node in the cluster, and the implementation principle and effects are the same. Therefore, the other nodes are not repeated here.

[0047] In some embodiments, before the node restart, the method further comprises:

[0048] During the normal operation of the node, the node information storage module in the node stores the communication link corresponding to the node.

[0049] When the node is powered off abnormally, the power preservation battery in the node supplies power for the node information preservation module in the node, so that the communication link corresponding to the node can be successfully stored.

[0050] In some embodiments, the communication link corresponding to the node is acquired by the node information preservation module in the node.

[0051] The communication link refers to a channel for communication connection between nodes.

[0052] When the cluster includes N nodes, for any node in the cluster, the communication link corresponding to the node is N-1, so as to ensure that each node in the cluster is in a full connection state, where N is an integer greater than or equal to 2.

[0053] For example, by Figure 3 For example, when the number of nodes in the cluster is 3, each node in the cluster is in a full connection state. Figure 3 A structural diagram of the full connection state of each node is provided for the embodiments of the present application, as shown in Figure 3 The nodes in the cluster include node A, node B and node C, wherein node A is connected with node B and node C through corresponding communication links respectively, and node B is connected with node C through a corresponding communication link.

[0054] In S202, the node state of the node is determined based on the communication link corresponding to the node, where the node state is used to indicate the recovery of the communication link of the node.

[0055] In some embodiments, before the node state of the node is determined based on the communication link corresponding to the node, the method can further include:

[0056] The link recovery module in the node recovers the original network configuration of the node.

[0057] The connection of the communication link corresponding to the node is recovered based on the communication link corresponding to the node.

[0058] The connection of the communication link corresponding to the node is recovered by starting to attempt to recover the connection of the communication link corresponding to the node.

[0059] The original network configuration of the node includes but is not limited to the IP address of the node, the address resolution protocol (ARP) table item, the routing table information, the Subnet Manager convergence state, and the protocol interaction state in the InfiniBand (IB) network.

[0060] In some embodiments, the node state of the node is determined based on the communication link corresponding to the node, including:

[0061] detecting the connection state of the communication link corresponding to each node one by one, determining whether the communication link corresponding to each node has been successfully recovered;

[0062] determining the current node state of the node based on whether the communication link corresponding to the node has been successfully recovered.

[0063] The node state indicates the recovery of the communication link of the node, which can be understood as indicating which communication links in the multiple communication links of the node have been recovered and which have not been recovered. For example, when the cluster includes N nodes, for any node in the cluster, the node needs to detect N-1 communication links between the node and other N-1 nodes. For each link, the recovery can be divided into two states: “recovered” or “not recovered”. “Recovered” means that the communication link has been successfully recovered and can normally transmit data and communicate; “not recovered” means that the communication link is still in the state of interruption, initialization or connection failure and cannot be used for communication.

[0064] S203, when the node state indicates that there is a first communication link that has been recovered and a second communication link that has not been recovered, starting a timer corresponding to the node.

[0065] In the embodiments of the present application, the node state is a dynamic information that is continuously updated with the advancement of the recovery process of the background communication link until the communication link of the node is recovered or the timer is timed out.

[0066] In the following, the processing procedure when the node state indicates other recovery conditions of the communication link of the node is described through Example 1A and Example 1B:

[0067] Example 1A, when the node state indicates that there is no recovered communication link of the node, waiting for a second time length; if there is a recovered communication link within the second time length, starting the timer corresponding to the node; if there is no recovered communication link within the second time length, stopping the starting of the timer corresponding to the node.

[0068] That is, when the first communication link of the node corresponding to the communication link is recovered after waiting for the second time length, the timer corresponding to the node is started; when the communication link of the node corresponding to the communication link is still not recovered after waiting for the second time length, the timer corresponding to the node is stopped.

[0069] Example 1B, when the node state indicates that the communication link corresponding to the node has been recovered, sending a second message based on the communication link corresponding to the node.

[0070] That is, when the communication link corresponding to the node has been recovered, the second message is sent based on the current recovered communication link (at this time, the current recovered communication link is the entire communication link corresponding to the node).

[0071] In some embodiments, the second time length is a preset configurable time window for waiting for initial recovery of at least one communication link. The second time length is set in consideration of the recovery time length of the communication link, so as to avoid triggering the abnormal processing too early due to slow recovery of the communication link.

[0072] In some embodiments, after performing example 1A, if the timer corresponding to the starting node is started, S204 is performed; if the timer corresponding to the starting node is stopped, it is determined that the restart result of the node is restart failure.

[0073] In some embodiments, after performing example 1B, step S208 is directly performed.

[0074] S204, before the timer expires, if there is a second communication link that has not been recovered, a first message is sent based on the first communication link and the second communication link that has been recovered, wherein the first message indicates that the node is not ready to join the cluster; if all the second communication links have been recovered, a second message is sent based on the first communication link and the second communication link, and the timer is canceled, wherein the second message indicates that the node is ready to join the cluster.

[0075] The first message and the second message are periodically sent packets. If the sending interval time length of the first message and the second message is a first time length, after the timer is started and before the timer expires, every interval of the first time length, it is determined whether there is a second communication link that has not been recovered, and then if there is a second communication link that has not been recovered, the first message is sent based on the first communication link and the second communication link that has been recovered, and if all the second communication links have been recovered (i.e. there is no second communication link that has not been recovered), the second message is sent based on the first communication link and the second communication link, and the timer is canceled.

[0076] Further, if the timer is canceled, S206 is directly performed.

[0077] Before the timer expires, when there is a second communication link that has not been recovered, the first message is continuously sent at a millisecond level interval; after all the second communication links have been recovered or the timer expires, the second message is continuously sent at a millisecond level interval. That is, the first time length can be at a millisecond level.

[0078] In some embodiments, the first message and the second message can be heartbeat packets. The heartbeat packet includes a node state flag bit, and the node state flag bit of the heartbeat packet indicates whether the node is ready to join the cluster. For example, when the node state flag bit of the heartbeat packet is a first value, it indicates that the node is not ready to join the cluster, and the first value is 0, for example. When the node state flag bit of the heartbeat packet is a second value, it indicates that the node is ready to join the cluster, and the second value is 1, for example.

[0079] In some embodiments, the first message is sent based on the first communication link and the currently recovered second communication link by the link heartbeat module.

[0080] Specifically, before the timer expires, the first message is sent based on the first communication link and the currently recovered second communication link by the node to the other node corresponding to the first communication link and the currently recovered second communication link.

[0081] Before the timer expires, if there is a second communication link that has not been recovered, the first message is sent based on the first communication link and the currently recovered second communication link, which means that the first message is sent based on the first communication link if all the second communication links have not been recovered.

[0082] Hereinafter, the process of sending the first message based on the first communication link by the node is exemplarily described.

[0083] For example, the cluster includes node A, node B and node C, and during the restart process of node A, the communication link between node A and node B is recovered first, while the communication link between node A and node C has not been recovered. At this time, the node state of node A indicates that there is one recovered communication link (i.e. the communication link between node A and node B) and one unrecovered communication link (i.e. the communication link between node A and node C). Node A starts the timer corresponding to node A, and before the timer expires, node A periodically sends the first message (i.e. the heartbeat packet carrying the specific node state flag) to node B through the communication link between node A and node B. The first message explicitly indicates to node B that node A currently detects that all the unrecovered communication links, and therefore node A is not ready to join the cluster.

[0084] Canceling the timer means that the node actively terminates the countdown running of the timer corresponding to the node, and releases the resources thereof.

[0085] It should be noted that the process of sending the first message based on the first communication link and the currently recovered second communication link is similar to the process of sending the first message based on the first communication link, which will not be described herein.

[0086] S205, after the timer expires, if there is a second communication link that has not been recovered, the second message is sent based on the first communication link and the second communication link that has been recovered during the timing process of the timer.

[0087] After the timer expires, if there is a second communication link that has not been recovered, it means that all the second communication links have not been recovered or part of the second communication links have not been recovered.

[0088] After the timer expires, the timer stops timing.

[0089] The second communication link recovered during the timer counting process can be understood as: the second communication link recovered during the process from the start of the timer to the stop of the timer counting.

[0090] It should be noted that the sending of the second message based on the first communication link and the second communication link recovered during the timer counting process is similar to the process of sending the first message based on the first communication link, which will not be described here.

[0091] In some embodiments, the node determines the restart result of the node based on the sent second message.

[0092] In some embodiments, the node determines the restart result of the node based on the sent second message.

[0093] In some embodiments, each node in the cluster receives the second message, and then the cluster re-determines the connection topology view of the cluster based on each node in the cluster, specifically, the state of the node in the connection topology view of the cluster is updated to re-determine the connection topology view of the cluster. For example, the state of the node is changed from a state such as “not ready” to a “ready” state.

[0094] The node not ready can be understood as the node not ready to join the cluster, and the ready can be understood as the node ready to join the cluster.

[0095] Further, each node in the cluster updates the connection topology view of the cluster maintained locally by each node, that is, the node completes the joining of the cluster.

[0096] In Figure 2In the illustrated embodiment, in the process of node restart, the communication link corresponding to the node is acquired; based on the communication link corresponding to the node, the current node state of the node is determined, the node state is used to indicate the recovery situation of the communication link of the node; when the node state indicates that there is a first communication link that has been recovered and a second communication link that has not been recovered, the timer corresponding to the node is started; before the timer expires, if there is a second communication link that has not been recovered, a first message is sent based on the first communication link and the current recovered second communication link, the first message indicating that the node is not ready to join the cluster; if all the second communication links have been recovered, a second message is sent based on the first communication link and the second communication link and the timer is cancelled, the second message indicating that the node is ready to join the cluster; after the timer expires, if there is a second communication link that has not been recovered, a second message is sent based on the first communication link and the second communication link that has been recovered in the timer counting process; based on the sent second message, the restart result of the node is determined. Through the above method, the cluster view shock problem caused by the recovery time difference of different types of communication links in the node restart process of the centralized storage cluster (i.e. in order to meet the full connection requirement of the centralized storage cluster, the centralized storage cluster will kick out the node) can be effectively avoided, and the stability and reliability of the cluster are significantly improved. Specifically, by sending the first message or the second message based on whether there is a second communication link that has not been recovered before the timer expires, it is realized that the node announces to join the cluster only after the communication link is fully recovered, by sending the second message after the timer expires, it is realized that the node announces to join the cluster after part of the communication link times out, which prevents the node from being mistakenly kicked out or repeatedly joined into the cluster when part of the communication link is not ready, thereby reducing the frequent changes of the connection topology graph of the cluster. At the same time, the node actively sends the first message (not ready) and the second message (ready) to the other nodes in the cluster, which realizes clear state synchronization with the other nodes in the cluster, so that the cluster can smoothly perceive and process the node restart process, and avoid the frequent changes of the connection topology graph of the cluster caused by the node state misjudgment of the node.

[0097] On the basis of any one of the above embodiments, below, in combination with Figure 4 The process of starting the timer corresponding to the node when the node state indicates that there is a first communication link that has been recovered and a second communication link that has not been recovered is described in detail.

[0098] Figure 4 A flowchart of a timer starting method provided by an embodiment of the present application is shown. Please refer to Figure 4 The method can include:

[0099] S401, when the node state indicates that the first communication link of the node has been recovered and the other communication links have not been recovered, the first duration of the timer corresponding to the node is determined.

[0100] The first communication link is a first communication link, and the other communication links are second communication links.

[0101] In some embodiments, determining the first duration of the timer corresponding to the node comprises:

[0102] Obtaining a type of the second communication link;

[0103] Determining the first duration based on the type of the second communication link.

[0104] The type of the communication link includes, but is not limited to, Non-Transparent Bridge (NTB), RDMA over Converged Ethernet (RoCE), and Fibre Channel (FC).

[0105] It should be noted that the method of obtaining the type of the communication link is similar to the method of obtaining the communication link corresponding to the node, and the storage of the type of the communication link before the node restarts is also similar to the storage of the communication link corresponding to the node, which will not be repeated here.

[0106] In some embodiments, determining the first duration based on the type of the second communication link comprises:

[0107] When the types of the second communication links are the same type, determining a recovery duration corresponding to the type; and determining the first duration based on the recovery duration corresponding to the type, in which case the first duration is greater than the recovery duration corresponding to the type.

[0108] When the types of the second communication links are multiple types, determining a recovery duration corresponding to each type; determining a maximum recovery duration among the recovery durations corresponding to the types; and determining the first duration based on the maximum recovery duration, in which case the first duration is greater than the maximum recovery duration.

[0109] The recovery duration corresponding to the type can be understood as the duration required for the communication link of the type to start recovery to successful recovery.

[0110] Optionally, determining the first duration based on the recovery duration corresponding to the type can be that a product of the recovery duration corresponding to the type and a preset safety factor is determined as the first duration, where the preset safety factor is greater than 1. For example, the preset safety factor can be 1.2.

[0111] Optionally, determining the first duration based on the recovery duration corresponding to the type can also be that a sum of the recovery duration corresponding to the type and a preset buffer time is determined as the first duration. For example, the preset buffer time can be 100 milliseconds.

[0112] It should be noted that the process of determining the first duration based on the maximum recovery duration is similar to the process of determining the first duration based on the recovery duration corresponding to the type, and will not be described here.

[0113] In some embodiments, the type of the communication link corresponds to the recovery duration one-to-one, and the node can query the recovery duration corresponding to the type in the pre-stored first correspondence relationship based on the type of the communication link. The first correspondence relationship includes a plurality of types of communication links, a plurality of recovery durations, and a correspondence relationship between the plurality of types of communication links and the plurality of recovery durations. For example, the first correspondence relationship is shown in Table 1 as follows.

[0114] Table 1 First correspondence relationship

[0115]

[0116] For example, when the type of the second communication link includes NTB and RoCE, based on Table 1, the recovery duration corresponding to NTB is 500 milliseconds, the recovery duration corresponding to RoCE is 1000 milliseconds, the maximum recovery duration is 1000 milliseconds, and further, the first duration can be 1200 milliseconds.

[0117] S402, based on the first duration, starting the timer corresponding to the node.

[0118] In some embodiments, based on the first duration, starting the timer corresponding to the node, includes:

[0119] configuring the first duration as the timeout threshold of the timer corresponding to the node;

[0120] starting the timer corresponding to the node.

[0121] After starting the timer corresponding to the node, the timer starts timing. The timeout of the timer can be understood as: the timer starts running continuously from the start, and the cumulative timing value of the timer is greater than the timeout threshold of the timer (i.e. the first duration).

[0122] After canceling the timer, the cumulative timing value of the timer is cleared, and the timeout threshold of the timer returns to the initial value (usually zero or an invalid placeholder value).

[0123] In Figure 4In the illustrated embodiment, when the node state indicates that the first communication link of the node has been restored and other communication links have not been restored, the first duration of the timer corresponding to the node is determined; and the timer corresponding to the node is started based on the first duration. Through the above method, the first duration of the timer is dynamically determined and set according to the type of the communication link that has not been restored, which can provide differentiated waiting time for the restoration of different types of communication links, effectively avoid the premature or late timeout problem that may be caused by the fixed timeout threshold of the timer, ensure that the communication link corresponding to the node has sufficient restoration time, and prevent unnecessary long waiting, thereby achieving the optimal balance between efficiency and reliability of the restart process in a complex network environment, and further improving the intelligence and adaptability of the node restart of the centralized storage cluster.

[0124] On the basis of any one of the above embodiments, below, in combination with Figure 5 The process of determining the restart result of the node based on the sent second message is described in detail.

[0125] Figure 5 A flowchart of a method for determining a restart result of a node according to an embodiment of the present application is provided. Please refer to Figure 5 The method can include:

[0126] S501, based on the sent second message, a third message is sent to the cluster management module in the node, wherein the third message indicates that the node receiving the second message is added in the cluster management module.

[0127] Below, at the interaction level of each node in the cluster, based on the sent second message, a third message is sent to the cluster management module in the node, which can be understood as:

[0128] The node receives the second message sent by other nodes based on the restored communication link, and after receiving the second message sent by other nodes, the node sends a third message to the cluster management module in the node, and the third message indicates that the other node sending the second message is added in the cluster management module of the node.

[0129] Correspondingly, the node sends a second message to other nodes based on the restored communication link, and after receiving the second message sent by the node, each of the other nodes sends a third message to the cluster management module in the other node, and the third message indicates that the node is added in the cluster management module of the other node.

[0130] Among them, the restored communication link includes the first communication link and the currently restored second communication link.

[0131] Through the symmetric message interaction mechanism, the nodes in the cluster can perceive and confirm the readiness state of each other, thereby cooperatively updating the local connection topology view of the cluster of each node, and finally realizing consistent update of the cluster membership.

[0132] S502, generating a connection topology view of the cluster based on information in the cluster management module.

[0133] In some embodiments, generating a connection topology view of the cluster based on information in the cluster management module comprises:

[0134] Based on the information in the cluster management module, determining the edges between the node and other nodes in the cluster, the edges being used to indicate that the node is communicatively connected with the other nodes;

[0135] Based on the edges between the node and the other nodes, generating a connection topology view of the cluster.

[0136] In the following, by way of example 2A and example 2B, when the nodes in the cluster include node A, node B and node C, taking node A as the restarted node as an example, the generation of the connection topology view of the cluster based on the information in the cluster management module is exemplarily described:

[0137] In example 2A, at the node A level, generating a connection topology view of the cluster based on information in the cluster management module can be understood as:

[0138] If the communication link between node A and node B and the communication link between node A and node C are both restored, the information in the cluster management module in node A includes node B and node C;

[0139] Based on the information in the cluster management module in node A, connecting node A and node B in the original connection topology view of the cluster to generate a connection topology view of the cluster;

[0140] The original connection topology view of the cluster is the original locally stored connection topology view of the cluster in node A.

[0141] In example 2B, at the interaction level of each node in the cluster, generating a connection topology view of the cluster based on information in the cluster management module can be understood as:

[0142] If the communication link between node A and node B and the communication link between node A and node C are both restored, the information in the cluster management module in node A includes node B and node C, the information in the cluster management module in node B includes node A and node C, and the information in the cluster management module in node C includes node A and node B;

[0143] Based on the information in the cluster management modules of the node A, the node B and the node C, the node A is connected with the node B, the node A is connected with the node C, and the node B is connected with the node C, a connection topology view of the cluster is generated.

[0144] S503, based on the connection topology view of the cluster, determining the restart result of the node.

[0145] In some embodiments, based on the connection topology view of the cluster, determining the restart result of the node comprises:

[0146] In the connection topology view of the cluster, determining whether there is an edge between the node and other nodes in the cluster;

[0147] If there is an edge between the node and other nodes in the cluster, it is determined that the restart result of the node is restart success;

[0148] If there is no edge between the node and any one of the nodes in the cluster, it is determined that the restart result of the node is restart failure;

[0149] When the restart result is restart failure, based on the connection topology view of the cluster, the number of nodes without an edge with the node is obtained;

[0150] When the number of nodes without an edge with the node is greater than or equal to a first threshold, a first request message is sent, and the first request message is used to request to disconnect the node in the cluster;

[0151] When the number of nodes without an edge with the node is less than the first threshold, a second request message is sent, and the second request message is used to request to disconnect the nodes without an edge with the node in the cluster, or used to request to disconnect the node in the cluster.

[0152] In some embodiments, determining whether there is an edge between the node and other nodes in the cluster means: traversing to query whether there is an edge with the node as an endpoint and any one of the other nodes in the cluster as the other end in the connection topology view. The existence of each edge represents that there is a restored communication link between the two end nodes.

[0153] If there is an edge between the node and other nodes in the cluster except the node, it indicates that the node has been fully connected with the cluster, and can normally participate in the activities of the cluster, so it is determined that the node restarts successfully.

[0154] If there is no edge between the node and any one of the other nodes in the cluster except the node, it indicates that the node fails to establish an effective connection with the other node, so it is determined that the node restarts failure.

[0155] In case of the restart failure, the severity of the failure needs to be further determined. The first threshold value can be a preset integer value, for example, 2.

[0156] In some embodiments, when the number of nodes that have no edge with the node is greater than or equal to the first threshold value, it indicates that the node cannot establish a connection with a majority of nodes in the cluster. At this time, the node sends a first request message to a master node in the cluster, requesting the master node to delete the node in the connection topology graph of the cluster, or the node sends a first request message to other nodes based on the restored communication link, requesting the cluster management module in the other nodes to delete the node.

[0157] In some embodiments, when the number of nodes that have no edge with the node is less than the first threshold value, it indicates that the node only has connection abnormality with a small number of nodes, and has normal connection with a majority of nodes in the cluster. At this time, the node sends a second request message to other nodes based on the restored communication link, and after receiving the second request message, the other nodes can have two processing methods: one is that the cluster management module in the other nodes deletes the nodes that cannot be connected with the node; the other is that the cluster management module in the other nodes deletes the node, or the node sends a second request message to a master node in the cluster, and after receiving the second request message, the master node can have two processing methods: one is that the master node deletes the nodes that cannot be connected with the node in the connection topology graph of the cluster; the other is that the master node deletes the node in the connection topology graph of the cluster.

[0158] The other nodes or the master node can randomly execute one of the two processing methods.

[0159] Further, each node in the cluster can update the connection topology graph of the cluster stored locally by each node based on the determined connection topology graph of the cluster.

[0160] The connection topology graph of the cluster stored locally by the node is the cluster view.

[0161] For example, by Figure 6 The processing method of the other nodes after receiving the second request message is described.

[0162] Figure 6 A schematic diagram of a connection topology view of a cluster with restart failure provided by an embodiment of the present application is shown in Figure 6 As shown, the nodes in the cluster include node D, node E and node F, wherein the node D is a restarted node.

[0163] An edge exists between node D and node E, meaning that a restored communication link exists between them. An edge exists between node E and node F, meaning that a restored communication link exists between them. No edge exists between node D and node F, meaning that no restored communication link exists between them.

[0164] Optionally, node D sends a second request message to node E based on the restored communication link with node E; after receiving the second request message, node E deletes either node D or node F in its cluster management module, thus achieving full connectivity among all nodes in the cluster.

[0165] exist Figure 5 In the illustrated embodiment, based on the sent second message, a third message is sent to the cluster management module in the node. The third message instructs the node that received the second message to be added to the cluster management module. Based on the information in the cluster management module, a connection topology view of the cluster is generated. Based on the connection topology view of the cluster, the restart result of the node is determined. Using this method, it is possible to accurately identify whether a node has successfully restored its connection with other nodes, thereby accurately distinguishing between successful and failed restarts.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0167] Figure 7 This is a schematic diagram of a node restart device for a centralized storage cluster, provided as an embodiment of this application. Figure 7 As shown, embodiments of this application also provide a node restart device 700 for a centralized storage cluster, comprising:

[0168] The acquisition module 701 is used to acquire the communication link corresponding to the node during the node restart process;

[0169] The first determining module 702 is used to determine the current node status of a node based on the communication link corresponding to the node. The node status is used to indicate the recovery status of the node's communication link.

[0170] Processing module 703 is used to start the timer corresponding to the node when the node status indicates that the node has a restored first communication link and an unrestored second communication link;

[0171] The first sending module 704 is configured to, before the expiration of the timer, if there is a second communication link that has not been recovered, send a first message based on the first communication link and the second communication link that has been recovered, the first message indicating that the node is not ready to join the cluster; and if all the second communication links have been recovered, send a second message based on the first communication link and the second communication link, and cancel the timer, the second message indicating that the node is ready to join the cluster.

[0172] The second sending module 705 is configured to, after the expiration of the timer, if there is a second communication link that has not been recovered, send the second message based on the first communication link and the second communication link that has been recovered during the timing of the timer.

[0173] The second determining module 706 is configured to determine the restart result of the node based on the sent second message.

[0174] The features of the embodiments of the node restart apparatus 700 of the centralized storage cluster can be referred to the related descriptions of the embodiments of the node restart method of the centralized storage cluster, which will not be repeated here.

[0175] In some embodiments, before the node is restarted, the obtaining module 701 is further configured to:

[0176] When the node is normally running, the node information storage module in the node stores the communication link corresponding to the node.

[0177] When the node is abnormally powered off, the power supply battery in the node supplies power to the node information storage module in the node, so that the communication link corresponding to the node can be successfully stored.

[0178] In some embodiments, the first determining module 702 is further configured to:

[0179] The link recovery module in the node recovers the original network configuration of the node.

[0180] Based on the communication link corresponding to the node, the connection of the communication link corresponding to the node is recovered.

[0181] In some embodiments, the processing module 703 is specifically configured to:

[0182] When the node status indicates that the first communication link of the node has been recovered and the other communication links have not been recovered, the first duration of the timer corresponding to the node is determined.

[0183] Based on the first duration, the timer corresponding to the node is started.

[0184] In some embodiments, the processing module 703 is further specifically configured to:

[0185] The type of the second communication link is obtained.

[0186] determine the first time length based on the recovery time length corresponding to the type when the types of the second communication links are the same type;

[0187] In some embodiments, the processing module 703 is further specific for:

[0188] when the types of the second communication links are the same type, determining a recovery time length corresponding to the type; and determining the first time length based on the recovery time length corresponding to the type, where the first time length is greater than the recovery time length corresponding to the type;

[0189] when the types of the second communication links are multiple types, determining recovery time lengths corresponding to the multiple types; determining a maximum recovery time length from the recovery time lengths corresponding to the multiple types; and determining the first time length based on the maximum recovery time length, where the first time length is greater than the maximum recovery time length.

[0190] In some embodiments, the processing module 703 is further specific for:

[0191] when the node status indicates that the node does not have a restored communication link, waiting for a second time length; if there is a restored communication link within the second time length, starting the timer corresponding to the node; and if there is no restored communication link within the second time length, stopping the timer corresponding to the node;

[0192] when the node status indicates that all communication links of the node have been restored, sending a second message based on the restored communication links.

[0193] In some embodiments, the first message and the second message are periodic messages, where before the timer expires, the first message is continuously sent at a millisecond level interval when there is an un-restored second communication link; and after all second communication links have been restored or the timer expires, the second message is continuously sent at a millisecond level interval.

[0194] In some embodiments, the second determining module 706 is specific for:

[0195] based on the sent second message, sending a third message to a cluster management module in the node, the third message indicating that the node receiving the second message is added in the cluster management module;

[0196] based on information in the cluster management module, generating a connection topology view of the cluster;

[0197] based on the connection topology view of the cluster, determining a restart result of the node.

[0198] In some embodiments, the second determining module 706 is further specific for:

[0199] based on information in the cluster management module, determining an edge between the node and other nodes in the cluster, the edge indicating that the node and the other nodes are communicatively connected;

[0200] Based on the edges between the node and other nodes, a connection topology view of the cluster is generated.

[0201] In some embodiments, the second determining module 706 is further specifically configured to:

[0202] In the connection topology view of the cluster, it is determined whether there is an edge between the node and other nodes in the cluster;

[0203] If there is an edge between the node and other nodes in the cluster, it is determined that the restart result of the node is restart success;

[0204] If there is no edge between the node and any one of the nodes in the cluster, it is determined that the restart result of the node is restart failure;

[0205] When the restart result is restart failure, based on the connection topology view of the cluster, the number of nodes that have no edge with the node is obtained;

[0206] When the number of nodes is greater than or equal to a first threshold, a first request message is sent, and the first request message is used to request to disconnect the node in the cluster;

[0207] When the number of nodes is less than the first threshold, a second request message is sent, and the second request message is used to request to disconnect the nodes that have no edge with the node in the cluster, or is used to request to disconnect the node in the cluster.

[0208] Figure 8 A structural schematic diagram of an electronic device is provided in an embodiment of the present application. As shown in the figure, Figure 8 The electronic device 800 provided in the embodiment includes at least one processor 801 and a memory 802. Optionally, the electronic device 800 further includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected through a bus.

[0209] In the specific implementation process, the at least one processor 801 executes the computer execution instructions stored in the memory 802, so that the at least one processor 801 executes the node restart method for the centralized storage cluster described above.

[0210] The specific implementation process of the processor 801 can refer to the method embodiments described above, which has similar implementation principles and technical effects, and will not be described here in detail.

[0211] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0212] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0213] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0214] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is configured to execute the steps in any of the above-mentioned node restart method embodiments of the centralized storage cluster when running.

[0215] In an example embodiment, the above-mentioned computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0216] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned node restart method embodiments of the centralized storage cluster.

[0217] The embodiment of the present application further provides another computer program product, comprising a nonvolatile computer readable storage medium, the nonvolatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps in any of the node restart method embodiments of the centralized storage cluster.

[0218] Those skilled in the art will further appreciate that the functions implemented by the elements and algorithms described herein can be implemented using electronic hardware, computer software, or any combination thereof. Depending on the particular application, different methods can be used to implement the functions described herein, and these methods can be implemented in different ways. The methods described herein are not limited to any particular implementation, and can be implemented in any combination of hardware, software, or firmware. Furthermore, the methods described herein can be implemented using any number of hardware devices, software modules, and / or firmware modules.

[0219] The above describes in detail the node restart method of the centralized storage cluster provided by the present application. The principles and implementation manners of the present application are described by applying specific examples herein, and the above description of the embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for node restart of a centralized storage cluster, characterized in that, The method comprises: during the process of node restart, obtaining the communication links corresponding to the node; based on the communication links corresponding to the node, determining the current node state of the node, the node state indicating the recovery situation of the communication links of the node; when the node state indicates that the node has a first communication link that has been recovered and a second communication link that has not been recovered, starting a timer corresponding to the node; before the timer expires, if there is a second communication link that has not been recovered, sending a first message based on the first communication link and the second communication link that has been recovered, the first message indicating that the node is not ready to join the cluster; if all the second communication links have been recovered, sending a second message based on the first communication link and the second communication link, and canceling the timer, the second message indicating that the node is ready to join the cluster; after the timer expires, if there is a second communication link that has not been recovered, sending the second message based on the first communication link and the second communication link that has been recovered during the counting process of the timer; based on the sent second message, determining the restart result of the node.

2. The method of claim 1, wherein, Before the node restart, the method further comprises: when the node is normally running, a node information saving module in the node stores the communication link information corresponding to the node; when the node is abnormally powered off, a backup battery in the node supplies power to the node information saving module in the node, so that the communication link information corresponding to the node can be successfully stored.

3. The method of claim 1, wherein, The starting of the timer corresponding to the node when the node state indicates that the node has a first communication link that has been recovered and a second communication link that has not been recovered comprises: when the node state indicates that the first communication link of the node has been recovered and the other communication links have not been recovered, determining a first duration of the timer corresponding to the node; based on the first duration, starting the timer corresponding to the node.

4. The method of claim 3, wherein, The determination of the first duration of the timer corresponding to the node comprises: obtaining the types of the second communication links; based on the types of the second communication links, determining the first duration.

5. The method of claim 4, wherein, The determination of the first duration based on the types of the second communication links comprises: when the types of the second communication links are the same type, determining the recovery duration corresponding to the type; based on the recovery duration corresponding to the type, determining the first duration, at this time, the first duration is greater than the recovery duration corresponding to the type; when the types of the second communication links are multiple types, determining the recovery durations corresponding to the types; determining the maximum recovery duration among the recovery durations corresponding to the types; based on the maximum recovery duration, determining the first duration, at this time, the first duration is greater than the maximum recovery duration.

6. The method of claim 3, wherein, The method further comprises: when the node state indicates that the node does not have a recovered communication link, waiting for a second duration; if there is a recovered communication link within the second duration, starting the timer corresponding to the node; if there is no recovered communication link within the second duration, stopping the starting of the timer corresponding to the node; When the node state indicates that all communication links corresponding to the node have been recovered, a second message is sent based on the communication links corresponding to the node.

7. The method of claim 1, wherein, The first message and the second message are periodically sent packets, wherein, before the timer expires, when there is a second communication link that has not been recovered, the first message is continuously sent at a millisecond level interval; after all second communication links have been recovered or the timer expires, the second message is continuously sent at a millisecond level interval.

8. The method of claim 1, wherein, The determination of the restart result of the node based on the sent second message comprises: Based on the sent second message, a third message is sent to a cluster management module in the node, the third message indicating that the node receiving the second message is added in the cluster management module; Based on information in the cluster management module, a connection topology view of the cluster is generated; Based on the connection topology view of the cluster, the restart result of the node is determined.

9. The method of claim 8, wherein, The generation of the connection topology view of the cluster based on the information in the cluster management module comprises: Based on the information in the cluster management module, edges between the node and other nodes in the cluster are determined, the edges being used to indicate that the node and the other nodes are in communication connection; Based on the edges between the node and the other nodes, the connection topology view of the cluster is generated.

10. The method of claim 9, wherein, The determination of the restart result of the node based on the connection topology view of the cluster comprises: In the connection topology view of the cluster, it is determined whether there are edges between the node and other nodes in the cluster; If there are edges between the node and the other nodes in the cluster, it is determined that the restart result of the node is restart success; If there is no edge between the node and any one of the other nodes in the cluster, it is determined that the restart result of the node is restart failure; When the restart result is restart failure, based on the connection topology view of the cluster, a number of nodes that have no edge with the node is obtained; When the number of nodes is greater than or equal to a first threshold, a first request message is sent, the first request message being used to request to disconnect the node in the cluster; When the number of nodes is less than the first threshold, a second request message is sent, the second request message being used to request to disconnect nodes that have no edge with the node in the cluster, or being used to request to disconnect the node in the cluster.

Images