Storage node fault judgment method and device

Abstract

The invention discloses a storage node fault judgment method and device, and relates to the technical field of distributed storage, and the method comprises the steps: obtaining state monitoring results of a plurality of monitoring nodes in a distributed storage system for a target storage node, the state monitoring results comprising a normal state and an abnormal state; acquiring continuous interruption duration of heartbeat connection between the target storage node and the master control node; and when the number of the abnormal states in all the state monitoring results exceeds a preset number threshold value and the continuous interruption duration exceeds a first preset duration, marking the target storage node as a DOWN state. According to the method, the state monitoring data of a plurality of monitoring nodes are introduced, the redundancy and reliability of judgment are enhanced, meanwhile, the continuous interruption duration of heartbeat connection is combined as the judgment basis, the influence of local network fluctuation on the judgment process is reduced, the misjudgment that the storage nodes are in the DOWN state is reduced, and the judgment efficiency is improved. Therefore, the accuracy of judging whether the storage node is in the DOWN state is improved.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and particularly relates to a method and device for judging the failure of storage nodes. Background Art

[0002] With the wide application of distributed storage systems in the fields of cloud computing, big data, etc., business continuity and data reliability have become core requirements. Especially in the scenario of judging the failure of storage nodes, how to accurately identify the abnormal state of storage nodes and avoid service interruption or data loss caused by misjudgment is the key research direction in the current technical field.

[0003] In related technologies, distributed storage systems usually judge the state of storage nodes based on mechanisms such as heartbeat detection and storage node monitoring reports. For example, when the number of heartbeat timeouts exceeds a set threshold, it is judged that the storage node fails, or when the storage node reports an abnormality, the storage node is marked as the DOWN state.

[0004] However, the above methods for judging the state of storage nodes are prone to misjudging the storage node as the DOWN state due to local network fluctuations or the failure of a single storage node. Summary of the Invention

[0005] This application provides a method and device for judging the failure of storage nodes, which can reduce the misjudgment of storage nodes as the DOWN state.

[0006] In a first aspect, an embodiment of this application provides a method for judging the failure of a storage node, and the method includes:

[0007] Obtain the state monitoring results of a target storage node by multiple monitoring nodes in a distributed storage system, where the state monitoring results include a normal state and an abnormal state;

[0008] Obtain the continuous interruption duration of the heartbeat connection between the target storage node and the master control node;

[0009] When the number of abnormal states in all state monitoring results exceeds a preset number threshold and the continuous interruption duration exceeds a first preset duration, mark the target storage node as the DOWN state.

[0010] In combination with the first aspect, in an implementation, obtaining the state monitoring results of a target storage node by multiple monitoring nodes in a distributed storage system includes:

[0011] Send a state query request to each monitoring node based on a preset polling period, and receive the state monitoring results fed back by each monitoring node;

[0012] When the received state monitoring result is an abnormal state, send a state query request to the corresponding monitoring node again;

[0013] When the status monitoring results of two queries are the same, determine that the status monitoring result is an abnormal state; otherwise, determine that the status monitoring result is a normal state.

[0014] Combined with the first aspect, in one implementation, obtaining the duration of continuous interruption of the heartbeat connection between the target storage node and the master node includes:

[0015] Record the first moment when the target storage node fails to respond to the heartbeat packet sent by the master node for the first time;

[0016] Calculate the first difference between the current moment and the first moment, and the first difference is the duration of continuous interruption of the heartbeat connection between the target storage node and the master node.

[0017] Combined with the first aspect, in one implementation, after marking the target storage node as the DOWN state, it further includes:

[0018] Trigger the redundancy recovery mechanism or write protection mechanism of the storage pool in the distributed storage system, and suspend the update operation of the storage resource allocation map in the distributed storage system;

[0019] The data stored in the storage resource allocation map is the mapping relationship of each storage node in the distributed storage system.

[0020] Combined with the first aspect, in one implementation, the redundancy recovery mechanism includes:

[0021] Create a copy of the data object carried by the target storage node on the non-failed storage nodes of the storage pool, so that the overall redundancy of the storage pool is maintained at a preset security level.

[0022] Combined with the first aspect, in one implementation, the write protection mechanism includes:

[0023] Intercept all write requests for the target storage node, and record the metadata change information of the data to be written corresponding to each write request;

[0024] Store the metadata change information in the temporary transaction log of the distributed storage system, and after the target storage node recovers, execute all intercepted write requests based on the metadata change information.

[0025] Combined with the first aspect, in one implementation, when it is monitored that the heartbeat connection between the target storage node and the master node is restored, it further includes:

[0026] Remove the DOWN state mark of the target storage node.

[0027] Second aspect, embodiments of the present application provide a storage node failure determination device, the device comprising:

[0028] A status monitoring module, configured to obtain status monitoring results of a plurality of monitoring nodes in a distributed storage system for a target storage node, the status monitoring results including a normal status and an abnormal status;

[0029] A duration acquisition module, configured to acquire a duration of continuous interruption of a heartbeat connection between the target storage node and a master control node;

[0030] A determination module, configured to mark the target storage node as being in a DOWN state when the number of abnormal statuses in all status monitoring results exceeds a preset number threshold and the duration of continuous interruption exceeds a first preset duration.

[0031] In combination with the second aspect, in an implementation manner, it further comprises:

[0032] A redundancy recovery module, configured to create a copy of a data object borne by the target storage node on an unfailed storage node in the storage pool, so that the overall redundancy of the storage pool is maintained at a preset security level.

[0033] In combination with the second aspect, in an implementation manner, it further comprises:

[0034] A write protection module, configured to intercept all write requests for the target storage node, record metadata change information of to-be-written data corresponding to each write request; and further configured to store the metadata change information in a temporary transaction log of the distributed storage system, and execute all intercepted write requests based on the metadata change information after the target storage node recovers.

[0035] The beneficial effects brought by the technical solutions provided by the embodiments of the present application include:

[0036] The present application determines whether a storage node is in a DOWN state by obtaining status monitoring results of a plurality of monitoring nodes in a distributed storage system for a target storage node and combining the duration of continuous interruption of a heartbeat connection between the target storage node and a master control node, and solves the technical problem in the related art that it is easy to misjudge a storage node as being in a DOWN state due to local network fluctuations or a single storage node failure. This solution enhances the redundancy and reliability of the determination by introducing status monitoring data of multiple monitoring nodes, and at the same time combines the duration of continuous interruption of the heartbeat connection as a determination basis, reduces the influence of local network fluctuations on the determination process, reduces the misjudgment of the storage node being in a DOWN state, and thus improves the accuracy of determining that the storage node is in a DOWN state. Description of the Drawings

[0037] Figure 1It is a schematic flowchart of the method for judging the failure of a storage node in an embodiment of the present application;

[0038] Figure 2 It is a schematic flowchart of the process for a monitoring node in an embodiment of the present application to generate a status monitoring result;

[0039] Figure 3 It is a schematic flowchart of the process for a master control node in an embodiment of the present application to process a status monitoring result;

[0040] Figure 4 It is a schematic structural diagram of the storage node failure judgment device in an embodiment of the present application. Detailed implementation manners

[0041] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0042] To make the purpose, technical solution and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0043] In the first aspect, please refer to Figure 1 , Figure 1 It is a schematic flowchart of the method for judging the failure of a storage node in an embodiment of the present application. The method for judging the failure of a storage node provided in this embodiment includes the following steps:

[0044] Step S1: Obtain the status monitoring results of a target storage node by multiple monitoring nodes in a distributed storage system. The status monitoring results include a normal state and an abnormal state.

[0045] Step S2: Obtain the duration of continuous interruption of the heartbeat connection between the target storage node and the master control node.

[0046] Step S3: When the number of abnormal states in all status monitoring results exceeds a preset number threshold and the duration of continuous interruption exceeds a first preset duration, mark the target storage node as the DOWN state.

[0047] This method determines whether the storage node is in the DOWN state by obtaining the status monitoring results of a target storage node by multiple monitoring nodes in a distributed storage system and combining the duration of continuous interruption of the heartbeat connection between the target storage node and the master control node, and solves the technical problem in the related art that it is easy to misjudge the storage node as the DOWN state due to local network fluctuations or the failure of a single storage node.

[0048] It should be noted that assuming there are multiple monitoring nodes in a distributed storage system to monitor the status of a target storage node, some of the monitoring nodes may report anomalies because they are temporarily unable to obtain the status monitoring results of the target storage node due to network congestion. However, if other nodes can obtain the status monitoring results normally and consider the target storage node to be normal, then it is impossible to directly determine that the target storage node is in the DOWN state. By comprehensively considering all status monitoring results and then judging the status of the target storage node, it is possible to effectively avoid mislabeling the target storage node as being in the DOWN state, thereby maintaining the normal operation of the distributed storage system and avoiding unnecessary IO interruptions.

[0049] The heartbeat mechanism is a commonly used means in a distributed storage system to detect the status of storage nodes. A normal heartbeat indicates that the storage node can communicate with the master node to a certain extent, can receive and execute some necessary instructions. Even if it may have some minor anomalies, as long as the heartbeat connection between the target storage node and the master node is normal, the storage node is not determined to be in the DOWN state. For example, a storage node may be performing some internal data sorting or resource recovery operations, resulting in delayed responses to some status query requests. However, as long as its heartbeat connection with the master node is normal, the distributed storage system considers it to be within the controllable range and will not interrupt its IO operations.

[0050] This method enhances the redundancy and reliability of the judgment by introducing the status monitoring data of multiple monitoring nodes. At the same time, by combining the continuous interruption duration of the heartbeat connection as a judgment basis, it reduces the impact of local network fluctuations on the judgment process, reduces the misjudgment of the storage node being in the DOWN state, and thus improves the accuracy of judging the storage node to be in the DOWN state.

[0051] In some embodiments, in the above step S1, obtaining the status monitoring results of multiple monitoring nodes in the distributed storage system for the target storage node includes the following steps:

[0052] S11: Send a status query request to each monitoring node based on a preset polling period, and receive the status monitoring results feedback by each monitoring node.

[0053] S12: When the received status monitoring result is an abnormal state, send a status query request to the corresponding monitoring node again.

[0054] S13: When the status monitoring results of the two queries are the same, determine that the status monitoring result is an abnormal state; otherwise, determine that the status monitoring result is a normal state.

[0055] In this embodiment, a status query request is periodically sent to each monitoring node through a preset polling period, so as to continuously and systematically collect the status information of the target storage node. When an abnormal status monitoring result is received for the first time, instead of immediately determining it as abnormal, a status query request is sent to the corresponding monitoring node again for confirmation. Adopting this dual verification mechanism helps to reduce misjudgments caused by single query errors or instantaneous failures, thereby improving the accuracy of monitoring.

[0056] In some embodiments, please refer to Figure 2 , Figure 2 which is a schematic flowchart of the process for the monitoring node in the embodiment of the present application to generate a status monitoring result. The process for the monitoring node in this embodiment to generate a status monitoring result includes the following steps:

[0057] Step A01: The monitoring node counts the information of normal and abnormal I / Os within a second preset duration, and generates a PoolBrNotify message based on the information of normal and abnormal I / Os. The PoolBrNotify is the status monitoring result.

[0058] Specifically, the second preset duration is preferably set to 5 seconds.

[0059] Step A02: The monitoring node sends the PoolBrNotify message to the pool leader. The pool leader is the master control node.

[0060] Step A03: The monitoring node clears the information of normal and abnormal I / Os counted this time.

[0061] Step A04: The monitoring node sleeps for the second preset time and yields the scheduling right. After step A04 ends, it returns to step A01.

[0062] In some embodiments, please refer to Figure 3 , Figure 3 which is a schematic flowchart of the process for the master control node in the embodiment of the present application to process the status monitoring result. The process for the master control node in this embodiment to process the status monitoring result includes the following steps:

[0063] Step B01: The master control node receives the status monitoring results within a third preset duration.

[0064] Specifically, the third preset duration is preferably set to 10 seconds.

[0065] Step B02: The master control node determines whether to mark the target storage node as the DOWN state. If so, it enters step B03; if not, it enters step B04.

[0066] Step B03: The master control node marks the target storage node as the DOWN state, updates the storage resource allocation map, and then enters step B04.

[0067] Step B04: The master node clears the status monitoring results of this statistics.

[0068] Step B05: The master node sleeps for a third preset time and yields the scheduling right. After step B05 ends, return to step B01.

[0069] In some embodiments, in the above step S2, obtaining the continuous interruption duration of the heartbeat connection between the target storage node and the master node includes the following steps:

[0070] S21: Record the first moment when the target storage node fails to respond to the heartbeat packet sent by the master node for the first time.

[0071] S22: Calculate the first difference between the current moment and the first moment, and the first difference is the continuous interruption duration of the heartbeat connection between the target storage node and the master node.

[0072] Specifically, after the master node sends a heartbeat packet, it will record the current timestamp as the sending time. If within the predetermined response time, the master node does not receive a response from the target storage node, then record the current timestamp as the first moment. After recording the first moment, the master node will continue to monitor the response of the target storage node. If the target storage node still does not respond in subsequent heartbeat cycles, the master node will calculate the difference between the current time and the first moment, and this difference represents the duration of the continuous termination of the heartbeat connection.

[0073] In some embodiments, in the above step S3, determining the preset quantity threshold according to the total number of monitoring nodes in the distributed storage system. For example, in a distributed storage system with 10 monitoring nodes, the preset quantity threshold is 3. Only when at least 3 nodes simultaneously feedback that the target storage node is in an abnormal state and the continuous interruption duration between the target storage node and the master node exceeds the first preset duration, the target storage node is marked as the DOWN state. This can reduce the misjudgment caused by a single monitoring node's false alarm or local network problems and improve the reliability of the distributed storage system.

[0074] In some embodiments, after marking the target storage node as the DOWN state in the above step S3, it further includes:

[0075] Trigger the redundancy recovery mechanism or write protection mechanism of the storage pool in the distributed storage system, and suspend the update operation of the storage resource allocation map in the distributed storage system, where the data stored in the storage resource allocation map is the mapping relationship of each storage node in the distributed storage system.

[0076] It should be noted that distributed storage systems are usually designed with redundancy recovery mechanisms, such as replicas or erasure codes, to ensure normal operation even when some storage nodes fail. After the target storage node is marked as the DOWN state, the redundancy recovery mechanism of the distributed storage system can be triggered, and the lost data replicas are copied from normal storage nodes to the target storage node, so as to quickly restore the data of the target storage node.

[0077] Another strategy for handling the failure of the target storage node is to trigger the write protection mechanism of the distributed storage system. The write protection mechanism will temporarily prevent write operations on the affected data area until the failed node recovers, ensuring that no new data is written during the data recovery process, which may cause data chaos.

[0078] The storage resource allocation map is a key data structure in the distributed storage system for tracking and managing the mapping relationships of each storage node. During the failure of the target storage node, in order to avoid resource allocation chaos caused by the change of the target storage node state, the distributed storage system will suspend the update operation of the storage resource allocation map, keep the resource allocation state stable, and thus avoid the occurrence of a chain failure reaction.

[0079] In some embodiments, the above redundancy recovery mechanism includes:

[0080] Create replicas of the data objects borne by the target storage node on the non-failed storage nodes in the storage pool, so that the overall redundancy of the storage pool is maintained at a preset security level.

[0081] It should be noted that data objects include basic storage units of distributed storage systems such as files, database records, and data blocks. When the distributed storage system selects non-failed storage nodes, it needs to consider the load status, network bandwidth, and storage capacity of the storage nodes to ensure the normal operation of the replica creation process. The selected non-failed storage node can be a single storage node or multiple storage nodes. During the replica creation process, by dynamically adjusting the number and distribution of replicas, the overall redundancy of the storage pool can be maintained at a preset security level, thus avoiding over-allocation of storage resources and improving the overall performance of the distributed storage system.

[0082] In some embodiments, the above write protection mechanism includes:

[0083] First, intercept all write requests for the target storage node, record the metadata change information of the data to be written corresponding to each write request, and then store the metadata change information in the temporary transaction log of the distributed storage system. After the target storage node recovers, execute all intercepted write requests based on the metadata change information.

[0084] In this embodiment, by intercepting write requests and recording metadata change information, the distributed storage system can perform write operations after a failed node recovers, avoiding data loss or corruption caused by node failures, thereby ensuring data integrity.

[0085] In some embodiments, when it is detected that the heartbeat connection between the target storage node and the master control node is restored, it further includes: removing the DOWN status flag of the target storage node.

[0086] In a second aspect, please refer to Figure 4 , Figure 4 which is a schematic structural diagram of the storage node failure determination device according to the embodiment of the present application. The storage node failure determination device provided in this embodiment includes the following modules:

[0087] A status monitoring module, configured to obtain the status monitoring results of multiple monitoring nodes in the distributed storage system for the target storage node, and the status monitoring results include a normal state and an abnormal state.

[0088] A duration obtaining module, configured to obtain the continuous interruption duration of the heartbeat connection between the target storage node and the master control node.

[0089] A judgment module, configured to mark the target storage node as the DOWN state when the number of abnormal states in all status monitoring results exceeds a preset number threshold and the continuous interruption duration exceeds a first preset duration.

[0090] In this method, the status monitoring module obtains the status monitoring results of multiple monitoring nodes in the distributed storage system for the target storage node, the duration obtaining module obtains the continuous interruption duration of the heartbeat connection between the target storage node and the master control node, and combines the status monitoring results and the continuous interruption duration to determine whether the storage node is in the DOWN state, solving the technical problem in the related art that it is easy to misjudge the storage node as the DOWN state due to local network fluctuations or single storage node failures.

[0091] In some embodiments, the above storage node failure determination device further includes a redundancy recovery module.

[0092] The redundancy recovery module is configured to create a copy of the data object borne by the target storage node on the non-failed storage nodes in the storage pool, so that the overall redundancy of the storage pool is maintained at a preset security level.

[0093] In some embodiments, the above storage node failure determination device further includes a write protection module.

[0094] A write protection module is used to intercept all write requests for a target storage node, record the metadata change information of the data to be written corresponding to each write request, and store the metadata change information in the temporary transaction log of the distributed storage system. After the target storage node is restored, all intercepted write requests are executed based on the metadata change information.

[0095] It should be noted that the serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments.

[0096] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices. The descriptions of terms such as "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit that "first", "second" and "third" are different types.

[0097] In the description of the embodiments of the present application, "exemplary", "for example" or "for instance" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary", "for example" or "for instance" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary", "for example" or "for instance" is intended to present related concepts in a specific manner.

[0098] In the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may represent A or B. The "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality of" means two or more than two.

[0099] In some processes described in the embodiments of the present application, a plurality of operations or steps appear in a specific order. However, it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or may be executed in parallel. The serial numbers of the operations are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in order or in parallel, and these operations or steps may be combined.

[0100] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes several instructions for causing a terminal device to execute the methods described in various embodiments of the present application.

[0101] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A method for judging storage node failures, characterized in that, The method includes: Obtaining the status monitoring results of a target storage node by multiple monitoring nodes in a distributed storage system, where the status monitoring results include a normal state and an abnormal state; Obtaining the duration of continuous interruption of the heartbeat connection between the target storage node and the master control node; When the number of abnormal states in all status monitoring results exceeds a preset quantity threshold and the duration of continuous interruption exceeds a first preset duration, marking the target storage node as the DOWN state.

2. The storage node fault judgment method according to claim 1, characterized in that The obtaining the status monitoring results of a target storage node by multiple monitoring nodes in a distributed storage system includes: Sending a status query request to each monitoring node based on a preset polling period, and receiving the status monitoring results fed back by each monitoring node; When the received status monitoring result is an abnormal state, sending a status query request to the corresponding monitoring node again; When the status monitoring results of two queries are the same, determining that the status monitoring result is an abnormal state; otherwise, determining that the status monitoring result is a normal state.

3. The storage node fault determination method according to claim 1, wherein The obtaining the duration of continuous interruption of the heartbeat connection between the target storage node and the master control node includes: Recording a first moment when the target storage node first fails to respond to the heartbeat packet sent by the master control node; Calculating a first difference between the current moment and the first moment, and the first difference is the duration of continuous interruption of the heartbeat connection between the target storage node and the master control node.

4. The storage node fault determination method according to claim 1, characterized in that, After marking the target storage node as the DOWN state, it further includes: Triggering a redundancy recovery mechanism or a write protection mechanism of a storage pool in the distributed storage system, and suspending the update operation of the storage resource allocation map in the distributed storage system; The data stored in the storage resource allocation map is the mapping relationship of each storage node in the distributed storage system.

5. The storage node fault determination method according to claim 4, wherein The redundancy recovery mechanism includes: Creating a copy of the data object carried by the target storage node on the non-failed storage nodes of the storage pool, so that the overall redundancy of the storage pool is maintained at a preset security level.

6. The storage node fault determination method according to claim 4, characterized in that, The write protection mechanism includes: Intercepting all write requests for the target storage node, and recording the metadata change information of the data to be written corresponding to each write request; Storing the metadata change information in the temporary transaction log of the distributed storage system, and after the target storage node recovers, executing all intercepted write requests based on the metadata change information.

7. The storage node fault judgment method according to claim 1, characterized in that When it is monitored that the heartbeat connection between the target storage node and the master control node is restored, it further includes: Removing the DOWN state mark of the target storage node.

8. A storage node fault determination device based on the method according to any one of claims 1-7, characterized in that, The device includes: A status monitoring module, configured to obtain the status monitoring results of a target storage node by multiple monitoring nodes in a distributed storage system, where the status monitoring results include a normal state and an abnormal state; A duration obtaining module, configured to obtain the duration of continuous interruption of the heartbeat connection between the target storage node and the master control node; A judgment module, configured to mark the target storage node as the DOWN state when the number of abnormal states in all status monitoring results exceeds a preset quantity threshold and the duration of continuous interruption exceeds a first preset duration.

9. The storage node fault determination device according to claim 8, wherein It further includes: A redundancy recovery module, which is used to create a copy of the data object borne by the target storage node on the non-failed storage nodes of the storage pool, so that the overall redundancy of the storage pool is maintained at a preset security level.

10. The storage node fault determination device according to claim 8, wherein It further includes: A write protection module, which is used to intercept all write requests for the target storage node and record the metadata change information of the data to be written corresponding to each write request; It is also used to store the metadata change information in the temporary transaction log of the distributed storage system, and after the target storage node recovers, execute all intercepted write requests based on the metadata change information.

Images