Method, system and device for processing node failure of distributed storage system and medium

Abstract

The application discloses a kind of processing methods of distributed storage system node failure, it is related to distributed storage field, to solve the problem that the whole failure of storage system or multiple node failure simultaneously causes unable to receive service read-write, in the application, lease message is sent to each spare monitor node when failure recovery;Obtain the lease response message returned by each spare monitor node to obtain the start information of each spare monitor node;Whether the majority node failure of storage system occurs is judged according to the start information of each spare monitor node;If yes, update object storage device maintenance table.It avoids that the object storage device is in offline state after failure, and the online state of object storage device before failure is wrongly updated in object storage device maintenance table in storage system, which causes storage system to be always in the process of promoting master selection, so that the front-end service served by storage system is faster to recover to the level before failure, and the stability of storage system in the majority node failure is improved.

Description

Technical Field

[0001] This application relates to the field of distributed storage, and in particular to a method, system, apparatus and medium for handling node failures in a distributed storage system. Background Technology

[0002] In distributed storage systems, to ensure the security of user data, the same data is replicated multiple times and stored on different storage nodes. When a storage node fails, the system can recover the data from the replicas on other storage nodes. For ease of management, the storage pool in a distributed storage system is divided into multiple Placement Groups (PGs). Each PG contains several object storage devices (OSDs), i.e., hard disks, according to certain replication or erasure rules. Taking a three-replica rule as an example, the same data is stored three times in a PG. A PG contains three hard disks, and each hard disk is managed by a secondary PG, resulting in three secondary PGs. These three secondary PGs are elected to serve as primary and backup PGs. The primary and secondary PGs are responsible for the data read and write processes. Therefore, the election of a primary and secondary PG by these three secondary PGs is a prerequisite for the normal read and write operation of the storage system. Generally, a change in the online or offline status of an object storage device (up or down in the object storage device maintenance table) will trigger the peering process for the secondary PG. During the peering process, all read and write information that is up in the object storage device maintenance table will be collected from each other. The secondary PG with the most records will be selected as the primary secondary PG. If a primary and secondary PG cannot be elected, the peering process will continue indefinitely, and the storage system will not be able to accept business read and write operations, affecting front-end services.

[0003] However, when both the object storage device service and the cluster's monitor daemon service fail simultaneously, the monitor service cannot complete the resolution of object storage device state changes. Therefore, it cannot mark the failed object storage devices as down. The status of these failed object storage devices in the database's object storage device maintenance table (ObjectStorageDevice Map, OSDMAP) is also up. When the fault is recovered, when the monitor starts, it reads the historical status of the object storage devices from the database for recovery. That is to say, when the object storage device is down, it is still incorrectly marked as up in the object storage device maintenance table. During the election of the secondary PG, these failed object storage devices in the down state are considered to be up and their read and write information is collected. However, since these failed object storage devices are down and unresponsive, the primary and secondary PGs cannot be elected, and the process remains in the peering process. The storage system cannot accept business read and write operations, affecting front-end services.

[0004] Therefore, resolving the issue of the Monitor service failing to update object storage device status information correctly due to overall storage system failure or simultaneous failure of multiple nodes, and consequently causing the cluster to become abnormal and unable to receive business read and write operations when the fault is recovered due to the failure of a few nodes, is a technical problem that urgently needs to be solved by those in this field. Summary of the Invention

[0005] The purpose of this application is to provide a method for handling node failures in a distributed storage system, which solves the problem that a failure of the entire storage system or multiple nodes at the same time can cause the Monitor service to malfunction and fail to update the object storage device status information correctly. This can lead to an abnormal overall cluster status when the system recovers, as a small number of nodes may fail to recover, making it impossible to receive business read and write operations.

[0006] To address the aforementioned technical problems, this application provides a method for handling node failures in a distributed storage system, applied to the master monitor node of the storage system, comprising:

[0007] During fault recovery, a lease message is sent to each backup monitor node;

[0008] Obtain the lease response messages returned by each backup monitor node to obtain the startup information of each backup monitor node;

[0009] Determine whether a majority of nodes in the storage system have failed based on the startup information of each backup monitor node;

[0010] If so, update all object storage devices in the object storage device maintenance table to "down" and publish the updated object storage device maintenance table.

[0011] Preferably, obtaining the lease response messages returned by each backup monitor node to obtain the startup information of each backup monitor node includes:

[0012] The marking status of each backup monitor node is obtained based on the lease response message. The marking status includes true and false.

[0013] The startup information of each backup monitor node is obtained by marking its status.

[0014] Preferably, determining whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node includes:

[0015] Get the first number of items whose flag state is true;

[0016] Determine whether the first quantity is greater than the first threshold;

[0017] If so, then it is determined that a majority of nodes in the storage system have failed;

[0018] If not, then it is determined that the storage system has not experienced a majority node failure.

[0019] Preferably, the backup monitor node used in the storage system includes:

[0020] During fault recovery, obtain the lease message sent by the main monitor node;

[0021] The lease response message returned to the primary monitor node allows the primary monitor node to obtain the startup information of the backup monitor nodes. This enables the primary monitor node to determine whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node. When a majority node failure occurs in the storage system, all object storage devices in the object storage device maintenance table are updated to "down" and the updated object storage device maintenance table is published.

[0022] Preferably, the lease response message returned to the main monitor node includes:

[0023] Return the marking status to the main monitor node. The marking status can be either true or false.

[0024] Preferably, returning the marking status to the main monitor node includes:

[0025] Calculate the time difference between the current time and the storage system's startup time;

[0026] The marking status is determined based on the time difference;

[0027] Return the flagged status to the main monitor node.

[0028] Preferably, the marking status is determined based on the time difference:

[0029] Determine whether the time difference is less than the second threshold;

[0030] If so, then the flag state is determined to be true;

[0031] If not, the flag status is determined to be false.

[0032] To address the aforementioned technical problems, this application also provides a system for handling node failures in a distributed storage system, applied to the master monitor node of the storage system, comprising:

[0033] The sending module is used to send lease messages to each backup monitor node when a fault is recovered;

[0034] The acquisition module is used to acquire the lease response messages returned by each backup monitor node to obtain the startup information of each backup monitor node;

[0035] The judgment module is used to determine whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node;

[0036] The update module is used to update all object storage devices in the object storage device maintenance table to "down" and publish the updated object storage device maintenance table when a majority of nodes in the storage system fail.

[0037] To address the aforementioned technical problems, this application also provides a device for handling node failures in a distributed storage system, including a memory for storing computer programs;

[0038] A processor is used to implement steps for handling node failures in a distributed storage system when executing a computer program.

[0039] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of a method for handling node failures in a distributed storage system.

[0040] The distributed storage system node failure handling method provided in this application involves the primary monitor node sending lease messages to each backup monitor node to obtain their startup information during system recovery. Based on this startup information, it determines whether a majority node failure has occurred. If a majority node failure is confirmed, all object storage devices in the object storage device maintenance table are updated to "down," and the updated table is published. This avoids the storage system mistakenly updating the object storage device maintenance table with the pre-failure online status of the objects when they are offline, preventing the system from being in a continuous peering process. This shortens the peering process, allowing the front-end services served by the storage system to recover to pre-failure levels more quickly, thus improving the stability of the storage system in majority node failure scenarios. It also enhances the stability of the storage system in data center failures or large-scale node failure scenarios, strengthening its resilience against risks.

[0041] The distributed storage system node fault handling system, device and medium provided in this application have the same beneficial effects as above. Attached Figure Description

[0042] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a method for handling node failures in a distributed storage system, as provided in this application embodiment;

[0044] Figure 2 A flowchart illustrating another method for handling node failures in a distributed storage system, provided as an embodiment of this application;

[0045] Figure 3 A structural diagram of a distributed storage system node failure handling system provided in this application embodiment;

[0046] Figure 4 This is a structural diagram of another distributed storage system node fault handling device provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0048] The core of this application is to provide a method, system, device, and medium for handling node failures in a distributed storage system.

[0049] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] With the rapid development of computer and network technologies, the amount of data requiring storage and processing is increasing exponentially. Traditional storage systems use centralized storage servers to store all data, making these servers a bottleneck for system performance and a focal point for reliability and security, failing to meet the needs of large-scale storage applications. Therefore, distributed storage solutions have emerged. Distributed storage is a data storage technology that distributes data across multiple independent devices. Distributed storage systems utilize multiple storage servers to share the storage load and location servers to locate stored information. This not only improves system reliability, availability, and access efficiency but also facilitates scalability. Distributed storage systems contain a large number of storage nodes (storage servers), and node failures are frequent occurrences.

[0051] In distributed storage systems, to ensure user data security, the same data is typically replicated multiple times and stored on different storage nodes. If a storage node fails, the system can recover the data from the other replicas. To facilitate data distribution and recovery management, storage pools in a typical storage system are divided into multiple Groups (PGs). Each PG contains several object storage devices according to certain replication or erasure rules. For example, creating a three-replica rule means that a piece of data is divided into three parts, requiring three hard drives. Therefore, each PG contains three hard drives, meaning each PG has three members distributed across three different object storage devices for data management. These three PGs are distinguished by a primary PG and replica PGs. Generally, data read and write operations are received and distributed by the primary PG; therefore, the primary PG can be elected as the prerequisite for normal read and write operations. Normally, whenever an object storage device comes online or offline, a peering process (called the Group Storage) is triggered. During peering, read and write information for all members listed as "up" in the object storage device maintenance table is collected, and the PG with the most records is selected as the primary PG. Simultaneously, this process marks which members are missing data. After peering is complete, the PG's status is updated to active, and data recovery is performed. If a PG fails to be elected during this process, its status will remain in the peering state, preventing it from receiving read and write operations, thus impacting front-end services.

[0052] As described above, the peering process of a PG is triggered by changes in the object storage device status recorded in the object storage device maintenance table. Under normal circumstances, whenever an object storage device fails (or recovers from a failure), it notifies the Monitor via a message. The Monitor marks it as down (or up) and proposes (data persistence) to publish a new object storage device maintenance table, pushing it to the object storage device. Upon receiving the new object storage device maintenance table version, the object storage device, based on the information in the new version, will trigger the peering process for placement groups whose members have changed. A new master will be re-elected, and the PG status will be updated to active. Data recovery will be triggered based on the list of missing objects, and the device will begin receiving front-end business. It can be seen that changes in the object storage device status depend on the Monitor service being functioning normally. Only when the Monitor service is functioning normally will it propose to update the object storage device maintenance table, thereby triggering the PG peering process.

[0053] Therefore, a scenario arises where both the object storage service and the monitor service fail simultaneously. Upon recovery, some object storage services may not have recovered, causing the cluster to fail to recover (peering gets stuck), blocking front-end services. For example, consider a scenario where the entire data center experiences an abnormal power outage. After recovery, one node fails to start. When the entire data center is powered down, the monitor service also fails because the object storage service failed. Therefore, the monitor service cannot process the object storage device status change protocol and cannot mark the failed object storage devices as down. These failed object storage devices remain in the up state in the object storage device maintenance table of the database. When the data center recovers, the monitor first reads and restores historical information from the database (at this point, the failed object storage devices in the object storage device maintenance table are all in the up state). Then, as the object storage service starts, the monitor receives some object storage device... The Monitor updates and publishes a new version of the object storage device maintenance table to reflect the status of these object storage devices (in this version, the object storage device of the node that failed to start is still in the up state). When this version of the object storage device maintenance table is pushed to the object storage device side, the members in its placement group assume that all object storage devices are up. Therefore, it needs to collect information from all PG records. However, some object storage device services are not running and therefore cannot respond to message interactions during peering. This causes the PG to wait for messages from the non-started PG members during peering, making it impossible to complete the peering process and receive business read / write operations, resulting in front-end business read / write operations being stuck.

[0054] Therefore, to address the issue that in certain use cases, a system-wide failure or simultaneous failure of multiple nodes in the storage system can cause the Monitor service to malfunction and fail to correctly update the object storage device status information. This can lead to a situation where, upon recovery, a small number of nodes fail to recover, resulting in an overall abnormal cluster state and inability to receive business read / write operations. This application provides a method for handling node failures in a distributed storage system, applied to the main monitor node (master monitor) of the storage system. Figure 1 As shown, it includes:

[0055] S11: When recovering from a fault, send lease messages to each backup monitor node.

[0056] Specifically, when a fault occurs, the Monitor service is also malfunctioning, making it impossible to determine the fault situation. Therefore, this embodiment applies to fault recovery. After fault recovery, the Monitor service will first start and complete the election. After the Monitor completes the election, a lease message will be sent between the primary and backup monitor nodes to maintain the heartbeat connection between the primary and backup monitor nodes.

[0057] The primary monitor node issues leases to backup monitors at fixed intervals. Only backup monitors holding leases can provide read services to external users; otherwise, the data on the monitor node is considered outdated and cannot be provided with services.

[0058] S12: Obtain the lease response messages returned by each backup monitor node to obtain the startup information of each backup monitor node.

[0059] The system startup information for each backup monitor node is collected based on the lease response messages returned by each backup monitor node.

[0060] S13: Determine whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node.

[0061] Whether a startup is normal can be determined based on the startup information of each backup monitor node. This embodiment does not limit the specific conditions for determining normal startup; they can be set according to actual needs.

[0062] In addition, this embodiment does not set the number or proportion of node failures under the condition of majority node failure; it can be set according to actual needs.

[0063] If a majority of nodes fail, execute S14: update all object storage devices in the object storage device maintenance table to "down" and publish the updated object storage device maintenance table.

[0064] The normal state of an object storage device is UP. However, when an object storage device malfunctions, such as a node shutdown or disk failure, the state of the object storage device will switch to DOWN (in this state, the object storage device cannot write data).

[0065] The object storage device maintenance table is updated based on the startup information of each backup monitor node. This avoids the storage system mistakenly updating the object storage device's online status in the object storage device maintenance table after the object storage device fails and goes offline, which would cause the storage system to be in a constant peering process. This shortens the peering process, allowing the front-end services served by the storage system to recover to the pre-failure level more quickly, and improves the stability of the storage system in the event of a failure of multiple nodes.

[0066] In this way, the object storage device will receive an object storage device maintenance table showing all object storage devices as down. At this point, because all members are offline, the PG will not trigger peering. Subsequently, as the services of each node's object storage device gradually start up, it will instruct the monitoring node to mark the corresponding object storage device status as up. Afterward, the object storage device will receive a new version of the object storage device maintenance table. In this version, object storage devices in the up state are all object storage devices that are functioning normally. During the peering process, the PG can smoothly collect the information recorded by each member through message exchange, elect a master PG, update the PG status, and start receiving business read and write operations, thus avoiding the problem of front-end business being blocked.

[0067] For potential mislabeling issues (such as some nodes not actually failing but being marked as down in the object storage device status), the corresponding object storage devices of these nodes will resend boot messages to inform the monitor node to restore the up state of their object storage device maintenance tables, and there will be no problem.

[0068] By collecting startup information from each monitor node system, it is determined whether it is a large-scale failure scenario. If so, it is necessary to proactively change the status of the object storage device in the object storage device maintenance table to solve the problem that the monitor service cannot accurately update the object storage device status when a failure occurs in a batch failure scenario.

[0069] The distributed storage system node failure handling method provided in this embodiment is applied to the primary monitor node of the storage system. Upon failure recovery, it sends lease messages to each backup monitor node; obtains lease response messages from each backup monitor node to acquire their startup information; determines whether a majority node failure has occurred based on the startup information of each backup monitor node; if so, it updates all object storage devices in the object storage device maintenance table to "down" and publishes the updated object storage device maintenance table. This avoids the storage system mistakenly updating the online status of object storage devices before the failure in the object storage device maintenance table after an object storage device fails and becomes offline, causing the storage system to be in a continuous peering process. This shortens the peering process, allowing the front-end services served by the storage system to recover to pre-failure levels more quickly, thus improving the stability of the storage system in majority node failure scenarios. It also improves the stability of the storage system in data center failures or large-scale node failure scenarios, enhancing the storage system's ability to withstand risks.

[0070] Based on the above embodiments, this embodiment provides a preferred solution, which includes obtaining the lease response messages returned by each backup monitor node to obtain the startup information of each backup monitor node, including:

[0071] The marking status of each backup monitor node is obtained based on the lease response message. The marking status includes true and false.

[0072] The startup information of each backup monitor node is obtained by marking its status.

[0073] If each backup monitor node starts successfully, it returns a lease response message with the status marked as true; if the startup fails, it returns a lease response message with the status marked as false.

[0074] The master node can accurately and quickly determine the startup information of the backup monitor node based on the flag status.

[0075] Based on the above embodiments, this embodiment provides a preferred solution, which includes determining whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node, including:

[0076] Get the first number of items whose flag state is true;

[0077] Determine whether the first quantity is greater than the first threshold;

[0078] If so, then it is determined that a majority of nodes in the storage system have failed;

[0079] If not, then it is determined that the storage system has not experienced a majority node failure.

[0080] The primary monitor node receives lease response messages from each backup monitor node, counts the number of nodes with a marked status of true, and records this as the first count. A first threshold is pre-set based on actual needs. If the number of nodes with a marked status of true exceeds the first threshold, it is determined that a majority node failure has occurred in the storage system; otherwise, it is determined that no majority node failure has occurred in the storage system.

[0081] Correspondingly, based on the backup monitor node, this embodiment provides another preferred solution, applied to the backup monitor node of the storage system, such as... Figure 2 As shown, the methods for handling node failures in a distributed storage system include:

[0082] S21: During fault recovery, obtain the lease message sent by the main monitor node;

[0083] S22: Return a lease response message to the primary monitor node so that the primary monitor node can obtain the startup information of the backup monitor nodes. This allows the primary monitor node to determine whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node. When a majority node failure occurs in the storage system, all object storage devices in the object storage device maintenance table are updated to "down" and the updated object storage device maintenance table is published.

[0084] According to the distributed storage system node failure handling method provided in this embodiment, during failure recovery, the system obtains the lease message sent by the primary monitor node and the lease response message returned to the primary monitor node, so that the primary monitor node can obtain the startup information of the backup monitor nodes. This allows the primary monitor node to determine whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node. When a majority node failure occurs, all object storage devices in the object storage device maintenance table are updated to "down," and the updated object storage device maintenance table is published. This avoids the storage system mistakenly updating the online status of the object storage devices before the failure in the object storage device maintenance table after the object storage devices fail and become offline, causing the storage system to be in a continuous peering process. This shortens the peering process, allowing the front-end services served by the storage system to recover to the pre-failure level more quickly, thus improving the stability of the storage system in majority node failure scenarios. This enhances the stability of the storage system in data center failures or large-scale node failure scenarios, strengthening the storage system's ability to withstand risks.

[0085] Correspondingly, according to the above embodiments, the lease response message returned to the main monitor node includes:

[0086] Return the marking status to the main monitor node. The marking status can be either true or false.

[0087] Each standby monitor node sets a flag status based on its actual startup status and returns it to the main monitor node.

[0088] Based on the above embodiments, this embodiment provides a preferred solution, in which returning the marking status to the main monitor node includes:

[0089] Calculate the time difference between the current time and the storage system's startup time;

[0090] The marking status is determined based on the time difference;

[0091] Return the flagged status to the main monitor node.

[0092] This embodiment determines whether startup was successful by using the time difference between the current time of each backup monitor node after receiving the lease message and the startup time of the storage system. If each backup monitor node starts successfully, it returns a lease response message with a flag of true; if startup fails, it returns a lease response message with a flag of false.

[0093] Specifically, the marking status is determined based on the time difference:

[0094] Determine whether the time difference is less than the second threshold;

[0095] If so, then the flag state is determined to be true;

[0096] If not, the flag status is determined to be false.

[0097] For example, after receiving the lease message, the standby monitor node calculates the time difference between the current time and the startup time of the storage system, diff_time. If diff_time is less than the second threshold (e.g., 200s), it is considered that the first election after the node restart has been completed, and the flag status is set to true; otherwise, the flag status is set to false.

[0098] In the above embodiments, the method for handling node failures in a distributed storage system has been described in detail. This application also provides embodiments corresponding to a device for handling node failures in a distributed storage system. It should be noted that this application describes the embodiments of the device part from two perspectives: one is based on functional modules, and the other is based on hardware.

[0099] From the perspective of functional modules Figure 3 A structural diagram of a distributed storage system node failure handling system provided in this application embodiment is shown below. Figure 3 As shown,

[0100] The sending module 31 is used to send lease messages to each backup monitor node when the fault is recovered;

[0101] The acquisition module 32 is used to acquire the lease response messages returned by each backup monitor node to obtain the startup information of each backup monitor node;

[0102] The judgment module 33 is used to determine whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node;

[0103] The update module 34 is used to update all object storage devices in the object storage device maintenance table to "down" and publish the updated object storage device maintenance table when a majority of nodes in the storage system fail.

[0104] The distributed storage system node failure handling system provided in this embodiment includes a sending module 31 that sends lease messages to each backup monitor node during failure recovery; an acquisition module 32 that acquires lease response messages returned by each backup monitor node to obtain the startup information of each backup monitor node; a judgment module 33 that determines whether a majority node failure has occurred in the storage system based on the startup information of each backup monitor node; and an update module 34 that, when a majority node failure occurs in the storage system, updates all object storage devices in the object storage device maintenance table to "down" and publishes the updated object storage device maintenance table. This avoids the storage system mistakenly updating the online status of the object storage devices before the failure in the object storage device maintenance table after the object storage devices fail and become offline, causing the storage system to be in a continuous peering process. This shortens the peering process, allowing the front-end services served by the storage system to recover to the pre-failure level more quickly, thus improving the stability of the storage system in majority node failure scenarios. It also improves the stability of the storage system in scenarios of data center failures or large-scale node failures, enhancing the storage system's ability to resist risks.

[0105] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0106] Figure 4 A structural diagram of another distributed storage system node fault handling device provided in the embodiments of this application is shown below. Figure 4 As shown, the device for handling node failures in a distributed storage system includes: a memory 40 for storing computer programs;

[0107] The processor 41 is used to implement the steps of the method for obtaining user operation habit information as described in the above embodiment (method for handling node failures in a distributed storage system) when executing a computer program.

[0108] The distributed storage system node failure handling device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0109] The processor 41 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 41 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 41 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 41 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 41 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0110] The memory 40 may include one or more computer-readable storage media, which may be non-transitory. The memory 40 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 40 is used to store at least the following computer program 401, which, after being loaded and executed by the processor 41, is capable of implementing the relevant steps of the distributed storage system node failure handling method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 40 may also include an operating system 402 and data 403, and the storage method may be temporary or permanent storage. The operating system 402 may include Windows, Unix, Linux, etc. The data 403 may include, but is not limited to, the data involved in implementing the distributed storage system node failure handling method.

[0111] In some embodiments, the device for handling node failures in a distributed storage system may further include a display screen 42, an input / output interface 43, a communication interface 44, a power supply 45, and a communication bus 46.

[0112] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the handling device for node failures in a distributed storage system and may include more or fewer components than shown.

[0113] The distributed storage system node failure handling apparatus provided in this application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: A distributed storage system node failure handling method, when the failed storage system recovers, the primary monitor node sends lease messages to each backup monitor node to obtain the startup information of each backup monitor node. Based on the startup information, it determines whether a majority node failure has occurred. When a majority node failure is confirmed, all object storage devices in the object storage device maintenance table are updated to "down" and the updated object storage device maintenance table is published. This avoids the storage system mistakenly updating the online status of the object storage devices before the failure in the object storage device maintenance table after the object storage devices fail and become offline, causing the storage system to be in a continuous peering process. This shortens the peering process, allowing the front-end services served by the storage system to recover to the pre-failure level more quickly, improving the stability of the storage system in majority node failure scenarios. It also improves the stability of the storage system in data center failures or large-scale node failure scenarios, enhancing the storage system's ability to resist risks.

[0114] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above-described embodiment of the method for handling node failures in a distributed storage system (which may be a method corresponding to the server side, a method corresponding to the diagnostic device side, or a method corresponding to both the server side and the diagnostic device side).

[0115] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The computer-readable storage medium provided in this embodiment stores a computer program. When the processor executes the program, it can implement the following method: a method for handling node failures in a distributed storage system. When the storage system recovers from a failure, the primary monitor node sends lease messages to each backup monitor node to obtain the startup information of each backup monitor node. Based on the startup information, it determines whether a majority node failure has occurred. When a majority node failure is confirmed, all object storage devices in the object storage device maintenance table are updated to "down," and the updated object storage device maintenance table is published. This avoids the storage system mistakenly updating the online status of the object storage devices before the failure in the object storage device maintenance table after the object storage devices fail and become offline, causing the storage system to be in a continuous peering process. This shortens the peering process, allowing the front-end services served by the storage system to recover to the pre-failure level more quickly, and improving the stability of the storage system in majority node failures. It also improves the stability of the storage system in scenarios of data center failures or large-scale node failures, enhancing the storage system's ability to resist risks.

[0117] The foregoing has provided a detailed description of the method, apparatus, system, and medium for handling node failures in a distributed storage system provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0118] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for handling node failures in a distributed storage system, characterized in that, The main monitor node used in the storage system includes: During fault recovery, a lease message is sent to each backup monitor node; Obtain the lease response message returned by each of the backup monitor nodes to obtain the startup information of each of the backup monitor nodes; Determine whether the storage system has experienced a majority node failure based on the startup information of each backup monitor node; If so, update all object storage devices in the object storage device maintenance table to "down" and publish the updated object storage device maintenance table; The step of obtaining the lease response messages returned by each of the backup monitor nodes to obtain the startup information of each backup monitor node includes: The marking status of each backup monitor node is obtained based on the lease response message, and the marking status includes true and false; The startup information of each backup monitor node is obtained through the marked status; The step of determining whether the storage system has experienced a majority node failure based on the startup information of each of the backup monitor nodes includes: obtaining a first number of flags with a status of true; determining whether the first number is greater than a first threshold; if yes, determining that the storage system has experienced a majority node failure; if no, determining that the storage system has not experienced a majority node failure. The backup monitor node is used to return a marking status to the primary monitor node, including: calculating the time difference between the current time and the startup time of the storage system; wherein the current time is the time after each backup monitor node receives the lease message; determining the marking status based on the time difference; and returning the marking status to the primary monitor node; The backup monitor node is used to determine the marking status based on the time difference value, including: determining whether the time difference value is less than a second threshold; if yes, then determining the marking status as true; if no, then determining the marking status as false.

2. A method for handling node failures in a distributed storage system, characterized in that, Backup monitor nodes used in storage systems include: During fault recovery, obtain the lease message sent by the main monitor node; The primary monitor node returns a lease response message to the primary monitor node so that it can obtain the startup information of the backup monitor nodes. Based on the startup information of each backup monitor node, the primary monitor node determines whether a majority node failure has occurred in the storage system. When a majority node failure occurs, the primary monitor node updates all object storage devices in the object storage device maintenance table to "down" and publishes the updated object storage device maintenance table. The primary monitor node obtains a first number of nodes with a flag of "true"; it then determines whether this first number is greater than a first threshold. If so, it determines that a majority node failure has occurred in the storage system; otherwise, it determines that a majority node failure has not occurred. The lease response message returned to the main monitor node includes: Return a marking status to the main monitor node, the marking status including true and false; The step of returning the marker status to the main monitor node includes: Calculate the time difference between the current time and the startup time of the storage system; where the current time is the time after each backup monitor node receives the lease message; The marking state is determined based on the time difference value; Return the marked status to the main monitor node; The step of determining the marker state based on the time difference includes: Determine whether the time difference is less than the second threshold; If so, then the flag state is determined to be true; If not, the flag state is determined to be false.

3. A system for handling node failures in a distributed storage system, characterized in that, The main monitor node used in the storage system includes: The sending module is used to send lease messages to each backup monitor node when a fault is recovered; The acquisition module is used to acquire the lease response messages returned by each of the backup monitor nodes to obtain the startup information of each of the backup monitor nodes; The judgment module is used to determine whether the storage system has experienced a majority node failure based on the startup information of each of the backup monitor nodes; The update module is used to update all object storage devices in the object storage device maintenance table to "down" and publish the updated object storage device maintenance table when a majority of nodes of the storage system fail. The step of obtaining the lease response messages returned by each of the backup monitor nodes to obtain the startup information of each backup monitor node includes: The marking status of each backup monitor node is obtained based on the lease response message, and the marking status includes true and false; The startup information of each backup monitor node is obtained through the marked status; The step of determining whether a majority node failure has occurred in the storage system based on the startup information of each of the backup monitor nodes includes: Obtain the first number of items whose flag state is true; Determine whether the first quantity is greater than the first threshold; If so, then it is determined that a majority of nodes in the storage system have failed; If not, then it is determined that the storage system has not experienced a majority node failure; The backup monitor node is used to return a marking status to the primary monitor node, including: calculating the time difference between the current time and the startup time of the storage system; wherein the current time is the time after each backup monitor node receives the lease message; determining the marking status based on the time difference; and returning the marking status to the primary monitor node; The backup monitor node is used to determine the marking status based on the time difference value, including: determining whether the time difference value is less than a second threshold; if yes, then determining the marking status as true; if no, then determining the marking status as false.

4. A device for handling node failures in a distributed storage system, characterized in that, Includes memory used to store computer programs; A processor, configured to implement the steps of the method for handling node failures in a distributed storage system as described in claim 1 or 2 when executing the computer program.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for handling node failures in a distributed storage system as described in claim 1 or 2.

Images