Fault processing method and apparatus, electronic device, and storage medium

Abstract

The present application relates to the technical field of computers. Disclosed are a fault processing method and apparatus, an electronic device, and a storage medium. The method comprises: when an error occurs during the execution of a target event, terminating the execution of the target event, modifying a cluster state to a recovery state, and modifying the event level of a node itself according to a preset rule, wherein after modifying the cluster state to the recovery state, a master node submits a first cluster recovery event and a second cluster recovery event in sequence; in the recovery state, processing a node state modification event by means of a target service module control layer of the node itself; when the first cluster recovery event is received, retaining critical information, and discarding non-critical information, wherein the critical information comprises configuration information; and when the second cluster recovery event is received, querying the target service module control layer of the node itself and updating the latest node state. The present application improves the efficiency of fault processing and cluster recovery.

Description

A fault handling method, apparatus, electronic device, and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese Patent Application No. 202410508482.5, filed on April 25, 2024, entitled "A Fault Handling Method, Apparatus, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and more specifically, to a fault handling method, apparatus, electronic device, and storage medium. Background Technology

[0004] Distributed storage clusters typically rely on a distributed consensus protocol to build a consensus framework. This framework ensures consistent state within the cluster, coordinating the behavior of each node to achieve high scalability and high availability. The consensus framework includes a set of control state machine sub-components and corresponding application clients. Under the coordination of the consensus framework, the control state machine on each node consistently reads and writes the cluster state and controls the application clients on their respective nodes to perform the same actions under the same state, thus achieving coordinated action among the nodes within the cluster.

[0005] The inventors realized that under normal operating conditions, all nodes within a cluster maintain a consistent state and behavior. However, if the cluster state undergoes an abnormal change, this outlier will be read by the state machines on all nodes. Because all state machines behave consistently, all nodes will terminate their business processes upon reading the same outlier, causing both the business and the cluster to crash. In related technologies, such problems heavily rely on manual recovery by implementation personnel, resulting in low efficiency in fault handling and cluster recovery.

[0006] Therefore, how to improve the efficiency of fault handling and cluster recovery is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0007] This application provides a fault handling method applied to nodes in a distributed storage cluster. The fault handling method includes:

[0008] When an error occurs during the execution of the target event, the execution of the target event is terminated, the cluster state is changed to a recovery state, and its own event level is modified according to preset rules; among them, after the master node changes the cluster state to a recovery state, it submits the first cluster recovery event and the second cluster recovery event in sequence;

[0009] In the recovery state, node state modification events are handled through the control layer of its own target business module.

[0010] When the first cluster recovery event is received, critical information is retained and non-critical information is discarded; the critical information includes configuration information.

[0011] When a second cluster recovery event is received, the system queries and updates the latest node status to the control layer of its target business module.

[0012] The fault handling method, after querying and updating the latest node status from the target business module control layer, also includes:

[0013] After receiving the target number of basic events broadcast by the master node, the target number of basic events is synchronized to update its own event level; where the target number is greater than the depth of the recent event queue.

[0014] In this process, after the target node in the distributed storage cluster queries and updates the latest node status to its own target business module control layer, it sends a target number of basic events to the master node so that the master node can broadcast the target number of basic events to the nodes participating in the cluster recovery.

[0015] After sending the basic event for the target quantity to the master node, the fault handling method also includes:

[0016] When a recovery completion event is received, the cluster status is changed from recovery status to normal status; specifically, the target node in the distributed storage cluster submits the recovery completion event after sending the target number of basic events to the master node.

[0017] After changing the cluster status from recovery to normal, the fault handling method also includes:

[0018] Nodes that did not participate in cluster recovery update their event levels by making a full copy of the cluster state.

[0019] The nodes participating in cluster recovery are determined based on the event level of the nodes in the distributed storage cluster after the event level of the nodes is modified according to preset rules.

[0020] Among them, the target number of basic events is the target number of empty events.

[0021] After the master node changes the cluster status to the recovery status, when there are no nodes in the suspended state, it submits the first cluster recovery event and the second cluster recovery event in sequence through its own business module control layer.

[0022] Among them, node status modification events include any one or a combination of any of the following: node addition event, node suspension event, node unsustainment event, and node removal event.

[0023] The key information includes any one or a combination of any of the following: disk array configuration information, storage pool configuration information, and virtual volume configuration information.

[0024] Non-critical information includes any one or a combination of items from disk array path status, virtual volume path status, node status, reset input / output statistics, log information, and error logs.

[0025] The fault handling methods also include:

[0026] If a node state modification event is received during the processing of the first cluster recovery event and the second cluster recovery event, the process will re-enter the step of processing the node state modification event through its own target business module control layer.

[0027] Among these, terminating the execution of the target event when an error occurs includes:

[0028] When a code assertion occurs when the target event is executed on the first cluster replica, the execution of the target event is terminated.

[0029] Among these, terminating the execution of the target event when an error occurs includes:

[0030] When the target event is executed on the first cluster replica, if a code assertion occurs in the business module control layer, the execution of the target event is terminated.

[0031] Before changing the cluster state to a recovery state, the fault handling methods also include:

[0032] A new first cluster replica is obtained by overwriting the first cluster replica with the second cluster replica.

[0033] Changing the cluster status to recovery status includes:

[0034] Change the cluster status of the new first cluster replica to the recovery state, switch to the second cluster replica, and change the cluster status of the second cluster replica to the recovery state.

[0035] The fault handling methods also include:

[0036] When an event to be executed is received, the cluster status is determined;

[0037] If the cluster status is in recovery state, determine whether the event to be executed is the first cluster recovery event, the second cluster recovery event, or a node status modification event;

[0038] If not, then skip the execution of the pending events.

[0039] After determining the cluster status, the fault handling methods also include:

[0040] If the cluster status is normal, determine whether the event to be executed is the first cluster recovery event or the second cluster recovery event;

[0041] If so, then skip the execution of the pending event;

[0042] If not, the pending event will be executed directly.

[0043] To achieve the above objectives, this application provides a fault handling device applied to a node in a distributed storage cluster. The fault handling device includes:

[0044] The first modification module is used to terminate the execution of the target event when an error occurs, change the cluster state to a recovery state, and modify its own event level according to preset rules; wherein, after the master node changes the cluster state to a recovery state, it submits the first cluster recovery event and the second cluster recovery event in sequence.

[0045] The processing module is used to handle node state modification events through its own target business module control layer in the recovery state;

[0046] The discard module is used to retain critical information and discard non-critical information when the first cluster recovery event is received; the critical information includes configuration information.

[0047] The first update module is used to query and update the latest node status from the control layer of its target business module when it receives the second cluster recovery event.

[0048] To achieve the above objectives, this application provides an electronic device, comprising:

[0049] Memory, used to store computer programs;

[0050] A processor is used to implement the steps of the fault handling method described above when executing a computer program. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings are used to provide a further understanding of this disclosure and constitute a part of the specification. They are used together with the following detailed description to explain this disclosure, but do not constitute a limitation of this disclosure. In the drawings:

[0052] Figure 1 is an architecture diagram of nodes in a distributed storage cluster according to one or more embodiments;

[0053] Figure 2 is a flowchart illustrating a fault handling method according to one or more embodiments;

[0054] Figure 3 is a flowchart illustrating another fault handling method according to one or more embodiments;

[0055] Figure 4 is a structural diagram of a fault handling device according to one or more embodiments;

[0056] Figure 5 is a structural diagram of an electronic device according to one or more embodiments. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, in the embodiments of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] This application applies to distributed storage clusters, which consist of multiple servers interconnected via a network. Each storage server has back-end disks or back-end disk enclosures. These disks can be shared within the storage cluster, which can implement multi-layered organization and virtualization on these back-end disks, including storage pools, RAID (Redundant Arrays of Independent Disks) groups, and virtual disks, to provide higher performance, throughput, availability, and other diverse data storage and access services compared to single-machine storage. The storage server cluster and front-end hosts are connected to the same front-end network, through which the storage servers provide storage services (virtual disks) to the front-end hosts.

[0059] In a distributed storage cluster, critical data needs to be shared within the cluster to avoid data inaccessibility due to single points of failure. To achieve this, each node in the storage cluster runs the same cluster and application software. The architecture of each node is shown in Figure 1, including a consistency protocol layer, a business module control layer, and a business module application layer.

[0060] The consistency protocol layer is used to maintain the existence of the cluster, collect and distribute cluster events, and provide a cluster state space to business modules.

[0061] Specifically, when the network links between all nodes in the cluster are normal, the consensus protocol layer maintains a cluster heartbeat to confirm that all nodes in the cluster are active. When a node loses its heartbeat or a link failure occurs, the consensus protocol layer on each node calculates whether the number of nodes in a certain network partition exceeds half of the total number of nodes in the previous stable cluster, and allows that network partition to take over the cluster.

[0062] The event collection of the consensus protocol layer is directed to each business module. Each business module can send events to the consensus protocol layer, and the event distribution of the consensus protocol layer can ensure that each node receives the same event sequence.

[0063] The consistency protocol layer provides a cluster state space to business modules, which can read and write data using the fixed interfaces provided by the consistency protocol layer. The consistency protocol layer guarantees that each node has the same initial cluster state and that all write operations performed by business modules on the cluster state are completely consistent, thus ensuring state consistency across all nodes. To ensure the atomicity of a series of write operations triggered by a single event, the cluster state has two completely consistent replicas: a first cluster replica and a second cluster replica. Business modules need to write to both replicas sequentially to complete the final state modification. Because there are two replicas, if a node fails while writing to either replica, the state of the other replica remains intact. Therefore, the cluster state can be restored to a consistent state by rolling back or rolling forward from the other replica.

[0064] Events in the cluster have monotonically increasing levels starting from zero. During event distribution, they are temporarily persisted on the nodes within the cluster. Once an event is executed, its purpose is fulfilled, and its level is updated to the latest cluster state level, thus no longer requiring storage space. Event sequence numbers grow indefinitely, but there isn't enough space to store all events indefinitely. Therefore, each node retains a limited number of events; these temporarily stored events are called recent events, and the maximum number of recent events is denoted as RENM (RECENT_EVENT_NUM_MAX). Recent events are persistently recorded and updated on a rolling basis, and their queue is called the recent event queue. Both copies of the cluster state and recent events are persistently stored and will not be lost due to hardware / software failures or power outages.

[0065] Successful event submission requires at least a majority of nodes in the cluster to return success. Submitted events are added to the tail of the recent event queue, awaiting execution. Events in the recent event queue are dequeued from the head for execution. Once an event completes its execution, it changes the cluster state, and then the next event is dequeued for execution. During normal operation, the execution order of events on all nodes in the cluster is consistent, but there is no hard guarantee of simultaneity.

[0066] The consensus protocol layer determines whether nodes belong to the same cluster based on two conditions: first, these nodes share a common unique cluster identifier; second, the nodes within the cluster have interconnected network links. Nodes meeting these conditions are automatically added to the cluster, while nodes that no longer meet these conditions are removed. When a node leaves, the consensus protocol layer checks if the remaining nodes in the cluster exceed half the number of nodes in the previous stable cluster. Only when the number of remaining nodes exceeds half can the node take over the cluster.

[0067] During cluster operation, if a node temporarily leaves the cluster due to network or software failure, its state upon returning may be outdated, requiring synchronization from other nodes. Depending on the node's state and the obsolescence of recent events, one of two recovery methods exists: If the node's recent events at the time of departure were in the range [N, N+RENM-1], and upon return, other nodes in the cluster still hold events at level N+RENM, then the missing events can simply be sent from the node containing those events to the lagging node. If the state levels of all cluster nodes have exceeded N+RENM (meaning N+RENM events are active and integrated into the cluster state), then synchronizing recent events is insufficient for the lagging node to catch up. In this case, the node must synchronize a complete cluster state from other nodes within the cluster.

[0068] Each business module can add a submodule to the business module control layer, the core of which is an event handling function. After receiving an event from the consistency protocol layer, the business module can invoke the corresponding logic to read and write the corresponding cluster state, and determine which specific actions to call in the business module application layer based on the state before and after the read and write.

[0069] Each business module can add several sub-modules to its application layer. These sub-modules are all controlled by the same control layer business sub-module. The application layer can send events to the consistency protocol layer as needed; each event can include a callback. This event is distributed by the consistency protocol layer to each node in the cluster, and then passed to the corresponding business module control layer within each node. After the control layer completes the writing of two replicas of the cluster state, it initiates the action of the corresponding application sub-module. Once the action is completed, it returns to the consistency protocol layer, which then finds the application layer sub-module that initiated the event and calls its callback included in the event.

[0070] The above architecture ensures that the cluster state is consistently replicated across all nodes within the cluster. Modifications to the cluster state are triggered by events, and the consistency protocol layer guarantees that each node in the cluster executes these events in the same order. This framework enables the storage cluster to perform complex node task coordination. However, if there is a problem in the event handling code of the business module control layer that causes node crashes, such as a valid value assertion in the code (the fault location is the business module control layer logic between steps 3-5 in Figure 1), then when this event is executed, because the consistency protocol layer distributes this event to all nodes in the cluster, all nodes will encounter this problem and crash. Worse still, because these events are persisted, this event will be re-executed after a node restart, ultimately causing all nodes in the cluster to crash repeatedly. Therefore, a single software problem can cause the cluster to crash, reducing the cluster's availability to a single-machine level and severely impacting business continuity.

[0071] A major challenge in cluster recovery is ensuring that all recovered nodes have the same copy of the cluster state upon completion of recovery, meaning that the cluster state is consistent across nodes. The following are some scenarios where inconsistency may occur:

[0072] 1. If only some nodes in the cluster have performed cluster recovery, the recovered cluster will consist of some nodes that performed recovery and others that did not. These two types of nodes will maintain different cluster states, so even if they receive the same sequence of events again, there is no guarantee that the nodes will respond in the same way.

[0073] 2. Inconsistency will also occur if all nodes perform cluster recovery but their initial cluster states differ. For example, a distributed storage cluster may have four nodes, A, B, C, and D, with three pending events X, Y, and Z. The consistency protocol layer guarantees that these three events are processed in order, but it cannot guarantee the precise timing of event processing. It is entirely possible that node A has completed processing all three events, but node B has only processed event X, while nodes C and D have not yet processed any events. If all nodes stop running and begin cluster recovery at this moment, it will also lead to inconsistencies in the cluster states across the nodes.

[0074] To avoid the inconsistency caused by starting cluster recovery at different points in event handling, one possible approach is to select one node in the cluster to perform the cluster recovery action and then replicate its state to all other nodes in the cluster after the recovery is complete.

[0075] However, cluster recovery also requires ensuring that the cluster state is consistent with the data stored on the backend disks. For example, if a data unit has been migrated from RAID A to RAID B and modified after the migration, it will cause data inconsistency if the cluster still believes that the data unit is on RAID A.

[0076] The cluster recovery method described above, which selects one node in the cluster to perform the cluster recovery action and then replicates its state to all other nodes in the cluster after recovery, can lead to inconsistencies between the cluster state and the data stored on the backend disk. For example, in scenario 2, assuming the cluster recovery only occurs on node B, the recovered cluster state will include the execution result of event X, but not the execution results of events Y and Z. However, node A will assume that events Y and Z have been completed and may have already issued I / O (input / output) requests or modified their read / write caches based on these results. Therefore, recovery from node B will invalidate these actions performed by node A.

[0077] To avoid the aforementioned inconsistency issues, all nodes can be stopped at the same precise point in the event sequence, triggering automatic cluster recovery. This recovery relies on the consistency of the consistency protocol layer to ensure all nodes return to a consistent state. However, even with this strategy, the exact number of nodes executing the recovery process is uncertain.

[0078] Consider the following scenarios:

[0079] 1. All nodes have the same cluster state, and a bad event will trigger an Assert statement on all nodes.

[0080] In this scenario, it might seem at first glance that all nodes would assert at the same location in the business control layer code and begin cluster reorganization simultaneously. In fact, this does happen in the most optimistic case. The reason this isn't inevitable is due to a boundary condition: nodes can only process events if a majority of nodes in the election set are online. Each node that successfully executes a bad event will assert, thus leaving the cluster. Once a majority of nodes leave the cluster, the remaining nodes will be unable to process any events and will also leave the cluster after their leases expire. Therefore, it's only possible to guarantee that a majority of nodes in the cluster can rejoin for cluster recovery.

[0081] In the aforementioned boundary case, before any node in the cluster processes the event, the consistency protocol layer ensures that every node in the view has obtained a persistent copy of the event. Once the master node confirms that every node in the view has obtained a copy of the event, it broadcasts a notification to all nodes that the event has been successfully committed and can be executed. The number of nodes that initiate cluster recovery depends on how many nodes receive this notification before a majority of nodes have processed the bad event and left the cluster via Assert. In the most optimistic case, all nodes receive this notification; in this scenario, all nodes will process the bad event, and therefore all nodes will initiate cluster recovery. The worst-case scenario is that the master node broadcasts a notification and immediately processes the bad event, causing the master node to assert itself. At this point, the broadcast has not yet reached any other nodes. The remaining nodes will elect a new master node, and the new master node will replay the bad event notification broadcast. In the worst-case scenario, the new master node can only broadcast the notification to itself. This process may continue until more than half of the nodes in the election set assert and exit. When less than half of the nodes remain in the cluster, the new master node will no longer be able to refresh the lease, and replay will not continue. All remaining nodes will have their leases expire.

[0082] 2. The cluster state of one node in the cluster is corrupted and inconsistent with that of other nodes, resulting in only this node being assigned an Assert.

[0083] At this point, the damaged node will begin cluster recovery, and the remaining nodes will operate normally. It is necessary to prevent the recovering node from rejoining the cluster or communicating with other nodes to prevent it from crashing the entire cluster and causing a single point of failure.

[0084] In addition, similar to the situation of entering cluster recovery, there are many issues regarding the timing of exiting cluster recovery.

[0085] First, cluster recovery exit must be coordinated in the same way as entering cluster recovery. The simplest method is to issue an event, which all nodes receive and exit recovery, returning to normal I / O processing. This method suffers from the same problem as mentioned above: it only guarantees that a majority of the nodes entering cluster recovery process the event and exit. The remaining nodes will have expired leases. However, as long as the exit is successful, nodes whose leases have expired will be able to replay the exit event and ignore it upon returning to the cluster. To achieve this, the control layer of the business module handling this event can notify the application layer of the business module on its own node to exit cluster recovery.

[0086] Secondly, resynchronization is required with nodes that did not participate in cluster recovery. While it's acceptable to have only a majority of nodes participate in cluster recovery, it's also essential that any other nodes within the cluster can rejoin once recovery occurs. Each node may hold a unique cache copy of a virtual volume; if a node fails to rejoin, its cached data will be lost. The consensus protocol layer has a robust mechanism for bringing temporarily missing nodes back into the cluster. Depending on the length of time a node has been away, the cluster uses one of two resynchronization methods: if the joining node is only a few events behind, these events are forwarded to it; if the joining node is significantly behind, the consensus protocol layer passes it a copy of the state from a node with the latest state. After cluster recovery, because a large number of state resets occur during recovery, all nodes that did not participate in cluster recovery rejoin the cluster by obtaining a copy of the cluster state, rather than replaying events. To ensure this, enough events can be processed during cluster recovery to prevent the consensus protocol layer from having sufficient event history space for replay, thus forcing a copy of the cluster state.

[0087] In some embodiments of this application, a fault handling method is disclosed, which improves the efficiency of fault handling and cluster recovery.

[0088] Referring to Figure 2, a flowchart of a fault handling method according to some embodiments is shown in Figure 2, including:

[0089] S101: When an error occurs during the execution of the target event, the execution of the target event is terminated, the cluster state is changed to the recovery state, and its own event level is modified according to preset rules; wherein, after the master node changes the cluster state to the recovery state, it submits the first cluster recovery event and the second cluster recovery event in sequence;

[0090] In some embodiments, the executing entities are the nodes in the distributed storage cluster. When the target event arrives at the business module control layer and an error occurs during the modification of the first cluster replica, the business module control layer terminates the execution of the target event and changes the cluster state from the normal state to the recovery state.

[0091] In some embodiments, when an error occurs during the execution of the target event, the execution of the target event is terminated, including: terminating the execution of the target event when a code assertion occurs during the execution of the target event on the first cluster replica. In a specific implementation, since the business module control layer modifies the first cluster replica and the second cluster replica in two completely identical phases of event execution, termination always occurs during the execution of the first cluster replica. When a failed assertion is encountered during the execution of the target event on the first cluster replica, the cluster state is changed from a normal state to a recovery state.

[0092] In some embodiments, before modifying the cluster state to a recovery state, the method further includes: overwriting the first cluster copy with a second cluster copy to obtain a new first cluster copy. In a specific implementation, all nodes in the cluster perform a state rollback, that is, overwriting the first cluster copy with a second cluster copy.

[0093] In some embodiments, changing the cluster state to a recovery state includes: changing the cluster state of a new first cluster replica to a recovery state, switching to a second cluster replica, and changing the cluster state of the second cluster replica to a recovery state. In a more specific implementation, the cluster state of the new first cluster replica is changed from a normal state to a recovery state, then the user switches to the second cluster replica, and the cluster state of the second cluster replica is changed from a normal state to a recovery state.

[0094] In some embodiments, when an error occurs during the execution of the target event, the execution of the target event is terminated, including: when executing the target event on the first cluster replica, if a code assertion occurs in the business module control layer, the execution of the target event is terminated. In a specific implementation, cluster recovery is automatically triggered when the business module control layer processes an event Assert. To avoid requiring all business module control layer developers to replace the Asserts in the business module control layer, the platform code and the consistency protocol layer cooperate to detect whether an Assert occurs during event processing. Asserts within the consistency protocol layer and Asserts in the business module application layer will not trigger cluster recovery. If cluster recovery is triggered, the IO process will exit to the outer process with a specific error code EXIT_RECOVER_CLUSTER. The outer process will recognize this exit code. When the IO process exits with this exit code, it will first create a coredump to capture enough information to locate the problem. After the coredump data file is written, the outer process will execute a new cluster_recovery process to perform cluster recovery. The behavior of the cluster_recovery process is largely consistent with that of the IO master process.

[0095] The `cluster_recovery` process initializes nodes as usual. During node initialization, the event levels of the nodes are modified according to the same preset rules to distinguish them from nodes that have not undergone cluster recovery. This ensures that nodes recovering do not form a cluster with unrecovered nodes. In other words, the nodes participating in cluster recovery are determined based on the event levels of the nodes in the distributed storage cluster after the event levels have been modified according to the preset rules. Nodes will start their communication links and ports in the usual way and attempt to form a cluster. Because the event levels of the nodes have been modified, nodes in the cluster will attempt to form a cluster with other nodes that have also entered cluster recovery mode, ignoring those nodes that have not entered recovery mode. Nodes will not take any further action until more than half of the nodes have entered cluster recovery mode and successfully formed a cluster. Once more than half of the nodes in the cluster have started the cluster recovery process and completed mutual discovery through the cluster communication links, the consistency protocol layer will start the cluster in the usual way and issue the necessary event sequence to synchronize all nodes with the cluster state. This process involves replaying uncommitted events and issuing corresponding node events based on node retention or demise. All components in the business module control layer register an alternative event handler to manage events during cluster recovery. With a few exceptions, the business module control layer ignores all events issued by the consistency protocol layer during the cluster_recovery process, so replaying bad events that triggered assertions during normal operation will not cause any problems.

[0096] For ordinary business module control layers other than the target business module control layer, only two events are processed during cluster recovery: the first cluster recovery event and the second cluster recovery event. The target business module control layer, a special control layer, exists in both the main IO process and the cluster_recovery process. In addition to processing the first and second cluster recovery events, it also processes node state modification events, ignoring all other events.

[0097] During the cluster recovery process, since the business module control layer ignores most events, this essentially puts the business module application layer into its initial dormant state.

[0098] In some embodiments, after the master node modifies the cluster state to a recovery state, when there are no nodes in a suspended state, it sequentially submits a first cluster recovery event and a second cluster recovery event through its own business module control layer.

[0099] In some embodiments, during cluster recovery, when no node is in a suspended state, the business module control layer of the master node broadcasts the first cluster recovery event and the second cluster recovery event in sequence. After receiving the first cluster recovery event and the second cluster recovery event, the business module control layer of the participating node calls the corresponding business module application layer to execute the first cluster recovery event and the second cluster recovery event in sequence.

[0100] S102: In the recovery state, node state modification events are handled through the control layer of its own target business module;

[0101] During cluster recovery, the control layer of the target business module on each node continues to process node modification events to ensure that the node status is always up-to-date. Node status modification events can include node addition events, node suspension events, node unsustainment events, node removal events, etc.

[0102] S103: When the first cluster recovery event is received, retain critical information and discard non-critical information; among which, critical information includes configuration information;

[0103] In some embodiments, when each node executes the first cluster recovery event, it retains critical information, performs an integrity check on the critical information, and then discards non-critical information. Critical information may include disk array configuration information, storage pool configuration information, virtual volume configuration information, etc. Disk array configuration information includes, for example, the backend controller, LUN (Logical Unit Number), heterogeneous LUN mode, and the naming of each disk array. Storage pool configuration information includes, for example, virtualization mapping tables and ongoing data migration tasks. Virtual volume configuration information includes, for example, the mapping between virtual volumes and hosts, subordinate IO groups, preferred nodes, and naming. Non-critical information may include disk array path status, virtual volume path status, node status, reset input / output statistics, log information, error logs, etc. In the cluster recovery state, the path status of all objects is set to offline. The relevant states of the application layer of the business module are discarded because all nodes are suspended. The cluster recovery business module control layer is queried to determine whether a node is online or offline, and its own business module control layer state is updated accordingly.

[0104] S104: When a second cluster recovery event is received, query the control layer of its target business module and update the latest node status.

[0105] In some embodiments, when each node executes the second cluster recovery event, the control layers of each business module are allowed to synchronize their states. Specifically, the target business module control layer provides a query function, which the aforementioned ordinary business module control layer can use to query and update the latest node state. The consistency protocol layer ensures through a two-phase execution mechanism of the first and second cluster recovery events that all events (including the first and second cluster recovery events) are either executed by all business module control layers or not executed by any business module control layer. This means that for the vast majority of updates to the cluster state, all business module control layers are consistent.

[0106] The fault handling method provided in this application terminates the execution of the target event when an error occurs, and modifies the cluster state to a recovery state. In the recovery state, the event level of each node is modified according to the same preset rules. Each node processes the node state modification event through its own target business module control layer. A first cluster recovery event retains critical information and discards non-critical information, while a second cluster recovery event queries and updates the latest node state. Therefore, the fault handling method provided in this application achieves automatic cluster recovery, improves fault handling and cluster recovery efficiency, and ensures that the event levels of nodes are consistent after recovery and that each node's state is the latest node state. This application also discloses a fault handling device, an electronic device, and a computer-readable storage medium, which can achieve the same technical effects.

[0107] In some embodiments, after querying and updating the latest node status from the target business module control layer, the method further includes: upon receiving a target number of basic events broadcast by the master node, synchronizing a target number of basic events to update its own event level; wherein the target number is greater than the recent event queue depth; wherein, after the target node in the distributed storage cluster queries and updates the latest node status from its own target business module control layer, it sends a target number of basic events to the master node, so that the master node broadcasts the target number of basic events to the nodes participating in cluster recovery.

[0108] In some embodiments, when executing a second cluster recovery event, one of the nodes in the distributed storage cluster, i.e., the target node, sends a target number of basic events to the master node. This target number is greater than the recent event queue depth, and its purpose is to elevate the node's event level. These basic events can be empty events. The master node broadcasts the target number of basic events to the nodes participating in cluster recovery. The nodes participating in cluster recovery execute the target number of basic events, synchronizing their event levels, thereby ensuring that the event levels of the nodes participating in cluster recovery are consistent after cluster recovery.

[0109] In some embodiments, after sending the target number of basic events to the master node, the method further includes: when a recovery completion event is received, changing the cluster state from recovery state to normal state; wherein, after the target node in the distributed storage cluster sends the target number of basic events to the master node, it submits the recovery completion event.

[0110] In some embodiments, after the target node sends a target number of basic events, the application layer of the target node's business module issues a recovery completion event. After receiving the recovery completion event, the control layer of the business module of other nodes calls its corresponding application layer to exit the cluster recovery state.

[0111] In some embodiments, after changing the cluster state from a recovery state to a normal state, the method further includes: nodes that did not participate in cluster recovery updating their own event levels by making a complete copy of the cluster state.

[0112] In some embodiments, after the cluster recovery is completed, in order for nodes that did not participate in the cluster recovery to rejoin the cluster, the nodes that did not participate in the cluster recovery need to synchronize their own event levels. Since the nodes that participated in the cluster recovery executed a target number of basic events, the rise in event levels is greater than the recent queue depth. Therefore, the nodes that did not participate in the cluster recovery update their own event levels by fully copying the cluster state replica, thereby enabling the nodes that did not participate in the cluster recovery to rejoin the cluster.

[0113] In some embodiments, the method further includes: if a node state modification event is received during the processing of a first cluster recovery event and a second cluster recovery event, then re-entering the step of processing the node state modification event through its own target business module control layer.

[0114] In some embodiments, if a node state modification event occurs during the processing of the first cluster recovery event and the second cluster recovery event, the process will re-enter step S102, process the node state modification event through the target business module control layer, and then execute the first cluster recovery event and the second cluster recovery event in sequence to perform node recovery again, ensuring that the event level of each node is consistent.

[0115] In some embodiments, the fault handling method provided in this application terminates the execution of the target event when an error occurs during the execution of the target event, and modifies the cluster state to a recovery state. In the recovery state, the event level of each node is modified according to the same preset rules. Each node processes the node state modification event through its own target business module control layer. A first cluster recovery event is used to retain critical information and discard non-critical information, and a second cluster recovery event is used to query and update the latest node state. Therefore, the fault handling method provided in this application embodiment achieves automatic cluster recovery, improves fault handling and cluster recovery efficiency, and ensures that the event levels of nodes are consistent after recovery and that the state of each node is the latest node state.

[0116] In some embodiments, this application discloses a fault handling method, specifically:

[0117] Referring to Figure 3, a flowchart of a fault handling method according to some embodiments is shown in Figure 3, including:

[0118] S201: When an event to be executed is received, determine the cluster status;

[0119] S202: If the cluster status is in recovery state, determine whether the event to be executed is the first cluster recovery event, the second cluster recovery event, or a node status modification event;

[0120] S203: If the event to be executed is a node state modification event, then the node state modification event is handled by the control layer of its own target business module;

[0121] S204: If the event to be executed is the first cluster recovery event, then retain the critical information and discard the non-critical information; the critical information includes configuration information.

[0122] S205: If the event to be executed is the second cluster recovery event, then query the control layer of its own target business module and update the latest node status;

[0123] S206: After receiving the target number of basic events broadcast by the master node, synchronize the target number of basic events in order to update its own event level;

[0124] S207: If the event to be executed is not any of the first cluster recovery event, the second cluster recovery event, or the node state modification event, then skip the execution of the event to be executed;

[0125] It should be noted that, in recovery mode, all events except for the first cluster recovery event, the second cluster recovery event, and the node state modification event will be ignored to avoid inconsistencies in node event levels caused by executing other events in recovery mode.

[0126] S208: When the cluster state is normal and the event to be executed is either the first cluster recovery event or the second cluster recovery event, skip the execution of the event to be executed. When the cluster state is normal and the event to be executed is neither the first cluster recovery event nor the second cluster recovery event, execute the event to be executed.

[0127] Specifically, when the cluster status is normal, it is determined whether the event to be executed is a first cluster recovery event or a second cluster recovery event. In response to determining that the event to be executed is a first cluster recovery event or a second cluster recovery event, the execution of the event to be executed is skipped. In response to determining that the event to be executed is neither a first cluster recovery event nor a second cluster recovery event, the event to be executed is executed directly.

[0128] It should be noted that under normal conditions, the business module control layer will track the status of nodes in the cluster, which can include node status, event level, index, etc. All cluster recovery events, including the first cluster recovery event and the second cluster recovery event, will be ignored to avoid erroneous cluster recovery under normal conditions.

[0129] The following describes a fault handling device provided by some embodiments of this application. The fault handling device described below and the fault handling method described above can be referred to each other.

[0130] Referring to FIG4, a structural diagram of a fault handling apparatus according to an exemplary embodiment is shown. As shown in FIG4, the fault handling apparatus includes:

[0131] The first modification module 100 is used to terminate the execution of the target event when an error occurs, modify the cluster state to a recovery state, and modify its own event level according to preset rules; wherein, after the master node modifies the cluster state to a recovery state, it submits the first cluster recovery event and the second cluster recovery event in sequence.

[0132] Processing module 200 is used to process node state modification events through its own target business module control layer in the recovery state;

[0133] The discard module 300 is used to retain critical information and discard non-critical information when the first cluster recovery event is received; the critical information includes configuration information.

[0134] The first update module 400 is used to query and update the latest node status from its target business module control layer when it receives a second cluster recovery event.

[0135] The fault handling apparatus provided in some embodiments of this application terminates the execution of the target event when an error occurs, and modifies the cluster state to a recovery state. In the recovery state, the event level of each node is modified according to the same preset rules. Each node processes the node state modification event through its own target business module control layer. A first cluster recovery event is used to retain critical information and discard non-critical information, while a second cluster recovery event is used to query and update the latest node state. Therefore, the fault handling apparatus provided in the embodiments of this application achieves automatic cluster recovery, improves fault handling and cluster recovery efficiency, and ensures that the event levels of nodes are consistent after recovery and that the state of each node is the latest node state.

[0136] In some embodiments, the fault handling apparatus further includes:

[0137] The synchronization module is used to synchronize a target number of basic events after receiving a target number of basic events broadcast by the master node, so as to update its own event level. Among them, the target number is greater than the recent event queue depth. After the target node in the distributed storage cluster queries and updates the latest node status from its own target business module control layer, it sends the target number of basic events to the master node so that the master node can broadcast the target number of basic events to the nodes participating in the cluster recovery.

[0138] In some embodiments, the fault handling apparatus further includes:

[0139] The second modification module is used to change the cluster status from recovery status to normal status when a recovery completion event is received; wherein, the target node in the distributed storage cluster submits the recovery completion event after sending the target number of basic events to the master node.

[0140] In some embodiments, the fault handling apparatus further includes:

[0141] The second update module is used by nodes that did not participate in cluster recovery to update their own event levels by making a complete copy of the cluster state.

[0142] In some embodiments, the nodes participating in cluster recovery are determined based on the event level of the nodes in the distributed storage cluster after modifying the event level of the nodes according to preset rules.

[0143] In some embodiments, the target number of basic events is the target number of empty events.

[0144] In some embodiments, after the master node modifies the cluster state to a recovery state, when there are no nodes in a suspended state, it sequentially submits a first cluster recovery event and a second cluster recovery event through its own business module control layer.

[0145] In some embodiments, node state modification events include any one or a combination of any of the following: node add event, node suspend event, node unsuspend event, and node remove event.

[0146] In some embodiments, the key information includes any one or a combination of any of the following: disk array configuration information, storage pool configuration information, and virtual volume configuration information.

[0147] In some embodiments, non-critical information includes any one or a combination of any of the following: disk array path status, virtual volume path status, node status, reset input / output statistics, log information, and error log.

[0148] In some embodiments, the fault handling apparatus further includes:

[0149] The receiving module is used to restart the workflow of the processing module 200 when a node state modification event is received during the processing of the first cluster recovery event and the second cluster recovery event.

[0150] In some embodiments, the first modification module 100 is specifically used to: terminate the execution of the target event when a code assertion occurs on the target event executed on the first cluster replica.

[0151] In some embodiments, the first modification module 100 is specifically used to: when executing a target event on a first cluster replica, if a code assertion occurs in the business module control layer, then terminate the execution of the target event.

[0152] In some embodiments, the fault handling apparatus further includes:

[0153] The overlay module is used to overwrite the first cluster copy with the second cluster copy to obtain a new first cluster copy.

[0154] In some embodiments, the first modification module 100 is specifically used to: modify the cluster state of the new first cluster replica to the recovery state, switch to the second cluster replica, and modify the cluster state of the second cluster replica to the recovery state.

[0155] In some embodiments, the fault handling apparatus further includes:

[0156] The determination module is used to determine the cluster status when an event to be executed is received;

[0157] The first judgment module is used to skip the execution of the event to be executed when the cluster is in a recovery state and the event to be executed is neither the first cluster recovery event, nor the second cluster recovery event, nor the node state modification event.

[0158] In some embodiments, the fault handling apparatus further includes:

[0159] The second judgment module is used to skip the execution of the event to be executed when the cluster status is normal and the event to be executed is either the first cluster recovery event or the second cluster recovery event; and to execute the event to be executed when the cluster status is normal and the event to be executed is neither the first cluster recovery event nor the second cluster recovery event.

[0160] Regarding the apparatus in some of the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0161] Based on the hardware implementation of the above program modules, and in order to implement the methods of some embodiments of this application, some embodiments of this application also provide a computer device including a memory and one or more processors. The memory stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors cause the one or more processors to perform the fault handling method provided in any of the above embodiments.

[0162] In one or more embodiments, the computer device may be a server, and its internal structure may be as shown in Figure 5. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-configuration power supply method for a protection circuit.

[0163] Those skilled in the art will understand that the structure shown in Figure 5 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0164] In one or more embodiments, this application also provides one or more non-volatile computer-readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the fault handling method provided in any of the above embodiments.

[0165] The computer-readable storage medium may be the computer-readable storage medium in the computer device shown in Figure 5.

[0166] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0167] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0168] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A fault handling method, characterized in that, The method, applied to nodes in a distributed storage cluster, includes: When an error occurs during the execution of the target event, the execution of the target event is terminated, the cluster state is changed to a recovery state, and its own event level is modified according to preset rules; wherein, after the master node changes the cluster state to a recovery state, it submits the first cluster recovery event and the second cluster recovery event in sequence; In the recovery state, node state modification events are processed through the target business module control layer. Upon receiving the first cluster recovery event, key information is retained while non-key information is discarded; wherein, the key information includes configuration information; and When the second cluster recovery event is received, the system queries and updates the latest node status to its target business module control layer.

2. The fault handling method according to claim 1, characterized in that, After querying and updating the latest node status from the control layer of the target business module, the method further includes: Upon receiving the target number of basic events broadcast by the master node, the system synchronizes the target number of basic events to update its own event level; wherein the target number is greater than the recent event queue depth; and In this process, after the target node in the distributed storage cluster queries and updates the latest node status to its own target business module control layer, it sends a target number of basic events to the master node, so that the master node can broadcast the target number of basic events to the nodes participating in the cluster recovery.

3. The fault handling method according to claim 2, characterized in that, After sending the target number of basic events to the master node, the method further includes: When a recovery completion event is received, the cluster status is changed from recovery status to normal status; wherein, the target node in the distributed storage cluster submits the recovery completion event after sending the target number of basic events to the master node.

4. The fault handling method according to claim 3, characterized in that, After changing the cluster state from a recovery state to a normal state, the method further includes: Nodes that did not participate in cluster recovery update their event levels by making a full copy of the cluster state.

5. The fault handling method according to claim 2, characterized in that, The nodes participating in cluster recovery are determined based on the event levels of the nodes in the distributed storage cluster after modifying the event levels according to preset rules.

6. The fault handling method according to claim 2, characterized in that, The target number of basic events is the target number of empty events.

7. The fault handling method according to claim 1, characterized in that, After the master node changes the cluster status to the recovery status, when there are no nodes in the suspended state, it submits the first cluster recovery event and the second cluster recovery event sequentially through its own business module control layer.

8. The fault handling method according to claim 1, characterized in that, The node status modification events include any one or a combination of any of the following: node addition event, node suspension event, node unsustainment event, and node removal event.

9. The fault handling method according to claim 1, characterized in that, The key information includes any one or a combination of any of the following: disk array configuration information, storage pool configuration information, and virtual volume configuration information.

10. The fault handling method according to claim 1, characterized in that, The non-critical information includes any one or a combination of any of the following: disk array path status, virtual volume path status, node status, reset input / output statistics, log information, and error log.

11. The fault handling method according to claim 1, characterized in that, The method further includes: When the node state modification event is received during the processing of the first cluster recovery event and the second cluster recovery event, the process re-enters the step of processing the node state modification event through its own target business module control layer.

12. The fault handling method according to claim 1, characterized in that, The step of terminating the execution of the target event when an error occurs includes: When a code assertion occurs when the target event is executed on the first cluster replica, the execution of the target event is terminated.

13. The fault handling method according to claim 12, characterized in that, The step of terminating the execution of the target event when an error occurs includes: When the target event is executed on the first cluster replica, and a code assertion occurs in the business module control layer, the execution of the target event is terminated.

14. The fault handling method according to claim 12, characterized in that, Before modifying the cluster state to a recovery state, the process also includes: A new first cluster replica is obtained by overwriting the first cluster replica with a second cluster replica.

15. The fault handling method according to claim 14, characterized in that, The step of changing the cluster state to a recovery state includes: Change the cluster status of the new first cluster replica to recovery status, switch to the second cluster replica, and change the cluster status of the second cluster replica to recovery status.

16. The fault handling method according to claim 1, characterized in that, The method further includes: When an event to be executed is received, the cluster status is determined; When the cluster state is in the recovery state, and the event to be executed is neither the first cluster recovery event, nor the second cluster recovery event, nor the node state modification event, the execution of the event to be executed is skipped.

17. The fault handling method according to claim 16, characterized in that, After determining the cluster state, the method further includes: When the cluster status is normal and the event to be executed is either the first cluster recovery event or the second cluster recovery event, the execution of the event to be executed is skipped. When the cluster is in a normal state, and the event to be executed is neither the first cluster recovery event nor the second cluster recovery event, the event to be executed is executed.

18. The fault handling method according to claim 1, characterized in that, The method further includes: The nodes participating in cluster recovery are determined based on the event level, so that the nodes participating in cluster recovery constitute a cluster.

19. A fault handling device, characterized in that, The device is used in nodes of a distributed storage cluster and includes: The first modification module is used to terminate the execution of the target event when an error occurs, modify the cluster state to a recovery state, and modify its own event level according to preset rules; wherein, after the master node modifies the cluster state to a recovery state, it submits the first cluster recovery event and the second cluster recovery event in sequence. The processing module is used to process node state modification events through its own target business module control layer in the recovery state. A discard module is used to retain critical information and discard non-critical information when the first cluster recovery event is received; wherein the critical information includes configuration information; and The first update module is used to query and update the latest node status from its target business module control layer when it receives the second cluster recovery event.

20. A computer device comprising a memory and one or more processors, the memory storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to perform the steps of any one of claims 1 to 18.

21. One or more non-volatile computer-readable storage media storing computer-readable instructions, characterized in that, When executed by one or more processors, the computer-readable instructions cause the one or more processors to perform the steps of the method of any one of claims 1 to 18.

Images