Method for arbitrating nodes of a storage system

Abstract

The application discloses a node arbitration method of a storage system, and relates to the technical field of electric digital data processing.The method comprises the following steps: determining the partition state of a plurality of node clusters based on the connection relationship between the plurality of node clusters under the condition that a preset network partition condition is met, initiating a serialized reading of the node clusters in each partition, determining the arbitration node address and version identifier corresponding to each node cluster, obtaining the version identifier from the partition serialized reading, promoting a new main arbitration node from the version dimension, and carrying out cluster arbitration in combination with the new main arbitration node and the plurality of node clusters.The method solves the technical problem that, in the related art, it is difficult to deal with the brain split problem of the storage system in the case of an even number of nodes, which leads to the paralysis of a business link, and enables the storage system to still realize effective arbitration and maintain the stable operation of the cluster when network partition occurs.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, and in particular to a node arbitration method for storage systems. Background Technology

[0002] Based on the majority vote election mechanism, storage systems need to avoid ties. However, for storage systems configured with an even number of nodes, it is difficult to completely avoid ties. Once a tie occurs, the system will be permanently paralyzed.

[0003] In related technologies, the tie problem can be solved by introducing arbitration. However, in single-arbitration application scenarios, it is difficult to avoid the blockage caused by single point of failure. In the case of network partitioning, multi-arbitration will result in the split-brain problem, that is, multiple master arbitrators are elected at the same time, which will lead to system data disorder, uncontrollable logic, and even paralysis of the entire business link. It is urgent to improve. Summary of the Invention

[0004] This application provides a node arbitration method for storage systems to at least address the technical problem in related technologies where, in the presence of an even number of nodes, it is difficult to cope with the split-brain problem in storage systems, leading to the paralysis of business links.

[0005] This application provides a node arbitration method for a storage system. The storage system is configured with multiple node clusters and multiple arbitration nodes. A master arbitration node is elected from among the multiple arbitration nodes. Based on the voting results of the master arbitration node and the multiple node clusters, the currently running business is executed. The method includes the following steps: under the condition that the storage system meets preset network partitioning conditions, the partitioning status of the multiple node clusters is determined based on the connection relationship between the multiple node clusters; based on the partitioning status, a serialization read is initiated on the node clusters in each partition to determine the arbitration node address and version identifier corresponding to each node cluster; based on the version identifier, a target version identifier that meets a preset maximum value condition is obtained, and a new master arbitration node is elected from among the multiple arbitration nodes according to the target version identifier and the arbitration node address; the new master arbitration node and the multiple node clusters are used to conduct cluster arbitration voting to execute the currently running business according to the voting results.

[0006] This application also provides a node arbitration device for a storage system. The storage system is configured with multiple node clusters and multiple arbitration nodes. A master arbitration node is elected from among the multiple arbitration nodes to execute the currently running business based on the voting results of the master arbitration node and the multiple node clusters. The device includes: a determining module, used to determine the partition status of multiple node clusters based on the connection relationship between the multiple node clusters when the storage system meets preset network partition conditions; an initiating module, used to initiate a serialization read for the node clusters in each partition based on the partition status to determine the arbitration node address and version identifier corresponding to each node cluster; and an electing module, used to obtain a target version identifier that meets a preset maximum value condition based on the version identifier, and elect a new master arbitration node from among the multiple arbitration nodes according to the target version identifier and the arbitration node address. The new master arbitration node and the multiple node clusters then conduct cluster arbitration voting to execute the currently running business according to the voting results.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the node arbitration method of any of the above-described storage systems when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the node arbitration method of any of the above-described storage systems.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the node arbitration method of any of the above-described storage systems.

[0010] This application enables the determination of partition status of multiple node clusters based on the connection relationships between them, under the condition of satisfying preset network partitioning. It then initiates serialization reads for each node cluster within a partition to determine the address and version identifier of the corresponding arbitrator node. Using the version identifier obtained from the partition serialization reads, a new master arbitrator node is elected based on the version. Cluster arbitration is then conducted using the new master arbitrator node and multiple node clusters, and the currently running business is executed based on the voting results of the cluster arbitration. This solves the technical problem in related technologies where the presence of an even number of nodes makes it difficult to handle the split-brain problem in the storage system, leading to the paralysis of business links. It allows the storage system to still achieve effective arbitration and maintain the robust operation of the cluster even when network partitions occur. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating the principle of the election mechanism in related technologies; Figure 2 This is a schematic diagram illustrating the network partitioning principle in related technologies; Figure 3 This is a flowchart of a node arbitration method for a storage system according to an embodiment of this application; Figure 4 A flowchart of a node arbitration method for a storage system according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the network partition arbitration principle of a node arbitration method for a storage system according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a node arbitration device for a storage system according to an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] Understandably, storage systems are designed for distributed environments. In a distributed architecture, the storage system primarily serves as a configuration center and coordination hub, undertaking functions such as global configuration management, service registration and discovery, node state maintenance, distributed locks, and leader election coordination, providing stable metadata support for upper-layer systems. With its simple architecture, ease of use, and stable performance, storage systems have been widely adopted in cloud-native and container orchestration fields, becoming a core foundational component of large-scale distributed systems.

[0017] Storage systems often require an odd number of nodes in many applications, such as... Figure 1 As shown, because the core of its election mechanism is majority vote, odd-numbered nodes will always have a majority of votes, while even-numbered nodes have a probability of a tie. A tie will cause the system to permanently crash. Therefore, a necessary choice in even-numbered node storage systems is to introduce arbitration technology. When a tie occurs, the arbitrator casts an additional vote to help the system continue operating. Of course, introducing arbitration also introduces more instability.

[0018] Even-numbered node clusters require arbitration; however, the introduction of arbitration also introduces uncertainty into the process. The probability of a single arbiter becoming disconnected or crashing becomes a single point of failure for the entire system, which is unacceptable in a production environment. To mitigate this risk, the number of arbitration nodes is usually increased in actual deployments, and primary and backup arbiters are set up. The storage system nodes dynamically select the primary arbiter by monitoring the working status of each arbiter in real time. This reduces the probability of arbiter failure to an extremely low level (the probability of multiple arbiters failing to work simultaneously is extremely low).

[0019] However, multiple arbitrations introduce a thorny and crucial issue, such as Figure 2 As shown, if a network partition occurs (i.e., the node network is split into several unconnected subsets), the node clusters within different partitions may independently elect different master arbiters based on local awareness, leading to catastrophic consequences, also known as split-brain. Split-brain refers to the emergence of two or more independent master arbiters in a system that requires all nodes to maintain complete consistency of stored data. Figure 2 As shown, partition A and partition B each elect a master arbitrator node. This split-brain state can lead to system data disorder, uncontrollable logic, and even paralysis of the entire business chain.

[0020] To address the aforementioned technical issues, the node arbitration method of the storage system in this application embodiment can, in the event of a split-brain scenario, combine the version identifier of the node cluster to elect the master arbitrator node, thereby locking in a unique decision source through algorithms or protocols in a multi-arbitration environment and ensuring the robust operation of the cluster.

[0021] In the node arbitration method of the storage system in this application embodiment, the storage system is equipped with multiple node clusters and multiple arbitration nodes to elect a master arbitration node from multiple arbitration nodes. The elected master arbitration node and the node clusters participate in the voting together to avoid a tie in the voting of an even number of node clusters.

[0022] In actual implementation, each node cluster of the storage system in this application embodiment can establish TCP (Transmission Control Protocol) connections with multiple arbitration nodes and confirm whether the connection is valid. For example, if the timeout threshold is set to 300ms, and a connection with the arbitration node is successfully established twice within 300ms, it is determined to be a valid connection.

[0023] This application embodiment can create an arbitration connection state cache locally after establishing a valid connection (where the key is the host address of the arbitration node, and the value contains information such as connection time and heartbeat cycle); this application embodiment can also submit a KV (Key-Value) write request to the node cluster of the storage system. The key format of KV is fixed as " / quorum / conns / add / etcd_member_id / quorum_host", where quorum is the arbitration node, conns is the connection, add is the addition, etcd_member_id is the unique identifier of the current node cluster, quorum_host is the IP (Internet Protocol) + port of the corresponding arbitration node, and value is the value (current system timestamp, accurate to milliseconds). The write request uses the storage system's native Put (write operation) interface to ensure the atomicity of the write.

[0024] After a valid connection is established, this embodiment of the application can also elect an arbitration node to determine a primary arbitration node as the business response.

[0025] like Figure 3 As shown, the node arbitration method of the storage system includes the following steps: In step S301, if the storage system meets the preset network partitioning conditions, the partitioning status of multiple node clusters is determined based on the connection relationship between multiple node clusters.

[0026] For example, when the key-value data under the " / quorum / conns" path changes (including connection establishment and connection termination), the current primary arbitrator node of the cluster will detect it in real time through the Watch interface of the storage system (the Watch listening path is set to " / quorum / conns / ", and the listening frequency is synchronized with the key-value change frequency). At this time, it can be confirmed that the internal environment of the storage system under test has changed, and a new primary arbitrator node needs to be elected.

[0027] In some embodiments, the primary arbitrator node can be determined by the number of effective connections between each arbitrator node and multiple node clusters, the score of each arbitrator node determined based on the number of effective connections, etc. However, there may still be cases where two primary arbitrator nodes are determined. For example, if there are a total of 4 node clusters, and these 4 node clusters elect a first primary arbitrator node and a second primary arbitrator node (the number of effective connections of the first primary arbitrator node and the second primary arbitrator node are the same, or the scores are the same), that is, the storage system meets the network partitioning conditions. In this case, the original primary arbitrator node cannot maintain communication with more than half of the nodes, triggering the Raft (consensus algorithm) protocol's step-off mechanism. At this time, the cluster has no unified primary arbitrator node, and regular serial reads (which rely on the primary arbitrator node's Range interface) cannot obtain the KV data under the " / quorum / active" path, resulting in each partition being unable to obtain a unified primary arbitrator identifier.

[0028] To avoid logical confusion, this application embodiment can determine the partition status of multiple node clusters based on the connection relationship between multiple node clusters. Node clusters that can establish effective connections with each other are divided into the same partition, and node clusters that cannot establish effective connections are divided into different partitions, resulting in multiple independent partitions corresponding to the clusters. This facilitates subsequent serialization reading on a partition-by-partition basis, collecting the arbitration node address and version identifier corresponding to each node cluster, and electing a new master arbitration node.

[0029] In step S302, based on the partition status, a serialization read is initiated for the node cluster in each partition to determine the arbitration node address and version identifier corresponding to each node cluster.

[0030] For cases where network partitioning conditions are met, this application embodiment can adopt a method of serialized read + revision (version identifier) ​​version number comparison. Serialized reads are initiated within the node cluster within each partition, without relying on the cluster master arbitrator node. Reads are initiated directly between nodes within the partition, reading the KV metadata cached locally on each node to determine the arbitrator node address and version identifier corresponding to each node cluster, which facilitates the subsequent election of a new master arbitrator node.

[0031] Optionally, in one embodiment of this application, initiating a serialized read for a node cluster in each partition includes: using any node cluster as the initiating node, sending a serialized read request carrying the node identifier of the initiating node and the request timestamp to other node clusters in the partition corresponding to any node cluster that meet the preset connection conditions; and receiving the local cached arbitration node address and the corresponding version identifier returned by other node clusters in response to the serialized read request.

[0032] For example, in this embodiment of the application, any node cluster can be used as the initiating node to send a serialized read request to all other connectable node clusters within the partition. The request format adopts the storage system's native Range interface, and the request parameter is " / quorum / active (currently effective)", while also carrying the initiating node identifier and request timestamp. Other node clusters that receive the request immediately read the locally cached " / quorum / active" KV value and the corresponding revision number, and return it to the initiating node through the response interface.

[0033] By combining the revision numbers of the initiating node (any node cluster) and the responding node (other node clusters) of each partition, the embodiments of this application can determine a new master arbitrator node.

[0034] This application embodiment can achieve unified selection of the main arbitrator through purely local serialization reading and version number sorting when each partition is physically isolated and unable to communicate. This avoids the risk of split-brain caused by network partitions and ensures that the cluster still has logically unified arbitration decision-making capabilities under extreme abnormal scenarios.

[0035] Optionally, in one embodiment of this application, after initiating a serialization read on the node cluster in each partition, the method further includes: counting the response time of multiple node clusters; identifying node clusters with response times greater than a first preset time threshold as invalid nodes, and prohibiting the initiation of serialization reads on invalid nodes.

[0036] In this embodiment, a threshold for response time, such as 300ms, can be set. If the response time of a node cluster exceeds 300ms, the node cluster is determined to have timed out and is considered an invalid node. This helps to eliminate node clusters within the partition that are slow to respond due to excessive disk load, sudden increase in network latency, or process lag, thereby improving the efficiency of leader selection decision-making in network partition scenarios.

[0037] In step S303, based on the version identifier, the target version identifier that meets the preset maximum value condition is obtained, and a new master arbitration node is elected from multiple arbitration nodes according to the target version identifier and the arbitration node address. The new master arbitration node and multiple node clusters are used to conduct cluster arbitration voting to execute the currently running business according to the voting results.

[0038] As one possible implementation method, in this embodiment of the application, the initiating node collects the " / quorum / active" KV value and the corresponding revision number from all valid responses (the revision number is a KV version identifier automatically generated by the node cluster of the storage system, which automatically increments as the KV value changes, without manual intervention); the initiating node starts the local comparison module, sorts all collected revision numbers in descending order, selects the KV value corresponding to the largest revision number (i.e., the latest master arbitrator node), and then determines the address of the arbitrator node corresponding to the largest revision number. Based on the arbitrator node address, a new master arbitrator node is determined, so as to use the new master arbitrator node and multiple node clusters to conduct cluster arbitration voting, avoiding the situation of a tie when an even number of node clusters vote.

[0039] This application embodiment can utilize the characteristic that the storage system generates a globally unique incrementing version number with each write operation, and use the version identifier as the nomination condition for the master arbitrator node. This ensures that no matter which nodes are contained in each partition, since the latest data has been synchronized to more than half of the nodes, the arbitrator node address obtained after each partition independently performs the maximum value operation is absolutely consistent, thus guaranteeing the uniqueness of the new master arbitrator node in the network partition state.

[0040] Optionally, in one embodiment of this application, the method further includes: obtaining the connection status between each node cluster and each arbitration node, so as to count the number of valid connections corresponding to each arbitration node based on the connection status; and determining whether the storage system meets the preset network partitioning conditions based on the number of valid connections.

[0041] In actual execution, after the primary arbitrator node detects a key-value (KV) change, it can immediately read all KV data under the " / quorum / conns / " path through the Range interface. It then analyzes the availability of each arbitrator node using a preset algorithm. For example, it counts the connection status between each arbitrator node and all node clusters, specifically the connection status between arbitrator node q1 and node clusters 1, 2, 3, and 4, and obtains the number of valid connections with a valid connection status. If arbitrator node q1 has valid connections with node clusters 2 and 3, but invalid connections with node clusters 1 and 4, then the number of valid connections is 2. The valid and invalid connection statuses will be described below.

[0042] In this embodiment, the number of node clusters that each arbitration node can effectively connect to can be summarized, which is the number of effective connections for the corresponding arbitration node. When the number of effective connections for each arbitration node differs significantly, and no arbitration node can maintain an effective connection with more than half of the node clusters, or multiple arbitration nodes correspond to different local connected subsets and form multiple sets of mutually isolated connected links, it is determined that the connectivity between cluster nodes is broken, data interaction cannot be globally synchronized, and the storage system meets the network partitioning conditions. Conversely, if there are arbitration nodes that can maintain an effective connection with the vast majority of node clusters, the overall connectivity of the cluster is complete, and there are no isolated subsets, it is determined that no network partitioning has occurred, and there is no need to enter the partition arbitration logic.

[0043] The embodiments of this application can count the effective connection scale of each arbitration node, and then accurately identify the cluster connectivity split based on the number of effective connections, avoiding the problems of arbitration logic disorder and multiple master nodes working in parallel due to misjudgment of connectivity status.

[0044] Optionally, in one embodiment of this application, obtaining the connection status between each node cluster and each arbitration node, and calculating the number of valid connections corresponding to each arbitration node based on the connection status, includes: obtaining multiple connection results between any node cluster and any arbitration node within a second preset time threshold; if all multiple connection results are successful, determining the connection status between any node cluster and any arbitration node as a preset valid connection status, so as to accumulate the number of valid connections corresponding to any arbitration node.

[0045] In some implementations, conditions can be set to determine whether the node cluster and the arbitration node are in a valid connection state or an invalid connection state.

[0046] When any node cluster (such as etcd-1, where etcd is the storage system) initiates a connection to any arbitration node (such as q1), this embodiment of the application can determine whether the connection is successful within a certain time threshold (such as 300ms). If the connection is successful multiple times (such as twice) within the time threshold, it is considered a valid connection, and the connection between etcd-1 and q1 is counted and added to the number of valid connections.

[0047] This application embodiment can avoid the problem of discrepancies between the count of effective connections and the actual situation caused by short-term network problems by combining multiple consecutive judgments and timeliness judgments. This allows the cumulative number of effective connections to characterize the network connectivity quality between each arbitration node and the node cluster, which can then serve as the basis for subsequent network partitioning conditions or the recommendation of the main arbitration node.

[0048] Optionally, in one embodiment of this application, after counting the number of valid connections corresponding to each arbitration node based on the connection status, the method further includes: obtaining the historical connection time of any node cluster stored in the cluster key-value storage space of the storage system; comparing the first connection initiation time and the historical connection time of any node cluster; and, if the first connection initiation time meets the preset timing conditions, writing the first connection initiation time, the address of any arbitration node, and the added identifier into the cluster key-value storage space.

[0049] It is understandable that there may be situations where out-of-order network connections cause old disconnection records to arrive later and incorrectly overwrite new connection records. To avoid this situation, the embodiments of this application can perform a time-series comparison, comparing the stored historical connection times with the first connection initiation times. After determining that the first connection initiation time is later than the historical connection times, the relevant content of the valid connections is written to the cluster key-value storage space to avoid data anomalies caused by out-of-order packets. This allows the number of valid connections to be used as a basis for subsequent network partitioning conditions or the recommendation of the master arbitrator node.

[0050] Optionally, in one embodiment of this application, obtaining the connection status between each node cluster and each arbitration node includes: obtaining multiple heartbeat interaction results between another node cluster and another arbitration node within a third preset time threshold; if the multiple heartbeat interaction results are all connection failures, determining that the other node cluster and the other arbitration node are disconnected, and determining that the connection status between the other node cluster and the other arbitration node is a preset invalid connection status.

[0051] Within a time threshold (e.g., 500ms), obtain the results of multiple (e.g., three) heartbeat interactions between another node cluster (e.g., etcd-3) and another arbitration node (e.g., q3). If no heartbeat response is received from q3 for three consecutive times, it is determined that q3 is offline and etcd-3 disconnects from q3.

[0052] The embodiments of this application can avoid misjudgment due to a single packet loss, and can serve as the basis for subsequent network partitioning conditions or the recommendation of the main arbitration node.

[0053] Optionally, in one embodiment of this application, after determining that another node cluster has disconnected from another arbitration node, the method further includes: marking a disconnection flag at the second connection initiation time of the other node cluster to obtain the flag initiation time; and writing the flag initiation time, the address of the other arbitration node, and the removal flag into the cluster key-value storage space of the storage system.

[0054] After the connection is determined to be broken, the KV data of another node cluster can be updated in this application embodiment. The disconnection identifier is marked on the time when the second connection was initiated, and the marked initiation time is obtained (such as KV value: 1743060060456#disconnect). The marked initiation time, the address of the other arbitration node and the removal identifier are written to facilitate subsequent traceability.

[0055] Optionally, in one embodiment of this application, the method further includes: when the storage system does not meet the preset network partitioning conditions, counting the number of valid connections corresponding to each arbitration node; sorting multiple arbitration nodes using the number of valid connections, so as to elect the arbitration node with the largest number of valid connections as the new master arbitration node.

[0056] If no partitioning scenario is determined, the embodiments of this application can elect a new master arbitrator node based on the number of valid connections between each arbitrator node and the node cluster.

[0057] For example, an embodiment of this application may include four node clusters (etcd-1, etcd-2, etcd-3, etcd-4) and three arbitration nodes (q1, q2, q3), wherein q1 establishes a valid connection with the three node clusters, q2 establishes a valid connection with one node cluster, and q3 establishes a valid connection with two node clusters.

[0058] In this embodiment of the application, q1 can be nominated as the new master arbitration node.

[0059] In this embodiment, a new master arbitrator node is nominated based on the number of effective connections to ensure that the master arbitrator node has the widest network coverage among all arbitrator nodes, so that the broadcast commands or data synchronization requests issued by the master arbitrator node can directly reach the largest node cluster with the highest probability.

[0060] Optionally, in one embodiment of this application, after counting the number of valid connections corresponding to each arbitration node, the method further includes: assigning a corresponding availability weight to each arbitration node based on the number of valid connections; calculating the average connection duration of each arbitration node based on the historical connection duration of each arbitration node, and assigning a corresponding stability weight to each arbitration node based on the average connection duration; combining the availability weight and the stability weight to determine the weight score of each arbitration node, and recommending a new master arbitration node based on the weight score.

[0061] Furthermore, embodiments of this application can also elect a new primary arbitration node through comprehensive calculation.

[0062] This application embodiment can count the number of valid connections in the node cluster for each arbitration node. The more valid connections, the higher the availability weight of the matching. This application embodiment can calculate the average connection duration of each arbitration node. The longer the connection duration, the higher the stability weight of the matching. This application embodiment can combine the two weights to select the arbitration node with the best availability (the one with the highest weight score).

[0063] By combining availability weights and stability weights for a comprehensive score, the newly elected master arbitrator node can possess both availability and stability, thereby increasing the cluster's stability and anti-interference capabilities in network fluctuation environments.

[0064] Optionally, in one embodiment of this application, after electing a new master arbitrator node from multiple arbitrator nodes, the method further includes: obtaining the current version identifier of multiple node clusters within the partition, and calculating the difference between the current version identifier and the target version identifier; marking node clusters with a difference greater than a preset difference as lagging node clusters; initiating an incremental data synchronization instruction to the lagging node clusters using the new master arbitrator node; and responding to the incremental data synchronization instruction by extracting target incremental data from the cluster key-value storage space of the storage system using the current version identifier and the difference as an index, so as to update the lagging node clusters using the target incremental data.

[0065] In actual implementation, the embodiments of this application can reuse the target version number obtained during the master selection process as the basis for data repair, automatically identify and locate the cluster of lagging nodes in the partition, i.e. the cluster of lagging nodes, and perform targeted incremental data backfilling, so that the lagging nodes can be quickly updated to the latest version without full data copying, shortening the node recovery time and improving the overall data consistency level in the partition.

[0066] The preset difference can be set by the expert in this field according to the actual situation.

[0067] Optionally, in one embodiment of this application, the method further includes: writing the address and election time of the new master arbitrator node to the cluster key-value storage space; obtaining write feedback data from multiple node clusters and accumulating the number of node clusters that have successfully submitted feedback; after the number of node clusters meets a preset quantity condition, updating the version identifiers of multiple node clusters, and broadcasting the address and election time of the new master arbitrator node to multiple node clusters using the new master arbitrator node.

[0068] The new primary arbitrator node can write its address (quorum_host) to the " / quorum / active" path via the Put interface, and set the response time of this key-value pair (consistent with the arbitrator node's heartbeat period, which is 500ms) to ensure that the primary arbitrator identifier is updated in real time. After the write is completed, it will notify all nodes in the cluster through a broadcast mechanism to synchronously update the primary arbitrator cache on each node.

[0069] Before writing, this embodiment of the application can first solicit feedback from all node clusters via broadcast. Only when more than half of the node clusters successfully respond will the final write be performed. This ensures that before the new master arbitrator node's data is written, more than half of the node clusters have confirmed the identity of the new master arbitrator node, avoiding the possibility of unauthorized master arbitrator node tampering.

[0070] Optionally, in one embodiment of this application, after broadcasting the address and election time of the new master arbitrator node to multiple node clusters using the new master arbitrator node, the method further includes: after the node clusters receive the broadcast, parsing the new version identifier corresponding to the address and election time of the new master arbitrator node; comparing the new version identifier corresponding to multiple node clusters with the actual version identifier of multiple node clusters; if the new version identifier is greater than the actual version identifier, responding to the broadcast to update multiple node clusters based on the address and new version identifier of the new master arbitrator node; if the new version identifier is less than or equal to the actual version identifier, rejecting the broadcast response and generating a version conflict reminder.

[0071] In actual implementation, this application embodiment can also set a version number verification mechanism to detect network anomalies or erroneous data conflicts in a timely manner.

[0072] In this embodiment of the application, the node cluster can compare the version identifier carried in the broadcast with the local actual version identifier when responding to a broadcast update from the master arbitrator node. If the new version identifier is indeed greater than the old version identifier, it indicates that the update is a normal update. This prevents data erroneous overwriting caused by network out-of-order issues and facilitates timely detection of anomalies through version conflict alerts, thereby enhancing the maintainability and fault diagnosability of the system.

[0073] Optionally, in one embodiment of this application, it further includes: accumulating the write duration of the address of the new master arbitrator node in the cluster key-value storage space; and generating a nomination instruction for the next master arbitrator node in response to the write duration exceeding a third preset duration threshold.

[0074] This application embodiment can avoid situations where there is no master arbitrator node for a long time by setting a write duration threshold (such as 500ms) for the address of the new master arbitrator node. When the address of the master arbitrator node is stored in the cluster key-value storage space for more than the duration threshold, this application embodiment can automatically trigger the next round of master election process, avoid system lag, enable the system to automatically correct errors, and ensure system stability.

[0075] Combination Figure 4 and Figure 5 As shown, the working principle of the node arbitration method of the storage system of this application is illustrated by an embodiment.

[0076] Taking a deployment of a 4-node cluster (etcd-1, etcd-2, etcd-3, etcd-4) and 3 independent arbitration nodes (arbitration node addresses: q1:192.168.1.10:8080, q2:192.168.1.11:8080, q3:192.168.1.12:8080) as an example, the environment configuration in this embodiment is as follows: Node cluster: 4-node cluster, Raft protocol default configuration, nodes can communicate with each other on the network, and the cluster identifier is unique; Arbitration nodes: 3 independent servers, deploying arbitration services, supporting TCP connections, 500ms heartbeat response, and offline judgment; Fixed parameters: TCP connection timeout 300ms, valid connection determination (2 consecutive successful connections), heartbeat cycle 500ms, offline determination (3 consecutive heartbeats without response), KV synchronization with more than half of the nodes committing, serialization read timeout 300ms; Data paths: Connection establishment KV: / quorum / conns / add / {etcd node ID} / {arbitration node address}; Connection termination KV: / quorum / conns / remove / {etcd node ID} / {arbitration node address}; Primary arbitration storage (primary arbitration node storage) KV: / quorum / active; Listening path: / quorum / conns / .

[0077] like Figure 4 As shown, embodiments of this application may include the following steps: Step S1: Determine the connection relationship between the node cluster and the arbitration node.

[0078] Taking the establishment of a valid connection between the etcd-1 node and the q1 arbitrator node as an example: etcd-1 initiates a TCP connection to q1. The connection succeeds within 300ms, and two consecutive connections succeed, indicating a valid connection. etcd-1's key-value write: Key-value key: / quorum / conns / add / etcd-1 / 192.168.1.10:8080; Key-value value: 1743060000123 (millisecond-level timestamp); Write method: native Put interface of the storage system, atomic write.

[0079] Taking the disconnection between the etcd-3 node and the q3 arbitrator node as an example: etcd-3 and q3 maintain a heartbeat interaction every 500ms. If no heartbeat response is received from q3 for 3 consecutive times, it is determined that q3 is offline. Etcd cluster KV write: KVkey: / quorum / conns / remove / etcd-3 / q3:192.168.1.12:8080; KVvalue: 1743060060456#disconnect (timestamp + disconnection flag).

[0080] Step S2: In response to a change in the KV path of any node cluster, an update instruction for the master arbitrator node is generated.

[0081] Step S3: Obtain the number of valid connections between the node cluster and the arbitration node.

[0082] Step S4: Determine if a network partition scenario has occurred.

[0083] In step S5, if no network partitioning occurs, a new primary arbitrator node is elected based on the number of valid connections and the average connection duration.

[0084] The cluster network is currently functioning normally, and the automatic selection process for the primary arbitrator is being executed.

[0085] Triggering condition: A change occurs in the key-value data under the / quorum / conns / path (e.g., etcd-2 establishes a q2 connection). The primary arbiter node detects the change in real time through the Watch interface. It then retrieves all data under the / quorum / conns / path, determines which arbiters have the most node connections based on the timestamp, sorts them according to byte order, selects the arbiter at the top of the sort (e.g., q2), and writes q2's address to / quorum / active.

[0086] Subsequently, in this embodiment of the application, the main arbitration storage and synchronization can be performed. The main arbitration node writes q2:192.168.1.11:8080 to the storage system cluster, with KV key= / quorum / active and value=q2:192.168.1.11:8080, and sets a response time of 500ms. This KV data is synchronized to more than half of the nodes, and the revision version number is automatically generated (e.g., revision=10001). The main arbitration node broadcasts a notification to all Etcd nodes, and each node updates its local main arbitration cache to q2. Under normal network conditions, the cluster uniformly uses q2 as the main arbitration node.

[0087] In step S6, if a network partition occurs, a new primary arbitrator node is elected based on the version identifier.

[0088] Assuming the cluster is divided into partition A (etcd-1, etcd-2) and partition B (etcd-3, etcd-4), the original master arbitrator node is unable to communicate with more than half of the nodes, triggering the master arbitrator node to abdicate. The cluster lacks unified leadership, regular serial reads fail, and each partition automatically executes a process that combines serialized reads with revision version identifier comparison.

[0089] Partition A: Serialization read process execution.

[0090] Triggering serialization read: Both etcd-1 and etcd-2 in partition A detect the absence of a master arbitrator node in the cluster and automatically initiate a serialization read; with etcd-1 as the initiating node, a Range request is sent to the connectable node (etcd-2) in partition A, with the parameter = / quorum / active, carrying the request identifier + timestamp, and a timeout of 300ms; etcd-2 reads the locally cached / quorum / active data and returns: {value:q2:192.168.1.11:8080,revision:10001}; valid response collection is complete.

[0091] Partition B: Serialization read process execution.

[0092] Triggering serialization read: When etcd-3 and etcd-4 in partition B detect that there is no Leader in the cluster, they automatically initiate a serialization read; with etcd-4 as the initiating node, a Range request is sent to the connectable node (etcd-3) in partition B, with the parameter = / quorum / active; etcd-3 reads the locally cached / quorum / active data and returns: {value:q2:192.168.1.11:8080,revision:10001}; valid response collection is complete.

[0093] like Figure 5As shown, whether it is partition A or partition B, the highest version identifier is 10001. The Candidate (candidate initiating node cluster) will obtain the local KV / quorum / active of all reachable nodes in the partition, so the cluster can guarantee a unified master arbitrator.

[0094] Comparison of Partition A: 1 valid response was collected, revision=10001 (maximum value), and the corresponding value=q2 was selected; Comparison of Partition B: 1 valid response was collected, revision=10001 (maximum value), and the corresponding value=q2 was selected.

[0095] A 4-node cluster requires more than half (≥3) of the nodes to synchronize the latest KV data. In partition A (2 nodes) and partition B (2 nodes), there must be a node storing revision=10001. Finally, the two partitions select the same master arbitrator node q2.

[0096] In a network partitioning scenario, the cluster does not have a unified master arbitrator node, but all nodes in partition A and partition B select q2 as the sole master arbitrator to ensure that only one node obtains the arbitration vote and is elected as the new master arbitrator node.

[0097] In summary, the embodiments of this application can ensure that each partition can obtain a unified and up-to-date master arbitrator identifier through the KV commit mechanism of the storage system cluster. Any successfully committed " / quorum / active" KV data in the storage system cluster will be synchronized to more than half of the nodes in the cluster. Combined with the partition characteristics of even-numbered node clusters (e.g., a 4-node cluster is divided into 2 partitions on average; if the partitions are not evenly divided, a leader can be elected normally), since KV commit requires synchronization of more than half of the nodes (a 4-node cluster requires at least 3 nodes to synchronize the latest data), regardless of how the network is partitioned, there will inevitably be nodes in both partitions that store the " / quorum / active" data corresponding to the latest revision number. This ensures that each partition ultimately selects a unified and usable master arbitrator node through the above comparison logic, which helps to prevent split-brain phenomena and ensures that all nodes in the cluster select only one master arbitrator node at the same time.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0099] like Figure 6As shown, embodiments of this application also provide a node arbitration device 10 for a storage system, wherein the storage system is configured with multiple node clusters and multiple arbitration nodes, and a master arbitration node is elected from the multiple arbitration nodes to execute the currently running business based on the voting results of the master arbitration node and the multiple node clusters.

[0100] The node arbitration device 10 of the storage system includes: Specifically, the determination module 100 is used to determine the partition status of multiple node clusters based on the connection relationship between multiple node clusters, provided that the storage system meets the preset network partition conditions.

[0101] The first initiation module 200 is used to initiate a serialization read on the node cluster in each partition based on the partition status, so as to determine the arbitration node address and version identifier corresponding to each node cluster.

[0102] The first recommendation module 300 is used to obtain the target version identifier that meets the preset maximum value condition based on the version identifier, and recommend a new master arbitration node from multiple arbitration nodes according to the target version identifier and the arbitration node address. Then, the new master arbitration node and multiple node clusters are used to conduct cluster arbitration voting to execute the currently running business according to the voting results.

[0103] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system includes: The acquisition module is used to obtain the connection status between each node cluster and each arbitration node, so as to count the number of valid connections corresponding to each arbitration node based on the connection status.

[0104] The second determining module is used to determine whether the storage system meets the preset network partitioning conditions based on the number of valid connections.

[0105] Optionally, in one embodiment of this application, the acquisition module includes: The acquisition unit is used to acquire multiple connection results between any node cluster and any arbitration node within a second preset time threshold. The accumulation unit is used to determine the connection status between any node cluster and any arbitration node as a preset valid connection status when multiple connection results are all successful, so as to accumulate the number of valid connections corresponding to any arbitration node.

[0106] Optionally, in one embodiment of this application, the acquisition module further includes: The first acquisition unit is used to acquire the historical connection time of any node cluster stored in the cluster key-value storage space of the storage system.

[0107] The comparison unit is used to compare the time of the first connection initiation with the historical connection times of any node cluster.

[0108] The first storage unit is used to write the first connection initiation time, the address of any arbitration node, and the added identifier into the cluster key-value storage space when the preset timing conditions are met at the time the first connection is initiated.

[0109] Optionally, in one embodiment of this application, the acquisition module includes: The second acquisition unit is used to acquire the results of multiple heartbeat interactions between another node cluster and another arbitration node within a third preset time threshold.

[0110] The determination unit is used to determine that another node cluster is disconnected from another arbitration node when multiple heartbeat interactions result in connection failure, and to determine that the connection status between the other node cluster and the other arbitration node is a preset invalid connection status.

[0111] Optionally, in one embodiment of this application, the acquisition module further includes: The third acquisition unit is used to mark the disconnection flag at the second connection initiation time of another node cluster, so as to obtain the marking initiation time.

[0112] The second storage unit is used to write the initiation time of the annotation, the address of the other arbitration node, and the removal flag to the cluster key-value storage space of the storage system.

[0113] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system further includes: The first statistics module is used to count the number of valid connections corresponding to each arbitration node when the storage system does not meet the preset network partition conditions.

[0114] The second recommendation module is used to sort multiple arbitration nodes based on the number of valid connections, and recommend the arbitration node with the largest number of valid connections to become the new master arbitration node.

[0115] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system further includes: The assignment module is used to assign a corresponding availability weight to each arbitration node based on the number of valid connections.

[0116] The first calculation module is used to calculate the average connection duration of each arbitration node based on the historical connection duration of each arbitration node, and to assign a corresponding stability weight to each arbitration node based on the average connection duration.

[0117] The third recommendation module is used to combine availability weight and stability weight to determine the weight score of each arbitration node, and recommend a new master arbitration node based on the weight score.

[0118] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system further includes: The second statistics module is used to count the response time of multiple node clusters.

[0119] The first control module is used to designate node clusters with response times exceeding a first preset time threshold as invalid nodes and prohibit serialization reads from invalid nodes.

[0120] Optionally, in one embodiment of this application, the first initiating module 200 includes: The initiating unit is used to send a serialized read request carrying the node identifier of the initiating node and the request timestamp to other node clusters within the partition corresponding to the node cluster that meet the preset connection conditions, with any node cluster as the initiating node.

[0121] The receiving unit is used to receive the arbitration node address and corresponding version identifier of the local cache returned by other node clusters in response to the serialization read request.

[0122] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system further includes: The second calculation module is used to obtain the current version identifier of multiple node clusters within the partition and calculate the difference between the current version identifier and the target version identifier.

[0123] The marking module is used to mark node clusters with a difference greater than a preset difference as lagging node clusters.

[0124] The second initiation module is used to initiate incremental data synchronization commands to the cluster of lagging nodes using the new master arbitrator node.

[0125] The first update module is used to respond to incremental data synchronization commands. It extracts the target incremental data from the cluster key-value storage space of the storage system using the current version identifier and the difference as an index, so as to update the lagging node cluster with the target incremental data.

[0126] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system further includes: The storage module is used to write the address and election time of the new master arbitrator node to the cluster key-value storage space.

[0127] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system further includes: The first accumulation module is used to obtain write feedback data from multiple node clusters and accumulate the number of node clusters that have successfully reported writes.

[0128] The second update module is used to update the version identifier of multiple node clusters after the number of node clusters meets the preset quantity condition, and to broadcast the address and election time of the new master arbitrator node to multiple node clusters using the new master arbitrator node.

[0129] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system further includes: The parsing module is used to parse the address of the new master arbitrator node and the new version identifier corresponding to the moment of election after the node cluster receives the broadcast.

[0130] The comparison module is used to compare the new version identifiers of multiple node clusters with the actual version identifiers of the multiple node clusters.

[0131] The second control module is used to respond to the broadcast when the new version identifier is greater than the actual version identifier, so as to update the multiple node cluster based on the address of the new master arbitrator node and the new version identifier.

[0132] The third control module is used to refuse to respond to the broadcast and generate a version conflict alert if the new version identifier is less than or equal to the actual version identifier.

[0133] Optionally, in one embodiment of this application, the node arbitration device 10 of the storage system further includes: The second accumulation module is used to accumulate the write duration of the new master arbitration node's address in the cluster key-value storage space.

[0134] The generation module is used to generate a recommendation instruction for the next primary arbitrator node in response to a write duration exceeding a third preset duration threshold.

[0135] For a description of the features of the node arbitration device in the embodiment of the storage system, please refer to the relevant description of the node arbitration method in the embodiment of the storage system, which will not be repeated here.

[0136] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the node arbitration method for the storage system.

[0137] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the node arbitration method embodiments of any of the above storage systems when it is run.

[0138] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0139] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the node arbitration method for the storage system.

[0140] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described storage system node arbitration method embodiments.

[0141] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] The node arbitration method for a storage system provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method of arbitrating nodes of a storage system, the method comprising: The storage system is configured with multiple node clusters and multiple arbitration nodes. A master arbitration node is elected from among the multiple arbitration nodes. Based on the voting results of the master arbitration node and the multiple node clusters, the currently running business is executed. The method includes the following steps: When the storage system meets the preset network partitioning conditions, the partitioning status of the multiple node clusters is determined based on the connection relationship between the multiple node clusters. Based on the partition status, a serialization read is initiated for the node clusters in each partition to determine the arbitration node address and version identifier corresponding to each node cluster. Based on the version identifier, a target version identifier that meets the preset maximum value condition is obtained. A new master arbitrator node is elected from multiple arbitrator nodes according to the target version identifier and the arbitrator node address. The new master arbitrator node and the multiple node clusters are used to conduct cluster arbitration voting to execute the currently running service according to the voting results.

2. The method of claim 1, wherein, Also includes: Obtain the connection status between each node cluster and each arbitration node, and count the number of valid connections corresponding to each arbitration node based on the connection status; Based on the number of valid connections, determine whether the storage system meets the preset network partitioning conditions.

3. The method of claim 2, wherein, The step of obtaining the connection status between each node cluster and each arbitration node, and then calculating the number of valid connections corresponding to each arbitration node based on the connection status, includes: Obtain the results of multiple connections between any node cluster and any arbitration node within a second preset time threshold. If multiple connection attempts all result in a successful connection, the connection status between any of the node clusters and any of the arbitration nodes is determined as a preset valid connection status, so as to accumulate the number of valid connections corresponding to any of the arbitration nodes.

4. The method of claim 3, wherein, After counting the number of valid connections corresponding to each arbitration node based on the connection status, the method further includes: Obtain the historical connection time of any node cluster stored in the cluster key-value storage space of the storage system; Compare the first connection initiation time and the historical connection time of any of the node clusters described above; If the first connection initiation time meets the preset timing conditions, the first connection initiation time, the address of any of the arbitration nodes, and the added identifier are written to the cluster key-value storage space.

5. The method according to claim 2, characterized in that, The step of obtaining the connection status between each node cluster and each arbitration node includes: Obtain the results of multiple heartbeat interactions between another node cluster and another arbitration node within a third preset time threshold. If multiple heartbeat interactions result in connection failure, it is determined that the other node cluster is disconnected from the other arbitration node, and the connection status between the other node cluster and the other arbitration node is determined to be a preset invalid connection status.

6. The method according to claim 5, characterized in that, After determining that another node cluster has disconnected from the other arbitration node, the process further includes: Mark the disconnection flag at the second connection initiation time of the other node cluster to obtain the marked initiation time; The time of initiation of the annotation, the address of the other arbitration node, and the removal identifier are written to the cluster key-value storage space of the storage system.

7. The method according to claim 2, characterized in that, Also includes: If the storage system does not meet the preset network partitioning conditions, count the number of valid connections corresponding to each arbitration node; The arbitration nodes are sorted according to the number of valid connections, and the arbitration node with the largest number of valid connections is elected as the new master arbitration node.

8. The method according to claim 7, characterized in that, After counting the number of valid connections corresponding to each arbitration node, the process further includes: Based on the number of valid connections, each arbitration node is assigned a corresponding availability weight; Based on the historical connection duration of each arbitration node, the average connection duration of each arbitration node is calculated, and a corresponding stability weight is assigned to each arbitration node based on the average connection duration. By combining the availability weight and the stability weight, a weight score is determined for each arbitration node, and a new master arbitration node is recommended based on the weight score.

9. The method according to claim 1, characterized in that, After initiating a serialized read on the node cluster in each of the partitions, the process also includes: Statistically analyze the response time of multiple node clusters. Clusters of nodes whose response time exceeds a first preset time threshold are designated as invalid nodes, and serialization reads are prohibited from being initiated on the invalid nodes.

10. The method according to claim 1, characterized in that, The step of initiating a serialized read on the node cluster in each partition includes: Take any node cluster as the initiating node and send a serialized read request carrying the node identifier and request timestamp of the initiating node to other node clusters in the partition corresponding to the any node cluster that meet the preset connection conditions. Receive the local cached arbitration node address and corresponding version identifier returned by the other node cluster in response to the serialized read request.

11. The method according to claim 1, characterized in that, After electing a new master arbitrator from among the multiple arbitrator nodes, the process also includes: Obtain the current version identifier of multiple node clusters within the partition, and calculate the difference between the current version identifier and the target version identifier; The node clusters whose difference is greater than a preset difference are marked as lagging node clusters; The new master arbitrator node is used to initiate an incremental data synchronization command to the cluster of lagging nodes. In response to the incremental data synchronization command, the target incremental data is extracted from the cluster key-value storage space of the storage system using the current version identifier and the difference as an index, so as to update the lagging node cluster using the target incremental data.

12. The method according to claim 1, characterized in that, Also includes: The address and election time of the new master arbitrator node are written to the cluster key-value storage space of the storage system.

13. The method according to claim 12, characterized in that, After writing the address and election time of the new master arbitrator node to the cluster key-value storage space of the storage system, the process further includes: Obtain write feedback data from multiple node clusters and accumulate the number of node clusters that successfully responded; After the number of node clusters meets the preset quantity condition, the version identifiers of multiple node clusters are updated, and the address and election time of the new master arbitrator node are broadcast to multiple node clusters using the new master arbitrator node.

14. The method according to claim 13, characterized in that, After broadcasting the address and election time of the new master arbitrator node to multiple node clusters using the new master arbitrator node, the process further includes: After the node cluster receives the broadcast, it parses the address of the new master arbitrator node and the new version identifier corresponding to the election time; Compare the new version identifiers corresponding to the multiple node clusters with the actual version identifiers of the multiple node clusters; If the new version identifier is greater than the actual version identifier, then respond to the broadcast to update multiple node clusters based on the address of the new primary arbitrator node and the new version identifier; If the new version identifier is less than or equal to the actual version identifier, the broadcast is rejected and a version conflict warning is generated.

15. The method according to claim 12, characterized in that, Also includes: The cumulative duration of the write operation of the address of the new master arbitrator node in the cluster key-value storage space; In response to the write duration exceeding a third preset duration threshold, a recommendation instruction for the next primary arbitration node is generated.

Images