Storage cluster arbitration state monitoring method and system, terminal and storage medium

Abstract

This invention relates to the field of storage cluster technology, specifically providing a method, system, terminal, and storage medium for monitoring the arbitration status of a storage cluster. The method includes: constructing a first bitmap based on the connection relationship between controllers and arbitration devices in the storage cluster; constructing a second bitmap based on the information of activated controllers in the storage cluster; determining whether the first bitmap covers all controller information in the second bitmap: if yes, the arbitration status is considered normal; if no, the arbitration status is considered abnormal. This invention can monitor the connection status between the storage cluster and the arbitration device in real time, and report an alarm when an anomaly is detected, prompting the customer to repair it promptly and avoiding losses due to arbitration failure in the event of a split-brain scenario.

Description

Technical Field

[0001] This invention belongs to the field of storage cluster technology, specifically relating to a storage cluster arbitration status monitoring method, system, terminal, and storage medium. Background Technology

[0002] Clustered storage aggregates storage space from multiple storage devices into a storage pool that provides a unified access and management interface for application servers. Applications can transparently access and utilize the disks on all storage devices through this interface, maximizing storage device performance and disk utilization. Data will be stored and retrieved from multiple storage devices according to certain rules to achieve higher concurrent access performance.

[0003] As society develops, data security is increasingly valued by enterprises. Different storage vendors employ various solutions for multi-controller cluster split-brain scenarios, with the use of third-party devices to provide arbitration services being widely adopted. These third-party devices connect to all storage controllers. When a storage cluster splits into multiple mutually invisible sub-clusters due to a failure, the device can preempt the sub-cluster that will be arbitrated to continue operating. Controllers outside the sub-cluster that leave the cluster can rejoin, preventing data errors caused by split-brain.

[0004] Therefore, monitoring the status of arbitration services is crucial for ensuring the stability of the storage cluster. Summary of the Invention

[0005] To address the lack of an effective method for monitoring arbitration status in existing technologies, this invention provides a storage cluster arbitration status monitoring method, system, terminal, and storage medium to solve the aforementioned technical problems.

[0006] In a first aspect, the present invention provides a method for monitoring the arbitration status of a storage cluster, comprising:

[0007] A first-order graph is constructed based on the connection relationship between the controller and the arbitration device in the storage cluster;

[0008] A second bitmap is constructed based on the information of the controllers that have been activated in the storage cluster;

[0009] Determine if the first bitmap contains all controller information from the second bitmap:

[0010] If so, the arbitration status is determined to be normal:

[0011] If not, the arbitration status is deemed abnormal.

[0012] Furthermore, a first-order graph is constructed based on the connection relationship between the controller and the arbitration device in the storage cluster, including:

[0013] Assign IDs to all arbitration devices in the storage cluster;

[0014] Each controller generates its own sub-bitmap based on the ID of the arbitration device it is connected to;

[0015] All sub-bitmaps are aggregated, and during the aggregation process, arbitration device IDs are matched on the sub-bitmaps to identify arbitration device IDs that are connected to multiple controllers simultaneously.

[0016] The first bitmap is generated based on the connection relationship between the controller and arbitration device ID obtained by summarizing all sub-bitmaps, and the first bitmap is shared with all controllers.

[0017] Furthermore, a second bitmap is constructed based on the information of the active controllers in the storage cluster, including:

[0018] Periodically identify all active controllers in the storage cluster and store the information of the most recently identified controller as a second bitmap.

[0019] Furthermore, determine whether the first bitmap covers all controller information in the second bitmap, including:

[0020] Determine whether the controller information in the second bitmap has an arbitration device with a connection relationship in the first bitmap.

[0021] Secondly, the present invention provides a storage cluster arbitration status monitoring system, comprising:

[0022] The first building unit is used to construct the first-level graph based on the connection relationship between the controller and the arbitration device in the storage cluster.

[0023] The second building unit is used to build a second bitmap based on the information of the controllers that have been activated in the storage cluster;

[0024] The bitmap judgment unit is used to determine whether the first bitmap covers all the controller information in the second bitmap.

[0025] The normal determination unit is used to determine that the arbitration status is normal if the first bit diagram covers all the controller information in the second bit diagram:

[0026] An anomaly determination unit is used to determine that the arbitration status is abnormal if the first bit map fails to cover all controller information in the second bit map.

[0027] Furthermore, the first building unit includes:

[0028] The ID allocation module is used to assign IDs to all arbitration devices in the storage cluster.

[0029] The bitmap generation module is used by each controller to generate its own sub-bitmap based on the arbitration device ID connected to it.

[0030] The connection identification module is used to summarize all sub-bitmaps. During the summarization process, the arbitration device IDs of the sub-bitmaps are matched to identify the arbitration device IDs that are connected to multiple controllers at the same time.

[0031] The bitmap sharing module is used to generate a first bitmap based on the connection relationship between the controller and arbitration device ID obtained by summarizing all sub-bitmaps, and to share the first bitmap with all controllers.

[0032] Furthermore, the second building unit includes:

[0033] The second building module is used to periodically identify all active controllers in the storage cluster and store the information of the most recently identified controller as a second bitmap.

[0034] Furthermore, the bitmap determination unit includes:

[0035] The bitmap matching module is used to determine whether the controller information in the second bitmap contains arbitration devices with connection relationships in the first bitmap.

[0036] Thirdly, a terminal is provided, including:

[0037] Processor, memory, among which,

[0038] This memory is used to store computer programs.

[0039] The processor is used to retrieve and run the computer program from memory, causing the terminal to perform the terminal method described above.

[0040] Fourthly, a computer storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the methods described in the above aspects.

[0041] The beneficial effects of the present invention are that the storage cluster arbitration status monitoring method, system, terminal and storage medium provided by the present invention can monitor the connection status between the storage cluster and the arbitration in real time, and report an alarm when an anomaly is detected, prompting the customer to repair it in time, so as to avoid losses caused by arbitration failure in the event of a split-brain situation.

[0042] Furthermore, the design principle of this invention is reliable, the structure is simple, and it has a very wide range of application prospects. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram illustrating the principle of a method according to an embodiment of the present invention.

[0046] Figure 3 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0047] Figure 4 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0048] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0049] Traditional storage systems are limited by their physical components (e.g., controller performance, bus performance, number of disk drives, number of connected servers, memory size, NAS head performance, etc.) and functional limitations (e.g., supported file system capacity, coupling of metadata and data processing pathways, number of snapshots or replications, etc.), leading to storage system bottlenecks. When encountering a storage system bottleneck, two options emerge: one is to use a single storage system with more powerful hardware; the other is to use several storage systems with average performance to form a "storage cluster." As a widely used architecture, "clustering," when applied to storage to form "clustered storage," can provide proportionally increased performance, capacity, reliability, and availability of storage resources, overcoming the limitations of single-machine devices. Traditional SAN and NAS provide two different levels of storage services: block storage and file storage. Clustered storage also includes clustered storage systems that provide both block and file storage services. The advantages of clustered storage mainly lie in improving the overall performance of parallel or partitioned I / O, especially for workflows, read-intensive tasks, and large file access, while reducing overall costs by using lower-cost servers.

[0050] In multi-controller storage clusters, arbitration devices play a crucial role. Different storage vendors employ various solutions for split-brain scenarios in multi-controller clusters, with the use of third-party devices to provide arbitration services being a widely adopted approach. These third-party devices connect to all storage controllers. When a storage cluster splits into multiple mutually invisible sub-clusters due to a failure, the device can preempt the arbitration sub-cluster to continue operating. Controllers outside the sub-cluster that leave the cluster can rejoin, preventing data errors caused by split-brain.

[0051] An arbitrator is a disk shared by two or more nodes that votes to arbitrate whether a cluster should run. A cluster can only run if a sufficient number of arbitrator votes are reached. When a cluster is split into several separate groups of nodes, the arbitrator is used to determine which groups constitute the new cluster. Both the cluster nodes and the arbitrator vote to form an arbitrator. By default, a cluster node has one arbitrator vote count when it boots up and becomes a cluster member. A node's vote count can be zero when it is being installed or when an administrator puts a node into maintenance mode. The arbitrator acquires its arbitrator vote count, which is based on the number of nodes connected to the device. When an arbitrator is set up, it acquires a maximum vote count of N-1, where N is the number of arbitrator votes connected to the device. For example, an arbitrator connected to two nodes with non-zero vote counts will have an arbitrator vote count of one (two minus one). Sun Cluster systems attempt to prevent data corruption and ensure data integrity. Because cluster nodes share data and resources, a cluster should never be split into multiple independent partitions that are active simultaneously. CMM ensures that only one cluster is active at any given time. Cluster partitioning can cause two types of problems: cluster splitting and amnesia. Cluster splits occur when the cluster interconnect between nodes fails and the cluster is split into multiple sub-clusters, each believing itself to be the only partition. Sub-clusters unaware of other sub-clusters can lead to shared resource conflicts, such as duplicate network addresses and data corruption. Amnesia occurs when all nodes believe the cluster is in an unstable state. Consider a two-node cluster with node A and node B. If node A shuts down, only the CCR configuration data in node B is updated, not in node A. If node B later shuts down and node A reboots, node A will run with the old CCR. This state is called amnesia, and it can cause the cluster to run with outdated configuration information. Cluster splits and amnesia can be avoided by assigning each node a vote and stipulating that only a majority of votes constitutes a valid cluster. The partition with the majority of votes has arbitration and is therefore allowed to run. This majority voting mechanism works for clusters with more than two nodes. In a two-node cluster, the majority is two. If such a cluster is split into two partitions, external voting is required to arbitrate one of the partitions. This external voting is provided by the arbitration device. The arbitration device can be any disk shared between the two nodes.

[0052] In a multi-controller storage cluster, multiple arbitration devices may provide arbitration services. If any of these devices fails, the arbitration service will be compromised, potentially leading to data integrity issues. Therefore, monitoring the status of the arbitration service is crucial for the stability of the storage cluster.

[0053] Figure 1This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a storage cluster arbitration status monitoring system.

[0054] like Figure 1 As shown, the method includes:

[0055] Step 110: Construct a first-order graph based on the connection relationship between the controller and the arbitration device in the storage cluster;

[0056] Step 120: Construct a second bitmap based on the information of the active controllers in the storage cluster;

[0057] Step 130: Determine whether the first bitmap covers all controller information in the second bitmap.

[0058] Step 140: If yes, then the arbitration status is determined to be normal.

[0059] Step 150: If not, the arbitration status is determined to be abnormal.

[0060] This method can monitor the connection status between the storage cluster and the arbitration in real time. When an anomaly is detected, an alarm is reported to prompt the customer to repair it in time and avoid losses caused by arbitration failure in the event of a split-brain situation.

[0061] To facilitate understanding of the present invention, the following description further illustrates the storage cluster arbitration status monitoring method provided by the present invention, based on the principle of the storage cluster arbitration status monitoring method and the process of monitoring the storage cluster arbitration status in the embodiments.

[0062] Specifically, the storage cluster arbitration status monitoring method includes:

[0063] S1. Construct the first-order graph based on the connection relationship between the controller and the arbitration device in the storage cluster.

[0064] Each controller assigns an ID to all arbitration devices in the storage cluster; each controller generates its own sub-bitmap based on the ID of the arbitration device it is connected to; all sub-bitmaps are aggregated, and during the aggregation process, arbitration device IDs are matched in the sub-bitmaps to identify arbitration device IDs that are connected to multiple controllers simultaneously; a first-order bitmap is generated based on the connection relationship between controllers and arbitration device IDs obtained from the aggregation of all sub-bitmaps, and the first-order bitmap is shared with all controllers.

[0065] Specifically, the cluster uniformly assigns IDs to all arbitration devices, incrementing the ID with each connection to ensure that different arbitration devices are assigned unique IDs. A connection bitmap between the arbitration devices and the controller is maintained; when the controller receives a connection request from an arbitration device, a new bitmap is added; when the controller disconnects from the arbitration device, the corresponding connection bitmap is cleared. When the same arbitration device connects to different nodes, arbitration information is shared across all controllers by matching the user's UID (user account).

[0066] S2. Construct a second bitmap based on the information of the controllers that have been activated in the storage cluster.

[0067] Periodically identify all active controllers in the storage cluster and store the information of the most recently identified controller as a second bitmap.

[0068] Specifically, the bitmap of all currently active controllers is maintained periodically as a second bitmap.

[0069] S3. Determine whether the first bitmap covers all controller information in the second bitmap: if yes, the arbitration status is normal; if no, the arbitration status is abnormal.

[0070] Determine whether the controller information in the second bitmap contains arbitration devices with connection relationships in the first bitmap. The specific determination method is as follows:

[0071] The controller counts the number of third-party arbitration devices and the number of arbitration devices connected to all controllers.

[0072] Set a timer for arbitration health checks and perform periodic checks;

[0073] Periodic checks will detect if an arbitration device is not connected to any of the controllers. If this is detected, the alarm presence flag will be enabled, triggering an alarm report and notifying the customer to correct it. If the alarm presence flag is enabled, it indicates that a previously existing alarm has not yet been fixed and does not need to be reported again.

[0074] When each arbitration is connected to the controller, it is determined whether the alarm presence flag is enabled. If enabled, the alarm is automatically repaired.

[0075] Both alarm reporting and alarm clearing use the same timed detection, but alarm clearing also has the following two methods:

[0076] When a new controller connects to the arbitrator, the alarm can be cleared if all controllers in the cluster are connected to the arbitrator service at that time; when a controller leaves the cluster, the alarm can be cleared if only that controller is connected to the arbitrator (the alarm is cleared if no controller is connected).

[0077] For a clearer explanation of the implementation of this invention, please refer to [link / reference]. Figure 2When the cluster has four controllers active, and controllers 1, 2, and 4 are connected to the arbitrator...

[0078] (1) The arbitration management module assigns arbitration IDs;

[0079] (2) Maintain the bitmap "1111" of the active controller and maintain the bitmap "1011" of the controller for connection arbitration;

[0080] (3) The scheduled task module starts a timer and calls the health detection module to perform a check every two minutes;

[0081] (4) The health detection module finds that the controller bitmap “1111” and the controller bitmap “1011” connected to the arbitration are inconsistent and reports an alarm to the alarm processing module.

[0082] (5) The alarm processing module determines whether there is an unreported alarm. If not, it reports the current alarm.

[0083] (6) When controller 3 is connected to the arbitration, the health detection module detects that the arbitration has been reconnected to all controllers and reports to the alarm processing module to automatically repair the alarm.

[0084] like Figure 3 As shown, the system 300 includes:

[0085] The first building unit 310 is used to build the first bit graph based on the connection relationship between the controller and the arbitration device in the storage cluster;

[0086] The second building unit 320 is used to build a second bitmap based on the controller information that has been activated in the storage cluster;

[0087] Bitmap determination unit 330 is used to determine whether the first bitmap covers all controller information in the second bitmap;

[0088] The normal determination unit 340 is used to determine that the arbitration status is normal if the first bit map covers all the controller information in the second bit map.

[0089] The anomaly determination unit 350 is used to determine that the arbitration status is abnormal if the first bit map fails to cover all controller information in the second bit map.

[0090] Optionally, as an embodiment of the present invention, the first building unit includes:

[0091] The ID allocation module is used to assign IDs to all arbitration devices in the storage cluster.

[0092] The bitmap generation module is used by each controller to generate its own sub-bitmap based on the arbitration device ID connected to it.

[0093] The connection identification module is used to summarize all sub-bitmaps. During the summarization process, the arbitration device IDs of the sub-bitmaps are matched to identify the arbitration device IDs that are connected to multiple controllers at the same time.

[0094] The bitmap sharing module is used to generate a first bitmap based on the connection relationship between the controller and arbitration device ID obtained by summarizing all sub-bitmaps, and to share the first bitmap with all controllers.

[0095] Optionally, as an embodiment of the present invention, the second building unit includes:

[0096] The second building module is used to periodically identify all active controllers in the storage cluster and store the information of the most recently identified controller as a second bitmap.

[0097] Optionally, as an embodiment of the present invention, the bitmap determination unit includes:

[0098] The bitmap matching module is used to determine whether the controller information in the second bitmap contains arbitration devices with connection relationships in the first bitmap.

[0099] Figure 4 This is a schematic diagram of the structure of a terminal 400 provided in an embodiment of the present invention. The terminal 400 can be used to execute the storage cluster arbitration status monitoring method provided in the embodiment of the present invention.

[0100] The terminal 400 may include a processor 410, a memory 420, and a communication unit 430. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0101] The memory 420 can be used to store the execution instructions of the processor 410. The memory 420 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 420 are executed by the processor 410, the terminal 400 is able to perform some or all of the steps in the above method embodiments.

[0102] The processor 410 serves as the control center of the storage terminal, connecting various parts of the electronic terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 420, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 410 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0103] The communication unit 430 is used to establish a communication channel, enabling the storage terminal to communicate with other terminals. It can receive user data sent by other terminals or send user data to other terminals.

[0104] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0105] Therefore, this invention can monitor the connection status between the storage cluster and the arbitration in real time. When an anomaly is detected, an alarm is reported to prompt the customer to repair it in time, so as to avoid losses caused by arbitration failure in the event of a split-brain situation. The technical effects achieved by this embodiment can be referred to in the description above, and will not be repeated here.

[0106] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0107] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0108] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should also be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. A method for monitoring arbitration state of a storage cluster, the method comprising: include: A first-order graph is constructed based on the connection relationship between the controller and the arbitration device in the storage cluster; A second bitmap is constructed based on the information of the controllers that have been activated in the storage cluster; Determine if the first bitmap contains all controller information from the second bitmap: If so, the arbitration status is determined to be normal: If not, the arbitration status is deemed abnormal; A first-order graph is constructed based on the connection relationship between the controller and the arbitration device in the storage cluster, including: Assign IDs to all arbitration devices in the storage cluster; Each controller generates its own sub-bitmap based on the ID of the arbitration device it is connected to; All sub-bitmaps are aggregated, and during the aggregation process, arbitration device IDs are matched on the sub-bitmaps to identify arbitration device IDs that are connected to multiple controllers simultaneously. The first bitmap is generated based on the connection relationship between the controller and arbitration device ID obtained by summarizing all sub-bitmaps, and the first bitmap is shared with all controllers; A second bitmap is constructed based on the information of the active controllers in the storage cluster, including: Periodically identify all active controllers in the storage cluster and store the information of the most recently identified controller as a second bitmap.

2. The method according to claim 1, characterized in that, Determine whether the first bitmap covers all controller information in the second bitmap, including: Determine whether the controller information in the second bitmap has an arbitration device with a connection relationship in the first bitmap.

3. A storage cluster arbitration state monitoring system, comprising: include: The first building unit is used to construct the first-level graph based on the connection relationship between the controller and the arbitration device in the storage cluster. The second building unit is used to build a second bitmap based on the information of the controllers that have been activated in the storage cluster; The bitmap judgment unit is used to determine whether the first bitmap covers all the controller information in the second bitmap. The normal determination unit is used to determine that the arbitration status is normal if the first bit diagram covers all the controller information in the second bit diagram: An anomaly determination unit is used to determine that the arbitration status is abnormal if the first bit diagram fails to cover all controller information in the second bit diagram. The first building unit includes: The ID allocation module is used to assign IDs to all arbitration devices in the storage cluster. The bitmap generation module is used by each controller to generate its own sub-bitmap based on the arbitration device ID connected to it. The connection identification module is used to summarize all sub-bitmaps. During the summarization process, the arbitration device IDs of the sub-bitmaps are matched to identify the arbitration device IDs that are connected to multiple controllers at the same time. The bitmap sharing module is used to generate a first bitmap based on the connection relationship between the controller and arbitration device ID obtained by summarizing all sub-bitmaps, and to share the first bitmap with all controllers; The second building unit includes: The second building module is used to periodically identify all active controllers in the storage cluster and store the information of the most recently identified controller as a second bitmap.

4. The system of claim 3, wherein, The bitmap determination unit includes: The bitmap matching module is used to determine whether the controller information in the second bitmap contains arbitration devices with connection relationships in the first bitmap.

5. A terminal, characterized by comprising: include: processor; Memory used to store the processor's execution instructions; The processor is configured to perform the method according to any one of claims 1-2.

6. A computer readable storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-2.

Images