A cluster brain split processing method, system, device and storage medium

Abstract

The application discloses a cluster brain split processing method, comprising the following steps: in response to detecting a cluster brain split, dividing the cluster into multiple sub-clusters according to the group attribute of each storage pool; adjusting the service link of the corresponding sub-cluster according to the group attribute of the each storage pool; in response to the sub-cluster receiving a service request, accessing the corresponding storage pool by using the adjusted service link. The application also discloses a system, a computer device and a readable storage medium. The scheme provided by the application is a brain split processing mechanism in a non-arbitration mode, that is, the cluster is split into sub-clusters according to the service resource attribution, and each sub-cluster provides services. In this way, the influence caused by the instability of the arbitration device is avoided in the non-arbitration mode, and the environmental dependence is reduced; when the cluster brain splits, the cluster is split, the service attribution relationship is determined according to the internal resource dependence, each sub-cluster provides the service belonging to itself, and the service interruption is avoided.

Description

Technical Field

[0001] This invention relates to the field of clustering, and more specifically to a cluster split-brain processing method, system, device, and storage medium. Background Technology

[0002] Currently, a non-fully redundant SAN storage multi-controller cluster architecture exists in the storage business. This involves storage heads (or gateways, referred to as "headers" below) mutually backing up to establish a cluster and provide SAN services, but each head has exclusive access to its own backend disk enclosure. The advantage of this clustering approach is increased bandwidth for front-end services, and the ability to mutually back up and take over services when the head's front-end network fails, as shown in the diagram below, where application servers can access the same disk enclosure through two heads. However, this approach also has a significant drawback: if one head fails, the corresponding backend disks become inaccessible, preventing other heads from taking over and providing backup. To prevent cluster split-brain in the event of an internal fault in this cluster architecture, arbitration is currently used to handle the situation, keeping one surviving node to provide services while shutting down the others.

[0003] Currently, arbitration is used to address the following scenarios: In scenario 1, when a storage site is indeed abnormal, the normal node can confirm the other party's status, thus ensuring that it continues to provide services; In scenario 2, when both storage sites are normal but the heartbeat network is abnormal, according to the arbitration principle, one site is selected to provide services normally, and the other site stops providing services.

[0004] When existing technologies are applied to SAN storage multi-controller clusters that are not fully mutually redundant (i.e., the storage heads mutually redundantly establish a cluster, but the back-end disk enclosures are each exclusively used by the head unit), the following problems may occur:

[0005] 1) In scenario 2, arbitration will shut down one storage site in the cluster, but the data on its corresponding disk enclosure will be inaccessible on another storage site, causing related services to be unavailable on the surviving storage site, resulting in service interruption.

[0006] 2) Providing an arbitration mechanism requires third-party storage or a stable IP address for a long period of time. In actual business deployment, this depends on the user environment, leading to customer dissatisfaction and increasing the risk of a split-brain scenario due to the instability of the arbitration equipment. Summary of the Invention

[0007] In view of this, in order to overcome at least one aspect of the above-mentioned problems, embodiments of the present invention propose a method for processing clustered split-brain syndromes, comprising the following steps:

[0008] In response to the detection of a cluster split-brain, the cluster is divided into multiple sub-clusters based on the group attributes of each storage pool;

[0009] Adjust the business links of the corresponding sub-clusters according to the group attributes of each storage pool;

[0010] In response to the sub-cluster receiving a service request, it accesses the corresponding storage pool using the adjusted service link.

[0011] In some embodiments, it also includes:

[0012] In response to the creation of a new storage pool, group attributes are set according to the home site corresponding to the new storage pool.

[0013] In some embodiments, adjusting the service links of the corresponding sub-clusters according to the group attributes of each storage pool further includes:

[0014] Mark the business links corresponding to other group attributes under each sub-cluster as unavailable, and only retain the business links corresponding to its own group attributes.

[0015] In some embodiments, it also includes:

[0016] In response to the cluster returning to normal, the service links corresponding to other group attributes that were marked as unavailable in each sub-cluster are remarked as available.

[0017] The cluster is reorganized using each sub-cluster.

[0018] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a clustered split-brain processing system, comprising:

[0019] The split module is configured to, in response to the detection of a cluster split-brain, divide the cluster into multiple sub-clusters based on the group attributes of each storage pool.

[0020] The adjustment module is configured to adjust the service links of the corresponding sub-clusters based on the group attributes of each storage pool.

[0021] The access module is configured to access the corresponding storage pool using the adjusted service link in response to a service request received by the sub-cluster.

[0022] In some embodiments, a creation module is also included, configured as follows:

[0023] In response to the creation of a new storage pool, group attributes are set according to the home site corresponding to the new storage pool.

[0024] In some embodiments, the adjustment module is further configured to:

[0025] Mark the business links corresponding to other group attributes under each sub-cluster as unavailable, and only retain the business links corresponding to its own group attributes.

[0026] In some embodiments, the adjustment module is further configured to:

[0027] In response to the cluster returning to normal, the service links corresponding to other group attributes that were marked as unavailable in each sub-cluster are remarked as available.

[0028] The cluster is reorganized using each sub-cluster.

[0029] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer device, comprising:

[0030] At least one processor; and

[0031] A memory storing a computer program executable on the processor, characterized in that the processor performs the following steps when executing the program:

[0032] In response to the detection of a cluster split-brain, the cluster is divided into multiple sub-clusters based on the group attributes of each storage pool;

[0033] Adjust the business links of the corresponding sub-clusters according to the group attributes of each storage pool;

[0034] In response to the sub-cluster receiving a service request, it accesses the corresponding storage pool using the adjusted service link.

[0035] In some embodiments, it also includes:

[0036] In response to the creation of a new storage pool, group attributes are set according to the home site corresponding to the new storage pool.

[0037] In some embodiments, adjusting the service links of the corresponding sub-clusters according to the group attributes of each storage pool further includes:

[0038] Mark the business links corresponding to other group attributes under each sub-cluster as unavailable, and only retain the business links corresponding to its own group attributes.

[0039] In some embodiments, it also includes:

[0040] In response to the cluster returning to normal, the service links corresponding to other group attributes that were marked as unavailable in each sub-cluster are remarked as available.

[0041] The cluster is reorganized using each sub-cluster.

[0042] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0043] In response to the detection of a cluster split-brain, the cluster is divided into multiple sub-clusters based on the group attributes of each storage pool;

[0044] Adjust the business links of the corresponding sub-clusters according to the group attributes of each storage pool;

[0045] In response to the sub-cluster receiving a service request, it accesses the corresponding storage pool using the adjusted service link.

[0046] In some embodiments, it also includes:

[0047] In response to the creation of a new storage pool, group attributes are set according to the home site corresponding to the new storage pool.

[0048] In some embodiments, adjusting the service links of the corresponding sub-clusters according to the group attributes of each storage pool further includes:

[0049] Mark the business links corresponding to other group attributes under each sub-cluster as unavailable, and only retain the business links corresponding to its own group attributes.

[0050] In some embodiments, it also includes:

[0051] In response to the cluster returning to normal, the service links corresponding to other group attributes that were marked as unavailable in each sub-cluster are remarked as available.

[0052] The cluster is reorganized using each sub-cluster.

[0053] This invention offers one of the following beneficial technical effects: The proposed solution is a brain-split processing mechanism without arbitration, which splits the cluster into sub-clusters based on the ownership of business resources, with each sub-cluster providing its own services. This eliminates the impact of unstable arbitration equipment and reduces environmental dependence; when a cluster experiences brain split, the cluster is split, and the business ownership relationship is determined based on internal resource dependencies, with each sub-cluster providing its own assigned services, ensuring uninterrupted service. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0055] Figure 1 A schematic flowchart of a clustered split-brain processing method provided in an embodiment of the present invention;

[0056] Figure 2 A schematic diagram illustrating the service path adjustment after a cluster splits into two parts, provided as an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the structure of the clustered split-brain treatment system provided in the embodiments of the present invention;

[0058] Figure 4 A schematic diagram of the structure of a computer device provided for an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the structure of a computer-readable storage medium provided for an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0061] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0062] According to one aspect of the present invention, embodiments of the present invention provide a method for processing clustered split-brain syndromes, such as... Figure 1 As shown, it may include the following steps:

[0063] S1, in response to detecting a cluster split-brain, the cluster is divided into multiple sub-clusters according to the group attributes of each storage pool;

[0064] S2, adjust the service links of the corresponding sub-clusters according to the group attributes of each storage pool;

[0065] S3, in response to the sub-cluster receiving a service request, access the corresponding storage pool using the adjusted service link.

[0066] The proposed solution in this invention is a split-brain handling mechanism without arbitration, which splits the cluster into sub-clusters based on the ownership of business resources, with each sub-cluster providing its own services. This eliminates the impact of unstable arbitration equipment and reduces environmental dependence by operating without arbitration. When a cluster experiences a split-brain event, the cluster is split, and the ownership of services is determined based on internal resource dependencies. Each sub-cluster provides its own assigned services, ensuring uninterrupted service.

[0067] In some embodiments, the solution proposed in this invention mainly includes setting the service ownership attribute according to disk resource dependency when creating a service, a heartbeat detection unit performing heartbeat detection and feedback, splitting the cluster into sub-clusters to provide services when the heartbeat is abnormal, and restoring the cluster when the heartbeat is restored.

[0068] In some embodiments, it also includes:

[0069] In response to the creation of a new storage pool, group attributes are set according to the home site corresponding to the new storage pool.

[0070] Specifically, when configuring storage services, the first step is to create a pool. According to an embodiment of the present invention, the pool's home site attribute is set when creating the pool, named the iogroup attribute. Subsequently, only disks and RAIDs belonging to this iogroup can be added to the pool, and volumes created based on this pool synchronously inherit the iogroup attribute of this pool.

[0071] In some embodiments, in step S1, in response to detecting a cluster split-brain, the cluster is divided into multiple sub-clusters according to the group attributes of each storage pool. Specifically, the heartbeat detection unit includes a heartbeat flag and a heartbeat detection function. A flag of 0 indicates that the current heartbeat detection is normal, and a flag of 1 indicates that the heartbeat detection is abnormal. If a normal heartbeat is detected when the flag is 0, the device does not perform any processing action; if a heartbeat abnormality is detected when the flag is 0, the flag is set to 1 (multiple detections can be set to prevent false alarms), and a heartbeat abnormality is reported; if a heartbeat abnormality is detected when the flag is 1, the device does not perform any processing action; if a normal heartbeat is detected when the flag is 1, the flag is set to 0 (multiple detections can be set to prevent false alarms), and a heartbeat recovery is reported.

[0072] In some embodiments, adjusting the service links of the corresponding sub-clusters according to the group attributes of each storage pool further includes:

[0073] Mark the business links corresponding to other group attributes under each sub-cluster as unavailable, and only retain the business links corresponding to its own group attributes.

[0074] Specifically, such as Figure 2 As shown, when the cluster receives a heartbeat anomaly signal from the heartbeat detection unit, the cluster splitting unit operates, splitting the cluster into two sub-clusters based on iogroups. Each sub-cluster provides its own iogroup attribute for its related services and marks service links with other iogroup attributes as unavailable. This changes the multipath status of the storage service volumes scanned by the application server from two sets of active paths to only one set of active paths, with the other set of paths scanned as Fail. Services on each storage site can still be accessed. Figure 2As shown, when a split-brain event occurs, the application server can only access disk cabinet 1 and disk cabinet 2 separately. It cannot access disk cabinet 2 through disk cabinet 1 or disk cabinet 1 through disk cabinet 2. In other words, the business links corresponding to other group attributes under each sub-cluster are marked as unavailable, and only the business links corresponding to its own group attributes are retained.

[0075] In some embodiments, it also includes:

[0076] In response to the cluster returning to normal, the service links corresponding to other group attributes that were marked as unavailable in each sub-cluster are remarked as available.

[0077] The cluster is reorganized using each sub-cluster.

[0078] Specifically, when the cluster receives a heartbeat recovery signal from the heartbeat detection unit, it performs status checks according to the original configuration file. Once the rebuild conditions are met, cluster reorganization is initiated. Each service completes path extension to the other's iogroup and marks service links previously marked as unavailable as available. Thus, the multipath status of the storage service volume scanned by the application server changes from one group of active paths and one group of failed paths to two groups of active valid paths, and the cluster status is restored.

[0079] The proposed solution in this invention is a split-brain handling mechanism without arbitration, which splits the cluster into sub-clusters based on the ownership of business resources, with each sub-cluster providing its own services. This eliminates the impact of unstable arbitration equipment and reduces environmental dependence by operating without arbitration. When a cluster experiences a split-brain event, the cluster is split, and the ownership of services is determined based on internal resource dependencies. Each sub-cluster provides its own assigned services, ensuring uninterrupted service.

[0080] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a clustered split-brain processing system 400, such as... Figure 3 As shown, it includes:

[0081] The splitting module 401 is configured to, in response to the detection of a cluster split-brain, divide the cluster into multiple sub-clusters according to the group attributes of each storage pool.

[0082] Adjustment module 402 is configured to adjust the service links of the corresponding sub-clusters according to the group attributes of each storage pool;

[0083] Access module 403 is configured to access the corresponding storage pool using the adjusted service link in response to a service request received by the sub-cluster.

[0084] In some embodiments, a creation module is also included, configured as follows:

[0085] In response to the creation of a new storage pool, group attributes are set according to the home site corresponding to the new storage pool.

[0086] In some embodiments, the adjustment module 402 is further configured to:

[0087] Mark the business links corresponding to other group attributes under each sub-cluster as unavailable, and only retain the business links corresponding to its own group attributes.

[0088] In some embodiments, the adjustment module 402 is further configured to:

[0089] In response to the cluster returning to normal, the service links corresponding to other group attributes that were marked as unavailable in each sub-cluster are remarked as available.

[0090] The cluster is reorganized using each sub-cluster.

[0091] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 4 As shown, embodiments of the present invention also provide a computer device 501, comprising:

[0092] At least one processor 520; and

[0093] Memory 510 stores a computer program 511 that can run on a processor. When the processor 520 executes the program, it performs the following steps:

[0094] S1, in response to detecting a cluster split-brain, the cluster is divided into multiple sub-clusters according to the group attributes of each storage pool;

[0095] S2, adjust the service links of the corresponding sub-clusters according to the group attributes of each storage pool;

[0096] S3, in response to the sub-cluster receiving a service request, access the corresponding storage pool using the adjusted service link.

[0097] The proposed solution in this invention is a split-brain handling mechanism without arbitration, which splits the cluster into sub-clusters based on the ownership of business resources, with each sub-cluster providing its own services. This eliminates the impact of unstable arbitration equipment and reduces environmental dependence by operating without arbitration. When a cluster experiences a split-brain event, the cluster is split, and the ownership of services is determined based on internal resource dependencies. Each sub-cluster provides its own assigned services, ensuring uninterrupted service.

[0098] In some embodiments, it also includes:

[0099] In response to the creation of a new storage pool, group attributes are set according to the home site corresponding to the new storage pool.

[0100] In some embodiments, adjusting the service links of the corresponding sub-clusters according to the group attributes of each storage pool further includes:

[0101] Mark the business links corresponding to other group attributes under each sub-cluster as unavailable, and only retain the business links corresponding to its own group attributes.

[0102] In some embodiments, it also includes:

[0103] In response to the cluster returning to normal, the service links corresponding to other group attributes that were marked as unavailable in each sub-cluster are remarked as available.

[0104] The cluster is reorganized using each sub-cluster.

[0105] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 5 As shown, embodiments of the present invention also provide a computer-readable storage medium 601, which stores a computer program 610. When the computer program 610 is executed by a processor, it performs the following steps:

[0106] S1, in response to detecting a cluster split-brain, the cluster is divided into multiple sub-clusters according to the group attributes of each storage pool;

[0107] S2, adjust the service links of the corresponding sub-clusters according to the group attributes of each storage pool;

[0108] S3, in response to the sub-cluster receiving a service request, access the corresponding storage pool using the adjusted service link.

[0109] The proposed solution in this invention is a split-brain processing mechanism without arbitration, which splits the cluster into sub-clusters based on the ownership of business resources, and each sub-cluster provides its own services.

[0110] In some embodiments, it also includes:

[0111] In response to the creation of a new storage pool, group attributes are set according to the home site corresponding to the new storage pool.

[0112] In some embodiments, adjusting the service links of the corresponding sub-clusters according to the group attributes of each storage pool further includes:

[0113] Mark the business links corresponding to other group attributes under each sub-cluster as unavailable, and only retain the business links corresponding to its own group attributes.

[0114] In some embodiments, it also includes:

[0115] In response to the cluster returning to normal, the service links corresponding to other group attributes that were marked as unavailable in each sub-cluster are remarked as available.

[0116] The cluster is reorganized using each sub-cluster.

[0117] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.

[0118] Furthermore, it should be understood that the computer-readable storage medium (e.g., memory) described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.

[0119] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0120] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0121] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0122] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0123] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0124] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method of cluster split processing, the method comprising: The method comprises the following steps: in response to detecting cluster split-brain, dividing the cluster into multiple sub-clusters according to group attributes of each storage pool; adjusting service links of the corresponding sub-cluster according to the group attributes of each storage pool; in response to the sub-cluster receiving a service request, accessing the corresponding storage pool by using the adjusted service links; adjusting service links of the corresponding sub-cluster according to the group attributes of each storage pool, further comprising: marking service links corresponding to other group attributes under each sub-cluster as unavailable, and only keeping service links corresponding to the own group attribute.

2. The method of claim 1, wherein, Further comprising: in response to creating a new storage pool, setting group attributes according to a home site corresponding to the new storage pool.

3. The method of claim 1, wherein, Further comprising: in response to the cluster returning to normal, re-marking service links corresponding to other group attributes in each sub-cluster as available; reorganizing the cluster by using each sub-cluster.

4. A cluster split-brain handling system, characterized by comprise: a splitting module configured to, in response to detecting cluster split-brain, divide the cluster into multiple sub-clusters according to group attributes of each storage pool; an adjusting module configured to adjust service links of the corresponding sub-cluster according to the group attributes of each storage pool; an accessing module configured to, in response to the sub-cluster receiving a service request, access the corresponding storage pool by using the adjusted service links; the adjusting module is further configured to: mark service links corresponding to other group attributes under each sub-cluster as unavailable, and only keep service links corresponding to the own group attribute.

5. The system of claim 4, wherein, Further comprising a creating module configured to: in response to creating a new storage pool, set group attributes according to a home site corresponding to the new storage pool.

6. The system of claim 4, wherein, The adjusting module is further configured to: in response to the cluster returning to normal, re-mark service links corresponding to other group attributes in each sub-cluster as available; reorganize the cluster by using each sub-cluster.

7. A computer device, comprising: at least one processor; and a memory, the memory storing a computer program executable on the processor, characterized in that the processor executes the program to perform the steps of the method of any one of claims 1-3.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to perform the steps of the method of any one of claims 1-3.

Images