Memory system and system construction method

Abstract

Maintain adequate availability of storage systems in a cloud environment. [Solution] The system is provided with multiple storage nodes that constitute a first node group spanning multiple failure domains in a cloud environment. For each node, the domain ID of the failure domain in which the node was generated is obtained, and a second node group is formed from the first node group using the necessary number of nodes with domain IDs that do not overlap as much as possible. The number of member nodes in the second node group that exist in the same failure domain is less than or equal to the redundancy level. The redundancy level is the maximum number of member nodes in the second node group that are allowed to stop simultaneously. In the first node group, nodes other than those in the second node group are spare nodes that can be selected as failback destination nodes.

Description

Technical Field

[0001] The present invention relates to a memory system and a system construction method.

Background Art

[0002] In recent years, as a platform for information processing systems, clouds (especially public clouds) are becoming increasingly popular. In such a public cloud, computer resources and storage resources are provided as IaaS (Infrastructure as a Service) by a public cloud vendor. Also, in order to improve the utilization efficiency of the storage capacity, the demand for SDS (Software Defined Storage) is increasing.

[0003] Generally, in an information processing system, a redundant configuration of server devices is adopted to improve availability and reliability. For example, Patent Document 1 discloses a rebuild method that can quickly return from a degraded configuration in SDS built on a public cloud when a failure occurs.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When a plurality of storage nodes as virtual server devices (virtual machine instances) are arranged in a memory system in a cloud environment, a cluster as a node group composed of two or more storage nodes operates. The cluster has a redundancy degree that means how many storage nodes can stop simultaneously at most and the processing (business) can still continue. If the number of storage nodes that stop exceeds the redundancy degree, the cluster will go down.

[0006] The placement of storage nodes in a storage system is a crucial element for maintaining availability. One perspective on storage node placement is the Fault Domain (FD). An FD is a set of hardware components (e.g., power supplies, servers, or storage devices) that share a single point of failure, such as a power boundary or a rack.

[0007] Storage systems with multiple failure domains are known. If any storage node in a cluster fails due to a node failure or other reason, it is necessary to add a failback storage node to operate in place of the failed storage node and integrate the added storage node into the cluster in order to restore the cluster's redundancy. However, at least one of the following problems (a) and (b) may occur. (a) If a failback storage node is added to the same FD as an existing storage node in the cluster, a failure in that FD will cause all of those storage nodes to shut down simultaneously. If the number of shut down storage nodes exceeds the redundancy limit, the cluster will go down. (b) The floppy disk (FD) to which the expansion will be added may not have enough space to add new storage nodes. For example, this could be the case if the FD to which the expansion will be added already has enough storage nodes used by one or more users other than the users using the cluster. [Means for solving the problem]

[0008] The system is equipped with multiple storage nodes that constitute a first storage node group spanning multiple failure domains in a cloud environment. For each storage node, the domain ID of the failure domain in which the storage node was generated is obtained, and a second storage node group is formed from the first storage node group using the necessary number of storage nodes with as few overlapping domain IDs as possible. The number of member storage nodes in the second storage node group that exist in the same failure domain is less than or equal to the redundancy level. The redundancy level is the maximum number of member storage nodes in the second storage node group that are allowed to fail simultaneously. In the first storage node group, storage nodes other than those in the second storage node group are spare storage nodes that can be selected as failback destination storage nodes. [Effects of the Invention]

[0009] According to the present invention, the availability of a storage system in a cloud environment can be appropriately maintained. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing the overall configuration of the storage system according to the embodiment. [Figure 2] A block diagram showing the general configuration of the storage nodes. [Figure 3] A block diagram illustrating the software and configuration information stored in the memory of a storage node. [Figure 4] A diagram showing the configuration of the storage node management table. [Figure 5] A diagram showing the configuration of the cluster management table. [Figure 6] A block diagram for explaining how to create a program. [Figure 7] A block diagram illustrating cluster creation. [Figure 8] A block diagram to explain failover. [Figure 9] A block diagram illustrating the redundant configuration recovery process. [Figure 10] Block diagram for explaining state change processing. [Figure 11] Flowchart showing cluster construction processing according to an embodiment. [Figure 12] Flowchart showing redundant configuration recovery processing (when a storage node fails). [Figure 13] Flowchart showing state change processing at the time of recovery of a failed storage node. [Figure 14] Block diagram showing another configuration of a cluster. [Figure 15] Block diagram showing the configuration of a cluster when using a spare FD. [Figure 16] Block diagram showing the configuration of a PG having a plurality of clusters. [Figure 17] Block diagram showing an example in which a plurality of PGs straddle a plurality of common FDs.

Mode for Carrying Out the Invention

[0011] The following describes in detail an embodiment of the present invention with reference to the drawings. The following description and drawings are examples for explaining the present invention and do not limit the technical scope of the present invention. In each figure, the same reference numerals are assigned to common configurations.

[0012] In the following description, various information may be described using expressions such as "table", but the various information may be represented by other data structures. In order to indicate independence from the data structure, "XX table", "XX list", etc. may be referred to as "XX information". When explaining the content of each information, expressions such as "identification information", "identifier", "name", "ID", "number", etc. are used, but these can be mutually replaced.

[0013] Also, in the following description, when explaining without distinguishing between elements of the same type, reference numerals or common numbers in the reference numerals are used, and when explaining while distinguishing between elements of the same type, the reference numerals of those elements may be used or the ID assigned to those elements may be used instead of the reference numerals.

[0014] In the following description, there may be cases where the processing performed by executing a program is described. However, the program is executed by at least one processor (e.g., a CPU), and by using an appropriate storage resource (e.g., a memory) and / or an interface device (e.g., a communication port), etc., to perform the defined processing. Therefore, the subject of the processing may be the processor. Similarly, the subject of the processing performed by executing the program may be a controller, a device, a system, a computer, a node, a storage system, a storage device, a server, a management computer, a client, or a host having a processor. The subject of the processing performed by executing the program (e.g., a processor) may include a hardware circuit that performs part or all of the processing. For example, the subject of the processing performed by executing the program may include a hardware circuit that performs encryption and decryption, or compression and decompression. The processor operates as a functional unit that realizes a predetermined function by operating according to the program. An apparatus and a system including the processor are apparatuses and systems including these functional units.

[0015] The program may be installed from a program source into a device such as a computer. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server includes a processor (e.g., a CPU) and a storage resource, and the storage resource may further store a distribution program and the program to be distributed. Then, by the processor of the program distribution server executing the distribution program, the processor of the program distribution server may distribute the program to be distributed to other computers. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] FIG. 1 is a block diagram showing the overall configuration of a storage system 1 according to an embodiment.

[0017] The storage system 1 may be a Software Defined Storage (SDS) based on the cloud system 4. For example, multiple (or one) host devices 3, the cloud system 4, and the cloud control device 5 may be interconnected in a communicative manner via a network 2 consisting of Ethernet (registered trademark) or a Local Area Network (LAN).

[0018] The host device 3 is a higher-level device that sends read requests and write requests (hereinafter, these will be collectively referred to as I / O (Input / Output) requests) to the storage node 10 within the cloud system 4 in response to user operations or requests from implemented application programs, and is composed of a general-purpose computer device. The host device 3 may be a physical computer device or a virtual computer device such as a virtual machine. Furthermore, the host device 3 may be integrated into the cloud system 4.

[0019] The cloud system 4 is a system based on a cloud infrastructure (computer system) having multiple physical computers, and includes a computer provision service 12 that provides multiple storage nodes 10 and a block storage provision service 14 that provides multiple storage devices 13. Each storage node 10 can communicate with at least one of the multiple storage devices 13 (for example, each storage device 13). The multiple storage devices 13 may comprise one or more redundancy groups. A redundancy group consists of two or more storage devices 13, and data redundancy is achieved using technologies such as RAID (Redundant Array of Independent (or Inexpensive) Disks) or EC (Erasure Coding). The storage devices 13 may consist of one or more types of high-capacity non-volatile storage devices. The storage devices 13 may provide physical or logical storage areas for reading and writing data in response to I / O requests from the host device 3. In this embodiment, the storage devices 13 are cloud block storage in the cloud system 4, but the present invention can also be applied to storage systems including cloud systems that provide storage other than block storage to storage nodes. Non-volatile storage devices can be, for example, SAS SSDs, NVMe SSDs, SAS HDDs, or SATA HDDs. SAS stands for Serial Attached SCSI. SCSI stands for Small Computer System Interface. SSD stands for Solid State Drive. NVMe stands for Non-Volatile Memory Express. SATA stands for Serial ATA. ATA stands for Advanced Technology Attachment.

[0020] The storage node 10 is a virtual server device (virtual machine instance) that provides storage space for reading and writing data to the host device 3. In practice, each storage node 10 is allocated one or more storage devices 13. The storage node 10 then virtualizes the storage space provided by these allocated storage devices 13 and provides it to the host device 3.

[0021] As shown in Figure 2, the storage node 10 comprises a CPU (Central Processing Unit) 21, a HI / F (Host Communication Device) 22, and a BI / F (Block Storage Communication Device) 23, all interconnected via an internal network 20, and a memory 24 connected to the CPU 21. Each storage node 10 has one or more CPUs 21, HI / Fs 22, BI / Fs 23, and memory 24. Since the storage node 10 is a virtual server device, the CPU 21, HI / F 22, BI / F 23, and memory 24 are all virtual devices. These virtual devices may be based on the physical computer where the storage node 10 is located.

[0022] The CPU 21 is a processor that controls the operation of the entire storage node 10. The memory 24 consists of volatile semiconductor memory such as SRAM (Static RAM (Random Access Memory)) and DRAM (Dynamic RAM), and is used to temporarily hold various programs and necessary data. By executing programs stored in the memory 24 with at least one CPU 21, various processes for the entire storage node 10, as described later, are performed.

[0023] HI / F22 is an interface that allows the storage node 10 to communicate with the host device 3, other storage nodes 10, or cloud control device 5 via the network 2, and is composed of, for example, a NIC (Network Interface Card). HI / F22 performs protocol control during communication with the host device 3, other storage nodes 10, or cloud control device 5.

[0024] BI / F23 is an interface for storage node 10 to communicate with storage device 13, and is composed of components such as a NIC, similar to BI / F23. BI / F23 controls the protocol during communication with storage device 13.

[0025] Cloud system 4 has multiple Fault Domains 11. Hereafter, Fault Domain may be abbreviated as "FD". An "FD" is a set of hardware components (e.g., power supplies and switches) that share a single point of failure, that is, a unit of independent hardware component sets. An FD 11 is generally equivalent to a rack. If two or more storage nodes 10 are located in two or more different FD 11s, then even if one FD 11 fails due to a power failure or the like, not all of those two or more storage nodes 10 will stop simultaneously. An FD 11 can be, for example, one or more physical computers.

[0026] Cloud system 4 is configured with one or more Placement Groups 16. Hereinafter, Placement Group may be abbreviated as "PG". A "PG" is a group composed of multiple storage nodes 10. The boundaries (typically power boundaries or rack boundaries) that follow FD 11 in PG 16 are visible to the user. Therefore, the user can know which FD 11 each of their storage nodes 10 is located on. PG 16 includes multiple storage nodes 10. Some of the storage nodes 10 in PG 16 become elements of one or more clusters 15, and the remaining storage nodes 10 become spare storage nodes 10 that are not included in any cluster 15. If any of the storage nodes 10 in a cluster 15 stop (for example, if it stops due to a failure), a spare storage node 10 can take over the stopped storage node 10. PG 16 may be an example of a first storage node group, and cluster 15 may be an example of a second storage node group. "PG" may be the same as Placement Group in AWS®. As a first storage node group other than PG, an Azure® Virtual Machine Scale Set may be used. Which spare storage node 10 will become operational if any of the storage nodes 10 in cluster 15 fails will depend on at least one of (p) and (q) below. (p) In at least part of PG16, the correspondence between spare storage nodes 10 and the storage nodes 10 that make up the cluster 15 (in other words, which spare storage node 10 will take over if any of the storage nodes 10 fail) is predetermined. The correspondence is 1:1, many:1, 1:many, or many:many. According to this correspondence, if any of the storage nodes 10 in the cluster 15 fail, the spare storage node 10 corresponding to that storage node 10 will take over. (q) In at least part of PG16, the correspondence between the spare storage node 10 and the storage nodes 10 that make up the cluster 15 is not predetermined. If any of the storage nodes 10 in the cluster 15 fail, a spare storage node 10 selected arbitrarily (or according to a predetermined policy) will take over. This selection may be made by the cloud control device 5.

[0027] The cloud control unit 5 is a general-purpose computer device that has the function of allowing the system administrator to control the computing service 12 and block storage service 14 within the cloud system 4. The cloud control unit 5, in response to the system administrator's operations, adds, deletes, or modifies the storage nodes 10 and clusters 15 within the computing service 12, and the storage devices 13 within the block storage service 14, via the network 2. The cloud control unit 5 may be a physical computer device or a virtual computer device such as a virtual machine. Furthermore, the cloud control unit 5 may be integrated into the cloud system 4.

[0028] The multiple storage nodes 10 in the cloud system 4 may include only one storage node 10 for a single user, but typically include two or more storage nodes 10 for two or more users. For example, the multiple storage nodes 10 may include two or more storage nodes 10 for user A (e.g., company A) and two or more storage nodes 10 for user B (e.g., company B).

[0029] Figure 3 is a block diagram illustrating the software and configuration information stored in the memory 24 of the storage node 10.

[0030] Memory 24 stores software that, when executed by the CPU 21, realizes functions such as the cluster control unit 33, storage control unit 34, cluster construction unit 35, redundant configuration recovery unit 36, and state change unit 37. These functions 33 to 37 may be realized by a single software or by multiple independent and different software programs. Details of these functions 33 to 37 will be described later.

[0031] Memory 24 stores cluster configuration information 30 as configuration information. Cluster configuration information 30 is often a database, for example, and includes a storage node management table 31 and a cluster management table 32.

[0032] Figure 4 shows the configuration of the storage node management table 31.

[0033] The storage node management table 31 contains information about the storage nodes 10. The storage node management table 31 has a record for each storage node 10. Each record contains information such as the storage node ID 100, cluster ID 101, PG ID 102, FD ID 103, and status 104. Taking one storage node 10 as an example, the information 100-104 is as follows:

[0034] In other words, Storage Node ID 100 represents the ID of Storage Node 10. Cluster ID 101 represents the ID of Cluster 15, which includes Storage Node 10. PG ID 102 represents the ID of PG 16, which includes Storage Node 10. FD ID 103 represents the ID of FD 11, where Storage Node 10 is located. State 104 represents the state of Storage Node 10.

[0035] As shown in the example in Figure 4, PG "0x01" includes cluster "0x01" and cluster "0x02". Storage node "0x0004" is a spare storage node 10 that is included in PG "0x01" but not in either cluster "0x01" or "0x02" because cluster ID 101 is "Not Allocated".

[0036] As state 104, "Running" means operational. "Blocked" means stopped due to a failure. "Hibernated" means stopped. Note that the values ​​representing the state of a storage node (virtual machine instance) and their meanings may differ depending on the cloud vendor, but in this embodiment, "Hibernated" (stopped) is defined as a state in which the storage node 10 needs to be started in order to operate, but can be kept in a state with low maintenance costs. A storage node 10 in "Hibernated" state can be in a state (e.g., power-off state) that consumes less power than a state (e.g., sleep state) that maintains power consumption so that the storage node 10 can become operational in a relatively short time (e.g., without needing to be started). From another perspective, state 104 "Hibernated" of the storage node 10 may be defined as a state in which the storage node 10 is assigned PG ID 102 and FD ID 103, but is not assigned cluster ID 101.

[0037] Each record in the storage node management table 31 may also include additional information, such as the instance type of the storage node 10 and the type of storage device 13 assigned to the storage node 10.

[0038] Figure 5 shows the configuration of the cluster management table 32.

[0039] The cluster management table 32 contains information about cluster 15. The cluster management table 32 has a record for each cluster 15. Each record contains information such as cluster ID 200, PG ID 201, number of storage nodes 202, redundancy 203, and status 204. Taking one cluster 15 as an example, the information 200-204 is as follows:

[0040] Specifically, Cluster ID 200 represents the ID of Cluster 15. PG ID 201 represents the ID of PG16, which includes Cluster 15. Storage Node Count 202 represents the number of storage nodes 10 included in Cluster 15. Redundancy 203 is the redundancy of Cluster 15, specifically the maximum number of storage nodes in Cluster 15 that can withstand failure. Even if the number of storage nodes 10 equal to or less than the number represented by Redundancy Count 203 fails (stops) in Cluster 15, processing (business operations) in Cluster 15 can continue. State 104 represents the state of Cluster 15. Note that in each Cluster 15, PG ID 201 and Storage Node Count 202 are information that can be identified from the Storage Node Management Table 31 and are therefore optional. Each of one or more users may be notified of information corresponding to that user from the Cluster Management Table 32 (for example, the record of Cluster 15 including the storage nodes 10 assigned to that user). The recipient of the information notification may be the host device 3 or a management device not shown.

[0041] As shown in the example in Figure 5, cluster "0x01" is included in PG "0x01" and includes five storage nodes 10, and can continue processing even if one of the storage nodes 10 fails.

[0042] As status 204, "Normal" means normal. "Warning" means that a failure has occurred in storage nodes 10 less than or equal to the number represented by redundancy 203. "Stopped" means that a failure has occurred in storage nodes 10 more than the number represented by redundancy 203, and cluster 15 is stopped. "Failover in progress" means that failover is in progress. "Failback in progress" means that failback is in progress. "Caution" means that all storage nodes 10 in cluster 15 are normal, but there is some problem with cluster 15. "Some problem" could be, for example, a configuration where cluster 15 goes down due to a single FD failure due to failover or failback, or there could be no storage node 10 to fail back to.

[0043] Figure 6 is a block diagram illustrating the creation of a program.

[0044] An interface for receiving instructions to create a program (PG) is provided to the user (e.g., the host device 3 or management device) by the PG function of the cloud control device 5. Instructions are received from the user via this interface, and in response to these instructions, a PG 16 spanning multiple FD 11s is created.

[0045] For example, the FD ID of the FD11 to which each storage node 10 belongs is obtained and this process continues until a certain criterion is met. The "certain criterion" is that a predetermined number or more storage nodes 10 are secured in each FD11. For example, the process continues until two or more storage nodes 10 are secured in each FD11. Therefore, at least two storage nodes 10 are secured in each FD11.

[0046] Subsequently, a PG16 is created, consisting of a predetermined number or more storage nodes 10 allocated in each of the multiple FD11s, and spanning multiple FD11s. Storage nodes 10 that are not needed as components of the PG16 may be deleted, or they may remain as components of the PG16 without being deleted.

[0047] As shown in the example in Figure 6, there are 14 storage nodes A to N in the five FD11s, and the created PG consists of 10 storage nodes A to J, with the remaining storage nodes K to N being deleted as they are no longer needed.

[0048] According to the PG function of this embodiment, it is not possible to reserve a storage node 10 by specifying an FD ID (i.e., the PG function cannot accept the specification of the FD to which the storage node 10 will be reserved (placed)). However, as a modified example, the storage node 10 may be reserved by specifying an FD ID. Furthermore, according to the PG function of this embodiment, a storage node 10 reserved on FD 11 can know the FD ID of the FD 11 on which the storage node 10 resides. The FD ID may be a physical rack number (for example, a location within a data center) or a relative index number within PG 16.

[0049] Figure 7 is a block diagram illustrating the creation of a cluster.

[0050] Two or more storage nodes 10 in PG16 are selected as elements (members) of cluster 15, and cluster 15 (number of nodes "5", redundancy "1" 1) is constructed from the two or more selected storage nodes 10. During the construction of cluster 15, the cluster control unit 33 and storage control unit 34 are constructed, and the storage devices 13 are attached to the storage nodes 10, etc. The number of storage nodes required for cluster 15 is selected so that the number of storage nodes aggregated in FD11 is less than or equal to the redundancy of cluster 15, and cluster 15 consisting of the selected storage nodes is constructed. Each storage node 10 in PG16 that was not selected as an element of cluster 15 is set to the state "Hibernated", that is, stopped.

[0051] In the example shown in Figure 7, there are 5 FDs, 5 storage nodes are required for the cluster configuration, and the redundancy is "1". Therefore, one storage node 10 is selected from each FD11. In other words, the number of storage nodes 10 included in cluster 15 that are allowed to be aggregated (duplicated) into the same FD11 is less than or equal to the redundancy of "1". To put it another way, it is avoided that more than 1 storage nodes (at least two storage nodes in this example) are aggregated into a single FD among the storage nodes 10 included in cluster 15. As a result, five storage nodes A to E located in five different FD11s are selected, and cluster 15 is formed from these five selected storage nodes A to E. The state of the remaining storage nodes F to J is "Hibernated".

[0052] In each storage node 10 within cluster 15, the cluster control unit 33 and storage control unit 34 are as follows, for example. In the diagram, "SC" is an abbreviation for storage control unit, and in the following explanation, the storage control unit 34 may be abbreviated as "SC".

[0053] The cluster control unit 33 manages or operates the state of each storage node 10 within the cluster 15. Specifically, for example, the cluster control unit 33 starts up the storage control unit 34, detects failures, performs failover, and so on.

[0054] The storage control unit 34 functions as a storage controller within the storage node 10. For example, the storage control unit 34 performs I / O to the storage device 13 in accordance with I / O requests from the host device 3. The storage control unit 34 is redundant across M storage nodes 10 (M is an integer of 2 or more) in the cluster 15, and is configured in an Active-Standby configuration. Specifically, a redundancy group is configured across M storage nodes 10, where the state of one storage control unit 34 is "Active" and the state of each of the (M-1) storage control units 34 is "Standby". In the illustrated example, M=2. Hereinafter, a redundancy group composed of SC-n (n=A, B, ...) may be referred to as "redundancy group n". Note that the "redundancy level" of cluster 15 may be synonymous with the number of SCs (Standby) in each redundancy group within cluster 15. For example, if the redundancy group includes three SCs, specifically one SC (Active) and two SCs (Standby), the redundancy of cluster 15 will be "2".

[0055] Furthermore, multiple SC(Active) instances in multiple redundancy groups may be distributed across multiple storage nodes 10 that make up the cluster 15. In other words, multiple SC(Active) instances in multiple redundancy groups may not be concentrated on a single storage node 10. This distributes the load across multiple storage nodes 10 (multiple FD11).

[0056] For each storage node 10, access (I / O) to the storage device 13 attached to (allocated to) that storage node 10 is processed by the SC (Active) within that storage node 10. For example, access to the storage device 13 attached to storage node A is performed by SC-A (Active).

[0057] If storage node 10 or FD11 fails and SC(Active) stops, a failover will occur. Specifically, SC(Standby) will be promoted to SC(Active), processing will be taken over from the failing SC(Active) to SC(Standby), and SC(Standby) will be promoted to SC(Active).

[0058] For example, suppose a node failure occurs in storage node A, as illustrated in Figure 8. In this case, processing is taken over from the Active SC-A in storage node A to SC-A(Standby) in redundancy group A, that is, SC-A(Standby) in storage node B, and SC-A(Standby) in storage node B is promoted to SC-A(Active).

[0059] Failover allows processing to continue using the Active SC in the redundant group with a reduced number of SCs. In the example shown in Figure 8, the redundant groups with a reduced number of SCs are redundant groups A and E.

[0060] However, as shown in the example in Figure 8, the same number of storage nodes 10 as the redundancy level "1" of cluster 15 have stopped due to node failure. Therefore, if another node failure occurs in storage node 10, cluster 15 will be unable to continue processing.

[0061] Therefore, as illustrated in Figure 9, a redundancy configuration recovery process (rebuild and failback) is performed in the cluster (5 nodes, 1 redundant). This process is performed by at least the redundant configuration recovery unit 36 ​​of the cluster control unit 33 and redundant configuration recovery unit 36 ​​in the representative storage node 10, which will be described later.

[0062] Specifically, a representative storage node 10 within the cluster 15 (for example, one of the storage nodes B to E other than storage node A, chosen randomly or according to a predetermined rule) requests the cloud control unit 5 to start up storage node F, which is located on a different FD11 from the four FD11s where the existing storage nodes B to E reside (in this case, the same FD11 as storage node A) among the spare storage nodes F to J. In response to this request, the cloud control unit 5 starts up storage node F. For example, the state of storage node F changes from "Hibernated" to "Running".

[0063] The representative storage node 10 requests the cloud control unit 5 to detach the storage device 13 that was assigned to storage node A (remove the assignment of the storage device 13 to storage node A) and to attach the storage device 13 to storage node F. In response to this request, the cloud control unit 5 attaches the storage device 13 that was assigned to storage node A to storage node F instead.

[0064] The necessary information for integrating storage node F into cluster 15 is copied to storage node F, for example, from at least one of the other storage nodes B to E within cluster 15, or from the storage device 13 attached to storage node F. The cluster control unit 33 and storage control unit 34 of storage node F are started up, and the cluster configuration information 30 in each of storage nodes B to F is updated.

[0065] The data in the storage device 13 attached to storage node F is restored to its most recent state. For example, differential data, which is data updated between the time storage device 13 was detached from storage node A and attached to storage node F, is restored. All data may be restored based on general RAID technology, or updated areas (blocks that were updated) may be identified using differential bitmaps or the like on other storage nodes B to E, and only the updated areas may be restored.

[0066] After this, the operation of SC-A (Active) on storage node B is taken over by the corresponding SC-A (Standby) on storage node F, and the state of SC-A on storage node B changes from "Active" to "Standby".

[0067] The redundancy recovery process described above restores the redundancy level, which had decreased due to the node failure of storage node A, back to its original redundancy level of "1". Additionally, although SC(Active) was temporarily consolidated on storage node B, SC(Active) has now been redistributed.

[0068] If storage node A recovers after the redundancy recovery process, the state of storage node A will change from, for example, "Blocked" to "Hibernated," as shown in Figure 10. That is, storage node A will become one of the spare storage nodes 10 in PG16. Alternatively, a redundancy recovery process may be performed in which storage node A is incorporated into cluster 15 in place of storage node F, and storage node F may once again become a spare storage node 10 (a "Hibernated" storage node 10).

[0069] Furthermore, if storage node A recovers before storage node F is selected during the redundant configuration recovery process (for example, if the downtime of storage node A is temporary and it recovers quickly), storage node A may be selected as the recovery destination instead of storage node F.

[0070] Furthermore, if storage node F fails before storage node A is recovered (before the state of storage node A is changed to "Hibernated"), storage node G may be selected as the recovery target. In this case, as an element of a single cluster 15, there will be two storage nodes 10 within the second FD11 from the left, and the state 204 of that cluster 15 may be set to "Caution".

[0071] The following describes an example of the process performed in the embodiment.

[0072] Figure 11 is a flowchart showing the cluster construction process according to the embodiment.

[0073] The cluster construction process may be initiated when the administrator of memory system 1 (for example, a user) issues an instruction to the cloud control device 5.

[0074] The cloud control unit 5 receives instructions to create a program (PG) through an interface provided to the administrator (e.g., the host device 3 or management device), and in response to these instructions, causes the computer provision service 12 to create the PG (S1101). The administrator may also know the number of FDs 111 that form the basis of the PG. For example, the administrator may query the cloud control unit 5 for the number of FDs, and the cloud control unit 5 may obtain the number of FDs from the computer provision service 12 and return it to the administrator.

[0075] The computer provision service 12 creates (allocates) a storage node 10 on FD11 and adds the said storage node 10 to PG16 (S1102).

[0076] The created storage node 10 (for example, the cluster building unit 35) obtains the FD ID of the FD 11 on which the storage node 10 resides (for example, by querying the computer provision service 12 and obtaining it from the computer provision service 12), and registers the obtained FD ID in the record corresponding to the storage node 10 in the storage node management table 31 (S1103).

[0077] In creating a Programming Program (PG), the administrator may specify at least one of the following: the number of FD11s that the created PG16 will span, and the number of storage nodes 10 to be included in the PG16 (i.e., the desired number for each of the FD11s and storage nodes 10). However, the administrator does not need to specify which FD11 the desired number of storage nodes 10 will be placed in. The computer provision service 12 may determine which FD11 the storage nodes 10 will be created (allocated). For example, the computer provision service 12 may create storage nodes 10 in the FD11 so that the storage nodes 10 are evenly distributed across the FD11 that the PG16 spans. If the number of storage nodes 10 generated exceeds the number of FDs, the administrator will not know which storage node 10 will be generated in which FD11. For this reason, it is technically significant for the generated storage node 10 to obtain the FD ID of the FD11 in which it was generated and update the storage node management table 31. In the redundancy recovery process, based on the storage node management table 31, a storage node 10 in a different FD11 than the one where the remaining storage nodes 10 in cluster 15 reside is selected as the storage node 10 to operate in place of the stopped storage node 10. This prevents two or more storage nodes 10 in cluster 15 from being consolidated in the same FD11, thereby appropriately restoring redundancy.

[0078] The computer provisioning service 12 determines whether the availability policy is met (S1104). The availability policy may be associated with instructions for creating a PG (e.g., instructions from an administrator or instructions from the cloud control device 5) or it may be predetermined. The availability policy may include at least "the number of storage nodes necessary to constitute the cluster exists." The availability policy may also be a policy regarding the number of FD11 that form the basis of the PG16, and / or the number of storage nodes 10 that constitute the PG16, and may include at least one of the following: the number of storage nodes 10 that constitute the PG16, the number of storage nodes 10 that should be reserved for one FD11, and the required number of FDs and storage nodes. S1102 and S1103 are performed for each storage node 10 according to the availability policy. The availability policy may be predetermined instead of being associated with instructions.

[0079] If the result of S1104 is false (S1104: NO), the process returns to S1102. If the availability policy cannot be met even after the specified number of retries of S1102 to S1104, the computer service 12 may issue an alert and repeat the retries, notify the administrator that the availability policy cannot be met even after the specified number of retries, wait for a certain period of time until an appropriate storage node can be selected, or terminate abnormally at this point.

[0080] If the result of the determination in S1104 is true (S1104:YES), the computer provision service 12 or one of the storage nodes 10 required by PG16 (for example, the cluster construction unit 35) deletes the storage node 10 that is not needed by PG16 (S1105). S1105 is optional.

[0081] One of the storage nodes 10 in PG16 (e.g., the cluster construction unit 35) selects two or more storage nodes 10 in PG16 as elements of a cluster 15 and constructs a cluster 15 consisting of the two or more selected storage nodes 10 (S1106). Based on the configuration of the cluster 15, each storage node 10 in the cluster 15 (e.g., the cluster construction unit 35) updates the storage node management table 31 and the cluster management table 32. The number of storage nodes 10 constituting the cluster 15 may be included in the availability policy. The storage nodes 10 constituting the cluster 15 may be selected evenly from the FD11 that PG16 spans. Therefore, if the number of storage nodes 10 constituting the cluster 15 is less than or equal to the number of FD11 that PG16 spans, storage nodes 10 from different FD11 may be selected, and two or more storage nodes 10 from the same FD11 may not be selected.

[0082] One of the storage nodes 10 in cluster 15 (for example, the cluster construction unit 35 or the state change unit 37) sets the status of the storage nodes 10 among the PG16 that are not included in cluster 15 to "Hibernated" and updates the status 104 in the storage node management table 31 to "Hibernated" (S1107). To reduce costs, the OS disk capacity, etc., of the "Hibernated" storage nodes 10 may be reduced.

[0083] The cluster configuration information 30 for all storage nodes 10 in the created PG16 may be considered to be the same.

[0084] Figure 12 is a flowchart showing the redundant configuration recovery process (in the event of a storage node failure).

[0085] This process is performed by at least the redundant configuration recovery unit 36 ​​of the cluster control unit 33 and redundant configuration recovery unit 36 ​​of the representative storage node 10 in cluster 15. At this point, failover is complete, and therefore, the SC (Standby) on the failback destination storage node 10 is promoted to SC (Active) and is operating in place of the SC (Active) on the failed storage node 10 (the storage node 10 that is stopped due to node failure).

[0086] The representative storage node 10 obtains the FD ID of each storage node 10 in the PG16, including the cluster 15, from the storage node management table 31 (S1201). The "representative storage node 10" can be any storage node 10 in the cluster 15 that has not experienced a node failure. The representative storage node 10 may update the cluster 15's status 204 to "Failback in progress".

[0087] The representative storage node 10 refers to the storage node management table 31 and the cluster management table 32 and selects one of the storage nodes 10 that meets the following requirements (x) and (y) as the failback storage node 10 (S1202). The "failback storage node 10" is the storage node 10 that will operate in place of the failed storage node 10. (x) Cluster ID 101 is “Not Allocated” and state 104 is “Hibernated”. (y) The FD containing cluster 15 is located in the FD with the minimum number of storage nodes, 202, among the FDs that span PG16.

[0088] In principle, the failback destination storage node 10 should be selected from the FD11 to which the stopped storage node 10 belongs. If the failed storage node 10 has recovered at this point, this recovered storage node 10 may be selected as the failback destination storage node 10. However, in S1202, a check is performed to see if the storage nodes 10 are evenly distributed across the multiple FD11s that PG16 spans. If the result indicates that they are not evenly distributed, a different storage node 10 (i.e., a storage node 10 that contributes to the even distribution of storage nodes 10) may be selected as the failback destination storage node 10. Furthermore, as will be described later, if there is a spare FD11, there may be an FD11 that should not be selected as the failback destination storage node 10, and the failback destination storage node 10 may be selected from the FD11s excluding the FD11. If selecting a failback destination storage node 10 results in a deterioration of availability (for example, if a number of storage nodes 10 exceeding the redundancy limit are concentrated on a particular FD11, causing a cluster downtime due to a single FD failure), the representative cluster control unit 33 may raise an alert (for example, by updating the status 204 of cluster 15 to "Caution") and continue this redundancy recovery process, or it may notify the administrator of the deterioration in availability and seek the administrator's judgment, or it may wait for a certain period of time until an appropriate failback destination storage node 10 can be selected, or it may terminate abnormally at this point.

[0089] The representative storage node 10 starts up the failback target storage node 10 selected in S1202 via, for example, the cloud control device 5 (S1203). As a result, the state 104 of the failback target storage node 10 is updated from "Hibernated" to, for example, "Running". In S1203, the configuration of the failback target storage node 10 is updated as needed. For example, if the OS disk capacity had been reduced, the OS disk capacity may be increased. Also, the storage device 13 detached from the failed storage node 10 is attached to the failback target storage node 10.

[0090] If the startup of the failback target storage node 10 fails (S1204:NO), the process returns to S1202. In other words, a different storage node 10 is selected as the failback target storage node 10.

[0091] If the failback target storage node 10 starts up successfully (S1204: YES), the representative storage node 10 copies the configuration information (for example, cluster configuration information 30 including information for incorporating the failback target storage node 10 into cluster 15) from the failback target storage node 10 (and / or the failed storage node 10) to the failback target storage node 10 (S1205). The representative storage node 10 instructs the cluster control unit 33 and storage control unit 34 of the failback target storage node 10 to start up (S1206). The representative storage node 10 updates the storage node management table 31 and the cluster management table 32 (S1207). As a result, the failback target storage node 10 is incorporated into cluster 15, and the redundancy of cluster 15 is restored.

[0092] Figure 13 is a flowchart showing the state change process when a failed storage node is recovered.

[0093] This process modifies the state of the failed storage node 10 when it is restored.

[0094] The cloud control unit 5 or the representative storage node 10 detects the recovery of the failed storage node (S1301). For example, the cloud control unit 5 or the representative storage node 10 may receive a notification from the cloud system 4, or it may periodically check the status of the failed storage node 10 and confirm that it has returned to normal.

[0095] The cloud control unit 5 or the representative storage node 10 determines whether a failback to the recovered storage node is necessary (S1302). For example, the cloud control unit 5 or the representative storage node 10 determines whether there is a problem ("Caution") in which an inappropriate storage node was selected as the failback destination storage node 10 during the redundant configuration recovery process (for example, there are more storage nodes than the redundancy limit within the same FD11 as storage node 10 in cluster 15), and whether the problem can be rectified by failbacking to the recovered storage node (returning to the state before the failure).

[0096] If the result of the S1302 determination is true (S1302:YES), the representative storage node 10 (for example, the redundant configuration recovery unit 36 ​​and / or the state change unit 37) copies each configuration information from, for example, the original failback target storage node to the failback target storage node (recovery storage node) (S1303). The representative storage node 10 instructs the cluster control unit 33 and the storage control unit 34 of the failback target storage node 10 to start up (S1304). The representative storage node 10 updates the storage node management table 31 and the cluster management table 32 (S1305). As a result, the recovered storage node 10 is incorporated into the cluster 15 in place of the original failback target storage node 10.

[0097] The representative storage node 10 sets the state of the original failback target storage node 10 to "Hibernated" and updates the state 104 in the storage node management table 31 to "Hibernated" (S1306). To reduce costs, the OS disk capacity, etc., of the "Hibernated" storage node 10 may be reduced.

[0098] The above is an example of the processing performed in this embodiment. Note that the configuration of cluster 15 (e.g., number of nodes and redundancy), the configuration of multiple FD11s, the configuration of PG16, etc., are not limited to the above example. For example, at least one of the examples shown in Figures 14 to 17 may be adopted.

[0099] As shown in the example in Figure 14, a cluster 15 may have a node count of "6" and a redundancy level of "2". As shown in the example in Figure 14, the number of storage nodes required for cluster 15 is greater than the number of FDs that the cloud system 4 has (supports). In this case, at least one FD 11 will contain two or more storage nodes 10 within the same cluster 15. In the example in Figure 14, the number of FDs is "5" and the number of storage nodes 10 required for cluster 15 is "6", so two storage nodes A and F within the same cluster 15 will be contained in one FD 11.

[0100] To prevent a cluster from going down due to a single FD failure, the cluster redundancy must be "2" or higher. Therefore, it is conceivable to have three or more SCs that make up the redundant group. For example, the storage control unit 34 could be configured as Active-Standby-Standby instead of Active-Standby. A redundancy of "1" may be adopted, although this would reduce FD failure tolerance (decrease availability).

[0101] In the construction of cluster 15, at least two storage nodes 10 are assigned to each FD11. Therefore, in the example shown in Figure 14, during PG creation, steps S1102 to S1104 in Figure 11 are repeated until at least three storage nodes 10 are secured per FD11.

[0102] In cluster construction, for example, the required number of storage nodes 10 are selected so that the storage nodes 10 can be distributed across as many FD11 as possible. In the example shown in Figure 14, only storage nodes A and F are aggregated on the same FD. In addition, at least one storage node ("Hibernated") may be further reserved within the FD11 so that a failback storage node can be selected from the same FD11 in the event of simultaneous failure of storage nodes A and F. A cluster 15 is constructed, consisting of the storage nodes A to F selected for cluster construction. In PG16, the state of each of the storage nodes G to O other than storage nodes A to F is set to "Hibernated".

[0103] As shown in the example in Figure 15, some of the multiple FD11s (one or more FD11s) in the cloud system 4 are designated as spare FD11s. Storage nodes 10 secured from spare FD11s are not components of cluster 15. All storage nodes 10 in spare FD11s are spare storage nodes 10 ("Hibernated"). The number of storage nodes 10 secured in spare FD11s may be the number of storage nodes in the FD11 with the largest number of storage nodes among the other FD11s, or the number of cluster storage nodes in the FD11 with the largest number of storage nodes in cluster 15 among the other FD11s. In the event of a node failure in any of the storage nodes 10 in cluster 15, the spare storage node 10 selected as the failback destination storage node 10 is the spare storage node 10 in spare FD11. This makes it possible to fail back all storage nodes in the failed FD to the storage nodes in the spare FD when an FD failure occurs, thus improving stability during FD failures. As shown in the example in Figure 15, if a failure occurs in FD11 which has storage nodes A and E, a failback is performed to storage nodes K and N in the spare FD11.

[0104] In the example shown in Figure 15, the process of securing storage nodes from each FD during PG creation can be the same as in the example shown in Figure 14. In cluster construction, the required number of storage nodes 10 (e.g., storage A to F) are selected from the FDs other than the select few, so that the storage nodes 10 can be distributed across as many FDs as possible, excluding some of the FDs. The "select few FDs" from which no storage nodes 10 are selected as elements in cluster 15 are designated as spare FDs. In the event of a node failure, the failback storage node is selected from the FDs other than the spare FDs.

[0105] In the example shown in Figure 16, multiple clusters 15 are included within a single PG 16. The number of storage nodes 10 constituting each cluster 15 may be the same, but as illustrated in Figure 16, the number of storage nodes 10 constituting each cluster 15 may differ between clusters.

[0106] Furthermore, the spare storage nodes 10 within PG16 are assigned to any of the clusters 15, and may be dedicated spare storage nodes 10 for the assigned cluster, or they may be common to multiple clusters 15. That is, each of the spare storage nodes K to T may be selected as the failback destination storage node 10 if a node failure occurs in a storage node 10 in either the cluster 15 consisting of storage nodes A to E or the cluster 15 consisting of storage nodes F to I. Note that if each spare storage node 10 is shared by multiple clusters, the number of spare storage nodes 10 may be less than the number of storage nodes 10 that make up the multiple clusters.

[0107] In the example shown in Figure 17, multiple PG16s span the same group of FDs (multiple FD15s). When multiple PG16s are provided, a failback target storage node 10 is reserved for each PG. In other words, if a node failure occurs in any of the storage nodes 10 in the cluster 15, a spare storage node 10 will be selected from the PG16 including the cluster 15 to serve as the failback target storage node 10. However, if the target cluster 15 supports the following functions, a "Hibernated" storage node 10 may be reserved outside the PG16, and a spare storage node 10 that can be selected as the failback target storage node 10 may be shared between clusters (between PGs), similar to Figure 14. • The FD ID of storage node 10 outside of PG16 can be obtained. It is possible to integrate any storage node 10 outside of PG16 into PG16 while maintaining the FD ID of that storage node 10 (for example, without changing the FD15 of the storage node 10, or without changing the FD ID even if the FD15 of the storage node 10 is not changed).

[0108] Although one embodiment has been described above, this is merely an example for the purpose of explaining the present invention and is not intended to limit the scope of the present invention to this embodiment alone. The present invention can also be implemented in various other forms, such as forms in which some of the components of each of the above embodiments are deleted, forms in which at least some of the components are replaced, forms in which components are added, or forms that combine some or all of the embodiments.

[0109] The above explanation can be summarized as follows. This summary may include supplementary explanations and variations of the above explanation.

[0110] The cloud environment is equipped with multiple storage nodes 10 that constitute one or more PG16 (an example of one or more first storage node groups) spanning multiple FD11. Specifically, for each of the one or more PG16, the FD ID (domain ID) of the FD11 on which the storage node 10 was generated is obtained for each storage node 10, and a cluster 15 (an example of a second storage node group) is formed from the necessary number of storage nodes, with as few overlapping domain IDs as possible. Each storage node 10 included in one or more clusters 15 is a member storage node 10, and a storage node 10 not included in any of the one or more clusters 15 is a spare storage node 10. Each member storage node 10 performs I / O to the storage device 13 assigned to it and maintains cluster configuration information 30 (an example of configuration information) that includes the correspondence between the storage node 10 and the FD ID. Each spare storage node 10 is a storage node that can be selected based on cluster configuration information 30 to operate in place of a member storage node 10 that has stopped due to an FD failure or node failure in one or more clusters 15 or a predetermined cluster 15. For each of the one or more clusters 15, the number of member storage nodes 10 residing in the same FD 11 within that cluster 15 is less than or equal to the redundancy level. This redundancy level is the maximum number of member storage nodes in that cluster 15 that are allowed to stop simultaneously. This ensures that the availability of the storage system in the cloud environment is properly maintained.

[0111] Multiple storage nodes 10 constituting a PG16 spanning multiple FD11 may be provided by at least one of the one or more computers (for example, a cloud control device 5, a computer provisioning service 12, and one or more storage nodes 10). For each storage node 10 generated (allocated) in an FD11, the storage node 10 may obtain the FD ID of the FD11 to which it belongs, and add the relationship between the storage node 10 and the obtained FD ID to the cluster configuration information 30. One or more clusters 15 may be configured by at least one of the one or more computers based on the generated multiple storage nodes 10 and the FD IDs of those multiple storage nodes, and a storage node 10 not included in any of the clusters 15 may be designated as a spare storage node 10. For the construction of a cluster 15, more than the number of storage nodes 10 required for the construction of the cluster 15 may be reserved in advance by at least one of the one or more computers, and the required number of storage nodes 10 may be selected so that the number of duplicate FD IDs is less than or equal to the redundancy of the cluster 15, and the cluster 15 may be constructed from the selected required number of storage nodes 10.

[0112] The state of the spare storage node 10 may be in a hibernation state, which is a stopped state where the spare storage node 10 needs to be started for it to operate, but consumes little power. This allows the storage system to be maintained at a low cost (with little power consumption). From another perspective, the state of the spare storage node 10 may be such that, in the cluster configuration information 30, the spare storage node 10 is assigned the ID of the PG16 including the spare storage node 10 and the domain ID of the FD11 where the member storage node 10 is located, but is not assigned the ID of any cluster 15. Each spare storage node 10 may be put into a hibernation state by at least one of more computers, for example, through a predetermined function in a cloud environment (for example, a function provided by a cloud vendor).

[0113] If a storage node 10 in any cluster 15 fails, the representative storage node 10, which is one of the storage nodes 10 in the cluster 15 other than the failing storage node 10, may, based on the cluster configuration information 30, select one of the spare storage nodes 10 as the failback storage node 10, and replace the failing storage node 10 with the selected spare storage node 10 as a member storage node 10 of the cluster 15. Specifically, the representative storage node 10 may, based on the cluster configuration information 30, select a spare storage node 10 in an FD11 such that the number of member storage nodes 10 of the cluster 15 remains below the redundancy of the cluster 15 for any FD11, as the failback storage node 10. For example, the FD ID of the FD11 to which the failing storage node 10 belongs may be identified from the cluster configuration information 30, and one of the spare storage nodes 10 may be selected as the failback storage node 10 so as not to reduce FD fault tolerance (so that the redundancy of the cluster 15 is restored). This ensures availability is maintained.

[0114] One or more of the multiple FD11s may be one or more spare FD11s. If one or more storage nodes 10 are located on each of the one or more spare FD11s, then at least one of the computers may choose not to select any of the storage nodes 10 as elements of any cluster 15, and may treat them all as spare storage nodes 10. The representative storage node 10 may select a spare storage node 10 to be used as a failback destination storage node 10 from among the one or more spare FD11s. This makes it possible to fail back all storage nodes 10 in the failed FD11 to the storage nodes 10 in the spare FD11 in the event of an FD failure, thus improving stability during FD failures.

[0115] At least one PG16 may include two or more clusters 15 and one or more spare storage nodes 10 common to those two or more clusters 15. If a storage node 10 in any of the two or more clusters 15 fails, the representative storage node 10 may select one of the common spare storage nodes 10 as the failback storage node 10. This is expected to reduce the number of spare storage nodes 10 and thereby reduce resource consumption.

[0116] While spare storage nodes 10 may be allocated dynamically (for example, during failback), one or more spare storage nodes 10 may be allocated in advance for at least one of the PG16s before any storage node 10 in any of the clusters 15 fails. This ensures that spare storage nodes are reliably allocated from FD11 in a way that does not reduce FD fault tolerance (i.e., restores the redundancy of the cluster 15). In other words, it is expected that the possibility of not being able to allocate a failback storage node from FD11 because such FD11 is being used, for example, by another user's member storage node 10, is eliminated. [Explanation of Symbols]

[0117] 1...Memory system, 4...Cloud system, 5...Cloud control unit, 10...Storage node, 11...Fault Domain, 13...Storage device, 15...Cluster, 16...Placement Group

Claims

1. Having multiple storage nodes that constitute one or more first storage node groups spanning multiple failure domains in a cloud environment, For each of the one or more first storage node groups, For each storage node, the domain ID of the failure domain in which the storage node was created is obtained. A second storage node group is formed from the necessary number of storage nodes, with as few overlapping domain IDs as possible. Each storage node included in one or more second storage node groups is a member storage node, and a storage node not included in any of the said one or more second storage node groups is a spare storage node. Each member storage node is configured to perform I / O to the storage device assigned to it, and maintains configuration information including the correspondence between storage nodes and domain IDs. Each spare storage node is a storage node that can be selected based on the configuration information to operate in place of a member storage node that is shut down in any of the one or more second storage node groups or in a predetermined second storage node group, in order to take over the operation of the member storage node. For each of the one or more second storage node groups, the number of member storage nodes in the same failure domain within that second storage node group is less than or equal to the redundancy, where the redundancy is the maximum number of member storage nodes in that second storage node group that are allowed to fail simultaneously. Memory system.

2. The state of the spare storage node is a "hibernation" state, which is a stopped state where the spare storage node needs to be started for it to function, but consumes little power. The storage system according to claim 1.

3. The state of the spare storage node is such that, in the configuration information, the spare storage node is assigned the ID of the first storage node group that includes the spare storage node and the domain ID of the fault domain where the spare storage node is located, but is not assigned the ID of any second storage node group. The storage system according to claim 2.

4. If a storage node in any of the second storage node groups fails, the representative storage node, which is one of the storage nodes in that second storage node group other than the failing storage node, Based on the aforementioned configuration information, select one of the spare storage nodes as the failback storage node. The selected spare storage node will be made a member storage node of the second storage node group in place of the storage node that is to be shut down. The storage system according to claim 1.

5. Based on the configuration information, the representative storage node selects the spare storage node in a failure domain as the failback destination storage node, such that the number of member storage nodes of the second storage node group remains less than or equal to the redundancy of the second storage node group in any failure domain. The storage system according to claim 4.

6. One or more of the aforementioned multiple failure domains are one or more spare failure domains. If, for each of the one or more spare failure domains, one or more storage nodes are located in that spare failure domain, then each of those one or more storage nodes is a spare storage node and is not selected as an element of any of the second storage node groups. The representative storage node selects a spare storage node to be used as the failback destination storage node from one or more spare failure domains. The storage system according to claim 4.

7. At least one first storage node group includes two or more second storage node groups and one or more spare storage nodes common to the two or more second storage node groups, The representative storage node, if a storage node in any of the two or more second storage node groups fails, selects one or more of the common spare storage nodes as the failback storage node. The storage system according to claim 4.

8. For at least one of the one or more first storage node groups, the one or more spare storage nodes are reserved in advance before any storage nodes in any of the second storage node groups fail. The storage system according to claim 4.

9. In a cloud environment, multiple storage nodes are generated in multiple failure domains, and these multiple storage nodes constitute one or more first storage node groups in a storage system that span across the multiple failure domains. In generating these multiple storage nodes, the domain ID of the failure domain in which the storage node was generated is obtained for each storage node. A second storage node group is formed from the necessary number of storage nodes, ensuring that the domain IDs do not overlap as much as possible. Doing so, For each of the one or more first storage node groups, Each storage node included in one or more second storage node groups is a member storage node, and a storage node not included in any of the said one or more second storage node groups is a spare storage node. Each member storage node is configured to perform I / O to the storage device assigned to it, and maintains configuration information including the correspondence between storage nodes and domain IDs. Each spare storage node is a storage node that can be selected based on the configuration information to operate in place of a member storage node that is shut down in any of the one or more second storage node groups or in a predetermined second storage node group, in order to take over the operation of the member storage node. For each of the one or more second storage node groups, the number of member storage nodes in the same failure domain within that second storage node group is less than or equal to the redundancy, where the redundancy is the maximum number of member storage nodes in that second storage node group that are allowed to fail simultaneously. System construction method.

10. The state of each member storage node is set to a "hibernation" state, which is a stopped state where the member storage node needs to be started for it to operate, but consumes little power. The system construction method according to claim 9.

11. The state of each member storage node is such that, in the configuration information, the member storage node is assigned the ID of the first storage node group that includes the member storage node and the domain ID of the fault domain where the member storage node is located, but is not assigned the ID of any of the second storage node groups. The system construction method according to claim 10.

12. If a storage node in any of the second storage node groups fails, the representative storage node, which is one of the storage nodes in that second storage node group other than the failing storage node, Based on the aforementioned configuration information, select one of the spare storage nodes as the failback storage node. The selected spare storage node will be made a member storage node of the second storage node group in place of the storage node that is to be shut down. The system construction method according to claim 9.

13. Based on the configuration information, the representative storage node selects the spare storage node in a failure domain as the failback destination storage node, such that the number of member storage nodes of the second storage node group remains less than or equal to the redundancy of the second storage node group in any failure domain. The system construction method according to claim 12.

14. One or more of the aforementioned multiple failure domains are one or more spare failure domains. If, for each of the one or more spare failure domains, one or more storage nodes are located in that spare failure domain, none of those one or more storage nodes shall be selected as spare storage nodes, and none of them shall be selected as elements of any of the second storage node groups. The representative storage node selects a spare storage node to be used as the failback destination storage node from one or more spare failure domains. The system construction method according to claim 12.

15. For at least one first storage node group, one or more spare storage nodes common to two or more second storage node groups are provided for two or more second storage node groups. If a storage node in any of the two or more second storage node groups fails, the representative storage node selects one or more of the common spare storage nodes as the failback storage node. The system construction method according to claim 12.

Images