Storage system and failure coping method therein

Abstract

To provide a storage system and a storage coping method therein that ensure high reliability while maintaining memory capacity efficiency.SOLUTION: In a storage system having a plurality of storage nodes and a plurality of storage devices that provide storage areas to the storage nodes, a cluster control unit, upon detecting occurrence of a failure in another storage node, requests an external control device to perform detachment processing to detach the storage device assigned to a failure storage node from the failure storage node, and attachment processing to assign the storage device to an alternative storage node. An alternative cluster control unit then restores, on the basis of a log stored in the storage device that has been assigned to the storage node that has last failed among the storage nodes that belong to a same redundancy group as a failure storage control unit of the failure storage node, contents of a memory of the alternative storage node.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a storage system and a method for dealing with a failure in a storage system. [Background technology]

[0002] Conventionally, storage systems often have a redundant configuration in order to improve availability and reliability.

[0003] For example, Patent Document 1 proposes a storage system with the following redundant configuration. That is, the storage system has one or more storage devices that provide storage areas to each storage node of the storage system, and one or more storage control units on which storage control software runs to read and write data to the storage devices in response to requests from a host device. Each storage control unit holds storage configuration information required to read and write data to the corresponding storage device in response to requests from the host device.

[0004] In this storage system, multiple storage control software programs are managed as one redundancy group. The configuration information held by each of the multiple storage control software programs that make up the same redundancy group is updated synchronously, and the software programs are placed on different storage nodes to distribute the load on each storage node.

[0005] In the conventional technology disclosed in Patent Document 1, active storage control software belonging to one redundancy group and standby storage control software belonging to another redundancy group are placed on the same storage node, which makes it possible to efficiently use the resources of the storage node while increasing the availability of the storage control software.

[0006] Furthermore, Patent Document 1 discloses that information in memory required for controlling storage control software is made redundant between active storage control software and standby storage control software. Information in memory includes cache data. This makes it possible to build a storage system using SDS (Software Defined Storage) that can continue reading and writing even in the event of a storage node failure. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2019-101703 Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the above-mentioned conventional technology, if a storage node failure causes all storage control software in the same redundancy group to become inoperable, dirty write data processed by the storage control software but not yet stored in the storage device will be lost. If dirty data that has been written to the storage system (for which a write completion response has been sent) is lost, the reliability of the storage system will be significantly impaired.

[0009] Therefore, by increasing the number of control software programs belonging to the same redundancy group, the risk of all storage control software programs in the same redundancy group becoming inoperable at the same time can be reduced, and the reliability of the storage system can be improved. However, because each storage control software program needs to store information in memory, there is a problem in that the efficiency of the memory capacity of the storage system decreases.

[0010] The present invention has been made in consideration of the above circumstances, and has as its object to ensure high reliability in a storage system while maintaining memory capacity efficiency. [Means for solving the problem]

[0011] In order to solve the above-mentioned problems and achieve the object of the present invention, in a storage system having a plurality of storage nodes and a plurality of storage devices that provide storage areas to the plurality of storage nodes, the storage nodes each have a memory that stores cache data relating to data that the storage nodes read from and write to the storage areas, a storage control unit that executes reading and writing of the data from and to the storage areas in response to requests from a host device, updates the cache data relating to the data in the memory, creates a log relating to the cache data, and stores the log in the storage device assigned to the storage node, and a cluster control unit that manages the plurality of storage control units in redundancy groups, distributes and manages the plurality of storage control units belonging to the same redundancy group across the plurality of storage nodes, and monitors the occurrence of failures in the other storage nodes, wherein the storage control units belonging to the same redundancy group synchronize the cache data stored in the memory, and the cache data of the other storage nodes is updated. the failure-detecting cluster control unit that is the cluster control unit that detects the occurrence of a failure requests an external control device to create an alternative storage node that is the storage node that substitutes for the failed storage node, the failure-detecting cluster control unit executes a detach process to separate the storage device assigned to the failed storage node from the failed storage node, and then requests the control device to execute an attach process to assign the storage device to the alternative storage node; the alternative cluster control unit that is the cluster control unit of the alternative storage node selects a specific storage node that last failed among the storage nodes having the storage control unit that belongs to the same redundancy group as the failed storage control unit of the failed storage node, and selects and executes a first recovery method for restoring storage contents in the memory of the alternative storage node based on the log stored in the storage device that was assigned to the specific storage node; and in the alternative storage node,The method is characterized in that an alternative storage control unit, which is the storage control unit that substitutes for the failed storage control unit, is started. [Effects of the Invention]

[0012] According to the present invention, in a storage system, it is possible to ensure high reliability while maintaining memory capacity efficiency. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating the overall configuration of a storage system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a schematic configuration of a storage node according to the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating the configuration of a memory according to the first embodiment. [Figure 4] FIG. 3 is a diagram for explaining a storage node management table according to the first embodiment. [Figure 5] FIG. 4 is a diagram illustrating a storage control unit management table according to the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating the configuration of the storage system according to the first embodiment in a normal state. [Figure 7] FIG. 2 is a diagram illustrating an outline of a system recovery process according to the first embodiment. [Figure 8] 10 is a flowchart of a control information update process according to the first embodiment. [Figure 9] 10 is a flowchart of a cache data update process according to the first embodiment. [Figure 10] 10 is a flowchart of a log creation process according to the first embodiment. [Figure 11] 10 is a flowchart of a log saving process according to the first embodiment. [Figure 12] 10 is a flowchart of a base image saving process according to the first embodiment. [Figure 13] 10 is a flowchart of a log recovery process according to the first embodiment. [Figure 14] 10 is a flowchart of a system recovery process according to the first embodiment. [Figure 15]10 is a flowchart of a valid log selection process according to the first embodiment. [Figure 16] FIG. 10 is a diagram illustrating the overall configuration of a storage system according to a second embodiment. [Figure 17] FIG. 10 is a diagram illustrating a schematic configuration of a storage node according to a second embodiment. [Figure 18] 11 is a flowchart of a storage node maintenance suspension process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the accompanying drawings, functionally identical elements may be designated by the same reference numerals. The accompanying drawings show specific embodiments and examples in accordance with the principles of the present invention. These embodiments and examples are intended to help understand the present invention and should not be used to interpret the present invention in a limiting manner.

[0015] In addition, in the following description, when describing elements of the same type without distinguishing between them, common reference symbols will be used, and when distinguishing between elements of the same type, reference symbols (or element IDs (e.g., identification numbers)) will be used. [Example]

[0016] <Configuration of storage system 1 according to first embodiment> 1 is a diagram illustrating the overall configuration of a storage system 1 according to Example 1. The storage system 1 includes a plurality of host devices 3 interconnected via a network 2 configured from, for example, Ethernet (registered trademark) or a LAN (Local Area Network), a cloud system 4, and a cloud control device 5.

[0017] The host device 3 is a higher-level device that sends read requests and write requests (hereinafter referred to as "I / O (Input / Output) requests") to the storage nodes 10 in the cloud system 4 in response to user operations or requests from application programs. The host device 3 is configured using a general-purpose computer device. The host device 3 may be a physical computer device or a virtual computer device such as a virtual machine. Furthermore, the host device 3 may be incorporated into the cloud system 4.

[0018] The cloud system 4 includes a computer providing service unit 11 having a plurality of storage nodes 10, and a block storage providing service unit 13 having a plurality of storage devices 12 (storage media). In the cloud system 4, the storage nodes 10 constituting the computer providing service unit 11 and the storage devices 12 constituting the block storage providing service unit 13 are interconnected via a block storage providing service network 14.

[0019] The storage node 10 is a physical or virtual server device that provides a storage area for reading and writing data to the host device 3. One or more storage devices 12 in the block storage provision service unit 13 are assigned to each storage node 10. The storage node 10 virtualizes the storage area provided by the assigned storage device 12 and provides it to the host device 3.

[0020] Fig. 2 is a diagram illustrating a schematic configuration of a storage node 10 according to the first embodiment. As shown in Fig. 2, the storage node 10 includes a CPU (Central Processing Unit) 21, a host-oriented communication device 22, and a block storage providing service-oriented communication device 23, which are interconnected via an internal network 20, and a memory 24 connected to the CPU 21. Each storage node 10 includes one or more of the CPU 21, the host-oriented communication device 22, the block storage providing service-oriented communication device 23, and the memory 24.

[0021] When the storage node 10 is a physical server device, the CPU 21, the host-oriented communication device 22, the block storage providing service-oriented communication device 23, and the memory 24 are each configured as physical devices. When the storage node 10 is a virtual server device, the CPU 21, the host-oriented communication device 22, the block storage providing service-oriented communication device 23, and the memory 24 are each configured as virtual devices.

[0022] The host-oriented communication device 22 is an interface through which the storage node 10 communicates with the host device 3, the cloud control device 5, or other storage nodes 10 via the network 2. The host-oriented communication device 22 is configured using, for example, a network interface card (NIC). The host-oriented communication device 22 also controls protocols used when communicating with the host device 3, other storage nodes 10, or the cloud control device 5.

[0023] The block storage provision service-oriented communication device 23 is an interface that enables the storage node 10 to communicate with the storage device 12 in the block storage provision service unit 13 via the block storage provision service-oriented network 14. Specifically, the block storage provision service-oriented communication device 23 is configured using an NIC or the like, similar to the host-oriented communication device 22. The block storage provision service-oriented communication device 23 controls the protocol when communicating with the storage device 12.

[0024] The host-oriented communication device 22 and the block storage providing service-oriented communication device 23 may be physically different communication devices, or may be physically the same but logically separated communication devices. Furthermore, the host-oriented communication device 22 and the block storage providing service-oriented communication device 23 may be physically or logically the same communication device.

[0025] Returning to the explanation of Figure 1, the storage device 12 is configured using one or more types of large-capacity nonvolatile storage devices, such as a SAS (Serial Attached SCSI (Small Computer System Interface)) SSD (Solid State Drive), an NVMe (Non Volatile Memory express) SSD, a SAS hard disk drive, or a SATA (Serial ATA (Advanced Technology Attachment)) hard disk drive. The storage device 12 provides a physical or logical storage area for reading and writing data in response to an I / O request from the host device 3.

[0026] The cloud control device 5 is a general-purpose computer device that allows a system administrator to control the computer providing service unit 11 and the block storage providing service unit 13 in the cloud system 4. In response to operations by the system administrator, the cloud control device 5 adds, deletes, or changes the configuration of the storage nodes 10 and clusters 15 in the computer providing service unit 11 and the storage devices 12 in the block storage providing service unit 13 via the network 2. The cloud control device 5 may be a physical computer device or a virtual computer device such as a virtual machine. Furthermore, the cloud control device 5 may be incorporated into the cloud system 4.

[0027] <Configuration diagram of memory 24 according to the first embodiment> 3 is a diagram illustrating the configuration of the memory 24 according to the embodiment 1. The memory 24 of each storage node 10 stores a cluster control unit 30, a storage control unit 31, storage control information 40, a log header management table 41, a cache data area 42, a control information log buffer 43, a cache data log buffer 44, a storage node management table 45, and a storage control unit management table 46.

[0028] The cluster control unit 30 and the storage control unit 31 are processing function units that are realized by the CPU 21 executing a predetermined program.

[0029] The cluster control unit 30 manages and operates the status of each storage node 10 constituting the cluster 15 and each storage device 12 assigned to these storage nodes 10. The cluster control unit 30 manages multiple storage control units 31 in redundancy groups, and manages multiple storage control units 31 belonging to the same redundancy group by distributing them across multiple storage nodes 10.

[0030] The cluster control unit 30 also manages and controls the startup of the storage control unit 31, which will be described later, and the operation mode (active mode or standby mode, which will be described later) of the storage control unit 31. The cluster control unit 30 may also be called cluster control software or a cluster control program.

[0031] Furthermore, each cluster control unit 30 monitors the status of the other cluster control units 30, and if a failure occurs in another storage node 10 due to some reason such as a power outage or network disruption, the cluster control units 30 have the function of detecting this and notifying the other cluster control units 30. Specific examples of a "storage node failure" include a failure in a system disk required for the operation of the OS (Operating System) implemented in the storage node 10, or a failure in the device itself that constitutes the storage node 10.

[0032] Furthermore, the cluster control unit 30 transfers an I / O request received from the host device 3 to the cluster control unit 30 of the corresponding other storage node 10 via the network for the block storage provision service. The cluster control unit 30 also passes an I / O request transferred from the cluster control unit 30 of the other storage node 10 to the storage control unit 31 of the corresponding redundancy group 50 (FIG. 6). The redundancy group 50 will be described later with reference to FIG. 6.

[0033] The storage control unit 31 controls the storage system. For example, the storage control unit 31 processes I / O requests received from the host device 3. In response to requests from the host device 3, the storage control unit 31 reads and writes data from and to storage areas provided by the multiple storage devices 12. The storage control unit 31 may also be called storage control software or a storage control program.

[0034] The storage control information 40 is an area where the storage control unit 31 stores control information for implementing various storage functions. The storage control information 40 includes cache control information such as the correspondence between the address of cache data and the logical address in the volume where the cached data is stored, and the state of the cache data (dirty data and clean data).

[0035] The log header management table 41 is a table that stores the log headers of the log 60 of all cache data on the storage device 12. The log 60 (FIG. 6) is composed of the update data itself and a log header, and is information that indicates how the control information and cache data on the memory 24 have been updated. The log header contains information that indicates the location and size of the data update, and the order of the updates.

[0036] The cache data area 42 is an area for storing cache data.

[0037] The control information log buffer 43 temporarily stores a log of storage control information.

[0038] The cache data log buffer 44 temporarily stores a log of the cache data.

[0039] The storage node management table 45 is data for managing the state of each storage node 10 that constitutes the cluster 15. The storage node management table 45 will be described later with reference to FIG.

[0040] The storage control unit management table 46 is data for managing the state of the storage control unit 31 belonging to each storage node 10 that constitutes the cluster 15. The storage control unit management table 46 will be described later with reference to FIG.

[0041] <Storage node management table 45 according to the first embodiment> 4 is a diagram illustrating a storage node management table 45 according to the first embodiment. The CPU 21 creates and manages the storage node management table 45. Each entry in the storage node management table 45 manages the status of each storage node 10 belonging to the cluster 15.

[0042] Each column of the storage node management table 45 will be explained below. Column 451 records an ID indicating a storage node 10. Column 452 records the status of the corresponding storage node 10. For example, the status value of a storage node 10 is set to "normal" if the storage node 10 is operating normally, and "abnormal" if a fault has occurred in the storage node 10.

[0043] Column 453 records the date and time of failure when a failure occurs in the corresponding storage node 10. When failures occur in multiple storage nodes 10, the order in which the storage node failures occurred can be determined from the date and time of the failure occurrence. Note that if no failure has occurred in the storage node 10, an invalid value such as "-" may be recorded.

[0044] <Storage control unit management table 46 according to the first embodiment> 5 is a diagram illustrating the storage control unit management table 46 according to the first embodiment. The cluster control unit 30 (FIG. 3) creates the storage control unit management table 46 and manages the redundancy groups of the storage control units 31 based on the storage control unit management table 46. Each entry in the storage control unit management table 46 manages the status of the storage control unit 31 operating in each storage node 10 belonging to the cluster 15.

[0045] Each column of the storage control unit management table 46 will be explained below. Column 461 records an ID indicating the storage control unit 31. Column 462 records the status of the corresponding storage control unit 31. The status value of the storage control unit 31 is set to "normal (active)" or simply "active" when the storage control unit 31 is operating normally and is in a state where it accepts I / O requests from the host device 3 (active system state, hereinafter referred to as "active mode"). Also, when the storage control unit 31 is operating normally and is in a state where it is not accepting I / O requests from the host device 3 (standby system state, hereinafter referred to as "standby mode"), the status is set to "normal (standby)" or simply "standby". On the other hand, when the storage control unit 31 is not operating, the status is set to "abnormal".

[0046] Column 463 records an ID indicating the redundancy group 50 to which the corresponding storage control unit 31 belongs. Column 464 records an ID indicating the storage node 10 on which the corresponding storage control unit 31 operates.

[0047] <Configuration of the storage system 1 according to the first embodiment in normal operation> Fig. 6 is a diagram illustrating the normal configuration of the storage system 1. In this embodiment, as shown in Fig. 6, the storage control unit 31 implemented in each storage node 10 configures a group for redundancy (hereinafter referred to as a "redundancy group 50") together with one or more storage control units 31 implemented in different storage nodes 10.

[0048] 6 shows an example in which the redundancy group 50 is configured by two storage control units 31. In the following explanation, it is also assumed that the redundancy group 50 is configured by two storage control units 31. The number of storage control units 31 that configure the redundancy group 50 is not limited to two, and may be three or more.

[0049] In the redundancy group 50, at least one storage control unit 31 is set to active mode. The remaining storage control units 31 that are not set to active mode are set to standby mode.

[0050] Therefore, in a redundancy group 50 consisting of two storage control units 31, either a configuration in which these two storage control units 31 are both set to active mode (hereinafter referred to as an "active-active configuration") or a configuration in which one storage control unit 31 is set to active mode and the other storage control unit 31 is set to standby mode (hereinafter referred to as an "active-standby configuration") is adopted.

[0051] In a redundancy group 50 employing an active-standby configuration, if a failure occurs in a storage control unit 31 set to active mode or in a storage node 10 on which the storage control unit 31 is operating, or if such a storage node 10 is removed, the storage control unit 31 that had been set to standby mode can be changed to active mode. As a result, the storage control unit 31 that had been set to standby mode takes over processing of I / O requests from the host device 3. This function is called a "failover function."

[0052] To achieve this failover function, storage control units 31 belonging to the same redundancy group 50 always hold the same storage control information 40. To achieve this, when the storage control information 40 is updated, a storage control unit 31 set to active mode transmits the difference between the storage control information before and after the update as differential data to the other storage control unit 31 belonging to the same redundancy group 50 as itself. Based on this differential data, the other storage control unit 31 updates the storage control information 40 held by that storage control unit 31. As a result, the storage control information 40 held by each storage control unit 31 constituting the same redundancy group 50 is always kept synchronized.

[0053] In this way, the two storage control units 31 that make up the redundancy group 50 always hold the same storage control information 40. As a result, even if a failure occurs in a storage control unit 31 set to active mode or in a storage node 10 on which the storage control unit 31 is operating, or if such a storage node is removed, another storage control unit 31 that belongs to the same redundancy group 50 as the storage control unit 31 can immediately take over the processing that had been executed by that storage control unit 31 up to that point.

[0054] As a means for realizing such processing, the cluster control unit 30 of each storage node 10 shares information such as which storage node 10 each storage control unit 31 in the cluster 15 is located on, which redundancy group 50 it belongs to, and whether its operating mode is set to active mode or standby mode. This information is managed by a storage node management table 45 and a storage control unit management table 46.

[0055] In the example shown in FIG. 6, the storage control unit 31 (storage control unit A) of the storage node 10 (storage node A) and the storage control unit 31 (storage control unit A) of the storage node 10 (storage node B) belong to the same redundancy group 50.

[0056] The storage control unit 31 (storage control unit A) of the storage node 10 (storage node A) holds storage control information 40 (storage control information A) in the storage node 10 (storage node A), and the storage control unit 31 (storage control unit A) of the storage node 10 (storage node A) writes out a log 60 (log A) to the storage device 12 (storage device A).

[0057] Similarly, the storage control unit 31 (storage control unit A) of the storage node 10 (storage node B) stores storage control information 40 (storage control information A) in the storage node 10 (storage node B) and writes out a log 60 (log A) to the storage device 12 (storage device B).

[0058] Furthermore, the storage control unit 31 (storage control unit C) of the storage node 10 (storage node A) and the storage control unit 31 of a storage node 10 (not shown) belong to the same redundancy group 50. The storage control unit 31 (storage control unit C) of the storage node 10 (storage node A) holds storage control information 40 (storage control information C) in the storage node 10 (storage node A) and writes a log 60 (log C) to the storage device 12 (storage device A).

[0059] Similarly, the storage control unit 31 (storage control unit B) of the storage node 10 (storage node B) and the storage control unit 31 of the storage node 10 (not shown) belong to the same redundancy group 50. The storage control unit 31 (storage control unit B) of the storage node 10 (storage node B) holds storage control information 40 (storage control information B) in the storage node 10 (storage node B) and writes a log 60 (log B) to the storage device 12 (storage device B).

[0060] <Outline of system recovery process according to the first embodiment> Next, a series of processing flows will be described when failures occur in multiple storage nodes 10, causing all storage control units 31 belonging to the same redundancy group 50 to stop. Fig. 7 is a diagram illustrating an overview of the system recovery processing according to the first embodiment.

[0061] Referring to Figure 7, we will explain the flow (hereinafter referred to as "system recovery processing") when a failure occurs in "storage node A" at time = t0, and then a failure occurs in "storage node B" at time = t1 (t1 ≧ t0).

[0062] First, the cluster control unit 30 of one of the storage nodes 10 connected to the cluster control unit 30 of "storage node A" detects that a failure has occurred in "storage node A." Then, the cluster control unit 30 that detected the failure sends a notification to the other storage nodes 10 that it has detected the failure in "storage node A." Then, the cluster control unit 30 of the other storage nodes 10 that have recognized the failure in "storage node A" switches the operation mode of the storage control unit 31 in their own storage nodes that belong to the same redundancy group 50 as the storage control unit 31 in "storage node A" from standby mode to active mode. In the example of FIG. 7, the cluster control unit 30 of "storage node B" changes the state of "storage control unit A (standby)" to active mode (step S701). As a result, the I / O processing that had been performed by "storage control unit A (active)" is taken over by "storage control unit A (standby)" (failover).

[0063] Furthermore, the cluster control unit 30 of another storage node 10 that recognizes the failure of "storage node A" instructs the "storage control unit A (standby)" or "storage control unit C (active)" in its own storage node, which belongs to the same redundancy group 50 as the "storage control unit A (active)" or "storage control unit C (standby)" in the "storage node A" where the failure occurred, not to transfer the differential data to the "storage control unit A (active)" and "storage control unit C (standby)" that belong to the same redundancy group 50, even if the storage control information 40 held by that storage control unit 31 is updated after time t1.

[0064] Next, the cluster control unit 30 of one of the storage nodes 10 connected to the cluster control unit 30 of "storage node B" detects that a failure has occurred in "storage node B." The cluster control unit 30 that detected the failure then sends a notification to the other storage nodes 10 that it has detected the failure in "storage node B." The cluster control unit 30 of the other storage nodes 10 that have recognized the failure in "storage node B" then switches the operation mode of the storage control unit 31 in its own storage node that belongs to the same redundancy group 50B as the storage control unit 31 in "storage node B" from standby mode to active mode. At this time, the cluster control unit 30 detects that all other storage control units 31 that belong to the same redundancy group 50A as "storage control unit A" have stopped due to the failure. Then, the cluster control unit 30 of another storage node 10 (hereinafter referred to as the "failure handling cluster control unit") that recognizes the failure of "storage node B" sends a request to the cloud control device 5 to create an alternative storage node in order to create a storage node 10 (hereinafter referred to as the "alternate storage node") that will replace the "storage node A" and "storage node B" that are in a failed state.

[0065] Upon receiving the request to create alternative storage nodes, the cloud control device 5 creates new storage nodes 10 ("storage node X" and "storage node Y" in FIG. 7) in the cluster 15 (steps S702 and S703).

[0066] Specifically, if the storage node 10 is a physical storage device, the cloud control device 5 incorporates the already existing physical storage device into the cluster 15, and if the storage node 10 is a virtual storage device, the cloud control device 5 generates a new virtual storage device and incorporates it into the cluster 15.

[0067] The failure handling cluster control unit also requests the cloud control device 5 to detach all storage devices 12 that have been assigned to the failed "storage node A" and "storage node B." Upon receiving the request, the cloud control device 5 detaches all storage devices 12 that had been assigned to "storage node A" and "storage node B" up until that point (step S704).

[0068] Furthermore, the failure response cluster control unit requests the cloud control unit 5 to attach all of the storage devices 12 detached from "storage node A" to the newly created "storage node X" in step S704. Similarly, the failure response cluster control unit requests the cloud control unit 5 to attach all of the storage devices 12 detached from "storage node B" to the newly created "storage node Y".

[0069] Upon receiving this request, the cloud control device 5 attaches all storage devices 12 that had been allocated to "storage node A" and "storage node B" to "storage node X" and "storage node Y" respectively (step S705).

[0070] On the other hand, when "storage node X" and "storage node Y" are added to the cluster 15 as described above, the cluster control units 30 in "storage node X" and "storage node Y" are started by the cloud control device 5 (steps S706, S707).

[0071] The cluster control units 30 of the started-up "storage node X" and "storage node Y" then start sharing the configuration information (storage node management table 45 and storage control unit management table 46) of each storage node 10, including its own storage node, in the cluster 15 with the cluster control units 30 of the other storage nodes 10.

[0072] As a result, the cluster control units 30 of "storage node X" and "storage node Y" (hereinafter referred to as "alternate cluster control units") perform a process to restore the memory contents of storage control units A of "storage node X" and "storage node Y" from the log ("memory data recovery from log" described below) since failures have occurred in all storage control units 31 that make up the redundancy group 50A to which storage control unit A belongs (i.e., memory contents have been lost) (steps S708, S709).

[0073] Specifically, the alternative cluster control unit refers to the storage node management table 45 and the storage control unit management table 46 in its own storage node, and identifies the storage nodes 10 to which each storage control unit 31 that constitutes the redundancy group 50A belongs ("storage node A" and "storage node B" in the example of Figure 7).

[0074] The alternative cluster control unit then identifies, among these storage nodes 10, the storage node 10 with the latest failure occurrence time as the storage node 10 linked to the log 60 reflecting the latest memory update (hereinafter referred to as the "valid log") ("storage node B" in the example of FIG. 7). Thereafter, the alternative cluster control unit restores the memory contents using the log of the storage device 12 ("storage device B" in the example of FIG. 7) that was linked to the storage node B thus identified (log recovery processing). Details of the log recovery processing will be described later with reference to FIG. 13.

[0075] Furthermore, since one of the storage controllers 31 constituting the redundancy group 50B to which storage controller B belongs is operating normally, the cluster controller 30 (alternate cluster controller) of "storage node Y" performs a process of restoring the memory contents of storage controller B from the redundancy group 50B of storage controller B ("memory data recovery from redundant system" to be described later) (not shown in the example of FIG. 7). Similarly, the storage controller C of "storage node X" performs a process of restoring the memory contents of storage controller C from the redundancy group 50C to which storage controller C belongs ("memory data recovery from redundant system" to be described later).

[0076] In this way, the lost memory contents can be restored to the memories 24 of "storage node X" and "storage node Y." After that, the cluster control units 30 of "storage node X" and "storage node Y" start up the storage control units 31 in their own storage nodes. Specifically, for example, "storage node X" starts up "storage control unit A" in active mode and starts up "storage control unit C" in standby mode. Furthermore, "storage node Y" starts up "storage control unit A" in standby mode and starts up "storage control unit B" in active mode.

[0077] Thereafter, the alternative cluster control unit updates the status of each storage node 10 in the storage node management table 45 and updates the status of each storage control unit 31 in the storage control unit management table 46 .

[0078] Through the above processing, the newly created “storage node X” and “storage node Y” can be used as part of the cluster 15.

[0079] <Control information update process according to the first embodiment> 8 is a flowchart of the control information update process according to the embodiment 1. The control information update process is executed when a request to update the storage control information 40 is made.

[0080] First, in step S801, the storage control unit 31 updates the storage control information 40 in the memory 24. Next, in step S802, the storage control unit 31 performs a log creation process. Details of the log creation process will be described later with reference to FIG.

[0081] <Cache data update process according to the first embodiment> 9 is a flowchart of a cache data update process according to the embodiment 1. The cache data update process is executed when data to be stored in the cache data area 42 occurs or when data stored in the cache data area 42 is updated.

[0082] First, in step S901, the storage control unit 31 updates the cache data in the memory. Specifically, for example, the data received from the host device 3 is written to the cache data area 42 already allocated in the memory 24.

[0083] Next, in step S902, the storage control unit 31 performs a log creation process. The log creation process is a process for creating a log 60 related to the updated cache data, and will be described later with reference to FIG.

[0084] Next, in step S903, the storage control unit 31 determines whether the cache data update in step S901 is an overwrite. That is, the storage control unit 31 checks whether a log 60 related to the cache data update of the address range included in the range of the currently updated cache data area 42 (hereinafter referred to as the "same address log") exists in the existing log 60, and if so, determines that it is an overwrite. If it is an overwrite (step S903 Yes), the storage control unit 31 proceeds to step S904, and if it is a new write (step S903 No), proceeds to step S905.

[0085] In step S904, the storage control unit 31 invalidates the address log written in the log header management table 41. In step S905, the storage control unit 31 updates the log header management table 41.

[0086] <Log Creation Process According to First Embodiment> FIG. 10 is a flowchart of the log creation process according to the first embodiment.

[0087] First, in step S1001, the storage control unit 31 allocates a new sequence number. The sequence number indicates the order in which the logs 60 were created, and is incremented by one each time a new log 60 is created.

[0088] Next, in step S1002, the storage control unit 31 secures a log buffer for temporarily storing the log 60. Specifically, if the data to be stored in the log 60 is control information, the storage control unit 31 allocates an area of a size necessary to store the log 60 to be created from the control information log buffer 43, or from the cache data log buffer 44 if the data to be stored in the log 60 is cache data.

[0089] Next, in step S1003, the storage control unit 31 creates a log header, which includes a sequence number, the address of the target data in the memory 24, the size of the target data, and so on.

[0090] Next, in step S1004, the storage control unit 31 stores the log data in the log buffers of the control information log buffer 43 and the cache data log buffer 44. Next, in step S1005, the storage control unit 31 performs a process to validate the created log 60. Specifically, for example, the log header includes a flag indicating whether the log 60 is valid or invalid, and the log 60 is validated by turning this flag ON.

[0091] <Log save process according to the first embodiment> 11 is a flowchart of the log evacuation process according to the embodiment 1. The log evacuation process is a process for writing the logs 60 accumulated in the log buffers of the control information log buffer 43 and the cache data log buffer 44 to the storage device 12, and is executed when it is necessary to write the logs 60 to the storage device 12, for example, before a host response is made in processing a write request from the host device 3.

[0092] First, in step S1101, the storage control unit 31 retrieves the unsaved logs, that is, the logs 60 that have not yet been written to the storage device 12, from the log buffer in the memory .

[0093] Next, in step S1102, the storage control unit 31 writes the log 60 acquired in step S1101 to the storage device 12. Next, in step S1103, the storage control unit 31 deletes the log 60 written to the storage device 12 in step S1102 from the log buffer.

[0094] <Base image saving process according to the first embodiment> 12 is a flowchart of a base image backup process according to Example 1. The base image process is a process of writing the entire memory area (base image) to be protected to the storage device 12. In this example, the base image process is used to protect control information, and is executed at a predetermined timing, for example, when the control information log on the storage device 12 has accumulated to a certain amount or more.

[0095] First, in step S1201, the storage control unit 31 refers to the sequence number and acquires the latest sequence number at the current time.

[0096] Next, in step S1202, the storage control unit 31 writes the entire base image in the memory 24 to the storage device 12. The area of the storage device 12 into which the base image is written is called the base image area. Once this process is complete, the old log 60 becomes unnecessary.

[0097] Next, in step S1203, the storage control unit 31 invalidates all logs 60 prior to the sequence number acquired in step S1201.

[0098] <Log recovery process according to the first embodiment> 13 is a flowchart of log recovery processing according to Example 1. The log recovery processing is processing for recovering control information and dirty data in the memory 24 from a log 60 stored in advance in the storage device 12 when information in the memory 24 is lost due to a factor such as a storage node failure. The log recovery processing is executed by the CPU 21 of the storage node 10 at the recovery destination before resuming acceptance of I / O.

[0099] First, in step S 1301 , the storage control unit 31 reads out a base image from the base image area on the storage device 12 and stores it in the storage control information 40 on the memory 24 .

[0100] Next, in step S1302, the storage control unit 31 reads the control information log 60 and the cache data log 60 from the storage device 12 and sorts them in order from oldest to newest according to their sequence numbers. Next, in step S1303, the storage control unit 31 reflects the logs 60 sorted in step S1302 in the respective areas of the control information log buffer 43 and the cache data log buffer 44 on the memory 24 in the sorted order according to the address information written in the headers of the logs 60.

[0101] <System recovery process according to the first embodiment> FIG. 14 is a flowchart of the system recovery process according to the first embodiment.

[0102] The system recovery process is a process for recovering the system when a failure occurs in a storage node 10 in a cluster 15 (hereinafter, the storage node 10 where the failure occurred is referred to as the "failed storage node") by preparing a storage node 10 (hereinafter, referred to as the "alternate storage node") to replace the failed storage node and restoring data that was in the memory 24 of the failed storage node to the memory of the alternative storage node.

[0103] The system recovery process is initiated when the cluster controller 30 (hereinafter referred to as the "failure-detecting cluster controller") of any storage node 10 in the cluster 15 detects a failure in another storage node (failed storage node).

[0104] First, in step S1401, the failure detection cluster control unit requests the cloud control device 5 to create an alternative storage node to substitute for the failed storage node.

[0105] Next, in step S1402, the failure detection cluster control unit selects a memory data recovery method. In step S1402, the failure detection cluster control unit identifies the storage control unit belonging to the failed storage node and identifies each redundancy group to which each of the storage control units belongs (hereinafter referred to as a "failure occurrence redundancy group"). Then, for each failure occurrence redundancy group, the failure detection cluster control unit determines whether a failure has occurred in all of the storage control units 31 belonging to the failure occurrence redundancy group. For a failure occurrence redundancy group for which it has determined that a failure has occurred in all of the storage control units 31 belonging to the failure occurrence redundancy group, the failure detection cluster control unit selects "memory data recovery by log" (first recovery method). On the other hand, for a failure occurrence redundancy group for which it has determined that a storage control unit 31 in a normal state exists among the storage control units 31 belonging to the failure occurrence redundancy group, the failure detection cluster control unit selects "memory data recovery from a redundant system" (second recovery method).

[0106] "Memory data recovery using logs" is a method of restoring memory data by using the log 60 written by the failed storage node to the storage device 12. "Memory data recovery from a redundant system" is a method of restoring memory data by duplicating the memory data that is made redundant in a storage control unit 31 in a normal state that belongs to the same redundancy group 50 as the failed storage node.

[0107] In general, the data access speed to the storage device 12 is often slower than the memory data copy speed between storage nodes 10. For example, if a failure occurs in all of the storage control units 31 that belong to the same redundancy group 50 as the failed storage node, memory data recovery using logs is selected. On the other hand, if there is a storage control unit 31 that is not failed among the storage control units 31 that belong to the same redundancy group 50 as the failed storage node, memory data recovery from the redundant system is selected. In this way, by using available memory data, the time required for system recovery processing can be shortened.

[0108] If the failure detection cluster control unit selects "memory data recovery using logs" in step S1402 (step S1403 Yes), the failure detection cluster control unit proceeds to step S1404. On the other hand, if the failure detection cluster control unit selects "memory data recovery from redundant system" in step S1402 (step S1403 No), the failure detection cluster control unit proceeds to step S1409. When proceeding to step S1409, the failure detection cluster control unit requests the cloud control device 5 to allocate the storage device 12 to an alternative storage node.

[0109] In step S1404, the failure detection cluster control unit selects one of the storage devices 12 (hereinafter referred to as the "selected storage device") that was assigned to the failed storage node, and requests the cloud control device 5 to perform a detachment process to detach the selected storage device from the failed storage node.

[0110] Next, in step S1405, the failure detection cluster control unit requests the cloud control device 5 to perform an attach process to attach the storage device 12 to an alternative storage node.

[0111] Next, in step S1406, the failure detection cluster control unit determines whether the processing of steps S1404 and S1405 has been completed for all storage devices 12 assigned to the failed storage node. If the processing has been completed for all storage devices 12 (step S1406: Yes), the failure detection cluster control unit proceeds to step S1407. On the other hand, if the processing has not been completed for all storage devices 12 (step S1406: No), the failure detection cluster control unit returns the processing to step S1404. In step S1404 to which the processing has been returned from step S1406, the failure detection cluster control unit sequentially switches the storage device 12 selected in step S1404, and repeats the processing of steps S1404 to S1406 until the determination result of step S1406 is Yes.

[0112] In step S1407, the cluster controller 30 of the alternative storage node (hereinafter referred to as the "alternative cluster controller") selects a storage node 10 having a valid log (specific storage node) by specifying the redundancy group 50 to which the failed storage node belongs and executing valid log selection processing in order to restore the memory contents lost in the failed storage node. Details of the valid log selection processing will be described later with reference to FIG. 15.

[0113] Next, in step S1408, the alternative cluster control unit executes log recovery processing (Figure 13) (first recovery processing) to restore the memory contents from the log in the storage device 12 associated with the storage node 10 having the "valid log" selected in step S1407.

[0114] In step S1409 (processing transferred from step S1403 NO), since the second recovery method has been selected by the failure-detecting cluster control unit, instead of the first recovery method, the alternative cluster control unit restores the memory contents in the memory 24 of the alternative storage node based on the memory contents stored in the memory 24 of the storage node 10 having a storage control unit 31 that is operating normally and belongs to the same redundancy group as the storage control unit 31 in which the failure occurred (referred to as the ``failed storage control unit'').

[0115] Next, in step S1410, the alternative cluster control unit determines whether restoration of the stored contents has been completed in the memories 24 of the storage control units belonging to all alternative storage nodes. If the processing has been completed for all memories 24 (Yes in step S1410), the alternative cluster control unit proceeds to step S1411. On the other hand, if the processing has not been completed for all memories 24 (No in step S1410), the alternative cluster control unit returns the processing to step S1403. In step S1403 to which the processing has been returned from step S1410, the failure detection cluster control unit switches the failed redundancy group selected in S1403 in order, and repeats the processing of steps S1403 to S1410 until the determination result in step S1410 becomes Yes.

[0116] Next, in step S1411, the alternative cluster control unit starts up each storage control unit 31 in the alternative storage node as an alternative storage control unit that substitutes for the failed storage control unit. The alternative cluster control unit then updates the status of each storage node 10 in the storage node management table 45, and updates the status of each storage control unit 31 in the storage control unit management table 46.

[0117] <Valid log selection process according to the first embodiment> FIG. 15 is a flowchart of the valid log selection process according to the first embodiment. The valid log selection process is a process for selecting a storage node 10 associated with the "valid log" of a specified redundancy group 50. As described above, the "valid log" refers to the log 60 that contains the latest memory update contents of the storage control unit 31 that belongs to the specified redundancy group 50. On the other hand, if a storage node 10 is unable to write out the log 60 due to a factor such as a storage node failure, the log 60 associated with the storage node 10 will no longer contain the latest memory update contents. Such a log 60 is called an "invalid log."

[0118] When all storage control units 31 of a certain redundancy group 50 stop operating due to factors such as a storage node failure, causing data in memory 24 to be lost, in order to restore the memory contents associated with that redundancy group 50, it is necessary to identify to which storage node 10 the storage device 12 containing the "valid log" is assigned.

[0119] First, in step S1501, the alternative cluster control unit refers to the storage node management table 45 and acquires the status of each storage node 10 in the cluster 15 that it controls.

[0120] Next, in step S1502, the alternative cluster control unit sorts the storage nodes 10 in the cluster 15 that it controls by the time of node failure. Next, in step S1503, the alternative cluster control unit selects the storage node 10 that last failed from among the storage nodes 10 belonging to the target redundancy group as the storage node 10 with the "valid log."

[0121] <Effects of Example 1> In the first embodiment, a specific storage node where a failure last occurred is selected from among storage nodes having a storage controller that belongs to the same redundancy group as the failed storage controller of the failed storage node. Then, based on the validity log stored in the storage device assigned to the specific storage node, the stored contents such as cache data and control information are restored in the memory of the alternative storage node. This makes it possible to avoid the loss of dirty data that has not yet been stored in the storage device as much as possible, maintain memory capacity efficiency, and realize a highly reliable storage system.

[0122] Furthermore, in the first embodiment, if any of the storage control units 31 belonging to the same redundancy group as the failed storage control unit is operating normally, the memory contents of the memory 24 of the alternative storage node are restored based on the memory contents of the memory 24 of the normally operating storage control unit 31. This allows for quicker recovery from a failure than when the memory contents of the memory 24 are restored from a log. In particular, if the storage system 1 is configured with a disk array device, it is not necessary to detach and attach the storage device, allowing for quicker recovery from a failure.

[0123] <Modification 1 of Example 1> For example, the system recovery process described with reference to Figure 14 has been described as being performed by the storage node 10 that first detects the failure in the storage node 10 that has failed among the storage nodes 10 that make up the same cluster 15, but the present invention is not limited to this.For example, it is also possible to select one representative storage node 10 (hereinafter referred to as the "representative storage node") from among the storage nodes 10 that make up the cluster 15, and have the selected representative storage node perform the relevant system recovery process.

[0124] 7, the storage control units 31 that are in active mode are started in a distributed manner for all alternative storage nodes ("storage node Y" and "storage node X" in FIG. 7) and then used as part of the cluster 15. This method has the advantage that load balancing among the storage nodes 10 becomes possible when the storage nodes 10 recover from the failure and resume I / O processing.

[0125] However, the present invention is not limited to this. After restoring the memory contents to one alternative storage node ("storage node Y" in FIG. 7), all storage controllers 31 ("storage controller A" and "storage controller B" in FIG. 7) in the alternative storage node may be started in active mode and used as part of the cluster 15. This method has the advantage of enabling faster recovery from a failure and resumption of I / O processing for the storage controllers 31. Even in this case, by starting the storage controllers 31 ("storage controller A" and "storage controller C" in FIG. 7) of the remaining alternative storage node ("storage node X" in FIG. 7) in standby mode and then performing a separate failover process, the storage controllers 31 in active mode can be relocated among the storage nodes 10, thereby balancing the load among the storage nodes 10. This method can achieve both quick resumption of I / O processing for the storage controllers 31 and load balancing among the storage nodes 10.

[0126] <Modification 2 of Example 1> For example, in steps S708 and S709 of the system recovery process described with reference to Fig. 7, an example is shown in which all storage control units A of "storage node X" and "storage node Y" perform memory data recovery using logs. This method has the advantage that the log recovery process of storage control unit A can be executed in parallel on each storage node.

[0127] However, the present invention is not limited to this. At least one storage control unit A may perform memory data recovery using logs, while the remaining storage control units A may perform "memory data recovery from redundant systems." In other words, if it is determined that a failure has occurred in all storage control units belonging to a certain failure-occurring redundancy group, at least one storage control unit within the failure-occurring redundancy group may select memory data recovery using logs, while the remaining storage control units may select memory data recovery from redundant systems. This method has the advantage of reducing the number of accesses to the storage devices involved in log recovery processing and the amount of CPU calculations.

[0128] <Modification 3 of Example 1> For example, in the system recovery process described with reference to Fig. 7, the first recovery method or the second recovery method is selected for each failed redundancy group. This method has the advantage of being able to quickly recover from a failure by selecting the second recovery method in a failed redundancy group in which at least one storage control unit is operating normally.

[0129] However, the present invention is not limited to this. Either the first recovery method or the second recovery method may be selected for each failed storage node. That is, if at least one storage controller belonging to an alternative storage node selects the first recovery method, the failure detection cluster controller may also select the first recovery method for all other storage controllers belonging to the alternative storage node. This method can reduce the amount of network bandwidth consumed by memory copies between storage nodes.

[0130] <Fourth Modification of First Embodiment> For example, in the system recovery process described with reference to Fig. 14, when the second recovery method is selected, data is copied (rebuild process) to a storage device that is pre-attached to the alternative storage node. This method has the advantage of reducing the time required for system recovery because it requires fewer detachment and attachment processes for storage devices.

[0131] However, the present invention is not limited to this. When the second recovery method is selected, a storage device of the failed storage node may be selected and detached (step S1404), and the detached storage device may be attached to an alternative storage node (step S1405), and data may be copied to the attached storage device (rebuild process). This method can shorten the time required for the rebuild process. [Example]

[0132] In the first embodiment, the storage device 12 is attached to one storage node 10 and is accessed only from this storage node 10. In contrast, in the second embodiment, the storage device 12 can be accessed from any storage node 10 via the network. In the following explanation of the second embodiment, the differences from the first embodiment will be mainly explained, and duplicate explanations of the same configurations and processes will be omitted.

[0133] <Configuration of storage system 1B according to embodiment 2> 16 is a diagram illustrating the overall configuration of a storage system 1B according to a second embodiment. In this embodiment, unlike in the first embodiment, the cloud system 4 is replaced with a storage device 7, and the cloud control device 5 is replaced with a storage management device 6. This shows that the present invention can be applied not only to cloud environments but also to on-premise environments. That is, the cloud system 4 and the cloud control device 5 in the first embodiment are replaced with the storage device 7 and the storage management device 6, respectively, in this embodiment.

[0134] The storage device 7 has a cluster 15 including a plurality of storage nodes 10, and a disk array device 17 including a plurality of storage devices 12. In the storage system 1B, the storage nodes 10 constituting the cluster 15 and the storage devices 12 constituting the disk array device 17 are connected to each other via a network 18 for the disk array.

[0135] Specifically, the disk array device 17 may be configured as a Fabric-attached Bunch of Flash (FBOF) or a Just a Bunch of Disks (JBOD), etc. Also, similar to the first embodiment, the disk array device 17 may be configured as a block storage providing service unit 13 of a cloud system.

[0136] The storage node 10 is a physical or virtual server device that provides a storage area for reading and writing data to the host device 3. In reality, one or more storage devices 12 in the disk array device 17 are allocated to each storage node 10. The storage node 10 then virtualizes the storage areas provided by these allocated storage devices 12 and provides them to the host device 3.

[0137] <Configuration of Storage Node According to Second Embodiment> Fig. 17 is a diagram illustrating a schematic configuration of a storage node according to Example 2. As shown in Fig. 17, the storage node 10 according to this example includes a CPU 21, a host-directed communication device 22, and a disk array-directed communication device 25, which are interconnected via an internal network 20, and a memory 24 connected to the CPU 21. Each storage node 10 includes one or more CPUs 21, one or more host-directed communication devices 22, one or more disk array-directed communication devices 25, and one or more memories 24.

[0138] The difference between the second embodiment (FIG. 17) and the first embodiment (FIG. 2) is that the block storage providing service-oriented communication device 23 is changed to a disk array-oriented communication device 25. Therefore, the process performed using the block storage providing service-oriented communication device 23 in the first embodiment is replaced with the disk array-oriented communication device 25 in the present embodiment.

[0139] The disk array communication device 25 is an interface through which the storage node 10 communicates with the storage devices 12 in the disk array device 17 via the disk array network 18. The disk array communication device 25 is configured using, for example, an NIC or the like, similar to the host communication device 22. The disk array communication device 25 controls the protocol when communicating with the storage devices 12.

[0140] The storage management device 6 is a general-purpose computer device that has the function of allowing a system administrator to control the storage nodes 10 and disk array device 17 in the cluster 15 in the storage device 7. The storage management device 6 adds, deletes, or changes the configuration of the storage nodes 10, cluster 15, and storage devices 12 in the disk array device 17 via the network 2 in response to operations by the system administrator. The storage management device 6 may be a physical computer device or a virtual computer device such as a virtual machine.

[0141] <System recovery process according to the second embodiment> The system recovery process according to the second embodiment will be described below with reference to Fig. 7 and Fig. 14. Unlike the first embodiment, the system recovery process according to the second embodiment does not require detaching the storage device 12 from the failed storage node (step S704 in Fig. 7, step S1402 in Fig. 14) and attaching the storage device 12 to an alternative storage node (step S705 in Fig. 7, step S1403 in Fig. 14).

[0142] <Effects of Example 2> In this embodiment, when recovering data from a valid log, it is not necessary to switch (detach and attach) the storage devices 12 between storage nodes 10, thereby reducing the time required to recover data from the log. Therefore, this is suitable for a configuration in which it takes a long time to switch the storage devices 12 (for example, an on-premise SDS configuration). [Example]

[0143] The third embodiment aims to maintain reliability after a storage node is stopped due to a maintenance operation or the like on the storage node.

[0144] Stopping a certain storage node 10 reduces the redundancy of the storage control unit 31 operating in that storage node 10 (the number of storage nodes for which that storage control unit 31 is made redundant). For example, in a configuration in which two storage control units 31 belong to a certain redundancy group 50, if one storage node 10 is stopped for maintenance or other reasons, the number of operating storage nodes 10 that belong to the same redundancy group 50 as that storage node 10 drops to one, and the reliability of the entire storage system decreases while one storage node 10 is stopped.

[0145] Therefore, in this embodiment, before shutting down a storage node 10, the storage control units 31 belonging to the same redundancy group 50 as the storage node 10 to be shut down are made redundant using another storage node 10, and then the storage node 10 to be shut down is shut down. This makes it possible to maintain the reliability of the entire storage system even when the storage node to be shut down is shut down.

[0146] In this embodiment, when any of the cluster controllers 30 constituting the cluster 15 receives a maintenance request for the storage node 10 from the cloud control device 5, it executes a storage node maintenance stop process, which will be described later.

[0147] <Storage node maintenance stop processing according to the third embodiment> 18 is a flowchart of a storage node maintenance stop process according to Example 3. The storage node maintenance stop process is executed, for example, every time a stop instruction is input to a target storage node for storage node maintenance.

[0148] First, in step S1801, the cluster control unit 30 (hereinafter referred to as the "cluster control unit to be maintained") of the storage node 10 to be shut down for maintenance (hereinafter referred to as the "storage node to be shut down") refers to the storage control unit management table 46 and creates a list of the storage control units 31 operating in the storage node to be shut down.

[0149] Next, in step S1802, the cluster control unit to be maintained secures a storage node 10 (hereinafter referred to as a "redundancy destination storage node") that will establish a redundant configuration for the storage control unit 31 operating in the storage node to be stopped. Next, in step S1803, the cluster control unit to be maintained makes each storage control unit 31 included in the list created in step S1802 redundant using the "redundancy destination storage node."

[0150] Next, in step S1804, the cluster control unit to be maintained starts processing to fail over each storage control unit 31 made redundant in step S1804 (hereinafter referred to as the "storage control unit to be made redundant") to the "redundancy destination storage node." Specifically, the cluster control unit to be maintained references the storage control unit management table 46 (FIG. 5), and changes the status (column 462) of each storage control unit 31 made redundant in step S1804 to "normal (active)."

[0151] Next, in step S1805, the cluster control unit to be maintained stops the storage node to be stopped. At the same time, the cluster control unit to be maintained updates the storage node management table 45. Specifically, the cluster control unit to be maintained updates the status (column 452) of the stopped storage node 10 to record that the storage node 10 has stopped (for example, updates the status to "stopped" or the like). Furthermore, the cluster control unit to be maintained updates the storage control unit management table 46. Specifically, the cluster control unit to be maintained updates the status (column 462) of each storage control unit 31 operating in the stopped storage node 10 to record that the storage control unit has stopped (for example, updates the status to "stopped" or the like).

[0152] Next, in step S1806, the cluster control unit to be maintained sends a notification to the cloud control device 5 requesting that the storage device 12 of the storage node 10 to be stopped be taken over by the "redundancy destination storage node." After receiving the notification, the cloud control device 5 takes over the storage device 12 in accordance with the request.

[0153] <Effects of Example 3> According to this embodiment, even during maintenance of the storage node 10, the redundancy of each storage control unit 31 in the cluster 15 can be maintained, and the reliability of the cluster can be maintained.

[0154] The present invention is not limited to the above-described embodiment, but includes various modifications. Furthermore, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0155] It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment, or to add, delete, or replace part of the configuration of each embodiment with the configuration of another embodiment.

[0156] Furthermore, the above-mentioned components, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Furthermore, the above-mentioned components, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in memory, a storage device such as a hard disk or SSD, or a storage medium such as an IC card or SD card.

[0157] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0158] 1, 1B: storage system, 3: host device, 5: cloud control device, 6: storage management device, 7: storage device, 10: storage node, 12: storage device, 14: network, 15: cluster, 17: disk array device, 18: network, 21: CPU, 24: memory, 30: cluster control unit, 31: storage control unit, 40: storage control information, 42: cache data area, 43: control information log buffer, 44: cache data log buffer, 45: storage node management table, 46: storage control unit management table, 50, 50A, 50B: redundancy group, 60: log.

Claims

1. A storage system having a plurality of storage nodes and a plurality of storage devices that provide storage areas to the plurality of storage nodes, The storage node a memory for storing cache data relating to data that the storage node reads and writes between the storage area; a storage control unit that executes reading and writing of the data from and to the storage area in response to a request from a host device, updates the cache data related to the data in the memory, creates a log related to the cache data, and stores the log in the storage device assigned to the storage node; a cluster control unit that manages the plurality of storage control units in redundancy groups, distributes the plurality of storage control units belonging to the same redundancy group among the plurality of storage nodes, and manages the plurality of storage control units, and monitors the occurrence of failures in other storage nodes; the storage control units belonging to the same redundancy group synchronize the cache data stored in the memory, The failure detection cluster control unit, which is the cluster control unit that detects the occurrence of a failure in another storage node, requests an external control device to create an alternative storage node, which is the storage node that substitutes for the failed storage node, which is the storage node in which the failure has occurred; the failure detection cluster control unit executes a detach process to detach the storage device assigned to the failed storage node from the failed storage node, and then requests the control device to execute an attach process to assign the storage device to the alternative storage node; an alternative cluster control unit that is the cluster control unit of the alternative storage node selects a specific storage node in which a failure has occurred last from among the storage nodes having a storage control unit that belongs to the same redundancy group as the failed storage control unit of the failed storage node, and selects and executes a first recovery method for restoring storage contents in the memory of the alternative storage node based on the log stored in the storage device assigned to the specific storage node; In the alternative storage node, an alternative storage control unit is started, which is the storage control unit that takes over for the failed storage control unit. A storage system comprising:

2. 2. The storage system according to claim 1, the memory stores control information related to the storage control unit; the log includes a log related to the control information, The storage control unit The control information is updated in the memory to create the log relating to the control information, and the log is stored in the storage device assigned to the storage node. A storage system comprising:

3. 2. The storage system according to claim 1, The failure detection cluster control unit determining whether a failure has occurred in all of the storage control units that belong to the same redundancy group as the failed storage control unit; When it is determined that a failure has occurred in all of the storage control units belonging to the same redundancy group as the failed storage control unit, the alternative cluster control unit selects and executes the first recovery method, When it is determined that any one of the storage control units belonging to the same redundancy group as the failed storage control unit is operating normally, the alternative cluster control unit selects and executes, instead of the first recovery method, a second recovery method for restoring storage contents in the memory of the alternative storage node based on storage contents stored in the memory of the storage node having the storage control unit that belongs to the same redundancy group as the failed storage control unit and is operating normally. A storage system comprising:

4. 2. The storage system according to claim 1, The failure detection cluster control unit determining whether a failure has occurred in all of the storage control units that belong to the same redundancy group as the failed storage control unit; If it is determined that a failure has occurred in all of the storage control units that belong to the same redundancy group as the failed storage control unit, the alternative cluster control unit selects and executes the first recovery method; If it is determined that any of the storage control units belonging to the same redundancy group as the failed storage control unit is operating normally, The alternative cluster control unit, instead of the first recovery method, requests the control device to perform a detach process to detach the storage device assigned to the failed storage node from the failed storage node, and then to perform an attach process to assign the storage device to the alternative storage node, and selects and executes a second recovery method to restore the storage contents in the memory of the alternative storage node based on the storage contents stored in the memory of the storage node having the storage control unit that is operating normally and belongs to the same redundancy group as the failed storage control unit. A storage system comprising:

5. 2. The storage system according to claim 1, a disk array device including the plurality of storage devices; the plurality of storage nodes and the disk array device are connected to each other via a predetermined network, The detachment process and the attachment process are This is executed as a deletion, addition, or configuration change of the storage device within the disk array device based on control from a management device of the storage system via the specified network. A storage system comprising:

6. 2. The storage system according to claim 1, The cluster control unit of the storage node to which the stop instruction is input, creating a list of the storage control units that are operating in the target storage node and are targets for redundancy; a redundancy destination storage node is secured to make the target storage node redundant; The storage control unit to be made redundant is made redundant in the redundancy destination storage node; After executing a detach process for detaching the storage device assigned to the target storage node from the target storage node, a request is made to the control device to execute an attach process for assigning the storage device to the redundancy destination storage node. A storage system comprising:

7. 2. The storage system according to claim 1, The alternative cluster control unit All of the alternative storage control units that are in the active mode out of the active mode and the standby mode in the redundancy group are started so as to be distributed and arranged on all of the alternative storage nodes. A storage system comprising:

8. 2. The storage system according to claim 1, The alternative cluster control unit All of the alternative storage control units that are in the active mode out of the active mode and the standby mode in the redundancy group are started so as to be centrally arranged on some of the alternative storage nodes. A storage system comprising:

9. 9. The storage system according to claim 8, The alternative cluster control unit After starting up all of the alternative storage control units that will be in the active mode so as to be centrally arranged on some of the alternative storage nodes, All of the alternative storage control units that are set to the active mode are relocated so as to be distributed to all of the alternative storage nodes. A storage system comprising:

10. A failure handling method for a storage system, the method being executed by a storage system having a plurality of storage nodes and a plurality of storage devices that provide storage areas to the plurality of storage nodes, comprising: The storage node a memory for storing cache data relating to data that the storage node reads and writes between the storage area; a storage control unit that executes reading and writing of the data from and to the storage area in response to a request from a host device, updates the cache data related to the data in the memory, creates a log related to the cache data, and stores the log in the storage device assigned to the storage node; a cluster control unit that manages the plurality of storage control units in redundancy groups, distributes the plurality of storage control units belonging to the same redundancy group among the plurality of storage nodes, and manages the plurality of storage control units, and monitors the occurrence of failures in other storage nodes; the storage control units belonging to the same redundancy group synchronize the cache data stored in the memory, a failure detection cluster control unit that is the cluster control unit that detects the occurrence of a failure in another storage node, requesting an external control device to create an alternative storage node that is the storage node that substitutes for the failed storage node, which is the storage node where the failure has occurred; An alternative cluster control unit that is the cluster control unit of the alternative storage node, execute a detach process to detach the storage device assigned to the failed storage node from the failed storage node, and then request the control device to execute an attach process to assign the storage device to the alternative storage node; select a specific storage node in which the failure last occurred from among the storage nodes having the storage control unit that belongs to the same redundancy group as the failed storage control unit of the failed storage node; and select and execute a first recovery method to restore storage contents in the memory of the alternative storage node based on the log stored in the storage device assigned to the specific storage node; In the alternative storage node, an alternative storage control unit is started, which is the storage control unit that takes over for the failed storage control unit. A fault handling method for a storage system, characterized by including each of the above methods.

Images