Information processing system and information processing method
The distributed storage system addresses communication delays and costs by using redundancy groups with local data management and failover, ensuring high availability and efficient I/O performance across geographically separated zones.
Patent Information
- Application Number
- JP2022212311
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-12-28
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-12-28
AI Technical Summary
The challenge of maintaining high availability in a distributed storage system across geographically separated cloud availability zones is exacerbated by communication delays and costs, which affect I/O performance and increase operational expenses due to inter-zone communication.
A distributed storage system with redundancy groups across multiple bases, where active and standby storage controllers manage data locally and switch roles upon failure, ensuring data locality and minimizing cross-zone communication.
This approach ensures high availability by enabling seamless failover within the same availability zone, reducing communication delays and costs, and maintaining I/O performance.
Smart Images

Figure 0007592063000001 
Figure 0007592063000002 
Figure 0007592063000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing system and an information processing method, which are suitable for application to, for example, a distributed storage system. [Background technology]
[0002] In recent years, the need for storage to manage data on the cloud has increased with the expansion of cloud usage. In particular, since the cloud is made up of multiple locations (hereinafter referred to as availability zones), there is a demand for highly available storage systems that can withstand failures on an availability zone basis.
[0003] As a technology for increasing the availability of a storage system, for example, Patent Document 1 discloses a technology for hierarchically making data redundant within / between data centers. Patent Document 2 discloses a technology for storing a code (parity) for data recovery in one or more storage nodes different from the storage destination of user data. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-071100 A [Patent Document 2] JP 2020-107082 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, cloud availability zones are usually geographically separated, and configuring a distributed storage system across availability zones creates the problem of communication between the availability zones, which affects I / O performance due to communication delays.In addition, charges are incurred according to the amount of communication between availability zones, so there is also the problem of high costs if the amount of communication is large.
[0006] The present invention has been made in consideration of the above points, and a primary object of the present invention is to propose a highly available information processing system and information processing method that can withstand failures at the base (availability zone) level, and another object of the present invention is to propose an information processing system and information processing method that can further suppress degradation of I / O performance caused by communication delays associated with communications between bases, and the occurrence of costs due to communications between bases. [Means for solving the problem]
[0007] In order to solve the above problem, the present invention provides an information processing system having a plurality of storage servers arranged at each of a plurality of bases connected by a network, the information processing system including: a storage device arranged at each of the bases for storing data; and a storage controller implemented in the storage server, providing a logical volume to a higher-level application and processing data read from and written to the storage device via the logical volume. 、 before A management server that manages the storage servers and forming a redundancy group including a plurality of the storage controllers arranged at different bases, the redundancy group including a storage controller in an active state that processes data, and a storage controller in a standby state that takes over the processing of the data when a failure occurs in the storage controller in the active state, the storage controller in the active state stores the data from a higher-level application arranged at the same base in the storage device arranged at the base, and executes a process for storing redundancy data for restoring data to be stored in the storage device at the same base in the storage device arranged at the other base where the storage controller in the standby state of the same redundancy group is arranged. The storage controller migrates the logical volume to another storage controller at the same base based on a predetermined condition, and when a failure occurs at the base where the active storage controller is located, a standby storage controller that belongs to the same redundancy group as the active storage controller at the base where the failure occurred and is located at the other base changes to an active state and takes over the processing of the data, and restores the data stored in the storage device at the base where the failure occurred to the storage device at the base where the storage controller that took over the processing of the storage controller is located, using redundant data stored in the storage device at the other base, and the management server starts an application that is the same as the upper level application at the base where the storage controller that took over the processing of the storage controller is located. I did so. Further, in the present invention, in an information processing system having a plurality of storage servers arranged at a plurality of bases connected by a network, a storage device arranged at each of the bases and storing data, a storage controller implemented in the storage server, providing a logical volume to an upper level application and processing data read and written to the storage device via the logical volume, and a capacity monitoring unit for monitoring a used capacity or remaining capacity of each of the storage servers in the base are provided, and a redundancy group is formed including the plurality of storage controllers arranged at different bases, and the redundancy group includes a storage controller in an active state that processes data, and a storage controller in a standby state that takes over the processing of the data when a failure occurs in the storage controller in the active state, and the storage controller in the active state receives the data from the upper level application arranged at the same base, and stores the data in the storage device arranged at the base, and executes a process of storing redundancy data for restoring the data to be stored in the storage device at the same base in the storage device arranged at the other base where a standby storage controller of the same redundancy group is arranged, and when the used capacity or the remaining capacity of any of the storage servers reaches a predetermined condition, the capacity monitoring unit expands the capacity of each of the storage servers in which the storage controllers constituting the redundancy group to which the storage controller arranged in the storage server belongs are respectively implemented, and when it is not possible to expand the capacity of each of the storage servers in which the storage controllers constituting the redundancy group to which the storage controller arranged in the storage server belongs are respectively implemented, the capacity monitoring unit moves a logical volume provided by the storage controller of the storage server to the other storage server installed at the same base.
[0008] Also, in the present invention, there is provided an information processing method executed in an information processing system having a plurality of storage servers arranged at each of a plurality of bases connected by a network, the information processing system including: a storage device arranged at each of the bases for storing data; and a storage controller implemented in the storage server, providing a logical volume to a higher-level application and processing data read from and written to the storage device via the logical volume. 、 before A management server that manages the storage servers and forming a redundancy group including a plurality of the storage controllers arranged at different bases, the redundancy group including a storage controller in an active state that processes data, and a storage controller in a standby state that takes over the processing of the data when a failure occurs in the storage controller in the active state, the storage controller in the active state stores data from a higher-level application arranged at the same base in the storage device arranged at the base, and executes a process of storing redundancy data for restoring data to be stored in the storage device at the same base in the storage device arranged at the other base where the storage controller in the standby state of the same redundancy group is arranged. 1. Steps a second step in which the storage controller moves the logical volume to another storage controller at the same base based on a predetermined condition, while, when a failure occurs at the base where the active storage controller is located, a standby storage controller that belongs to the same redundancy group as the active storage controller at the base where the failure occurred and is located at the other base changes to an active state and takes over the processing of the data, the data stored in the storage device at the base where the failure occurred is restored to the storage device at the base where the storage controller that took over the processing of the storage controller is located by using redundant data stored in the storage device at the other base, and the management server starts up an application that is the same as the upper level application at the base where the storage controller that took over the processing of the storage controller is located; We have decided to set up the following. In the present invention, An information processing method executed in an information processing system having a plurality of storage servers arranged at each of a plurality of bases connected by a network, the information processing system comprising: a storage device arranged at each of the bases and storing data; a storage controller implemented in the storage server, providing a logical volume to an upper-level application and processing data read from and written to the storage device via the logical volume; and a capacity monitoring unit for monitoring, for each of the bases, a used capacity or remaining capacity of each of the storage servers in the base; and a redundancy group is formed including a plurality of the storage controllers arranged at different bases, the redundancy group including an active storage controller that processes data, and a standby storage controller that takes over the processing of the data when a failure occurs in the active storage controller, and the active storage controller transmits the data from the upper-level application arranged at the same base to the standby storage controller. The method includes a first step of storing data in the storage device located at a base, and executing a process of storing redundancy data for restoring data to be stored in the storage device at the same base in the storage device located at another base where a standby storage controller of the same redundancy group is located; and a second step of, when the used capacity or the remaining capacity of any of the storage servers reaches a predetermined condition, expanding the capacity of each of the storage servers in which the storage controllers constituting the redundancy group to which the storage controller implemented in the storage server belongs, and, when it is not possible to expand the capacity of each of the storage servers in which the storage controllers constituting the redundancy group to which the storage controller implemented in the storage server belongs, moving the logical volume provided by the storage controller of the storage server to the other storage server installed at the same base.
[0009] According to the information processing system and information processing method of the present invention, it is possible to store redundant data in another base while ensuring data locality. Therefore, even if a base-level failure occurs in the base where the active storage controller is located, the processing that was being performed by the active storage controller until then can be taken over by a standby storage controller that is part of the same redundant group. Effect of the Invention
[0010] According to the present invention, it is possible to realize a highly available information processing system and information processing method that can withstand failures at individual bases. [Brief description of the drawings]
[0011] [Figure 1] 1 is a block diagram showing an overall configuration of a storage system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a hardware configuration of a storage server. [Diagram 3] FIG. 2 is a block diagram showing a logical configuration of a storage server. [Figure 4] 13 is a diagram illustrating a storage configuration management table. [Diagram 5] FIG. 13 is a conceptual diagram illustrating a redundancy group. [Figure 6] FIG. 11 is a conceptual diagram illustrating a chunk group. [Figure 7] 1 is a conceptual diagram illustrating redundancy of user data in a storage system. [Figure 8] 13 is a diagram showing a storage controller management table. [Figure 9] 13 is a diagram showing a chunk group management table. [Figure 10] 1 is a conceptual diagram explaining a method for controlling access from an application to a host volume. FIG. [Figure 11] 13 is a diagram illustrating a host volume management table. [Figure 12] FIG. 13 is a conceptual diagram illustrating a failover when a failure occurs in a data center. [Figure 13] 11 is a conceptual diagram explaining switching of an access path to a host volume accompanying the movement of an application. FIG. [Figure 14] 13 is a flowchart showing a processing procedure for a server failure recovery process. [Figure 15] FIG. 13 is a diagram showing an example of the screen configuration of a host volume creation screen. [Figure 16]13 is a flowchart showing a processing routine for host volume creation processing. [Figure 17] 13 is a flowchart showing a procedure for a server capacity expansion process. [Figure 18] 13 is a flowchart showing a processing routine for server usage capacity monitoring processing; [Figure 19] 13 is a flowchart showing the processing routine for volume movement processing. [Figure 20] FIG. 11 is a block diagram showing the overall configuration of a storage system according to a second embodiment. [Figure 21] FIG. 11 is a block diagram showing a logical configuration of a storage server according to a second embodiment. [Figure 22] 13 is a flowchart showing a processing routine for host volume creation processing according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] An embodiment of the present invention will be described in detail below with reference to the drawings. Note that the following description and drawings are merely examples for explaining the present invention and do not limit the technical scope of the present invention. In addition, the same reference numerals are used to refer to common components in each drawing.
[0013] In the following explanation, various information may be explained using expressions such as "table," "list," and "queue," but the various information may be expressed in other data structures. To indicate independence from the data structure, "XX table," "XX list," and the like may be referred to as "XX information." When explaining the content of each piece of information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.
[0014] In addition, in the following description, when describing elements of the same type without distinguishing between them, reference signs or common numbers in reference signs will be used, and when describing elements of the same type while distinguishing between them, the reference signs of those elements will be used or an ID assigned to those elements will be used instead of the reference signs.
[0015] In the following description, the process performed by executing a program may be described, but the program is executed by at least one processor (e.g., CPU) to appropriately perform a predetermined process using a storage resource (e.g., memory) and / or an interface device (e.g., communication port), and therefore the subject of the process may be the processor. Similarly, the subject of the process performed by executing a program may be a controller, device, system, computer, node, storage system, storage device, server, management computer, client, or host having a processor. The subject of the process performed by executing a program (e.g., processor) may include a hardware circuit that performs a part or all of the process. For example, the subject of the process performed by executing a program may include a hardware circuit that performs encryption and decryption, or compression and decompression. The processor operates as a functional unit that realizes a predetermined function by operating according to the program. The device and system including the processor are devices and systems that include these functional units.
[0016] A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server includes a processor (e.g., a CPU) and a storage resource, and the storage resource may further store a distribution program and a program to be distributed. Then, the processor of the program distribution server may execute the distribution program, thereby distributing the program to be distributed to other computers. Also, in the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.
[0017] (1) Configuration of the Storage System According to the Present Embodiment (1-1) Configuration of the Storage System According to this Embodiment In Fig. 1, 1 indicates the cloud system according to this embodiment as a whole. This cloud system 1 is configured with first, second and third data centers 2A, 2B and 2C that are installed in different availability zones. In the following, when there is no need to particularly distinguish between the first to third data centers 2A to 2C, these will be collectively referred to as data center 2.
[0018] These data centers 2 are connected to each other via a dedicated network 3. A management server 4 is also connected to the dedicated network 3, and a user terminal 6 is connected to the management server 4 via a network 5 such as the Internet. Each of the data centers 2A to 2C is also provided with one or more storage servers 7 and one or more network drives 8 that constitute a distributed storage system. The configuration of the storage server 7 will be described later.
[0019] The network drives 8 are configured from large-capacity nonvolatile storage devices such as SAS (Serial Attached SCSI (Small Computer System Interface)), SSD (Solid State Drive), NVMe (Non Volatile Memory express), or SATA (Serial ATA (Advanced Technology Attachment)). Each network drive 8 is logically connected to one of the storage servers 7 in the same data center 2, and provides a physical storage area to the storage server 7 to which it is connected.
[0020] The network drives 8 may be housed in each storage server 7 or may be provided separately from the storage server 7, but in the following, they are assumed to be provided separately from the storage server 7 as shown in Fig. 3. Each storage server 7 is physically connected to each network drive 8 in the same data center 2 via an intra-data center network 34 (Fig. 3) such as a LAN (Local Area Network).
[0021] A host server 9 on which an application 33 (FIG. 3) such as a database application is implemented is also placed in each data center 2. The host server 9 is configured from a physical computer device or a virtual machine, which is a virtual computer device.
[0022] The management server 4 is composed of a general-purpose computer device incorporating a CPU (Central Processing Unit), memory, communication devices, etc., and is used by an administrator of a storage system 10 composed of each storage server 7 located in each data center 2 and the management server 4, to manage the storage system 10.
[0023] The management server 4, for example, sends commands in response to operational input by an administrator or requests from users (users) of the storage system 10 via user terminals 6 to the storage servers 7 of each data center 2, performs various settings and changes to those settings for these storage servers 7, and collects necessary information from the storage servers 7 of each data center 2.
[0024] The user terminal 6 is a communication terminal device used by a user of the storage system 10, and is configured from a general-purpose computer device. The user terminal 6 transmits requests, etc., in response to user operations, to the management server 4 via the network 5, and displays information transmitted from the management server 4.
[0025] 2 shows the physical configuration of the storage server 7. The storage server 7 is a general-purpose server device that has a function of reading / writing user data to a storage area provided by a network drive 8 in response to an I / O request from an application 33 (FIG. 3) installed in a host server 9.
[0026] As shown in FIG. 2, the storage server 7 is configured with one or more CPUs 21, intra-data center communication devices 22, and inter-data center communication devices 23 interconnected via an internal network 20, and one or more memories 24 connected to the CPU 21.
[0027] The CPU 21 is a processor that controls the operation of the storage server 7. The data center communication device 22 is an interface for the storage server 7 to communicate with other storage servers 7 in the same data center 2 and to access the network drives 8 in the same data center 2, and is composed of, for example, a LAN card or a NIC (Network Interface Card).
[0028] The inter-datacenter communication device 23 is an interface for the storage server 7 to communicate with the storage servers 7 in other data centers 2 via the dedicated network 3 (FIG. 1), and is configured by, for example, a NIC or a fiber channel card.
[0029] The memory 24 is composed of volatile semiconductor memory such as SRAM (Static RAM (Random Access Memory)) or DRAM (Dynamic RAM), and is used to temporarily store various programs and necessary data. The CPU 21 executes the programs stored in the memory 24, thereby executing various processes of the storage server 7 as a whole, as described below. Storage control software 25, which will be described later, is also stored and held in this memory 24.
[0030] Fig. 3 shows the logical configuration of the storage server 7. As shown in Fig. 3, each storage server 7 arranged in each data center 2 has one or more storage controllers 30 constituting a Software Defined Storage (SDS). The storage controller 30 is a functional unit realized by a CPU 21 (Fig. 2) executing storage control software 25 (Fig. 2) stored in a memory 24 (Fig. 2). The storage controller 30 has a data plane 31 and a control plane 32.
[0031] The data plane 31 is a functional unit that has the ability to read / write user data to the network drive 8 via the data center network 34 in response to write requests and read requests (hereinafter, collectively referred to as I / O (Input / Output) requests) from an application 33 implemented in the host server 9.
[0032] In practice, in this storage system 10, a virtual logical volume (hereinafter referred to as a host volume) HVOL, which is obtained by virtualizing a physical storage area provided by a network drive 8 within storage server 7, is provided to an application 33 implemented in host server 9 as a storage area for reading / writing user data. Furthermore, this host volume HVOL is associated with one of the storage controllers 30 within storage server 7 in which the host volume HVOL was created.
[0033] When the data plane 31 receives from an application 33 of a host server 9 a write request specifying a write destination within a host volume HVOL associated with the storage controller (hereinafter referred to as its own storage controller) 30 of which it is equipped, and user data to be written, the data plane 31 dynamically allocates a physical storage area provided by a network drive 8 logically connected to a storage server 7 in which its own storage controller 30 is implemented, to the virtual storage area specified as the write destination within the host volume HVOL, and stores the user data in that physical area.
[0034] In addition, when a read request specifying a read destination within the host volume HVOL is given by an application 33 of the host server 9, the data plane 31 reads user data from the corresponding physical area of the corresponding network drive 8 assigned to that read destination within the host volume HVOL, and transmits the read user data to the application 33.
[0035] The control plane 32 is a functional unit having a function of managing the configuration of the storage system 10. For example, the control plane 32 manages information such as what kind of storage servers 7 are arranged in each data center 2 and which network drives 8 are logically connected to these storage servers 7, by using a storage configuration management table 35 shown in FIG.
[0036] As shown in FIG. 4, the storage configuration management table 35 includes a data center ID column 35A, a server ID column 35B, and a network drive ID column 35C.
[0037] The data center ID column 35A stores an identifier (data center ID) that is assigned to each data center 2 and is unique to that data center 2. The server ID column 35B is divided into sections corresponding to the storage servers 7 arranged in the corresponding data centers 2, and each divided column (hereinafter, these will be referred to as a server column) stores an identifier (server ID) that is assigned to the corresponding storage server 7 and is unique to that storage server 7.
[0038] Furthermore, the network drive ID column 35C is divided to correspond to each server ID column 35B, and stores the identifiers (network drive IDs) of all network drives 8 that are logically connected to the storage server 7 (that is available to the storage server 7) whose server ID is stored in the corresponding server ID column 35B.
[0039] Therefore, in the example of Figure 4, for example, a data center 2 assigned a data center ID of "000" is configured with a storage server 7 assigned a server ID of "000" and a storage server 7 assigned a server ID of "001", and it is shown that a network drive 8 assigned a network drive ID of "000" and a network drive 8 assigned a network drive ID of "001" are logically connected to the storage server 7 "000".
[0040] 5 shows an example of a redundant configuration of the storage controller 30 in the storage system 10. In the storage system 10, each storage controller 30 implemented in the storage server 7 is managed as one group 36 for redundancy (hereinafter, this is called a redundancy group) together with one or more other storage controllers 30 implemented in any of the storage servers 7 in different data centers 2.
[0041] 5 shows an example in which one redundancy group 36 is configured by three storage controllers 30 in different data centers 2. In the following description, it is assumed that one redundancy group 36 is configured by these three storage controllers 30, but the redundancy group 36 may be configured by two or four or more storage controllers 30.
[0042] In the redundancy group 36, a priority is set for each storage controller 30. The storage controller 30 with the highest priority is set to an operation mode in which its data plane 31 (FIG. 3) can accept I / O requests from the host server 9 (active system state, hereinafter referred to as active mode), and the remaining storage controllers 30 are set to an operation mode in which its data plane 31 does not accept I / O requests from the host server 9 (standby system state, hereinafter referred to as standby mode). In FIG. 5, the storage controller 30 set to the active mode is indicated by "A", and the storage controller 30 set to the standby mode is indicated by "S".
[0043] In the redundancy group 36, if a failure occurs in the storage controller 30 set to active mode or in the storage server 7 in which that storage controller 30 is implemented, the operation mode of the storage controller 30 with the highest priority among the remaining storage controllers 30 that were set to standby mode up until that point is switched to active mode. As a result, even if the storage controller 30 set to active mode becomes unable to operate, the I / O processing that was being executed by that storage controller 30 can be taken over by another storage controller 30 that was set to standby mode up until that point (failover function).
[0044] In order to realize such a failover function, the control plane 32 of the storage controllers 30 belonging to the same redundancy group 36 always holds the same metadata. The metadata is information necessary for the storage controller 30 to execute processes related to various functions such as a capacity virtualization function, a hierarchical storage control function for moving frequently accessed data to a storage area with a faster response speed, a deduplication function for deleting duplicate data from stored data, a compression function for compressing and storing data, a snapshot function for retaining the state of data at a certain point in time, and a remote copy function for synchronously or asynchronously copying data to a remote location for disaster prevention. The metadata also includes the storage configuration management table 35 described above with reference to FIG. 4, the storage controller management table 40 described below with reference to FIG. 8, the chunk group management table 41 described below with reference to FIG. 9, and the host volume management table 52 described below with reference to FIG. 11.
[0045] When the metadata of a storage controller 30 in active mode constituting a redundancy group 36 is updated due to a configuration change or the like, the control plane 32 (FIG. 3) of that storage controller 30 transfers the difference between the metadata before and after the update as differential data to the other storage controllers 30 constituting the redundancy group 36, and based on this differential data, the metadata held by that storage controller 30 in that other storage controller 30 is updated by the control plane 32 of that storage controller 30. This ensures that the metadata of each storage controller 30 constituting the redundancy group 36 is always kept synchronized.
[0046] In this way, each storage controller 30 constituting a redundancy group 36 always holds the same metadata. Therefore, even if a failure occurs in a storage controller 30 set to active mode or in the storage server 7 on which the storage controller 30 is running, the processing that was being performed by that storage controller 30 can be immediately taken over by another storage controller 30 that constitutes the same redundancy group 36 as the storage controller 30.
[0047] 6 shows a method for managing storage areas in the storage system 10. In this storage system 10, the storage area provided by each network drive 8 is divided into physical areas of a fixed size (e.g., several hundred GB) and managed. Hereinafter, this physical area is referred to as a physical chunk 37.
[0048] The physical chunk 37 is managed as a group (hereinafter referred to as a chunk group) 38 for making user data redundant, together with one or more other physical chunks 37 defined in any of the network drives 8 in different data centers 2.
[0049] Figure 6 shows an example in which one chunk group 38 is composed of three physical chunks 37 (physical chunks 37 indicated by diagonal lines in the figure) each of which exists in a different data center 2, and the following explanation will be given assuming that one chunk group 38 is composed of three physical chunks 37 each of which exists in a different data center 2.
[0050] In principle, each physical chunk 37 constituting the same chunk group 38 is assigned to a storage controller 30 in the same data center 2 as the physical chunks 37 constituting the same redundancy group 36 .
[0051] Therefore, for example, a physical chunk 37 in a first data center 2A constituting a certain chunk group 38 is assigned to a storage controller 30 in the first data center 2A constituting a certain redundancy group 36. Also, a physical chunk 37 in a second data center 2B constituting that chunk group 38 is assigned to a storage controller 30 in the second data center 2B constituting that redundancy group 36, and a physical chunk 37 in a third data center 2C constituting that chunk group 38 is assigned to a storage controller 30 in the third data center 2C constituting that redundancy group 36.
[0052] User data is written to the chunk group 38 in accordance with a preset data protection policy. Data protection policies applied to the storage system 10 of this embodiment include mirroring and erasure coding (EC). "Mirroring" is a method of storing exactly the same user data as that stored in a certain physical chunk 37 in another physical chunk 37 that constitutes the same chunk group 38 as the physical chunk 37. As for "EC", there are a first method that does not guarantee data locality and a second method that guarantees data locality, and in this embodiment, the second method that guarantees data locality within the data center 2 is applied.
[0053] In other words, in the storage system 10 of this embodiment, regardless of whether mirroring or EC is specified as the data protection policy for the chunk group 38, the user data used by the application 33 (Figure 3) implemented in the host server 9 and metadata related to that user data are stored in the same data center 2 as the application 33.
[0054] An example of the second EC method applied to the storage system 10 will be specifically described with reference to Fig. 7. In the case of the second EC method according to this example, it is not necessary to assign the physical chunks 37 constituting the same chunk group 38 to each of the storage controllers 30 constituting the redundancy group 36.
[0055] In the following, as shown in FIG. 7, it is assumed that the host server in the first data center 2A writes first user data D1 (data consisting of "a" and "b" in the figure) to the host volume HVOL in the first storage server 7A, and this first user data D1 is stored in the first physical chunk 37A in the first storage server 7A.
[0056] Furthermore, a second physical chunk 37B that constitutes the same chunk group 38 as the first physical chunk 37A exists in a second storage server 7B in the second data center 2B, and second user data D2 (data composed of "c" and "d" in the figure) is stored in a memory area within the second physical chunk 37B that is the same memory area as the memory area in which the first user data D1 in the first physical chunk 37 is stored.
[0057] Similarly, a third physical chunk 37C exists in a third storage server 7C in a third data center 2C, which constitutes the same chunk group 38 as the first physical chunk 37A, and third user data D3 is stored in a memory area within the third physical chunk 37C that is the same as the memory area in which the first user data D1 in the first physical chunk 37A is stored.
[0058] In such a configuration, when a first application 33A implemented in a first host server 9A in a first data center 2A writes first user data D1 to a first host volume HVOL1 assigned to it, the first user data D1 is stored directly in a first physical chunk 37A by the data plane 31A of the corresponding storage controller 30A.
[0059] In addition, the data plane 31A divides the first user data D1 into two partial data D1A, D1B of the same size, "a" and "b," and transfers one of the partial data D1A ("a" in the figure) to a second storage server 7B that provides a second physical chunk 37B in the second data center 2B, and transfers the other partial data D1B ("b" in the figure) to a third storage server 7C that provides a third physical chunk 37C in the third data center 2C.
[0060] Furthermore, the data plane 31A reads one of the two partial data D2A, D2B of the same size, "c" and "d", obtained by dividing the second user data D2 from the second physical chunk 37B via the data plane 31B of the corresponding storage controller 30B of the second storage server 7B in the second data center 2B, the partial data D2A ("c" in the figure) being one of the two partial data D3A, D3B of the same size, "e" and "f", obtained by dividing the third user data D3 from the third physical chunk 37C via the data plane 31C of the corresponding storage controller 30C of the third storage server 7C in the third data center 2C. Then, the data plane 31A generates parity P1 from the partial data D2A called "c" and the partial data D3A called "e" that have been read, and stores the generated parity P1 in the first physical chunk 37A.
[0061] When the partial data D1A of "a" is transferred from the first storage server 7A, the data plane 31B of the storage controller 30B associated with the second physical chunk 37B in the second storage server 7B reads one of the partial data D3A, D3B of "e" and "f" ("f" in the figure) from the third physical chunk 37C via the data plane 31C of the corresponding storage controller 30C of the third storage server 7C in the third data center 2C. The data plane 31B also generates parity P2 from the read partial data D3B of "f" and the partial data D1A of "a" transferred from the first storage server 7A, and stores the generated parity P2 in the second physical chunk 37B.
[0062] Furthermore, when partial data D1B designated "b" is transferred from the first storage server 7A, the data plane 31C of the storage controller 30C associated with the third physical chunk 37C in the third storage server 7C reads one of the partial data D2A, D2B designated "c" and "d" ("d" in the figure) from the second physical chunk 37B via the data plane 31B of the corresponding storage controller 30B of the second storage server 7B arranged in the second data center 2B. Furthermore, the data plane 31B generates parity P3 from the read partial data D2B designated "d" and the partial data D1B designated "b" transferred from the first storage server 7A, and stores the generated parity P3 in the third physical chunk 37C.
[0063] The above processing is also performed when, in the second data center 2B, a second application 33B implemented in a second host server 9B writes user data D2 to a second host volume HVOL2 of a second storage server 7B, or when, in the third data center 2C, a third application 33C implemented in a third host server 9C writes user data D3 to a third host volume HVOL3 of a third storage server 7C.
[0064] By such a redundancy process of the user data D1 to D3, the first to third user data D1 to D3 used by the first to third applications 33A to 33C implemented in the first to third host servers 9A to 9C can be made redundant, and the first to third user data D1 to D3 can always be held in the same first to third data centers 2A to 2C as the first to third applications 33A to 33C. When a failure occurs in the host server 9, the user data stored in the host server 9 can be restored using the parity and the user data that is the basis for generating the parity and is stored in another host server 9. This prevents data transfer between the first to third data centers 2A to 2C of the first to third user data D1 to D3 used by the first to third applications 33A to 33C, and prevents a decrease in I / O performance and an increase in communication costs due to such data transfer. The number of user data and the number of parities are not limited to 2D1P, and can be set to any number.
[0065] In order to manage such redundancy groups 36 (FIG. 5) and chunk groups 38 (FIG. 6), the control plane 32 of each storage controller 30 manages a storage controller management table 40 as shown in FIG. 8 and a chunk group management table 41 as shown in FIG. 9 as part of the above-mentioned metadata.
[0066] The storage controller management table 40 is a table for managing the above-mentioned redundancy groups 36 set by an administrator, a user, etc., and is configured with a redundancy group ID column 40A, an active server ID column 40B, and a standby server ID column 40C, as shown in Fig. 8. In the storage controller management table 40, one row corresponds to one redundancy group 36.
[0067] The redundancy group ID column 40A stores an identifier (redundancy group ID) unique to the corresponding redundancy group 36, and the active server ID column 40B stores the server ID of the storage server 7 in which the storage controller 30 set to active mode is implemented within the corresponding redundancy group 36. The standby server ID column 40C stores the server ID of the storage server 7 in which the storage controller 30 set to standby mode is implemented within the redundancy group 36.
[0068] Therefore, in the example of Figure 8, in a redundancy group 36 assigned a redundancy group ID of "1", the storage controller 30 set to active mode is implemented in a storage server 7 assigned a server ID of "100", and the remaining two storage controllers 30 set to standby mode are respectively implemented in a storage server 7 assigned a server ID of "200" and a storage server 7 assigned a server ID of "300".
[0069] The chunk group management table 41 is a table for managing the chunk groups 38 set by an administrator, a user, or the like, and is configured with a chunk group ID column 41A, a data protection policy column 41B, and a physical chunk ID column 41C, as shown in Fig. 9. In the chunk group management table 41, one row corresponds to one chunk group 38.
[0070] The chunk group ID column 41A stores an identifier (chunk group ID) unique to the corresponding chunk group 38, and the data protection policy column 41B stores a data protection policy set for the chunk group 38. Data protection policies include "mirroring" and "second method of EC" that store the same data. In these methods, user data is stored in the storage server 7 in the own data center 2, and therefore user data can be read without performing inter-availability communication, resulting in high read performance and low network load.
[0071] 9, the data protection policy of the chunk group 38 assigned a chunk group ID of "0" is "mirroring", and it is shown that the chunk group 38 is made up of a physical chunk 37 assigned a physical chunk ID of "100", a physical chunk 37 assigned a physical chunk ID of "200", and a physical chunk 37 assigned a physical chunk ID of "300". In other words, data is stored in the own data center 2, and mirror data is transferred to and stored in the other data center 2.
[0072] These storage controller management table 40 and chunk group management table 41 are updated by the control plane 32 of the storage controller 30 that holds the storage controller management table 40 or chunk group management table 41, for example, when a failover occurs in one of the redundancy groups 36 and the configuration of the redundancy group 36 changes, or when a new network drive 8 is logically connected to the storage server 7.
[0073] 10 shows a method of controlling access from an application 33 implemented in a host server 9 to a host volume HVOL in a storage server 7. In this storage system 10, a host volume HVOL is created in the storage server 7 in which the storage controller 30 is implemented, in association with each of the storage controllers 30 constituting a redundancy group 36. In addition, these host volumes HVOL are provided to an application 33 implemented in the host server 9 as the same host volume HVOL. In the following, a collection of host volumes HVOLs created in association with each of the storage controllers 30 constituting a redundancy group 36 is referred to as a host volume group 50.
[0074] Then, based on information notified from the storage controller 30 associated with each host volume HVOL in each storage server 7 when logging in to that host volume HVOL, the application 33 sets, among the paths 51 to each provided host volume HVOL, the path 51 to the host volume HVOL associated with the storage controller 30 set to active mode in the corresponding redundancy group 36 as an optimized path as a path 51 to be used for accessing user data, and sets the paths 51 to the other host volumes HVOL as non-optimized paths. Furthermore, the application 33 always accesses user data via the optimized path. Therefore, the access from the application 33 to the host volume HVOL is always made to the host volume HVOL associated with the storage controller 30 set to active mode.
[0075] In this case, the storage controller 30 set to the active mode stores the user data in a physical storage area provided by the network drive 8 (FIG. 1) in the same data center 2 connected to the storage server 7 in which the storage controller 30 is implemented as described above, so that the user data always exists in the same data center 2 as the application 33. As a result, no data transfer between data centers 2 occurs when the application 33 accesses the user data, and it is possible to avoid a decrease in I / O performance and an increase in communication costs due to such data transfer.
[0076] 11 shows a host volume management table 52 used to manage the host volumes HVOL created in the storage server 7 as described above. This host volume management table 52 is a table used to manage the location of a host volume HVOL (hereinafter referred to as an owner host volume) HVOL associated with a storage controller 30 set to active mode, among multiple host volumes HVOL provided as the same host volume HVOL to an application 33 implemented in the host server 9.
[0077] In practice, the host volume management table 52 is configured to include a host volume (HVOL) ID column 52A, an owner data center ID column 52B, an owner server ID column 52C, and a size column 52D. In the host volume management table 52, one row corresponds to one owner host volume HVOL provided to the application 33 implemented in the host server 9.
[0078] The host volume ID column 52A stores the volume ID of a host volume (including an owner host volume) HVOL provided to the application 33 implemented in the host server 9, and the size column 52D stores the volume size of that host volume HVOL. The owner data center ID column 52B stores the data center ID of the data center (owner data center) 2 in which the owner host volume HVOL of that host volume HVOL exists, and the owner server ID column 52C stores the server ID of the storage server (owner server) 7 in which that owner host volume HVOL was created.
[0079] Therefore, in the example of Figure 11, the size of the host volume HVOL recognized by application 33 with a host volume ID of "1" is "100 GB", and it is shown that the owner host volume HVOL has been created in a storage server 7 with a server ID of "100" in a data center 2 with a data center ID of "1".
[0080] (1-2) Flow of failover when a failure occurs Next, a description will be given of the flow of failover processing executed when a failure occurs at a data center in the storage system 10 of this embodiment. Fig. 12 shows the state of failover executed when a failure occurs at a data center in any of the data centers 2 (here, the first data center 2A) from the normal state shown in Fig. 5.
[0081] In this storage system 10, the control plane 32 (FIG. 3) of each storage controller 30 exchanges heartbeat signals at a predetermined cycle with the control planes 32 of the other storage controllers 30 that constitute the same redundancy group 36 as the storage controller 30, thereby monitoring the liveness of each storage server 7 in which the other storage controllers 30 are implemented. If the control plane 32 fails to receive a heartbeat signal from the control plane 32 of the storage server 7 it is monitoring for a certain period of time, it determines that a failure has occurred in that storage server 7, and blocks that storage server 7 (hereinafter, referred to as the failed storage server) 7.
[0082] Furthermore, if the blocked failed storage server 7 has a storage controller 30 in active mode in any of the redundancy groups 36, the operating mode of the storage controller 30 with the next highest priority in that redundancy group 36 after that storage controller 30 is switched to active mode, and the processing performed by the original active mode storage controller 30 (hereinafter referred to as the original active storage controller) 30 is taken over by the storage controller newly set to active mode (hereinafter referred to as the new active storage controller) 30.
[0083] For example, in the redundancy group 36 shown at the left end of Fig. 12 and the fourth redundancy group 36 from the left, the storage controller 30 arranged in the first data center 2A where the failure occurred was in active mode, so an example is shown in which the operation mode of the storage controllers 30 (each storage controller 30 shown by diagonal lines in Fig. 12) in the second data center 2B constituting the same redundancy group 36 is switched to active mode. Therefore, in this case, the I / O processing that had been executed by the original active storage controller 30 implemented in the blocked storage server 7 is taken over and executed by the new active storage controller 30 in the second data center 2B.
[0084] Therefore, when the data protection policy applied to the physical chunk 37 in which the user data was stored is the second EC method described above, the new active storage controller 30 that took over the processing of the former active storage controller 30 restores the user data using data, parity, etc. present in the remaining data centers 2B, 2C in which no failure has occurred. In addition, the new active storage controller 30 stores the restored user data in the physical chunk 37 (FIG. 6) associated with the host volume HVOL in its own storage server 7 that constitutes the same host volume group 50 (FIG. 10) as the host volume (hereinafter, referred to as the failed host volume) HVOL in the failed storage server in which the user data was originally stored. When the data protection policy is mirroring, the mirror data is used as user data. When the data centers 2 are different, the mirror data that has become user data is moved to the same data center 2 as the new active storage controller 30.
[0085] Furthermore, if the management server 4 detects that a data center-wide failure or a storage server 7-wide failure has occurred in any of the data centers 2, as shown in FIG. 13, it starts an application 33 that is the same as the application 33 (hereinafter referred to as the failed application 33) of the host server 9 that was reading / writing user data to the host volume (failed host volume) HVOL in the storage server (failed storage server) 7 where the failure occurred, on the host server 9 in the data center 2 in which the new active storage controller 30 is located, and has that application 33 take over the processing that the failed application 33 had been performing until then.
[0086] Then, from the application 33 that has taken over the processing of the failed application 33, the path 51 to the host volume HVOL associated with the new active storage controller 30 is set to an optimized path, and the paths 51 to the other host volumes HVOLs are set to non-optimized paths. This allows the application 33 that has taken over the processing of the failed application 33 to access the restored user data.
[0087] In this manner, in this storage system 10, in order to enable the new active storage controller 30 that took over the processing of the original active storage controller 30 when a failure occurs to start up the same application 33 as the failed application 33 in the same data center 2 as the new active storage controller 30 and for that application 33 to continue processing, within a group made up of host servers 9 in each data center 2 (hereinafter referred to as a host server group), each host server 9 holds the same application 33 and the information required for that application 33 to execute processing (hereinafter referred to as application meta information).
[0088] In the host server group, when application meta information of any application 33 implemented in any host server 9 is updated, the difference between the application meta information before and after the update is transferred as differential data to other host servers 9 belonging to the host server group. Furthermore, when the differential data is transferred to the other host servers 9, the other host servers 9 update the application meta information held by the host server 9 based on the differential data. This ensures that the contents of the application meta information held by each host server 9 constituting the same host server group are always kept the same.
[0089] In this way, each host server 9 constituting a host server group always holds the same application meta-information. Therefore, even if a host server 9 or storage server 7 in one of the data centers 2 becomes inoperable due to a failure, the processing that had been performed by the application 33 implemented in that host server 9 can be immediately taken over by the same application 33 implemented in a host server 9 in another data center 2.
[0090] FIG. 14 shows the flow of server failure recovery processing executed when the control plane 32 of a storage controller 30 detects a failure (including a data center-level failure) of a storage server 7 that is implemented with another storage controller 30 that constitutes the same redundancy group 36 as the storage controller 30.
[0091] If the control plane 32 fails to receive a heartbeat signal from the control plane 32 of another storage controller 30 that is in the same redundancy group 36 as the own storage controller 30 for a certain period of time, it initiates the server failure recovery process shown in Figure 14.
[0092] Then, the control plane 32 first executes a blocking process to block the storage server 7 in which the storage controller 30 from which a heartbeat signal has not been received for a certain period of time (hereinafter, this will be referred to as a failed storage controller 30) is implemented (S1). This blocking process also includes, for example, a process of updating the storage configuration management table 35 described above with reference to FIG. 4.
[0093] Next, the control plane 32 refers to the storage controller management table 40 (Figure 8) to determine whether the failed storage controller 30 is a storage controller set to active mode in the redundancy group 36 to which the own storage controller 30 belongs (S2).
[0094] If the control plane 32 obtains a positive result in this determination, it determines, based on the metadata it manages, whether or not its own storage controller 30 has the next highest priority after the failed storage controller 30 in the redundancy group 36 to which its own storage controller 30 belongs (S3).
[0095] If the control plane 32 obtains a positive result in this determination, it executes a failover process (S4) for making its own storage controller 30 take over the process that had been performed by the failed storage controller 30. This failover process includes switching the operation mode of its own storage controller 30 to active mode, notifying storage controllers 30 other than the failed storage controller 30 in the same redundancy group 36 that its own storage controller 30 has entered active mode, and updating necessary metadata including the storage controller management table 40 described above with reference to Fig. 8, the chunk group management table 41 described above with reference to Fig. 9, and the host volume management table 52 described above with reference to Fig. 11.
[0096] Next, the control plane 32 sets the path to the host volume (hereinafter referred to as the failover destination host volume) HVOL in the storage server 7 in which its own storage controller 30 is implemented, which constitutes the same host volume group 50 as the host volume (hereinafter referred to as the failed host volume) HVOL associated with the failed storage controller 30, as an optimized path (S5).
[0097] As a result, when the same application 33 that was reading / writing data to the failed host volume HVOL in the data center 2 where the failure occurred is started by the management server 4 in the data center 2 in which the control plane 32 exists and the application 33 logs in to its own storage controller 30, the control plane 32 notifies the application 33 to set the path to the corresponding host volume HVOL in its own storage controller 30 to the optimized ("Optimized") path. In response to this notification, the application 33 sets the path to the host volume HVOL to the optimized ("Optimized") path. This ends the server failure recovery process.
[0098] On the other hand, if the control plane 32 obtains a negative result in step S2, it notifies the storage controller 30 with the highest priority among the redundancy group 36 to which the storage controller 30 belongs (here, the storage controller 30 in active mode) that the storage server 7 in which the failed storage controller 30 is implemented has been blocked (S6).
[0099] As a result, the storage controller 30 that received this notification executes a predetermined process such as updating the necessary metadata including the storage controller management table 40 described above with reference to Fig. 8 in accordance with the contents of this notification. This completes the server failure recovery process.
[0100] Furthermore, if the control plane 32 obtains a negative result in step S3, it notifies the storage controller 30 with the next highest priority after the failed storage controller 30 in the redundancy group 36 to which the storage controller 30 belongs that the storage server 7 in which the failed storage controller 30 is implemented has been blocked (S6).
[0101] As a result, the control plane 32 of the storage controller 30 that received this notification executes the same processes as steps S4 and S5, and then this server failure recovery process ends.
[0102] (1-3) Flow of creating a host volume Next, a flow of operations until a user creates an owner host volume HVOL of a desired volume size in a desired data center 2 will be described.
[0103] 15 shows an example of the configuration of a host volume creation screen 60 that can be displayed on the user terminal 6 (FIG. 1) by a predetermined operation. This host volume creation screen 60 is a screen for the user to create a host volume (owner host volume) HVOL to be associated with the storage controller 30 in active mode, among the host volumes HVOL provided to the application 33 implemented in the host server 9.
[0104] This host volume creation screen 60 comprises a volume number designation field 61 , a volume size designation field 62 , a destination data center designation field 63 , and an OK button 64 .
[0105] On the host volume creation screen 60, the user can operate the user terminal 6 to specify the volume ID (here, a number) of the owner host volume HVOL to be created by inputting it into a volume number designation field 61, and can specify the volume size of that owner host volume HVOL by inputting it into a volume size designation field 62.
[0106] Furthermore, on the host volume creation screen 60, a pull-down menu 66 listing the data center IDs of each data center 2 can be displayed by clicking on a pull-down menu 65 provided to the right of the destination data center specification field 63.
[0107] The user can then select by clicking the data center ID of a desired data center 2 from the data center IDs displayed in this pull-down menu 66, and specify that data center 2 as the data center 2 in which the owner host volume HVOL is to be created. At this time, the data center ID of the selected data center 2 is displayed in the destination data center designation field 63.
[0108] Then, on the host volume creation screen 60, by specifying the volume ID, volume size, and destination data center 2 of the owner host volume HVOL as described above and then clicking the OK button 64, the user can instruct the management server 4 to create an owner host volume HVOL of that volume ID and volume size in that data center 2.
[0109] In practice, when the OK button 64 on the host volume creation screen 60 is clicked, a volume creation request including the volume ID, volume size, and information on the destination data center 2 specified by the user on the host volume creation screen 60 at that time is created on the user terminal 6 that is displaying the host volume creation screen 60, and the created volume creation request is sent to the management server 4 (Figure 1).
[0110] When such a volume creation request is given, the management server 4 creates an owner host volume HVOL of the requested volume ID and volume size in one of the storage servers 7 in the specified data center 2, following the processing procedure shown in Figure 16.
[0111] In practice, when such a volume creation request is received, the management server 4 initiates the host volume creation process shown in FIG. 16, and first determines (S10) whether or not there is a storage server 7 in the data center 2 specified in the volume creation request as the destination for creating the owner host volume HVOL (hereinafter referred to as the specified data center 2) that has the capacity to create an owner host volume HVOL of the volume size specified by the user.
[0112] Specifically, the management server 4 inquires of any one of the storage controllers 30 implemented in each storage server 7 in the specified data center 2 about the capacity and current capacity used of that storage server 7. Then, based on the capacity and current capacity used of each of the storage servers 7 notified by the control planes 32 of the storage controllers 30 in response to this inquiry, the management server 4 determines whether or not it is possible to create an owner host volume HVOL of the specified volume size.
[0113] If the management server 4 obtains a positive result in this determination, it then determines in the same manner as in the case of the owner host server HVOL described above whether each of the storage servers 7 in other data centers 2 that are equipped with other storage controllers 30 that constitute the same redundancy group 36 as the storage controller 30 (e.g., an existing storage controller 30 or a newly created storage controller 30) that is to be associated with the owner host volume HVOL in the storage server 7 that is capable of creating the owner host volume HVOL is capable of creating a host volume HVOL of the specified volume size (S11).
[0114] If the management server 4 obtains a positive result in this determination, it selects one storage server 7 from among the storage servers 7 that were determined in step S10 to be capable of creating an owner host volume HVOL and from among the storage servers 7 that also obtained a positive result in step S11, and issues an instruction to create the owner host volume HVOL to the storage controller 30 in that storage server 7 that corresponds to the owner host volume HVOL (S15). As a result, the storage controller 30 creates an owner host volume HVOL of the specified volume size in that storage server 7, in association with that storage controller 30. Also, the operation mode of that storage controller 30 is set to the active mode.
[0115] The management server 4 then issues an instruction to create a host volume HVOL of the same volume size as the owner host volume HVOL to each of the storage controllers 30 in the other data centers 2 constituting the same redundancy group 36 as the storage controller 30 (S16). As a result, these storage controllers 30 create host volumes HVOL of the same volume size as the owner host volume HVOL in association with each of the storage controllers 30, respectively, in the same storage server 4 as the storage controllers 30. The operation mode of these storage controllers 30 is also set to standby mode.
[0116] In the above-mentioned steps S15 and S16, the association of each host volume HVOL (including the owner host volume HVOL) newly created in each data center 2 with the storage controller 30, and the setting of the operation mode (active mode or standby mode) of the storage controller 30 associated with each of these host volumes HVOL may be manually performed by an administrator or user of the storage system 10. The same applies hereinafter.
[0117] On the other hand, if the management server 4 obtains a negative result in the judgment of step S10 or step S11, it determines whether or not there is any storage server 7 among the storage servers 7 in the specified data center 2 that can expand its capacity to the point where the specified host volume HVOL can be created (S12).
[0118] Specifically, the management server 4 inquires of any storage controller 30 implemented in any storage server 7 in the specified data center 2 about the number of network drives 8 (FIG. 1) logically connected to each storage server 7 in the specified data center 2. Since the number of network drives 8 that can be logically connected to a storage server 7 is fixed, this is to confirm whether or not it is possible to expand the capacity by connecting an additional network drive 8 to each storage server 7.
[0119] The management server 4 also queries the storage controller 30 about the number of network drives 8 that are arranged in the designated data center 2 and that are not logically connected to any storage server 7, and the capacity of these network drives 8. Then, based on each piece of information obtained as described above, the management server 4 determines whether or not there is a storage server 7 in the designated data center 2 that can expand the capacity to the point where a designated host volume HVOL can be created by additionally connecting a network drive 8 in the designated data center 2.
[0120] At this time, if a certain storage server 7 is expandable, the management server 4 judges whether or not the same capacity can be expanded for storage servers 7 in other data centers 2 in which the storage controller 30 to which the owner host volume HVOL is to be associated in that storage server 7 and the other storage controllers 30 constituting the redundancy group 36 are respectively implemented. This is because it is necessary to create host volumes HVOLs of the same volume size as the owner storage controller 30 for these storage servers 7 as well.
[0121] If the management server 4 obtains a negative result in the determination in step S12, it sends an error notification to the user terminal 6 that sent the volume creation request (S13), and then terminates this volume creation process. As a result, based on the error notification, a warning to the effect that the designated host volume HVOL cannot be created is displayed on the user terminal 6.
[0122] In response to this, if the management server 4 obtains a positive result in the judgment in step S12, it selects one storage server 7 from among the storage servers 7 in the specified data center 2 that have been determined in step S12 to be capable of expanding capacity (including capacity expansion of corresponding storage servers 7 in other data centers 2), and executes a server capacity expansion process to expand the capacity of the selected storage server 7 by additionally logically connecting a network drive 8 to the selected storage server 7 (hereinafter referred to as the selected storage server 7) (S14).
[0123] The management server 4 also issues an instruction to create an owner host volume HVOL to the storage controller 30 to be associated with the owner host volume HVOL in the selected storage server 7 whose capacity has been expanded (S15). As a result, an owner host volume HVOL of the volume size specified by the storage controller 30 is created in the same storage server 7 as the storage controller 30, in association with the storage controller 30. Also, the operation mode of the storage controller 30 is set to the active mode.
[0124] Furthermore, the management server 4 then instructs each storage controller 30 in the other data centers 2 constituting the same redundancy group 36 as the storage controller 30 to create a host volume HVOL of the same volume size as the owner host volume HVOL (S16). As a result, these storage controllers 30 create host volumes HVOL of the same volume size as the owner host volume HVOL in association with each storage controller 30, respectively, in the same storage server 7 as the storage controllers 30. Furthermore, the operation mode of these storage controllers 30 is set to standby mode.
[0125] Then, the management server 4 thereafter ends this host volume creation process.
[0126] The flow of the server capacity expansion process executed by the management server 4 in step S14 of this host volume creation process is shown in FIG.
[0127] When the management server 4 proceeds to step S14 of FIG. 16, it starts the server capacity expansion process shown in FIG. 17, and first determines (S20) the expansion capacity of each storage server 7 in each data center 2 (hereinafter, these storage servers 7 are referred to as storage servers 7 to be expanded) in which each storage controller 30 to be associated with each host volume HVOL (including the owner host volume HVOL) that constitutes the host volume group 50 (FIG. 10) to which the above-mentioned owner host volume HVOL belongs.
[0128] Next, the management server 4 determines the network drives 8 to be logically connected to each of the storage servers 7 to be expanded in capacity so that the capacity of each storage server 7 to be expanded in capacity can be expanded equally (S21), and logically connects the determined network drives 8 to the corresponding storage servers 7 to be expanded in capacity (S22).
[0129] Specifically, the management server 4 notifies the storage controllers 30 of each data center 2, which correspond to each host volume HVOL constituting the host volume group, that the network drive 8 has been logically connected. The management server 4 also instructs the active mode storage controllers 30 of each redundancy group 36 in the storage system 10 to update the network drive ID column 35C corresponding to the storage server 7 to be subject to capacity expansion in the storage configuration management table 35 described above with reference to Fig. 4 to add the network drive ID of the network drive 8 logically connected at that time.
[0130] Next, the management server 4 creates chunk groups 38 (FIG. 6) between storage areas provided by each network drive 8 connected to each storage server 7 to be expanded, and issues an instruction to each of the active mode storage controllers 30 of each redundancy group 36 in the storage system 10 to update the created chunk groups 38 to the state registered in the chunk group management table 41 described above with reference to FIG. 9 (S23). Then, the management server 4 thereafter ends this server capacity expansion process and proceeds to step S15 of the host volume creation process.
[0131] (1-4) Server usage capacity monitoring process flow On the other hand, Figure 18 shows the flow of a server usage capacity monitoring process that is periodically executed by the control plane (hereinafter referred to as a specific control plane) 32 of a specific storage controller 30 implemented in any of the storage servers 7 in each data center 2.
[0132] The specific control plane 32 monitors the usage capacity of each storage server 7 in the data center (hereinafter referred to as its own data center) 2 in which its own storage controller 30 resides according to the processing procedure shown in FIG. 18, and when the usage capacity of any storage server 7 exceeds a preset threshold (hereinafter referred to as the usage capacity threshold), executes processing to expand the capacity of that storage server 7.
[0133] In practice, when the specific control plane 32 starts the server capacity monitoring process shown in Fig. 18, it first obtains the capacity and current usage capacity of each storage server 7 from any one of the storage controllers 30 implemented in that storage server 7 in its own data center 2. At this time, it also obtains the capacity and current usage capacity of the storage server 7 in which its own storage controller 30 is implemented (S30).
[0134] Next, the specific control plane 32 judges whether or not the usage capacity of any of the storage servers 7 in its own data center 2 has exceeded the above-mentioned usage capacity threshold based on the acquired information (S31). If the specific control plane 32 obtains a negative result in this judgment, it ends this storage server usage capacity monitoring process.
[0135] In contrast, if the specific control plane 32 obtains a negative result in the judgment of step S31, it judges (S32) whether the storage server 7 whose usage capacity has exceeded the usage capacity threshold (hereinafter referred to as the overused storage server 7) is expandable in the same manner as step S12 of the host volume creation process described above with reference to Figure 16.
[0136] If the specific control plane 32 obtains a positive result in this determination, it executes processing similar to the server capacity expansion processing described above with reference to FIG. 17 to expand the capacity of the overutilized storage server 7, the capacity of the storage controller 30 implemented in the overutilized storage server 7, and the capacity of the storage servers 7 in which other storage controllers 30 that constitute the redundancy group 36 are implemented (S33), and then terminates this server utilization capacity monitoring processing.
[0137] In response to this, if the specific control plane 32 obtains a negative result in the judgment of step S32, it executes a host volume movement process to move the host volume HVOL to a storage server 7 in the same or a different data center 2 as the overutilized storage server 7 and that has free capacity to which the host volume HVOL in the overutilized storage server 7 can be moved (S34), and then terminates this server utilization capacity monitoring process.
[0138] The specific processing contents of the host volume migration process are shown in Fig. 19. When the specific control plane 32 proceeds to step S34 of the server usage capacity monitoring process, it starts the host volume migration process shown in Fig. 19.
[0139] Then, based on the capacity and current usage capacity of each storage server 7 in its own data center 2 obtained in step S30 of the server usage capacity monitoring process, the specific control plane 32 first selects one storage server 7 from among the storage servers 7 in its own data center 2 that has enough free capacity to move any of the host volumes HVOL in the overused storage server 7 (S40).
[0140] Then, the specific control plane 32 judges whether or not such a storage server 7 has been selected in this step S40 (S41), and if so, proceeds to step S43.
[0141] On the other hand, if the specific control plane 32 obtains a negative result in the determination in step S41, it acquires the capacity and currently used capacity of each storage server 7 in the data center 2 by inquiring of the control plane 32 of any storage controller 30 of any storage server 7 in each data center 2 other than its own data center 2. Then, based on this acquired information, the management server 4 selects a storage server 7 having free capacity to which any host volume HVOL in the overused storage server 7 can be moved from among the storage servers 7 in the data center 2 other than its own data center 2 (S42).
[0142] Next, the specific control plane 32 selects a host volume HVOL to be moved to another storage server 7 from among the host volumes HVOL in the overutilized storage server 7 (hereinafter referred to as the target host volume for movement), and copies the data of the selected target host volume HVOL to the storage server 7 selected in step S40 or step S42 (S43).
[0143] Specifically, the specific control plane 32 first creates a host volume HVOL in the storage server 7 that is the migration destination of the migration target host volume HVOL, and associates the created host volume HVOL with any of the active mode storage controllers 30 implemented in the storage server 7. Then, the specific control plane 32 copies data of the migration target host volume HVOL to this host volume HVOL.
[0144] Furthermore, the specific control plane 32 associates this storage controller 30 with other storage controllers (hereinafter referred to as associated storage controllers) 30 in other data centers 2 that constitute the redundancy group 36, and creates host volumes HVOLs that constitute a host volume group 50 (Figure 10) together with the host volume HVOL to which the data of the host volume HVOL to be migrated is copied, in the storage server 7 in which the associated storage controller 30 is implemented.
[0145] The specific control plane 32 then associates each of these created host volumes HVOL with an associated storage controller 30 in the same storage server 7 .
[0146] Next, the specific control plane 32 sets the path from the application 33 that had been reading / writing user data from the migration target host volume HVOL until then to the host volume HVOL to which the data of the migration target host volume HVOL has been copied (hereinafter referred to as the data copy destination host volume) to an optimized ("Optimized") path (S44).
[0147] As a result, thereafter, when the application 33 logs in to the data copy destination host volume HVOL, a notification is given to the application 33 to the effect that the path should be set as the optimized path, and based on this notification, the application 33 sets the path as the optimized path and sets the other paths as non-optimized paths. The management server then terminates this volume migration process.
[0148] Note that, apart from capacity, migration processing of host volumes HVOL may be performed between storage servers 7 within the data center 2 for the purpose of rebalancing the load on the volumes.
[0149] (1-5) Effects of this embodiment According to the storage system 10 of this embodiment having the above configuration, it is possible to store redundant data in another data center 2 (another availability zone) while ensuring data locality, so that even if a failure occurs on a data center basis (availability zone basis) in the data center 2 in which the storage controller 30 in active mode is located, the processing that had been performed by that storage controller 30 up until that point can be taken over by the storage controller 30 set to standby mode that constitutes the same redundancy group 36. Thus, according to this embodiment, it is possible to realize a highly available storage system 10 that can withstand failures on an availability zone basis.
[0150] Furthermore, according to the present storage controller 30, the application 33 and the user data used by the application 33 can always exist in the same availability zone, so that it is possible to prevent communication across availability zones when the storage controller 30 in active mode processes an I / O request from the application 33. Therefore, according to the present storage system 10, it is possible to prevent a decrease in I / O performance caused by communication delays associated with communication between availability zones and the occurrence of costs due to communication between bases.
[0151] Furthermore, according to the present storage system 10, even if a failure occurs on a data center basis, not only does the storage controller 30 fail over, but the application 33 and user data are also moved to the failover destination data center 2, making it possible to realize the construction of a highly available system that can withstand failures on an availability zone basis. Although communication between the data centers 2 is necessary during normal operation for failover, the present storage system 10 is designed to reduce the amount of communication.
[0152] (2) Second embodiment 20, in which parts corresponding to those in Fig. 1 are given the same reference numerals, shows a cloud system 70 according to the second embodiment. This cloud system 70 is configured to include first to third data centers 71A, 71B, and 71C that are installed in different availability zones.
[0153] These first to third data centers 71A to 71C are connected to one another via a dedicated network 3. A management server 72 is also connected to the dedicated network 3, and a storage system 73 is configured by the first to third data centers 71A to 71C and the management server 72. In the following, when there is no need to particularly distinguish between the first to third data centers 71A to 71C, they will be collectively referred to as the data center 71.
[0154] Each of the first and second data centers 71A and 71B has a plurality of storage servers 74 constituting a distributed storage system and a plurality of network drives 8. The third data center 71C does not have a network drive 8, but has at least one storage server 75. The hardware configuration of these storage servers 74 and 75 is similar to that of the storage server 4 of the first embodiment described above with reference to FIG. 2, and therefore a description thereof will be omitted here.
[0155] Fig. 21, in which the same reference numerals are used for parts corresponding to those in Fig. 3, shows the logical configuration of storage servers 74, 75 respectively arranged in each data center 71. As shown in Fig. 21, each storage server 74 arranged in the first and second data centers 71A, 71B has the same logical configuration as the storage server 7 of the first embodiment.
[0156] In practice, the storage server 74 is configured to include one or more storage controllers 76 having a data plane 77 and a control plane 78. The data plane 77 is a functional unit having a function of reading / writing user data from / to the network drive 8 via the intra-datacenter network 34 in response to an I / O request from an application 33 implemented in the host server 9. The control plane 78 is a functional unit having a function of managing the configuration of the storage system 73 (FIG. 20).
[0157] The operations of the data plane 77 and the control plane 78 are similar to those executed by the storage controllers 30 installed in the storage servers 7 in the remaining two data centers 2 when a data center-based failure occurs in one data center 2 in the storage system 10 of the first embodiment, and therefore will not be described here. Note that redundancy of user data in this embodiment is always achieved by mirroring.
[0158] On the other hand, the storage server 75 arranged in the third data center 71C is configured to include one or more storage controllers 79 having only a control plane 80. For this reason, in the storage system 73 of this embodiment, the storage server 75 in the third data center 71C cannot perform I / O processing of user data. For this reason, neither the host server 9 nor the network drive 8 exists in the third data center 71C, and no host volume HVOL is created. In other words, in the case of this storage system 73, user data cannot be held in the third data center 71C.
[0159] The control plane 80 of the storage controller 79 has the function of monitoring the health of the storage servers 74 in the first and second data centers 71A, 71B by exchanging heartbeat signals with the control planes 78 of the storage controllers 76 that constitute the same redundancy group 36 (Figure 5) implemented in the storage servers 74 in the first and second data centers 71A, 71B.
[0160] FIG. 22 shows the processing steps of a host volume creation process executed by the management server 72 of this embodiment in the storage system 73 of this embodiment instead of the host volume creation process of the first embodiment described above with reference to FIG.
[0161] In this storage system 73, the user also uses the host volume creation screen 60 described above in FIG. 15 to specify the volume ID, volume size, and destination data center 71 of the owner host volume HVOL to be created, as described above, and then clicks the OK button 64 to instruct the management server 72 to create the owner host volume HVOL.
[0162] As a result, a volume creation request including the volume ID, volume size, and information on the destination data center 71 specified by the user is created on the user terminal 6 (Figure 20) that is displaying the host volume creation screen 60, and the created volume creation request is sent to the management server 72.
[0163] When such a volume creation request is given, the management server 72 creates an owner host volume HVOL of the requested volume ID and volume size in one of the storage servers 74 in the data center (designated data center) 71 specified in the volume creation request as the destination for the owner host volume HVOL, in accordance with the processing procedure shown in FIG.
[0164] Specifically, when such a volume creation request is given, the management server 72 initiates the host volume creation process shown in FIG. 22, and first determines whether the specified data center 71 in the volume creation request is a data center 71 that can hold user data (S50).
[0165] For example, by querying the control plane 78, 80 of one of the storage controllers 76 in the data center (designated data center) 71 designated in the volume creation request as the destination for creating the owner host volume HVOL, regarding the number of network drives 8 logically connected to each of the storage servers 74, 75 in the designated data center 71, it is possible to determine whether the designated data center 71 is a data center capable of holding data.
[0166] If the management server 72 obtains a negative result in this determination, it sends an error notification to the user terminal 6 that sent the volume creation request (S54), and then terminates this host volume creation process. As a result, a warning is displayed on the user terminal 6 to the effect that the host volume HVOL cannot be created in the data center 71 specified by the user.
[0167] On the other hand, if the management server 72 obtains a positive result in the determination at step S50, it executes the processes of steps S51 to S57 in the same manner as steps S10 to S16 of the host volume creation process of the first embodiment described above with reference to Fig. 16. As a result, a host volume HVOL with the host volume ID and volume size specified by the user is created in any storage server 7 or the like in the data center 71 specified by the user. Then, the management server 72 thereafter terminates this host volume creation process.
[0168] According to the storage system 73 of this embodiment having the above configuration, it is possible to obtain the same effects as the storage system 10 of the first embodiment even when performing I / O processing of user data in two data centers 2.
[0169] (3) Other embodiments In the above embodiment, the user interface presentation device that presents the host volume creation screen 60 described above in relation to Figure 15, which is a user interface for specifying an availability zone in which to create a host volume HVOL to be associated with the active storage controller 30, is described as being the user terminal 6. However, the present invention is not limited to this, and such a host volume creation screen 60 may be displayed on the management server 4, 72, and an administrator may display the host volume creation screen 60 in response to a request from a user.
[0170] In the above embodiment, the storage controller 30 in each data center 2 is used as a capacity monitoring unit that monitors the usage capacity of each storage server 7, 74 in the data center 2, but the present invention is not limited to this, and a capacity monitoring device having the function of such a capacity monitoring unit may be substituted by the monitoring servers 4, 72, or such a capacity monitoring device may be provided in each data center 2 separately from the storage server 7. Also, the storage controller 30 or such a capacity monitoring device may monitor the remaining capacity of each storage server 7, 74 instead of monitoring the usage capacity of each storage server 7, 74 in the data center 2. [Industrial Applicability]
[0171] The present invention relates to an information processing system, and can be widely applied to a distributed storage system made up of a plurality of storage servers arranged in different availability zones. [Explanation of symbols]
[0172] 1,70...Cloud system, 2,2A~2C,71,71A~71C...Data center, 4,72...Management server, 6...User terminal, 7,74,75...Storage server, 8...Network drive, 9...Host server, 10,73...Storage system, 30,76...Storage controller, 31,77...Data plane, 32,78...Control plane, 33...Application, 36...Redundancy group, 37...Physical chunk, 38...Chunk group, 50...Host volume group, 51...Path, 60...Host volume creation screen, HVOL...Host volume.
Claims
1. In an information processing system having a plurality of storage servers arranged at each of a plurality of bases connected to a network, A storage device that is arranged at each of the bases and stores data; a storage controller implemented in the storage server, providing a logical volume to a higher-level application and processing data read from and written to the storage device via the logical volume; a management server for managing the storage server; Equipped with forming a redundancy group including a plurality of the storage controllers arranged at different bases, the redundancy group including an active storage controller for processing data and a standby storage controller for taking over the processing of the data when a failure occurs in the active storage controller; The active storage controller includes: storing the data from the upper level application disposed at the same site in the storage device disposed at the site; execute a process for storing redundancy data for restoring data to be stored in a storage device at the same site in a storage device arranged at another site where a standby storage controller of the same redundancy group is arranged; The storage controller migrates the logical volume to another storage controller at the same site based on a predetermined condition, When a failure occurs in the site where the active storage controller is located, a standby storage controller located at another site and belonging to the same redundancy group as the active storage controller at the site where the failure occurred, changes to an active state and takes over the processing of the data; restoring the data stored in the storage device of the site where the failure occurred to the storage device of the site where the storage controller that has taken over the processing of the storage controller is located, by using redundant data stored in the storage device of the other site; The management server includes: At the base where the storage controller that has taken over the processing of the storage controller is located, an application that is the same as the upper application is started. An information processing system comprising:
2. Among paths from the upper level application to each of the logical volumes, the path associated with the storage controller in the active state is set as an optimized path for the upper level application to read and write data; When a failure occurs at the base and the processing of the active storage controller is taken over by another storage controller in the same redundancy group, a path from the upper application to the storage controller that has taken over the processing is set as a path for an application started to take over the upper application to read and write the data.
2. The information processing system according to claim 1 .
3. a user interface presenting unit presenting a user interface for specifying the base for creating the logical volume to be associated with the storage controller in the active state; 3. The information processing system according to claim 2.
4. the redundant data stored in the storage device located at the other base is mirror data or parity generated based on a plurality of data stored at different bases, The active controller comprises: Transferring the data to be stored in the storage device at the same site to another site where the redundant data is stored in order to generate the mirror data or the parity; By storing data related to the logical volume in a storage device in the same base as the logical volume, the data can be read without transferring data to any other base.
2. The information processing system according to claim 1 .
5. An information processing system having a plurality of storage servers each located at a plurality of bases connected by a network, A storage device that is arranged at each of the bases and stores data; a storage controller implemented in the storage server, providing a logical volume to a higher-level application and processing data read from and written to the storage device via the logical volume; a capacity monitoring unit for monitoring a used capacity or a remaining capacity of each of the storage servers in each of the bases; Equipped with forming a redundancy group including a plurality of the storage controllers arranged at different bases, the redundancy group including an active storage controller for processing data and a standby storage controller for taking over the processing of the data when a failure occurs in the active storage controller; The active storage controller includes: storing the data from the upper level application disposed at the same site in the storage device disposed at the site; execute a process for storing redundancy data for restoring data to be stored in a storage device at the same site in a storage device arranged at another site where a standby storage controller of the same redundancy group is arranged; The capacity monitoring unit is When the used capacity or the remaining capacity of any one of the storage servers reaches a predetermined condition, the capacity of each of the storage servers in which the storage controllers constituting the redundancy group to which the storage controller implemented in the storage server belongs is expanded; When the capacity of each of the storage servers in which the storage controllers constituting the redundancy group to which the storage controller implemented in the storage server belongs cannot be expanded, the logical volume provided by the storage controller of the storage server is moved to another storage server installed at the same base. An information processing system comprising:
6. 1. An information processing method executed in an information processing system having a plurality of storage servers arranged at each of a plurality of bases connected via a network, comprising: The information processing system includes: A storage device that is arranged at each of the bases and stores data; a storage controller implemented in the storage server, providing a logical volume to a higher-level application and processing data read from and written to the storage device via the logical volume; a management server for managing the storage server; having forming a redundancy group including a plurality of the storage controllers arranged at different bases, the redundancy group including an active storage controller for processing data and a standby storage controller for taking over the processing of the data when a failure occurs in the active storage controller; a first step in which the storage controller in an active state stores data from a higher-level application arranged at the same base in the storage device arranged at the base, and executes a process for storing redundancy data for restoring the data to be stored in the storage device at the same base in the storage device arranged at another base where a storage controller in a standby state of the same redundancy group is arranged; a second step in which, when the storage controller moves the logical volume to another storage controller at the same base based on a predetermined condition, while a failure occurs at the base where the active storage controller is located, a standby storage controller that belongs to the same redundancy group as the active storage controller at the base where the failure occurred and is located at the other base changes to an active state and takes over the processing of the data, the data stored in the storage device at the base where the failure occurred is restored to the storage device at the base where the storage controller that took over the processing of the storage controller is located by using redundant data stored in the storage device at the other base, and the management server starts up an application that is the same as the upper level application at the base where the storage controller that took over the processing of the storage controller is located; An information processing method comprising:
7. An information processing method executed in an information processing system having a plurality of storage servers arranged at each of a plurality of bases connected by a network, comprising: The information processing system includes: A storage device that is arranged at each of the bases and stores data; a storage controller implemented in the storage server, providing a logical volume to a higher-level application and processing data read from and written to the storage device via the logical volume; a capacity monitoring unit for monitoring a used capacity or a remaining capacity of each of the storage servers in each of the bases; having forming a redundancy group including a plurality of the storage controllers arranged at different bases, the redundancy group including an active storage controller for processing data and a standby storage controller for taking over the processing of the data when a failure occurs in the active storage controller; a first step in which the storage controller in an active state stores the data from a higher-level application arranged at the same base in the storage device arranged at the base, and executes a process of storing redundancy data for restoring the data to be stored in the storage device at the same base in the storage device arranged at another base where a storage controller in a standby state of the same redundancy group is arranged; a second step of the capacity monitoring unit expanding the capacity of each of the storage servers in which the storage controllers constituting the redundancy group to which the storage controller implemented in the storage server belongs when the used capacity or the remaining capacity of any of the storage servers reaches a predetermined condition, and migrating a logical volume provided by the storage controller of the storage server to another storage server installed at the same site when the capacity monitoring unit cannot expand the capacity of each of the storage servers in which the storage controllers constituting the redundancy group to which the storage controller implemented in the storage server belongs; An information processing method comprising:
Citation Information
Patent Citations
Redundancy method and device for data files in database
CN107402850A
Distributed storage system
JP2019071100A
Information processing system, managing method for information processing system, and program
JP2020021277A
Storage system
JP2020107082A
Storage system and method for controlling the same
JP2021036450A
