Computer system and storage control method
The computer system and storage control method address communication failures in cloud environments by pairing storage controllers and rerouting I/O to minimize inter-data center communication, thereby reducing costs and latency.
Patent Information
- Application Number
- JP2023078726
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-21
AI Technical Summary
In cloud computing environments, communication failures between storage systems across multiple data centers result in increased billing charges and response times due to the need for inter-data center communication, which is more costly and latency-prone compared to intra-data center communication.
A computer system and storage control method that pairs storage controllers across different data centers, collects communication characteristics, and uses statistical processing to reroute I/O operations to minimize inter-data center communication by stopping affected storage nodes and rerouting I/O to standby controllers.
Reduces billing charges and response times by minimizing inter-data center communication during failures, ensuring seamless operation with reduced costs and latency.
Smart Images

Figure 2024162814000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a computer system and a storage control method, and is suitable for application to a computer system including a storage system constructed across a plurality of data centers, and to the storage control method thereof. [Background technology]
[0002] Conventionally, in cloud computing environments, systems are constructed across multiple data centers so that IT services can continue even if a failure occurs at the data center level. Therefore, storage systems that operate on the cloud are constructed across multiple data centers and have a configuration that is resistant to data center failures.
[0003] In a storage system constructed by distributing multiple storages, if a communication failure occurs between storage nodes or storage clusters that constitute the storage system, the entity that detects the failure may not be able to distinguish whether a communication failure has occurred or whether a failure has occurred with the communication partner, and may not be able to implement appropriate failure control (blocking, failover, etc.). As a method for solving this problem, the failure classification can be determined and control can be carried out through a third-party intermediary.
[0004] For example, if a communication failure occurs between storage nodes, as described in Patent Document 1, an intermediary can be used to share state information between the storage nodes to determine whether or not to implement failure control.
[0005] Also, for example, if a communication failure occurs between storage clusters, failure control can be performed by transmitting a control command to the other cluster through an intermediary, as described in Patent Document 2 or Patent Document 3. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] Special Publication No. 2018-536229 [Patent Document 2] Patent Publication No. 2022-122993 [Patent Document 3] International Publication No. 2015 / 198449 Summary of the Invention [Problem to be solved by the invention]
[0007] Incidentally, in order to cope with communication failures within an on-premise storage system, there are measures that utilize the conventional technologies disclosed in Patent Documents 1 to 3 as described above to control the storage system when a communication failure occurs. However, in the case of a storage system in which multiple computers are connected via a network across multiple data centers (a storage system operating in a cloud environment), the following problems exist that cannot be solved by the conventional technologies:
[0008] In a cloud environment, inter-datacenter communications incur charges according to the amount of communication, and the communication latency is greater than that of intra-datacenter communications. Therefore, in general, the volumes provided by a storage system and the applications that use them are configured to be able to communicate within the same datacenter. If a communication failure occurs in a storage system configured in this way, the volumes may move across datacenters by stopping part of the storage system through failure control. As a result, if the volumes and applications are configured to span datacenters after failure control, inter-datacenter communications will occur for each I / O, which can result in issues such as charges being incurred and increased response times.
[0009] The present invention has been made in consideration of the above points, and aims to propose a computer system and a storage control method that can perform fault control to suppress charges associated with I / O processing and increases in responses when a specified communication fault occurs in a storage system built across multiple data centers. [Means for solving the problem]
[0010] In order to solve the above problem, the present invention provides a computer system that is configured by connecting a plurality of computers arranged in a plurality of data centers via a network, and includes a compute node that executes an application, a storage node having a storage controller that processes I / O from the compute node and inputs / outputs data to / from a storage device, and a storage cluster controller that controls the storage node, in which storage controllers arranged in different data centers form a pair, and communicate with each other via a communication path between the data centers so that one storage controller of the pair can take over the processing of the other storage controller, and the communication characteristics between the application and the storage controllers are collected, and predetermined statistical processing is performed to generate statistical information. A computer system is provided which has an I / O monitor, and when a communication failure occurs in a communication path between the paired storage controllers, a tiebreaker in the storage cluster controller determines failure control to take over data input / output from one of the paired storage controllers to the other storage controller based on statistical information on the communication characteristics of the storage controllers generated by the I / O monitor, and to stop a storage node having one of the storage controllers, and the storage cluster controller executes the failure control determined by the tiebreaker, and an application which was performing I / O to the storage controller of the stopped storage node sends I / O to the storage controller which has taken over the processing.
[0011] In order to solve the above problem, the present invention provides a storage control method using a computer system configured by connecting a plurality of computers arranged in a plurality of data centers via a network, the computer system having a compute node that executes an application, a storage node having a storage controller that processes I / O from the compute node and inputs / outputs data to / from a storage device, and a storage cluster controller that controls the storage node, the storage controllers arranged in different data centers form a pair, and communication is performed via a communication path between the data centers so that one storage controller of the pair can take over the processing of the other storage controller, and the communication characteristics between the application and the storage controllers are collected and predetermined statistical processing is performed. and an I / O monitor which implements a communication characteristic analysis and generates statistical information, and when a communication failure occurs in a communication path between the paired storage controllers, a tiebreaker in the storage cluster controller determines failure control to take over data input / output from one of the paired storage controllers to the other storage controller based on the statistical information of the communication characteristics of the storage controllers generated by the I / O monitor, and to stop a storage node having the one storage controller, the storage cluster controller executes the failure control determined by the tiebreaker, and an application which had been performing I / O to the storage controller of the stopped storage node sends I / O to the storage controller which has taken over the processing. Effect of the Invention
[0012] According to the present invention, when a specified communication failure occurs in a storage system constructed across multiple data centers (e.g., a storage system operating in a cloud environment), it is possible to execute failure control that suppresses the occurrence of charges associated with I / O processing and increases in responses. [Brief description of the drawings]
[0013] [Figure 1] 1 is a conceptual diagram of a computer system 1 according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a conceptual diagram of a storage cluster 120. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of a storage system 100. [Figure 4] FIG. 1 is a diagram illustrating a communication failure in the storage system 100. [Diagram 5] 1 is a conceptual diagram for explaining collection of I / O log information in the storage system 100. FIG. [Figure 6] FIG. 2 is a conceptual diagram of communication failure control in the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of the configuration of a layout module 141. [Figure 8] 2 is a diagram showing an example of a storage controller layout table 212. FIG. [Figure 9] 13 is a diagram showing an example of a controller pair layout table 213. FIG. [Figure 10] FIG. 2 illustrates an example of a compute node layout table 214. [Figure 11] FIG. 13 is a diagram illustrating an example of a registration screen for compute node information. [Figure 12] FIG. 13 illustrates an example of the configuration of an I / O monitor 132. [Figure 13] FIG. 2 is a diagram showing an example of an I / O information record 255. [Figure 14] 10 is a flowchart illustrating an example of a processing procedure for a communication characteristic recording process according to the first embodiment. [Figure 15] FIG. 13 is a diagram illustrating an example of the configuration of a tiebreaker 142. [Figure 16] 10 is a flowchart illustrating an example of a processing procedure for determining a control method for a communication failure in the first embodiment. [Figure 17] FIG. 13 is a conceptual diagram of a computer system according to a modified example of the first embodiment. [Figure 18]18 is a diagram for explaining a communication failure in the storage cluster 290 shown in FIG. [Figure 19] FIG. 11 is a conceptual diagram of a computer system 2 according to a second embodiment of the present invention. [Figure 20] 1 is a conceptual diagram for explaining collection of I / O log information in the storage system 300. FIG. [Figure 21] FIG. 11 is a conceptual diagram of communication failure control in the second embodiment. [Figure 22] FIG. 2 is a diagram showing an example of the configuration of a layout module 341. [Figure 23] FIG. 4 shows an example of a volume layout table 412. [Figure 24] 13 is a diagram showing an example of a volume pair layout table 413. [Diagram 25] 3A and 3B are diagrams illustrating examples of the configuration of an I / O monitor 322 and an I / O state table 333. [Figure 26] FIG. 4 is a diagram showing an example of an I / O information record 433. [Figure 27] 13 is a flowchart illustrating an example of a processing procedure for a communication characteristic recording process according to the second embodiment. [Figure 28] FIG. 13 is a diagram illustrating an example of the configuration of a tiebreaker 342. [Figure 29] 13 is a diagram showing an example of a volume state table 332. FIG. [Diagram 30] 13 is a flowchart illustrating an example of a processing procedure for a control method decision process for a communication failure in the second embodiment. [Diagram 31] FIG. 11 is a conceptual diagram of a computer system 3 according to a third embodiment of the present invention. [Diagram 32] 13 is a diagram illustrating an example of the configuration of a migration proposer 544. FIG. [Diagram 33] FIG. 11 is a conceptual diagram of a computer system 4 according to a fourth embodiment of the present invention. [Diagram 34] 13 is a diagram illustrating an example of the configuration of a migration proposer 745. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
[0015] The following description and drawings are illustrative for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. In addition, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. The present invention is not limited to the embodiments, and all application examples that conform to the idea of the present invention are included in the technical scope of the present invention. A person skilled in the art can make various additions and modifications to the present invention within the scope of the present invention. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be multiple or singular.
[0016] In the following explanation, various information may be explained using expressions such as "table," "list," and "queue," but the various information may be expressed in other data structures. To indicate independence from the data structure, "XX table," "XX list," and the like may be referred to as "XX information." When explaining the content of each piece of information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.
[0017] In the following description, the process performed by executing a program may be described, but the program is executed by at least one processor (e.g., CPU) to appropriately perform a predetermined process using a storage resource (e.g., memory) and / or an interface device (e.g., communication port), and therefore the subject of the process may be the processor. Similarly, the subject of the process performed by executing a program may be a controller, device, system, computer, node, storage system, storage device, server, management computer, client, or host having a processor. The subject of the process performed by executing a program (e.g., processor) may include a hardware circuit that performs a part or all of the process. For example, the subject of the process performed by executing a program may include a hardware circuit that performs encryption and decryption, or compression and decompression. The processor operates as a functional unit that realizes a predetermined function by operating according to the program. The device and system including the processor are devices and systems that include these functional units.
[0018] A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a non-transitory storage medium readable by a computer. When the program source is a program distribution server, the program distribution server includes a processor (e.g., a CPU) and a non-transitory storage resource, and the storage resource may further store a distribution program and a program to be distributed. Then, the processor of the program distribution server may execute the distribution program, thereby distributing the program to be distributed to other computers. Also, in the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.
[0019] In the following description, when elements of the same kind are described without distinction, the common part (part excluding the subscript and the branch number) of the reference sign including the subscript and the branch number is used, and when elements of the same kind are described with distinction, the reference sign including the subscript and the branch number may be used. For example, when a storage node is described without distinction, it is described as "storage node 130", whereas when each storage node 130 is described with distinction, it may be described with a subscript such as "storage node 130A" or "storage node 130B". Also, each element may be described using an identifier (ID) or the like assigned to the element in question. Specifically, for example, the storage node 130A may be described as "storage node #1" and the storage node 130B as "storage node #2".
[0020] (1) First embodiment Fig. 1 is a conceptual diagram of a computer system 1 according to a first embodiment of the present invention. The computer system 1 shown in Fig. 1 is composed of a storage system 100 (storage cluster 120) in which a plurality of computers (servers, described later) are connected via a network, and a plurality of computers (compute nodes 110) that use the storage system 100. The configuration of the storage system 100 (storage cluster 120) will be described in detail again after the explanation of Figs. 2 to 4.
[0021] Fig. 2 is a conceptual diagram of a storage cluster 120. As shown in Fig. 2, the storage cluster 120 is configured by connecting a plurality of storage nodes 130 and a storage cluster controller 140 via a network switch 160. The storage cluster controller 140 may be executed inside the storage node 130. The configuration of the storage cluster shown in Fig. 2 is also common to other embodiments described later. In the storage system 100 shown in Fig. 1, since one storage cluster 120 is configured as a whole, the storage cluster 120 may be read as the storage system 100, or vice versa.
[0022] Fig. 3 is a diagram showing an example of a hardware configuration of the storage system 100. As also shown in Fig. 1, the storage system 100 is configured across multiple data centers 10, and in each data center 10, multiple servers 20 are connected by an intra-data center network (Intra-DC Network) 30. The data centers 10 (for example, between data center #1 and data center #2) are connected by an inter-data center network (Inter-DC Network) 40, so that the multiple data centers 10 are connected so as to be able to communicate with each other.
[0023] The server 20 is configured to include a CPU 21 which is a processor, a memory 22 which is a main storage device that stores programs and data executed by the CPU 21, a network interface 23 which has a communication function between the server 20 and the outside, and a drive 24 which is an auxiliary storage device that stores programs and data. The CPU 21 may be a processor other than a CPU (Central Processing Unit). One storage node 130 corresponds to one server 20.
[0024] The storage nodes 130 and storage cluster controller 140 constituting the storage cluster 120 are executed on the server 20. The storage cluster 120 may be configured in a specific data center 10, or may be configured across multiple data centers 10. Specifically, for example, the storage cluster 120 may be configured in the data center 10A, or may be configured from multiple data centers 10 such as the data centers 10A and 10B. The hardware configuration of the storage system described using FIG. 3 is also common to other embodiments described later.
[0025] Fig. 4 is a diagram for explaining a communication failure in the storage system 100. As shown in Fig. 4, data centers 10A, 10B that provide a cloud environment are composed of a plurality of racks 50 such as storage racks, and are connected by a plurality of network paths. A plurality of storage nodes 130 included in a storage cluster 120 are distributed and arranged in different racks 50, and the connections between the respective storage nodes 130 take different paths.
[0026] By configuring the communication paths as described above, in the case where a failure (marked with an x in FIG. 4) occurs in one of the communication paths connecting the storage nodes 130, the storage system 100 experiences a communication failure only between the storage node 130 on data center #1 and some of the storage nodes 130 on data center #2 (i.e., only the communication on the communication path where the failure occurred is cut off), and communication can continue on the other communication paths. The configuration of the communication paths of the storage system described using FIG. 4 is also common to other embodiments described later.
[0027] With reference to FIG. 1, the configuration of a storage system 100 (storage cluster 120) configured by connecting a plurality of computers (servers 20) via a network will be described in detail.
[0028] As shown in FIG. 1, the storage cluster 120 includes a storage node 130A, a storage node 130B, and a storage cluster controller 140.
[0029] The storage node 130A (storage node #1) has storage controllers 131A, 131C, and an I / O monitor 132A, and is located in the data center 10A. The storage node 130B (storage node #2) has storage controllers 131B, 131D, and an I / O monitor 132B, and is located in the data center 10B.
[0030] The storage cluster controller 140 is a controller that controls the storage cluster 120 as a whole, and includes a layout module 141, a tie breaker 142, and a network monitor 343. The storage cluster controller 140 is executed, for example, on a data center 10 different from the data centers 10A and 10B, and is connected to the storage nodes 130A and 130B arranged in each data center 10 by an inter-data center network 40. As described above, the storage cluster controller 140 may be configured to be executed on the data center 10A or the data center 10B. In that case, an application using a distributed algorithm may share configuration information and control commands of the storage cluster 120 between the data centers 10A and 10B and the third data center 10.
[0031] The storage controllers 131 are paired with each other (controller pair 134) so as to straddle the data center #1 and the data center #2, and manage the volumes 133.
[0032] For example, the controller pair 134A (controller pair #1) is composed of a storage controller 131A (storage controller #1) and a storage controller 131B (storage controller #2) and has an active-standby configuration. Normally, the storage controller #1 is in an active state and manages the volumes 133A and 133B (volumes #1 and #2). The storage controller #2 is normally on standby, and when the storage controller #1, which is the other pair, stops, it becomes active and takes over the management of the volumes #1 and #2. Similarly, the storage controller 131C (storage controller #3) and the storage controller 131D (storage controller #4), which constitute the controller pair 134B (controller pair #2), also have an active-standby configuration. Normally, the storage controller #4 is in an active state and manages the volume 133C (volume #3). The storage controller #3 is normally on standby, and when the storage controller #4, which is the other pair, stops, it becomes active and takes over the management of the volume #3.
[0033] The compute node 110A (compute node #1) is an application that uses the storage system 100, and runs on the data center 10A. As shown in Fig. 1, the compute node 110A and the storage system 100 are connected by an intra-data center path 151A and an inter-data center path 152A, and access to the volumes 133 (133A, 133B, 133C) can be implemented from the compute node 110A via these paths.
[0034] The compute node 110B (compute node #2) is an application that uses the storage system 100, and runs on the data center 10B. As shown in Fig. 1, the compute node 110B and the storage system 100 are connected by an intra-data center path 151B and an inter-data center path 152B, and access to the volumes 133 (133A, 133B, 133C) can be implemented from the compute node 110B via these paths.
[0035] The intra-datacenter path 151 is a communication path within the same datacenter 10, and if access from the compute node 110 to the volume 133 is intra-datacenter communication that passes only through the intra-datacenter path 151, no charges are incurred in the cloud environment.
[0036] The inter-datacenter path 152 is a communication path between different data centers 10, and when access from a compute node 110 to a volume 133 is inter-datacenter communication via the inter-datacenter path 152, charges are incurred in the cloud environment, for example according to the amount of communication.
[0037] FIG. 5 is a conceptual diagram for explaining collection of I / O log information in the storage system 100. As shown in FIG.
[0038] 5, when storage controller #1 arranged in data center #1 receives an I / O from compute nodes 110A, 110B to a volume 133 (volume #1 or volume #2) that it manages, it sends log information of the I / O to I / O monitor #1. Then, using the received I / O log information and information from layout module 141 of storage cluster controller 140, I / O monitor #1 determines whether the I / O is an intra-datacenter communication or an inter-datacenter communication, and records the I / O log information.
[0039] Similarly, when storage controller #2 arranged in data center #2 receives an I / O from compute nodes 110A, 110B to volume 133 (volume #3) that it manages, it sends log information of the I / O to I / O monitor #2. Then, using the received I / O log information and information from layout module 141 of storage cluster controller 140, I / O monitor #2 determines whether the I / O is an intra-datacenter communication or an inter-datacenter communication, and records the I / O log information.
[0040] With the above configuration, in each storage node 130, in cooperation with the storage controller 131, the I / O monitor 132 can accumulate and hold log information of I / O to the volume 133 (I / O statistical information).
[0041] FIG. 6 is a conceptual diagram of communication failure control in the first embodiment. Communication failure control means control that is performed when a communication failure occurs, and is hereinafter also referred to as failure control. The state of the storage cluster 120 shown in FIG. 6 is a state in which the controller pair #1 and the controller pair #2 are disconnected due to a failure occurring in the communication path between the storage node #1 and the storage node #2. When communication between the controller pairs becomes impossible in this way, the storage system 100 according to the first embodiment stops all the storage controllers 131 belonging to the storage node #1 or the storage node #2, and puts the storage node #1 or the storage node #2 in a stopped state.
[0042] More specifically, the control when a communication failure occurs will be described. When a communication failure as described above occurs, first, the network monitor 143 operating on the storage cluster controller 140 detects that the controller pair #1 and the controller pair #2 are unable to communicate, and requests the tiebreaker 142 to stop the related storage controllers 131.
[0043] Next, the tiebreaker 142 obtains configuration information of the storage system 100 from the layout module 141, and further obtains I / O statistical information of the storage controllers #1 and #4 (the active storage controllers 131 in the disconnected controller pair 134) from the I / O monitors #1 and #2.
[0044] Then, based on the acquired information, the tiebreaker 142 determines a method for shutting down the storage controller 131 so that the amount of communication (throughput) between the data centers after shutting down the storage nodes #1 and #2 is minimized, and requests the storage cluster controller 140 to perform control in accordance with the determined shutting down method. For example, in the case of Fig. 6, the tiebreaker 142 determines that shutting down the storage node #2 will minimize the amount of communication (throughput) between the data centers, and requests the storage cluster controller 140 to shut down the storage controllers #2 and #4 of the storage node #2 according to the determination result.
[0045] Then, the storage cluster controller 140 performs control to stop the storage controller 131 in accordance with the request from the tiebreaker 142, and stops the storage node 130 (storage node #2 in this example) to which one of the controller pair that has become unable to communicate belongs. As a result, as shown in Fig. 6, in the controller pair #1, the storage controller #2 that was in standby state stops, and the storage controller #1 continues to operate in active state (management of volumes #1 and #2). Also, in the controller pair #2, the storage controller #4 that was in active state stops, and the storage controller #3 goes from standby state to active state and takes over management of volume #3.
[0046] As a result of the above-mentioned communication failure control, the path of compute node #1 to storage system 100 becomes intra-datacenter path 151A, and the path of compute node #2 to storage system 100 becomes inter-datacenter path 152B. At this time, when the communication volumes of the two paths involved in I / O processing are compared, the communication volume of intra-datacenter path 151A is greater than the communication volume of inter-datacenter path 152, and the charges incurred for inter-datacenter communication via inter-datacenter path 152 are minimized.
[0047] Fig. 7 is a diagram showing an example of the configuration of the layout module 141. As shown in Fig. 7, the layout module 141 includes a layout updater 211, a storage controller layout table 212, a controller pair layout table 213, and a compute node layout table 214.
[0048] The storage controller layout table 212 is data for managing the arrangement information of the storage controller 131, and receives the arrangement information of the storage controller 131 as an input from the configuration file 220 that holds the configuration information of the storage cluster 120, and holds this. When the arrangement information of the storage controller 131 held in the storage controller layout table 212 is updated, the layout module 141 sends the updated information to the tiebreaker 142 and the I / O monitor 132. The storage location of the configuration file 220 is, for example, within the storage cluster controller 140, but is not particularly limited thereto.
[0049] Fig. 8 is a diagram showing an example of the storage controller layout table 212. In the case of Fig. 8, the storage controller layout table 212 is composed of a data center ID 2121 indicating an identifier (data center ID) allocated to each data center 10, a storage node ID 2122 indicating an identifier (storage node ID) allocated to each storage node 130, and a storage controller ID 2123 indicating an identifier (storage controller ID) allocated to each storage controller 131. These pieces of information can be acquired from the configuration file 220, and it is possible to know the storage node 130 and data center 10 to which the storage controller 131 belongs.
[0050] The controller pair layout table 213 is data for managing information related to the controller pairs 134, and receives and holds information related to the controller pairs 134 as input from the configuration file 220 that holds the configuration information of the storage cluster 120. When the information related to the controller pairs 134 held in the controller pair layout table 213 is updated, the layout module 141 sends the updated information to the tiebreaker 142.
[0051] Fig. 9 is a diagram showing an example of the controller pair layout table 213. In the case of Fig. 9, the controller pair layout table 213 is composed of a controller pair ID 2131 indicating an identifier (controller pair ID) assigned to each controller pair 134, an active storage controller ID 2132 indicating the storage controller ID of the storage controller 131 in an active state in the controller pair 134, and a standby storage controller ID 2133 indicating the storage controller ID of the storage controller 131 in a standby state in the controller pair 134. This information can be obtained from the configuration file 220, and it is possible to know the storage controllers 131 constituting the controller pair 134 and their status (active state or standby state).
[0052] The compute node layout table 214 is data for managing placement information of the compute nodes 110, and receives and holds the placement information of the compute nodes 110 as input from the layout updater 211. When the placement information of the compute nodes 110 held in the compute node layout table 214 is updated, the layout module 141 sends the updated information to the I / O monitor 132.
[0053] The layout updater 211 has a function of updating the placement information of the compute node 110 held in the compute node layout table 214. The layout updater 211 may be software or hardware. The layout updater 211 receives an input of a compute node ID (an identifier of the compute node 110) and an instance ID from a user interface 230 that allows a user to perform an input operation, queries the data center platform 240 for placement information of the compute node 110 (information of the data center 10 to which the compute node 110 belongs), and sends the processing results to the compute node layout table 214. Note that the layout updater 211 may be configured to provide information obtained by querying the data center platform 240 to the user side from the user interface 230. FIG. 11, which will be described later, shows an example of a registration screen of compute node information provided to the user by the user interface 230.
[0054] The instance ID refers to server management information in the data center infrastructure 240, and an instance ID is assigned to a server that executes the compute node 110 or the like, or to a virtual machine that executes on the server. The data center infrastructure 240 is a function on the cloud provided by the data center 10.
[0055] Fig. 10 is a diagram showing an example of the compute node layout table 214. In the case of Fig. 10, the compute node layout table 214 is composed of a data center ID 2141 indicating a data center ID, and a compute node ID 2142 indicating an identifier (compute node ID) assigned to an individual compute node 110. As described above, this information can be acquired from the user interface 230 and the data center platform 240 via the layout updater 211, and it is possible to know the data center 10 to which the compute node 110 belongs.
[0056] Fig. 11 is a diagram showing an example of a registration screen for compute node information. The registration screen 280 shown in Fig. 11 is a screen for inputting information provided to the user by the user interface 230, and is displayed on a display or the like connected to the storage cluster 120 via a network. When a user performs a predetermined operation on the registration screen 280, information about the compute node 110 is additionally input, and the information held in the compute node layout table 214 of the layout module 141 is updated.
[0057] The registration screen 280 shown in Figure 11 is configured to include an input field 281 for entering a compute node ID, an input field 282 for entering an instance ID, a search button 283 for searching for the data center ID of the data center 10 to which the compute node 110 belongs, an output field 284 in which the data center ID is displayed after operating the search button 283, a delete button 285 for deleting the input / output content of the target row, and an apply button 286 for applying (registering) the placement information of the compute node 110 based on the input / output content displayed on the registration screen 280 to the compute node layout table 214.
[0058] On the registration screen 280, when the user inputs a compute node ID and an instance ID in input fields 281, 282 and then presses a search button 283, the layout updater 211 uses the instance ID to inquire of the data center infrastructure 240, thereby obtaining placement information (data center ID) of the compute node 110, and displays it in an output field 284. Then, when the user presses an apply button 286, the layout updater 211 combines the input / output contents (compute node ID and data center ID) displayed on the registration screen 280 and transmits them to the compute node layout table 214, and the information is applied (registered).
[0059] Fig. 12 is a diagram showing an example of the configuration of the I / O monitor 132. As shown in Fig. 12, the I / O monitor 132 is configured to include an I / O recorder 251, a statistics module 252, an intra-data center I / O table 253, and an inter-data center I / O table 254.
[0060] The I / O recorder 251 has a function of identifying whether the I / O received from the storage controller 131 is an intra-datacenter communication or an inter-datacenter communication, and transmitting the I / O log to a table according to the identification result. The I / O recorder 251 may be software or hardware.
[0061] To explain in more detail, the I / O recorder 251 first receives and stores information on the storage controller layout table 212 and the compute node layout table 214 from the layout module 141.
[0062] When the I / O recorder 251 receives an I / O log (I / O log 261) from the storage controller 131, it identifies whether the I / O was processed by intra-datacenter communication or inter-datacenter communication by referring to the storage controller ID 2123 in the storage controller layout table 212 and the compute node ID 2142 in the compute node layout table 214. The I / O log includes, for example, the storage controller ID, the compute node ID, and the I / O size, as in the I / O log 261 shown in FIG.
[0063] Then, depending on the result of the above identification, if the I / O was processed by intra-datacenter communication, the I / O recorder 251 sends the I / O log to the intra-datacenter I / O table 253, and if the I / O was processed by inter-datacenter communication, it sends the I / O log to the inter-datacenter I / O table 254. Details of the process (communication characteristics recording process) by the above I / O recorder 251 are shown in Fig. 14, which will be described later.
[0064] The intra-datacenter I / O table 253 records log information of I / O processed in intra-datacenter communication received from the I / O recorder 251 (see I / O information record 255 shown in FIG. 13). The intra-datacenter I / O table 253 increments and holds the I / O size for each storage controller ID. Furthermore, when the intra-datacenter I / O table 253 receives a reset signal from the statistics module 252, it resets the held I / O size to zero.
[0065] The inter-datacenter I / O table 254 records log information of I / O processed in inter-datacenter communication received from the I / O recorder 251. The inter-datacenter I / O table 254 increments and holds the I / O size for each storage controller ID. Furthermore, when the inter-datacenter I / O table 254 receives a reset signal from the statistics module 252, it resets the held I / O size to zero.
[0066] The statistics module 252 is a program that has the function of responding with I / O statistical information (I / O traffic volume for intra-datacenter communications and I / O traffic volume for inter-datacenter communications) corresponding to a specified storage controller ID in response to a request from the tiebreaker 142.
[0067] More specifically, when the statistics module 252 receives a storage controller ID from the tiebreaker 142, it reads the I / O size corresponding to the storage controller ID from the intra-datacenter I / O table 253 and the inter-datacenter I / O table 254, respectively.
[0068] Then, the statistics module 252 converts each of the read I / O sizes into the traffic volume (throughput) of intra-datacenter communication associated with I / O processing and the traffic volume (throughput) of inter-datacenter communication associated with I / O processing by dividing (subtracting) each of the read I / O sizes by the time elapsed since each table was last reset, and responds with these conversion results as statistical information (I / O statistical information 262) to the tiebreaker 142. As shown in Fig. 12, the I / O statistical information 262 includes the traffic volume within the datacenter (intra-datacenter I / O throughput) and the traffic volume between datacenters (inter-datacenter I / O throughput) for each storage controller ID.
[0069] In addition, when the statistical module 252 receives a reset request from the tiebreaker 142 or at regular intervals, it sends a reset signal to the intra-datacenter I / O table 253 and the inter-datacenter I / O table 254, and resets the I / O sizes held in each table to zero.
[0070] If a reset request is received from the tiebreaker 142 immediately after the I / O size is reset by periodic processing or the like, the I / O size will be recorded for only a short period of time, and there is a possibility that sufficient I / O statistical information will not be obtained to determine a method for stopping the storage controller 131 so as to minimize the amount of communication between data centers. Therefore, when resetting at regular intervals, it is preferable to retain the I / O size before the reset for a specified period of time.
[0071] FIG. 13 is a diagram showing an example of the I / O information record 255. The I / O information record 255 is data that records I / O log information held by the intra-datacenter I / O table 253 and the inter-datacenter I / O table 254. In the case of FIG. 13, the I / O information record 255 is composed of a storage controller ID 2551 indicating a storage controller ID, and an I / O size 2552 indicating a cumulative total of the data amount of I / O processed by the storage controller 131. The I / O information record 255 (the intra-datacenter I / O table 253 and the inter-datacenter I / O table 254) increments the processed I / O size for each storage controller 131 and holds it in the I / O size 2552, and when a reset signal is received from the statistics module 252, resets the value of the I / O size 2552 to zero.
[0072] Fig. 14 is a flowchart showing an example of a processing procedure of a communication characteristics recording process in the first embodiment. The communication characteristics recording process shown in Fig. 14 is a process in which the I / O monitor 132 (particularly the I / O recorder 251) records an I / O log in the intra-datacenter I / O table 253 or the inter-datacenter I / O table 254 as a record of communication characteristics accompanying I / O processing for each storage controller 131 in the storage system 100, and is started upon receiving an I / O log from the storage controller 131.
[0073] 14, first, the I / O monitor 132 checks the data center ID of the data center 10 to which the storage controller 131 indicated by the storage controller ID belongs, based on the storage controller ID included in the I / O log (step S101). When checking the data center ID from the storage controller ID, the storage controller layout table 212 shown in FIG. 8 may be referred to.
[0074] Next, the I / O monitor 132 checks the data center ID of the data center 10 to which the compute node 110 indicated by the compute node ID belongs based on the compute node ID included in the I / O log (step S102). When checking the data center ID from the compute node ID, it is sufficient to refer to the compute node layout table 214 shown in Fig. 10. The order of execution of steps S101 and S102 is not particularly limited, and they may be processed in parallel.
[0075] Next, the I / O monitor 132 compares the data center IDs confirmed in steps S101 and S102 to determine whether the storage controller 131 and the compute node 110 indicated in the I / O log belong to the same data center 10 (step S103).
[0076] If it is determined in step S103 that the storage controller 131 and the compute node 110 belong to the same data center 10 (YES in step S103), the I / O monitor 132 records the I / O log received at the start of processing in the I / O table 253 within the data center (step S104), and terminates the communication characteristics recording processing.
[0077] On the other hand, if it is determined in step S103 that the storage controller 131 and the compute node 110 belong to different data centers 10 (NO in step S103), the I / O monitor 132 records the I / O log received at the start of the processing in the inter-data center I / O table 254 (step S105), and terminates the communication characteristics recording processing.
[0078] Fig. 15 is a diagram showing an example of the configuration of the tiebreaker 142. As shown in Fig. 15, the tiebreaker 142 is configured to include a collector 271, a calculator 272 (272A, 272B), and an indicator 273. Each of these components may be realized by software or hardware.
[0079] The collector 271 has a function of receiving and storing information of the controller pair layout table 213 from the layout module 141, and further has a function of performing the following processing when a controller pair ID that requires fault control is received from the network monitor 143.
[0080] When the collector 271 receives two controller pair IDs that require failure control from the network monitor 143 that has detected a communication failure between storage nodes, the collector 271 queries the I / O monitors 132A and 132B for I / O statistical information for the active storage controller IDs. Then, when the collector 271 receives the I / O statistical information (I / O statistical information 262 shown in FIG. 12) queried from the I / O monitors 132A and 132B, it transmits the information to the computers 272A and 272B.
[0081] As a specific example, FIG. 15 shows collector 271 receiving controller pair ID #1 and controller pair ID #2 from network monitor 143, recognizing storage controller ID #1 and storage controller ID #4 as the IDs of active storage controllers, querying I / O monitor 132A for I / O statistical information about storage controller ID #1, and querying I / O monitor 132B for I / O statistical information about storage controller ID #4.
[0082] The calculator 272 (272A, 272B) has the function of calculating the inter-datacenter communication volume (inter-datacenter I / O throughput) when the storage controller 131 that is the target of failure control stops based on I / O statistical information, and sending it to the indicator 273.
[0083] Specifically, in the case of Fig. 15, the computer 272A refers to I / O statistical information and adds up the I / O throughput of the intra-datacenter communication in the storage controller ID #1 and the I / O throughput of the inter-datacenter communication in the storage controller ID #4 to calculate the inter-datacenter communication volume ("T1" in Fig. 15) when the storage controller with storage controller ID #1 (storage controller #1 shown in Fig. 1) stops. Also, the computer 272B refers to I / O statistical information and adds up the I / O throughput of the intra-datacenter communication in the storage controller ID #4 and the I / O throughput of the inter-datacenter communication in the storage controller ID #1 to calculate the inter-datacenter communication volume ("T2" in Fig. 15) when the storage controller with storage controller ID #4 (storage controller #4 shown in Fig. 1) stops.
[0084] The indicator 273 has a function of receiving and storing information on the storage controller layout table 212 and the controller pair layout table 213 from the layout module 141, and further has a function of performing the following processing when it receives the calculation results (T1, T2 in FIG. 15) of the throughput between data centers when the active storage controller is stopped from the computers 272A, 272B.
[0085] The indicator 273 compares the inter-data center traffic (inter-data center I / O throughput) T1 received from the computer 272A with the inter-data center traffic (inter-data center I / O throughput) T2 received from the computer 272B, determines to stop the storage controller 131 so that the inter-data center traffic (throughput) becomes smaller, and requests the storage cluster controller 140 to stop the target storage controller 131 according to the determination result.
[0086] In the specific example shown in FIG. 15, when the value of T1 is large (T1≧T2), the indicator 273 determines to stop all the storage controllers 131 of the storage node (storage node #2) to which the storage controller ID #4 belongs. On the other hand, when the value of T2 is large (T1<T2), the indicator 273 determines to stop all the storage controllers 131 of the storage node (storage node #1) to which the storage controller ID #1 belongs. The control result when the value of T1 is large corresponds to the state illustrated in FIG. 6 described above. Specifically, the storage controllers #2 and #4 (storage controllers 131B and 131D) belonging to the storage node #2 (storage node 130B) are stopped.
[0087] FIG. 16 is a flowchart showing an example of the processing procedure of the control method determination process for communication failure in the first embodiment. The control method determination process shown in FIG. 16 is a process executed by the breaker 142. The process is started when the breaker 142 receives two controller pair IDs that require failure control from the network monitor 143 that has detected a communication failure between the storage nodes.
[0088] 15, first, the collector 271 of the tiebreaker 142 extracts an active storage controller ID from the two controller pair IDs received from the network monitor 143 (step S201). In step S201, the collector 271 can extract the active storage controller ID by obtaining the corresponding active storage controller ID 2132 from the controller pair layout table 213 using the received controller pair ID as a key. In this description, it is assumed that "SC001" is extracted as the storage controller ID #1 corresponding to the storage controller #1, and "SC004" is extracted as the storage controller ID #4 corresponding to the storage controller #4.
[0089] Next, the collector 271 queries the I / O monitor 132 to acquire information on communication characteristics (I / O statistical information 262) of the two storage controllers 131 corresponding to the storage controller IDs extracted in step S201 (step S202). Then, the collector 271 transmits the acquired I / O statistical information 262 of the two storage controllers 131 (storage controller IDs #1 and #4) to the computers 272A and 272B.
[0090] Next, the computer 272A uses the I / O statistics information 262 received in step S202 to calculate the traffic volume "T1" of the inter-datacenter communication that is expected when the active storage controller (for example, storage controller #1) in the first controller pair stops (step S203). The method of calculating "T1" is as described in the explanation of the computer 272A with reference to FIG. 15.
[0091] Next, the computer 272B uses the I / O statistics information 262 received in step S202 to calculate the traffic volume "T2" of the inter-datacenter communication that is expected when the active storage controller (for example, storage controller #4) in the second controller pair stops (step S204). The method of calculating "T2" is as described in the explanation of the computer 272B with reference to FIG. 15.
[0092] Then, the computer 272A transmits the inter-data center communication volume "T1" calculated in step S203 to the indicator 273, and the computer 272B transmits the inter-data center communication volume "T2" calculated in step S204 to the indicator 273. Note that the order of execution of steps S203 and S204 is not particularly limited, and they may be processed in parallel.
[0093] Next, the indicator 273 compares the calculated inter-datacenter communication volume "T1" with the inter-datacenter communication volume "T2" (step S205), and judges whether the value of T1 is equal to or greater than the value of T2 (step S206). If the value of T1 is equal to or greater than the value of T2 (YES in step S206), the process proceeds to step S207, and if the value of T1 is less than the value of T2 (NO in step S206), the process proceeds to step S208.
[0094] Here, the relationship "T1≧T2" means that when the active storage controller #1 in the first controller pair of two storage controller pairs is stopped, the amount of inter-datacenter communication occurring between data centers 10 due to I / O processing will be greater than when the active storage controller #4 in the second controller pair is stopped.
[0095] Therefore, when "T1 ≥ T2" in step S206 (YES in step S206), as a control method for reducing the inter-data center traffic after failure control (i.e., after the storage controller stops), for the storage node #2 to which the active storage controller #4 in the second controller pair belongs, the indicator 273 determines to stop all the storage controllers 131 (storage controllers #2 and #4), requests the storage cluster controller 140 to perform control according to this decision (step S207), and ends the control method determination process.
[0096] On the other hand, the relationship of "T1 < T2" means that when the active storage controller #4 in the second controller pair among the two storage controller pairs is stopped, the inter-data center traffic generated between the data centers 10 during the I / O process becomes larger than when the active storage controller #1 in the first controller pair is stopped.
[0097] Therefore, when "T1 < T2" in step S206 (NO in step S206), as a control method for reducing the inter-data center traffic after failure control (i.e., after the storage controller stops), for the storage node #1 to which the active storage controller #1 in the first controller pair belongs, the indicator 273 determines to stop all the storage controllers 131 (storage controllers #1 and #3), requests the storage cluster controller 140 to perform control according to this decision (step S208), and ends the control method determination process.
[0098] As described above, by executing the communication characteristic recording process shown in Fig. 14 and the control method determination process shown in Fig. 16, in the computer system 1 according to this embodiment, in two storage controller pairs (controller pairs #1 and #2 in Fig. 6) formed between the storage controllers 131 belonging to the storage nodes 130 of different data centers 10, in which the active storage controllers 131 in each pair belong to the storage nodes 130 of different data centers 10, when a communication failure occurs between the controller pairs, a failure control method can be determined for stopping all the storage controllers 131 belonging to any of the storage nodes 130 so as to reduce the communication volume (throughput) between the data centers associated with I / O processing. Then, when the storage cluster controller 140 stops the storage controllers 131 according to the determined failure control method, the management of the volume 133 by the active storage controller 131 to be stopped is taken over by the standby storage controller 131 of the storage controller pair in the active state. Then, after the failure control is executed, the application that was performing I / O to the stopped storage controller 131 sends I / O to the storage controller 131 that has taken over the processing. As a result, as explained with reference to the conceptual diagram of Fig. 6, the storage system 100 can handle I / O requests from one or more compute nodes 110 with the storage controller 131 arranged in one data center 10 while suppressing the amount of communication between data centers via the inter-data center path 152, and can realize failure control that suppresses the occurrence of charges and an increase in responses associated with I / O processing.
[0099] 1 and 5, the storage nodes 130 constituting the storage cluster 120 are arranged one for each data center 10, but the storage system 100 according to this embodiment is not limited in the number of storage nodes 130 arranged for each data center 10, and multiple storage nodes 130 may be arranged in each data center 10. A computer system including a storage system 100 configured in this way will be described as a modified example of the first embodiment with reference to FIGS. 17 to 18.
[0100] 17 is a conceptual diagram of a computer system in a modified example of the first embodiment. In FIG.
[0101] As shown in Fig. 17, the storage cluster 290 is configured to include a plurality of storage nodes 130 arranged in each of the data centers 10A and 10B, and one storage cluster controller 140. More specifically, in the case of Fig. 17, at least the storage node 130A (storage node #1) and the storage node 130C (storage node #3) operate in the data center 10A (data center #1), and at least the storage node 130B (storage node #2) and the storage node 130D (storage node #4) operate in the data center 10B (data center #2). Note that even more storage nodes 130 may belong to the storage cluster 290 (storage system 100).
[0102] Furthermore, even if a storage cluster 290 includes a plurality of storage nodes 130 in one data center 10, controller pairs 134 across different data centers 10 are configured only between specific storage nodes 130, similar to the storage cluster 120 shown in Fig. 1. Specifically, in the case of Fig. 17, the storage controller 131 of the storage node #1 arranged in the data center #1 forms controller pairs 134A and 134B (controller pair #1 and #2) with the storage controller 131 of the storage node #2 arranged in the data center #2, and the storage controller 131 of the storage node #3 arranged in the data center #1 forms controller pairs 134C and 134D (controller pair #3 and #4) with the storage controller 131 of the storage node #4 arranged in the data center #2.
[0103] Fig. 18 is a diagram for explaining a communication failure in the storage cluster 290 shown in Fig. 17. In the storage cluster 290 shown in Fig. 18, a failure occurs in the communication path between storage node #1 and storage node #2, and the controller pair #1 and #2 are disconnected. Furthermore, a failure also occurs in the communication path between storage node #3 and storage node #4, and the controller pair #3 and #4 are disconnected.
[0104] When a communication failure as shown in Fig. 18 occurs, the storage cluster 290 performs failure control for each of the two storage nodes forming the controller pair 134. Therefore, the storage cluster 290 can independently perform failure control for storage node #1 and storage node #2, and failure control for storage node #3 and storage node #4. Details of each failure control can be considered to be similar to the failure control described above with reference to Figs. 15 and 16, and therefore description thereof will be omitted.
[0105] Therefore, even in the configuration of the modified example of the first embodiment in which a plurality of storage nodes 130 are arranged in each data center 10, it is possible to obtain the same effects as the storage system 100 of the first embodiment described above.
[0106] (2) Second embodiment In the second embodiment, a storage system 300 including a storage cluster 320 and a quorum disk 330 will be described. The second embodiment will be described mainly with respect to the differences from the first embodiment. Therefore, in the second embodiment, the same components as those in the first embodiment will be denoted by the same reference numerals and the description thereof will be omitted as appropriate. In addition, the description of the processes performed by the individual components similar to those in the first embodiment will be omitted as appropriate.
[0107] Fig. 19 is a conceptual diagram of a computer system 2 according to the second embodiment of the present invention. The computer system 2 shown in Fig. 19 is composed of a storage system 300 in which multiple computers (servers 20) are connected via a network, and multiple computers (compute nodes 110) that use the storage system 300.
[0108] The storage system 300 is configured to include a storage cluster 320A (storage cluster #1), a storage cluster 320B (storage cluster #2), and a quorum disk 330. Although not shown in Fig. 19, each storage cluster 320 may be considered to include one or more storage nodes, similar to the storage clusters 120 and 290 shown in the first embodiment.
[0109] The storage cluster #1 includes a storage controller 321A (storage controller #1), an I / O monitor 322A (I / O monitor #1), and a storage cluster controller 340A (storage cluster controller #1), and is located in a data center 10A (data center #1).
[0110] The storage cluster #2 includes a storage controller 321B (storage controller #2), an I / O monitor 322B (I / O monitor #2), and a storage cluster controller 340B (storage cluster controller #2), and is located in a data center 10B (data center #2).
[0111] The quorum disk 330 is a recording medium having an information recording function accessible from the data center #1 and the data center #2, and is configured to have a layout table 331, a volume state table 332, and an I / O state table 333. The quorum disk 330 may be configured to be held by a node executed on the server 20.
[0112] Storage controller #1 manages volume 323A (volume #1) and volume 323C (volume #3), and storage controller #2 manages volume 323B (volume #2) and volume 323D (volume #4). In storage system 300, two volumes 323 arranged in different storage clusters 320 are paired (volume pair 324). The two volumes 323 constituting volume pair 324 hold the same data and operate in an active-active state. Specifically, for example, volumes #1 and #2 form volume pair 324A (volume pair #1) and hold the same data. I / O access may be performed to either volume #1 or volume #2, and the write result is synchronized between the two volumes. Similarly, volumes #3 and #4 form volume pair 324B (volume pair #2) and hold the same data.
[0113] The storage cluster controller 340 is provided for each storage cluster 320 and is a controller that controls the entire storage cluster 320 to which it belongs. The storage cluster controller #1 is configured with a layout module 341A, a tie breaker 342A, a network monitor 343A, and a state checker 344A. Similarly, the storage cluster controller #2 is configured with a layout module 341B, a tie breaker 342B, a network monitor 343B, and a state checker 344B. The state checker 344 has a function of periodically monitoring the volume state table 332 of the quorum disk 330, and when there is a change in the table, requesting its own storage cluster controller 340 to perform control according to the state.
[0114] The compute node 110A (compute node #1) in Fig. 19 is an application that uses the storage system 300, and runs on the data center #1. As shown in Fig. 19, the compute node #1 and the storage system 300 are connected by an intra-data center path 151A and an inter-data center path 152A, and access to the volumes 323 (323A, 323B, 323C, 323D) can be implemented from the compute node #1 via these paths.
[0115] The compute node 110B (compute node #2) in Fig. 19 is an application that uses the storage system 300, and runs on the data center #2. As shown in Fig. 19, the compute node #2 and the storage system 300 are connected by an intra-data center path 151B and an inter-data center path 152B, and access to the volumes 323 (323A, 323B, 323C, 323D) can be implemented from the compute node #2 via these paths.
[0116] FIG. 20 is a conceptual diagram for explaining collection of I / O log information in the storage system 300. As shown in FIG.
[0117] As shown in FIG. 20, when storage controller #1 located in data center #1 receives I / O from compute nodes 110A, 110B for a volume 323 (volume #1 or volume #3) that it manages, it sends log information of the I / O to I / O monitor #1.
[0118] Then, the I / O monitor #1 uses the received I / O log information and information from the layout module 341A of the storage cluster controller #1 to determine whether the I / O is an intra-datacenter communication or an inter-datacenter communication, and records the I / O log information. Furthermore, the I / O monitor #1 also writes the recorded I / O log information to the I / O state table 333. The I / O monitor #1 also periodically monitors the information in the I / O state table 333, and when there is a change, obtains the difference and records it in itself.
[0119] 20, the storage controller #2 arranged in the data center #2 operates in the same manner as the storage controller #1 on the storage cluster #2, and the I / O monitor #2 that receives the I / O log information from the storage controller #2 operates in the same manner as the I / O monitor #1. Therefore, the I / O monitor #1 and the I / O monitor #2 share the same information via the I / O state table 333.
[0120] 20, the layout module 341A operating on the storage cluster controller #1 has a function of sending data center arrangement information of the volumes 323 and the compute nodes 110 to the I / O monitor #1. If there is a change in this information, the layout module 341A also writes the information in the layout table 331. In addition, the layout module 341A periodically monitors the layout table 331, and if there is a change, obtains and records the difference.
[0121] On the other hand, the layout module 341B operating on the storage cluster controller #2 performs the same operation as the layout module 341A on the storage cluster #2 as shown in Fig. 20. Therefore, the layout module 341A of the storage cluster controller #1 and the layout module 341B of the storage cluster controller #2 share the same layout information via the layout table 331.
[0122] Fig. 21 is a conceptual diagram of communication failure control in the second embodiment. The state of the storage cluster 320 shown in Fig. 21 is a state in which volume pair #1 and volume pair #2 are disconnected due to a failure occurring in the communication path between storage node #1 and storage node #2. When communication between the volumes becomes impossible in this way, synchronization between the volumes becomes impossible. When this state occurs, the storage system 300 according to the second embodiment stops one of the volumes constituting the disconnected volume pair (for example, volume #2) and continues to accept I / O only through the other volume (in this case, volume #1).
[0123] The above example will be used to specifically explain failure control by the storage system 300. When a communication failure as described above occurs, first, the network monitor 343A operating on the storage cluster controller #1 detects that the volume pair #1 is unable to communicate, and requests the tiebreaker 342A to stop the related volume 323.
[0124] Next, the tiebreaker 342A obtains configuration information of the storage system 300 from the layout module 341A, and further obtains I / O statistical information of volumes #1 and #2 that make up the volume pair #1 from the I / O monitor #1.
[0125] Then, based on the acquired information, the tiebreaker 342A determines a volume stopping method so that the amount of communication (throughput) between the data centers after stopping either of volumes #1 and #2 constituting volume pair #1 is minimized, and requests control according to the determined stopping method from the storage cluster controller #1, or writes information according to the determined stopping method in the volume state table 332. For example, in the case of Fig. 21, the tiebreaker 342A determines that stopping volume #2 will minimize the amount of communication between the data centers thereafter, and writes the stopping of volume #2 in the volume state table 332 according to the determination result.
[0126] Here, the state checker 344B operating on the storage cluster controller #2 of the storage cluster #2 periodically monitors the volume state table 332, and if there is a change, requests the storage cluster controller #2 to perform control according to the state. Therefore, as described above, when the tiebreaker 342A writes the suspension of volume #2 to the volume state table 332, the state checker 344B recognizes this and requests the storage cluster controller #2 to suspend volume #2. Then, upon receiving the request, the storage cluster controller #2 suspends volume #2 via the storage controller #2.
[0127] As described above, in the computer system 2 according to the second embodiment, when volume pair #1 is disconnected due to a communication failure, one of the volumes constituting volume pair #1 can be stopped so that the amount of communication between data centers after the stoppage is minimized, and I / O reception can be continued only by the other volume that was not stopped (volume #1 in this example). Note that, even when volume pair #2 is disconnected due to a communication failure, similar processing is performed, making it possible to continue receiving I / O using one of the volumes constituting volume pair #2.
[0128] Fig. 22 is a diagram showing an example of the configuration of the layout module 341. As shown in Fig. 22, the layout module 341 is configured to include a layout updater 411, a volume layout table 412, a volume pair layout table 413, and a compute node layout table 414.
[0129] The volume layout table 412 is data for managing configuration information of the volume 323, and reads the configuration information of the volume 323 held by the storage system 300 from the layout table 331 and holds this information. When the configuration information of the volume 323 held in the volume layout table 412 is updated, the layout module 341 sends the updated information to the tiebreaker 342 and the I / O monitor 322.
[0130] Fig. 23 is a diagram showing an example of the volume layout table 412. In the case of Fig. 23, the volume layout table 412 is composed of a data center ID 4121 indicating an identifier (data center ID) allocated to each data center 10, a storage controller ID 4122 indicating an identifier (storage controller ID) allocated to each storage controller 321, and a volume ID 4123 indicating an identifier (volume ID) allocated to a volume managed by the storage controller 321. By referring to such a volume layout table 412, it is possible to know the storage controller 321 and the data center 10 to which the volume 323 belongs.
[0131] The volume pair layout table 413 is data for managing information related to the volume pairs 324, and reads configuration information of the volume pairs 324 held by the storage system 300 from the layout table 331 and holds this information. When the configuration information of the volume pairs 324 held in the volume pair layout table 413 is updated, the layout module 341 sends the updated information to the tiebreaker 342.
[0132] Fig. 24 is a diagram showing an example of the volume pair layout table 413. In the case of Fig. 24, the volume pair layout table 413 is composed of a volume pair ID 4131 indicating an identifier (volume pair ID) assigned to each volume pair 324, and a volume ID 4132 indicating the volume ID of the volume 323 constituting the volume pair 324. As described above, in the storage system 300 according to the second embodiment, the two volumes 323 constituting the volume pair 324 operate in an active-active state. By referring to such a volume pair layout table 413, it is possible to know the volumes 323 forming the volume pair 324.
[0133] The compute node layout table 414 is data for managing placement information of the compute nodes 110, and receives placement information of the compute nodes 110 as input from the layout updater 411 and holds it. When the placement information of the compute nodes 110 held in the compute node layout table 414 is updated, the layout module 341 sends the updated information to the I / O monitor 322. The data configuration of the compute node layout table 414 may be similar to that of the compute node layout table 214 shown in Fig. 10, and detailed description thereof will be omitted.
[0134] The layout updater 411 has a function of updating the placement information of the compute node 110 held in the compute node layout table 414. The layout updater 411 may be software or hardware. The layout updater 411 receives an input of a compute node ID and an instance ID from a user interface 230 that allows a user to perform an input operation, queries the data center platform 240 for placement information of the compute node 110 (information of the data center 10 to which the compute node 110 belongs), and sends the processing results to the compute node layout table 414. Furthermore, the layout updater 411 also writes the above processing results to the layout table 331. Note that the information obtained by the layout updater 411 querying the data center platform 240 may be configured to be provided to the user side from the user interface 230. Furthermore, the layout updater 411 periodically checks the information in the layout table 331, and if there is a change, reflects the difference in the compute node layout table 414.
[0135] Fig. 25 is a diagram showing an example of the configuration of the I / O monitor 322 and the I / O state table 333. As shown in Fig. 25, the I / O monitor 322 is configured to include an I / O recorder 421 and a statistics module 422, and the I / O state table 333 is configured to include an intra-datacenter I / O table 431 and an inter-datacenter I / O table 432.
[0136] The I / O recorder 421 can identify the data center 10 to which the volume 323 and the compute node 110 belong by receiving and holding information on the volume layout table 412 and the compute node layout table 414 from the layout module 341. When the I / O recorder 421 receives an I / O log from the storage controller 321, the I / O recorder 421 uses the volume ID and the compute node ID to identify whether the I / O was processed by intra-datacenter communication or inter-datacenter communication. If the I / O was processed by intra-datacenter communication, the I / O recorder 421 sends the I / O log to the intra-datacenter I / O table 431, and if the I / O was processed by inter-datacenter communication, the I / O recorder 421 sends the I / O log to the inter-datacenter I / O table 432. Details of the process (communication characteristic recording process) by the I / O recorder 421 are shown in FIG. 27, which will be described later.
[0137] The intra-datacenter I / O table 431 and inter-datacenter I / O table 432 increment and hold the I / O size processed for each volume ID (see I / O information record 433 shown in FIG. 26). Furthermore, when the intra-datacenter I / O table 431 and inter-datacenter I / O table 432 receive a reset signal from the statistics module 422, they reset the held I / O size to zero.
[0138] The statistics module 422 is a program that has the function of responding with I / O statistical information (I / O traffic volume within a data center and I / O traffic volume between data centers) corresponding to a specified volume ID in response to a request from the tiebreaker 342.
[0139] More specifically, when the statistics module 422 receives a volume ID from the tiebreaker 342, it reads out the I / O size corresponding to the volume ID from the intra-datacenter I / O table 431 and the inter-datacenter I / O table 432. Then, the statistics module 422 converts each of the read I / O sizes into the traffic volume (throughput) of intra-datacenter communication associated with I / O processing and the traffic volume (throughput) of inter-datacenter communication associated with I / O processing by dividing the I / O size by the time elapsed since each table was last reset, and responds with these conversion results to the tiebreaker 342 as statistical information (I / O statistical information 442).
[0140] Furthermore, the statistics module 422 sends a reset signal to the intra-datacenter I / O table 431 and the inter-datacenter I / O table 432 at the timing when a reset request is received from the tiebreaker 342 or at regular intervals, and resets the I / O size held in each table to zero. If a reset request is received from the tiebreaker 342 immediately after the I / O size is reset by periodic processing or the like, the I / O size will be recorded for only a short period of time, and there is a possibility that sufficient I / O statistical information cannot be obtained to determine a method for stopping the volume 323 so as to minimize the amount of communication between datacenters. Therefore, when resetting at regular intervals, it is preferable to hold the I / O size before the reset for a specified period of time.
[0141] FIG. 26 is a diagram showing an example of the I / O information record 433. The I / O information record 433 is data that records I / O log information held by the intra-datacenter I / O table 431 and the inter-datacenter I / O table 432. In the case of FIG. 26, the I / O information record 433 is composed of a volume ID 4331 indicating a volume ID, and an I / O size 4332 indicating a cumulative total amount of data of I / O in the volume. The I / O information record 433 (the intra-datacenter I / O table 431 and the inter-datacenter I / O table 432) increments the processed I / O size for each volume 323 and holds it in the I / O size 4332, and when a reset signal is received from the statistics module 422, resets the value of the I / O size 4332 to zero.
[0142] Fig. 27 is a flowchart showing an example of a processing procedure of a communication characteristics recording process in the second embodiment. The communication characteristics recording process shown in Fig. 27 is a process in which the I / O monitor 322 (particularly the I / O recorder 421) records an I / O log in the intra-datacenter I / O table 431 or the inter-datacenter I / O table 432 as a record of communication characteristics accompanying I / O processing for each volume 323 in the storage system 300, and is started upon receiving an I / O log from the storage controller 321.
[0143] 27, first, the I / O monitor 322 checks the data center ID of the data center 10 to which the volume 323 indicated by the volume ID belongs, based on the volume ID included in the I / O log (step S301). When checking the data center ID from the volume ID, the volume layout table 412 shown in FIG. 23 can be referenced.
[0144] Next, the I / O monitor 322 checks the data center ID of the data center 10 to which the compute node 110 indicated by the compute node ID belongs based on the compute node ID included in the I / O log (step S302). When checking the data center ID from the compute node ID, it is sufficient to refer to the compute node layout table 414. The order of execution of steps S301 and S302 is not particularly limited, and they may be processed in parallel.
[0145] Next, the I / O monitor 322 compares the data center IDs confirmed in steps S301 and S302 to determine whether the volume 323 and the compute node 110 indicated in the I / O log belong to the same data center 10 (step S303).
[0146] If it is determined in step S303 that the volume 323 and the compute node 110 belong to the same data center 10 (YES in step S303), the I / O monitor 322 records the I / O log received at the start of processing in the I / O table 431 within the data center (step S304), and terminates the communication characteristics recording process.
[0147] On the other hand, if it is determined in step S303 that the volume 323 and the compute node 110 belong to different data centers 10 (NO in step S303), the I / O monitor 322 records the I / O log received at the start of the processing in the inter-data center I / O table 432 (step S305), and terminates the communication characteristics recording processing.
[0148] 28 is a diagram showing an example of the configuration of the tie breaker 342. As shown in FIG. 28, the tie breaker 342 includes a collector 451, a calculator 452 (452A, 452B), and an indicator 453.
[0149] The collector 451 has the function of receiving and storing information of the volume pair layout table 413 from the layout module 341, and further has the function of performing the following processing when it receives a volume pair ID that requires fault control from the network monitor 343.
[0150] When the collector 451 receives a volume pair ID requiring failure control from the network monitor 343 that has detected a communication failure between the volume pair 324, it identifies the two volumes 323 that make up the pair based on the volume pair ID, and queries the I / O monitor 322 for I / O statistical information on each of the volumes 323. Then, when the collector 451 receives the I / O statistical information queried from the I / O monitor 322 (the I / O statistical information 442 shown in Fig. 25), it transmits the information to the computers 452A, 452B.
[0151] The computer 452 (452A, 452B) has the function of calculating the inter-datacenter communication volume (inter-datacenter I / O throughput) when the volume 323 subject to failure control stops based on I / O statistical information, and sending it to the indicator 453.
[0152] Specifically, in the case of Fig. 28, the computer 452A refers to the I / O statistical information and adds up the I / O throughput of the intra-datacenter communication in volume ID #1 and the I / O throughput of the inter-datacenter communication in volume ID #2 to calculate the inter-datacenter communication volume ("T1" in Fig. 28) when the first volume of the volume pair (volume #1 in this example) stops. Also, the computer 452B refers to the I / O statistical information and adds up the I / O throughput of the intra-datacenter communication in volume ID #2 and the I / O throughput of the inter-datacenter communication in volume ID #1 to calculate the inter-datacenter communication volume ("T2" in Fig. 28) when the second volume of the volume pair (volume #2 in this example) stops.
[0153] The indicator 453 has a function of receiving and storing information on the volume layout table 412 and the volume pair layout table 413 from the layout module 341, and further has a function of performing the following processing when it receives the calculation results (T1, T2 in Figure 28) of the throughput between the data centers when one of the volumes constituting the volume pair stops from the computers 452A, 452B.
[0154] The indicator 453 compares the inter-datacenter communication volume (inter-datacenter I / O throughput) T1 received from the computer 452A with the inter-datacenter communication volume (inter-datacenter I / O throughput) T2 received from the computer 452B, judges whether to stop the volume 323 so as to reduce the inter-datacenter communication volume (throughput), and requests the storage cluster controller 340 to stop the target volume 323 according to the judgment result (for details, see FIG. 30 described later). Furthermore, the indicator 453 writes to the volume state table 332 that the target volume 323 is to be put into a stopped state.
[0155] Fig. 29 is a diagram showing an example of the volume state table 332. The volume state table 332 is a table that holds state information for each volume 323, and in the case of Fig. 29, is made up of a volume ID 3321 indicating the volume ID, and a state 3322 indicating the state of the volume. In the state 3322, the state of the volume is indicated as an active state ("Active") or a blocked state ("Block"). The active state indicates that the volume 323 is in a state where I / O is possible, and the blocked state indicates that the volume 323 is in a state where I / O is not possible.
[0156] Fig. 30 is a flowchart showing an example of a processing procedure of a control method decision process for a communication failure in the second embodiment. The control method decision process shown in Fig. 30 is a process executed by the tiebreaker 342, and is started when the tiebreaker 342 receives a volume pair ID that requires failure control from the network monitor 343 that has detected a communication failure between the volumes.
[0157] 30, first, the collector 451 of the tiebreaker 342 extracts the volume IDs of the two paired volumes 323 based on the volume pair ID received from the network monitor 343 (step S401). In step S401, the collector 451 can extract the volume IDs of the two paired volumes 323 by obtaining the corresponding volume ID 4132 from the volume pair layout table 413 using the received volume pair ID as a key. In this description, as a specific example using FIG. 28 etc., it is assumed that volume ID #1 "V001" and volume ID #2 "V002" are extracted based on the reception of volume pair ID #1.
[0158] Next, the collector 451 queries the I / O monitor 322 to acquire information on communication characteristics (I / O statistical information 442) of the two volumes 323 corresponding to the volume IDs extracted in step S401 (step S402).The collector 451 then transmits the acquired I / O statistical information 442 of the two volumes 323 (volumes #1 and #2) to the computers 452A and 452B.
[0159] Next, the computer 452A uses the I / O statistical information 442 received in step S402 to calculate "T1", the estimated traffic volume of communication between data centers when the first volume 323 (e.g., volume #1) in the volume pair 324 in which a communication failure was detected stops (step S403). The method of calculating "T1" is as described in the explanation of the computer 452A with reference to Fig. 28.
[0160] Next, the computer 452B uses the I / O statistics information 442 received in step S402 to calculate "T2", the estimated traffic volume between data centers when the first volume 323 (e.g., volume #2) in the volume pair 324 in which a communication failure was detected stops (step S404). The method of calculating "T2" is as described in the explanation of the computer 452B with reference to Fig. 28.
[0161] Then, the computer 452A transmits the inter-datacenter communication volume "T1" calculated in step S403 to the indicator 453, and the computer 452B transmits the inter-datacenter communication volume "T2" calculated in step S404 to the indicator 453. Note that the order of execution of steps S403 and S404 is not particularly limited, and they may be processed in parallel.
[0162] Next, the indicator 453 compares the calculated inter-datacenter communication volume "T1" with the inter-datacenter communication volume "T2" (step S405), and judges whether the value of T1 is equal to or greater than the value of T2 (step S406). If the value of T1 is equal to or greater than the value of T2 (YES in step S406), the process proceeds to step S407, and if the value of T1 is less than the value of T2 (NO in step S406), the process proceeds to step S408.
[0163] Here, the relationship "T1≧T2" means that if the first volume #1 of the two volumes #1, #2 that make up volume pair #1 is stopped, the amount of inter-datacenter communication occurring between datacenters 10 due to I / O processing will be greater than if the second volume #2 is stopped.
[0164] Therefore, when "T1≧T2" in step S406 (YES in step S406), as a control method for reducing the inter-data center communication volume after failure control (after volume stop), the indicator 453 determines to stop the second volume #2, requests the storage cluster controller 340 to perform control in accordance with this determination (or writes to the volume state table 332 to set the volume #2 (volume ID "V002") to the stopped state) (step S407), and ends the control method determination process.
[0165] On the other hand, the relationship "T1<T2" means that when the second volume #2 among the two volumes #1 and #2 constituting the volume pair #1 is stopped, the inter-data center communication volume generated between the data centers 10 due to I / O processing becomes larger than when the first volume #1 is stopped.
[0166] Therefore, when "T1<T2" in step S406 (NO in step S406), as a control method for reducing the inter-data center communication volume after failure control (after volume stop), the indicator 453 determines to stop the first volume #1, requests the storage cluster controller 340 to perform control in accordance with this determination (or writes to the volume state table 332 to set the volume #2 (volume ID "V002") to the stopped state) (step S408), and ends the control method determination process.
[0167] As described above, in the computer system 2 according to the second embodiment, by executing the communication characteristic recording process shown in Fig. 27 and the control method determination process shown in Fig. 30, in the volume pair 324 formed between the volumes 323 in the storage clusters 310 of different data centers 10, when a communication failure occurs between the volumes, the failure control method for stopping one of the two volumes 323 constituting the volume pair 324 can be determined so as to reduce the communication volume (throughput) between the data centers involved in I / O processing. Here, since the two volumes 323 constituting the volume pair 324 operated in synchronization with data in an active-active state, even after the storage cluster controller 340 stops one of the volumes 323 according to the determined failure control method, the other volume 323 can receive I / O access. As a result, as explained with reference to the conceptual diagram of Figure 21, the storage system 300 can respond to I / O requests from one or more compute nodes 110 using volumes 323 that have not been suspended while suppressing the amount of inter-datacenter communication via the inter-datacenter path 152, and can achieve fault control that suppresses the occurrence of charges associated with I / O processing and increases in responses.
[0168] (3) Third embodiment The third embodiment is an extension of the first embodiment, and adds a function of proposing a placement method of the compute nodes 110 that can further reduce the amount of communication between data centers after the storage controller is stopped by communication failure control. In the following explanation, the configuration unique to the third embodiment will be mainly explained based on the configuration according to the first embodiment.
[0169] Fig. 31 is a conceptual diagram of a computer system 3 according to a third embodiment of the present invention. Fig. 31 also shows an image of communication failure control in the third embodiment.
[0170] As shown in Fig. 31, a storage system 500 (storage cluster 520) included in the computer system 3 is different from the storage system 100 (storage cluster 120) in the first embodiment shown in Fig. 1 in that it includes a storage cluster controller 540 whose internal configuration is different from that of the storage cluster controller 140. Also, a compute manager 560 is added to the computer system 3 as a difference from the computer system 1 according to the first embodiment.
[0171] The storage cluster controller 540 in the third embodiment is configured to include a layout module 541, a tiebreaker 542, a network monitor 543, and a migration proposer 544. Among these, the configurations other than the migration proposer 544 are similar to the configurations of the same names that the storage cluster controller 140 in the first embodiment has, but there is a difference from the first embodiment in a part of the processing by the tiebreaker 542. The storage cluster controller 540 will be described in detail below.
[0172] When a communication failure occurs in the controller pair 134, the tiebreaker 542, like the tiebreaker 142 in the first embodiment, determines a method for stopping the storage controller 131, and requests the storage cluster controller 540 to control stopping the determined storage controller 131. At this time, the tiebreaker 542 also passes the ID of the active storage controller to be stopped to the migration proposer 544.
[0173] Specifically, in the case of FIG. 31, the tiebreaker 542 decides to shut down storage controller #2 and storage controller #4 belonging to storage node #2, and passes the storage controller ID "SC004" of storage controller #4, which is in an active state, to the migration proposal 544.
[0174] The migration proposer 544 has a function of proposing a placement method of the compute nodes 110 that is expected to further reduce the amount of communication between data centers in a state after a specific storage controller 131 is stopped. When the migration proposer 544 receives a storage controller ID from the tiebreaker 542, the migration proposer 544 acquires configuration information of the storage system 500 and placement information of the compute nodes 110 from the layout module 541. Then, based on the acquired information, the migration proposer 544 calculates a placement method of the compute nodes 110 that is expected to reduce the amount of communication between data centers in a state after the storage controller 131 is stopped due to failure control, and proposes this placement method to the compute manager 560.
[0175] The compute manager 560 is a mechanism for controlling the placement and operation of the compute nodes 110. When the compute manager 560 receives a proposal for a placement method of the compute nodes from the migration proposer 544, the compute manager 560 judges whether or not to implement the proposal. The judgment may be made based on a program prepared in advance, or may be made by accepting a judgment by a user. Then, when it is decided to implement the proposal, the compute manager 560 changes the placement of the compute nodes 110 according to the contents of the proposal.
[0176] Specifically, in the case of FIG. 31, the compute manager 560 controls the compute node #1 (compute node 110A) and the compute node #2 (compute node 110B). In a state before a communication failure occurs and failure control is performed, the compute node #1 is placed in the data center 10A, and the compute node #2 is placed in the data center #2. After that, the storage controllers #2 and #4 in the data center #2 are stopped by the failure control, and then the migration proposer 544 proposes placing the compute node #2 in the data center #1 as a placement method for further reducing the amount of communication between the data centers. In response to this proposal, the compute manager 560 moves the compute node #2 that was placed in the data center #2 to the data center #1.
[0177] As a result of the placement control of the compute nodes 110 being performed by the compute manager 560 as described above, the compute nodes #1 and #2 are placed in the data center #1. In this state, in the storage system 500, the paths of the compute nodes #1 and #2 to the storage cluster 520 (storage system 500) are only the intra-data center paths 551A and 551B in the data center #1 as shown in Fig. 31, and communication between the data centers can be reduced to zero.
[0178] Fig. 32 is a diagram showing a configuration example of the migration proposer 544. As shown in Fig. 32, the migration proposer 544 is configured to include a data center searcher 611 (individually, a data center searcher 611A and a data center searcher 611B), a compute node searcher 612, a storage controller searcher 613, a proposal generator 614, and a filter 615.
[0179] The data center searcher 611 receives and holds information on the storage controller layout table from the layout module 541. The internal configuration of the layout module 541 may be considered to be similar to that of the layout module 141 shown in FIG. 7 in the first embodiment, and includes a layout updater 211, a storage controller layout table 212, a controller pair layout table 213, and a compute node layout table 214. Therefore, the information on the storage controller layout table received by the data center searcher 611 may be considered to be the information on the storage controller layout table 212 shown in FIG. 7 or FIG. 8. When the data center searcher 611 receives a storage controller ID ("SC ID (Active)" shown in FIG. 32) from the tiebreaker 542 as an input, the data center searcher 611 uses the stored storage controller layout table to search for and output the data center ID of the data center 10 in which the storage controller 131 indicated by the input storage controller ID is located ("DC ID" shown in FIG. 32).
[0180] The compute node searcher 612 receives and holds information of a compute node layout table (for example, the compute node layout table 214 shown in FIG. 10) from the layout module 541. When the compute node searcher 612 receives a data center ID ("DC ID" shown in FIG. 32) as input from the data center searcher 611, the compute node searcher 612 uses the compute node layout table to search for the compute node IDs of all the compute nodes 110 arranged in the data center 10 indicated by the input data center ID, and outputs the compute node IDs as a list ("CN ID" shown in FIG. 32).
[0181] The storage controller searcher 613 receives and holds information of a controller pair layout table (for example, the controller pair layout table 213 shown in FIG. 9) from the layout module 541. When the storage controller searcher 613 receives a storage controller ID ("SC ID (Active)" shown in FIG. 32) as an input from the tiebreaker 542, it uses the controller pair layout table it holds to search for and output the storage controller ID of the other storage controller 131 in the controller pair 134 with which the storage controller 131 indicated by the input storage controller ID is paired ("SC ID (Standby)" shown in FIG. 32).
[0182] When the migration proposer 544 having the above internal configuration receives one active storage controller ID as input from the tiebreaker 542, it uses the data center searcher 611A and the compute node searcher 612 to obtain a list of IDs of compute nodes 110 that belong to the same data center 10 as the storage controller 131 indicated by the received storage controller ID. This list of compute node IDs is a list including the compute nodes 110 to be moved in the placement method proposed by the migration proposer 544, and is internally input to the proposal generator 614.
[0183] Furthermore, the migration proposer 544 uses the storage controller searcher 613 and the data center searcher 611B to obtain the ID of the data center 10 to which the other storage controller 131 that forms a pair with the storage controller 131 indicated by the active storage controller ID received as an input from the tiebreaker 542 belongs. This data center ID is the ID of the data center 10 to which the compute node 110 is to be moved in the placement method proposed by the migration proposer 544, and is internally input to the proposal generator 614.
[0184] Next, the process within the migration proposer 544 after the list of compute node IDs "CD IN" and the data center ID "DC ID" are input to the proposal generator 614 as described above will be described.
[0185] The proposal generator 614 generates a proposal to move the compute node 110 based on the input list of compute node IDs and the data center ID, and outputs the proposal to the filter 615. Specifically, for example, if the list of compute node IDs is composed of "CN002 (compute node ID #2)" and the data center ID is "DC001 (data center ID #1)", a proposal to "move compute node #2 to data center #1" is generated.
[0186] The filter 615 judges whether or not to pass the proposal generated by the proposal generator 614 to the compute manager 560. In the initial state, the filter 615 is in the off state (Filter OFF), and the filter 615 in the off state passes the proposal to the compute manager 560. On the other hand, once a proposal has been passed to the compute manager 560, the filter 615 becomes in the on state (Filter ON), and the filter 615 in the on state rejects all subsequent proposals without passing them to the compute manager 560 (until it next changes to the off state).
[0187] The reason why the filter 615 rejects the proposal of the proposal generator 614 is that if a change in the placement of the compute node 110 is proposed multiple times while the control to stop the storage controller 131 is being performed, there is a possibility that the compute node 110 will not stay at its fixed position. Therefore, by filtering the second and subsequent proposals, the movement of the compute node during a communication failure is limited to one time. The on state of the filter 615 is switched to the off state when a predetermined instruction is issued from the storage cluster controller 540. Specifically, the storage cluster controller 540 instructs the filter 615 to be switched to the off state when the storage cluster 520 (storage system 500) recovers from a failure.
[0188] As described above, in the computer system 3 according to the third embodiment, similarly to the computer system 1 according to the first embodiment, when a communication failure occurs between a controller pair, not only can the computer system 3 determine a failure control method for stopping all storage controllers 131 belonging to any of the storage nodes 130 so as to reduce the amount of communication (throughput) between data centers accompanying I / O processing, and can stop the storage controllers 131 according to the determined failure control method, but also can propose a placement method for the compute nodes 110 so as to further reduce (reduce to zero) the amount of communication between data centers. Then, the compute manager 560 can change the placement of the compute nodes 110 if the proposal is accepted.
[0189] (4) Fourth embodiment The fourth embodiment is an extension of the second embodiment, and adds a function of proposing a placement method of the compute nodes 110 that can further reduce the amount of communication between data centers after a volume has been stopped by communication failure control. In the following explanation, the configuration unique to the fourth embodiment will be mainly explained based on the configuration according to the second embodiment.
[0190] Fig. 33 is a conceptual diagram of a computer system 4 according to a fourth embodiment of the present invention. Fig. 33 also shows an image of communication failure control in the fourth embodiment.
[0191] As shown in Fig. 33, the storage system 700 included in the computer system 4 is different from the storage system 300 in the second embodiment shown in Fig. 19 in that it includes a storage cluster controller 740 whose internal configuration is different from that of the storage cluster controller 340. Also, a compute manager 760 is added to the computer system 4 as a difference from the computer system 2 according to the second embodiment.
[0192] The storage cluster controller 740 in the fourth embodiment is configured to include a layout module 741, a tiebreaker 742, a network monitor 743, a state checker 744, and a migration proposer 745. Among these, the configurations other than the migration proposer 745 are similar to the configurations of the same names that the storage cluster controller 340 in the second embodiment has, but there is a difference from the second embodiment in a part of the processing by the tiebreaker 742. The storage cluster controller 740 will be described in detail below.
[0193] When a communication failure occurs in the volume pair 324, the tiebreaker 742, like the tiebreaker 342 in the second embodiment, determines a method for stopping the volume 323, and requests the storage cluster controller 740 to control stopping the determined volume 323. At this time, the tiebreaker 742 also passes the ID of the volume 323 to be stopped to the migration proposer 745.
[0194] Specifically, in the case of Figure 33, the tiebreaker 742A of the storage cluster controller 740A (storage cluster controller #1) decides to stop volume #2 and volume #4 located in storage cluster #2, and passes the IDs of these two volumes, "V002" and "V004", to the migration proposer 745.
[0195] The migration proposer 745 has a function of proposing a placement method of the compute nodes 110 that is expected to further reduce the amount of communication between data centers in a state after a specified volume 323 is stopped. When the migration proposer 745 receives a volume ID from the tiebreaker 742, it acquires configuration information of the storage system 700 and placement information of the compute nodes 110 from the layout module 741. Then, based on the acquired information, the migration proposer 745 calculates a placement method of the compute nodes 110 that is expected to reduce the amount of communication between data centers in a state after the volume 323 is stopped by failure control, and proposes this placement method to the compute manager 760.
[0196] The compute manager 760 is a mechanism for controlling the placement and operation of the compute nodes 110. When the compute manager 760 receives a proposal for a placement method of the compute nodes from the migration proposer 745, the compute manager 760 judges whether or not to implement the proposal. The judgment may be made based on a program prepared in advance, or may be made by accepting a judgment by a user. Then, when it is decided to implement the proposal, the compute manager 760 changes the placement of the compute nodes 110 according to the contents of the proposal.
[0197] Specifically, in the case of FIG. 33, the compute manager 760 controls the compute node #1 (compute node 110A) and the compute node #2 (compute node 110B). In a state before a communication failure occurs and failure control is performed, the compute node #1 is placed in the data center 10A, and the compute node #2 is placed in the data center #2. After that, the volumes #2 and #4 in the data center #2 are stopped by the failure control, and then the migration proposer 745 proposes placing the compute node #2 in the data center #1 as a placement method for further reducing the amount of communication between the data centers. In response to this proposal, the compute manager 760 moves the compute node #2 that was placed in the data center #2 to the data center #1.
[0198] As a result of the placement control of the compute nodes 110 being performed by the compute manager 760 as described above, the compute nodes #1 and #2 are placed in the data center #1. In this state, in the storage system 700, the paths of the compute nodes #1 and #2 to the storage system 700 are only the intra-data center paths 751A and 751B in the data center #1 as shown in Fig. 33, and communication between the data centers can be reduced to zero.
[0199] Fig. 34 is a diagram showing a configuration example of the migration proposer 745. As shown in Fig. 34, the migration proposer 745 is configured to include a data center searcher 811 (individually, a data center searcher 811A and a data center searcher 811B), a compute node searcher 812, a volume searcher 813, a proposal generator 814, and a filter 815.
[0200] The data center searcher 811 receives and holds volume layout table information from the layout module 741. The internal configuration of the layout module 741 may be considered to be similar to that of the layout module 341 shown in FIG. 22 in the second embodiment, and includes a layout updater 411, a volume layout table 412, a volume pair layout table 413, and a compute node layout table 414. Therefore, the volume layout table information received by the data center searcher 811 may be considered to be the information of the volume layout table 412 shown in FIG. 22 or FIG. 23. When the data center searcher 811 receives a volume ID ("Vol ID" shown in FIG. 34) from the tiebreaker 742 as an input, it uses the held volume layout table to search for and output the data center ID of the data center 10 in which the volume 323 indicated by the input volume ID is located ("DC ID" shown in FIG. 34).
[0201] The compute node searcher 812 receives and holds information on a compute node layout table (for example, the compute node layout table 214 shown in FIG. 10) from the layout module 741. When the compute node searcher 812 receives a data center ID ("DC ID" shown in FIG. 34) as input from the data center searcher 811, the compute node searcher 812 uses the compute node layout table to search for the compute node IDs of all the compute nodes 110 arranged in the data center 10 indicated by the input data center ID, and outputs the results as a list ("CN ID" shown in FIG. 34).
[0202] The volume searcher 813 receives and holds information on a volume pair layout table (for example, the volume pair layout table 413 shown in FIG. 24) from the layout module 741. When the volume searcher 813 receives a volume ID ("Vol ID" shown in FIG. 34) as an input from the tiebreaker 742, it uses the volume pair layout table it holds to search for and output the volume ID of the other volume 323 in the volume pair 324 with which the volume 323 indicated by the input volume ID is paired ("Vol ID" shown in FIG. 34).
[0203] When the migration proposer 745 having the above internal configuration receives one volume ID as input from the tiebreaker 742, it uses the data center searcher 811A and the compute node searcher 812 to obtain a list of IDs of compute nodes 110 that belong to the same data center 10 as the volume 323 indicated by the received volume ID. This list of compute node IDs is a list including the compute nodes 110 to be migrated in the placement method proposed by the migration proposer 745, and is internally input to the proposal generator 814.
[0204] Furthermore, the migration proposer 745 uses the volume searcher 813 and the datacenter searcher 611B to obtain the ID of the datacenter 10 to which the other volume 323 that forms a pair with the volume 323 indicated by the volume ID received as an input from the tiebreaker 742 belongs. This datacenter ID is the ID of the datacenter 10 to which the compute node 110 is to be migrated in the placement method proposed by the migration proposer 544, and is internally input to the proposal generator 814.
[0205] Next, the process within the migration proposer 745 after the list of compute node IDs "CD IN" and the data center ID "DC ID" are input to the proposal generator 814 as described above will be described.
[0206] The proposal generator 814 generates a proposal to move the compute node 110 based on the input list of compute node IDs and data center ID, and outputs the proposal to the filter 815. Specifically, for example, if the list of compute node IDs is composed of "CN002 (compute node ID #2)" and the data center ID is "DC001 (data center ID #1)", a proposal to "move compute node #2 to data center #1" is generated.
[0207] The filter 815 judges whether or not to pass the proposal generated by the proposal generator 814 to the compute manager 760. In the initial state, the filter 815 is in the off state (Filter OFF), and the filter 815 in the off state passes the proposal to the compute manager 760. On the other hand, once a proposal has been passed to the compute manager 760, the filter 815 becomes in the on state (Filter ON), and the filter 815 in the on state rejects all subsequent proposals without passing them to the compute manager 760 (until it next changes to the off state).
[0208] The reason why the filter 815 rejects the proposal of the proposal generator 814 is that if a change in the placement of the compute node 110 is proposed multiple times while the control to stop the volume 323 is being performed, there is a possibility that the compute node 110 will not stay at its fixed position. Therefore, by filtering the second and subsequent proposals, the movement of the compute node during a communication failure is limited to one time. The on state of the filter 815 is switched to the off state when a predetermined instruction is issued from the storage cluster controller 740. Specifically, the storage cluster controller 740 instructs the filter 815 to be in the off state when the storage system 700 recovers from the failure.
[0209] As described above, in the computer system 4 according to the fourth embodiment, similarly to the computer system 2 according to the second embodiment, when a communication failure occurs between a volume pair, not only can it determine a failure control method for stopping one of the volume pair disconnected due to a communication failure so as to reduce the amount of communication (throughput) between data centers accompanying I / O processing, and can stop the volume 323 according to the determined failure control method, but it can also propose a placement method for the compute nodes 110 so as to further reduce (reduce to zero) the amount of communication between data centers.The compute manager 760 can then change the placement of the compute nodes 110 if it accepts the proposal.
[0210] Various embodiments and modifications of the computer system according to the present invention have been described above, but these are merely examples for explaining the present invention, and the scope of the present invention is not limited to these embodiments. The present invention may be implemented in various other forms, or by combining the various embodiments as appropriate. As shown in these embodiments, the computer system according to the present invention includes an I / O monitor that collects and records communication characteristics related to I / O with an application (compute node) for a storage controller or a volume in each storage node (or storage cluster) arranged in a plurality of data centers, and a mechanism (tiebreaker) that determines failure control when a communication failure occurs. When the storage system recognizes a communication failure in the communication path between the pair, the failure control determination mechanism collects statistical information on communication characteristics from the I / O monitor, and determines failure control to stop the storage controller or the volume so that the amount of communication between the data centers is minimized. With such a configuration, when a communication failure occurs in a storage system (e.g., a storage system operating in a cloud environment) in which multiple computers are connected via a network across multiple data centers, the storage system performs failure control in accordance with the determination by the above mechanism to minimize the amount of communication between the data centers, thereby suppressing the occurrence (or increase) of charges associated with I / O processing and suppressing (and most preferably minimizing) increases in responses in I / O processing. [Explanation of symbols]
[0211] 1,2,3,4 Computer System 10 Data Center 20 Servers 21 CPU 22 Memory 23 Network Interface 24 Drive 30 Data Center Network 40 Inter-datacenter network 100, 300, 500, 700 Storage Systems 110 Compute Nodes 120,290,320,520 Storage Cluster 130 storage nodes 131,321 Storage Controllers 132,322 I / O monitor 133,323 Volume 134 Controller Pairs 140,340,540,740 Storage Cluster Controller 141,341,541,741 Layout Module 142,342,542,742 Tiebreaker 143,343,543,743 Network Monitor 151,551,751 Datacenter Paths 152 Inter-datacenter Paths 160 Network Switch 211,411 Layout Updater 212 Storage Controller Layout Table 213 Controller Pair Layout Table 214,414 Compute Node Layout Table 220 Configuration File 230 User Interface 240 Data Center Infrastructure 251,421 I / O Recorder 252,422 Statistics Module 253,431 I / O table in data center 254,432 Datacenter I / O table 255,433 I / O records 261,441 I / O logs 262,442 I / O statistics 271,451 Collectors 272,452 calculator 273,453 Indicator 280 Registration screen 324 volume pairs 330 Quorum Disk 331 Layout Table 332 Volume State Table 333 I / O State Table 344,744 State Checker 412 Volume Layout Table 413 Volume Pair Layout Table 560,760 Compute Manager 611,811 Data Center Searcher 612,812 Compute Node Searcher 613 Storage Controller Searcher 614,814 Proposal Generator 615,815 Filters 813 Volume Searcher
Claims
1. It is composed of multiple computers located in multiple data centers connected via a network, Compute nodes that run applications; A storage node having a storage controller that processes I / O from the compute node and inputs / outputs data to / from a storage device; a storage cluster controller that controls the storage nodes; A computer system comprising: Storage controllers arranged in different data centers form a pair, and one storage controller of the pair communicates with the other storage controller via a communication path between the data centers so that the other storage controller can take over the processing of the other storage controller; an I / O monitor that collects communication characteristics between the application and the storage controller and performs predetermined statistical processing to generate statistical information; When a communication failure occurs in a communication path between the pair of storage controllers, a tiebreaker included in the storage cluster controller determines failure control for taking over data input / output from one of the pair of storage controllers to the other storage controller based on statistical information of the communication characteristics of the storage controllers generated by the I / O monitor, and for stopping a storage node having the one storage controller; The storage cluster controller executes the failure control determined by the tiebreaker; The application that was performing I / O to the storage controller of the stopped storage node sends the I / O to the storage controller that took over the processing.
1. A computer system comprising:
2. The compute node and the storage node are connected to other data centers by different communication paths, An application on the compute node can send an I / O request to a storage controller of a storage node in another data center; It has multiple applications and multiple pairs of storage controllers. A storage controller that processes the application and its I / O is disposed in each data center so as to reduce the amount of communication between the data centers; The tiebreaker determines which storage node to stop so that, when the storage node is stopped, an increase in the amount of communication between the data centers is reduced.
2. The computer system according to claim 1 .
3. When storage controllers arranged in different data centers form a controller pair via a communication path between the data centers, The two storage controllers constituting the controller pair are configured such that one manages a volume in an active state and the other waits in a standby state; When a first storage controller in an active state is stopped due to execution of the failure control, a second storage controller in a standby state that forms a controller pair with the first storage controller becomes active and takes over management of the volume by the first storage controller.
3. The computer system according to claim 2.
4. When the controller pair is configured by storage controllers of storage nodes arranged in different data centers, and a communication failure occurs in a communication path between the data centers in the controller pair, The tiebreaker determines the failure control in such a manner that all storage controllers in a storage node having one of the storage controllers constituting the controller pair are stopped, so that the amount of communication between the data centers after the stoppage is minimized.
4. The computer system according to claim 3.
5. When the controller pair is configured by storage controllers of storage nodes arranged in different data centers, and a communication failure occurs in a communication path between data centers in two or more sets of storage nodes that configure the controller pair, The tiebreaker determines the failure control independently for each set of storage nodes in which the controller pair is configured.
4. The computer system according to claim 3.
6. The storage controller provides volumes to the applications for I / O; When volumes located in different data centers form a volume pair via a communication path between the data centers, The two volumes constituting the volume pair are both configured to operate in an active state and synchronize data; When the first volume constituting the volume pair is stopped due to the execution of the failure control, the second volume forming the volume pair with the first volume continues to receive I / O.
3. The computer system according to claim 2.
7. When the volume pair is configured with volumes owned by storage nodes arranged in different data centers, and a communication failure occurs in a communication path between data centers in two or more of the volume pairs, The tiebreaker determines the failure control independently for each of the volume pairs.
7. The computer system according to claim 6.
8. The storage cluster controller further includes a migration proposer that proposes a change in the placement of the application in the plurality of data centers based on storage system configuration information and application placement information when the storage controller or a volume managed by the storage controller is stopped due to execution of the failure control, so as to minimize the amount of communication in a communication path between the data centers after the change.
8. The computer system according to claim 1, wherein the computer system comprises:
9. The migration proposer outputs a proposal for changing the placement of the application to a mechanism that controls the placement and operation of the application.
9. The computer system according to claim 8.
10. A storage control method for a computer system configured by connecting a plurality of computers disposed in a plurality of data centers via a network, comprising: The computer system comprises: Compute nodes that run applications; A storage node having a storage controller that processes I / O from the compute node and inputs / outputs data to / from a storage device; a storage cluster controller that controls the storage nodes; having Storage controllers arranged in different data centers form a pair, and one storage controller of the pair communicates with the other storage controller via a communication path between the data centers so that the other storage controller can take over the processing of the other storage controller; an I / O monitor that collects communication characteristics between the application and the storage controller and performs predetermined statistical processing to generate statistical information; When a communication failure occurs in a communication path between the pair of storage controllers, a tiebreaker included in the storage cluster controller determines failure control for transferring data input / output from one of the pair of storage controllers to the other storage controller based on statistical information of the communication characteristics of the storage controllers generated by the I / O monitor, and for stopping a storage node having the one storage controller; The storage cluster controller executes the failure control determined by the tiebreaker; The application that was performing I / O to the storage controller of the stopped storage node sends the I / O to the storage controller that took over the processing. A storage control method comprising:
Citation Information
Patent Citations
Methods, devices, and media for performing switching operations between computing nodes
JP2018536229A
Synchronously replicating datasets and other managed objects to a cloud-based storage system
JP2022122993A
Storage system
WO2015198449A1
