Storage system and storage system control method

Abstract

To reduce an influence of a maintenance event on performance.SOLUTION: In a storage system including a plurality of storage nodes each having an arithmetic device and a memory, the plurality of storage nodes take over a failed storage node by failover when detecting occurrence of a failure in another storage node in the storage system, and when a maintenance event occurs in the storage system, the plurality of storage nodes change a detection condition of the failure for a storage node related to the maintenance event based on maintenance event information that is information on the maintenance event, and restrict data input / output processing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a storage system and a control method for a storage system. [Background technology]

[0002] Storage systems are known that are configured by connecting storage nodes operating on computers via a network. In recent years, storage systems are sometimes configured using computing resources in a cloud computing environment. When using computing resources in a cloud computing environment, it is necessary to consider maintenance performed by the cloud vendor. Virtual machines, which are typical computing resources, may be forcibly terminated or restarted. Technologies for dealing with such maintenance are available, such as those described in Patent Document 1 and Patent Document 2.

[0003] Patent Document 1 states that "In a cluster system, availability is ensured while reducing operational costs by using instances that may be forcibly terminated." and "The second server 30 is a server to which service execution is failed over when a failure occurs in the execution of a service on the first server 20. The second server 30 is a virtual server created in the cloud environment 50 as a first type instance that may be forcibly terminated by the cloud service provider. The instance monitoring means 22 monitors whether the second server 30 is to be forcibly terminated. If the instance operation means 23 detects that the second server 30 is to be forcibly terminated, it causes the cloud environment 50 to create a third server 40 as a first type instance. The instance operation means 23 causes the third server 40 to take over the functions provided by the second server 30."

[0004] Patent Document 2 states, "We provide a storage system that can perform maintenance that enables stable operation of a storage cluster based on a maintenance plan for the storage cluster." and "A processor operates a storage node on each of multiple servers, groups multiple storage nodes together to set up a storage cluster, compares the maintenance plan for the storage cluster with the state of the storage cluster, and can change the maintenance plan based on the comparison results, and maintenance for the storage cluster is performed based on the changed maintenance plan." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-129184 [Patent Document 2] Japanese Patent Application Publication No. 2023-151189 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, it is possible to detect that a virtual machine is being forcibly terminated and create a substitute virtual machine to take over the functions. In Patent Document 2, it is possible to receive a maintenance plan to restart a virtual machine, and if the storage system cannot accept the maintenance plan, it is possible to request a change to the plan. As mentioned above, there are technologies that take measures against maintenance performed by cloud vendors, but there are problems that cannot be solved by these technologies.

[0007] Some maintenance performed by cloud vendors does not involve restarting virtual machines. In this case, the CPU (Central Processing Unit) and network are temporarily stopped while maintaining the contents of the virtual machine's memory. When the CPU and network are stopped, the storage node targeted for maintenance is unable to respond to the storage system's liveness monitoring function and is determined to have failed. Therefore, maintenance that does not involve restart requires countermeasures such as shutting down the target storage node in advance. However, simply shutting down the storage node as a countermeasure reduces the redundancy and availability of the storage system. Furthermore, because the impact of maintenance that does not involve restart on virtual machines is minimal, it is often not possible to change the plan for such maintenance. In other words, the plan cannot be changed as described in Patent Document 2. Alternatively, when performing maintenance to restart a virtual machine, measures such as shutting down the target storage node beforehand are required. In this case, the number of operating storage nodes required for the storage system to operate may not be met, causing the storage system to come to an emergency shutdown, which could result in data integrity not being guaranteed.

[0008] In this way, when maintenance is performed on the computing resources that make up a storage system, performance is affected, such as a decrease in redundancy, a decrease in availability, or a shutdown of the storage system. Therefore, an object of the present invention is to reduce the impact on performance caused by a maintenance event in a storage system. [Means for solving the problem]

[0009] In order to achieve the above object, one representative storage system of the present invention is a storage system having a plurality of storage nodes each having an arithmetic unit and a memory, wherein when the plurality of storage nodes detects a failure in another storage node within the storage system, the plurality of storage nodes take over from the storage node in which the failure occurred by failover, and when a maintenance event occurs in the storage system, the plurality of storage nodes change the failure detection conditions for the storage node associated with the maintenance event based on maintenance event information, which is information related to the maintenance event, and restrict data input / output processing. Furthermore, one representative control method for a storage system of the present invention is a control method for a storage system having a plurality of storage nodes each having an arithmetic unit and a memory, wherein, when the plurality of storage nodes detect a failure in another storage node within the storage system, the plurality of storage nodes take over from the storage node in which the failure occurred by failover, and when a maintenance event occurs in the storage system, the plurality of storage nodes change the failure detection conditions for the storage node associated with the maintenance event based on maintenance event information, which is information related to the maintenance event, and restrict data input / output processing. [Effects of the Invention]

[0010] According to the present invention, it is possible to reduce the impact on performance of a storage system caused by a maintenance event. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiment. [Brief explanation of the drawings]

[0011] [Figure 1] Storage system configuration diagram [Figure 2] Storage system network configuration diagram [Figure 3]Storage system hardware configuration diagram [Figure 4] Event monitoring mechanism configuration diagram [Figure 5] Event Data Table [Figure 6] Event monitoring mechanism operation flowchart [Figure 7] Configuration diagram of the event control mechanism of the first embodiment [Figure 8] Resource Information Table and Capacity Group Information Table [Figure 9] 1 is a control flowchart of the event control mechanism according to the first embodiment. [Figure 10] Freeze event control flowchart of the event control mechanism [Figure 11] Control end flowchart of the event control mechanism of the first embodiment [Figure 12] Freeze event control end flowchart for the event control mechanism [Figure 13] Example of displaying the storage cluster status using event control [Figure 14] Configuration diagram of the event control mechanism of the second embodiment [Figure 15] Storage cluster information tables [Figure 16] Control flowchart of the event control mechanism of the second embodiment [Figure 17] Control end flowchart of the event control mechanism of the second embodiment [Figure 18] Front-end configuration diagram of the third embodiment [Figure 19] Volume information tables [Figure 20] 10 is a flowchart showing the I / O classification of the front end of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present invention is implemented in a storage system configured by connecting storage nodes operating on computers via a network.

[0013] 1 is a configuration diagram of a storage system. A platform service 110, a compute node 120, and a storage cluster 200 operate on a platform 100. The storage cluster 200 is composed of multiple storage nodes 220. The storage node 220 is composed of a front end 1000, a storage controller 230, a back end 240, a database 250, a collaboration service 260, a cluster controller 270, a node controller 280, and an event monitoring mechanism 3000. Furthermore, the cluster controller 270 includes an event control mechanism 2000.

[0014] The platform service 110 is a service for obtaining and controlling information on the platform 100, and may have multiple operation interfaces such as a command line interface and a REST API. The compute node 120 is an application that uses the storage cluster 200 and issues I / O (Input / Output) to the storage node 220 via the network. The I / O is a request for data input / output processing.

[0015] A plurality of storage nodes 220 constitute a capacity group 210. The capacity group 210 is a group for data protection in the storage cluster 200. For example, the capacity group 210A (capacity group #1) is made up of a storage node 220A (storage node #1), a storage node 220B (storage node #2), and a storage node 220C (storage node #3), and redundancy of the storage controllers and data is implemented within these three storage nodes. The number of storage nodes 220 constituting the capacity group 210 may be any number depending on the configuration of the storage system.

[0016] The storage nodes 220 are classified into several types. Storage node 220A (storage node #1) serves as a primary master, has a cluster controller 270 operating, and has the function of controlling the storage cluster 200. Storage node 220B (storage node #2) and storage node 220C (storage node #3) serve as secondary masters, and take over the function of controlling the storage cluster 200 if storage node 220A (storage node #1) stops functioning. The cluster controller 270 in the secondary master is on standby. Storage node 220D (storage node #4) serves as a worker. Unlike the primary and secondary masters, services necessary for storage cluster management, such as the database 250, collaboration service 260, and cluster controller 270, do not operate. The number of storage nodes such as primary masters, secondary masters, and workers may be any number depending on the configuration of the storage system.

[0017] The frontend 1000 receives I / O from the compute node 120, and transfers I / O between the storage nodes 220 that make up the storage cluster 200, and to the storage controller 230. The storage controller 230 processes I / O from the frontend 1000. The storage controller 230 is configured redundantly among multiple storage nodes 220, so that even if one storage controller 230 stops, processing can be taken over by another storage controller 230. The backend 240 protects data of I / O processed by the storage controller 230. Specifically, the backend 240 protects data by writing data to a disk device external to the storage node 220. I / O data is made redundant among multiple storage nodes 220, so that even if one storage node 220 stops, the I / O data can be restored from another storage node 220.

[0018] The database 250 stores configuration information and control information for the storage cluster 200. The database 250 operates as a distributed database among multiple storage nodes 220. The collaboration service 260 is responsible for collaborative processing among the storage nodes 220. For example, it monitors the health of the storage nodes 220, selects a primary master to serve as the leader from the multiple storage nodes 220, and transmits control information between the storage nodes 220. The cluster controller 270 controls the storage cluster 200. The node controller 280 performs intra-node control in each storage node 220.

[0019] The event monitoring mechanism 3000 periodically acquires event information of each storage node 220 from the platform service 110 and notifies the event control mechanism 2000. Based on the information received from the event monitoring mechanism 3000, the event control mechanism 2000 determines and executes the control required for the storage cluster 200.

[0020] 2 is a network configuration diagram of a storage system. A storage cluster 200 is configured by connecting multiple storage nodes 220 via an inter-node network 300. A compute node 120 is connected to the multiple storage nodes 220 via a compute network 290.

[0021] FIG. 3 is a hardware configuration diagram of a storage system. The server 400 is composed of a CPU 410, memory 420, a network interface 430, and a drive 440. The compute node 120 and the storage node 220 can be considered as software running on the server 400. In other words, when the CPU 410 executes a storage node control program, the server 400 realizes the functions of a storage node and executes a storage node control method. In this case, the computer resources seen from the compute node 120 and the storage node 220 may be physical resources or resources abstracted by virtual machines or the like. Multiple servers 400 are connected via a network 450. The platform 100 hosts the servers 400 and the network 450.

[0022] 4 is a configuration diagram of the event monitoring mechanism. The event monitoring mechanism 3000 shown in FIG.

[0023] The periodic timer 3100 periodically activates the event array acquirer 3200. The event array acquirer 3200 sends an event information acquisition request to the platform service 110, and receives an event array 500 in response, the event array 500 having event data 510 as an element.

[0024] The event sequence analyzer 3300 analyzes the event sequence 500 in cooperation with the event data table 3400. The event data table 3400 records information about the event sequence 500 acquired in the previous cycle. The event sequence analyzer 3300 compares the contents of the event sequence 500 in the current cycle with those in the previous cycle to determine whether an event targeted at the node itself has been scheduled or whether an event targeted at the node itself has ended. If either of the above conditions is confirmed, the event sequence analyzer 3300 passes notification information to the transmitter 3500. Furthermore, the event data table 3400 is overwritten with the contents of the event sequence 500 in the current cycle. When the transmitter 3500 receives a notification from the event sequence analyzer 3300, it transmits the content to the event control mechanism 2000.

[0025] 5 shows an event data table. The event data table 3400 is composed of an event ID 3410, a status 3420, a type 3430, a target resource 3440, and an execution time 3450. This information is contained in the event data 510, which is an element of the event sequence 500. The event data table 3400 records all the event data 510 received from the event sequence analyzer 3300.

[0026] The information of the event data 510 recorded in the event data table 3400 is maintenance information provided by the platform 100. An example of a method for recording maintenance information will be described using an example of the event data table 3400. An event whose event ID 3410 is E001 indicates that "a Freeze event targeting instance 1 is currently running." The target resource 3440 is the name of the resource targeted by the event, which is instance 1. The type 3430 indicates the event type and indicates "Freeze." Freeze indicates maintenance that does not require a reboot. The status 3420 indicates the event state and indicates "Started." "Started" indicates that the event is in an executing state. The execution time 3450 indicates the time when the event becomes executable. That is, the execution time 3540 of an event in an executing state indicates the time when execution started. The execution time 3540 of an event in an executing state may be another value, such as "Started." An event whose event ID 3410 is E002 indicates that "a Reboot event targeting instance 2 is scheduled to be executed." The target resource 3440 is the name of the resource that is the target of the event, which is instance 2. The type 3430 is the event type and indicates Reboot. Reboot indicates maintenance that requires a restart. The status 3420 is the event state and indicates Scheduled. Scheduled indicates that the event is scheduled to be executed. The execution time 3450 indicates the time when the event becomes executable. In other words, the execution time 3540 for a scheduled event indicates the scheduled time when execution will begin.

[0027] 6 is a flowchart showing the operation of the event monitoring mechanism. The event monitoring mechanism 3000 is started by the periodic timer 3100 (4000). Next, the event array 500 is acquired from the platform service 110 (4010). Next, the contents of the event array 500 and the event data table 3400 are compared (4020), and it is determined whether there is a difference in the contents (4030). If YES, proceed to steps 4040 and 4050 where parallel processing is possible; if NO, proceed to step 4090.

[0028] If the answer is YES in step 4030, it is determined whether an event targeting the own node has been scheduled (4040). If the answer is YES in step 4040, the scheduled event data 510 is sent to the event control mechanism 2000 (4060). If the answer is NO, the process proceeds to step 4080. Here, the determination in step 4040 can be made by checking whether the event data 510 holds content in which the status 3420 is Scheduled (event execution planned) and the target resource 3440 is the resource name indicating the own node.

[0029] If the result is YES in step 4030, then in parallel with step 4040, it is determined (4050) whether the event targeted at the node itself has ended. If the result is YES in step 4050, an event completion notification 520 is sent to the event control mechanism 2000 (4070). If the result is NO, the process proceeds to step 4080. Here, the determination in step 4050 can be made by confirming that the event data 510 containing content in which the status 3420 is Started (event being executed) and the target resource (3440) is the resource name indicating the node itself has disappeared.

[0030] Next, in step 4080, the contents of the event data table 3400 are updated with the contents of the acquired event array 500. Finally, the event monitoring mechanism 3000 enters a standby state until it is next started by the periodic timer 3100 (4090).

[0031] In this way, the storage node 220 can monitor the schedule of events targeted at the node itself and the completion of events targeted at the node itself using the event monitoring mechanism 3000. Then, by controlling to limit I / O processing based on the event monitoring results, it is possible to prevent a decrease in redundancy and availability of the storage system and to safely shut down the storage node and storage cluster. [Example]

[0032] Example 1 shows a method for dealing with maintenance that does not require rebooting without shutting down the storage node in an embodiment of the present invention. This embodiment is implemented on the configurations shown in Figures 1 to 6.

[0033] 7 is a block diagram of the event control mechanism 2000. The event control mechanism 2000 is made up of an event data analyzer 2100, a resource information table 2200, an event control transmitter 2300, a capacity group information table 2400, and a notification receiver 2500.

[0034] The event data analyzer 2100 begins operation by receiving event data 510 from the event monitoring mechanism 3000. Next, it identifies the ID of the storage node 220 to be controlled from the event data 510 and the contents of the resource information table 2200. Next, it checks whether the type 3430 of the event data 510 is Freeze (maintenance that does not require a reboot). If the type is anything other than Freeze, it ends the processing. If the type is Freeze, it requests control from the event control transmitter 2300.

[0035] The event control transmitter 2300 begins operation upon receiving a control request from the event data analyzer 2100. Next, it requests the collaboration service 260 to extend the liveness monitoring timeout of the storage node 220 to be controlled. Next, it identifies the capacity group 210 to which the storage node 220 to be controlled belongs from the capacity group information table 2400. Next, it acquires the IDs of all storage nodes 220 belonging to that capacity group 210. Next, it requests the node controllers 280 of all acquired storage nodes 220 to stop accepting I / O from the frontend 1000 and to stop issuing asynchronous I / O. Finally, it requests the notification receiver 2500 to wait for a notification to the storage node 220 to be controlled. Asynchronous I / O refers to data input / output processing that is not requested by the compute node 120 but is issued within the storage system 200. For example, rebalancing processing to adjust the amount of data stored in multiple storage nodes 220, and processing to create a snapshot of stored data are examples of such processing.

[0036] The notification receiver 2500 receives a request from the event control transmitter 2300 and waits for a notification. The notification receiver 2500 receives an event completion notification 520 from the event monitoring mechanism 3000 or a timeout notification 530 from the cluster controller 270. When a notification is received, the notification receiver 2500 checks whether it is a notification for the storage node 220 to be controlled. If it is a notification for the target storage node 220, it requests control from the event control transmitter 2300. If not, it continues to wait for a notification.

[0037] The event control transmitter 2300 also starts operating when it receives a control request from the notification receiver 2500. Next, it requests the collaboration service 260 to cancel the extension of the liveness monitoring timeout of the storage node 220 to be controlled. Next, it identifies the capacity group 210 to which the storage node 220 to be controlled belongs from the capacity group information table 2400. Next, it obtains the IDs of all storage nodes 220 that belong to that capacity group 210. Next, it requests the node controllers 280 of all the obtained storage nodes 220 to resume I / O reception by the frontend 1000 and resume issuing asynchronous I / O.

[0038] Here, the effects of the control performed by the event control mechanism 2000 will be described. First, the effect of extending the liveness monitoring timeout of the storage node 220 will be described. This prevents the network of the storage node 220 from being temporarily stopped due to maintenance, causing a timeout. This makes it possible to avoid the storage node 220 being determined to have a fault due to maintenance. In this case, the extension time of the liveness monitoring timeout is set to be longer than the execution time of the maintenance. However, to minimize the impact on liveness monitoring of the storage cluster 200, an extremely long extension time should be avoided. Furthermore, even if the timeout is extended, the liveness monitoring may still time out, in which case a timeout notification will be received from the cluster controller 270.

[0039] Next, the effect of stopping the frontend 1000 from accepting I / O and stopping the issuance of asynchronous I / O for the storage nodes 220 belonging to the capacity group 210 will be shown. This prevents the network of the storage node 220 from temporarily stopping due to maintenance, causing I / O processing to time out. Additionally, the capacity group 210 is a data protection group in the storage cluster 200, and implements redundancy of the storage controller 230 and data. Therefore, inter-node communication is essential for the storage nodes 220 belonging to the same capacity group 210 to process I / O. By stopping the frontend 1000 from accepting I / O and stopping the issuance of asynchronous I / O, inter-node communication can be suppressed.

[0040] The front end 1000 can stop accepting I / O by keeping the I / O in a queue in memory within the front end 1000. I / O requests accepted while I / O acceptance is stopped are stored in the queue without being processed. After I / O acceptance is resumed, processing resumes in the order of I / O that was stored in the queue. I / O accepted before I / O acceptance was stopped is processed until it is completed.

[0041] 8 shows a resource information table and a capacity group information table. The resource information table 2200 is composed of a resource ID 2210, a resource name 2220, and a storage node ID 2230. The resource information table 2200 shows the correspondence between the management information of the platform 100 and the node management information of the storage cluster 200. The resource ID 2210 and the resource name 2220 are information managed by the platform 100 and can be obtained from the platform service 110. The storage node ID 2230 is the ID of the storage node 220.

[0042] The capacity group information table 2400 is composed of a capacity group ID 2410 and a storage node ID 2420. The capacity group information table 2400 shows a list of storage nodes 220 that constitute a capacity group 210. The capacity group ID 2410 is the ID of the capacity group 210. The storage node ID 2420 is a list of the IDs of the storage nodes 220 that constitute each capacity group 210.

[0043] 9 is a control flowchart of the event control mechanism of the first embodiment. The event control mechanism 2000 is started by receiving event data 510 from the event monitoring mechanism 3000 (5000). Next, the event target storage node 220 is identified from the contents of the event data 510 (5010). Next, it is determined from the contents of the event data 510 whether it is a Freeze event (5020). If YES, Freeze event control processing is performed (5030) and the process proceeds to step 5040. If NO, the process proceeds to step 5040. In step 5040, the process enters a state of waiting for the event to complete.

[0044] Fig. 10 is a flowchart of the freeze event control of the event control mechanism. This corresponds to the freeze event control process (5030) in Fig. 9. When the freeze event control process starts (6000), the process proceeds to steps 6010 and 6020, which can be processed in parallel.

[0045] In step 6010, a request is made to the collaboration service 260 to extend the timeout period of the event-target storage node 220, and the process proceeds to step 6060. In step 6020, the capacity group 210 to which the event-target storage node 220 belongs is acquired. Next, all storage nodes 220 belonging to the capacity group 210 are identified (6030). Next, the process proceeds to steps 6040 and 6050, where parallel processing is possible.

[0046] In step 6040, the node controller 280 of each storage node 220 is requested to stop issuing asynchronous I / O. The node controller 280 that receives the request controls the storage controller 230 to stop issuing asynchronous I / O. In step 6050, the node controller 280 of each storage node 220 is requested to stop accepting I / O at the frontend 1000. Finally, after steps 6010, 6040, and 6050 are performed, the Freeze event control process ends (6060).

[0047] 11 is a control termination flowchart of the event control mechanism of the first embodiment. The event control mechanism 2000 starts control by receiving the event completion notification 520 from the event monitoring mechanism 3000 (7000) or the timeout notification 530 from the cluster controller 270 (7010). Next, the event control mechanism 2000 identifies the storage node 220 that is the target of the event from the contents of the event data 510 (7020). Next, it determines from the contents of the received notification whether the notification is for a storage node 220 that is waiting for completion of a Freeze event (7030). If YES, it performs Freeze event control termination processing (7040) and proceeds to step 7050. If NO, it proceeds to step 7050. In step 7050, it enters a state of waiting for reception of a notification.

[0048] Figure 12 is a flowchart showing the end of Freeze event control by the event control mechanism. This corresponds to the Freeze event control end process (7040) in Figure 11. When the event control mechanism 2000 starts control (8000), it proceeds to steps 8010 and 8020, which can be processed in parallel.

[0049] In step 8010, the coordination service 260 is requested to cancel the extension of the timeout period for the event target storage node 220. In step 8020, the capacity group 210 to which the event target storage node 220 belongs is acquired. Next, all storage nodes 220 belonging to the capacity group 210 are identified (8030). Next, the process proceeds to steps 8040 and 8050, where parallel processing is possible.

[0050] In step 8040, the node controller 280 of each storage node 220 is requested to resume issuing asynchronous I / O. In step 8050, the node controller 280 of each storage node 220 is requested to resume accepting I / O at the front end 1000. Finally, after steps 8010, 8040, and 8050 are performed, the Freeze event control end process is terminated (8060).

[0051] 13 is an example of displaying the storage cluster status under event control. The management screen 600 is a screen for displaying information about the storage cluster 200, and is displayed on a display or the like connected to the storage cluster 200 via a network. The management screen 600 is an example of displaying the status of the storage nodes 220 for each capacity group 210.

[0052] The status of a storage node 220 on the management screen 600 consists of a storage node ID 610, a status 620, and a message 630. Using this information, the user can check the status of each storage node 220 and the cause of that status. For example, the storage node 220 displayed on the management screen 600 can display a state such as "normal" or "I / O acceptance suspended" as the status 620. If the status 620 shows "I / O acceptance suspended," this indicates that a Freeze event is being controlled by the event control mechanism 2000. Furthermore, the user can check from the message 630 which storage node 220 is experiencing a Freeze event and has suspended I / O acceptance.

[0053] In this way, in the first embodiment, when a Freeze event occurs, the timeout period of the storage node targeted by the event is extended, and the issuance of asynchronous I / O and the acceptance of I / O are stopped in the capacity group to which the storage node targeted by the event belongs, thereby preventing the storage node from being stopped. [Example]

[0054] Example 2 shows a method of determining and implementing a control method for maintenance by taking into account information about the storage cluster 200 in an embodiment of the present invention. This embodiment is shown in a form that expands Example 1 in addition to the configurations shown in Figs. 1 to 6. However, the gist is that a control method for maintenance is determined by taking into account information about the storage cluster 200, and there is no need to be limited to the form of Example 1.

[0055] 14 is a configuration diagram of the event control mechanism of embodiment 2. The event control mechanism 2000 is composed of an event data analyzer 2100, a resource information table 2200, an event control determiner 9000, a storage cluster information table group 9100, an event control transmitter 2300, a capacity group information table 2400, and a notification receiver 2500.

[0056] The event data analyzer 2100 begins operation by receiving event data 510 from the event monitoring mechanism 3000. Next, it identifies the ID of the storage node 220 to be controlled from the event data 510 and the contents of the resource information table 2200. Next, it checks whether the type 3430 of the event data 510 is Freeze (maintenance that does not require a reboot) or Reboot (maintenance that requires a reboot). If the type is anything other than Freeze or Reboot, it ends the processing. If it is Freeze or Reboot, it requests the event control decision unit 9000 to make a control decision.

[0057] The event control determiner 9000 starts operation when requested to make a control determination by the event data analyzer 2100. Next, if a Freeze event is scheduled to be executed in the storage node 220 to be controlled, it determines whether the Freeze event control process 5020 can be executed. If it can be executed, it requests the event control transmitter 2300 to execute the Freeze event control process 5020. If it cannot be executed, it determines that blocking processing of the storage node 220 is necessary. If a Reboot event is scheduled to be executed in the storage node 220 to be controlled, it also determines that blocking processing of the storage node 220 is necessary.

[0058] Next, a decision is made regarding the blocking process of the storage node 220. Based on the information in the storage cluster information table group 9100 and the capacity group information table 2400, it is decided whether blocking the storage node 220 will result in a failure that exceeds the redundancy of the storage cluster 200. If a failure that exceeds the redundancy of the storage cluster 200 will result, the event control transmitter 2300 is requested to perform blocking process of the storage cluster 200. If not, the event control transmitter 2300 is requested to perform blocking process of the storage node 220.

[0059] The event control transmitter 2300 starts operation when it receives a control request from the event control determiner 9000. Next, it performs control in accordance with the control content requested by the event control determiner 9000. When it is requested to perform the Freeze event control process 5020, it performs the control as explained in FIG. 7 and FIG. 10. When it is requested to perform the blockage process of the storage cluster 200, it requests the cluster controller 270 to perform the process and then terminates. When it is requested to perform the blockage process of the storage node 220, it requests the node controller 280 to perform the process. Finally, it requests the notification receiver 2500 to wait for a notification to the storage node 220 to be controlled.

[0060] The notification receiver 2500 receives a request from the event control transmitter 2300 and waits for a notification. The notification receiver 2500 receives an event completion notification 520 from the event monitoring mechanism 3000 or a timeout notification 530 from the cluster controller 270. When a notification is received, the notification receiver 2500 checks whether it is a notification for the storage node 220 to be controlled. If it is a notification for the target storage node 220, it requests control from the event control transmitter 2300. If not, it continues to wait for a notification.

[0061] The event control transmitter 2300 also starts operation when it receives a control request from the notification receiver 2500. In the case of a notification from a storage node 220 that has performed freeze event control processing, it performs freeze event control end processing 7040. This executes the control as explained in Figures 7 and 12. In the case of a notification from a storage node 220 that has performed node blocking control, it requests the cluster controller 270 to perform node recovery processing.

[0062] Here, we will provide additional information about the control performed by the event control mechanism 2000. First, an example of a factor determining whether or not the Freeze event control process 5020 can be performed is when the grace period until the Freeze event is performed is shorter than the processing time of the Freeze event control process 5020. If the Freeze event control process 5020 cannot be executed, the storage node 220 is blocked.

[0063] Next, the blocking process of a storage node 220 means stopping the process of the target node, restarting the node, and putting it into a standby state. Blocking the storage node 220 prevents the maintenance from affecting the storage cluster 200. At this time, I / O processing and the like that should be performed by the blocked storage node 220 is temporarily taken over by another storage node 220. Furthermore, the blocked storage node 220 can be returned to the storage cluster 200 by recovery processing from the cluster controller 270.

[0064] Finally, the storage cluster 200 blocking process is to stop the storage cluster 200. This process is necessary when a failure occurs that exceeds the redundancy that allows the storage cluster 200 to continue operating. Whether the storage cluster 200 can continue operating can be determined based on the number of failed storage nodes 220 and the number of drives 440 used by the storage nodes 220. If the number of failed units in the capacity group 210 exceeds the redundancy of the storage cluster 200, continued operation becomes impossible. The storage cluster 200 blocking process can stop the storage cluster 200 before a failure occurs that exceeds the redundancy.

[0065] 15 shows a storage cluster information table group. The storage cluster information table group 9100 is made up of a storage node information table 9110 and a drive information table 9120. The storage node information table 9110 is made up of a storage node ID 9111 and a status 9112. The drive information table 9120 is made up of a drive ID 9121, a status 9122, and a storage node ID 9123. The information in the storage cluster information table group 9100 is acquired from the information of the storage cluster 200 managed in the database 250.

[0066] The storage node information table 9110 indicates the status of each storage node 220. If the status 9112 is "normal", the storage node 220 is operating normally, and if the status 9112 is "blocked", it can be determined that the storage node 220 has experienced a failure.

[0067] The drive information table 9120 indicates the status of each drive 440. If the status 9122 is "normal," the drive 440 is operating normally, and if the status 9122 is "blocked," it can be determined that the drive 440 has experienced a failure. Furthermore, the storage node 220 to which each drive 440 belongs can be determined by checking the storage node ID 9122.

[0068] 16 is a control flowchart of the event control mechanism of the second embodiment. The event control mechanism 2000 is started by receiving event data 510 from the event monitoring mechanism 3000 (5000). Next, the event target storage node 220 is identified from the contents of the event data 510 (5010). Next, it is determined from the contents of the event data 510 whether it is a Freeze event (5020). If YES, proceed to step 10000. If NO, proceed to step 10010.

[0069] In step 10000, it is determined whether or not Freeze event control is possible. If YES, Freeze event control processing (5030) is performed and the process waits for the event to complete (5040). If NO, the process proceeds to step 10020. In step 10010, it is determined whether or not it is a Reboot event. If YES, the process proceeds to step 10020. If NO, the process waits for the event to complete (5040).

[0070] In step 10020, it is determined whether blocking the storage node 220 that is the target of the event will result in a failure that exceeds the redundancy of the storage cluster 200. If the result is YES, the cluster controller 270 is requested to perform blocking processing for the storage cluster 200 (10030), and the processing ends (10050). If the result is NO, the node controller 280 is requested to perform blocking processing for the storage node 220 (10040), and the process waits for the event to complete (5040).

[0071] 17 is a control termination flowchart of the event control mechanism of embodiment 2. The event control mechanism 2000 starts control by receiving (7000) an event completion notification 520 from the event monitoring mechanism 3000 or by receiving (7010) a timeout notification 530 from the cluster controller 270. Next, the event target storage node 220 is identified from the contents of the event data 510 (7020).

[0072] Next, it is determined from the contents of the received notification whether it is a notification for a storage node 220 waiting for completion of a Freeze event (7030). If YES, a Freeze event control end process (7040) is performed and the storage node 220 enters a standby state (7050). If NO, the process proceeds to step 11000.

[0073] In step 11000, it is determined whether the notification is to a storage node 220 that is waiting for completion of the reboot event. If the result is YES, the cluster controller 270 is requested to perform recovery processing for the storage node 220 that is the target of the event (11010), and the node enters a standby state (7050). If the result is NO, the node remains in a standby state (7050).

[0074] In this way, in the second embodiment, control is determined based on the type of event and information on the storage cluster 200. If the event type is a Freeze event, as in the first embodiment, the timeout period of the storage node targeted by the event is extended, and the issuance of asynchronous I / O and the acceptance of I / O are stopped in the capacity group to which the storage node targeted by the event belongs, thereby preventing the storage node from being stopped. If the event type is a reboot event, it is determined whether the storage node or the storage cluster should be stopped and the appropriate action is taken, so that the storage node and the storage cluster can be stopped safely. [Example]

[0075] Example 3 shows an enhancement of a method for dealing with the maintenance that does not require a reboot shown in Example 1 in an embodiment of the present invention without stopping the storage node. The enhancement is to classify received I / O when I / O reception is stopped at the front end, and continue processing only I / O that can be processed. This embodiment is based on the configurations shown in Figures 1 to 6, and on the premise that a maintenance that does not require a reboot is handled as in Example 1 and Example 2.

[0076] 18 is a configuration diagram of a front end of embodiment 3. The front end 1000 is composed of an I / O acceptance queue 1200, an I / O processing queue 1300, an I / O standby queue 1400, an I / O response queue 1500, an I / O classifier 1600, and a volume information table group 1700.

[0077] The I / O acceptance queue 1200 receives and stores I / O from the compute node 120. The I / O classifier 1600 classifies the I / O stored in the I / O acceptance queue 1200 into an I / O processing queue 1300 and an I / O standby queue 1400 based on information in a volume information table group 1700. However, classification is only performed when the node controller 280 requests the frontend 1000 to stop accepting I / O.

[0078] The I / O classifier 1600 classifies I / O based on whether the I / O can be processed without communication between storage nodes. For example, if the I / O is a write, data must be made redundant between multiple storage nodes 220, so inter-node communication is required. Therefore, the write is classified into the I / O waiting queue 1400. If the I / O is a read, if the read data is stored within the own node, it can be processed without inter-node communication. If the read data is not stored within the own node, inter-node communication is required. Therefore, the read is classified into the I / O processing queue 1300 and the I / O waiting queue 1400 depending on the location of the read data.

[0079] The I / O standby queue 1400 is a queue for holding I / O without processing it. When I / O acceptance is suspended, it holds I / O that cannot be processed, and when I / O acceptance resumes, it passes the I / O to the storage controller 230 and processes it.

[0080] The I / O processing queue 1300 is a queue for processing I / O. The I / O held in the I / O processing queue 1300 is passed to the storage controller 230 in order and processed.

[0081] The I / O response queue 1500 is a queue for responding to I / O to the compute node 120. The I / O is passed from the front end 1000 to the storage controller 230 and then to the back end 240, where it is processed. When the processing is complete, a response is returned from the back end 240 and then the storage controller 230, and the response is entered into the I / O response queue 1500 in the front end 1000. The I / O passed to the I / O response queue 1500 is passed to the compute node 120 as a response.

[0082] 19 shows a group of volume information tables. The group of volume information tables 1700 is made up of a volume owner information table 1710 and a storage controller information table 1720. The volume owner information table 1710 is made up of a volume ID 1711, an owner storage controller ID 1712, a data owner storage node ID 1713, a data status 1714, and a parity status 1715. The storage controller information table 1720 is made up of a storage controller ID 1721, a status 1722, and a storage node ID 1723.

[0083] The volume owner information table 1710 shows volume owner information. A volume is a virtual drive that the storage cluster 200 presents to the compute node 120. Each volume has an owner storage controller 230, which is indicated by an owner storage controller ID 1712. I / O to the volume is processed by the owner storage controller 230. Each volume has a data owner storage node 220, which stores the data for that volume. The data status 1714 shows the status of the data for that volume; "normal" means that reading and writing are possible, and "blocked" means that reading and writing are not possible. The parity status 1715 shows the status of the parity of the data for that volume; "normal" means that reading and writing are possible, and "blocked" means that reading and writing are not possible.

[0084] The storage controller information table 1720 shows information about the storage controller. The status 1722 shows the status of the storage controller 230. If the status 1722 is Active, it is operating normally, and if it is Standby, it is in a standby state and can take over processing if an abnormality occurs in the active storage controller. The storage node ID 1723 shows the storage node 220 to which the storage controller 230 belongs.

[0085] Here, an example is shown in which the I / O classifier 1600 classifies I / O using the volume information table group 1700. When the frontend 1000 receives a read for a certain volume, it checks whether the owner storage controller and data owner storage node are its own node. Next, it checks whether the data status 1714 is normal. If all of these conditions are met, it can be determined that the I / O can access the data without performing inter-node communication.

[0086] 20 is a flowchart of I / O classification for the frontend of the third embodiment. The frontend 1000 starts control by receiving I / O from the compute node 120 (12000). Next, it determines whether the I / O is a read based on the contents of the I / O. If YES, proceed to step 12020. If NO, proceed to step 12060.

[0087] In step 12020, it is determined whether or not there is an owner storage controller in the own node for the volume that is the target of the I / O. If YES, proceed to step 12030. If NO, proceed to step 12060. In step 12030, it is determined whether or not the own node is the data owner storage node for the volume that is the target of the I / O. If YES, proceed to step 12040. If NO, proceed to step 12060. In step 12040, it is determined whether or not the data status of the volume that is the target of the I / O is normal. If YES, the I / O is moved to the I / O processing queue 1300 (12050) and waits for the next I / O (12070). If NO, proceed to step 12060. In step 12060, the I / O is moved to the I / O waiting queue 1400 and waits for the next I / O (12070).

[0088] I / Os in the I / O processing queue 1300 are processed even if other nodes in the capacity group are frozen. I / Os in the I / O waiting queue 1400 wait for other nodes in the capacity group to unfreeze before being processed.

[0089] In this way, in the third embodiment, I / O that can be processed within one node can be processed even when a freeze event occurs, and a decrease in availability can be prevented.

[0090] As described above, the storage system comprises a plurality of storage nodes 220 each having an arithmetic unit (CPU 410) and a memory 420, and when the plurality of storage nodes 220 detects a failure in another storage node within the storage system, they take over from the storage node in which the failure occurred by failover, and when a maintenance event occurs in the storage system, the plurality of storage nodes change the failure detection conditions for the storage node associated with the maintenance event based on maintenance event information, which is information related to the maintenance event, and restrict data input / output processing. This configuration and operation allows the storage system to reduce the impact on performance caused by maintenance events.

[0091] Furthermore, when the maintenance event occurs, the storage node extends the timeout period of the liveness monitoring, which is a condition for detecting the failure. The extended liveness check timeout period is longer than the maintenance event period. Therefore, when a maintenance event is carried out, it is possible to avoid stopping the target storage node and prevent a decrease in redundancy and availability of the storage system.

[0092] Furthermore, when the maintenance event occurs, the storage node stops the data input / output processing related to the processing by the storage node involved in the maintenance event, and continues to execute the data input / output processing not related to the processing by the storage node involved in the maintenance event without stopping it. Specifically, the storage node stores in memory the data input / output requests received while data input / output processing is stopped, and processes the data input / output requests whose processing has been stopped after the maintenance event ends. Furthermore, when the maintenance event occurs, the storage node suspends write processing and read processing involving other storages among the data input / output processing, but continues to execute read processing performed only by the storage node itself. This reduces the performance degradation when a maintenance event is carried out.

[0093] Furthermore, based on the content of the maintenance event, the storage node determines whether to change the fault detection conditions when the maintenance event occurs, or whether another storage node will take over from the storage node affected by the maintenance event through failover. Specifically, if the maintenance event does not include a restart of the storage node, the storage node decides to change the failure detection conditions when the maintenance event occurs, and if the maintenance event includes a restart, decides that another storage node will take over from the storage node affected by the maintenance event by failover. Therefore, the optimum operation can be selected depending on the maintenance event, and the performance degradation when the maintenance event is carried out can be reduced.

[0094] Although several embodiments have been described above, these are merely examples for explaining the present invention, and it is not intended that the scope of the present invention be limited to these embodiments. The present invention can be implemented in various other forms. [Explanation of symbols]

[0095] 100: Platform 110: Platform Services 120: Compute node 200: Storage cluster 210: Capacity Group 220: Storage node 230: Storage controller 240: Backend 250: Database 260: Collaborative Services 270: Cluster Controller 280: Node controller 290: Compute Network 300: Internode Network 400:Server 410:CPU 420:Memory 430: Network Interface 440: Drive 450: Network 1000: Front end 1600:I / O classifier 1700: Volume information tables 2000: Event control mechanism 2100: Event Data Analyzer 2200: Resource information table 2300: Event-controlled transmitter 2400: Capacity Group Information Table 2500: Notification receiver 3000: Event monitoring mechanism 3100: Periodic timer 3200: Event sequence acquirer 3300: Event Sequence Analyzer 3500: Transmitter 9000: Event control judgement device 9100: Storage cluster information tables 9110: Storage node information table

Claims

1. In a storage system including a plurality of storage nodes each having a computing device and a memory, when the plurality of storage nodes detects the occurrence of a failure in another storage node within the storage system, the plurality of storage nodes take over the storage node in which the failure occurred by failover; When a maintenance event occurs in the storage system, the storage nodes Based on maintenance event information, which is information about the maintenance event, the fault detection condition for the storage node related to the maintenance event is changed and data input / output processing is restricted. A storage system comprising:

2. 2. The storage system according to claim 1, The storage node extends a liveness monitoring timeout period as a condition for detecting the failure when the maintenance event occurs. A storage system comprising:

3. 3. The storage system according to claim 2, The extended liveness timeout duration is longer than the maintenance event duration. A storage system comprising:

4. 2. The storage system according to claim 1, When the maintenance event occurs, the storage node stops data input / output processing related to the processing by the storage node involved in the maintenance event, and continues to execute data input / output processing not related to the processing by the storage node involved in the maintenance event. A storage system comprising:

5. 5. The storage system according to claim 4, The storage node stores in a memory a data input / output request that has been received while the data input / output processing is stopped, and processes the data input / output request whose processing has been stopped after the maintenance event ends. A storage system comprising:

6. 5. The storage system according to claim 4, When the maintenance event occurs, the storage node stops write processing and read processing involving other storages among the data input / output processing, but continues to execute read processing performed only by the storage node itself. A storage system comprising:

7. 2. The storage system according to claim 1, The storage node determines, based on the content of the maintenance event, whether to change the failure detection conditions when the maintenance event occurs, or whether another storage node will take over the storage node affected by the maintenance event by failover. A storage system comprising:

8. 8. The storage system according to claim 7, The storage node determines to change the failure detection condition when the maintenance event occurs if the maintenance event does not include a reboot of the storage node, and determines to have another storage node take over the storage node involved in the maintenance event by failover if the reboot is included. A storage system comprising:

9. A control method for a storage system including a plurality of storage nodes each having a computing device and a memory, comprising: when the plurality of storage nodes detects the occurrence of a failure in another storage node within the storage system, the plurality of storage nodes take over the storage node in which the failure occurred by failover; When a maintenance event occurs in the storage system, the storage nodes When the maintenance event occurs, the fault detection condition for the storage node related to the maintenance event is changed based on maintenance event information, which is information related to the maintenance event, and data input / output processing is restricted. A storage system control method comprising:

Images