Apparatus for managing verification process in storage system

Abstract

To more appropriately control verification processing of a storage system.SOLUTION: An apparatus for managing a verification process in a storage system includes a processor and a storage device. The processor determines an amount of data stored in a target logical device of the verify processing, determines a time-out time of the target logical device based on the amount of data, and controls the verify processing of the target logical device based on the determined time-out time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to management of verification processes in storage systems. [Background technology]

[0002] Background art of the present disclosure is Japanese Patent Application Laid-Open No. 2000-293318, which discloses a disk array device that reduces the number of media errors that can be remedied by retry processing being detected as timeout errors by gradually increasing the time monitoring setting value of the hard disk device (see, for example, the abstract).

[0003] Verification is one type of storage system test that checks the consistency of data distributed across parity groups (also known as RAID (Redundant Array of Independent Disks) groups) primarily to check for write leakage.

[0004] The test device specifies the target LDEV (also called a logical device or volume) to the storage system and instructs it to perform a verify. The storage system then checks the consistency of the data in the specified LDEV. For example, a data consistency check compares the user data in two LDEVs in a RAID1 configuration, or compares one or two parities generated from the user data with the parity stored on the drive in a RAID5 or RAID6 configuration. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-293318 Summary of the Invention [Problem to be solved by the invention]

[0006] For example, in testing a storage system, the test device receives the verification results from the storage system, but the verification may not progress, causing the test to stop. [Means for solving the problem]

[0007] One aspect of the present invention is an apparatus for managing verification processing in a storage system, comprising a processor and a storage device, wherein the processor determines the amount of data stored in a logical device that is the target of the verification processing, determines a timeout period for the logical device based on the amount of data, and controls the verification processing of the logical device based on the determined timeout period. [Effects of the Invention]

[0008] According to a representative embodiment of the present invention, the verification process of a storage system can be controlled more appropriately. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiment. [Brief explanation of the drawings]

[0009] [Figure 1] 1 illustrates a test environment for a storage system according to an embodiment of the present specification. [Figure 2] 1 shows an example of the configuration of a storage system. [Figure 3] FIG. 1 illustrates an example hardware configuration of a test execution client according to an embodiment of the present specification. [Figure 4] 10 shows an example of the configuration of a timeout time derivation table. [Figure 5] 10 shows an example of the configuration of a timeout time derivation table. [Figure 6] 10 shows a flowchart of an example process of verify management and control by a maintenance fault tool. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following description of the embodiments will be given with reference to the accompanying drawings. First, the prerequisites for the following description will be described.

[0011] First, the embodiments described below do not limit the invention according to the claims, and not all combinations of elements described in the embodiments are necessarily essential to the solution of the invention.

[0012] Second, in the following description, data structures such as "tables" and "lists" are used to describe how data and control information are stored, but other data structures that can provide equivalent representations may also be used. Also, in the following description, integer IDs are sometimes assigned to each item in a table or list to distinguish them, but these IDs may be expressed in other unique ID formats. Examples of other ID formats include GUIDs (Globally Unique IDs) and character strings.

[0013] Third, in the following explanation, we may use the term "program" to describe processing. However, a program is interpreted and executed by a CPU (Central Processing Unit), which controls components such as memory and ports as necessary to execute the processing described in the program. Furthermore, instead of executing the processing described in the program itself, the CPU may use an appropriate hardware accelerator to execute the processing, depending on the content of the processing. Examples of hardware accelerators include a compression accelerator that compresses and decompresses data on behalf of the CPU, and a DMA engine that performs data communication on behalf of the CPU.

[0014] Fourth, in the following explanation, the operation of physical components and the operation of logical data structures may be described without distinguishing between them. However, operations on logical data structures are performed by the operation of physical components abstracted by those data structures, while the operation of physical components also involves appropriate operations on the logical data structures that abstract those components. For example, when a storage controller inputs or outputs data to or from a drive, the storage controller not only sends and receives data to and from the drive, but also updates the control information area in memory and the metadata in non-volatile memory so that the status changes associated with the data input / output are appropriately reflected in logical data structures such as the thin provisioning pool that abstracts the drive and the parity group to which the drive belongs.

[0015] One embodiment of this specification manages the verification process of a logical device in a storage system. The verification process management sets a timeout period, and when the elapsed time from the start of the verification process reaches the timeout period, the verification process is retried or stopped. This reduces unnecessary lost time when the verification process cannot be completed due to some kind of failure. Furthermore, repeated retries can reduce the frequency of verification process failures.

[0016] In one embodiment of this specification, a timeout period is set for multiple drives that make up a RAID (Redundant Array of Independent Disks) group (parity group) based on the RAID level. In another embodiment of this specification, the timeout period is further set based on the drive types that make up the RAID group. By taking these items into consideration, a more appropriate timeout period can be set. The RAID level and drive type are each drive attributes.

[0017] FIG. 1 schematically illustrates a test environment for a storage system according to an embodiment of this specification. A test executor executes test cases to verify the functionality of the storage system 1 and confirms that it operates as designed. A test execution client 3 executes test cases instructed by the test executor on the storage system 1 and verifies the operation of the storage system 1. A maintenance PC (Personal Computer) 5 monitors the status of and maintains the storage system 1. The storage system 1, the test execution client 3, and the maintenance PC 5 can communicate with each other via a network.

[0018] The maintenance PC 5 stores an object file 51, and installs it in the storage system 1 in accordance with instructions from the test execution client 3. The object file 51 is a program executed by the storage system 1, and performs verification processing of data stored in the storage system 1. After testing of the storage system 1 is completed, the object file 51 may be deleted from the storage system 1.

[0019] A combination of a maintenance PC 5 and a storage system 1 is implemented in the execution environment. Although Fig. 1 shows one set of a maintenance PC 5 and a storage system 1, the test execution client 3 can simultaneously execute tests on multiple combinations of a maintenance PC 5 and a storage system 1.

[0020] The test execution client 3 stores a maintenance fault tool 31 and a timeout time derivation table 32. The maintenance fault tool 31 is a program that issues maintenance operation commands, including commands related to the verify process, to the storage system 1. The maintenance fault tool 31 manages the execution of the verify process by the storage system 1 by referring to the timeout time derivation table 32. The timeout time derivation table 32 manages the timeout time of the verify process.

[0021] The storage system 1 includes one or more storage controllers (CTL) 12 and one or more LDEVs (logical devices) 10. In the configuration example shown in FIG.

[0022] The LDEV 10 is a logical storage area, also called a volume. A storage area is allocated to the LDEV 10 from one or more physical drives, and host data received from a host (not shown) is stored therein.

[0023] The storage controller 12 processes IO requests from a host (not shown). Specifically, in accordance with a write request, the storage controller 12 stores host data received from the host at the address of the specified LDEV 10, and reads the host data from the address of the specified LDEV and sends it to the host.

[0024] When testing the storage system 1, the storage controllers 12 execute an object file 51 that has been installed and loaded from the maintenance PC 5. In the configuration example shown in Fig. 1, each storage controller 12 can access all of the LDEVs 10. Normally, one of the storage controllers 12 processes IO requests and executes the object file 51, and if a failure occurs in one of the storage controllers 12, the other takes over and executes the object file 51.

[0025] Each storage controller 12 further executes processing in response to a command from the test execution client 3. In one embodiment of this specification, each storage controller 12 executes verification processing of a specified LDEV in response to a command from the test execution client 3. The verification processing is executed under the management and control of the test execution client 3.

[0026] Fig. 2 shows an example of the configuration of the storage system 1. The storage system 1 includes one or more storage controllers 12 and one or more physical drives 13. In the example configuration of Fig. 2, two storage controllers 12 and a plurality of physical drives 13 are implemented.

[0027] The storage controller 12 is connected to other devices, such as a host, a test execution client 3, and a maintenance PC 5, via a front-end interface 16, and is capable of receiving various commands and sending and receiving data. The host interface connecting to the host and the management interface connecting to other management devices, such as the test execution client 3 and the maintenance PC 5, may be different. An example of a connection topology between the storage controller 12 and the host is an IP-SAN (Storage Area Network). An example of a connection topology between the storage controller 12 and the test execution client 3 and the maintenance PC 5 is a LAN.

[0028] The storage controller 12 is connected to the physical drives 13 via one or more back-end interfaces 17 , and is capable of issuing various commands to the physical drives 13 and transmitting and receiving data to and from the physical drives 13 .

[0029] The physical drive 13, simply referred to as a drive, is a non-volatile storage device. Examples of the drive 13 include a solid state drive (SSD) and a hard disk drive (HDD). The drive 13 may be stored in a drive box 19 independent of the storage controller 12, as shown in FIG. 2, or may be built into the storage controller 12. An example of a connection between the storage controller 12 and the drive 13 is a backend switch 100 that can connect multiple NVMe drives to a single PCIe port.

[0030] The connection between the storage controllers 12 and the drives 13 does not necessarily require a logical communication path to be secured between all storage controllers 12 and all drives 13 as illustrated in Figure 2, and each storage controller 12 may have a logical communication path secured only between some of the drives 13.

[0031] The storage controllers 12 are connected to each other by an inter-controller bus, and commands and data can be exchanged via the inter-controller bus. For hosts and drives 13 with which no logical communication path is established, each storage controller 12 can exchange commands and data with other storage controllers 12 via the inter-controller bus, thereby indirectly exchanging commands and data with the host or drive 13.

[0032] The storage controller 12 includes a processor 14 and a memory 15, and the processor 14 executes a control program on the memory 15. The processor 14 uses a cache area on the memory 15 as a storage area for temporary data, and also uses a partial area on the memory 113 as a storage area for control information. The processor 14 exchanges data and commands with external devices and the drives 13 in accordance with the descriptions in the control program.

[0033] The control programs, control information, and data in the cache area in the memory 15 are made nonvolatile as necessary. The storage controller 12 may be equipped with a dedicated nonvolatile memory for making the control programs, control information, and data in the cache area in the memory 15 nonvolatile. Examples of nonvolatile memory include an SSD (Solid State Drive) and an SCM (Storage Class Memory).

[0034] 3 is a diagram showing an example of the hardware configuration of the test execution client 3 according to an embodiment of the present specification. An example of the hardware configuration of the test execution client 3 will be described below, but the maintenance PC 53 may also have a similar configuration.

[0035] The test execution client 3 includes a CPU (processor) 301 that executes various programs, a memory (main storage device) 302 that stores the various programs, and an auxiliary storage device 303 that stores various data. The CPU 301 can include one or more cores, and the memory 302 is, for example, a DRAM that includes a volatile storage area. The auxiliary storage device 303 is, for example, an HDD (hard disk drive) or flash memory, and can provide a non-volatile storage area.

[0036] The test execution client 3 further includes an output device 304 for presenting information to the user of the device, an input device 305 for inputting instructions, images, etc. from the user, and a communication device 306 for communicating with other devices. These are interconnected by a bus 307.

[0037] The CPU 301 reads and executes various programs from the memory 302 as necessary. The memory 302 can store the maintenance troubleshooting tool 31, an OS (not shown), and other application programs. Each program is loaded into the memory 302 from the auxiliary storage device 303, for example, and executed by the CPU 301. At least a part of the functions of the test execution client 3 may be configured using logic circuits.

[0038] The auxiliary storage device 303 stores data that is referenced or managed by various programs, such as a timeout time derivation table 32.

[0039] The output device 304 is composed of devices such as a display, printer, and speaker. The input device 305 is composed of devices such as a keyboard, mouse, and microphone. The output device 304 presents input results from the user and also presents processing results by the test execution client 3. Instructions from the user are input to the test execution client 3 via the input device 305.

[0040] The communication device 306 receives data transmitted from other devices connected via a network including the storage system 1, and transmits processing results by the test execution client 3 to other devices. Note that some devices may be omitted. Also, the hardware configuration of the maintenance PC 5 may be the same as that described with reference to FIG. 3.

[0041] 4 and 5 show an example of the configuration of the timeout time derivation table 32. The timeout time derivation table 32 defines the timeout time for each storage configuration (including drives) that provides an LDEV. In the example shown in FIGS. 4 and 5, the timeout time derivation table 32 indicates the timeout time (h) per 1 TB of data. The timeout time derivation table 32 also indicates coefficients for calculating the predicted timeout time for the actual verification process for each LDEV.

[0042] 4, the timeout period derivation table 32 has a model column 321, a drive type column 322, a RAID1(h) column 323, a RAID5(h) column 324, and a RAID6(h) column 325. Note that information on other RAID levels may also be included.

[0043] The model column 321 indicates the model of the storage system 1 (or storage controller 12). Here, mid-range and high-end are shown as examples, but other levels may be included, and the model may be defined in more detail, such as by model number. Differences in model indicate, for example, differences in the performance of the storage controller 12. A storage system 1 with higher performance is capable of faster processing and requires less time for verification processing. This example makes it possible to define a timeout period that is appropriate for the performance of the storage controller 12.

[0044] The drive type column 322 indicates the type of drive 13 that provides the storage area to the LDEV. Because different types of drives 13 may have different IO performance, it is possible to define a timeout period that is appropriate for each. For example, the time required for verifying data stored on an SSD is shorter than the time required for verifying data stored on an HDD.

[0045] The RAID1(h) column 323, RAID5(h) column 324, and RAID6(h) column 325 each indicate the timeout period (hours) for a different RAID level. The process for verifying data integrity differs depending on the RAID level. An appropriate timeout period can be defined depending on the RAID level.

[0046] For example, data verification for RAID1 compares the actual data between mirrored drives. Data verification for RAID5 compares one parity created from host data stored on multiple drives 13 with one parity stored on another drive 13. Data verification for RAID6 compares two parities created from host data stored on multiple drives 13 with two parities stored on another drive 13. Therefore, the time required for verification under the same conditions (same model, drive type, and data volume) is shortest for RAID5 and longest for RAID1.

[0047] Fig. 5 shows a simplified example of the configuration of the timeout time derivation table 32. The model column 321 and drive type column 322 have been omitted from the example configuration shown in Fig. 4. The RAID level has a greater impact on the time required for verification than the model and drive type. This makes it possible to efficiently manage and control the verification process appropriately. It is also possible to omit only one of the model column 321 and the drive type column 322. Depending on the design, the RAID level columns 323-325 may be omitted, or the timeout time derivation table 32 may be omitted. The timeout time is determined according to the amount of data stored in the LDEV.

[0048] The processing performed by the maintenance fault tool 31 will be explained below. The maintenance fault tool 31 instructs the storage controller 12 in which the object file 51 is installed to perform verification processing for each LDEV, and manages and controls this processing. The maintenance fault tool 31 determines the timeout time for each LDEV by referencing the timeout time derivation table 32. The timeout time is calculated according to the following formula. Timeout period = Coefficient of timeout time derivation table x LDEV capacity x LDEV usage rate [%]

[0049] The coefficients obtained from the timeout time derivation table 32 are defined according to the model, drive type, and RAID level of the storage system 1 in the configuration example of Fig. 4, and are defined according to the RAID level in the example of Fig. 5. In the following, the timeout time derivation table 32 of the configuration example shown in Fig. 4 is assumed.

[0050] FIG. 6 shows a flowchart of an example of a process for verify management and control by the maintenance trouble tool 31. In FIG.

[0051] In pre-processing, the maintenance fault tool 31 defines the timeout time derivation table 32 (S11). Specifically, the maintenance fault tool 31 registers the maximum value of processing time per 1 TB (timeout time) for each combination of model, drive type, and RAID level in the timeout time derivation table 32, in accordance with input from the user (the user who executes the test of the storage system 1).

[0052] Next, the maintenance fault tool 31 receives input of a list of LDEVs for which verification processing is to be executed (S12). For example, the user inputs the identifiers of multiple LDEVs for which verification processing is to be executed, separated by "," (comma).

[0053] Next, the maintenance trouble tool 31 acquires information about the model from the storage system 1 (S13). For example, the maintenance trouble tool 31 logs in to the controller 12 and issues a command to acquire the model information. The maintenance trouble tool 31 acquires the model information from the controller 12. For example, the model number is acquired from the storage system 1, and the maintenance trouble tool 31 determines the model level in the timeout period derivation table 32 by referring to correspondence information between the model number and the model level (mid-range, high-end, etc.) that is stored in advance.

[0054] The maintenance fault tool 31 executes the following steps for each LDEV indicated in the LDEV list: The maintenance fault tool 31 acquires from the storage system 1 information on the drive type and RAID level of the parity group (RAID group) that allocates storage space to the target LDEV (S14, S15).

[0055] Furthermore, the maintenance fault tool 31 obtains information on the size (capacity) and usage rate (data storage rate) of the target LDEV from the storage system 1 (S16, S17). The amount of data stored in the LDEV is determined from the capacity and usage rate. The usage rate indicates the usage status of the drive that stores the data of the LDEV. It is assumed that test data is stored in advance in the LDEV. The maintenance fault tool 31 can use storage management software executed on the test execution client 3 to obtain information on these items. The storage management software obtains the information from the storage system 1 on behalf of the test execution client 3.

[0056] Next, the maintenance trouble tool 31 calculates the timeout time for the target LDEV (S18). Specifically, the maintenance trouble tool 31 acquires the coefficient for the target LDEV from the timeout time derivation table 32. The coefficient is determined from the storage model, drive type, and RAID level of the target LDEV. The maintenance trouble tool 31 calculates the timeout time from the acquired coefficient, LDEV size, and its usage rate in accordance with the above formula.

[0057] Next, the maintenance fault tool 31 issues a command to the storage system 1 to instruct it to execute a verify process (S19). The maintenance PC 5 may load a verify object file 51 into the storage system 1 immediately before step S19. In other words, the verify object file 51 may be loaded and deleted for each LDEV, or may be deleted after verification of all test target LDEVs in the storage system 1 is completed.

[0058] Next, the maintenance trouble tool 31 acquires the verification start time (S20). The start time may be acquired from the storage system 1, or may be the time when the verify command is issued.

[0059] Next, the maintenance trouble tool 31 repeatedly executes steps S21 to S23 until it exits the loop. First, the maintenance trouble tool 31 waits a specified number of seconds (S21), and then determines whether the verify process has ended or is still ongoing (S22).

[0060] For example, when the storage system 1 completes the verification process, it sends a Service Information Message (SIM) indicating the result to the test execution client 3. The maintenance trouble tool 31 determines whether the verification process has completed or is still ongoing based on whether the SIM has been received. Alternatively, the maintenance trouble tool 31 may issue a verification progress check command to the storage system 1 and make the determination based on the response.

[0061] If the verify process is ongoing (S22: Y), the maintenance trouble tool 31 acquires the current time and calculates the time that has elapsed since the verify start time. The maintenance trouble tool 31 compares the elapsed time with the calculated timeout time to determine whether a timeout has occurred (S25). If the elapsed time has not reached the timeout time (S25: N), the flow returns to step S21.

[0062] If the elapsed time has reached the timeout time (S25: Y), it is determined to be in a no-response state, and the flow returns to step S17. If the LDEV usage rate has changed due to IO processing or background processing during the timeout time, the timeout time is recalculated. Note that if a timeout has occurred (S25: Y), the flow may return to step S19.

[0063] If the verify process has finished (S22: N), the maintenance fault tool 31 refers to the SIM to determine whether the verify process has finished normally or whether an error has been detected (S23). If the verify process has finished normally (S23: Y), the verify process for the target LDEV finishes, and the next LDEV is selected from the LDEV list. If the verify process has finished other than normally (S23: N), the maintenance fault tool 31 outputs an error message to the output device 304, for example, a display device (S26). After that, the next LDEV is selected from the LDEV list.

[0064] In the above example, the process executed by the maintenance fault tool 31 may be executed by the controller 12 of the storage system 1 instead. The verify process may be executed during testing of the storage system 1, or may be executed in the storage system 1 while it is in operation.

[0065] In the above example, retries are repeated when the verification process times out. Generally, timeouts are often caused by temporary failures, so this ensures that the verification process is completed. In another example, an upper limit may be set for the number of retries. When the number of retries reaches the upper limit, the maintenance fault tool 31 stops the verification process for the target LDEV and outputs an error message.

[0066] The maintenance trouble tool 31 may shorten the timeout period in accordance with repeated retries (increase in the number of retries). The timeout period may be shortened for each retry, or may be shortened after multiple retries. The shortened period may be constant or may change as the number of retries increases. By shortening the timeout period, it is possible to reduce the waiting time when the verification process is stalled.

[0067] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0068] Furthermore, the above-mentioned components, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Furthermore, the above-mentioned components, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in memory, a storage device such as a hard disk or SSD, or a storage medium such as an IC card or SD card.

[0069] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0070] 1. Storage System 3 Storage System Test Execution Client 5 Maintenance PC 12 Storage Controller 13 physical drives 31 Maintenance Fault Tools 32 Timeout Derivation Table 51 object files 301 CPU 302 memory 304 Output Device 305 Input Device

Claims

1. An apparatus for managing a verification process in a storage system, comprising: a processor; a storage device, The processor: determining the amount of data stored in the logical device to be verified; determining a timeout period for the target logical device based on the amount of data; The apparatus controls a verify process for the target logical device based on the determined timeout period.

2. 10. The apparatus of claim 1, The processor executes a retry of the verify process when the elapsed time of the verify process reaches the timeout time.

3. 10. The apparatus of claim 1, the storage device stores timeout management information for managing information for determining a timeout period for the verification process; The timeout management information stores a coefficient associated with a drive attribute; The processor acquires a coefficient according to an attribute of a drive that stores data of the target logical device from the timeout management information, The apparatus determines a timeout period for the target logical device based on the acquired coefficient and the amount of data.

4. 4. The apparatus of claim 3, The drive attribute indicates a RAID level of a RAID group that stores data of a logical device.

5. 4. The apparatus of claim 3, The drive attribute indicates a drive type of a drive that stores data of a logical device.

6. 4. The apparatus of claim 3, The timeout management information further associates the model of the storage system with the coefficient, The drive attribute indicates a RAID level and a drive type of a RAID group that stores data of the logical device, The processor references the timeout management information and determines the timeout period based on the amount of data, RAID level, and drive type of the target logical device, and the model of the storage system.

7. 10. The apparatus of claim 1, The processor: repeatedly retrying the verify process when the timeout period has elapsed; The device shortens the timeout period in response to repeated retries.

8. 10. The apparatus of claim 1, the device is a test device that performs a test on the storage system, A device in which the program for the storage system to execute the verify process is loaded from a device different from the test device.

9. A method for controlling a verification process of a storage system by an apparatus, comprising: The device determines the amount of data stored in the logical device that is the target of the verification process; The device determines a timeout period for the target logical device based on the amount of data; The method further comprises: controlling a verify process for the target logical device based on the determined timeout period.

Images