Storage management system and storage management method

Abstract

To appropriately carry out identification of a failed job and recovery from the failure when integrally managing storages that are installed over a plurality of sites in a hybrid cloud environment.SOLUTION: The present application is applied to a system that manages storages installed in a plurality of sites with lower APIs in the storages, and has an upper API that unifies the lower APIs. The storage management system comprises: an inter-API dependence data generation unit that generates dependence data describing information on the mutual dependence between the APIs in calling the lower APIs according to the state of use of the lower APIs; an inter-job dependence data generation unit that generates dependence meta data when the upper API succeeds in executing a job; and a failure identification unit that, when the upper API fails to execute the job, identifies a failure in the lower API by using the dependence data structure generated by the inter-API dependence data generation unit and the dependence meta data generated by the inter-job dependence data generation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a storage management system and a storage management method. [Background technology]

[0002] In recent years, hybrid cloud environments have seen the centralized management of storage installed across multiple locations. In such cases, a method for managing each storage is to provide a higher-level API (Application Programming Interface) that unifies the storage at each location, in addition to the lower-level APIs that manage the storage at each location. This allows storage users to use the higher-level API to access storage at all locations, making it easier to manage storage installed across multiple locations.

[0003] Patent Document 1 describes a technology for appropriately managing stacks of virtual resource groups. The technology described in Patent Document 1 performs a rollback after a failure in managing a virtual resource group occurs, and analyzes the failure. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-170344 Summary of the Invention [Problem to be solved by the invention]

[0005] When managing storage installed across multiple locations using multiple APIs, including lower-level and upper-level APIs, if an error occurs when reading from each storage device during execution of the lower-level API, it has been difficult to identify the problem and recover from the upper-level API operated by the user. In other words, since the higher-level API does not know the dependencies between the lower-level APIs that are currently running and the dependencies between jobs, it is difficult for the higher-level API to identify the faulty job and perform failure recovery.

[0006] For example, as described in Patent Document 1, there is a technology that performs rollback and analyzes errors, but when using lower-level APIs and upper-level APIs, it is not possible to grasp the dependencies between the APIs in advance. Therefore, it has been difficult with conventional technology to identify and recover from failures that occur during the execution of upper-level APIs.

[0007] The present invention aims to provide a storage management system and a storage management method that can appropriately identify faulty jobs and perform fault recovery when centrally managing storage installed across multiple locations in a hybrid cloud environment. [Means for solving the problem]

[0008] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above problems, and one example is a storage management system that manages storage installed at multiple locations using a lower-level storage management interface provided for each storage, and has an upper-level storage management interface that controls each of the lower-level storage management interfaces. A storage management system as an example of the present application includes an inter-storage management interface dependency data generation unit that generates dependency data describing interdependency information between storage management interfaces when calling a lower storage management interface in accordance with the usage status of the lower storage management interface; an inter-job dependency data generation unit that generates dependency metadata when execution of a job by the upper storage management interface is successful; and a failure identification unit that identifies a failure in the lower storage management interface using the dependency data structure generated by the inter-storage management interface dependency data generation unit and the dependency metadata generated by the inter-job dependency data generation unit when execution of a job by the upper storage management interface fails. [Effects of the Invention]

[0009] According to the present invention, it is possible to identify the dependencies between multiple storage management interfaces to be executed and the dependencies between jobs, thereby simplifying management operations and speeding up recovery responses when an error occurs. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a configuration diagram showing an example of an entire system according to an embodiment of the present invention; [Figure 2] 1 is a configuration diagram showing an example of a storage management system according to an embodiment of the present invention; [Figure 3] FIG. 10 is a diagram showing an example of an execution history table of user requests according to an embodiment of the present invention. [Figure 4] FIG. 2 is a configuration diagram showing an example of a master server control unit and a slave server control unit according to an embodiment of the present invention. [Figure 5] 1 is a configuration diagram showing an example of a failure detection and recovery system according to an embodiment of the present invention. [Figure 6]FIG. 2 is a configuration diagram illustrating an example of a job dependency relationship data generation unit according to an embodiment of the present invention. [Figure 7] FIG. 10 is a configuration diagram illustrating an example of an inter-site dependency data asynchronous unit according to an embodiment of the present invention. [Figure 8] 3 is a configuration diagram showing an example of an API failure identification and recovery unit according to an embodiment of the present invention. FIG. [Figure 9] FIG. 2 is a configuration diagram illustrating an example of a user notification unit according to an embodiment of the present invention. [Figure 10] 10 is a flowchart showing an example of the overall flow of processing in a master server and a group of slave servers according to an embodiment of the present invention. [Figure 11] 10 is a flowchart showing an example of processing in an inter-API dependency data generation unit according to an embodiment of the present invention. [Figure 12] 10 is a flowchart illustrating an example of processing in a job dependency relationship data generation unit according to an embodiment of the present invention. [Figure 13] 10 is a flowchart illustrating an example of processing in an inter-base dependency data asynchronous unit according to an embodiment of the present invention. [Figure 14] 10 is a flowchart showing an example of processing in an API failure identification and recovery unit according to an embodiment of the present invention. [Figure 15] 10 is a flowchart illustrating an example of processing in a user notification unit according to an embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing an example of a display screen for fault notification information according to an embodiment of the present invention. [Figure 17] 10A and 10B are diagrams illustrating examples of dependency data asynchronization and failure recovery states according to an embodiment of the present invention (Example 1: when cloud storage is not used). [Figure 18] FIG. 10 is a diagram showing an example of asynchrony of dependency data and a failure recovery state (example 2: when cloud storage is used) according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] A storage management system and a storage management method according to an embodiment of the present invention (hereinafter referred to as "this embodiment") will be described below with reference to the accompanying drawings.

[0012] [System-wide configuration] FIG. 1 shows an example of the overall configuration of a hybrid cloud system 1 of this embodiment. The hybrid cloud system 1 has an on-premise provider 10a, a cloud provider 10b, and a hybrid cloud provider 10c as providers 10. The terminals of the respective providers 10a, 10b, and 10c are configured to be able to access the storage management system 20 via the cloud environment. In the following description, the providers 10a, 10b, and 10c are referred to as the first data center provider, the second data center provider, and the third data center provider, respectively.

[0013] The storage management system 20 includes a user request input unit 21, a master server control unit 30a, a slave server group control unit 30b, a storage group 33, and a database 5. The user request input unit 21 inputs and processes requests from the user terminals 1000a, 1000b, and 1000c, and requests from the businesses 10a, 10b, and 10c.

[0014] The master server control unit 30a controls the first server 31a. The slave server group control unit 30b controls the slave servers such as the second server 31b and the third server 31c. The storage group 33 includes a first storage 33a, a second storage 33b, and a third storage 33c that store data under the control of the respective servers 31a, 31b, and 31c.

[0015] 1 shows two slave servers and storages, but any number of slave servers and storages may be prepared depending on the data storage capacity, etc. In the case of the hybrid cloud system 1, the storages 33a, 33b, and 33c constituting the storage group 33 are installed at multiple locations via a network. The storage management system 20 of this example also includes a database 5. The database 5 stores the failure detection information and recovery information notified from the failure detection and recovery system 40.

[0016] When each server 31a, 31b, 31c controls each storage 33a, 33b, 33c, it uses a storage management interface called an API (Application Programming Interface). In the following description, the storage management interface will be referred to as API.

[0017] In this example, storages 33a, 33b, and 33c installed at multiple locations are controlled by lower-level APIs provided in servers 31a, 31b, and 31c, respectively. Furthermore, each server 31a, 31b, and 31c is provided with a higher-level API that manages the lower-level APIs, and the higher-level APIs execute the interoperation process between the storages and servers at each location. The configuration for control using these APIs is described in Figure 4.

[0018] The hybrid cloud system 1 of this example includes a failure detection and recovery system 40. The fault detection and recovery system 40 includes an inter-API dependency data generation unit 50, an inter-job dependency data generation unit 60, an inter-site dependency data asynchronous unit 70, an API fault identification and recovery unit 80, and a user notification unit 90. The fault detection and recovery system 40 also includes a database 6. The database 6 stores fault detection information and associated recovery information obtained by the fault detection and recovery system 40.

[0019] The configurations of the processing units 50 to 90 that make up the failure detection and recovery system 40 will be described with reference to FIGS. The hardware configuration of the fault detection and recovery system 40 includes a CPU (Central Processing Unit) 41, a storage unit 42, an interface 43, and the like, all of which are connected via a bus line so that data can be transferred between them. When the CPU 41 executes a program stored in the storage unit 42, the above-mentioned processing units 50 to 90 and database 6 are configured in the storage unit 42.

[0020] In the following configuration and processing, we will explain the case where data (source data) stored in storage 33a as the first server (master server) is copied (destination data) to storage 33b, 33c as the second or third server (slave server).

[0021] [Configuration of user request input section] 2 shows the configuration of the user request input unit 21 of the storage management system 20. The user request input unit 21 shown in FIG. The user request input unit 21 includes a user request execution program 100, a higher level abstract API request extraction program 110, and a request source and copy destination identification program 120 from the request.

[0022] The user request execution program 100 stores user requests in a database as a user request execution history table 200 . The higher level abstract API request extraction program 110 extracts requests for higher level APIs from the user requests stored in the user request execution history table 200 . The request-based copy source and copy destination identification program 120 identifies the data copy source storage and copy destination storage based on the user request.

[0023] FIG. 3 shows an example of the data structure of the user request execution history table 200. As shown in FIG. As shown in FIG. 3, the user request execution history table 200 has columns for user 201, node 202, request ID 203, requested content 204, and timestamp 205. The column for user 201 stores data for identifying user terminals 1000a, 1000b, and 1000c of the first user, second user, and so on.

[0024] The node 202 field stores the node that executed the request, such as the master or the first slave. The request ID 203 field stores an ID assigned to each request from a user terminal. The requested content 204 field stores the specific content of the request. The timestamp 205 field stores the date and time when the request was issued.

[0025] [Storage Management System Server Configuration] FIG. 4 shows the configuration of the master server control unit 30a and the slave server group control unit 30b of the storage management system 20. As shown in FIG. The master server control unit 30a includes a higher-level API call execution unit 35 and a lower-level API call execution unit 36. The storage device of the higher level API call execution unit 35 holds the higher level abstract API call program 210 for the copy source. The storage device of the lower level API call execution unit 36 ​​holds the lower level batch execution API call program 220 for the copy source.

[0026] A slave server group control unit 30b is provided for each of the slave storages 33b, 33c, etc. shown in Fig. 1. Each slave server group control unit 30b includes a higher-level API call execution unit 37 and a lower-level API call execution unit 38. The storage device of the higher level API call execution unit 37 holds a higher level abstract API call program 230 for a plurality of copy destinations. The storage device of the lower level API call execution unit 38 holds a lower level abstract execution API call program 240 for a plurality of copy destinations.

[0027] [Fault detection and recovery system] Next, the configuration of the failure detection and recovery system 40 will be described with reference to FIGS. FIG. 5 shows the configuration of an inter-API dependency data generator (inter-storage management interface dependency data generator) 50 in the failure detection and recovery system 40. As shown in FIG. The inter-API dependency data generation unit 50 includes a master server side processing unit 51a and a slave server side processing unit 51b. The storage device of the master server side processing unit 51 a stores a root dependency relationship data structure creation program 310 . The storage device of the slave server side processing unit 51b holds the storage dependency relationship data structure creation program 320. Note that the number of slave server side processing units 51b provided is equal to the number of slaves.

[0028] FIG. 6 shows the configuration of a job dependency relationship data generation unit (job dependency relationship data generation unit) 60 in the failure detection and recovery system 40. As shown in FIG. The job dependency relationship data generating unit 60 includes a master server side processing unit 61a and a slave server side processing unit 61b. The storage device of the master server-side processing unit 61a stores a lower-level combination API execution program 410a, a lower-level API request detection program 420a, and a job dependency relationship data structure creation program 430a. The storage device of the master server-side processing unit 61a also stores a job sequential execution program 440a and an update program 450a for job dependency relationships.

[0029] The storage device of the slave server-side processing unit 61b holds a lower-level combination API execution program 410b, a lower-level API request detection program 420b, and a job dependency relationship data structure creation program 430b. The storage device of the slave server-side processing unit 61b also holds a job sequential execution program 440b and an update program 450b for job dependency relationships.

[0030] FIG. 7 shows the configuration of the inter-site dependency relationship data asynchronization unit 70 in the failure detection and recovery system 40. The storage device of the inter-base dependency data asynchronous unit 70 holds an update program 710 for root dependency and job dependency, and an update program 720 for storage dependency and job dependency. The storage device of the inter-base dependency data asynchronous unit 70 holds an asynchronous processing program 730 for root dependency and inter-storage dependency, and an integration program 740 for integrating information on the execution results of each job between APIs into the root dependency data.

[0031] FIG. 8 shows the configuration of the failure identification and recovery unit 80 in the failure detection and recovery system 40. The storage device of the fault identification and recovery unit 80 holds a program 610 for detecting faults between lower-level combination APIs, a program 620 for determining whether or not to roll back to a faulty job from the dependency data structure, and a program 630 for identifying a faulty job from root dependency data. The rollback here is for finding an automatic recovery checkpoint where there is no fault, and returning to that automatic recovery checkpoint to perform an automatic rollback and retrying the process. If automatic rollback is enabled, it is preferable to periodically notify the higher-level API of the dependency relationships between jobs of the lower-level API and the execution status information of each task. The storage device of the fault identification and recovery unit 80 also holds a recovery program 640 for the faulty job or a job that has a dependency relationship with the faulty job.

[0032] In this example, the fault identification and recovery unit 80 is configured to have a fault identification function as a fault identification unit that identifies a fault and a fault recovery function that recovers from the identified fault. On the other hand, if, for example, a different system performs recovery from the fault, the fault identification and recovery unit 80 may be configured to have only the function as a fault identification unit.

[0033] FIG. 9 shows the configuration of the user notification unit 90 in the failure detection and recovery system 40. The storage device of the user notification unit 90 holds a program 510 for outputting the status of jobs executed between APIs, and a program 520 for extracting fault information of faulty jobs and jobs having dependencies with the faulty jobs from the root dependency data structure. The storage device of the user notification unit 90 also holds an extraction program 530 for recovery information of a faulty job and jobs having a dependency relationship with the faulty job from the root dependency relationship data structure, and a result notification program 540 for the batch execution API.

[0034] [Overall processing flow between the master server and the slave server group] FIG. 10 is a flowchart showing the overall processing flow in the master server and the slave server group. 10 shows an example in which the first data center operator 10a transmits a user request in step S1, and the second data center operator 10b transmits a copy configuration change request in step S10. That is, the example in Fig. 10 shows an example of processing by the master node (master server control unit) 30a and slave node (slave server control unit) 30b of the storage management system 20 when these requests are transmitted.

[0035] First, the first data center operator 10a transmits a user request to the master node 30a (step S1). The master node 30a receives the user request and detects a higher-level abstract API request (step S2). The master node 30a then determines whether a dependency relationship has been determined (step S3). If the dependency relationship determination has not been completed in step S3 (No in step S3), the process returns to step S2.

[0036] If the determination of the dependency relationship is completed in step S3 (Yes in step S3), the master node 30a causes the slave node 30b to execute a remote copy of the higher-level abstract API (step S4). The master node 30a then performs a process of generating a root dependency data structure (step S5), and the slave node 30b also performs a process of generating a root dependency data structure (step S6). The process of generating a root dependency data structure in steps S5 and S6 will be described in detail later with reference to FIG.

[0037] Next, the master node 30a performs a process of generating a job dependency relationship data structure (step S7), and the slave node 30b also performs a process of generating a job dependency relationship data structure (step S8). Details of the process of generating a job dependency relationship data structure in steps S7 and S8 will be described later with reference to FIG.

[0038] Next, the master node 30a and the slave node 30b execute the lower-level APIs in a batch (step S9). Now, let us assume that the second data center provider 10b has sent a copy configuration change request (step S10). At this time, the slave node 30b updates the dependency relationship between the storage and the job (step S11). The master node 30a also updates the dependency relationship between the storage and the job (step S12). Thereafter, the slave node 30b sends an information update request to the master node 30a (step S13). The master node 30a, which has received the information update request, performs asynchronous processing of the dependency relationship between the root and the storage (step S14). Furthermore, the master node 30a executes a fault identification and recovery mechanism (step S15) and determines whether a fault has been detected (step S16).

[0039] If it is not determined in step S16 that a fault has been detected (No in step S16), the slave node 30b restores the faulty job (step S18). If it is determined in step S16 that a fault has been detected, the master node 30b restores jobs that have a dependency relationship with the faulty job (step S19). Then, the slave node 30b notifies the user (the second data center operator 10b) of the recovery result (step S20), and the master node 30 also notifies the user of the recovery result (step S21).

[0040] FIG. 11 is a flowchart showing an example of processing (storage management interface dependency relationship data generation processing) in the inter-API dependency relationship data generation unit 50 (see FIG. 5) in the failure detection and recovery system 40. The inter-API dependency data generation unit 50 detects a request to a higher-level abstract API (step S31). Having detected the request, the inter-API dependency data generation unit 50 determines whether or not the copy pair relationship has been determined from the request (step S32). If the copy pair relationship cannot be determined in step S32 (No in step S32), the inter-API dependency data generation unit 50 returns to the request detection process of step S31.

[0041] If the copy pair relationship can be determined in step S32 (Yes in step S32), the master server side processing unit 51a creates route dependency relationship data for the copy source node from the node information included in the request (step S33). Also, the slave server side processing unit 51b creates storage dependency relationship data for the copy destination node from the node information included in the request (step S34). Multiple copies of storage dependency relationship data for the copy destination are created in step S34 according to the number of copy destinations.

[0042] FIG. 12 is a flowchart showing an example of processing (inter-job dependency relationship data generation processing) in the inter-job dependency relationship data generation unit 60 (see FIG. 6) in the failure detection and recovery system 40. The job dependency data generating unit 60 generates dependency data between jobs of the executed lower-level APIs (step S41). Next, the job dependency relationship data generating unit 60 detects a request to the lower combination API (step S42). Thereafter, the storage management system 20 executes all jobs of the lower level API (step S43).

[0043] FIG. 13 is a flowchart showing an example of processing in the inter-site dependency relationship data asynchronization unit 70 (see FIG. 7) in the failure detection and recovery system 40. In FIG. The inter-base dependency data asynchronization unit 70 updates the storage dependency relationship and the job dependency relationship (step S51). Also, the inter-base dependency data asynchronization unit 70 updates the route dependency relationship and the job dependency relationship (step S52). Then, the inter-site dependency data asynchronous unit 70 asynchronously processes the root dependency and the dependency between storages each time a slave server completes job execution (step S53). Through this asynchronous processing, information on the execution results of each job between APIs and an integrated file are created in the root dependency data.

[0044] FIG. 14 is a flowchart showing an example of processing in the API failure identification and recovery unit 80 (see FIG. 8) in the failure detection and recovery system 40. First, the API failure identification and recovery unit 80 detects an error in a lower-level API and starts processing (step S100). When the process starts in step S100, the API failure identification and recovery unit 80 determines whether or not the faulty job has an automatic rollback from the dependency data structure (step S200).

[0045] When it is determined in step S200 that automatic rollback is not to be performed (step S301), the API failure identification and recovery unit 80 identifies the faulty job from the root dependency relationship data (step S302). Then, the API failure identification and recovery unit 80 determines whether or not the dependency relationship between the faulty job identified in step S302 and the jobs having the dependency relationship therebetween has been confirmed (step S303).

[0046] If the dependency relationship can be confirmed in step S303, the API failure identification and recovery unit 80 recovers from the failure by re-executing the faulty job and modifying the job based on the route-dependent data relationship (step S304). If the dependency relationship cannot be confirmed in step S303, the API failure identification and recovery unit 80 recovers from the failure by re-executing the job that has a dependency relationship with the failure and by changing the job based on the route dependency data relationship (step S305). Then, the API failure identification and recovery unit 80 recovers from the failure by re-executing or changing the failed job (step S306).

[0047] Furthermore, when it is determined in step S200 that a rollback is to be performed (step S401), the API failure identification and recovery unit 80 identifies the faulty job from the root dependency relationship data (step S402). Then, the API failure identification and recovery unit 80 stops the execution of the identified failed job (step S403). Then, the API failure identification and recovery unit 80 determines whether or not the dependency relationship between the identified failed job and the jobs having the dependency relationship therebetween has been confirmed (step S404).

[0048] If the dependency relationship can be confirmed in step S404, the API failure identification and recovery unit 80 recovers from the failure by re-executing automatic rollback from the automatic recovery checkpoint of the failed job (step S405). If the dependency relationship cannot be confirmed in step S407, the API failure identification and recovery unit 80 recovers from the failure by re-executing the rollback of the job that has a dependency relationship with the failure (step S406). Then, the API failure identification and recovery unit 80 recovers the failed job by re-executing the automatic rollback (step S407).

[0049] FIG. 15 is a flowchart showing an example of processing in the user notification unit 90 (see FIG. 9) in the failure detection and recovery system 40. The user notification unit 90 extracts meta information of the job executed between APIs (step S501). Next, the user notification unit 90 extracts the fault information of the faulty job and the jobs that have a dependency relationship with the faulty job from the root dependency relationship data structure (step S502). Furthermore, the user notification unit 90 extracts recovery information of the faulty job and jobs that have a dependency relationship with the faulty job from the root dependency relationship data structure (step S503). Then, the user notification unit 90 notifies the user (data center operator) of the result of the batch execution API (step S504). The terminals of the operators 10a, 10b, and 10c display the result notified in step S507.

[0050] [Notification screen by user notification section] FIG. 16 shows an example of a notification screen 1000 displayed on the terminals of the businesses 10a, 10b, and 10c upon notification by the user notification unit 90. 16 has display item setting buttons 1010 on the left side of the screen. The display item setting buttons 1010 include a copy policy setting button, a fault monitoring / management button, and a system setting button.

[0051] Specific display items of the notification screen 1000 include a user setting area 1020 and a fault notification area 1030. The user setting area 1020 corresponds to the user request sent in step S1 of the flowchart in FIG. 10, and displays a policy description for remote copying between bases. Specifically, provisioning settings 1021 such as a data copy source address, a copy destination address, etc. are input from a terminal and displayed in the user setting area 1020. Then, they are sent to the storage management system 20 in response to a user request (step S1 in FIG. 10).

[0052] A fault notification area 1030, which is an area where notifications are sent by the user notification unit 90, displays a fault monitoring / recovery record and a fault notification title 1031. Details of a copy source table 1033 and a copy destination table 1034 are written and displayed in a fault monitoring / recovery record column 1032 of the fault notification area 1030. Furthermore, the fault monitoring and recovery record field 1032 displays a dependency relationship JSON data download button 1035. By selecting the download button 1035 on the terminal displaying this screen, the dependency relationship JSON data can be downloaded and displayed. JSON is a format used when processing text data with an API.

[0053] In the fault notification column 1041, detailed fault information 1042 is displayed. Specifically, the detailed information 1042 displays the circumstances under which the error occurred and how the recovery was carried out for each failure.

[0054] As described above, according to this embodiment, in a configuration in which storage devices installed at multiple locations are managed by upper and lower APIs, a failure that occurs in each storage device can be easily detected and recovered. In other words, if a failure occurs in a lower API during execution of the upper API, the upper API can identify the failure and notify the user (business operator). Furthermore, recovery from a failure in the lower API can be achieved through control by the upper API.

[0055] [Variations] The embodiment examples described so far have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. For example, in the system 1 shown in Figure 1, each storage is a cloud storage, but it can also be applied to a case where each storage is installed at multiple locations, or where only some of the multiple storages are cloud storage. Examples of such cases are shown below in Figures 17 and 18.

[0056] The example of FIG. 17 shows an example of a configuration having an on-premise data center 2001 as a master, and on-premise data centers 2002 and 2003 as a first slave and a second slave, respectively, at different locations. In the configuration of Fig. 17, cloud storage is not used, but the storage configuration has multiple bases. In the example of Fig. 17, each base has a higher-level API (higher-level abstract API) and a lower-level API, so the example of Fig. 17 can be said to have substantially the same configuration as system 1 shown in Fig. 1.

[0057] Each of the data centers 2001, 2002, and 2003 has an upper abstract API and a lower API, and remote copying is performed between bases in response to a request from the first business operator, for example. As explained in FIG. 1 and other figures, the data centers 2001, 2002, and 2003 are configured to acquire dependency data structures and the like using a failure detection and recovery system to identify and recover from failures.

[0058] For example, the master data center 2001 acquires dependency information between storage and jobs from slave data centers 2002 and 2003 through asynchronous processing of dependency information between the root and storage. It is preferable to periodically acquire this dependency information between storage and jobs. When a failure occurs in data center 2002 or data center 2003, data center 2001 identifies the failure based on the acquired dependency information, etc., and executes recovery processing. Furthermore, data center 2001 notifies the user (first business operator) of the failure identification and recovery results.

[0059] 18 shows an example of a configuration having an on-premise data center 3001 as a master, and on-premise data centers 3002 and 3003 as a first slave and a second slave. In the example of Fig. 18, only the on-premise data center 3003 as the second slave is cloud storage.

[0060] In the example of FIG. 18, the data centers 3001, 3002, and 3003 also have a configuration having a higher level abstract API and a lower level API, and remote copying is performed between bases in response to a request from the first business operator, for example. As explained in FIG. 1 and other figures, the data centers 3001, 3002, and 3003 are configured to acquire dependency data structures and the like using a failure detection and recovery system to identify and recover from failures.

[0061] 18, the master data center 3001 obtains dependency information between storage and jobs from slave data centers 3002 and 3003 by asynchronously processing dependency information between the root and storage. It is preferable to obtain this dependency information between storage and jobs periodically. When a failure occurs in the data center 3002 or the data center 3003, the data center 3001 identifies the failure based on the acquired dependency information and the like, and performs recovery processing.

[0062] Here, the data center 3003, which is cloud storage, is also used by another business operator, so there is a possibility that a request to change the copy configuration (request 2 shown in FIG. 18) may be made from the second business operator, as shown in Fig. 18. If this request 2 causes a change in the lower-level API of the data center 3003, the dependency information between the storage and the job is updated. Therefore, the data center 3003 acting as the master can identify the failure when it occurs based on the obtained dependency information, and can recover from the identified failure. As described above, the present invention can also be applied to a configuration having storage at multiple locations that is not cloud storage, or a configuration that combines in-house storage and cloud storage.

[0063] 1, the failure detection and recovery system 40 detects a failure and recovers from the detected failure. However, the failure detection and recovery system 40 may detect a failure, notify a user such as a business operator of the detected failure, and recover from another system based on the failure information.

[0064] In addition, in the configuration diagrams shown in Figure 1 and elsewhere, only the control lines and information lines considered necessary for explanation are shown, and not all control lines and information lines in the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected. Furthermore, the flowcharts shown in FIG. 10 and the like are also examples, and as long as the processing results are the same, the order of some of the processing may be changed or multiple processes may be executed simultaneously.

[0065] Furthermore, the fault detection and recovery system 40 described in each of the above-described embodiments may be configured by implementing a program that executes the processing shown in the flowchart of Fig. 10, and in this case, the program is prepared in the storage unit 42 shown in Fig. 1. Alternatively, the program executed by the computer serving as the fault detection and recovery system 40 may be stored in a recording medium such as an external memory, an IC card, an SD card, or an optical disk, and transferred to the computer that functions as the fault detection and recovery system 40. [Explanation of symbols]

[0066] 1. Hybrid cloud system 5,6…Database 10,10a,10b,10c…Business operator 20...Storage management system 21...User request input section 30a...master server control unit (master node), 30b... Slave server group control unit (slave node) 31a...First server 31b...Second server 31c...Third server 33...Storage group 33a…1st Storage 33b…Second storage 33c…Third storage 35...High-level API call execution unit 36... Lower-level API call execution unit 37...High-level API call execution unit 38... Lower-level API call execution unit 40...Fault detection and recovery system 41...CPU 42...Storage section 43...Interface 50...Inter-API dependency data generation section 51a...Master server side processing unit 51b...Slave server side processing unit 60...Job dependency data generation unit 61a...Master server side processing unit 61b...Slave server side processing unit 70...Inter-site dependency data asynchronous section 80...API Fault Identification and Recovery Department 90...User notification section 100...execution program 100b...memory 100c…Storage 101...Work Plan Database 109...Work continuation determination unit 110...Request extraction program 120...Identification Program 200...User request execution history table

Claims

1. A storage management system that manages storages installed at a plurality of locations using a lower-level storage management interface provided for each storage, and has a higher-level storage management interface that controls each of the lower-level storage management interfaces, a storage management interface dependency data generation unit that generates dependency data describing interdependency information between storage management interfaces when the lower level storage management interface is called, in accordance with a usage status of the lower level storage management interface; a job dependency data generation unit that generates dependency metadata when the job is successfully executed by the upper storage management interface; a failure identification unit that, when execution of a job by the upper storage management interface fails, identifies a failure in the lower storage management interface by using the dependency data structure generated by the inter-storage management interface dependency data generation unit and the dependency metadata generated by the inter-job dependency data generation unit. Storage management system.

2. The failure identification unit acquires dependency data of the job being executed from the storage management interface dependency data generation unit and identifies the job in which the failure occurred. The storage management system according to claim 1 .

3. The dependency data generated by the storage management interface dependency data generation unit includes data on whether or not an automatic rollback is performed; When the execution of a job by the upper storage management interface fails and there is no automatic rollback, the failure identification unit acquires dependency data of the job being executed from the data generated by the storage management interface dependency data generation unit, and notifies the upper storage management interface of the dependency relationships between the jobs of the lower storage management interface and error-related information. The storage management system according to claim 2 .

4. Furthermore, the failure identification unit has a failure recovery function, Restarting a failed job with an auto-recovery checkpoint The storage management system according to claim 3 .

5. The failure identification unit periodically notifies the upper storage management interface of the dependency relationships between jobs of the lower storage management interface and execution status information of each task when automatic rollback is enabled. The storage management system according to claim 3 .

6. Furthermore, the failure identification unit has a failure recovery function, Restart jobs that have dependencies on the failed job to recover from the failure The storage management system according to claim 3 .

7. A storage management method in which storage devices installed at multiple locations are managed by a lower-level storage management interface provided for each storage device, and a higher-level storage management interface is provided to control each of the lower-level storage management interfaces, a storage management interface dependency data generation process for generating dependency data describing interdependency information between storage management interfaces when the lower level storage management interface is called, in accordance with the usage status of the lower level storage management interface; a job dependency data generation process for generating dependency metadata when the job is successfully executed by the upper storage management interface; and a failure identification process for identifying a failure in the lower storage management interface when execution of a job by the upper storage management interface fails, using the dependency data structure generated by the storage management interface dependency data generation process and the dependency metadata generated by the job dependency data generation process. Storage management methods.

Images