Storage system and volume migrating method in storage system

Abstract

To migrate a volume between nodes without suspending an I / O request from a host in a storage system that includes a plurality of storage nodes.SOLUTION: In a system S that includes a first storage system and a second storage system, there is a nonsynchronous-copy pair relation between a first volume of the first storage system and a second volume of the second storage system. The system S creates, with the second volume being as a volume subjected to migration, a copy volume of the second volume in the other storage node in the second storage system. Next, the system reflects, in the copy volume, difference data between the first volume and a snapshot of the first volume. Subsequently, the system newly creates a nonsynchronous-copy pair relation between the first volume and the copy volume in such a way that data on an I / O request from a host to the first volume is reflected on the copy volume in a nonsynchronous manner.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present invention relates to a storage system and a volume migration method in a storage system. [Background technology]

[0002] In a storage system having multiple storage nodes, it is necessary to move volumes between nodes when rebalancing the load or capacity between the nodes, removing nodes, etc. For example, Patent Document 1 discloses a conventional technology in which one of two volumes for which a copy pair of a volume is created is moved between storage controllers operating on different nodes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 9,563,383 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above-mentioned conventional technology, when moving a paired volume between storage controllers, I / O requests from the host had to be temporarily suspended when switching the journals used before and after the move, which caused a problem of affecting user operations related to host I / O.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to move paired volumes between storage nodes in a storage system having multiple storage nodes without stopping I / O requests from the host. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention provides a storage system configured to include storages each having a storage node including a storage controller that processes I / O requests from a host and a storage drive, wherein a first storage has a first volume in a first storage node, and a second storage has a second volume in a second storage node, and a first pair relationship of asynchronous copy is created between the first volume and the second volume, in which data related to the I / O request from the host to the first volume is asynchronously reflected in the second volume, and the storage controller creates a snapshot of the first volume in the first storage, and after creating the snapshot, The first pair relationship is deleted to stop the asynchronous copy, and while continuing to process the I / O request from the host to the first volume, the first volume or the second volume is moved as a volume to be moved to another storage node of the storage that has the volume to be moved, and after the movement of the first volume or the second volume to the other storage node is completed, a second pair relationship of the asynchronous copy is created between the first volume and the second volume, and after the second pair relationship is created, the first volume and the snapshot are compared to identify difference data between the first volume and the snapshot, and the identified difference data is reflected in the second volume. [Effects of the Invention]

[0007] According to the present invention, in a storage system having a plurality of storage nodes, a paired volume can be moved between storage nodes without stopping I / O requests from the host. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram showing an outline of processing in the storage system according to the first embodiment. [Figure 2]FIG. 2 is a diagram showing the configuration of a storage node according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing the configuration of a memory of a storage node according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing the configuration of a storage management table according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing the configuration of a pool management table according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing the configuration of a volume management table according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing the configuration of a snapshot management table according to the first embodiment. [Figure 8] FIG. 3 is a diagram showing the configuration of a node management table according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing the configuration of a volume mapping table according to the first embodiment. [Figure 10] FIG. 4 is a diagram showing the configuration of a snapshot mapping table according to the first embodiment. [Figure 11] FIG. 10 is a diagram showing the configuration of an asynchronous copy pair table according to the first embodiment. [Figure 12] FIG. 10 is a diagram showing the configuration of a synchronous copy pair table according to the first embodiment. [Figure 13] FIG. 2 is a diagram showing the configuration of a journal table according to the first embodiment. [Figure 14] FIG. 4 is a diagram showing a differential bitmap according to the first embodiment. [Figure 15] FIG. 4 is a diagram showing a copy area management bitmap according to the first embodiment. [Figure 16] FIG. 10 is a sequence diagram showing a copy destination volume migration process according to the first embodiment. [Figure 17] 10 is a flowchart showing a snapshot creation process according to the first embodiment. [Figure 18] 10 is a flowchart showing a pair creation process by differential copying according to the first embodiment. [Figure 19] 10 is a flowchart showing a copy area management bitmap creation process according to the first embodiment. [Figure 20] 10 is a flowchart showing a difference acquisition process according to the first embodiment. [Figure 21] 10 is a flowchart showing a copy metadata creation process according to the first embodiment. [Figure 22] FIG. 10 is a diagram showing an outline of processing in a storage system according to a second embodiment. [Figure 23] FIG. 10 is a sequence diagram showing a source volume migration process according to the second embodiment. [Figure 24A] FIG. 11 is a diagram showing an outline of processing in a storage system according to a third embodiment. [Figure 24B] FIG. 11 is a diagram showing an outline of processing in a storage system according to a third embodiment. [Figure 25] FIG. 11 is a sequence diagram showing a pair deletion process that takes into consideration a CTG according to the third embodiment. [Figure 26] FIG. 11 is a sequence diagram showing a resync process for pair creation taking into consideration the CTG according to the third embodiment. [Figure 27] FIG. 11 is a sequence diagram showing a resync process according to the third embodiment. [Figure 28A] FIG. 10 is a diagram showing an outline of processing in a storage system according to a fourth embodiment. [Figure 28B] FIG. 10 is a diagram showing an outline of processing in a storage system according to a fourth embodiment. [Figure 29] FIG. 13 is a sequence diagram showing a pair deletion process that takes into consideration the CTG according to the fourth embodiment. [Figure 30] FIG. 13 is a sequence diagram showing a resync process for pair creation taking into consideration the CTG according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment of the present invention will be described in detail below with reference to the drawings.

[0010] In the following description, an "interface device" may refer to one or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., an NIC and an HBA (Host Bus Adapter)).

[0011] In the following description, "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0012] In the following description, a "storage drive" is an example of one or more persistent storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and specifically may be, for example, a hard disk drive (HDD), a solid state drive (SSD), or a non-volatile memory express (NVMe) drive.

[0013] Furthermore, in the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a hardware circuit that performs part or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0014] In the following description, information that provides an output for an input may be described using expressions such as "xxx table." This information may be data of any structure (e.g., structured data or unstructured data), or may be a learning model such as a neural network, genetic algorithm, or random forest that generates an output for an input. Therefore, an "xxx table" may be referred to as "xxx information." In the following description, the structure of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0015] In the following description, processing may be described using a "program" as the subject. A program is executed by a processor to perform a predetermined process using a storage device and / or an interface device, etc., as appropriate. Therefore, the subject of processing may be the processor (or a device such as a controller having the processor). A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] Furthermore, in the following description, when describing elements of the same type without distinguishing between them, the common portion (the portion excluding the branch number) of the reference sign including the branch number may be used, and when describing elements of the same type while distinguishing between them, the reference sign including the branch number may be used. For example, when storage nodes are not particularly distinguished, they may be referred to as "storage node 10," and when individual storage nodes such as the source and destination of a volume are distinguished, they may be referred to as "storage node 10a" and "storage node 10b." Another method of describing elements of the same type while distinguishing between them is to use the element's ID (e.g., identification number). Specifically, for example, "journal 10Ja" and "journal 10Ja2" may be referred to as "journal #1" and "journal #2." [Example]

[0017] A first embodiment of the present invention will be described below with reference to FIGS.

[0018] (Processing Overview of System S According to First Embodiment) Fig. 1 is a diagram illustrating an overview of the processing of a system S according to Example 1. Fig. 1 illustrates control in the system S when a copy destination volume of a copy destination site, which forms an asynchronous copy pair with a copy source volume of a copy source site, is moved between storage nodes of the copy destination site.

[0019] A volume 10Va in a storage node 10a in a copy source site 1a is copied to a storage node 10b in a copy destination site 1b. The data in the volume 10Va is written to a storage drive (not shown) via a pool 10Pa. At this time, a journal 10Ja (journal #1) created in the copy source storage node 10a and a journal 10Jb (journal #3) created in the copy destination storage node 10b are used.

[0020] The copy source site 1a and the copy destination site 1b are not limited to being different remote sites, but may be the same site. That is, the storage system including the storage node 10a and the storage system including the storage nodes 10b and 10b2 may be storage systems located at the same site.

[0021] The journal 10Ja temporarily stores data related to updates to the copy source volume 10Va. In the copy destination storage node 10b, the data accumulated in the journal 10Ja is read and written to the journal 10Jb. The data accumulated in the journal 10Jb is then reflected in the copy destination volume 10Vb. As a result, data is asynchronously copied from the copy source volume 10Va to the copy destination volume 10Vb. The data in volume 10Vb is written to a storage drive (not shown) via pool 10Pb.

[0022] The storage of the copy destination site 1b is configured to include a plurality of storage nodes 10b and 10b2. In the copy destination site 1b, in order to equalize the load between the storage nodes 10b and 10b2, the volume 10Vb is rebalanced between the storage nodes 10b and 10b2.

[0023] In this case, if data is re-copied from the source site 1a to the destination site 1b to recreate the asynchronous copy pair in order to maintain the asynchronous copy pair relationship, the network load between the source site 1a and the destination site 1b will increase, which is inconvenient.

[0024] Therefore, in this embodiment, to avoid re-copying, volume 10Vb2 is created at the copy destination site 1b as a synchronous copy volume (second copy volume) of volume 10Vb. Then, while maintaining the asynchronous copy pair relationship, the volume is migrated from volume 10Vb in storage node 10b to volume 10Vb2 in storage node 10b2.

[0025] The method for moving the copy destination volume from volume 10Vb to volume 10Vb2 is as follows: First, in step S1, copy destination volume 10Vb is synchronously copied between storage nodes 10b and 10b2 (data D0 and data D1 are copied to volume Vb2).

[0026] Next, in step S2, a snapshot 10Sa of the source volume 10Va is created on the source storage node 10a. When the snapshot 10Sa is created, the asynchronous copy pair relationship between the volumes 10Va and 10Vb is deleted. The snapshot 10Sa is used to identify the differences in the subsequent stages. Note that the copy of the source volume is not limited to a snapshot, and other types of copies can be used as long as they allow for the identification of the differences in the subsequent stages.

[0027] Here, the copy source volume 10Va contains differential data D2 that has not yet been copied to the copy destination volume 10Vb. If only the differential data D2 is copied, the new copy destination volume 10Vb2 will have the same data as the copy source volume 10Va. In other words, the copy destination volume can be moved with the asynchronous copy pair relationship established.

[0028] Therefore, in step S3, snapshot 10Sa is compared with source volume 10Va, and it is determined that differential data D2 is the difference. Next, in step S4, the determined differential data D2 is copied to volume 10Vb2, and an asynchronous copy pair relationship is established between the source volume 10Va and the new destination volume 10Vb2.

[0029] At this time, journal 10Jb2 (journal #4) is created in storage node 10b2, the destination of the volume, and journal 10Ja2 (journal #2), which is a pair of journal 10Jb2, is created in storage node 10a, the source of the copy. Differential data D2 is copied to volume 10Vb2 via journals 10Ja2 and 10Jb2. The data in volume 10Vb2 is written to a storage drive (not shown) via pool 10Pb2.

[0030] (Configuration of the storage node 10 according to the first embodiment) 2 is a diagram showing the configuration of a storage node 10 according to the first embodiment. A storage ST including one or more storage nodes 10 is arranged at each site. The storage node 10 is connected to a host 20 via a network N1 and receives I / O (Input / Output) requests from the host 20. The storage ST is an example of a first storage system including a storage node included in the system S (FIG. 1), and a second storage system including a storage node.

[0031] The storage node 10 is also connected to a management computer 30 via a network N2, and the storage configuration and the like are managed by the management computer 30. The storage node 10 is also connected to storage STs at other sites via a network N3, and performs asynchronous copying of volumes between the storage STs at the other sites. The networks N1, N2, and N3 may be the same network or different networks.

[0032] The storage node 10 has one or more storage controllers 11. The storage controller 11 has one or more processors 12, one or more volatile or non-volatile memories 13, front-end I / Fs (Interfaces) 14 and 16, a management I / F 15, and a back-end I / F 17. One or more storage drives 18 having physical persistent storage areas are connected to the storage node 10.

[0033] The processor 12 receives data related to an I / O request from the host 20 via the front-end I / F 14, caches the data in the memory 13, and accesses and processes the storage drive 18 via the back-end I / F 17. The processor 12 also manages the storage drive 18 via the back-end I / F 17 based on commands related to management of the configuration, etc., received from the management computer 30 via the management I / F 15.

[0034] The processor 12 also reads data stored in a volume logically constructed from the storage area of ​​the storage drive 18 provided to the host 20, and copies the data to the storage node 10b at another site via the front-end I / F 16.

[0035] (Configuration of the memory 13 of the storage node 10 according to the first embodiment) 3 is a diagram illustrating the configuration of the memory 13 of the storage node 10 according to Example 1. The memory 13 includes a control information section 131 that is a storage area for control information, a program section 132 that is a storage area for programs, and a cache section 133 that is a temporary storage area for data related to I / O from the host 20.

[0036] The control information unit 131 stores a storage management table 131a, a pool management table 131b, a volume management table 131c, a snapshot management table 131d, a node management table 131e, and a volume mapping table 131f. The control information unit 131 also stores a snapshot mapping table 131g, an asynchronous copy pair management table 131h, a synchronous copy pair management table 131i, and a journal management table 131j. The control information unit 131 also stores a differential bitmap 131k and a copy area management bitmap 131l.

[0037] 4, the storage management table 131a has columns for "storage ID," "total capacity," "used capacity," "address," and "management port." The storage management table 131a is management information for the storage nodes 10 that constitute the storage ST.

[0038] "Storage ID" is identification information for each storage ST that is configured to include a storage node 10. "Total capacity" is capacity information for the storage ST in question. "Used capacity" is used capacity information for the storage ST in question. "Address" is information for accessing each storage ST for I / O, and is shown as an IP address as an example. "Management port" is information for accessing the storage ST in question for management of configuration, etc., and is shown as an IP address as an example.

[0039] 5, the pool management table 131b has the columns of "Page ID," "Start Address," "Status," "Allocated Volume ID / Snapshot ID," and "Allocated Address." The pool management table 131b contains information for managing the usage status of the pool in units of pages, which are fixed-size areas.

[0040] "Page ID" is identification information for each page. "Start address" indicates the address (position) of the page in the pool. "Status" indicates the usage status of the page, i.e., whether the page has already been allocated to a volume or snapshot. "Assigned volume ID / snapshot ID" indicates the ID of the volume or snapshot to which the page has been allocated. "Assigned address" indicates the allocation location (address within the volume or snapshot) of the volume or snapshot to which the page has been allocated.

[0041] As shown in FIG. 6, the volume management table 131c has columns for "Volume ID," "Size," and "Storage Node ID." The volume management table 131c is control information for managing volumes. "Volume ID" is identification information for each volume. "Size" is size information for the corresponding volume. "Storage Node ID" is identification information for the storage node that manages the corresponding volume.

[0042] 7, the snapshot management table 131d has columns for "snapshot ID," "parent volume ID," and "storage node ID." The snapshot management table 131d is control information for managing snapshots.

[0043] "Snapshot ID" is the identification information for each snapshot. "Parent Volume ID" is the identification information for the parent volume from which the snapshot was created. "Storage Node ID" is the identification information for the storage node that owns the snapshot.

[0044] The snapshot management table 131d can be omitted if it is clear that the snapshot is managed by the same storage node 10 as the parent volume. In this case, the storage node is identified by referring to the "volume ID" corresponding to the parent volume in the volume management table 131c.

[0045] 8, the node management table 131e has columns of "storage node ID," "total capacity," "used capacity," "address," and "management port." The node management table 131e contains information for managing the storage nodes 10.

[0046] "Storage node ID" is identification information for each storage node 10. "Total capacity" is capacity information for the storage node 10 in question. "Used capacity" is used capacity information for the storage node 10 in question. "Address" is information for accessing each storage node 10 for I / O, and is shown as an IP address as an example. "Management port" is information for accessing the storage node 10 in question for management of configuration, etc., and is shown as an IP address as an example.

[0047] The volume mapping table 131f has columns for "volume ID," "start address," "status," and "reference address," as shown in Fig. 9. The volume mapping table 131f is information for managing the allocation status of pages to volumes.

[0048] "Volume ID" is identification information for each volume. "Start address" indicates the address of the relevant volume (the start position of the volume). "Status" indicates whether a page has been allocated or not. "Referenced address" is identification information for an allocated page. A page is allocated when data is first written to a volume, so an area to which data has never been written is an unallocated page.

[0049] 10, the snapshot mapping table 131g has the columns of "snapshot ID," "start address," "status," and "reference address." Similar to the volume mapping table 131f that manages the allocation status of pages to volumes, the snapshot mapping table 131g is information that manages the allocation status of pages to snapshots. Each column of the snapshot mapping table 131g is similar to the volume mapping table 131f, except that the target is changed from a volume to a snapshot.

[0050] As shown in Fig. 11, the asynchronous copy pair management table 131h has columns for "Copy Source Volume ID," "Copy Source Storage ID," and "Copy Source Journal ID." The asynchronous copy pair management table 131h also has columns for "Copy Destination Volume ID," "Copy Destination Storage ID," "Copy Destination Journal ID," "CTG ID," and "Status." The asynchronous copy pair management table 131h contains information for managing asynchronous copy pairs.

[0051] "Source volume ID" is identification information of the volume that is the source of the copy. "Source storage ID" is identification information of the storage that is the source of the copy. "Source journal ID" is identification information of the journal that is the source of the copy. "Destination volume ID," "destination storage ID," and "destination journal ID" indicate the identification information of the volume that is the destination of the copy, the identification information of the storage that is the destination of the copy, and the identification information of the journal that is the destination of the copy, respectively.

[0052] "CTGID" is identification information for a CTG (Consistency Group). A CTG is a group of multiple copy pairs configured to maintain consistency (write order) when copying multiple volumes. "Status" indicates the data copy status of the corresponding copy pair: Syncing, Synced, or Suspended.

[0053] 12, the synchronous copy pair management table 131i has the columns of "Source Volume ID," "Source Storage ID," "Destination Volume ID," "Destination Storage ID," and "Status." The synchronous copy pair management table 131i is information for managing synchronous copy pairs.

[0054] "Source volume ID" is identification information for the volume that is the source of the copy. "Source storage ID" is identification information for the storage that is the source of the copy. "Destination volume ID" and "destination storage ID" indicate identification information for the volume that is the destination of the copy and identification information for the storage that is the destination of the copy, respectively. "Status" indicates whether the data copy status of the corresponding copy pair is Syncing, Synced, or Suspended.

[0055] As shown in FIG. 13, the journal management table 131j has columns for "journal ID" and "volume ID." The journal management table 131j contains information for managing journals used for asynchronous copying. "Journal ID" is identification information for each journal. "Volume ID" is identification information for the volume used to temporarily store the data to be copied.

[0056] As shown in FIG. 14, the differential bitmap 131k has columns for "start address" and "difference bit." The differential bitmap 131k is information that manages the results of comparing a volume with its snapshot for each page of the volume and identifying differential portions where there are differences in the data. The "start address" indicates the start address of the page of the volume. The "difference bit" indicates whether or not there is a difference.

[0057] As shown in FIG. 15, the copy area management bitmap 131l has columns for "volume ID," "start address," and "copy required bit." The copy area management bitmap 131l is information indicating whether data copying is required for each page of the copy source volume. "Volume ID" is identification information for each copy source volume. "Start address" is the start address of the page of the copy source volume. "Copy required bit" indicates whether copying of the corresponding page is required. If the "copy required bit" is "1" (copy required), metadata for copying the corresponding page is saved in the journal when the copy pair is created. When the copy source storage receives a request for the copy destination storage to read data from the journal, if this metadata is stored in the journal, the copy source storage reads the corresponding page based on this metadata and returns the data to the copy destination storage. This copies the data of the corresponding page from the copy source storage to the copy destination storage.

[0058] The program section 132 stores a volume migration program 132a, a snapshot creation program 132b, an asynchronous copy program 132c, a synchronous copy program 132d, and a difference acquisition program 132e.

[0059] (Copy destination volume migration process according to the first embodiment) 16 is a sequence diagram showing a copy destination volume migration process according to Example 1. The copy destination volume migration process is realized by the storage controller 11 that executes the volume migration program 132a (FIG. 3). The process is executed by the storage controller 11 of the copy destination storage ST when rebalancing or node removal is performed due to the load on the copy destination storage node 10 exceeding a threshold.

[0060] First, in step S1601, the storage controller 11 of the destination storage ST (hereinafter referred to as the "destination storage controller") creates a volume in the destination storage node 10 to which the destination storage node 10 is to be moved. Then, a synchronous copy pair is created in which the volume to be moved is the copy source and the created volume (destination volume) is the copy destination.

[0061] Next, in step S1602, the copy destination storage controller waits until the copy between the synchronous copy pair created in step S1601 is completed (the copy state becomes Synced). When the copy is completed (step S1602 YES), the copy destination storage controller proceeds to step S1603, and repeats step S1602 while the copy is not completed (step S1602 NO).

[0062] In step S1603, the destination storage controller requests the source storage ST to create a snapshot of the source volume. The storage controller 11 of the source storage ST (hereinafter referred to as the "source storage controller") executes snapshot creation processing in accordance with the instruction of step S1603. Details of the snapshot creation processing will be described later with reference to FIG. 17.

[0063] Next, in step S1605, when the destination storage controller receives a completion notification of step S1604 from the source storage controller, it deletes the current asynchronous copy pair in its own storage.

[0064] Next, in step S1606, the destination storage controller requests the source storage controller to delete the current asynchronous copy pair. In step S1607, the source storage controller deletes the asynchronous copy pair requested to be deleted in step S1606. In the example of FIG. 1, the asynchronous copy pair deleted in steps S1605 and S1607 is the pair relationship between volume 10Va and volume 10Vb.

[0065] In step S1607, all data that has been stored in the journal but has not yet been copied is copied and reflected in the copy destination volume of the copy destination storage.

[0066] Next, in step S1608, the destination storage controller deletes the synchronous copy pair created in step S1601. In the example of Fig. 1, the source volume 10Vb may also be deleted.

[0067] Next, in step S1609, the destination storage controller requests the source storage controller to create an asynchronous copy pair by differential copying, with the destination volume created in step S1601 as the destination and the current source volume as the source. If a journal for copying data to the destination volume does not exist in either the source storage or the destination storage, a journal is created so that it exists in both the source storage and the destination storage before processing in step S1609.

[0068] In step S1610, the source storage controller executes asynchronous copy pair creation processing by differential copying in accordance with the request in step S1609. Details of asynchronous copy pair creation processing by differential copying will be described later with reference to FIG.

[0069] Next, in step S1611, upon receiving a completion notification of step S1610 from the source storage controller, the destination storage controller creates an entry for the asynchronous copy pair in the asynchronous copy pair management table 131h based on the received notification.

[0070] Next, in step S1612, the destination storage controller starts copying the asynchronous copy pair for which an entry has been created in the asynchronous copy pair management table 131h. Copying is performed by reading data from the journal of the source storage, temporarily saving it in the journal of the destination storage, and then reflecting it in the destination volume. At this time, copying based on the metadata created based on the copy area management bitmap 131l is as described in Figure 15.

[0071] Next, in step S1613, the destination storage controller determines whether the data read from the journal contains special data instructing a transition to the Synced state. If the data contains special data instructing a transition to the Synced state (step S1613 YES), the destination storage controller proceeds to step S1614, and if the data does not contain special data (step S1613 NO), the destination storage controller repeats step S1613.

[0072] Next, in step S1614, the destination storage controller changes the state of the asynchronous copy pair to Synced. Next, in step S1615, the destination storage controller notifies the source storage controller of the completion of the change to Synced.

[0073] Next, in step S1616, upon receiving a notification of completion of change to Synced from the destination storage controller, the source storage controller changes the state of the asynchronous copy pair managed by the source storage to Synced.

[0074] (Snapshot creation process according to the first embodiment) Fig. 17 is a flowchart showing a snapshot creation process according to Example 1. The snapshot creation process is realized by the storage controller 11 that executes the snapshot creation program 132b (Fig. 3).

[0075] First, in step S1701, the copy source storage controller determines whether or not dirty data that has been written to a specified volume but not yet reflected in the storage drive 18 exists in the cache memory (cache unit 133). If dirty data exists in the cache memory (step S1701 YES), the copy source storage controller proceeds to step S1702, and if dirty data does not exist (step S1701 NO), the copy source storage controller proceeds to step S1704.

[0076] In step S1702, the source storage controller updates the pool management table 131b to allocate a pool page to the dirty data. Next, in step S1703, the source storage controller destages the dirty data from the cache memory and writes it to the storage drive 18. Next, in step S1704, the source storage controller maps the page allocated in step S1702 to a volume and updates the volume mapping table 131f.

[0077] Next, in step S1705, the source storage controller copies the entry of the volume for which a snapshot is to be created in the volume mapping table 131f to the snapshot mapping table 131g, and creates an entry for the snapshot. Next, in step S1706, information about the snapshot created in step S1705 is added to the snapshot management table 131d.

[0078] (Pair creation process by differential copy according to the first embodiment) FIG. 18 is a flowchart illustrating a pair creation process based on differential copying according to the first embodiment.

[0079] First, in step S1801, the source storage controller creates information for managing the target asynchronous copy pair in the asynchronous copy pair management table 131h. In step S1801, "Status=Syncing" is registered in the asynchronous copy pair management table 131h.

[0080] Next, in step S1900, the copy source storage controller executes a copy area management bitmap creation process. Details of the copy area management bitmap creation process will be described later with reference to Fig. 19. Next, in step S1803, the copy source storage controller executes a copy metadata creation process. Details of the copy metadata creation process will be described later with reference to Fig. 21.

[0081] (Copy Area Management Bitmap Creation Process According to First Embodiment) 19 is a flowchart showing copy area management bitmap creation processing according to Example 1. The copy area management bitmap creation processing is processing called from the pair creation processing by differential copy (step S1802 in FIG. 18), and creates a bitmap for managing the copy area.

[0082] First, in step S1901, the source storage controller executes a difference acquisition process. Details of the difference acquisition process will be described later with reference to Fig. 20. Next, in step S1902, the source storage controller selects one unselected entry from the difference bitmap 131k created in step S1901.

[0083] Next, in step S1903, the copy source storage controller determines whether the differential bit in the differential bitmap 131k is 1. The copy source storage controller checks the entry in the differential bitmap 131k, and if the differential bit in the differential bitmap 131k is 1, proceeds to step S1904, and if the differential bit is 0, proceeds to step S1905.

[0084] In step S1904, the copy source storage controller updates the copy required bit in the copy area management bitmap 131l to 1 for the same address as the address indicated by the entry whose difference bit was 1 in step S1903, ie, which indicates that there is a difference.

[0085] Next, in step S1905, if there is an entry in the differential bitmap 131k that has not been selected in step S1902, the copy source storage controller returns the process to step S1902 and selects an unselected entry. If all entries in the differential bitmap 131k have been selected in step S1902, the copy source storage controller ends the copy area management bitmap creation process.

[0086] (Difference Acquisition Process According to the First Embodiment) 20 is a flowchart showing the difference acquisition process according to Example 1. The difference acquisition process is realized by the storage controller 11 that executes the difference acquisition program 132e (FIG. 3).

[0087] First, in step S2001, the copy source storage controller compares the reference addresses of the same logical address of the target volume and the target snapshot in the volume mapping table 131f and snapshot mapping table 131g.

[0088] Next, in step S2002, the copy source storage controller determines whether the reference addresses compared in step S2001 are different. If the reference addresses are different (step S2002 YES), the copy source storage controller proceeds to step S2003, and if they match (step S2002 NO), the copy source storage controller proceeds to step S2004.

[0089] In step S2003, the source storage controller updates the differential bit in the differential bitmap 131k corresponding to the reference address identified as different in step S2002 to 1. Next, in step S2004, the source storage controller determines whether comparison of all logical addresses has been completed in step S2001. If comparison of all logical addresses has been completed in step S2001 (YES in step S2004), the source storage controller ends the difference acquisition process, and if comparison has not been completed (NO in step S2004), the source storage controller returns the process to step S2001.

[0090] (Copy metadata creation process according to the first embodiment) Fig. 21 is a flowchart showing copy metadata creation processing related to Example 1. The copy metadata creation processing is processing in which metadata for data copying is created in a journal by the source storage when a synchronous copy pair is created (step S1601 of the destination volume migration processing (Fig. 16)).

[0091] The journal normally stores the data written to the source volume and its metadata (the volume and its write location (logical address)). However, during the initial copy when creating a pair, storing all of the volume's data in the journal would be a heavy load, so only the metadata is stored. When the destination storage calls for the metadata from the journal, the data is read from the source volume and sent to the destination storage.

[0092] First, in step S2101, the copy source storage controller selects one unselected entry in the copy area management bitmap 131l. Next, in step S2102, the copy source storage controller determines whether the copy required bit of the entry selected in step S2101 is 1. If the copy required bit is 1 (step S2102 YES), the copy source storage controller proceeds to step S2103, and if the copy required bit is 0 (step S2102 NO), the copy source storage controller proceeds to step S2104.

[0093] In step S2103, the copy source storage controller creates in the journal metadata for copying the area (page) for which the copy required bit was determined to be 1 in step S2102. Next, in step S2104, if there is an entry in the copy area management bitmap 131l that was not selected in step S2101, the copy source storage controller returns the process to step S2102 and selects an unselected entry. If all entries in the copy area management bitmap 131l have been selected in step S2101, the copy source storage controller moves the process to step S2105.

[0094] In step S2105, when the source storage controller has finished creating metadata for all areas (pages) whose copy-needed bits are 1, it creates metadata in the journal instructing to change the copy state to Synced. When the destination storage controller reads this metadata, it changes the state of the asynchronous copy pair to Synced. [Example]

[0095] Second Embodiment A second embodiment of the present invention will be described below with reference to Figures 22 and 23. In the description of the second embodiment, differences from the first embodiment will be mainly described, and overlapping descriptions will be omitted.

[0096] (Processing Overview of Storage System According to Second Embodiment) Fig. 22 is a diagram illustrating an overview of processing in a storage system according to Example 2. Fig. 22 illustrates control in a storage system 2S when a source volume of a source site that forms an asynchronous copy pair with a destination volume of a destination site is moved between storage nodes of the source site.

[0097] The storage of the copy source site 1a is configured to include a plurality of storage nodes 10a, 10a2. In the copy source site 1a, in order to equalize the load between the storage nodes 10a, 10a2, the volume 10Va is rebalanced between the storage nodes 10a, 10a2.

[0098] In this embodiment, volume 10Va2 is created at the source site 1a as a synchronous copy volume (first copy volume) of volume 10Va. Then, while maintaining the asynchronous copy pair relationship, the volume is migrated from volume 10Va of storage node 10a to volume 10Va2 of storage node 10a2, thereby avoiding recopying of data.

[0099] The method for moving the copy source volume from volume 10Va to volume 10Va2 is as follows: First, in step S11, the copy source volume 10Va is synchronously copied between the storage nodes 10a and 10a2 (data D0 and data D1 are copied to volume Va2).

[0100] Next, in step S12, a snapshot 10Sa2 of the source volume 10Va2 is created on the source storage node 10a2. When the snapshot 10Sa2 is created, the asynchronous copy pair relationship between volumes 10Va and 10Vb is deleted. The snapshot 10Sa2 is used to identify differences in the subsequent stages. Note that the copy of the source volume is not limited to a snapshot, and other types of copies can be used as long as they allow for subsequent differences to be identified.

[0101] Next, in step S13, the access destination for I / O requests from host 20a is switched to synchronous copy destination volume 10Va2. Because host 20a continues updating data, subsequent data updates are added to synchronous copy destination volume 10Va2 (differential data D2 has been added in Figure 22).

[0102] Next, in step S14, snapshot 10Sa2 is compared with volume 10Va2, and it is determined that differential data D2 is the difference. Next, in step S15, the determined differential data D2 is copied to volume 10Vb, and an asynchronous copy pair relationship is established between the new copy source volume 10Va2 and the copy destination volume 10Vb.

[0103] At this time, a journal 10Ja2 (journal #2) is created in the storage node 10a2 to which the volume is to be moved, and a journal 10Jb2 (journal #4) that is a pair with the journal 10Ja2 is created in the storage node 10b to which the volume is to be copied. Differential data D2 is copied to volume 10Vb2 via journals 10Ja2 and 10Jb2. The data in volume 10Va2 is written to a storage drive (not shown) via pool 10Pa2.

[0104] (Copy source volume movement processing according to the second embodiment) Fig. 23 is a sequence diagram showing a copy source volume migration process according to Example 2. The copy source volume migration process is realized by the storage controller 11 that executes the volume migration program 132a (Fig. 3). The process is executed by the storage controller 11 of the copy source storage ST when rebalancing or node removal is performed due to the load on the copy source storage node 10 exceeding a threshold.

[0105] First, in step S2301, the source storage controller creates a volume on the destination node and creates a synchronous copy pair with the volume to be migrated as the source and the created volume as the destination. This destination volume is the destination volume.

[0106] Next, in step S2302, the copy source storage controller waits until the copy between the synchronous copy pair created in step S2301 is completed (the copy state becomes Synced). When the copy is completed (step S2302 YES), the copy source storage controller proceeds to step S2303, and repeats step S2302 while the copy is not completed (step S2302 NO).

[0107] In step S2303, the copy source storage controller executes a snapshot creation process. Details of the snapshot creation process are as shown in Fig. 17. The execution entity of the snapshot creation process called in step S2303 is the copy source storage controller.

[0108] Next, in step S2304, the source storage controller switches the access destination of the host 20a from the migration target volume to the migration destination volume. Next, in step S2305, the source storage controller deletes the current asynchronous copy pair in its own storage.

[0109] Next, in step S2306, the source storage controller requests the destination storage to delete the current asynchronous copy pair. In step S2307, the destination storage controller deletes the asynchronous copy pair requested to be deleted in step S2306. In the example of Figure 22, the asynchronous copy pair deleted in steps S2305 and S2307 is the pair relationship between volume 10Va and volume 10Vb.

[0110] In step S2307, the destination storage controller copies all data that has been stored in the journal but has not yet been copied to the destination volume of the destination storage, thereby reflecting the data.

[0111] Next, in step S2308, the source storage controller deletes the synchronous copy pair created in step S2301. In the example of Fig. 22, source volume 10Va may also be deleted.

[0112] Next, in step S2309, the source storage controller executes a process for creating a copy pair by differential copying. The details of the process for creating a copy pair by differential copying are as shown in Fig. 18. In step S2309, the source storage controller executes the creation of an asynchronous copy pair by differential copying, with the destination volume created in step S2301 as the source and the current destination volume as the destination.

[0113] If a journal for copying data to the destination volume does not exist in either the source storage or the destination storage, a journal is created in both the source storage and the destination storage before the processing of step S2309 so that the journal exists.

[0114] Next, in step S2310, the source storage controller requests the destination storage controller to create an asynchronous copy pair. In step S2311, upon receiving the asynchronous copy pair creation request from the source storage controller, the destination storage controller creates an entry for the asynchronous copy pair in the asynchronous copy pair management table 131h.

[0115] Next, in step S2312, the destination storage controller starts copying for the asynchronous copy pair for which an entry has been created in the asynchronous copy pair management table 131h. Copying is performed by reading data from the journal of the source storage, temporarily storing it in the journal of the destination storage, and then reflecting it in the destination volume.

[0116] Next, in step S2313, the destination storage controller determines whether the data read from the journal contains special data instructing a transition to the Synced state. If the data contains special data instructing a transition to the Synced state (step S2313 YES), the destination storage controller proceeds to step S2314, and if not (step S2313 NO), the destination storage controller repeats step S2313.

[0117] Next, in step S2314, the destination storage controller changes the state of the asynchronous copy pair to Synced. Next, in step S2315, the destination storage controller notifies the source storage controller of the completion of the change to Synced.

[0118] Next, in step S2316, when the source storage controller receives a notification of completion of change to Synced from the destination storage controller, it changes the state of the asynchronous copy pair managed by the source storage to Synced.

[0119] (Effects of Examples 1 and 2) In the above-described first and second embodiments, a snapshot is used to identify the difference between the source volume and the destination volume, and an asynchronous copy pair is created by copying only the difference, thereby migrating volumes between storage nodes. By moving volumes with this asynchronous copy pair between storage nodes, it is possible to level the load between storage nodes without temporarily stopping I / O requests from hosts and avoiding performance impacts on applications. In addition, a storage node with a volume with an asynchronous copy pair can be moved to another storage node without temporarily stopping I / O requests from hosts and avoiding performance impacts on applications, and the corresponding storage node can be removed. This reduces excess storage resources and enables cost reduction. [Example]

[0120] Hereinafter, a third embodiment of the present invention will be described with reference to Figures 24A to 27. In the description of the third embodiment, differences from the first and second embodiments will be mainly described, and overlapping descriptions will be omitted.

[0121] (Processing Overview of Storage System According to Third Embodiment) 24A and 24B are diagrams showing an overview of the processing of a storage system according to a third embodiment. When a storage system employs a CTG, copying must be performed consistently within the same CTG. In the present invention, copying is temporarily suspended during volume migration, and therefore, in consideration of consistency with other asynchronous copy pairs in the same CTG, copying for the other asynchronous copy pairs must also be suspended. In this embodiment, taking this into consideration, copying for all asynchronous copy pairs belonging to the same CTG is suspended during volume migration.

[0122] Fig. 24A and Fig. 24B show, as a premise of this embodiment, how the CTG configuration changes due to volume migration.

[0123] When multiple asynchronous copy pairs belonging to the same CTG are copies within the same storage node, volume migration changes the configuration so that the CTG spans storage nodes. As shown in the "(A) Node Distribution" diagram in the upper part of Figure 24A, two asynchronous copy pairs P1 and P2 are copies within the same storage nodes N11 and N21. From this state, the copy destination volume of asynchronous copy pair P2 is migrated from storage node N21 to storage node N22. At this time, the CTG, which was completed within storage nodes N11 and N21, is changed to node distribution so that it spans storage nodes N11, N21, and N12 and N22. A CTG configured across multiple storage nodes is sometimes called an ExCTG.

[0124] Furthermore, when multiple asynchronous copy pairs belonging to the same CTG are copies that span multiple storage nodes, volume migration changes the settings so that the CTG is completed within the same storage node. As shown in the "(B) Node Consolidation" diagram in the lower part of Figure 24A, at the copy source site, asynchronous copy pair P1 is a copy within the same storage nodes N11 and N21. Furthermore, asynchronous copy pair P2 is a copy that spans storage nodes N11, N21 and N12, N22. From this state, the copy destination volume of asynchronous copy pair P2 is moved from storage node N22 to storage node N21. At this time, the CTG, which previously spanned storage nodes N11, N21 and N12, N22, is node consolidated so that it is completed within storage nodes N11 and N21.

[0125] Furthermore, as shown in the "(C) Maintaining Consolidation" diagram in the upper part of Figure 24B, the two asynchronous copy pairs P1 and P2 are copied within the same storage nodes N11 and N21. From this state, the copy destination volumes of the asynchronous copy pair P1 and P2 are both moved from storage node N21 to storage node N22 while maintaining consolidation. At this time, the CTG, which was configured from storage nodes N11 and N21, is changed to be configured from storage nodes N11 and N22.

[0126] As shown in the "(D) Maintaining Distribution" diagram at the bottom of Figure 24B, asynchronous copy pair P1 is a copy within the same storage nodes N11 and N21, and asynchronous copy pair P2 is a copy spanning storage nodes N11, N21 and N12, N22. From this state, the copy destination volume of asynchronous copy pair P2 is moved from storage node N22 to storage node N23 while maintaining distribution. At this time, the CTG, which previously spanned storage nodes N11, N21 and N12, N22, is changed to span storage nodes N11, N21, N12, N22 and N13, N23 (or storage nodes N11, N21 and N13, N23).

[0127] (Pair deletion process taking CTG into consideration according to the third embodiment) Fig. 25 is a sequence diagram showing the pair deletion processing that takes CTG into consideration according to Example 3. The pair deletion processing that takes CTG into consideration shown in Fig. 25 is processing that replaces steps S1605 to S1607 of the copy destination volume movement processing of Fig. 16. The pair deletion processing that takes CTG into consideration is processing that suspends pairs other than the target pair (temporarily suspends copying) before deleting the asynchronous copy pair.

[0128] First, in step S2501 following step S1604, the source storage controller and the destination storage controller select an unselected asynchronous copy pair from among the asynchronous copy pairs in the same CTG.

[0129] Next, in step S2502, the destination storage controller requests the source storage controller to suspend the asynchronous copy pair selected in step S2501. Next, in step S2503, the source storage controller adds metadata to the journal instructing the suspension of the target asynchronous copy pair.

[0130] Next, in step S2504, upon receiving the metadata added to the journal in step S2503, the destination storage controller changes the pair state of the asynchronous copy pair in its own storage selected in step S2501 to "Suspended." Next, in step S2505, the destination storage controller changes the pair state of the asynchronous copy pair in its own storage selected in step S2501 to "Suspended."

[0131] Next, in step S2506, if there are any asynchronous copy pairs that have not been selected in step S2501, the source storage controller and the destination storage controller return the process to step S2501 and select an unselected entry.If all asynchronous copy pairs of the CTG that are the selection target have been selected in step S2501, the source storage controller and the destination storage controller return the process to step S2501.

[0132] In step S2507, the destination storage controller deletes the asynchronous copy pair to which the migration target volume belongs within its own storage. Then, it requests the source storage to delete the asynchronous copy pair to which the migration target volume belongs within its own storage. In step S2508, in response to the deletion request from the destination storage controller in step S2507, the source storage deletes the asynchronous copy pair to which the migration target volume belongs within its own storage.

[0133] When writing to the copy source volume occurs during suspension, the updated position is recorded in the copy area management bitmap 1311. That is, 1 is set in the copy required bit for the write destination logical address in the copy area management bitmap creation process.

[0134] (Resync process for pair creation taking into consideration CTG according to the third embodiment) Fig. 26 is a sequence diagram showing the resync process related to pair creation that takes into account the CTG according to Example 3. The resync process related to pair creation that takes into account the CTG shown in Fig. 26 is a process that is executed between steps S1610 and S1611 of the copy destination volume migration process in Fig. 16.

[0135] First, in step S2601 following step S1610, the source storage controller and the destination storage controller select an unselected asynchronous copy pair from among the asynchronous copy pairs in the same CTG. Next, in step S2602, the destination storage controller requests a resync from the source storage. Next, in step S2603, the source storage controller and the destination storage controller execute a resync process in response to the resync request. Details of the resync process will be described later with reference to FIG. 27.

[0136] (Resync process according to the third embodiment) 27 is a sequence diagram showing the resync process according to Example 3. The resync process according to this example resynchronizes data between two volumes related to all volume pairs belonging to a consistency group.

[0137] First, in step S2701, the copy source storage controller executes copy metadata creation processing. Details of the copy metadata creation processing are as described in detail with reference to Fig. 21. Next, in step S2702, the copy source storage controller changes the state of the asynchronous copy pair to Syncing.

[0138] Next, in step S2703, the destination storage controller changes the state of the asynchronous copy pair in its own storage, which is the same asynchronous copy pair as in step S2702, to Syncing. Next, in step S2704, the destination storage controller starts asynchronous copying. Next, in step S2705, the destination storage controller determines whether the data read from the journal created in step S2701 contains special data that instructs a transition to the Synced state. If the special data instructs a transition to the Synced state is contained (step S2705 YES), the destination storage controller proceeds to step S2708, and if it is not contained (step S2705 NO), it repeats step S2705.

[0139] Next, in step S2706, the destination storage controller changes the state of the asynchronous copy pair to Synced, and then in step S2706, the destination storage controller notifies the source storage controller of the completion of the change to Synced.

[0140] Next, in step S2708, when the source storage controller receives a notification of completion of change to Synced from the destination storage controller, it changes the state of the asynchronous copy pair managed by the source storage to Synced.

[0141] (Effects of Example 3) According to the above-described third embodiment, it is possible to move the copy destination volume of an asynchronous copy pair between storage nodes of the copy destination storage while maintaining the CTG. [Example]

[0142] Hereinafter, a fourth embodiment of the present invention will be described with reference to Figures 28A to 30. In the description of the fourth embodiment, differences from the first to third embodiments will be mainly described, and overlapping descriptions will be omitted.

[0143] (Processing Overview of Storage System According to Fourth Embodiment) 28A and 28B are diagrams showing an overview of processing in a storage system according to Example 4. As in Example 3 which aims to distribute the load on copy-destination storage nodes, when distributing the load on copy-source storage nodes, if the storage employs a CTG, copying must be performed with consistency within the same CTG.

[0144] Fig. 28A and Fig. 28B show, as a premise of this embodiment, how the CTG configuration changes due to volume migration.

[0145] As shown in the "(A) Node Distribution" diagram in the upper part of Figure 28A, two asynchronous copy pairs P1 and P2 are copied within the same storage nodes N11 and N21. From this state, the copy source volume of one of the asynchronous copy pairs, P2, is moved from storage node N11 to storage node N12. At this time, the CTG, which was completed within storage nodes N11 and N21, is changed to node distribution so that it spans storage nodes N11, N21, and N12 and N22.

[0146] As shown in the "(B) Node Consolidation" diagram at the bottom of Figure 28A, asynchronous copy pair P1 is a copy within the same storage nodes N11 and N21, and asynchronous copy pair P2 is a copy across storage nodes N11, N21 and N12, N22. From this state, the copy source volume of asynchronous copy pair P2 is moved from storage node N12 to storage node N11. At this time, the CTG, which spanned storage nodes N11, N21 and N12, N22, is node consolidated so that it is completed within storage nodes N11 and N21.

[0147] Furthermore, as shown in the "(C) Maintaining Aggregation" diagram in the upper part of Figure 28B, the two asynchronous copy pairs P1 and P2 are copied within the same storage nodes N11 and N21. From this state, the copy source volumes of the asynchronous copy pair P1 and P2 are both moved from storage node N11 to storage node N12 while maintaining aggregation. At this time, the CTG, which was completed within storage nodes N11 and N21, is changed to a configuration in which it is completed within storage nodes N12 and N21.

[0148] As shown in the "(D) Maintaining Distribution" diagram at the bottom of Figure 24B, asynchronous copy pair P1 is a copy within the same storage nodes N11 and N21, and asynchronous copy pair P2 is a copy spanning storage nodes N11, N21 and N12, N22. From this state, the copy source volume of asynchronous copy pair P2 is moved from storage node N11 to storage node N12 while maintaining distribution. At this time, the CTG is changed to a setting spanning storage nodes N11, N21 and N12, N22 before and after the volume movement.

[0149] (Pair deletion process taking into consideration CTG according to the fourth embodiment) Fig. 29 is a sequence diagram showing the pair deletion processing that takes CTG into consideration related to Example 4. The pair deletion processing that takes CTG into consideration shown in Fig. 29 is processing that replaces steps S2305 to S1607 of the source volume movement processing of Fig. 23. The pair deletion processing that takes CTG into consideration is processing that suspends pairs other than the target pair (temporarily suspends copying) before deleting the asynchronous copy pair.

[0150] First, in step S2305a following step S2304, the source storage controller and the destination storage controller select an unselected asynchronous copy pair from among the asynchronous copy pairs in the same CTG.

[0151] Next, in step S12305b, the source storage controller adds metadata to the journal instructing the suspension of the asynchronous copy pair selected in step S2305a.

[0152] Next, in step S2305c, upon receiving the metadata added to the journal in step S2305b, the destination storage controller changes the pair state of the asynchronous copy pair in its own storage selected in step S1605a to "Suspended." Next, in step S2305d, the source storage controller changes the pair state of the asynchronous copy pair in its own storage selected in step S2305a to "Suspended."

[0153] Next, in step S2305e, if there are any asynchronous copy pairs that have not been selected in step S2305a, the source storage controller and the destination storage controller return to step S2305a to select an unselected asynchronous copy pair.If all asynchronous copy pairs of the CTG that are the selection target have been selected in step S2305a, the source storage controller and the destination storage controller proceed to step S2306a.

[0154] In step S2306a, the source storage controller deletes the asynchronous copy pair to which the migration target volume belongs. Next, in step S2307a, the destination storage controller deletes the asynchronous copy pair to which the migration target volume belongs.

[0155] (Resync process for pair creation taking into account CTG according to the fourth embodiment) Fig. 30 is a sequence diagram showing the resync process related to pair creation taking into consideration the CTG according to Example 4. The resync process related to pair creation taking into consideration the CTG shown in Fig. 30 is a process executed between steps S2310 and S2311 of the source volume migration process in Fig. 23.

[0156] First, in step S3001 following step S2310, the source storage controller and the destination storage controller select an unselected asynchronous copy pair from among the asynchronous copy pairs in the same CTG. Next, in step S3002, the source storage controller and the destination storage controller execute a resync process. Details of the resync process are as described in detail with reference to FIG. 27.

[0157] (Effects of Example 4) According to the above-described fourth embodiment, the source volume of a synchronous copy pair can be moved between storage nodes of the source storage while maintaining the CTG.

[0158] The present invention is not limited to the above-described examples, and includes various modifications. Furthermore, the above-described examples have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the described configurations. Furthermore, not only are such configurations removed, but replacement or addition of configurations is also possible. Furthermore, embodiments of the present invention also include combinations of some or all of the above-described examples in a consistent manner. Furthermore, the execution order of each processing step shown in the flowcharts of the above-described examples can be changed as appropriate as long as the processing results remain unchanged.

[0159] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a recording medium on which the program code is recorded is provided to a computer, and the processor of the computer reads the program code stored in the recording medium.

[0160] In this case, the program code itself read from the recording medium will realize the functions of the above-described embodiment, and the program code itself and the recording medium on which it is stored constitute the present invention. Examples of recording media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0161] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, and Java (registered trademark).

[0162] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected. [Explanation of symbols]

[0163] S,2S: storage system, ST: storage, 10: storage node, 10Sa, 10Sa2: snapshot, 11: storage controller, 20, 20a: host.

Claims

1. A storage system including storage nodes each including a storage controller that processes I / O requests from a host and a storage drive, the first storage has a first volume in a first storage node; the second storage has a second volume on a second storage node; a first pair relationship of asynchronous copy in which data related to the I / O request from the host to the first volume is asynchronously reflected in the second volume is created between the first volume and the second volume; The storage controller creating a snapshot of the first volume in the first storage; After creating the snapshot, deleting the first pair relationship and stopping the asynchronous copy; While continuing the processing of the I / O request from the host to the first volume, the first volume or the second volume is moved as a volume to be moved to another storage node of the storage having the volume to be moved; after the migration of the first volume or the second volume to the other storage node is completed, creating a second pair relationship of the asynchronous copy between the first volume and the second volume; After creating the second pair relationship, comparing the first volume with the snapshot to identify differential data between the first volume and the snapshot; The identified differential data is reflected in the second volume. A storage system comprising:

2. 2. The storage system according to claim 1, The storage controller After the asynchronous copy is stopped, while continuing to process the I / O request from the host to the first volume, create a copy volume by synchronous copying of the second volume in another storage node of the second storage that is different from the second storage node, with the second volume as the volume to be moved; The copy volume is set as the new second volume, and the second pair relationship is created between the first volume and the second volume. A storage system comprising:

3. 2. The storage system according to claim 1, The storage controller creating a copy volume of the first volume by synchronous copying in another storage node of the first storage that is different from the first storage node while continuing to process the I / O request from the host to the first volume; creating the snapshot of the copy volume in the other storage, deleting the first pair relationship and stopping the asynchronous copy; after stopping the asynchronous copy, switching the access destination of the I / O request from the host from the first volume to the copy volume; After switching the access destination of the I / O request to the copy volume, the copy volume is set as the new first volume, and the second pair relationship is created between the first volume and the second volume. A storage system comprising:

4. 2. The storage system according to claim 1, the first storage has a plurality of the first volumes; the second storage has a plurality of the second volumes; The storage controller When a consistency group for maintaining consistency of write order is set for a plurality of volume pairs of the first volume and the second volume for which the first pair relationship is created, the copy related to the I / O request to the first volume and the second volume for all volume pairs belonging to the consistency group is temporarily stopped, and then the first pair relationship is deleted. A storage system comprising:

5. 5. The storage system according to claim 4, The storage controller After creating the second pair relationship, a resync is performed to resynchronize data between the first volume and the second volume for all volume pairs belonging to the consistency group. A storage system comprising:

6. A volume migration method in a storage system including storages each having a storage node including a storage controller that processes I / O requests from a host and a storage drive, comprising: the first storage has a first volume in a first storage node; the second storage has a second volume on a second storage node; a first pair relationship of asynchronous copy in which data related to the I / O request from the host to the first volume is asynchronously reflected in the second volume is created between the first volume and the second volume; the storage controller: creating a snapshot of the first volume in the first storage; After creating the snapshot, deleting the first pair relationship and stopping the asynchronous copy; While continuing the processing of the I / O request from the host to the first volume, the first volume or the second volume is moved as a volume to be moved to another storage node of the storage having the volume to be moved; after the migration of the first volume or the second volume to the other storage node is completed, creating a second pair relationship of the asynchronous copy between the first volume and the second volume; After creating the second pair relationship, comparing the first volume with the snapshot to identify differential data between the first volume and the snapshot; The identified differential data is reflected in the second volume. A volume migration method in a storage system, comprising the steps of:

7. 7. A volume migration method in a storage system according to claim 6, the storage controller: After the asynchronous copy is stopped, while continuing to process the I / O request from the host to the first volume, create a copy volume by synchronous copying of the second volume in another storage node of the second storage that is different from the second storage node, with the second volume as the volume to be moved; The copy volume is set as the new second volume, and the second pair relationship is created between the first volume and the second volume. A volume migration method in a storage system, comprising the steps of:

8. 7. A volume migration method in a storage system according to claim 6, the storage controller: creating a copy volume of the first volume by synchronous copying in another storage node of the first storage that is different from the first storage node while continuing to process the I / O request from the host to the first volume; creating the snapshot of the copy volume in the other storage, deleting the first pair relationship and stopping the asynchronous copy; after stopping the asynchronous copy, switching the access destination of the I / O request from the host from the first volume to the copy volume; After switching the access destination of the I / O request to the copy volume, the copy volume is set as the new first volume, and the second pair relationship is created between the first volume and the second volume. A volume migration method in a storage system, comprising the steps of:

9. 7. A volume migration method in a storage system according to claim 6, the first storage has a plurality of the first volumes; the second storage has a plurality of the second volumes; the storage controller: When a consistency group for maintaining consistency of write order is set for a plurality of volume pairs of the first volume and the second volume for which the first pair relationship is created, the copy related to the I / O request to the first volume and the second volume for all volume pairs belonging to the consistency group is temporarily stopped, and then the first pair relationship is deleted.

1. A volume migration method in a storage system, comprising the steps of:

10. 10. A volume migration method in a storage system according to claim 9, the storage controller: After creating the second pair relationship, a resync is performed to resynchronize data between the first volume and the second volume for all volume pairs belonging to the consistency group.

1. A volume migration method in a storage system, comprising the steps of:

Images