Resynchronization to the endpoint of a file system synchronous replication relationship

Abstract

A technique for resynchronizing a synchronous replication relationship is provided. An asynchronous incremental transfer is performed to copy data of a storage object to a replicated storage object. During a last asynchronous incremental transfer, incoming write requests targeting the storage object are logged in a dirty region log. During the last asynchronous incremental transfer, metadata operations performed on the storage object are logged in the metadata log. Sequence numbers are assigned to the metadata operations based on an execution order. The metadata operations are copied to the replicated storage object for execution according to the sequence numbers, and dirty regions are copied to the replicated storage object in response to the metadata operations having been copied to the replicated storage object. The storage object and the replicated storage object are transitioned to a synchronous replication state in which incoming operations are synchronously replicated to the replicated storage object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 191,700, filed on November 15, 2018, entitled “RESYNCHRONIZATION TO ASYNCHRONOUS REPLICATION RELATIONSHIP,” which claims priority to Indian Patent Application No. 201841029907, filed on August 9, 2018, entitled “RESYNCHRONIZATION TO A SYNCHRONOUS REPLICATION RELATIONSHIP,” which applications are incorporated herein by reference. Background Art

[0003] Many storage systems can implement data replication and / or other redundant data access techniques for data loss protection and uninterrupted client access. For example, a first computing device can be configured to provide clients with primary access to data stored on the first storage device and / or other storage devices. In the event of a failure of the first computing device, a second computing device can be configured as a backup for the first computing device. Data can be replicated from the first computing device to the second computing device. In this way, in the event of a failure of the first computing device, the second computing device can provide clients with access to the replicated data.

[0004] One type of replication is asynchronous replication. When a first computing device receives an operation from a client device, it transmits a copy of the operation to a second computing device for execution. Regardless of whether the second computing device has already performed the copy, once the first computing device has executed the operation, it transmits confirmation of the successful execution of the operation to the client device.

[0005] Another type of replication, synchronous replication, provides a higher level of data protection guarantees. This is because the first computing device does not transmit an acknowledgment until the first computing device has performed the operation and the second computing device has performed or confirmed the replication operation. In this way, two copies of the data and / or metadata generated by the operation are maintained until the client receives confirmation that the operation was successful.

[0006] Unfortunately, the first computing device and the second computing device may become out of sync due to network transmission errors, computer failures, and / or other issues that cause the data maintained by the first computing device to differ from the replicated data maintained by the second computing device. Therefore, the data protection guarantees provided by synchronous replication cannot be provided until the storage of the first computing device and the second computing device are returned to a synchronous replication state. The current resynchronization process can be extremely disruptive to clients because client operations will be stalled (e.g., client I / O operations will be blocked, fail, aborted, or queued for later execution) during various stages of resynchronization. Blocking client I / O can cause applications to timeout and encounter errors, increase the latency experienced by clients, and disrupt access to data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a component block diagram illustrating an example cluster network in which embodiments of the present invention may be implemented.

[0008] Figure 2 is a component block diagram illustrating an example data storage system in which embodiments of the present invention may be implemented.

[0009] Figure 3 is a flow chart illustrating an example method for resynchronization of a synchronous replication relationship.

[0010] Figure 4 is a component block diagram illustrating an example system for resynchronization of synchronous replication relationships.

[0011] Figure 5 is an example of a computer-readable medium in which embodiments of the present invention may be implemented.

[0012] Figure 6 is a component block diagram illustrating an example computing environment in which embodiments of the invention may be implemented. DETAILED DESCRIPTION

[0013] Some examples of the claimed subject matter will now be described with reference to the accompanying drawings, wherein similar reference numerals are generally used throughout to refer to similar elements. In the following description, for purposes of explanation, numerous specific details are set forth to provide an understanding of the claimed subject matter. However, it is clear that the claimed subject matter can be practiced without these specific details. Nothing in this detailed description should be considered to be prior art.

[0014] In asynchronous replication, incremental changes to a storage object (such as a volume, file, directory, a defined set of files or directories, a file system, or a storage virtual machine comprising multiple volumes stored across one or more nodes of a cluster) are replicated from the storage object to a replicated storage object. In synchronous replication, when an operation (e.g., a write operation targeting a storage object) is received from a client device, the operation is split to create a replicated operation that is a copy of the operation. The operation is performed on the storage object, such as by a first computing device that manages the storage object. The replicated operation is performed on the replicated storage object, such as by a second computing device that manages the replicated storage object. The completion of the operation is not confirmed to the client device until both the operation and the replicated operation have been successfully performed on the storage object and the replicated storage object.

[0015] Synchronous replication can be implemented relatively easily for new volumes. This is because pending client I / Os that are making real-time changes and modifications to the new volume do not need to be processed to keep the replica of the new volume consistent with the new volume. Otherwise, these pending I / Os would need to be processed so that the new volume and the replica have the same data as the baseline and can begin synchronously replicating new incoming client I / Os.

[0016] However, for an existing volume that already includes data that is being modified in real time by incoming client I / O, the replicated volume must be synchronized with respect to the existing volume so that the replicated volume has the same data as the existing volume. Since the existing volume is actively processing client I / O, the replicated volume will lag behind the existing volume as the client I / O modifies the existing volume. Therefore, conventional techniques for transitioning from asynchronous replication to synchronous replication must pause client I / O (e.g., suspend, block, fail, or queue the client I / O for later execution), which increases latency (e.g., increases latency while client I / O is queued). This can also affect operations of client devices accessing data in the existing volume (e.g., applications may time out or encounter errors when data access operations attempting to access the data are blocked or fail).

[0017] Thus, provided herein are methods and / or systems that can transition a storage object from an asynchronously replicated state or other non-synchronized state (e.g., an out-of-sync state) to a synchronously replicated state in a manner that can mitigate client interruption and latency. That is, the transition (resynchronization) can be performed without holding client I / O (e.g., without pausing, blocking, failing, or queuing for later execution), which reduces client latency that might otherwise be experienced if client I / O were held during the transition.

[0018] The dirty region log is used to track regions within a storage object that have been modified by data operations, such as write operations performed during the last asynchronous incremental transfer (e.g., an asynchronous incremental transfer may be performed first to incrementally transfer data from a storage object to a replicated storage object to help make the replicated storage object include more of the same data as the storage object). The dirty region log may include a bit that can be set to a dirty indicator or a clean indicator. The bit may be mapped to a region within a storage object. Thus, the bit may be set to a dirty indicator to indicate that a data operation has modified the region (e.g., the region now includes data that has not yet been copied to the replicated storage object). The bit may be set to a clean indicator to indicate that the region is now clean (e.g., the region has not been modified with data that has not yet been copied to the replicated storage object, and therefore the region includes the same data as the corresponding region within the replicated storage object).

[0019] The metadata log is used to track metadata operations that modify storage objects, such as create operations (e.g., create file operations, create LUN operations, etc.), link operations, unlink operations, rename operations (e.g., file rename operations, etc.), set attribute operations (e.g., set volume size operations, assign permissions operations, etc.), etc. In particular, a copy of the metadata operation performed on the storage object during the last asynchronous transfer is inserted into the metadata log.

[0020] The metadata operations within the metadata log are copied to the replicated storage object based on the order in which the metadata operations are performed on the storage object to maintain consistency. In particular, sequence numbers are assigned to the metadata operations based on the order in which the metadata operations are performed on the storage object by the first computing system. The metadata operations are inserted into the metadata log together with the sequence numbers. In one embodiment, the metadata operations are sorted within the metadata log based on the sequence number, or the metadata operations are inserted into the metadata log based on the sequence number. After the metadata operations are copied to the second computing device for execution on the replicated storage object based on the sequence number, the dirty region identified by the dirty region log is copied from the storage object to the replicated storage object. That is, data within the dirty region (e.g., "dirty" data that has not yet been copied to the replicated storage object) is transferred to the second computing device for storage in the corresponding region within the replicated storage object. The replication of the dirty region is triggered based on the completion of the replication of the metadata operations.

[0021] In one embodiment, a dirty region log is created as a core region log (e.g., maintained in memory) for each storage object in a consistency group, such as for each file in the consistency group. Furthermore, a core splitter object (e.g., configured to intercept and replicate operations) is set for each replication endpoint (e.g., a replicated storage object hosted by a second computing device) and is set to a dirty region logging state. This ensures that incoming client write operations are intercepted by the splitter object and a dirty bit is set in the dirty region log for each region modified by the incoming client write operation. Thus, from this point on, dirty regions will be captured by the core. During the cutover phase, the splitter object will be changed to a cutover split state. From this point on, for each incoming client I / O, the corresponding dirty region log of the target storage object is evaluated. If the write operation targets a dirty region of the storage object, the write operation is performed on the storage object. If the write operation targets a non-dirty region or a partially dirty region, the data of the write operation is written to the storage object and split / replicated to the replicated storage object. A cutover scanner can be executed to read the core dirty region log. For each dirty region identified, the dirty data is copied to the replica storage object. During the copy, a lock may be acquired for the dirty region to prevent any writes to the dirty region. Once the second computing device writes the copied dirty data to the replica storage object, the lock is released.

[0022] The present system overcomes technical challenges associated with maintaining correctness (e.g., data and metadata consistency) when metadata operations are replicated before data operations. If the metadata operation and the write data operation target different storage objects, there is no correctness issue (e.g., no data or metadata inconsistencies). If the metadata operation occurs before the write data operation, there is also no correctness issue. If the write data operation is followed by a metadata operation on the file, correctness issues may arise. In particular, due to subsequent metadata operations on the file, some (or all) of the dirty region log content may become invalid. For example, an unlink metadata operation may cause a file to be deleted and invalidate all of the dirty region log content for that file. In another example, a set attribute metadata operation may truncate a file and invalidate a portion of the dirty region log. In these cases, attempts to read these portions on the source storage object (file) will fail. As provided herein, the present system ignores such failures and replicates the remaining valid content. This approach provides data consistency between the source storage object (file) and the destination storage object (mirror copy / replica of the file) at the end of the transition to the synchronous replication state. Thus, the dirty region log is a lightweight mechanism for tracking changes, while the metadata log contains as much detail as the initial operation and takes up a minimal amount of memory.

[0023] In addition, file / directory timestamp replication is provided. This type of replication is used to replicate changes to creation time (e.g., the time a file was created), state change time of a file or directory, data change time, and so on. In this way, the system provides a lightweight method for tracking metadata operations during the transition from asynchronous to synchronous replication, so that metadata operations are replicated to the replicated storage objects while maintaining file system consistency at the replicated storage objects. In addition, the system tracks file and directory timestamp changes in the metadata log during the transition period, so that file and directory timestamp changes are replicated to the replicated storage objects.

[0024] In an embodiment of timestamp replication and the associated transition state challenges, timestamp changes due to metadata operations are logged in the metadata log. However, timestamp changes due to data operations are not logged in the dirty region log. During metadata log drain, timestamps are replicated to the second computing device (destination) as part of file system message execution using a push file operation applied at the replicated storage object. During a dirty region log scan, timestamps are read from the active file system as part of a file system rotation network protocol read message (or any other type of read message). The timestamp is transmitted to the second computing device (destination) as part of the push data operation and conditionally applied to the second computing device if the timestamp is higher than the timestamp in the inode of the replicated storage object (destination file). In this way, a dirty region log scan sends the latest timestamp, rather than the timestamp at the time of the write. During the dirty region log scan, other data operations and / or metadata operations can be performed on the storage object (file) of interest.

[0025] If data operations are performed during a dirty region log scan, the push data operations will carry the latest timestamp. The dirty region log scan will carry the latest timestamp because the timestamp carried in the message is the same as the timestamp at the destination inode of the copied storage object (destination file), so the latest timestamp will not be applied. If metadata operations are performed on the storage object (source file) during a dirty region log scan, this is mutually exclusive with file system rotation network protocol read messages (or any other type of read message) caused by the dirty region log.

[0026] To provide resynchronization of synchronous replication relationships, Figure 1An embodiment of a clustered network environment 100 or a network storage environment is shown. However, it will be appreciated that the techniques described herein, etc., can be implemented in a clustered network environment 100, a non-clustered network environment, and / or various other computing environments (such as a desktop computing environment). That is, the present disclosure, including the scope of the appended claims, is not intended to be limited to the examples provided herein. It will be appreciated that if the same or similar components, elements, features, items, modules, etc. are shown in subsequent figures but have been previously discussed with respect to previous figures, their similar (e.g., redundant) discussion may be omitted when describing the subsequent figures (e.g., for the purpose of simplicity and ease of understanding).

[0027] Figure 1 1 is a block diagram illustrating a cluster network environment 100 in which at least some embodiments of the techniques and / or systems described herein may be implemented. The cluster network environment 100 includes data storage systems 102 and 104 coupled via a cluster fabric 106, such as a computing network embodied as a private Infiniband, Fibre Channel (FC), or Ethernet, to facilitate communications between the data storage systems 102 and 104 (and one or more modules, components, etc. therein, such as nodes 116 and 118). It should be understood that although the Figure 1 104 and two nodes 116 and 118 are shown, but any suitable number of such components is contemplated. In one example, nodes 116, 118 include storage controllers (e.g., node 116 may include a primary or local storage controller, while node 118 may include a secondary or remote storage controller) that provide access to data stored in data storage devices 128, 130 to client devices such as host devices 108, 110. Similarly, the same applies to other modules, elements, features, items, etc. referenced herein and / or shown in the accompanying drawings, unless otherwise specifically provided herein. That is, the specific number of components, modules, elements, features, items, etc. disclosed herein should not be interpreted in a limiting manner.

[0028] It will be further appreciated that the cluster network is not limited to any particular geographic area and can be clustered locally and / or remotely. Thus, in one embodiment, the cluster network can be distributed across multiple storage systems and / or nodes located in multiple geographic locations. In one embodiment, the cluster network can include data storage systems (e.g., 102, 104) that reside in the same geographic location (e.g., in a single on-site rack of data storage devices).

[0029] In the example shown, one or more host devices 108, 110 (which may include, for example, client devices, personal computers (PCs), computing devices for storage (e.g., storage servers), and other computers or peripheral devices (e.g., printers)) are coupled to respective data storage systems 102, 104 via storage network connections 112, 114. The network connections may include a local area network (LAN) or a wide area network (WAN) utilizing, for example, a network attached storage (NAS) protocol (such as the Common Internet File System (CIFS) protocol or the Network File System (NFS) protocol for exchanging data packets), a storage area network (SAN) protocol (such as the Small Computer System Interface (SCSI) or the Fibre Channel Protocol (FCP)), an object protocol (such as S3), etc. Illustratively, the host devices 108, 110 may be general-purpose computers running applications and may interact with the data storage systems 102, 104 using a client / server model to exchange information. That is, the host device can request data from the data storage system (e.g., data on a storage device managed by a network storage control that is configured to process I / O commands issued by the host device for the storage device), and the data storage system can return the results of the request to the host device via one or more storage network connections 112, 114.

[0030] The nodes 116, 118 on the clustered data storage systems 102, 104 may include network or host nodes that are interconnected as a cluster to provide data storage and management services, such as to an enterprise with a remote location, cloud storage (e.g., storage endpoints may be stored in a data cloud), etc. For example, such a node in the clustered network environment 100 may be a device attached to the network as a connection point, a redistribution point, or a communication endpoint. A node may be capable of sending, receiving, and / or forwarding information over a network communication channel and may include any device that meets any or all of these conditions. An example of a node may be a data storage and management server connected to a network, wherein the server may include a general-purpose computer or computing device that is specifically configured to function as a server in a data storage and management system.

[0031] In one example, a first cluster of nodes, such as nodes 116, 118, can be located at a first storage site (e.g., a first set of storage controllers configured to provide access to a first storage set comprising a first logical grouping of one or more storage devices). A second cluster of nodes, not shown, can be located at a second storage site (e.g., a second set of storage controllers configured to provide access to a second storage set comprising a second logical grouping of one or more storage devices). The first cluster of nodes and the second cluster of nodes can be configured according to a disaster recovery configuration, wherein if a disaster occurs at a disaster storage site including the disaster cluster of nodes, the surviving cluster of nodes provides switched access to the storage devices of the disaster cluster of nodes (e.g., if a disaster occurs at the second storage site, the first cluster of nodes provides client devices with switched data access to the storage devices of the second storage set).

[0032] As shown in the cluster network environment 100, the nodes 116, 118 may include various functional components that cooperate to provide a distributed storage architecture for the cluster. For example, the nodes may include network modules 120, 122 and disk modules 124, 126. The network modules 120, 122 may be configured to allow the nodes 116, 118 (e.g., network storage controllers) to connect to the host devices 108, 110 via the storage network connections 112, 114, for example, to allow the host devices 108, 110 to access data stored in the distributed storage system. In addition, the network modules 120, 122 may provide connectivity to one or more other components through the cluster fabric 106. For example, in Figure 1 In the example embodiment, the network module 120 of the node 116 may access the second data storage device by sending a request through the disk module 126 of the node 118 .

[0033] The disk modules 124, 126 can be configured to connect one or more data storage devices 128, 130 (such as disks or disk arrays, flash memory, or some other form of data storage) to the nodes 116, 118. The nodes 116, 118 can be interconnected via the cluster fabric 106, for example, to allow a respective node in the cluster to access data on a data storage device 128, 130 connected to a different node in the cluster. Typically, for example, the disk modules 124, 126 communicate with the data storage devices 128, 130 according to a SAN protocol such as SCSI or FCP. Thus, as seen from an operating system on the node 116, 118, the data storage devices 128, 130 appear to be locally attached to the operating system. In this way, different nodes 116, 118, etc., can access blocks of data through the operating system without explicitly requesting abstract files.

[0034] It should be understood that although cluster network environment 100 illustrates an equal number of network and disk modules, other embodiments may include different numbers of these modules. For example, there may be multiple interconnected network and disk modules in a cluster without a one-to-one correspondence between networks and disk modules. In other words, different nodes may have different numbers of network and disk modules, and the same node may have a different number of network modules than disk modules.

[0035] In addition, the host devices 108, 110 can be networked with the nodes 116, 118 in the cluster via the storage networking connections 112, 114. As an example, the respective host devices 108, 110 networked to the cluster can request services (e.g., exchange information in the form of data packets) of the nodes 116, 118 in the cluster, and the nodes 116, 118 can return the results of the requested services to the host devices 108, 110. In one embodiment, the host devices 108, 110 can exchange information with network modules 120, 122 residing in the nodes 116, 118 (e.g., network hosts) in the data storage systems 102, 104.

[0036] In one embodiment, data storage device 128,130 comprises volume 132, and volume 132 is the realization that information is stored on disk drive or disk array or other storage (for example, flash memory) as the file system for data.In one example, disk array can comprise the combination of all traditional hard disk drives, all flash drives, or traditional hard disk drives and flash drives.For example, volume can span a part, disk set or partial disk of disk, and defines the overall logical arrangement of file storage on the disk space in storage system usually.In one embodiment, volume can comprise the storage data of one or more files in the hierarchical directory structure as residing in this volume.

[0037] Volumes are typically configured in a format that can be associated with a specific storage system, and the corresponding volume format typically includes features that provide functionality for the volume, such as providing the ability for the volumes to form clusters. For example, where a first storage system may use a first format for its volumes, a second storage system may use a second format for its volumes.

[0038] In clustered network environment 100, host devices 108 and 110 can utilize data storage systems 102 and 104 to store and retrieve data from volume 132. In this embodiment, for example, host device 108 can send data packets to network module 120 in node 116 within storage device 102. Node 116 can use disk module 124 to forward the data to data storage device 128, where data storage device 128 includes volume 132A. In this manner, in this example, host devices can use data storage system 102 connected via storage network connection 112 to access volume 132A to store and / or retrieve data. Furthermore, in this embodiment, host device 110 can exchange data with network module 122 in node 118 within data storage system 104 (e.g., which can be remote from data storage system 102). Node 118 can use disk module 126 to forward the data to data storage device 130, thereby accessing volume 132B associated with data storage device 130.

[0039] It will be appreciated that replay of metadata and data operations can be implemented within the cluster network environment 100. In one example, the operations can be executed at the node 116 and replayed at the node 118. It will be appreciated that replay of metadata and data operations can be implemented for and / or between any type of computing environment and can be transportable between physical devices (e.g., nodes 116, nodes 118, desktop computers, tablet computers, laptop computers, wearable devices, mobile devices, storage devices, servers, etc.) and / or cloud computing environments (e.g., remote from the cluster network environment 100).

[0040] Figure 2 is a data storage system 200 (e.g., Figure 1 102, 104) in which further details are provided of embodiments of components that may implement one or more techniques and / or systems described herein. The data storage system 200 includes a node 202 (e.g., Figure 1 116, 118) and a data storage device 234 (e.g., Figure 1 Node 202 may be, for example, a general-purpose computer or some other computing device specifically configured to function as a storage server. Figure 1 108, 110) can be connected to the node 202 via the network 216, for example, to provide access to files and / or other data stored on the data storage device 234. For example, the node 202 includes a storage controller that provides access to data stored within the data storage device 234 to client devices such as the host device 205.

[0041] The data storage device 234 may include a mass storage device, such as the disks 224, 226, 228 of the disk arrays 218, 220, 222. It should be understood that the techniques and systems described herein are not limited to this example embodiment. For example, the disks 224, 226, 228 may include any type of mass storage device, including but not limited to disk drives, flash memory, and any other similar media suitable for storing information, including, for example, data (D) and / or parity (P) information.

[0042] The node 202 includes one or more processors 204, a memory 206, a network adapter 210, a cluster access adapter 212, and a storage adapter 214 interconnected via a system bus 242. The data storage system 200 also includes an operating system 208 installed in the memory 206 of the node 202. The operating system 208 can, for example, implement a redundant array of independent (or inexpensive) disks (RAID) optimization technique to optimize the process of reconstructing data from a failed disk in the array.

[0043] The operating system 208 may also manage communications for the data storage system and may be located in a cluster network such as attached to a cluster fabric 215 (eg, Figure 1 106 in the cluster network system). Thus, nodes 202, such as network storage controllers, can respond to host device requests to manage data on data storage devices 234 (e.g., or other cluster devices) in accordance with these host device requests. For example, the operating system 208 can typically establish one or more file systems on the data storage system 200, wherein the file system can include software code and data structures that implement a persistent hierarchical name space for files and directories. As an example, when a new data storage device (not shown) is added to the cluster network system, the operating system 208 is notified where new files associated with the new data storage device will be stored in the existing directory tree. This is generally referred to as "mounting" the file system.

[0044] In the example data storage system 200, the memory 206 may include storage locations addressable by the processor 204 and adapters 210, 212, 214 for storing relevant software application code and data structures. The processor 204 and adapters 210, 212, 214 may, for example, include processing elements and / or logic circuitry configured to execute software code and manipulate data structures. An operating system 208 (portions of which typically reside in the memory 206 and are executed by the processing elements) functionally organizes the storage system by invoking storage operations to support file services implemented by the storage system, etc. It will be clear to those skilled in the art that other processing and storage mechanisms (including various computer-readable media) may be used to store and / or execute application instructions related to the techniques described herein. For example, the operating system may also utilize one or more control files (not shown) to help provide virtual machines.

[0045] Network adapter 210 includes the mechanical, electrical, and signaling circuitry required to connect data storage system 200 to host device 205 via network 216, which may include a point-to-point connection or a shared medium, such as a local area network. Figure 1 108, 110) can be a general-purpose computer configured to execute an application. As described above, the host device 205 can interact with the data storage system 200 according to a client / host model of information transfer.

[0046] The storage adapter 214 cooperates with the operating system 208 executing on the node 202 to access information requested by the host device 205 (e.g., access data on a storage device managed by a network storage controller). The information can be stored on any type of attached writable media array, such as a disk drive, flash memory, and / or any other similar media suitable for storing information. In the example data storage system 200, the information can be stored in data blocks on the disks 224, 226, 228. The storage adapter 214 may include input / output (I / O) interface circuitry that is coupled to the disks via an I / O interconnect arrangement such as a storage area network (SAN) protocol (e.g., small computer system interface (SCSI), iSCSI, hyperSCSI, Fibre Channel Protocol (FCP)). The information is retrieved by the storage adapter 214 and processed, if necessary, by one or more processors 204 (or the storage adapter 214 itself) and then forwarded via the system bus 242 to the network adapter 210 (and / or cluster access adapter 212, if sending to another node in the cluster), where the information is formatted into data packets and returned to the host device 205 via the network 216 (and / or returned to another node attached to the cluster via the cluster fabric 215).

[0047] In one embodiment, information storage on disk arrays 218, 220, 222 can be implemented as one or more storage volumes 230, 232, which are composed of clusters of disks 224, 226, 228 that define an overall logical arrangement of disk space. The disks 224, 226, 228 comprising one or more volumes are typically organized into one or more RAID groups. As an example, volume 230 comprises a collection of disk arrays 218 and 220, which comprise clusters of disks 224 and 226.

[0048] In one embodiment, to facilitate access to disks 224, 226, 228, operating system 208 may implement a file system (e.g., a write-anywhere file system) that logically organizes information into a hierarchy of directories and files on the disks. In this embodiment, a respective file may be implemented as a set of disk blocks configured to store information, while a directory may be implemented as a specially formatted file in which information about other files and directories is stored.

[0049] Regardless of the underlying physical configuration within the data storage system 200, data may be stored as files in physical and / or virtual volumes, which may be associated with corresponding volume identifiers, such as file system identifiers (FSIDs), which may be 32 bits in length in one example.

[0050] A physical volume corresponds to at least a portion of a physical storage device whose address, addressable space, location, etc. does not change, such as at least some of one or more data storage devices 234 (e.g., a Redundant Array of Independent (or Inexpensive) Disks (RAID system). Typically, the location of a physical volume does not change because the address(es) (ranges) used to access the physical volume typically remain unchanged.

[0051] Instead, a virtual volume is stored on a collection of different portions of different physical storage devices. A virtual volume can be a collection of different available portions of different physical storage device locations, such as some available space from each of disks 224, 226, and / or 228. It should be understood that because a virtual volume is not "bound" to any one specific storage device, it can be said that the virtual volume includes an abstraction layer or virtualization layer that makes it resizable and / or flexible in certain aspects.

[0052] Furthermore, a virtual volume may include one or more logical unit numbers (LUNs) 238, directories 236, Qtrees 235, and files 240. These features (specifically, LUNs) allow, among other things, the identification and grouping of different storage locations in which data is stored, such as into data storage units. Thus, LUNs 238 may be characterized as constituting a virtual disk or drive on which the data within the virtual volume is stored. For example, LUNs are often referred to as virtual drives, allowing them to emulate hard drives in general-purpose computers, while actually comprising blocks of data stored in various portions of the volume.

[0053] In one embodiment, one or more data storage devices 234 may have one or more physical ports, each of which may be assigned a target address (e.g., a SCSI target address). To represent a corresponding volume stored on the data storage device, the target address on the data storage device may be used to identify one or more LUNs 238. Thus, for example, when a node 202 is connected to a volume 230, 232 via a storage adapter 214, a connection is created between the node 202 and the one or more LUNs 238 underlying the volume.

[0054] In one embodiment, a corresponding target address may identify multiple LUNs, such that a target address may represent multiple volumes. For example, an I / O interface, which may be implemented as circuitry and / or software in storage adapter 214 or as executable code residing in memory 206 and executed by processor 204, may connect to volume 230 using one or more addresses identifying one or more LUNs 238.

[0055] It will be appreciated that replay of metadata and data operations can be implemented for the data storage system 200. In one example, the node 202 can perform operations on NVRAM and replay the operations to flush the NVRAM to storage. It will be appreciated that replay of metadata and data operations can be implemented for and / or between any type of computing environment and can be transportable between physical devices (e.g., the node 202, the host device 205, a desktop computer, a tablet computer, a laptop computer, a wearable device, a mobile device, a storage device, a server, etc.) and / or a cloud computing environment (e.g., remote from the node 202 and / or the host device 205).

[0056] One embodiment of resynchronizing a synchronous replication relationship is by Figure 3 The exemplary method 300 is shown and further combined with Figure 4400. In one embodiment, a storage object 402 at a first computing environment may have an asynchronous replication relationship with a replicated storage object 404 at a second computing environment. The storage object may correspond to a file, a directory, a file system, a volume, a storage virtual machine including multiple volumes hosted across one or more nodes of the first computing environment, and the like. In one embodiment, the storage object 402 and the replicated storage object 404 may have an asynchronous replication relationship, a synchronous replication relationship that has become out of sync, or no replication relationship. A resynchronization process may be performed to transition the storage object 402 and the replicated storage object 404 to a synchronous replication state.

[0057] At 301, a snapshot creation operation 407 is performed to prepare for the subsequent last asynchronous incremental transfer as part of the resynchronization process. In particular, incoming metadata operations are maintained (e.g., inserted) into a splitter queue. The splitter queue can be associated with a splitter object that is configured to split (copy) operations targeting storage object 402 to a second computing environment for submission to a replicated storage object 404. A snapshot of storage object 402 is created for the last asynchronous incremental transfer. Once the snapshot is created, incoming metadata operations are dequeued from the splitter queue. The last exception copy is performed. Incoming operations are allowed onto storage object 402 and are logged in the dirty region log and metadata log. In one embodiment, the snapshot creation operation 407 is performed before, during, or after the asynchronous incremental transfer 460.

[0058] At 302, an asynchronous replication process is executed to perform an asynchronous incremental transfer 406 of data from storage object 402 to replicated storage object 404. For example, the asynchronous incremental transfer may transfer incremental data of storage object 402 between the time a first snapshot of storage object 402 is created and the time a second subsequent snapshot of storage object 402 is created. The incremental data is identified by comparing the first snapshot with the second snapshot to identify the data differences between the first snapshot and the second snapshot as incremental data. Asynchronous incremental transfer 406 includes the last asynchronous incremental transfer associated with the snapshot created by snapshot creation operation 401.

[0059] Performing an asynchronous incremental transfer to copy the delta data from storage object 402 to replicated storage object 404 helps bring replicated storage object 404 closer to a state with the same data as storage object 402. However, the delta data corresponds only to the differences between storage object 402 and replicated storage object 404 at the time the second snapshot was created. Therefore, storage object 402 may be further modified by data and / or metadata modification operations performed after the second snapshot was created. Therefore, as will be described in further detail, the dirty region log and the metadata log are used to track these data operations and metadata operations performed after the second snapshot was created and / or to track new / different data and metadata created by these operations for replication to replicated storage object 404.

[0060] At 304, incoming write operations targeting storage object 402 are logged in a dirty region log. The incoming write operations were logged during the last asynchronous incremental transfer. For example, the dirty region log includes a bitmap having bits corresponding to regions of storage object 402. The bits for a region can be set to a first value to indicate that the region has not been modified by the write operation (e.g., the region includes data that is the same as data within the corresponding region of replicated storage object 404), or to a second value to indicate that the region is now a dirty region having dirty data based on the write operation that modified the region (e.g., a "dirty" region includes data that has not yet been copied to replicated storage object 404, and therefore the dirty region includes data that is different from the data stored within the corresponding region within replicated storage object 404). Therefore, when a write operation is performed on storage object 402 during the last asynchronous incremental transfer, bits corresponding to regions modified by the write operation are set to the second value to indicate that these are now dirty regions having modified data that has not yet been copied to replicated storage object 402.

[0061] In one embodiment, a signature is associated with (assigned to) a write operation to indicate that the dirty region log will account for the write operation. The signature can include a predefined signature shared with the file system hosting storage object 402, or a dynamically determined signature based on various factors that can also be calculated by the file system (e.g., a hash of the data to be written by the write operation). In one embodiment, the write operation is intercepted above the file system that implements the dirty region log (e.g., within a layer of the storage stack above the operating system) and can therefore be assigned a signature before being forwarded to the file system (e.g., the operating system). The file system hosting storage object 402 is configured to fail write operations that are not assigned a signature. This is because the dirty region log does not account for and track such write operations that lack a signature. If the file system were to perform these write operations that are not accounted for by the dirty region log, then when the dirty region log is used to replicate dirty data in dirty regions within storage object 402, the modifications made by the write operation (dirty data in storage object 402 that has not yet been replicated to replicated storage object 404) would not be replicated to replicated storage object 404 because the dirty region log never accounts for and tracks write operations. Additionally, a signature is associated with (assigned to) the metadata operation to indicate that the metadata operation log will list the metadata operation. The file system will fail metadata operations that do not have a signature assigned to them.

[0062] At 306, metadata operations performed on storage object 402 during the last asynchronous incremental transfer are logged in a metadata log. Metadata operations may include create operations (e.g., create file operations, create LUN operations, etc.), link operations, unlink operations, rename operations (e.g., file rename operations, etc.), set attribute operations (e.g., file resize operations), etc. Metadata operations are logged in the metadata log by inserting the metadata operations into the metadata log. At 308, sequence numbers are assigned to the metadata operations in the metadata log based on the order in which the metadata operations were executed. In one embodiment, metadata operations may be sorted within the metadata log based on the sequence numbers. In one embodiment, metadata operations are inserted into the metadata log based on the sequence numbers. In one embodiment, the metadata log includes a queue. Metadata operations are logged into the queue based on the assigned sequence numbers or are sorted within the queue based on the assigned sequence numbers. In this manner, metadata operations performed on storage object 402 are tracked for subsequent replication to replicated storage object 402.

[0063] At 310, metadata operations in the metadata log are copied 408 to the second storage environment hosting the replicated storage object 404 for execution on the replicated storage object 404 in sequence according to the sequence number. Additionally, the mtime (e.g., the modification time at which the contents of the storage object were modified) and ctime (e.g., the change time at which the contents or properties of the storage object were changed) of the storage object 402 are copied. That is, metadata operations are executed by the second computing environment on the replicated storage object 404 in the same order as they were executed by the first computing environment on the storage object 402. In one embodiment, incoming metadata operations are prevented from being executed (e.g., blocked, failed, queued for later execution, etc.) until a snapshot can be created for the last asynchronous incremental transfer. That is, a snapshot of the storage object 402 is created as part of the last asynchronous incremental transfer to capture the current state of the storage object 402. Preventing the execution of incoming metadata operations on the storage object 402 avoids capturing metadata operations in both the metadata log (e.g., the execution of incoming metadata operations is captured in the metadata log for subsequent replication to the replicated storage object 404) and the snapshot.

[0064] In one embodiment, timestamp changes for storage objects 402 are logged in a metadata log. Timestamp changes can correspond to changes in time values ​​such as creation time (e.g., time when a file is created), time when a file or directory's state changes, time when a data change occurs (e.g., time when a file was last modified), etc. Timestamp changes are replicated to replicated storage objects 404 using the metadata log.

[0065] At 312, dirty regions of the storage object 402 identified by the dirty region log are copied 410 to the replica storage object 404. In one embodiment, the dirty regions are copied 410 after the replication 408 of metadata operations within the metadata log is completed on the replica storage object 404. That is, the completion of the replication 408 of metadata operations within the metadata log triggers the start of the replication 410 of dirty data within the dirty regions (e.g., dirty data that has not yet been replicated from the storage object 402 to the replica storage object 404 because the dirty data was written to the storage object 402 by a data operation (such as a write operation) that occurred after the last asynchronous replication transfer).

[0066] During the replication of dirty regions 410, incoming write operations targeting storage object 402 are processed on a case-by-case basis. Write operations targeting non-dirty regions (e.g., regions whose bits are set to a first value to indicate that such regions have not been modified by a write operation since logging of incoming write operations using the dirty region log began) are performed on storage object 402 and directly replicated and performed on replicated storage object 404. Write operations targeting dirty regions are performed only on storage object 402 and logged in the dirty region log so that the dirty region log can be used to replicate these changes to replicated storage object 404 through subsequent write operations. Portions of the dirty region log that become invalid due to subsequently performed metadata operations are not replicated to replicated storage object 404.

[0067] The present system improves upon existing techniques for establishing or reestablishing a synchronous replication relationship between storage objects because client I / O is not held (e.g., blocked, failed, aborted, or queued for later execution) during the transition to the synchronous replication state. Instead, dirty region logs and metadata logs are used to track and replicate such client I / O, such as data operations and metadata operations performed on storage objects 402. In this manner, the present system establishes the synchronous replication state in a non-destructive manner, which can reduce wait times that would otherwise occur if client I / O were held.

[0068] Another embodiment 500 involves a computer-readable medium comprising processor-executable instructions configured to implement one or more techniques presented herein. Figure 5 An example embodiment of a computer-readable medium or computer-readable device designed in these ways is shown in FIG, wherein the implementation includes a computer-readable medium 508 having computer-readable data 506 recorded thereon, such as a recordable compact disc (CD-R), a recordable digital versatile disc (DVD-R), a flash drive, a hard drive platter, etc. The computer-readable data 506 (such as binary data including at least one of 0 or 1) in turn includes processor-executable computer instructions 504 configured to operate according to one or more principles set forth herein. In some embodiments, for example, the processor-executable computer instructions 504 are configured to perform a method 502, such as Figure 3 In some embodiments, for example, the processor-executable computer instructions 504 are configured to implement a system such as Figure 4 It is contemplated that many such computer-readable media will operate in accordance with the techniques presented herein.

[0069] Figure 6is a diagram illustrating an example operating environment 600 in which embodiments of the techniques described herein may be implemented. In one example, the techniques described herein may be implemented within a client device 628, such as a laptop computer, tablet computer, personal computer, mobile device, wearable device, or the like. In another example, the techniques described herein may be implemented within a storage controller 630 (such as a node) configured to manage storage and access to data on behalf of the client device 628 and / or other client devices. In another example, the techniques described herein may be implemented within a distributed computing platform 602 (such as a cloud computing environment (e.g., a cloud storage environment, a multi-tenant platform, etc.) configured to manage storage and access to data on behalf of the client device 628 and / or other client devices.

[0070] In yet another example, at least some of the techniques described herein are implemented on one or more of the client device 628, the storage controller 630, and the distributed computing platform 602. For example, the client device 628 can transmit operations (such as data operations to read and write data, and metadata operations (e.g., create file operations, rename directory operations, resize operations, set attribute operations, etc.)) to the storage controller 630 via the network 626 for implementation by the storage controller 630 upon storage. The storage controller 630 can store data associated with operations within volumes or other data objects / structures hosted in locally attached storage, remote storage hosted by other computing devices accessible via the network 626, storage provided by the distributed computing platform 602, and the like. The storage controller 630 can replicate the data and / or operations to other computing devices to maintain one or more replicas, such as a destination storage volume maintained as a replica of a source storage volume. Such replicas can be used for disaster recovery and failover.

[0071] By transferring the data to the distributed computing platform 602, the storage controller 630 may store the data or a portion thereof in storage hosted by the distributed computing platform 602. In one example, the storage controller 630 may store frequently accessed data locally in locally attached storage. Data that is less frequently accessed may be transferred to the distributed computing platform 602 to be stored in the data storage layer 608. The data storage layer 608 may store the data in the service data repository 620 and may store client-specific data in client data repositories assigned to such data, such as the client (1) data repository 622 for storing data for client (1) and the client (N) data repository 624 for storing data for client (N). The data storage may be a physical storage device or may be defined as a logical storage that may be defined across one or more physical storage devices, such as a virtual volume, LUN, or other logical organization of data. In another example, the storage controller 630 transfers and stores all client data to the distributed computing platform 602. In yet another example, the client device 628 transfers and stores the data directly to the distributed computing platform 602 without using the storage controller 630.

[0072] Management of storage and access to data may be performed by one or more storage virtual machines (SVMs) or other storage applications providing software as a service (SaaS), such as storage software services. In one example, the SVMs may be hosted within the client device 628, within the storage controller 630, or within the distributed computing platform 602, such as by the application server layer 606. In another example, one or more SVMs may be hosted on one or more of the client device 628, the storage controller 630, and the distributed computing platform 602.

[0073] In one example of the distributed computing platform 602, one or more SVMs can be hosted by the application server layer 606. For example, server (1) 616 is configured to host an SVM for executing an application (such as a storage application) that manages data storage for client (1) within client (1) data repository 622. Thus, the SVM executing on server (1) 616 can receive data and / or operations from client device 628 and / or storage controller 630 via network 626. The SVM executes the storage application to process the operations and / or store the data within client (1) data repository 622. The SVM can transmit a response, such as a success message or an error message, back to the client device 628 and / or storage controller 630 via network 626. In this manner, the application server layer 606 can use server (1) 616, server (N) 618, etc. to host SVMs, services, and / or other storage applications.

[0074] The user interface layer 604 of the distributed computing platform 602 can provide the client device 628 and / or the storage controller 630 with access to user interfaces associated with storing and accessing data and / or other services provided by the distributed computing platform 602. In one example, a service user interface 610 can be accessible from the distributed computing platform 602 to access services subscribed to by the client and / or storage controller, such as data replication services, application hosting services, data security services, human resources services, warehouse tracking services, accounting services, etc. For example, client user interfaces can be provided to corresponding clients, such as client (1) user interface 612, client (N) user interface 614, etc. Client (1) can access various services and resources subscribed to by client (1) through client (1) user interface 612, such as access to web services, development environments, human resources applications, warehouse tracking applications, and / or other services and resources provided by the application server layer 606, which can use data stored in the data storage layer 608.

[0075] Client device 628 and / or storage controller 630 can subscribe to certain types and amounts of services and resources provided by distributed computing platform 602. For example, client device 628 can establish a subscription to access three virtual machines, a certain amount of storage, a certain type / amount of data redundancy, a certain type / amount of data security, certain service level agreements (SLAs) and service level objectives (SLOs), latency guarantees, bandwidth guarantees, access for executing or hosting certain applications, etc. Similarly, storage controller 630 can establish a subscription to access certain services and resources of distributed computing platform 602.

[0076] As shown, various clients incorporated with and / or into various computing devices, such as client device 628 and storage controller 630, may communicate with distributed computing platform 602 via one or more networks, such as network 625. For example, the clients may be incorporated with and / or into a client application (e.g., software) that is implemented at least in part by one or more computing devices.

[0077] Examples of suitable computing devices include personal computers, server computers, desktop computers, nodes, storage servers, storage controllers, laptop computers, notebook computers, tablet computers or personal digital assistants (PDAs), smartphones, cell phones, and consumer electronic devices incorporating one or more computing device components, such as one or more electronic processors, microprocessors, central processing units (CPUs), or controllers. Examples of suitable networks include networks utilizing wired and / or wireless communication technologies, and networks operating according to any suitable networking and / or communication protocols (e.g., the Internet). In use cases involving the delivery of customer support services, the labeled computing device represents the endpoint of the customer support delivery process, i.e., the consumer's device.

[0078] A distributed computing platform 602 (such as a multi-tenant business data processing platform or a cloud computing environment) may include multiple processing layers, including a user interface layer 604, an application server layer 606, and a data storage layer 608. The user interface layer 604 may maintain multiple user interfaces, including graphical user interfaces and / or web-based interfaces. The user interfaces may include a service user interface 610 for a service to provide access to applications and data of clients (e.g., "tenants") of the service. The user interfaces may also include one or more user interfaces that have been specialized / customized according to the specific requirements of the user and may be accessed via one or more APIs.

[0079] The service user interface 610 may include components that enable a tenant to manage its participation in the functions and capabilities provided by the distributed computing platform 602, such as accessing data, causing the execution of specific data processing operations, etc. Each processing layer may be implemented using a set of computers, a virtualized computing environment such as a storage virtual machine or storage virtual server, and / or computer components including computer servers and processors, and may perform various functions, methods, processes, or operations determined by the execution of software applications or sets of instructions.

[0080] The data storage layer 608 may include one or more data repositories, which may include a service data repository 620 and one or more client data repositories. Each client data repository may contain tenant-specific data used as part of providing a range of tenant-specific business and storage services or functions, including but not limited to ERP, CRM, e-commerce, human resource management, payroll, storage services, etc. The data repositories may be implemented using any suitable data storage technology, including a relational database management system (RDBMS) based on structured query language (SQL), a file system hosted by an operating system, object storage, etc.

[0081] According to one embodiment of the present invention, the distributed computing platform 602 may be a multi-tenant and service platform operated by an entity to provide a set of business-related applications, data storage, and functionality to multiple tenants. These applications and functionality may include applications and functionality used by an enterprise to manage various aspects of its operations. For example, the applications and functionality may include providing web-based access to business information systems, thereby allowing users with a browser and an internet or intranet connection to view, input, process, or modify certain types of business information or any other type of information.

[0082] In one embodiment, the described methods and / or their equivalents can be implemented using computer-executable instructions. Thus, in one embodiment, a non-transitory computer-readable / storage medium is configured with stored computer-executable instructions of an algorithm / executable application that, when executed by a (multiple) machine, causes the (multiple) machine (and / or related components) to perform the method. Example machines include, but are not limited to, processors, computers, servers operating in a cloud computing system, servers configured in a software-as-a-service (SaaS) architecture, smartphones, and the like. In one embodiment, a computing device is implemented using one or more executable algorithms configured to perform any of the disclosed methods.

[0083] It should be understood that the processes, architectures and / or procedures described herein can be implemented in hardware, firmware and / or software. It should also be understood that the provisions set forth herein can apply to any type of special-purpose computer (e.g., a file host, storage server and / or storage service device) and / or general-purpose computer, including stand-alone computers or portions thereof, that is embodied as or includes a storage system. In addition, the teachings herein can be configured as a variety of storage system architectures, including but not limited to network-attached storage environments, and / or storage area networks and disk components directly attached to a client or host. Therefore, in addition to any subsystems configured to perform storage functions and associated with other devices or systems, a storage system should be broadly considered to include such arrangements.

[0084] In some embodiments, the method described and / or shown in the present disclosure can be implemented in whole or in part on a computer-readable medium.Computer-readable medium can include a processor executable instruction configured to implement one or more methods proposed herein, and can include any mechanism for storing the data, which can be read by a computer system afterwards.The example of a computer-readable medium includes a (hard) drive (e.g., accessible via a network attached storage (NAS)), a storage area network (SAN), volatile and non-volatile memory, such as a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM) and / or flash memory, a compact disc read-only memory (CD-ROM), CD-R, a rewritable compact disc (CD-RW), DVD, a cassette tape, a magnetic tape, a disk storage, an optical or non-optical data storage device, and / or any other medium that can be used to store data.

[0085] Although the subject matter has been described in language specific to structural features or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing at least some of the claims.

[0086] Various operations of the embodiments are provided herein. The order in which some or all operations are described should not be interpreted as implying that these operations are necessarily related to the order. In view of the benefits of this specification, alternative orderings will be understood. In addition, it will be understood that not all operations must be present in each embodiment provided herein. Moreover, it will be understood that in some embodiments, not all operations are required.

[0087] Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard application or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. As used herein, the term "article of manufacture" is intended to encompass a computer application accessible from any computer-readable device, carrier, or media. Of course, many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.

[0088] As used in this application, the terms "component," "module," "system," "interface," and the like are generally intended to refer to a computer-related entity, whether hardware, a combination of hardware and / or hardware, software, or software execution. For example, a component includes a process running on a processor, a processor, an object, an executable file, a thread of execution, an application, or a computer. As an illustration, both an application running on a controller and the controller can be components. One or more components residing within a process or thread of execution can also be located on one computer or distributed between two or more computers.

[0089] In addition, "exemplary" is used herein to mean used as an example, instance, illustration, etc., and is not necessarily advantageous. As used in this application, "or" is intended to mean an inclusive "or" rather than an exclusive "or". In addition, "a" and "an" used in this application are generally interpreted to mean "one or more" unless otherwise specified or clearly pointed to in the singular form from the context. Moreover, at least one of A and B, etc. generally refers to A or B and / or both A and B. In addition, if "includes", "having", "has", "with" or its variants are used, such terms are intended to be inclusive in a manner similar to the term "comprising".

[0090] Many modifications may be made to the present disclosure without departing from the scope or spirit of the claimed subject matter. Unless otherwise indicated, "first," "second," and the like do not imply temporal, spatial, sequential, or the like. Rather, such terms are used merely as identifiers, names, and the like of features, elements, items, and the like. For example, a first information set and a second information set generally correspond to information set A and information set B, or two different or two identical information sets, or the same information set.

[0091] In addition, although the present disclosure has been shown and described with respect to one or more implementations, other persons skilled in the art will be able to conceive of equivalent changes and modifications based on a reading and understanding of this specification and the drawings. The present disclosure includes all such modifications and changes and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (e.g., elements, resources, etc.), unless otherwise indicated, the terms used to describe such components are intended to correspond to (e.g., be functionally equivalent to) any component that performs the specified function of the described component, even if not structurally equivalent to the disclosed structure. In addition, although a particular feature of the present disclosure may have been disclosed for only one of several implementations, such feature may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application.

Claims

1. A method comprising: performing an asynchronous incremental transfer of data of the storage object from the first node to the replicated storage object at the second node; assigning a signature to a write operation received for execution by the first node to indicate that the write operation is logged by a dirty region log; Because the write operation to which the signature is not assigned is not listed and tracked by the dirty region log, the file system of the first node fails the write operation to which the signature is not assigned; During a last asynchronous incremental transfer in the asynchronous incremental transfer, performing a metadata operation on the storage object based on the first node, and recording the metadata operation and a timestamp log of the metadata operation in a metadata log by the first node; The first node replicates the metadata operation and the timestamp in the metadata log to the second node for execution on the replicated storage object, wherein a modification time when the content in the storage object is modified and a change time when the attribute of the storage object is changed are replicated to the second node; configuring a core splitter object according to a dirty region logging state, wherein the core splitter object sets a dirty bit in the dirty region log for a region modified by the write operation; responsive to implementing a cutover phase, transitioning the core splitter object to a cutover split state to process incoming write operations on a case-by-case basis based on whether the incoming write operations target a dirty region of the storage object, wherein if the incoming write operation targets a dirty region of the storage object, the incoming write operation is performed on the storage object and logged in the dirty region, and wherein if the incoming write operation targets a non-dirty region or a partially dirty region of the storage object, the incoming write operation is written to the storage object and replicated to the replicated storage object; In response to completion of the replicated metadata operation, during the cutover phase, triggering replication of dirty regions of the storage object identified by the dirty region log and replication of a latest timestamp read from the active file system of the first node to the replicated storage object, wherein the latest timestamp is conditionally applied only if the latest timestamp is greater than a timestamp in an inode of the replicated storage object; as well as Responsive to the cutover phase being complete, a transition is made to a synchronous replication state in which incoming operations targeting the storage object are synchronously replicated to the replicated storage object.

2. The method of claim 1 , wherein the second node maintains a plurality of replication endpoints, and wherein the method comprises: For each replication endpoint, a core splitter object for the replication endpoint is set, wherein the replication endpoint corresponds to the replication storage object.

3. The method according to claim 1, comprising: A cutover scanner is executed to read the dirty region log and copy dirty data to the replicated storage object as replicated dirty data, wherein a lock is obtained from the dirty region log for a dirty region identified by the cutover scanner to prevent writes to the dirty region while the dirty region is copied, and wherein the lock is removed in response to the second node writing the replicated dirty data to the replicated storage object.

4. The method according to claim 1 , wherein triggering the replication of the dirty region of the storage object comprises: Based on the write operation corresponding to a non-dirty region, a write operation is performed on the storage object and the replicated storage object.

5. The method according to claim 1 , wherein triggering the replication of the dirty region of the storage object comprises: A write operation is performed on the storage object, and based on the write operation corresponding to a dirty region identified by the dirty region log, replication of the write operation to the replicated storage object is avoided. The method of claim 1 , wherein the metadata log is a queue, within which the metadata operations are ordered based on a sequence number.

7. The method according to claim 1, comprising: Record a timestamp change log for the storage object in the metadata log.

8. The method according to claim 7, comprising: The timestamp changes in the metadata log are copied to the replicated storage object.

9. The method according to claim 1, comprising: Incoming metadata operations are held until a snapshot is created for the last asynchronous incremental transfer.

10. The method of claim 1 , wherein copying the dirty region of the storage object comprises: Avoid copying portions of the dirty region log that become invalid due to subsequently performed metadata operations.

11. A non-transitory machine-readable medium comprising instructions for performing a method, the instructions, when executed by a machine, causing the machine to: performing an asynchronous incremental transfer of data of the storage object from the first node to the replicated storage object at the second node; assigning a signature to a write operation received for execution by the first node to indicate that the write operation is logged by a dirty region log; Because the write operation to which the signature is not assigned is not listed and tracked by the dirty region log, the file system of the first node fails the write operation to which the signature is not assigned; During a last asynchronous incremental transfer in the asynchronous incremental transfer, performing a metadata operation on the storage object based on the first node, and recording the metadata operation and a timestamp log of the metadata operation in a metadata log by the first node; The first node replicates the metadata operation and the timestamp in the metadata log to the second node for execution on the replicated storage object, wherein a modification time when the content in the storage object is modified and a change time when the attribute of the storage object is changed are replicated to the second node; configuring a core splitter object according to a dirty region logging state, wherein the core splitter object sets a dirty bit in the dirty region log for a region modified by the write operation; responsive to implementing a cutover phase, transitioning the core splitter object to a cutover split state to process incoming write operations on a case-by-case basis based on whether the incoming write operations target a dirty region of the storage object, wherein if the incoming write operation targets a dirty region of the storage object, the incoming write operation is performed on the storage object and logged in the dirty region, and wherein if the incoming write operation targets a non-dirty region or a partially dirty region of the storage object, the incoming write operation is written to the storage object and replicated to the replicated storage object; In response to completion of the replicated metadata operation, during the cutover phase, triggering replication of dirty regions of the storage object identified by the dirty region log and replication of a latest timestamp read from the active file system of the first node to the replicated storage object, wherein the latest timestamp is conditionally applied only if the latest timestamp is greater than a timestamp in an inode of the replicated storage object; as well as Responsive to the cutover phase being complete, a transition is made to a synchronous replication state in which incoming operations targeting the storage object are synchronously replicated to the replicated storage object.

12. The non-transitory machine-readable medium of claim 11, wherein the instructions cause the machine to: A splitter queue is associated with each core splitter object, wherein incoming metadata operations received by the core splitter object are inserted into the splitter queue until a snapshot is created in preparation for performing the latest asynchronous replication transfer.

13. The non-transitory machine-readable medium of claim 11, wherein the metadata operation comprises a link operation.

14. The non-transitory machine-readable medium of claim 11 , wherein the instructions cause the machine to: Based on the write operation corresponding to a non-dirty region, a write operation is performed on the storage object and the replicated storage object.

15. The non-transitory machine-readable medium of claim 11, wherein the instructions cause the machine to: A write operation is performed on the storage object, and based on the write operation corresponding to a dirty region identified by the dirty region log, replication of the write operation to the replicated storage object is avoided. 16 . The non-transitory machine-readable medium of claim 11 , wherein the metadata log is a queue within which the metadata operations are ordered based on assigned sequence numbers.

17. A computing device comprising: a memory comprising machine executable code for performing the method; as well as a processor coupled to the memory, the processor configured to execute the machine-executable code such that the processor: assigning a signature to a write operation received for execution by the first node to indicate that the write operation is enumerated by the dirty region log; Because the write operation to which the signature is not assigned is not listed and tracked by the dirty region log, the file system of the first node fails the write operation to which the signature is not assigned; During a last asynchronous incremental transfer of data of a storage object from the first node to a replicated storage object at a second node, performing a metadata operation on the storage object based on the first node, and recording, by the first node, the metadata operation and a timestamp log of the metadata operation in a metadata log; The first node replicates the metadata operation and the timestamp in the metadata log to the second node for execution on the replicated storage object; configuring a core splitter object according to a dirty region logging state, wherein the core splitter object sets a dirty bit in the dirty region log for a region modified by the write operation; responsive to implementing a cutover phase, transitioning the core splitter object to a cutover split state to process incoming write operations on a case-by-case basis based on whether the incoming write operations target a dirty region of the storage object, wherein if the incoming write operation targets a dirty region of the storage object, the incoming write operation is performed on the storage object and logged in the dirty region, and wherein if the incoming write operation targets a non-dirty region or a partially dirty region of the storage object, the incoming write operation is written to the storage object and replicated to the replicated storage object; In response to completion of the replicated metadata operation, during the cutover phase, triggering replication of dirty regions of the storage object identified by the dirty region log and replication of the latest timestamp read from the active file system of the first node to the replicated storage object; as well as Responsive to the cutover phase being complete, a transition is made to a synchronous replication state in which incoming operations targeting the storage object are synchronously replicated to the replicated storage object.

18. The computing device of claim 17, wherein the metadata log is a queue, within which the metadata operations are ordered based on a sequence number.

19. The computing device of claim 17, wherein the machine-executable code causes the processor to: The timestamp changes in the metadata log are copied to the replicated storage object.

20. The computing device of claim 17, wherein the machine-executable code causes the processor to: Incoming metadata operations are held until a snapshot is created for the last asynchronous incremental transfer.

Images