Multi-protocol, multi-site replication

Abstract

Disclosed are techniques for multi-protocol multi-site file replication, including a method that includes capturing IO operations performed on a file system at a gateway device, the method may further include enqueuing the captured IO operations into a number of queues, each of the multiple queues associated with one of a number of target sites, and independently managing replication of the IO operations to the target sites using inode bits.

Description

[Technical field]

[0001] This disclosure relates to file replication, and more particularly, to multi-protocol multi-site file replication. [Background technology]

[0002] The development of the EDVAC system in 1948 is often cited as the beginning of the computer age. Since then, computer systems have evolved into highly complex devices. Today's computer systems typically contain a combination of sophisticated hardware and software components, application programs, operating systems, processors, buses, memory, and input / output devices. As advances in semiconductor processing and computer architectures continue to push performance ever higher, more advanced computer software has evolved to exploit the relatively higher performance of those capabilities, and as a result, today's computer systems are much more powerful than they were just a few years ago.

[0003] These new computer systems have become an important part of many business processes and the lives of individual users. As a result, it is increasingly desirable to keep multiple copies of any information stored in these systems to protect the information against unwanted deletion, hardware failure, power outages, natural disasters, or other events that may render the data unavailable temporarily and / or permanently, or a combination thereof. These copies may be stored on local storage devices (e.g., mirroring one hard drive or partition in one system to a second hard drive or partition in the same system), or may be physically and geographically separated and data transferred across computer networks (e.g., replicating information to a backup system physically located in a different city). The backup process may be performed periodically (e.g., on physical media as weekly copies) or may be performed virtually continuously. Summary of the Invention

[0004] According to an embodiment of the present disclosure, a method for multi-site replication from a single production site includes capturing, at a gateway device, IO operations performed on a file system, the file system having inode bits. The method may further include enqueuing the captured IO operations into a number of queues, each of the multiple queues being associated with one of a number of target sites, and independently managing replication of the IO operations to the target sites using the inode bits. In some embodiments, the number of inode bits may be based on a number of target sites. In some embodiments, one of the inode bits may encode a number of replication states for the arrival of the remote site. In some embodiments, the replication state may be selected from the group consisting of: replication pending, and replication completed. In some embodiments, a first target site of the multiple target sites may include a first backend, and a second target site of the multiple target sites may include a second backend, the second backend being different from the first backend. In some embodiments, managing the replication of the IO operations may include maintaining a parent handler and a child handler on the gateway device. In some embodiments, each of the multiple queues may be responsible for replicating data to a target cloud. In some embodiments, the method may further include metering the replicated IO operations and generating a bill based on the metered usage.

[0005] A further aspect provides a computer program for implementing the method.

[0006] A further aspect provides a corresponding computer program product.

[0007] A further aspect provides a system for implementing the method.

[0008] The above summary is not intended to describe each illustrated embodiment or every implementation of the present disclosure.

[0009] The drawings included in this application are incorporated in and constitute a part of this specification. They illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. The drawings are merely illustrative of particular embodiments and are not intended to be limiting of the disclosure. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an embodiment of a data processing system (DPS) in accordance with some embodiments. [Diagram 2] FIG. 1 illustrates an embodiment of a cloud environment suitable for enabling a confidentiality-based intelligent task routing service mesh, according to some embodiments. [Diagram 3] FIG. 1 illustrates a set of functional abstraction layers provided by a cloud computing environment, according to some embodiments. [Figure 4] FIG. 1 is a system diagram of a replication system operating in SW mode, according to some embodiments. [Diagram 5] FIG. 1 is a system diagram of a replication system operating in MW mode, according to some embodiments. [Figure 6] 1 is a flowchart illustrating a multi-protocol, multi-site replication method according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] While the invention is susceptible to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. It is to be understood, however, that it is not intended to limit the invention to the particular embodiments described. On the contrary, it is intended to cover all modifications, equivalents, and alternatives falling within the scope of the invention.

[0012] Aspects of the present disclosure relate to file replication, and more particularly to multi-protocol, multi-site file replication. Although the present disclosure is not necessarily limited to such applications, various aspects of the present disclosure can be understood through a discussion of various examples using this context.

[0013] In some data replication environments, one site may be designed to act as a production site cluster (also known as a cache site) and one site as a backup site cluster (also known as a home site). The production site may capture all input / output (IO) operations performed on its local filesets and / or file systems and maintain a local queue of IO operations organized in a first-in, first-out (FIFO) queue order. This local queue may be maintained on only one node of the production site / cluster, which may be designated as the "gateway node" or "gateway device" for the cluster. The gateway device may maintain the queue in its memory. All application nodes in the cluster may generate remote procedure calls (RPCs) when local operations are to be performed, which in turn updates the queue on the designated gateway device with the operations to be performed on the filesets. The gateway device may also apply various optimization techniques to the queued operations (for example, coalescing successive smaller write operations into a single write operation) before asynchronously sending the operations to the remote cluster (also known as the home cluster or "secondary site").

[0014] In clustered file system embodiments, replication may occur between a clustered file system (e.g., an on-premise site) and another clustered file system (e.g., an off-premise site), between a clustered file system and cloud object storage, between a clustered file system and multiple backends (e.g., a local backup and two different cloud providers), etc. Such replication may occur via a protocol based on open standards, such as Network File System ("NFS"), or may be supported by the file system protocol itself. In the latter case, both the production file system and the backup file system may be the same file system type, for example, IBM® SpectrumScale™ available from IBM Corporation, Armonk, New York.

[0015] The backend systems in some embodiments may utilize a Portable Operating System Interface (POSIX) compliant protocol, such as the NFS or GPFS file systems available from IBM Corporation of Armonk, New York. These protocols may be desirable in many applications because POSIX compliant operations accepted by the production system may also be replicated to the backup site's file system via the underlying protocol. Furthermore, administrators of the production system may expect remote file systems to have such support. Alternatively, some or all of the backend systems in some embodiments may utilize various cloud object protocols that are not POSIX compliant, but where at least the most common POSIX compliant operations (e.g., create, write, attribute set, rename, remove, hard link, and symbolic link) are supported by the protocol.

[0016] In some embodiments, a cache site hosting a POSIX file system can have three different operating modes: 1. When the cache generates the data (for example at a production site) and the remote site is merely a backup site that receives the data in the form of objects sent from the cache site. This mode of operation of the cache is called Single Writer (SW) mode. 2. If the home site can generate data and the cache site is a receiver, the cache site can pull the data for backup or processing at the cache site. This mode of operation of the cache is called Read Only (RO) mode. 3. A hybrid mode (described below) that supports both pull and push mechanisms. This cache operation mode is called the Multi Writer (MW) mode. Some aspects of the present disclosure may be particularly desirable for use with SW and MW modes because these modes can support pushing data from local cache sites to remote backup sites. Some embodiments may also be desirable because they can support many-to-one mapping in MW mode through an extension of the one-to-one SW mode mapping. That is, the many-to-one mapping in MW mode in some embodiments can extend SW mode to allow multiple cache sites to point to a single home site. Furthermore, because MW mode can both push and pull, some embodiments can maintain accuracy and integrity of data at any one site by combining locally generated data and remotely retrieved data in a seamless manner.

[0017] Some embodiments may be desirable because they can enable replication from a single cache site to multiple backup sites, for example, to maintain multiple backup copies of data. With respect to cloud object stores and / or multiple third-party storage services, some embodiments can enable a cache site that can capture and send data to multiple remote sites (e.g., multiple clouds offered by different cloud service vendors). Additionally or alternatively, some embodiments enable one or more POSIX-compliant backends, as well as applications where the cache can replicate to POSIX-compliant systems (e.g., multiple NFS / native file system backends, one NFS and one native file system backend, one NFS and multiple cloud object backends, etc.).

[0018] Some embodiments may be desirable because they allow a target remote site to be managed separately from another target remote site with respect to replication. Some embodiments may manage these multiple backends by using specific bits (e.g., create / dirty / setattr bits) on their inode file data structures. The bits may indicate whether the IO operation in question (e.g., create or write) was successfully replicated with respect to the particular single remote site in question. For example, in some embodiments, if the create bit is on, it may mean that the associated target system has not yet completed replication of the create operation to the home site. Upon successful completion of replication with the remote site, the home site may receive an acknowledgement (ACK) from the target site. The ACK may then be used to turn off the associated inode bit.

[0019] In some multi-site replication embodiments, some embodiments may maintain multiple queues, one per target site (e.g., on the gateway device). These queues may contain a set of IO operations captured by the gateway device that represent IO operations performed on local file systems / filesets. When a new IO operation is performed in the cache, each of the multiple queues may be enqueued with this IO operation. When that operation is replicated to an associated backup site, the IO operation is dequeued from (and only from) the associated queue. Advantageously, this decision may be binary in nature.

[0020] In some embodiments, when the home system gets an ACK response indicating that the file has been replicated to the cache site, the successful acknowledgment of the operation may be reflected in the inode bit of the cache system. In some embodiments, the status information of the entire collection of target sites may be stored in a file structure, such as a single inode bit, stored in memory of the gateway device. This inode bit in such an embodiment may encode the state of the fileset such that the system and / or administrator can understand whether the fileset has completed any particular replication operation. The replication structure stored in the gateway device may also include various configurable information for the queue and the backend. Based on communication with the home site, the replication state of the fileset may be reflected in this memory structure based on whether the queue of operations is attempted to be performed or remains stalled because the home site is not yet ready to accept the operation.

[0021] Thus, one feature and advantage of some embodiments is the ability to push data from cache sites to remote sites, adapted for both one-to-one mapping in SW mode and many-to-one mapping in MW mode. Moreover, many-to-one mapping may simply be an extension of the one-to-one mapping in SW mode, where multiple cache sites can point to a single home site. Furthermore, since MW mode allows for both pushing and pulling, some embodiments can maintain accuracy and completeness of data at any one site by combining locally generated data with remotely retrieved data in a seamless manner. Furthermore, some embodiments can provide a way to accurately identify remote sites and all connected targets and their protocols, replicate data using file handles, and provide records / status (e.g., files are being replicated, pending, etc.) for all connected targets and files.

[0022] Embodiments of the present disclosure may be provided as part of a service agreement with a client company, non-profit organization, government entity, internal organizational structure, etc. Aspects of these embodiments may include configuring a computer system to perform some or all of the methods described herein, as well as deploying software, hardware, and web services that implement such methods. Aspects of these embodiments also include analyzing a client's operations, making recommendations responsive to the analysis, building a system that implements some of the recommendations, integrating the system into existing processes and infrastructure, metering usage of the system, allocating costs to users of the system, and billing for use of the system.

[0023] Although each of the above embodiments of the present disclosure has been described by setting forth its individual advantages, the present disclosure is not limited to any particular combination thereof. On the contrary, such embodiments may also be combined in any manner and number depending on the intended deployment of the present disclosure without losing their beneficial effects.

[0024] Data Processing System (DPS) FIG. 1 is a diagram of an embodiment of a data processing system (DPS) 100a, 100b (generally referred to herein as DPS 100) according to some embodiments. FIG. 1 depicts only representative major components of DPS 100, each of which may be more complex than depicted in FIG. 1. In some embodiments, DPS 100 may be implemented as a processor embedded in a larger device such as a personal computer, server computer, laptop or notebook computer, portable computer such as a personal digital assistant (PDA), tablet computer, or smartphone, an automobile, an airplane, a teleconferencing system, a home appliance, a smart device, or any other suitable type of electronic device. Additionally, there may be components other than or in addition to those depicted in FIG. 1, and the number, type, and configuration of such components may vary.

[0025] The DPS 100 of FIG. 1 may be comprised of multiple processing units 110a-110d (generally processors 110 or CPUs 110) that may be connected to a main memory 112, a mass storage interface 114, a terminal / display interface 116, a network interface 118, and an input / output ("I / O") interface 120 by a system bus 122. The mass storage interface 114 in this embodiment may connect the system bus 122 to one or more mass storage devices, such as a direct access storage device 140, a USB drive 141, or a readable / writable optical disk drive 142, or a combination thereof. The network interface 118 allows the DPS 100a to communicate with another DPS 100b over a network 106. The main memory 112 may include an operating system 124, multiple application programs 126, and program data 128.

[0026] The embodiment of DPS 100 of FIG. 1 may be a general-purpose computing device. In these embodiments, processor 110 may be any device capable of executing program instructions stored in main memory 112, and may itself be constructed from one or more microprocessors and / or integrated circuits. In some embodiments, DPS 100 may include multiple processors and / or processing cores, as typically found in larger, more powerful computer systems, while in other embodiments, DPS 100 may only include a single processor system, or a single processor designed to emulate a multiprocessor system, or both. Additionally, processor 110 may be implemented using a number of heterogeneous data processing systems in which main processor 110 exists on a single chip with secondary processors. As another illustrative example, processor 110 may be a symmetric multiprocessor system that includes multiple processors 110 of the same type.

[0027] When the DPS 100 boots, the associated processor 110 may first execute program instructions that make up the operating system 124. The operating system 124 may manage the physical and logical resources of the DPS 100. These resources may include main memory 112, mass storage interface 114, terminal / display interface 116, network interface 118, and system bus 122. As with the processor 110, some DPS 100 embodiments may utilize multiple system interfaces 114, 116, 118, 120, and bus 122, each of which may include its own separate fully programmed microprocessor.

[0028] The instructions for the operating system 124 and / or application programs 126 (generally referred to as "program code", "computer usable program code", or "computer readable program code") may initially be located on a mass storage device in communication with the processor 110 through the system bus 122. The program code in different embodiments may be embodied on different physical or tangible computer readable media, such as the memory 112 or a mass storage device. In the illustrative example of FIG. 1, the instructions may be stored in a functional form of persistent storage on the direct access storage device 140. These instructions may then be loaded into the main memory 112 for execution by the processor 110. However, the program code may also be located in a functional form on a computer readable medium, such as a selectively removable, direct access storage device 140 or a read / writeable optical disk drive 142, in some embodiments. Such instructions may be loaded into or transferred to the DPS 100 for execution by the processor 110.

[0029] 1, the system bus 122 may be any device that facilitates communication between and among the processor 110, the main memory 112, and the interfaces 114, 116, 118, 120. Moreover, although the system bus 122 in this embodiment is a relatively simple single bus structure that provides a direct communication path between the system buses 122, other bus structures are consistent with this disclosure, including, but not limited to, point-to-point links, multiple hierarchical buses, parallel paths, redundant paths, and the like in a hierarchical, star, or web configuration.

[0030] The main memory 112 and the mass storage device 140 may cooperate to store the operating system 124, the application programs 126, and the program data 128. In some embodiments, the main memory 112 may be a random access semiconductor memory device ("RAM") capable of storing data and program instructions. While FIG. 1 conceptually depicts the main memory 112 as a single monolithic entity, the main memory 112 in some embodiments may be a more complex arrangement, such as a hierarchy of caches and other memory devices. For example, there may be multiple levels of caches in the main memory 112, and these caches may be further divided by functionality, such that one cache holds instructions while another holds non-instruction data used by the processor 110. The main memory 112 may be further distributed and associated with different processors 110 or sets of processors 110, as is known in any of a variety of so-called non-uniform memory access (NUMA) computer architectures. Additionally, some embodiments may utilize a virtual addressing mechanism that allows the DPS 100 to behave as if it has access to a single large storage entity, rather than access to multiple smaller storage entities (such as the main memory 112 and the mass storage device 140).

[0031] 1 as being contained within main memory 112 of DPS 100a, some or all of them may be located on a different physical computer system (e.g., DPS 100b) and, in some embodiments, may be accessed remotely, e.g., via network 106. Moreover, operating system 124, application programs 126, and program data 128 are not necessarily all contained entirely within the same physical DPS 100a at the same time, but may even reside in the physical or virtual memory of the other DPS 100b.

[0032] The system interfaces 114, 116, 118, 120 in some embodiments can support communication with a variety of storage and I / O devices. The mass storage interface 114 can support attachment of one or more mass storage devices 140, including rotating magnetic disk drive storage devices, solid state storage devices (SSDs) that use integrated circuit assemblies as memory, typically flash memory, to persistently store data, or a combination of the two. In addition, the mass storage device 140 can also include other devices and assemblies, including an array of disk drives (commonly referred to as a RAID array) configured to appear to a host as a single mass storage device, or hard disk drives, tape (e.g., miniDV), writable compact disks (e.g., CD-R and CD-RW), digital versatile disks (e.g., DVD, DVD-R, DVD+R, DVD+RW, DVD-RAM), holographic storage systems, blue laser disks, archival storage media such as IBM® Millipede devices, or combinations thereof. I / O interface 120 may support the attachment of one or more I / O devices, such as a keyboard, a mouse, a modem, or a printer (not shown).

[0033] The terminal / display interface 116 may be used to connect one or more displays 180 directly to the DPS 100. These displays 180 may be non-intelligent (i.e., dumb) terminals, such as LED monitors, or may themselves be fully programmable workstations, allowing IT administrators and users to communicate with the DPS 100. However, it should be noted that while the display interface 116 may be provided to support communication with one or more displays 180, the DPS 100 does not necessarily require a display 180, as all necessary interaction with users and other processes may occur over the network 106.

[0034] The network 106 may be any suitable network or combination of networks and may support any suitable protocol suitable for communicating data and / or code between the multiple DPSs 100. Thus, the network interface 118 may be any device that facilitates such communication, regardless of whether the network connection is made using current analog or digital or both technologies, or through some future network mechanism. Suitable networks 106 include, but are not limited to, networks implemented using one or more of the "InfiniBand" or IEEE (Institute of Electrical and Electronics Engineers) 802.3x "Ethernet®" specifications, cellular transmission networks, wireless networks implementing one of the IEEE 802.11x, IEEE 802.16, General Packet Radio Service ("GPRS"), Family Radio Service (FRS), or Bluetooth specifications, ultra-wideband ("UWB") technologies as described in FCC 02-48, and the like. Those skilled in the art will appreciate that many different networks and transmission protocols may be used to implement the network 106. The Transmission Control Protocol / Internet Protocol ("TCP / IP") suite includes appropriate network and transmission controllers.

[0035] Cloud Computing 2 illustrates one embodiment of a cloud environment suitable for implementing a cloud object storage system suitable for receiving replication requests from one or more DPSs, such as DPS 100. Although this disclosure includes detailed descriptions of cloud computing, it should be understood that practice of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0036] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computational resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) and for rapidly provisioning and releasing these resources with minimal administrative effort or interaction with the service provider. This cloud model includes at least five characteristics, at least three service models, and at least four deployment models.

[0037] The features are as follows: On-Demand Self-Service: Cloud customers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without the need for human interaction with the service provider. Broad network access: Cloud capabilities are available over the network and can be accessed using standard mechanisms, facilitating usage by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and PDAs). Resource Pool: The provider's computing resources are pooled and offered to multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Consumers typically have a sense of location independence in that they have no control or knowledge regarding the exact location of the resources offered to them, although at a higher level of abstraction they can specify a location (e.g., country, state, or data center). Rapid Elasticity: Capacity can be provisioned quickly and elastically, in some cases automatically, to scale out quickly, and released quickly to scale in quickly. Capacity available for provisioning often appears to the consumer as unlimited, available for purchase in any amount at any time. Metered Services: Cloud systems leverage metering to automatically control and optimize resource usage at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0038] The service model is as follows: SaaS (Software as a Service): The capability offered to the consumer is the use of the provider's applications running on a cloud infrastructure. Those applications can be accessed from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer has no management or control over the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application functionality, except for the possibility of limited user-specific application configuration settings. · PaaS (Platform as a Service): The capability provided to a consumer is to deploy applications that the consumer creates or acquires, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The capability provided to a consumer is the provisioning of processing, storage, network, and other basic computing resources over which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and in some cases, limited control over selected network components (e.g., host firewalls).

[0039] The deployment model is as follows: Private Cloud: This cloud infrastructure is operated solely for an organization. It can be managed by that organization or a third party and can exist on-premise or off-premise. Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organization or a third party and can exist on-premise or off-premise. Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services. Hybrid Cloud: The cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain a unique entity but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0040] A cloud computing environment is a service-oriented environment with an emphasis on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0041] Referring now to FIG. 2, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers (e.g., a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N, etc.). The nodes 10 can communicate with each other. They can be physically or virtually grouped in one or more networks (not shown), such as a private, community, public, or hybrid cloud, or combination thereof, as described herein above. This allows the cloud computing environment 50 to provide an infrastructure, platform, and / or SaaS that does not require the cloud consumer to maintain resources on a local computing device. The types of computing devices 54A-54N illustrated in FIG. 2 are intended as examples only, and it will be understood that computing node 10 and cloud computing environment 50 may communicate with any type of computer-controlled device via any type of network and / or network-addressable connection (e.g., a connection using a web browser).

[0042] Referring now to Figure 3, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 2) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 3 are intended to be merely exemplary, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0043] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, Reduced Instruction Set Computer (RISC) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0044] The virtualization layer 70 provides an abstraction layer over which the following examples of virtual entities may be provided: virtual servers 71 , virtual storage 72 , virtual networks including virtual private networks 73 , virtual applications and operating systems 74 , and virtual clients 75 .

[0045] As an example, the management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computational and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking and billing or invoices for the utilization of resources within the cloud computing environment. As an example, the resources may include application software licenses. Security provides identity verification of cloud users and tasks, as well as protection of data and other resources. User portal 83 provides access to the cloud computing environment to users and system administrators. Service level management 84 provides allocation and management of cloud computational resources to meet required service levels. Service level agreement (SLA) planning and fulfillment 85 provides pre-provisioning and procurement of cloud computational resources in anticipation of upcoming demands in accordance with SLAs.

[0046] The Workload Layer 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may come from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and metering services for storage and input / output (I / O) utilization 96.

[0047] Replication System FIG. 4 is a system diagram of a replication system 400 operating in SW mode, according to some embodiments. The replication system 400 may include a primary / cache cluster 410 and a target system 430. The primary / cache cluster 410 may include multiple application nodes 415A-415N and a gateway device 420. The gateway device 420 includes a replication queue 425. The queue 425 may be associated with the target system 430 via a remote mount from the gateway device 420. The target system 430 may include a single DPS 100, a storage cluster of DPSs 100, or a cloud object storage system (see FIGS. 2-3) in some embodiments.

[0048] During operation, an application 440 executing on an application node 415A-415N may issue multiple input / output (I / O) commands 445A-445N (only a portion of which are shown for clarity). These I / O commands 445A-445N may be added to a queue 425 as enqueued I / O commands 425A-425N. The gateway device 420 may utilize the queue 425 to replicate each of the enqueued I / O commands 425A-425N to the target system 430, respectively. The gateway device 420 may also utilize the queue 425 to replicate those enqueued I / O commands 4251-425N to the target system 430, respectively. N The replication status for each of the enqueued I / O commands 445A-445N may be tracked, which may include receiving an acknowledgment 455A-455N from the target system 430 upon completion of each enqueued I / O command 445A-445N.

[0049] FIG. 5 is a system diagram of a replication system 500 operating in MW mode, according to some embodiments. The replication system 500 may include a primary / cache cluster 510 and multiple target systems 530A-N. The primary / cache cluster 510 may include multiple application nodes 515A-N and a gateway device 520. The gateway device 520 includes multiple queues 525A-N. Each of the queues 525A-N may be associated with one of the multiple target systems 530A-N. For example, queue 525A may be associated with target 530A via a remote mount from the primary gateway device 520, queue 525B may be associated with target 530B via a remote mount from the primary gateway device 520, etc. Each of target systems 530A-530N may, in some embodiments, include a single DPS 100, a storage cluster of DPSs 100, a cloud object storage system (see Figures 2-3), or a combination thereof.

[0050] During operation, an application 540 executing on an application node 515A-515N may issue a number of input / output (I / O) commands 545A-545N, etc. (only some of which are shown for clarity). These I / O commands 545A, 545B, etc. may be added to each of a number of queues 525A-525N as enqueued commands I / O commands 545A1, 545A2, 545B1, 545B2, etc. (only some of which are labeled for clarity). The gateway device 520 utilizes these queues 525A-525N to send enqueued commands 550A1-550M to each of the respective targets 530A-530N. NThe gateway device 520 can also use these queues 525A-525N to track the replication status for each of its I / O commands 545A, 545B, etc., with respect to each of the targets individually, which can include receiving an acknowledgment 555A1, 555A2, 555B1, 555B2, etc. (only ACK 555 is shown for clarity) from each of the targets 530A-530N upon completion of replication.

[0051] More specifically, the embodiment of Figure 5 may include a separate cache record that may store the target path and associated configuration (e.g., internet address, connection port, communication protocol, etc.) as part of each fileset for each replication target 530A-530N. For example, if a primary / cache cluster 510 for one fileset is connected to three targets 530A-530C, the cache record for this fileset may be chained into three entries and referenced according to the target sites 530A-530N. This on-disk information may change as target sites 530A-530N are added or removed.

[0052] In some embodiments, the target systems 530A-530N may be processed in the order specified in the cache record (e.g., may be read as target1, target2, target3, ..., targetN). This order is used as a priority for replication to that target system. That is, the specified order may be used in part by the gateway device 520 for which operations 545 are prioritized for which target systems 530A-530N. For example, target1 530A may have the highest priority and targetN 530N may have the lowest priority in terms of the order and speed in which replication is performed. One feature and advantage of these embodiments is that it may be possible to ensure, for example, that target1 530A gets this data first (i.e., even before target2 530B) and continues through to targetN 530N. In this regard, it may help reduce or minimize inconsistencies in performing local updates regarding replication state to the local inodes of the files / objects being replicated.

[0053] The embodiment of FIG. 5 may use inode bits to track the replication state of each operation 545 in each of a number of queues 525A-N independently with respect to each associated target system 530A-N. These may be stored in a cache system as part of the local inode. In one embodiment, the bits on the inode that are set for replication may include the following: Create bit - Set when a file is created on the primary / cache cluster 510 and needs to be replicated to the associated target sites 530A-530N. The create bit can be reset once the file's create IO operation has been successfully replicated to the associated remote sites. Dirty Bit - Set when in-place data is written to a file in the primary / cache cluster 510, which needs to be replicated to the associated target sites 530A-530N. The dirty bit can be reset once the file write has been successfully replicated to the associated target sites 530A-530N. ·Append bit - Set when data is written beyond the last known offset on a file in the primary / cache cluster 510 and needs to be replicated to the associated target sites 530A-530N. The append bit can be reset once the file writes have been successfully replicated to the associated target sites 530A-530N. SetAttr bit - Set when there is an attribute change to a file in the primary / cache cluster 510 that needs to be replicated to the associated target sites 530A-530N. The setattr bit can be reset once the file attribute change has been successfully replicated to the target sites 530A-530N. ·State - A bit that is set when a file is created on the primary / cache cluster 510 and needs to be replicated to the target site 530A-530N. The state bit can return the remote attributes of the file's counterpart at the target site 530A-530N. When the state bit is set, it can indicate that the file is available at the target site 530A-530N and that the local counterpart has attributes related to its remote counterpart. When the file is renamed, removed, and / or recreated at the target site 530A-530N, the state bit can reset and then needs to be revalidated back to the primary / cache cluster 510. The state bit may not be used in some embodiments operating in DR mode because the backup site is typically passive and cannot receive updates directly from the application. Instead, the DR function may not require the state bit to be set when the file is first created (e.g., the create bit is on) and the state bit may be set when the file is replicated to the backup site. Some embodiments may not reset the state bit unless the file is removed or renamed. Cached bit - set when the file is fully available in the primary / cache cluster 510. Since all IO is performed in Disaster Recovery (DR) mode at the production site, the cached bit may not be used and / or reset in some DR setup embodiments. ·Local bit - set if the file update should remain local within the primary / cache cluster 510. Thus, some embodiments may not replicate the file update to the target sites 530A-530N. Additionally, this bit may be applicable in some embodiments only to certain special directories on the primary / cache cluster 510 that maintain local replication information and in certain modes of replication. · Reset - This inode bit may be used in some embodiments to represent a combination of which target sites 530A-530N an IO command has and has not been replicated to. Since the target sites 530A-530N may be sequenced from 1 to N, some embodiments may define an order in which the inode bits are appended, such that a simple logical AND operation with this reset bit reflects whether the inode operation represented by the bit was replicated to the target site 530A-530N. The order number in some embodiments may be determined using a power of 2(N-1), where N is the target in question. For example, the order in which the inode replication status bits are appended may be 1 for target1, 2 for target2, 4 for target3, 8 for target4, and so on, or more generally, 2 for targetN.

[0054] In the single remote site case described with reference to Figure 4, the reset inode bit can take a binary form, for example, a value of 1 indicates that replication is pending and a value of 0 indicates that replication is complete. In the multi-target replication case described with reference to Figure 5, the reset inode bit can go through various values. For example, in a simple scenario of a cache replicating to three target sites (target1, target2, and target3), the inode reset bit can take on values ​​from 0 to 7 (i.e., 2^3-1). In this example, the reset bit values ​​can represent the following states: A reset inode value of 0 may indicate that the inode operation in question, as represented by the inode bits (e.g. create / dirty / setattr), has been replicated to all remote sites in question. A reset inode value of 1 may represent that the inode is pending replication for target1 and completed for target2 and target3. A reset inode value of 2 may represent that the inode is pending replication to target2 and completed to target1 and target3. A reset inode value of 3 may represent that the inode is pending replication to both target1 and target2, but completed to target3. A reset inode value of 4 may represent that the inode is pending replication to target3 and completed to target1 and target2. A reset inode value of 5 could represent that the inode is pending replication to both target1 and target3, but completed to target2. A reset inode value of 6 could represent that the inode is pending replication to both target2 and target3, but completed to target1. A reset inode value of 7 may represent that the inode is pending replication to all three targets, namely target1, target2, and target3. In the single replication scenario described with reference to Figure 4, each inode bit used for allocation takes up only one byte on the inode. In the multi-target replication scenario described with reference to Figure 5, each of the seven inode bits may take seven bytes on the inode to store (i.e., because they are Boolean in nature).

[0055] To improve scaling in applications with many target sites 530A-N, the reset bit in some embodiments may be an Int8 or Int16 value, either of which occupies 4 bytes on the inode for each replication. For even larger applications, a 32-bit Int may be used. Advantageously, even a 32-bit Int in such an embodiment would only require 4 bytes on the inode, so in this case all seven replication bits can take up to 28 bytes (compared to 7 bytes in the single target case), which can easily fit into the inode size specified by most current file systems.

[0056] Some embodiments may further include a replication data structure (also called a handler). In case of multi-site replication, the handler may consist of two separate components: (i) a fixed component, which may hold all common settings (e.g., config values) for the local cache sites. These may remain common to all target sites 530A-530N, and thus one copy in memory may be used for replication on the gateway device 520. This may start from the configured method to reach the target sites 530A-530N (e.g., http / https protocol part, e.g., in case of Cisco Cloud Object Storage (COS) backend, COS server address and bucket name at the target site, in case of NFS backend, NFS server name, IP address, port number, and remote exportPath name, in case of native file system backend, unique mount path, etc.); and (ii) a queue per target site 530A-530N to track / manage the replication. The replication structure may include a unique copy of the queues stored in the memory of the gateway device 520 for each target site 530A-530N, and as one operation completes for each target site 530A-530N, the handler may simply dequeue the relevant operation from the queue associated with that particular remote site, independent of the queues for other remote sites. Thus, for example, when one queue is dequeued, other running queues are not affected. Thus, one feature and advantage of these embodiments is that, since the target sites may be composed of different backends, not all target sites 530A-530N will behave the same, and thus one queue may be faster to flush while other queues may take longer to flush. Similarly, some remote sites may report replication errors and even have one of the queues become stale.

[0057] In some embodiments, the handlers on the gateway device 520 for a given file system / fileset that replicates to multiple remote sites may have the following characteristics: (i) a parent handler may hold all common configuration settings for a given replication system for all remote sites involved, so that any one of these configuration changes may be applied to all N remote replication sites with which the system may interact; and (ii) multiple child handlers, each of which may depend on the target site 530A-N to which it replicates. These child handlers may have configuration that applies only to the particular target site 530A-N to which it replicates, and the queue of operations that need to be replicated to the associated target site 530A-N.

[0058] Additionally, some embodiments may enable quality of service (QoS) based resource allocation per queue based on a weighting calculated for each target site (where the weighting may be calculated based on the cost and resource sharing factors of the target sites 530A-530N and the waiting tolerance of applications resident at secondary / cloud sites).

[0059] 6 is a flow chart illustrating a multi-protocol, multi-site replication method 600 according to some embodiments. In operation 605, the system may be configured to replicate data from a production site / cluster to multiple backends. Some of the backends may be managed by a first cloud vendor and some may be managed by a second, different cloud vendor. Cloud vendor connection information may be stored in a file system in some embodiments.

[0060] In operation 610, a plurality of IO operations are performed on a file system of a production site / cluster. A gateway device associated with the production site / cluster then captures the IO operations in operation 620. In response, the gateway device may enqueue the captured IO operations into a plurality of queues in operation 630. Each of the plurality of queues may be associated with one of the target sites. In operation 640, the gateway device may independently manage replication of the IO operations to the target sites using the plurality of queues stored in memory. In some embodiments, the gateway device may also independently track a replication status of each captured IO operation using inode bits of its local file system for each target site in operation 650. In some embodiments, one of the inode bits may encode a plurality of replication states for the reach of the remote site, including whether replication is pending and whether replication is completed. The number of inode bits may be based on the number of target sites.

[0061] Computer Program Products The present invention may be a system, method, and / or computer program product, in any possible level of technical detail integration. The computer program product may include a computer readable storage medium (or multiple computer readable storage media) having computer readable program instructions for causing a processor to implement aspects of the present invention.

[0062] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved structures on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0063] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0064] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0065] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0066] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium having instructions stored thereon, directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions for implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0067] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operable steps to create a computer-implemented process such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0068] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowcharts or block diagrams may represent a subsystem, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be performed as one step, may be performed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, or the blocks may sometimes be performed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a special-purpose hardware-based system that performs the specified functions or actions or executes a combination of special-purpose hardware and computer instructions.

[0069] Overview The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many changes and modifications will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification have been selected to explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable other skilled in the art to understand the embodiments disclosed herein. The presently described embodiments are therefore to be considered in all respects as illustrative and not restrictive, and reference should be made to the appended claims for determining the scope of the invention.

Claims

1. 1. A method for multi-site replication from a single production site, comprising: capturing, in a gateway device, IO operations performed on a file system, the file system having an inode bit; enqueuing the captured IO operations into a plurality of queues, each of the plurality of queues being associated with one of a plurality of target sites; independently managing replication of said IO operations to said target site using said inode bits; A method comprising:

2. 2. The method of claim 1, wherein the number of inode bits is based on a number of target sites.

3. 2. The method of claim 1, wherein one of the inode bits encodes a plurality of replication states for each of the remote sites.

4. The method of claim 3 , wherein the replication state is selected from the group consisting of: replication pending and replication completed.

5. 2. The method of claim 1 , wherein a first target site of the plurality of target sites includes a first backend and a second target site of the plurality of target sites includes a second backend, the second backend being different from the first backend.

6. The method of claim 1 , wherein managing the replication of the IO operations includes maintaining parent and child handlers on the gateway device.

7. The method of claim 1 , wherein each of the multiple queues is responsible for replicating data to a target cloud.

8. Metering replicated IO operations; generating a bill based on said metered usage; and The method of claim 1 further comprising:

9. 1. A system for multi-site replication from a single production site, the system comprising: one or more processors; a memory communicatively coupled to the one or more processors; Equipped with The memory, when executed by the one or more processors, causes the one or more processors to: capturing, in a gateway device, IO operations performed on a file system, the file system having an inode bit; enqueuing the captured IO operations into a plurality of queues, each of the plurality of queues being associated with one of a plurality of target sites; independently managing replication of said IO operations to said target site using said inode bits; A system comprising instructions for carrying out a method comprising:

10. 10. The system of claim 9, wherein the number of inode bits is based on a number of target sites.

11. 10. The system of claim 9, wherein one of the inode bits encodes a plurality of replication states for each of the remote sites.

12. The system of claim 11 , wherein the replication state is selected from the group consisting of: replication pending and replication completed.

13. 10. The system of claim 9, wherein a first target site of the plurality of target sites includes a first backend and a second target site of the plurality of target sites includes a second backend, the second backend being different from the first backend.

14. 10. The system of claim 9, wherein managing the replication of the IO operations includes maintaining parent and child handlers on the gateway device.

15. The system of claim 9 , wherein each of the plurality of queues is responsible for replicating data to a target cloud.

16. The method further comprising: Metering replicated IO operations; generating a bill based on said metered usage; and The system of claim 9 further comprising:

17. 1. A computer program product for multi-site replication from a single production site, the computer program product including a computer readable storage medium having program instructions embodied therein, the program instructions executable by a computer comprising: In a gateway device, capturing IO operations performed on a file system, the file system having an inode bit; the number of inode bits is based on a number of target sites; said capturing, one of said inode bits encoding a plurality of replication states for each of said remote sites; enqueuing the captured IO operations into a plurality of queues, each of the plurality of queues being associated with one of the target sites; independently managing replication of said IO operations to said target site using said inode bits; 23. A computer program product executable to perform a method comprising:

18. 20. The computer program product of claim 17, wherein a first one of the target sites includes a first backend and a second one of the target sites includes a second backend, the second backend being different from the first backend.

19. 20. The computer program product of claim 17, wherein managing the replication of the IO operations comprises maintaining parent and child handlers on the gateway device.

20. The method further comprising: Metering replicated IO operations; generating a bill based on said metered usage; and 20. The computer program product of claim 17, further comprising:

Images