Hybrid cloud asynchronous data synchronization

Abstract

In some examples, a computing device can communicate with multiple network storage systems, such as a first network storage system provided by a first service provider employing a first storage protocol and a second network storage system provided by a second service provider employing a second storage protocol different from the first storage protocol. The computing device receives a first object and determines, for the first object, a first remote bucket at the first network storage system and a second remote bucket at the second network storage system. The computing device can add a synchronization event to a queue for copying the first object to the first remote bucket and the second remote bucket. Based on consuming the synchronization event from the queue, the computing device copies data and metadata of the first object to the first remote bucket and the second remote bucket.

Description

Technical Field

[0001] This disclosure relates to the technical field of data storage. Background Technology

[0002] Synchronizing data across hybrid cloud systems is a recent concept that offers storage users cost management flexibility, for example, by supporting backup and data movement across various storage services in the cloud. In a hybrid cloud system, for instance, network storage products from various service providers can be used to store and synchronize data. However, synchronizing data across multiple heterogeneous systems can be challenging because the software stack used in each of these systems can be configured and controlled by different entities, and replication-specific changes may not necessarily work as expected across all the different systems. Summary of the Invention

[0003] Some implementations include a computing device capable of communicating with multiple network storage systems via a network. For example, a first network storage system may be provided by a first service provider employing a first storage protocol, and a second network storage system may be provided by a second service provider employing a second storage protocol different from the first storage protocol. The computing device can receive a first object and can determine a first remote bucket at the first network storage system and a second remote bucket at the second network storage system for the first object. The computing device can add synchronization events to a queue for copying the first object to the first and second remote buckets. Based on consuming the synchronization events from the queue, the computing device copies the data and metadata of the first object to the first and second remote buckets. Attached Figure Description

[0004] Detailed description will be provided with reference to the accompanying drawings. In the drawings, the leftmost numeral of the reference numeral indicates the drawing in which the reference numeral first appears. The use of the same reference numerals in different drawings indicates similar or identical terms or features.

[0005] Figure 1 An example architecture of a system, based on some implementations, is shown that can store data and metadata across multiple non-homogeneous storage systems.

[0006] Figure 2 This is a block diagram illustrating an example logical configuration for performing synchronization in the system, based on some implementations.

[0007] Figure 3 This is a block diagram illustrating an example of performing a synchronous-to-synchronous operation in a system, based on some implementations.

[0008] Figure 4 This is a diagram showing an example of pseudocode based on some implementations.

[0009] Figure 5 Figure 1 is a diagram illustrating an example of pseudo code according to some implementations.

[0010] Figure 6 Figure 2 illustrates an example data structure for a local system object version according to some implementations.

[0011] Figure 7 Figure 3 illustrates an example data structure for a networked storage system object version according to some implementations.

[0012] Figure 8 Figure 4 illustrates an example of versioned replication according to some implementations.

[0013] Figure 9 Figure 5 illustrates an example of versioned replication according to some implementations.

[0014] Figure 10 Figure 6 illustrates an example of applying filters to bucket replication according to some implementations.

[0015] Figure 11 Figure 7 illustrates an example of applying filters to bucket replication according to some implementations.

[0016] Figure 12 Figure 8 illustrates a flow diagram of synchronization to a process according to some implementations.

[0017] Figure 13 Figure 9 illustrates a flow diagram of from a process to synchronization according to some implementations.

[0018] Figure 14 Figure 10 illustrates a selected example set of components of a service computing device that can be used to implement at least some of the functionality of the systems described herein. DETAILED DESCRIPTION

[0019] Some implementations herein relate to techniques and structures for distributed computer systems that include hybrid cloud infrastructures that are capable of utilizing multiple differently configured networked storage systems as targets for synchronizing data, e.g., for backing up data, replicating data, retrieving data, etc. For example, multiple heterogeneous networked storage systems enable users to utilize a greater range of services provided by multiple different cloud storage providers. Implementations herein can use mechanisms to seamlessly synchronize user data and provide synchronization grouping controls to ensure data safety to support data synchronization targets across heterogeneous multiple storage service providers. Thus, implementations herein can support one-to-many synchronization of user data across heterogeneous networked based storage systems that can include commercial cloud storage, proprietary cloud storage, private wide area network storage, local storage, and any combination thereof.

[0020] Additionally, to enable one-to-many heterogeneous synchronization, some examples include using one or more open standard application programming interfaces (APIs), such as an S3-based protocol API for replicating data, which can provide a basic replication or data replication mechanism. However, conventional APIs fall short in terms of efficiency and accuracy of replication. For example, conventional APIs can not provide accurate metadata copies, which are desirable in addition to accurate data copies for a number of data management purposes. In addition to using basic data replication, some examples herein use data versioning and specific event triggers to help ensure accurate metadata updates to maintain the accuracy of metadata along with related data. Additionally, implementations here can include a notification mechanism for buckets or other containers of remote ends. Moreover, in some cases, solution capabilities can be extended by leveraging different services on all systems to provide an efficient and accurate data management solution.

[0021] Some examples herein can support a hybrid architecture that includes a proprietary local storage system in communication with multiple remote public and / or private cloud storage systems. Moreover, implementations herein can provide novel approaches for performing multiway replication across hybrid cloud systems running a variety of different software stacks from a variety of different providers.

[0022] To enable synchronization across multiple different provider systems that do not run the same software stack, some examples herein ensure that all communications are performed using open standard APIs. Additionally, information exchange can follow open standard messaging protocols. Moreover, in some cases, multiple different systems can be individually authenticated since not all types of network storage systems allow for federation of identity management for all various types of systems. Additionally, some examples herein can limit the amount or frequency of updates to remote systems and can reduce content transfer for optimizing performance.

[0023] Additionally, different network storage systems can operate at different speeds to receive, store, and / or provide data with variable response times. Accordingly, some examples herein can provide very fine-grained permission levels or roles to control communications across various network storage systems, such as providing system administrators with flexible control of access to multiple corporate network domains.

[0024] For purposes of discussion, some example implementations are described in the context of one or more service computing devices in communication with multiple different network storage systems from different storage providers, for managing storage of data and metadata. However, implementations herein are not limited to the particular examples provided, and can extend to other types of computing system architectures, other types of storage environments, other storage providers, other types of client configurations, other types of data, and so forth, as will be apparent to those of skill in the art in light of the disclosure herein.

[0025] Figure 1 An example architecture of a system 100 that is capable of storing data and metadata to multiple non-homogenous storage systems is shown, in accordance with some implementations. The system 100 includes multiple service computing devices 102 that are capable of communicating with or otherwise being coupled to multiple network storage systems, such as a first provider network storage system 104(1), a second provider network storage system 104(2), a third provider network storage system 104(3), a fourth provider network storage system 104(4), and so forth, e.g., via one or more networks 106. As noted above, each provider of network storage systems can be a different entity from the other providers. Examples of commercial network storage providers include Amazon Web Services, Microsoft AZURE, Google Cloud, IBM Cloud, and Oracle Cloud, among others. The network storage systems 104(1) through 104(4) can be referred to as “cloud storage” or “cloud-based storage” in some examples, and can implement a lower cost per gigabyte storage solution than local storage available at the service computing devices 102 in some cases. Additionally, in some examples, a storage provider can be a private or otherwise proprietary storage provider, such as for providing access only to a particular user, entity, and so forth associated with the service computing devices 102, for example. An example of a proprietary system can include a configuration of the Hitachi Content Platform.

[0026] Further, the service computing devices 102 are capable of communicating with one or more user computing devices 108 and one or more administrator computing devices 110 via the networks 106. The user devices 108 and administrator devices 110 can be any of various types of computing devices, as discussed further below.

[0027] In some examples, the service computing devices 102 can include one or more servers that can be implemented in any number of ways. For example, at least a portion of the programs, other functional components, and data stores of the service computing devices 102 can be implemented on at least one server, such as in a server cluster, server farm, data center, cloud-hosted computing service, and so forth, although other computer architectures can additionally or alternatively be used. Additional details of the service computing devices 102 are discussed below with reference to Figure 14 the example architecture of FIG. 1.

[0028] The one or more networks 106 can include any suitable network, including a wide area network such as the Internet; a local area network (LAN) such as an intranet; a wireless network such as a cellular network, a local wireless network such as Wi-Fi, and / or a short-range wireless communication such as Bluetooth®

[0029] Additionally, the service computing devices 102 can be capable of communicating with each other over one or more networks 107. In some cases, the one or more networks 107 can be a LAN, a private network, or the like, while in other cases, the one or more networks 107 can include any of the networks 106 discussed above.

[0030] The service computing devices 102 can be configured to provide storage and data management services to users 112 via the user devices 110, respectively. As a number of non-limiting examples, the users 112 can include users performing functions for an enterprise, a business, an organization, a government entity, an academic entity, or the like, and in some examples, the users 112 can include storing very large amounts of data. However, implementations herein are not limited to any particular use or application of the system 100 and other systems and structures described herein.

[0031] Each of the user devices 108 can be any suitable type of computing device, such as a desktop, a laptop, a tablet computing device, a mobile device, a smartphone, a wearable device, a terminal, and / or any other type of computing device capable of sending data over a network. The users 112 can be associated with the user devices 108, for example, through respective user accounts, user login credentials, or the like. Further, the user devices 108 can be capable of communicating with the service computing devices 102 over the one or more networks 106, over separate networks, or over any other suitable type of communication connection. Those of skill in the art, having the benefit of the disclosure herein, will appreciate a number of other variations.

[0032] ​Further, each user device 108 can include a respective instance of a user application 114 executable on the user device 108, e.g., for communicating with a user web application 116 executable on the service computing device 102, such as for sending user data for storage on the network storage system 104 and / or for receiving stored data from the network storage system 104 through data instructions 118, etc. In some cases, the application 114 can comprise a browser or can operate through a browser, while in other cases, the application 114 can comprise any other type of application having a communication functionality that enables communication with the user web application 116 through the one or more networks 106.

[0033] In some examples of the system 100, the users 112 can store data to and receive data from the service computing device 102 with which their respective user devices 108 communicate. Thus, the service computing device 102 can provide storage for the users 112 and the respective user devices 108. During steady state operation, there can be periodic communication of the users 108 with the service computing device 102.

[0034] Additionally, the administrator devices 110 can be any suitable type of computing device, such as a desktop, a laptop, a tablet computing device, a mobile device, a smartphone, a wearable device, a terminal, and / or any other type of computing device capable of sending data through a network. The administrators 120 can be associated with the administrator devices 110, e.g., through respective administrator accounts, administrator login credentials, etc. Further, the administrator devices 110 can be capable of communicating with the service computing device 102 through the one or more networks 106, 107, through separate networks, or through any other suitable type of communication connection.

[0035] Further, each administrator device 110 can include a respective instance of an administrator application 122 executable on the administrator device 110, such as for communicating with an administration web application 124 executable on the service computing device 102, such as for sending administration instructions for managing the system 100, and for sending administration data for storage on the network storage system 104 and / or for receiving stored administration data from the network storage system 104, e.g., through administration instructions 126, etc. In some cases, the administrator application 122 can comprise a browser or can operate through a browser, while in other cases, the administrator application 122 can comprise any other type of application having a communication functionality that enables communication with the administration web application 124 through the one or more networks 106.

[0036] The service computing device 102 can execute a storage program 130 that can provide a gateway to the network storage system 104, such as for sending data to be stored to the network storage system 104 and for retrieving requested data from the network storage system 104 or from local storage 132. In addition, the storage program 130 can manage data stored by the system 100, such as for managing data retention periods, data protection levels, data replication, etc.

[0037] The service computing device 102 can also include a metadata database (DB) 134 that can be divided into multiple metadata DB portions, e.g., partitions, and can be distributed across multiple service computing devices 102. For example, the metadata DB 134 can be used to manage object data 136 stored at the network storage system 104 and local object data 137 stored at the local storage 132. The metadata DB 134 can include a number of metadata about the object data 136, such as information about individual objects, how to access individual objects, storage protection levels of objects, storage retention periods, object owner information, object sizes, object types, etc. In addition, a metadata gateway program 138 can manage and maintain the metadata DB 134, such as for updating the metadata DB 134 when new objects are stored, old objects are deleted, objects are migrated, etc., and in response to requests to access data.

[0038] In addition, the service computing device 102 can include a policy engine program 140 that can be executed to perform data synchronization with the network storage systems 104(1) through 104(4). In some cases, the policy engine program 140 can be a module of the storage program 130, while in other cases, the policy engine program 140 can be a separate program. In some cases, the policy engine program 140 can operate in cooperation with a message queuing program 142 for synchronizing the object data 136 with the network storage systems 104. Examples of message queuing programs that can be used in some examples herein include open source multi-protocol message brokers, such as RABBITMQ and ACTIVEMQ.

[0039] In addition, the policy engine program 140 can synchronize the object data 136 asynchronously with the network storage systems 104. For example, the storage program 130 can store the object data as local object data 137 to the local storage 132, can add metadata for the object to the local metadata database 132, and can reply to the requesting device with a completion message, etc., rather than having to perform certain storage operations on the network storage systems 104 and complete those operations before reporting to the user. The storage program 130 can then send the data to a queue for later synchronization of the data with one or more network storage systems 104 via one or more data synchronization operations 144.

[0040] The implementations herein can divide the synchronization process into two parts, including a sync-to process that synchronizes data to the network storage system 104 and a sync-from process that synchronizes data back from the network storage system 104. This approach has several advantages, such as providing a large amount of flexibility for operators (e.g., users 112 and / or administrators 120) to control the level of replication, and further providing the ability to prevent the formation of loops. For example, the system can provide validation within the system to validate that, for example, the operators have not created a loop or topology at setup that forms a loop that causes infinite replication, which would then require additional intervention to prevent system instability.

[0041] In the system 100, event-driven synchronization is employed, guided by the policy engine program 140, which is configured to apply actions such as rate control limits and / or quality of service management to ensure that the speed of data synchronization can be managed by the administrator. Further, some examples can employ versioning of data, for example, each update can produce a different version, which can then help provide ordering of object data 136 on the network storage system 104. For example, each network storage system 104 can utilize a different versioning scheme, so it can be difficult to maintain ordering of object data 136 using provider tools. However, providing versioning capabilities via the policy engine program 140 enables operators / applications to compare and eliminate version conflicts by manually deleting certain versions or then updating versions with the correct content.

[0042] Further, in some examples, user metadata for each version of a data object can be synchronized with the object data, which can enable applications to add any specific information needed for further data management. As one example, this feature allows for maintaining an audit trail of the application / operator that generated the data. For example, metadata tags can include information about the originating source or originating application, as well as the time at which the application created the data, which can then be used with the data version to enable reordering of versions based on the timestamp included in the metadata for that object.

[0043] Additionally, some examples herein allow for the setting of bucket-level filters, which can be applied to, for example, implement synchronization of only a subset of the stored data, such as can be specified by an operator. Synchronization herein can also work without filters, but filters can provide an additional level of flexibility to limit the amount of data that is replicated.

[0044] In some cases, the service computing devices 102 can be arranged in one or more groups, clusters, systems, etc., such as at a site 150. Among other things, multiple sites 150 can be geographically dispersed from one another, such as for providing data replication, disaster recovery protection, etc. Moreover, in some cases, the service computing devices 102 at multiple different sites 150 can be configured for securely communicating with one another, such as for providing a federation of the multiple sites 150.

[0045] Figure 2 is a block diagram illustrating an example logical configuration for performing synchronization in the system 100, according to some implementations. In some examples, the system 100 can correspond to the system 100 discussed above, or any of a variety of other possible computing system architectures, as will be apparent to those of skill in the art having the benefit of the disclosure herein. The system 100 can implement distributed object storage, and can include using a web application as a front end for users and administrators. In some cases, the system 100 can store objects on the network storage system 104 in buckets that can be created by users and / or administrators. The system 100 can use resources distributed across local and cloud systems to enable complex management and storage of data.

[0046] In implementations herein, the synchronization process can be performed using two steps (e.g., synchronization to and from the network storage system 104). For example, dividing the data synchronization process into a two-step process based on direction enables implementations herein to provide multi-synchronization and multi-target support, and can also provide the ability to configure combinations of these steps. Moreover, some implementations herein include a two-level asynchronous event-driven execution framework for multi-target replication to multiple network storage systems 104.

[0047] Figure 2 Elements of an example of synchronizing data to the network storage system 104 are illustrated, and include an overview of the data replication flow as part of synchronization to direction. Some examples herein can be based at least in part on executing microservices that run in containers that can be used as independent computing engines. For example, multiple instances or containers of each service (e.g., the storage program 130, the metadata gateway program 138, and / or the policy engine program 140) can be executed across a distributed cluster (e.g., in multiple service computing devices 102). Multiple microservice instances can enable scaling of the policy engine across a distributed cluster or other distributed computing architecture, such as can be needed due to different application workloads.

[0048] As an example, when data is ingested from an application (e.g., via user web application 116 or management web application 124), two synchronous actions are performed in real time. For example, suppose a user has saved object A 202 via user web application 116. First, as indicated at 204, object metadata 206 can be created for object A. Object metadata 206 includes information such as the object name, version ID, timestamp, object size, user metadata tag, access control list, etc., entered by the application. Additionally, as shown at 208, also synchronously, object data 210 of object A is stored in local storage 132 as part of local object data 137, and the stored object locator ID is added to other system metadata as part of object metadata 206. Metadata gateway program 138 can add object A metadata 206 to metadata database 134. Figure 2 (Not shown in the image). Because these two actions are performed in real time, the stored procedure 130 can return a response to the user indicating that the storage of object A202 has been successfully completed.

[0049] As shown in 212, the metadata gateway 138 can then asynchronously generate synchronization events for object A 202 and can add these synchronization events to one or more queues 214 provided by the message queuing framework 216 described above, 142. For example, similar to the rest of the microservices described herein, queues 214 can scale up or down independently of other services. For example, as provided by the message queuing framework 216, queues 214 can be stateful and persistent. For example, if a queue 214 becomes corrupted or becomes unusable, one or more configurable redundant copies of the events in queue 214 can be maintained across at least one other queue 214 for recovery in case of failure. The event-driven message queuing framework 216 reduces traffic on the metadata gateway 138, which is executed to perform metadata storage and retrieval, so that the metadata gateway 138 is not overwhelmed by synchronous queries. Furthermore, the queuing framework helps isolate mirrored events from all other synchronous events in the system.

[0050] The policy engine program 140 can be executed to provide a set of containerized services that operate in the background to poll and consume events from the queues 214 for each synchronization target, as indicated at 220. For example, the policy engine program 140 can asynchronously pull synchronization events from the queues 214 and can synchronize events in the queues 214 to the respective network storage systems 104. For example, each synchronization event can have a separate target based on a particular one of the network storage systems 104 and a named bucket therein. Thus, based on techniques discussed further below, the policy engine program 140 can synchronize the object A (including, in some cases, metadata for the object A, as indicated at 222) to the intended synchronization targets, which in this example can include the network storage systems 104(1) through 104(4), each of which can be provided by a different service provider with different storage configurations, protocols, etc.

[0051] In some implementations, the outgoing synchronization events (i.e., sync-to events) can be segregated into one or more outgoing queues, and the synchronization events for retrieving data from the network storage systems 104 (i.e., sync-from events) can be segregated into one or more synchronization queues. Isolating the communication by direction using separate queues 214 can help ensure that if one of the network storage systems 104 fails or if there are network connectivity issues, the communication with other ones of the network storage systems 104 that are functioning properly is not affected. Given that the network storage systems 104 are provided by multiple different service providers, the system 100 can not be able to easily distinguish between a temporary outage or a complete outage of a network storage system 104. Also, it can not be practical to rely on a particular network storage system 104 to provide an accurate cause of an error or failure, which can be due to, for example, resource or quota limitations, a server being under maintenance, or a bug in the system.

[0052] Additionally, some examples herein can include isolating synchronization events based on categories. For example, the queuing framework 216 can include a separate set of queues 214 for each separate remote target domain (e.g., sync-to queues and sync-from queues), e.g., the queuing framework 216 can include a first set of queues 214 for the first provider network storage system 104(1), a second set of queues 214 for the second provider network storage system 104(2), a third set of queues 214 for the third provider network storage system 104(3), etc. Thus, the policy engine program 140 can perform one-to-many synchronization of data from the service computing device 102 to the network storage systems 104.

[0053] Further, in some cases, the policy engine program 140 can apply one or more filters to the synchronization events in the queues 214. For example, the policy engine program can be configured by a user to apply user-specified rules, which can include filtering certain objects based on object tags associated with the objects. Based on these user-specified rules, the policy engine program 140 can consume a synchronization event from one of the queues 214, compare any metadata tags included in the data targeted by the synchronization event, and if the metadata tags match the specified rules, the policy engine program 140 can discard the synchronization event, rather than the synchronization event being performed.

[0054] Further, in some examples, when the policy engine program 140 consumes a synchronization event from one of the queues 214, the policy engine program can first determine which buckets on which network storage systems 104 are targeted by the synchronization event. After determining the target buckets and target network storage systems 104, the policy engine program 140 can access authentication credentials in order to make access calls to the target network storage systems 104. After authentication, the policy engine program 140 can access the target network storage systems 104, for example, using provider-specific protocols and processes corresponding to the target network storage systems 104.

[0055] Figure 3 is a block diagram illustrating an example of performing synchronization-from to synchronization-to in the system 100, according to some implementations. For example, a synchronization event can include retrieving data from network storage systems 104 hosted by various different service providers. Configuring synchronization-from can allow for recovery of remotely stored data to the system 100, for example, to the local storage 132 and / or to a client device, such as the user device 108 or the administrator device 110. In the system 100, the policy engine program 140 can act as a set of consumers that process synchronization events by pulling events from the queuing framework 216. One reason for processing synchronization-from events separately from synchronization-to events is to support network storage systems 104 that do not provide the same type of consumption services as other network storage systems 104 in the network storage systems 104. For example, some network storage systems 104, such as the second provider network storage system 104(2) in the illustrated example, can provide a publisher mechanism 302 such that when an object is placed in a bucket in the second provider network storage system 104(2), the object can include a set of producers that can provide a notification of the change to the data bucket.

[0056] Other network storage systems 104 can provide a queuing notification system where new changes are sent to a queue and can be queried through the queue. In addition, other network storage systems 104 can incorporate a completely different approach and can not provide any notification or event consumption mechanism. Thus, the implementations herein are able to replicate back from any type of network storage system 104, whether or not there are event notifications, or whether or not there are any of a variety of different techniques for different notification models.

[0057] In some examples, the system can proactively look for new data on the network storage system 104 to perform a replication back operation. Thus, data on the network storage system 104 can be replicated back to the local system 100 using the replication back functionality herein, such as by directly accessing a queue provided by some network storage systems, or alternatively, based on received messages that can be added to a local replication back queue provided by the queuing framework 216.

[0058] The event-driven queuing and notification consumption framework discussed above can also be used in this example with the policy engine program 140 that pulls synchronization events out of the queue 214 in the queuing framework 216. For example, the internal queuing framework 216 can be used to abstract away remote notification differences. In cases where the network storage system 104 provides its own queuing framework, some of the implementations herein can use the queuing framework provided directly by the service provider to avoid introducing any additional latency that can be added by an additional local queuing framework 216. In cases where the remote system does not provide any notification services, some of the examples herein can use a thread that lists remote user data entries from a data bucket and creates entries in a producer queue.

[0059] In the illustrated example, assume that the first provider network storage system 104(1) is Amazon Web Services (AWS), and that the provider includes an AWS queuing service 304 (also known as "Simple Queue Service" (SQS)) that provides a queue 306. The queue 306 can be a scalable hosted queue for storing messages as they travel between computers, such as for moving data between distributed components of a system without losing messages or requiring each component to always be available. In addition, in the illustrated example, assume that the fourth provider network storage system 104(4) is a Microsoft WINDOWS AZURE system that provides a WINDOWS AZURE queuing service (WAQS) 308 that provides a queue 310 that can provide reliable, durable messaging between applications, etc. For example, the queue 310 can be used to transfer messages between applications or services.

[0060] As noted above, in examples herein, the policy engine program 140 can use the queues 306 and 310 directly to retrieve data from the network storage systems 104(1) and 104(4), respectively. For example, as indicated at 312, the policy engine program 140 can poll and consume directly from the queues 306 and 310. On the other hand, with respect to the second provider network storage system 104(2), as indicated at 314, when that system uses the publisher mechanism 302, the policy engine program 140 polls and consumes from the queue 214 in the queuing framework 216, which includes a synchronization event to enable the policy engine program 142 to consume the published data published by the publisher mechanism 302, as indicated at 316. Similarly, with respect to the third provider network storage system 104(3), assuming that storage system includes metadata, and as indicated at 318, the policy engine program 140 polls and consumes from the queue 214 in the queuing framework 216 to read the metadata, as indicated at 320.

[0061] The data received by the policy engine 140 can be stored to local storage 132, as indicated at 324. Further, as indicated at 326, the metadata gateway program 138 can update metadata in the metadata database. Thus, the synchronization operation can be one-to-one or many-to-one. Any objects added to a bucket on one of the network storage systems 104 can be retrieved according to implementations herein.

[0062] Figure 4 FIG. 4 is a diagram illustrating an example of pseudo code 400 according to some implementations. In some examples herein, an administrator in the system 100 can be able to configure replication settings and other synchronization settings in the policy engine program 140. For example, the administrator can enable or disable a remote replication target and set security measures, such as by allowing the communication to flow through a proxy host, so that the organization can apply appropriate security measures at the proxy host before the data leaves the local system 100.

[0063] In this example, as indicated at 402, the administrator can also limit the destinations to which data is being replicated. Information such as provider type can be stored to enable an understanding of the possible types of communication with each service provider, such as whether notifications can be targeted to that destination or whether the network storage system 104 of a particular provider allows any additional metadata requests, etc.

[0064] Figure 5 FIG. 5 is a diagram illustrating an example of pseudo code 500 according to some implementations. In this example, the synchronization rules of the policy engine program 140 can be established by an application user ingesting data into the system. For example, as indicated at 502, the name of a bucket and a service provider can be specified for synchronization from. Further, as indicated at 504, the name of a bucket and a service provider can also be specified for synchronization to.

[0065] Figure 6 An example data structure 600 for local system object versions is shown, in accordance with some implementations. For example, a user can enable object versioning when defining sync-to-copy rules or from sync-copy rules to help manage conflicts between source and destination targets. Other advantages of object versioning can include enabling single instantiation of data or support for limited copies when content is unchanged. In this example, the data structure 600 includes information for a local system object version, including a version ID 602, a content hash 604, a size of the object 606, a replication status of the object 608, and an update time 610. The content hash 604 can be determined, for example, using any suitable low collision hash function on at least a portion of the object data and / or metadata for each object. Examples of hash functions include SHA-1, SHA-256, MD5, FarmHash Fingerprint64, and the like.

[0066] As an example of a particular use case, a user can create a duplicate of a previously ingested object to store into the same bucket. This can be done to change the object metadata on the network storage system 104 to indicate that the object is still hot, so that the object does not become a target for tiering to colder storage on, for example, a remote system. Another reason to create a duplicate can be to support different access permissions, for example, creating read / write permissions for one object version and read-only permissions for another version of the object. Thus, versioning can allow the system to provide a more fine-grained level of control to the user. Object versions can be propagated, for example, each version can be replicated to one or more network storage systems 104. In some cases, if the system herein determines that the same content has already been replicated to a network storage system, a link to the original version including the same content can be created, if allowed by the remote system, rather than propagating the same content again.

[0067] Figure 7 An example data structure 700 for network storage system object versions is shown, in accordance with some implementations. In this example, the data structure includes a version ID 702, a metadata tag 704, a content size 706, a storage size 708, and a timestamp 710. As described above, the remote system object version data structure 700 can be used to determine whether other versions of an object exist on the remote network storage system 104 to which the data structure belongs. Further, as described below, the metadata tag 704 can actually correspond to a content hash for a particular object.

[0068] In some examples, because the version IDs 602 and 702 on the data structures 600 and 700, respectively, are controlled by different systems (i.e., local and remote), the version IDs 602 and 702 for a particular object version can not match. However, if the same version is maintained, the content hash and size on both systems will be the same. Additionally, some examples herein can use the S3 protocol, which supports creating metadata tags for object versions. For example, because not all network storage systems 104 can support retrieval of content hashes, the content hash can be added as a metadata tag to the remotely stored object, which can also be used for comparison when determining whether a same version of a particular object already exists. The state of object versions on the local system can be maintained to prevent the occurrence of re-duplication. If a new version of an object with matching content hash and size already exists in the local storage system and is re-ingested by a user, for example, as shown in 612 in FIG. 6, instead of copying the same version of the object to the network storage system more than once, only the link to the object is copied, as shown in 712 in FIG. 7, and the content is not re-transmitted. For example, the link can point to a version of the content that already exists on the network storage system. Figure 6 Figure 7

[0069] Figure 8 An example 800 of versioned replication is shown, according to some implementations. For example, assume that a first user 112(1) of a first user device 108(1) uses a user application 114(1) to generate an object and store the object to the system 100 described above. Further, assume that the first user 112(1) configures a local application bucket 802 in the local storage 132 and a remote application bucket 804 for receiving replication of the user object on one or more network storage systems 104.

[0070] When the policy engine program 140 copies the object from the local application bucket 802 to the remote application bucket 804, the policy engine program 140 can add a metadata tag to the metadata of the object being copied. As shown in 806, the metadata tag can include the content hash that was previously determined for the object in the local application bucket 802. Thus, during the copy, the policy engine program 140 can send all of the content of the object, including the object data and user metadata, to the network storage device for storage in the remote application bucket 804. As shown in 808, the network storage system can assign its own version ID to the copied object stored in the remote application bucket 804.

[0071] Figure 9 ​​An example 900 of versioned replication according to some implementations is shown. In this example, assume that a first user 112(1) re-ingests a copy of the same object into the local application bucket 802. The local system will assign a new version ID as indicated at 902, and update the timestamp of the object in the data structure 600, but the content hash and size are the same as the previously ingested object. When the new version of the object is replicated, then as shown at 904, the object content is not sent, but a link is created to force a timestamp update on the network storage system, and also a user metadata update is replicated. Thus, when the new version of the object is added to the remote application bucket 804, as indicated at 906, a new version ID is assigned, and as indicated at 908, the storage size is indicated as zero due to the link. Figure 8

[0072] Figure 10 An example 1000 of applying filters to bucket replication according to some implementations is shown. For example, as discussed above, filters can be implemented by a user on a bucket during synchronization of the policy engine program 140 to the object name, metadata tags, or based on various other metadata applications.

[0073] In this example, assume that an operator (such as a user or administrator) configures the bucket replication to include a filter rule to allow replication of objects whose name starts with "foo." Subsequently, assume that the user application 114(1) ingests an object named foo2 to the local application bucket 802. The object foo2 is stored to the local application bucket 802. The policy engine program 140 can then replicate all of the content of the object to the remote application bucket 804, as the filter in place at the time of replication to the remote application bucket 804 allows fo2.

[0074] Figure 11 An example 1100 of applying filters to bucket replication according to some implementations is shown. In the example discussed above Figure 10 In this example, assume that the user 112(1) ingests an object whose object name is "bar" to the local application bucket 802 using the application 114(1). Before replicating the object to the remote application bucket 804, the policy engine will check the filter and determine based on the filter that the replication of the object is not allowed. Thus, as shown at 1102, the object is not replicated to the remote application bucket 804 because the filter does not allow the object.

[0075] Figure 12 Figure 13 ​​is a flow diagram that shows an example process for routing metadata requests, in accordance with some implementations. The process is shown as a collection of blocks in a logical flow graph, which represents a sequence of operations, some or all of which can be implemented in hardware, software or a combination thereof. In the context of software, the blocks can represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, program the processors to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the blocks are described is not intended to be limiting. Any number of the described blocks can be combined in any order and / or in parallel to implement the process, or alternate processes, and not all of the blocks need to be executed. For discussion purposes, the process is described with reference to the environments, frameworks, and systems described in the examples herein, although the process can be implemented in a variety of other environments, frameworks and systems.

[0076] Figure 12 A flow diagram is shown that illustrates synchronizing to the process 1200, in accordance with some implementations. In Figure 12 In an example, the process 1200 can be performed, at least in part, by one or more service computing devices 102 executing the storage program 130, the metadata gateway program 138, and / or the policy engine program 140.

[0077] At 1202, an administrator can provide users in group 1 permission to access a target system. For example, the target system can be the network storage system 104, as described above.

[0078] At 1204, user 1 from user group 1 attempts to add a bucket replication to target 1.

[0079] At 1206, user 2 from user group 2 attempts to add a bucket replication to target 1. However, this effort fails because user 2 belongs to user group 2 and not user group 1, and therefore cannot set up replication to target 1.

[0080] At 1208, the system determines whether the setup of the replication was successful. If so, the process proceeds to 1212. If not, the process proceeds to 1210.

[0081] At 1210, if the setup of the replication was not successful, the system can send a warning to the user about the failure.

[0082] At 1212, if the setup of the replication was successful, user 1 can ingest an object into the local system, such as into a local bucket.

[0083] At 1214, after the object is ingested into the local system, a synchronization event is queued for synchronization of the object to target 1.

[0084] At 1216, the system can attempt to synchronize the object to the remote bucket on target 1.

[0085] At 1218, the system can determine whether the object synchronization was successful. If so, the process proceeds to 1222. If not, the process proceeds to 1220.

[0086] At 1220, if the synchronization was not successful, the system determines whether the object synchronization retry limit has been reached. If so, the system returns to 1214 and queues another synchronization event for the object. If, on the other hand, the retry limit has not been reached, the process proceeds to 1216 and the system makes another object synchronization attempt for target 1.

[0087] At 1222, if the object synchronization was successful, the object appears on the remote system and the synchronization event message is dequeued from the queue.

[0088] Figure 13 A flowchart from the synchronization process 1300 is shown, in accordance with some implementations. In the example of FIG. 13, the process 1300 can be performed, at least in part, by one or more service computing devices 102 executing the storage program 130, the metadata gateway program 138, and / or the policy engine program 140. Figure 13

[0089] At 1302, an administrator can provide users in user group 1 with permission to access a target system. For example, the target system can be the network storage system 104, as described above.

[0090] At 1304, user 1 from user group 1 attempts to add a bucket replication to target 1.

[0091] At 1306, user 2 from user group 2 attempts to add a bucket replication to target 1. However, this effort fails because user 2 belongs to user group 2 and not user group 1, and thus cannot set up replication to target 1.

[0092] At 1308, the system determines whether the replication setup was successful. If so, the process proceeds to 1312. If not, the process proceeds to 1310.

[0093] At 1310, if the replication setup was not successful, the system can send a warning to the user about the failure.

[0094] At 1312, if the replication setup was successful, then, subsequently, a synchronization from the consumer can be initiated for the object.

[0095] At 1314, an object placement message can be received on a remote queue corresponding to target 1.

[0096] At 1316, an object synchronization attempt can be made.​

[0097] At 1318, the system can determine whether the object synchronization was successful. If so, the process proceeds to 1322. If not, the process proceeds to 1320.

[0098] At 1320, if the synchronization was not successful, the system determines whether the object synchronization retry limit has been reached. If so, the system returns to 1314 and queues another object placement message onto the remote queue. If, on the other hand, the retry limit has not been reached, the process proceeds to 1316 and the system makes another object synchronization attempt.

[0099] At 1322, if the object synchronization was successful, the object appears on the local system and the synchronization event message is dequeued from the remote queue.

[0100] The example processes described herein are example processes only. Numerous other variations would be apparent to those of skill in the art given the disclosure herein. In addition, while the disclosure herein sets forth several examples of suitable frameworks, architectures, and environments for performing the processes, implementations of the present disclosure are not limited to the particular examples shown and discussed herein. Furthermore, as described and illustrated in the accompanying drawings, the present disclosure provides various example implementations. However, the present disclosure is not limited to the implementations described and illustrated herein, but can extend to other implementations known or known to those of ordinary skill in the art.

[0101] Figure 14 A select example set of components of a service computing device 102 that can be used to implement at least some of the functionality of the systems described herein is shown. The service computing device 102 can comprise one or more servers or other types of computing devices that can be implemented in any number of ways. For example, in the case of a server, the programs, other functional components, and data can be implemented on a single server, a cluster of servers, a server farm or data center, a cloud-hosted computing service, etc., although other computer architectures can additionally or alternatively be used. Multiple service computing devices 102 can be located together or apart, and for example, organized as virtual servers, server banks, and / or server farms. The described functionality can be provided by servers of a single entity or enterprise, or can be provided by servers and / or services of multiple different entities or enterprises.

[0102] In the illustrated example, the service computing device 102 includes or can be associated with one or more processors 1402, one or more computer-readable media 1404, and one or more communication interfaces 1406. Each processor 1402 can be a single processing unit or a plurality of processing units, and can include single or multiple computing units or processing cores. The processors 1402 can be implemented as one or more central processing units, microprocessors, microcomputers, microcontrollers, digital signal processors, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. As an example, the processors 1402 can include one or more hardware processors and / or any suitable type of logic circuitry that is particularly programmed or configured to execute the algorithms and processes described herein. The processors 1402 can be configured to fetch and execute computer-readable instructions stored in the computer-readable media 1404, which can program the processors 1402 to perform the functions described herein.

[0103] The computer-readable media 1404 can include volatile and nonvolatile memory and / or removable and non-removable media implemented in any type of technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer-readable media 1404 can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid state storage, magnetic tape, magnetic disk storage, RAID storage systems, storage arrays, network attached storage, storage area networks, cloud storage, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Depending on the configuration of the service computing device(s) 102, the computer-readable media 1404 can be tangible non-transitory media in a degree that, when referred to, non-transitory computer-readable media excludes media such as energy, carrier signals, electromagnetic waves, and / or signals per se. In some cases, the computer-readable media 1404 can be located in the same physical location as the service computing device 102, while in other examples, the computer-readable media 1404 can be partially remote from the service computing device 102. For example, in some cases, the computer-readable media 1404 can include portions of storage in the network storage system 104 discussed above with respect to the service computing device 102. Figure 1 The computer-readable media 1404 can include volatile and nonvolatile memory and / or removable and non-removable media implemented in any type of technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer-readable media 1404 can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid state storage, magnetic tape, magnetic disk storage, RAID storage systems, storage arrays, network attached storage, storage area networks, cloud storage, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Depending on the configuration of the service computing device(s) 102, the computer-readable media 1404 can be tangible non-transitory media in a degree that, when referred to, non-transitory computer-readable media excludes media such as energy, carrier signals, electromagnetic waves, and / or signals per se. In some cases, the computer-readable media 1404 can be located in the same physical location as the service computing device 102, while in other examples, the computer-readable media 1404 can be partially remote from the service computing device 102. For example, in some cases, the computer-readable media 1404 can include portions of storage in the network storage system 104 discussed above with respect to the service computing device 102.

[0104] The computer-readable media 1404 can be used to store any number of functional components that are executable by the processor 1402. In many implementations, these functional components comprise instructions or programs that are executable by the processor 1402, and that, when executed, specifically program the processor 1402 to perform the actions attributed to the service computing device 102 herein. The functional components stored in the computer-readable media 1404 can include the user web application 116, the administrative web application 124, the storage program 130, the metadata gateway program 138, the policy engine program 140, and the message queuing program 142, each of which can include one or more computer programs, applications, executable code, or portions thereof. Moreover, while these programs are shown together in this example, during use, some or all of these programs can be executed on separate service computing devices 102.

[0105] Additionally, the computer-readable media 1404 can store data, data structures, and other information used in performing the functions and services described herein. For example, the computer-readable media 1404 can store the metadata database 134 and the local object data 144. Moreover, while these data structures are shown together in this example, during use, some or all of these data structures can be stored on separate service computing devices 102. The service computing device 102 can also include or maintain other functional components and data, which can include programs, drivers, and the like, as well as data used or generated by the functional components. Moreover, the service computing device 102 can include many other logical, programmatic, and physical components, of which those described above are merely examples related to the discussion herein.

[0106] The one or more communication interfaces 1406 can include one or more software and hardware components for enabling communications with various other devices, for example, over one or more networks 106 and 107. For example, the communication interfaces 1406 can enable communications over a LAN, the Internet, a cable network, a cellular network, a wireless network (e.g., Wi-Fi), and a wired network (e.g., Fibre Channel, fiber optic, Ethernet), a direct connection, and one or more of short-range communications such as Bluetooth®, and the like, as additionally enumerated elsewhere herein.

[0107] The various instructions, methods, and techniques described herein can be considered in the general context of computer-executable instructions, such as program modules and applications stored on computer-readable media, and executed by the processors herein. Generally, a program or application can be interchangeable with a routine, method, object, component, data structure, executable code, and / or the like. These programs, applications, and the like can be executed as native code or can be downloaded and executed, such as in a virtual machine or other just-in-time compilation environments. Generally, the functionality of the programs and applications can be combined or distributed as desired in various implementations. Implementations of these programs, applications, and techniques can be stored on computer storage media or transmitted across some form of communication media.

[0108] While the subject matter has been described above in terms of specific embodiments, it is not intended to be limited to the embodiments described. Rather, the subject matter described herein is expected to address a wide range of issues. Thus, various modifications and changes can be made as would be obvious to one skilled in the art, and all modifications and changes fall within the scope of the claims.

Claims

1. A system comprising: a first computing device capable of communicating with a plurality of network storage systems over a network, wherein a first network storage system is provided by a first service provider employing a first storage protocol and a second network storage system is provided by a second service provider employing a second storage protocol different from the first storage protocol, the first computing device configured by executable instructions to perform operations comprising: receiving, at the first computing device, a first object; determining, for the first object, a first remote bucket at the first network storage system and a second remote bucket at the second network storage system; adding a synchronization event to a first queue for synchronizing the first object to the first remote bucket and the second remote bucket; and based on consuming the synchronization event from the first queue, replicating data and metadata of the first object to the first remote bucket and the second remote bucket.

2. The system of claim 1, the operations further comprising: generating a second queue that maintains synchronization events for synchronizing data from the first network storage system and the second network storage system to the first computing device; and maintaining, in the first queue, synchronization events for synchronizing data from the first computing device to the first network storage system and the second network storage system.

3. The system of claim 2, the operations further comprising: receiving, at the first network storage system, a publication of an indication of data for synchronization; based on the publication, adding a synchronization event to the second queue; and based on the synchronization event in the second queue, replicating the data from the first network storage system.

4. The system of claim 2, wherein, the second queue receiving an indication of data for replication from the first network storage system as a synchronization event, the operations further comprising: replicating the data from the first network storage system to local storage at the first computing device.

5. The system of claim 1, wherein, the first network storage system generating a queue at the first network storage system, the operations further comprising: consuming, at the first network storage system, synchronization events from the queue; and based on the synchronization events, replicating data from the first network storage system to the first computing device.

6. The system of claim 1, the operations further comprising: determining a filter associated with at least one of the first remote bucket or the second remote bucket; and based on determining that the filter does not apply to the first object, replicating the data and metadata.

7. The system of claim 6, wherein, the filter for at least one of an object name or object metadata for limiting replication to at least one of the first remote bucket or the second remote bucket.

8. The system of claim 1, the operations further comprising: receiving, at the first computing device, a second object; determining that a size and content hash associated with the second object matches a size and content hash associated with the first object; and copying metadata for the second object to the first remote bucket and the second remote bucket and indicating a link to the first object in place of content of the second object.

9. The system of claim 1, the operations further comprising: based on determining the first remote bucket and the second remote bucket, providing first authentication credentials to the first network storage system and second authentication credentials to the second network storage system, wherein the first authentication credentials are different from the second authentication credentials.

10. The system of claim 1, the operations further comprising executing a policy engine program on the first computing device, the policy engine program configuring the first computing device to: consume a synchronization event from the first queue; and determine a first storage protocol for copying the first object to the first remote bucket and a second storage protocol for copying the first object to the second remote bucket.

11. The system of claim 1, the operations further comprising: in response to receiving the first object, storing data of the first object in real-time in local memory; sending a response in real-time to a source of the first object; and subsequently copying the first object asynchronously to the first network storage system and the second network storage system.

12. A method comprising the steps of: receiving, by a first computing device, a first object, the first computing device being capable of communicating with a plurality of network storage systems over a network, wherein a first network storage system is provided by a first service provider employing a first storage protocol and a second network storage system is provided by a second service provider employing a second storage protocol different from the first storage protocol; determining, by the first computing device, for the first object, a first remote bucket at the first network storage system and a second remote bucket at the second network storage system; adding, by the first computing device, a synchronization event to a first queue for copying the first object to the first remote bucket and the second remote bucket; and based on consuming the synchronization event from the first queue, copying, by the first computing device, data and metadata of the first object to the first remote bucket and the second remote bucket.

13. The method of claim 12, the method further comprising the steps of: generating a second queue, the second queue maintaining synchronization events for synchronizing data from the first network storage system and the second network storage system to the first computing device; and maintaining, in the first queue, synchronization events for synchronizing data from the first computing device to the first network storage system and the second network storage system.

14. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a first computing device, configure the first computing device to perform the following operations: receiving a first object by the first computing device, the first computing device being capable of communicating with a plurality of network storage systems over a network, wherein, a first network storage system is provided by a first service provider employing a first storage protocol and a second network storage system is provided by a second service provider employing a second storage protocol different from the first storage protocol; determining, by the first computing device, a first remote bucket at the first network storage system and a second remote bucket at the second network storage system for the first object; adding, by the first computing device, a synchronization event to a first queue for copying the first object to the first remote bucket and the second remote bucket; and based on consuming the synchronization event from the first queue, copying, by the first computing device, data and metadata of the first object to the first remote bucket and the second remote bucket.

15. The one or more non-transitory computer-readable media of claim 14, the operations further comprising: generating a second queue that maintains synchronization events for synchronizing data from the first network storage system and the second network storage system to the first computing device; and maintaining, in the first queue, synchronization events for synchronizing data from the first computing device to the first network storage system and the second network storage system.

Images