Managed file synchronization with stateless sync nodes

Abstract

Disclosed herein is a system and method for managed file synchronization with a stateless sync node, wherein a client endpoint device has direct access to files stored on a managed file storage. When a client wishes to synchronize with a file stored on a managed file storage, the process identifies a sync role that can handle the request. A particular sync role does not need to maintain state with the client device or the file storage. The state necessary for the processing of the synchronization process is kept separate from the sync role, so that any sync role in the system can pick up a request and process it. Once a role is selected, the synchronization process proceeds as normal. Because state is not maintained by the sync role, multiple roles can handle the request, or two roles can handle different parts of the process.

Description

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2016 / 049441, international application date August 30, 2016, date of entry into the Chinese national phase March 8, 2018, and national application number 201680052057.3. Technical Field

[0002] The present disclosure relates generally to electronic security and, more particularly, to managed file synchronization with a stateless sync node. Background Art

[0003] Various file synchronization solutions have been produced over the years. In these solutions, the synchronization endpoints are accessible to users and applications. When files are created, modified, or deleted, the active-active synchronization engine is responsible for synchronizing these changes and bringing all endpoints to a consistent state. Some synchronization solutions support having hosted file endpoints. However, these solutions are affected by one or both of the following limitations. First, the hosted endpoints are not accessible to end users or applications. Instead, the hosted endpoints can only serve as a "hub" for synchronizing files between other (non-cloud) endpoints. Second, the hosted endpoints may be accessible, but only via access protocols specific to the synchronization solution. Basically, there is no "direct access" to the file at the hosted endpoint. Instead, the application or service must implement a synchronization solution-specific protocol and access the file through the synchronization solution as an intermediary. In addition, the synchronization session requires that a certain state remain unchanged throughout the session. This requires that the state be maintained between the service node and the two endpoints, so that in a scalable system, the process is bound to a specific node. Summary of the invention

[0004] In order to provide the reader with a basic understanding, a brief summary of the present disclosure is given below. This summary is not an extended overview of the present disclosure, and it does not identify the key / important elements of the present invention or describe the scope of the present invention. Its sole purpose is to present some concepts disclosed herein in a simplified form as a prelude to a more detailed description presented later.

[0005] The present disclosure proposes a system and method that can be used to synchronize files from a hosted file system, wherein a client endpoint device has direct access to files stored on a hosted file storage / system. When a client wishes to synchronize with a file stored on a hosted file storage, the process identifies a synchronization role on the hosting platform that can handle the request. A specific synchronization role does not need to maintain state or file storage. The state necessary for processing the synchronization process is maintained separately from the synchronization role, so that any synchronization role in the system can pick a request and process the request. Once the role is selected, the synchronization process will proceed as usual. Since the state is not maintained by the synchronization role, multiple roles can handle the request, or two roles can handle different parts of the synchronization process. This allows the core service to scale the size of the synchronization role up or down based on need and efficiently manage the use of resources.

[0006] Many of the attendant features may be better understood as they become better appreciated by reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present specification will be better understood from the following detailed description when read in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is a high-level block diagram illustrating components of a file synchronization system according to one illustrative embodiment.

[0009] Figure 2 is a block diagram of a file synchronization system in which client devices have direct access to hosted file storage according to an illustrative embodiment.

[0010] Figure 3 is a flow chart illustrating a process for synchronizing files of a client and a hosted endpoint with a stateless sync role according to one illustrative embodiment.

[0011] Figure 4 A component diagram of a computing device according to one embodiment is shown.

[0012] In the drawings, like reference numerals are used to denote like parts. DETAILED DESCRIPTION

[0013] The detailed description provided below in conjunction with the accompanying drawings is intended as a description of the current example, and is not intended to represent the only form in which the current example can be constructed or utilized. The description sets forth the functions of the example and the sequence of steps for constructing and operating the example. However, the same or equivalent functions and sequences may be accomplished by different examples.

[0014] When elements are referred to as being "connected" or "coupled," the elements may be directly connected or coupled together, or one or more intervening elements may also be present. In contrast, when elements are referred to as being "directly connected" or "directly coupled," there are no intervening elements.

[0015] The subject matter may be embodied as an apparatus, system, method, and / or computer program product. Thus, some or all of the subject matter may be implemented in hardware and / or software (including firmware, resident software, microcode, state machine, gate array, etc.). In addition, the subject matter may take the form of a computer program product on a computer-usable or computer-readable storage medium having a computer-usable or computer-readable program code embodied in the medium for use by or in conjunction with an instruction execution system. In the context of this document, a computer-usable or computer-readable medium may be any medium capable of containing, storing, transmitting, propagating, or transmitting the program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0016] The computer usable or computer readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus, device or propagation medium. By way of example and not limitation, computer readable media may include computer storage media and communication media.

[0017] Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disks (DVD) or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the instruction execution system. Note that computer usable or computer readable media can be paper or other suitable media on which the program can be imprinted, as the program can be electronically captured via, for example, optical scanning of paper or other suitable media, and then compiled, interpreted, or otherwise processed in an appropriate manner if necessary, and then stored in a computer memory.

[0018] Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. This is different from computer storage media. The term "modulated data signal" can be defined as a signal that has one or more of its characteristics set or changed in a manner to encode information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media, but not within computer storage media.

[0019] When the subject matter is embodied in the general context of computer-executable instructions, the embodiments may include program modules that are executed by one or more systems, computers, or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, in various embodiments, the functionality of the program modules may be combined or distributed as desired.

[0020] Figure 1 1 is a high-level block diagram illustrating components of a file synchronization system 100. The file synchronization system 100 includes a synchronization endpoint 110 and a synchronization endpoint 150. The synchronization endpoint 110 is connected to a synchronization database 120 and is associated with a file system 130. Similarly, the synchronization endpoint 150 is connected to a synchronization database 160 and is associated with a file system 170. The synchronization endpoint 110 includes a file synchronization provider 111, a synchronization metadata component 112, a data receiver 113, a change updater 114, and a coordinator 115. The synchronization endpoint 150 includes a file synchronization provider 151, an applicator component 152, a conflict resolver / detector 153, a synchronization applicator target 154, and a synchronization metadata component 155. For the purpose of discussion, the components are arranged in an upload scenario from the endpoint 110 to the endpoint 150.

[0021] Before discussing the specific components of the file sync provider 111 or 151, the different types of participants that can provide data will be discussed. A participant is a location that retrieves information from a data source. A participant can be anything from a network service to a laptop to a USB thumb drive.

[0022] The way a provider integrates synchronization will vary based on the capabilities of a particular device. At a minimum, the device should be able to programmatically return information when needed. Ultimately, it will be determined whether the device can enable information to be stored and manipulated on the device or in current data stores and allow applications to execute directly from the device.

[0023] It is important to distinguish between the types of actors to understand whether the actor can store arbitrary state information required by the provider and whether the provider can be executed directly from the device. Ideally, the actor model is generic. Therefore, a full actor can be configured as a partial actor or a simple actor.

[0024] A full participant is a device that allows developers to create applications and new data stores directly on the device. A laptop or smartphone is an example of a full participant because new applications can be executed directly from the device, and new data stores can be created to retain information when needed. A partial participant is a device that is able to store data in an existing data store or other data store on the device. However, these devices do not have the ability to launch executable files directly from the device. Some examples of these participants are thumb drives or SD cards. These devices act like hard drives where information can be created, updated, or deleted. However, they do not usually provide an interface that allows applications to be executed directly on them. A simple participant is a device that is only able to provide information when requested. These devices cannot store or manipulate new data, nor can they support the creation of new applications. RSS feeds and web services provided by external organizations such as Amazon or EBay are examples of simple participants. These organizations can provide the ability to execute or call web services and get results back, but they cannot provide the ability to create data stores for specific users, nor can they provide the ability to create applications that execute inside their web servers.

[0025] File Sync Provider 111 and Provider 151 are similar components found on both Sync Endpoint 110 and Sync Endpoint 150. File Sync Provider can be used to synchronize files and folders in many different file systems such as NTFS, FAT, or SMB file systems. In addition, the directories to be synchronized can be local or remote. They do not have to have the same file system. Applications can use static filters to exclude or include files by explicitly listing or by using wildcards (such as *.txt). Or applications can set filters that exclude entire subfolders. Applications can also register to receive notifications of file synchronization progress.

[0026] Coordinator 115 is a component of system 100 configured to initiate and control a synchronization session between two endpoints or participants. The coordinator communicates with both providers 111 and 151 to start the synchronization process and report back the progress of the synchronization. The actual process used by the coordinator is well known in the synchronization process, and the coordinator 115 can use any process.

[0027] The change updater 114 is a component of the system 100 that identifies local changes to the file system that have occurred without synchronization since the last time the change updater 114 ran. Detection / identification of changes can be done by simply comparing the timestamps associated with the corresponding last synchronization time. Other schemes and methods can be used to determine changes made in the namespace.

[0028] Synchronization databases 120 and 160 are components of system 100 that store metadata about files in a file system. Synchronization databases 120 and 160 provide metadata about specific files to be synchronized between a client and a server. These databases may also be referred to as metadata stores.

[0029] The synchronization database 120, 160 provides the ability to store information about the state and change information about the file system and objects within the file system.The metadata of the file system can be decomposed into five components (concurrency tracking attributes): version, knowledge, tick count, replica ID and tombstone.

[0030] For each item being synchronized, a small amount of information is stored that describes where and when the item was changed. This metadata consists of two versions: a creation version and an update version. A version consists of two parts: a tick count assigned by the data store and a replica ID of the data store. As an item is updated, the tick count is incremented by the data store and the new current tick count is applied to the item. The replica ID is a unique value that identifies a specific data store. When an item is created, the creation version is the same as the update version. Subsequent updates to the item modify the update version. That is, the creation version remains unchanged while the update version changes.

[0031] There are two main ways that version control can be implemented. The first is called online tracking. In this approach, change tracking information for an item is updated when changes are made. For example, in the case of a database, a trigger can be used to update the change tracking table immediately after a row is updated. The second approach is called asynchronous tracking. In this approach, there is an external process that runs and scans for changes. Any updates found are added to the version information. The process can be part of a scheduled process or can be performed before synchronization. This process is often used when there is no internal mechanism to automatically update the version information when an item is updated (for example, when it is not possible to inject logic into the update pipeline). A common way to check for changes is to store the state of the item and compare it to the current state. For example, it might check whether the last write time or file size has changed since the last update. Of course, other version control approaches can also be used.

[0032] All change tracking must happen at least at the level of items. In other words, each item must be independently versioned. In the case of file synchronization, an item can be a file, but it can be anything else that can be synchronized, such as a folder. In some scenarios, more granular tracking may be desirable, as it reduces the likelihood of data conflicts (two users updating the same item on different replicas). The downside is that it increases the amount of change tracking information stored.

[0033] Another concept is that of knowledge. Knowledge is a concise representation of changes that a replica is aware of. As version information is updated, knowledge about the data store is also updated. Providers such as providers 111 and 151 use replica knowledge to enumerate changes (determine which changes another replica is not aware of) and detect conflicts (determine which operations are performed without knowledge of each other).

[0034] Each replica should also maintain tombstone information for each item that is deleted. This is important because when synchronization is performed, if the item no longer exists, the provider will not be able to tell that the item has been deleted, and therefore cannot propagate the change to other providers. The tombstone can contain the following information: global ID, update version, and creation version. Because the number of tombstones will grow over time, in order to save space, some implementations can create a process to clean up the storage after a period of time. That is, these deleted files will be removed from the list of files maintained using metadata information. To prevent this from happening, the system can implement a method for capturing these files. The method starts by first determining whether the conditions are met when there may be a missing tombstone. The replica keeps another copy of the knowledge, which can be considered or referred to as "forgotten knowledge". When a tombstone is removed / cleaned from the database, the forgotten knowledge is set / updated. This forgotten knowledge can keep track of which tick counts have been cleaned up. This can provide hints about when a tombstone may have been lost. Then, if synchronization does not occur for a period of time (the replica becomes stale), the forgotten knowledge helps detect that the replica may be stale. The stale replica can start a "full enumeration synchronization session". This full enumeration is a time intensive and expensive synchronization session whereby all file metadata is transferred between the participants of the synchronization session.

[0035] Applicator 152 is a component of the system that applies specific changes indicated as necessary to complete the synchronization process. These changes are based on the changes recorded by change updater 114 during the synchronization process based on the metadata associated with each file. Depending on the direction of the synchronization process (upload or download), applicator 152 will operate on the corresponding synchronization endpoint. Figure 1 , an upload process is illustrated in which client 110 uploads its changes to client 150.

[0036] The conflict resolver 153 resolves conflicts detected between data files that have been stored and purported updated versions of data files received as part of the synchronization process. Basically, a conflict occurs if a change is made to the same item on two replicas between synchronization sessions. A conflict specifically occurs when the source knowledge does not contain a target version for the item (it is understood that the target knowledge does not contain any source version that was sent). If the version is contained in the target's knowledge, the change is considered out of date. Replicas are free to implement various policies for resolving items that are in conflict across the synchronization community. In some schemes, each of the replicas makes the same resolution regardless of where the conflict occurs or where the conflict is detected.

[0037] Here are some examples of common resolution strategies: Source wins: In the event of a conflict, changes made by the local replica always win. Destination wins, the remote replica always wins. Specified replica ID always wins, i.e. no matter who changes the item, the replica with the specified ID always wins. Last writer wins, this is based on the assumption that all replicas are trusted to make changes and that the wall clocks are synchronized. Therefore, the last writer to a file is allowed to win. When merging, in the event that two duplicate items conflict, the system merges the information from one file into the other. Log conflicts, in this scheme, the system chooses to simply log or postpone the conflict.

[0038] The sync target application 154 is a component of the system that applies the indicated changes after resolving any conflicts determined by the conflict resolver. The specific changes are divided into two groups. The first group is the actual data that is changed to the file. This information is processed by the applicator 154 and provided to the file system 170, which makes the appropriate changes to the underlying file. The second set of changes is the changes made to the synchronization metadata. These are applied to the synchronization database 160 through the metadata component 155, where metadata about specific files is stored.

[0039] Hosting platforms typically implement many instances of a particular service or scaling unit. Each instance of a service can be associated with a specific client or a specific service subscription. Traditional hosting platforms can add additional resources to the instance in response to increased load. However, the additional resources result in an increase in all resources that make up the service, rather than an increase in the individual components that make up the service. That is, all components of the service are increased, as opposed to a single component that only requires more resources. Services represent independent units of lifecycle and management in the platform and can be deployed and served in complete isolation from other services. Communication between services can only be performed via public endpoints, while communication between service roles can be performed via public and internal / protected endpoints. From a network perspective, each service has a single load-balanced virtual IP (VIP). Each VIP is a separate entity for detecting and preventing distributed denial of service (DDoS) attacks. In order to avoid requiring clients to remap non-default HTTP ports (80, 443) for communicating with different role endpoints within a single service, one solution uses a separate service to expose endpoints on the default port. The second solution uses a front-end service to route requests on the default port to roles within a single service based on the received uniform resource locator (URL). Both services and roles in the current platform consist of logically related groups of components. Separate services are created when the functionality or deployment of services are significantly separated from each other, or when the functionality of related services is utilized.

[0040] Figure 2 is a block diagram of a platform 200 implementing services and roles according to one approach. Services include subscription services 220, core services 210, monitoring services 230, analysis services 240, metadata storage 250, management data storage 260, file storage 280, and runtime state storage 270. However, in other approaches, additional services may be added to the platform based on the needs and expectations of specific end users. Client devices 290 connect to the platform to access these services.

[0041] The core service 210 implements multiple different roles within the service. These roles include a front-end role 211, an administration role 212, a synchronization / recall role 213, a background processing role 214, a data access discovery role 215, and a backup / restore role 216. It should be noted that Figure 2A single instance of a core service 210 and each role within the core service 210 is shown. However, in the present platform, the core service 210 may exist on any number of nodes, and these nodes may be distributed in different locations around the world or in a data center. Requests to any particular role may be serviced by any of these particular instances of a node. In addition, none of the roles presented herein are dependent on a particular instance of a file store 280 or metadata table. Further, it should be noted that, depending on the needs of the platform, certain roles may be extended to more nodes without similarly extending the remaining roles.

[0042] The front-end role 211 is configured in one solution to implement a thin front-end based on the gatekeeper pattern. The gatekeeper pattern is designed to protect applications and services by using a dedicated host instance that acts as a proxy between the client and the application or service. It verifies and cleans requests and passes requests and data between them. This provides an additional layer of security and can limit the attack surface of the system. The front-end role 211 performs validation of incoming requests and routes these requests to the management role 212, synchronization role 213, data access role 215, or backup / restore role 216 based on the content of the request. The front-end role 211 or synchronization role 213 also queues requests for long-running tasks to the background processing role 214 in the background task queue.

[0043] In some scenarios, the front-end role 211 also implements additional security measures or schemes to enhance the protection of client data and the overall security of the system. The front-end role 211 will interact with the security library 218 to implement additional security. The security library 218 can implement security mechanisms such as mutual authentication, SSL / TLS encryption, RMS data encryption, shared secrets, distributed denial of service (DDoS) protection, and shared access signatures. Other security mechanisms can be adopted.

[0044] The management role 212 is configured in one approach to provide provisioning, management and configuration capabilities to other services and roles in the platform. The management role 212 provides service endpoints for management access to core services and the ability to modify services through command input. The associated data for the management role 212 is stored in the management data store 260.

[0045] The sync / withdraw role, referred to herein as the sync role 213, is a component of the platform that is configured to service synchronization and file withdrawal foreground requests from synchronization and withdrawal clients (e.g., client devices 290) provided to it in a stateless manner from the front-end role 211. The client device 290 issues a synchronization request to the front-end role 211, which then provides the request to the sync role 213. The sync role 213 interfaces with a load balancer that balances the request for efficiency and ensures that the response is returned before a predetermined timeout. The sync role 213 during synchronization accesses data from the metadata storage 250, the file storage 280, and the runtime state storage 270 located on the platform background. Any number of sync roles 213 operations can exist on the platform at a given time. This allows efficient processing of synchronization requests. Any one of these sync roles 213 is capable of processing the request. Because such requests from client devices 290 can go to different instances of the sync role 213 at different times. As previously described, the synchronization role 213 is a stateless role relative to the file service and file storage 280 .

[0046] The core service also includes a load balancer, which implements load balancing of front-end requests based on stateless synchronization / recovery processing and an efficient cache system. If not cared for over time, each instance of the synchronization role 213 will eventually cache data for each request from the client device 290 connected or communicating with it. That is to say, each partner will cause additional cache. Therefore, the load balancer implements a cache strategy that prevents the synchronization role 213 from exceeding the threshold limit on its CPU and memory utilization and also ensures that the internal data structure remains below the threshold level. In one embodiment, the load balancer uses a round-robin strategy to route requests. That is, the next synchronization role 213 for a service without a request is assigned to handle the request. The load balancer can also use heuristic methods to optimize the location and resource usage of the locally used high-speed cache. For example, a request to the synchronization role 213 can be routed based on a hash of a tenant or synchronization folder that provides affinity to the request without introducing a state of fullness to the request.

[0047] Background processing role 214 is a component of the system that handles long-running tasks and synchronization role 213 unloads tasks from synchronization role 213 in an attempt to balance its workload. There are tasks that take a long time and may affect the overall performance of the system if their operations are completed in the foreground. Long-running tasks can include tasks such as change detection using enumeration, backup and recovery of the system, etc. Background processing role 214 receives various tasks to be executed through background task queues from front-end role 211. Background task queues implement sharding and priority queues for managing tasks given to background processing role 214. It uses performance metrics related to the throughput of tasks. Priority queues are mainly used for requests / tasks that require low latency, such as file recalls. The results created by background processing role 214 are asynchronously reported to calling client device 290 through notifications to the device.

[0048] The Data Access Discovery Role is a system component that provides location and data access capabilities to the platform. This can include providing secure access to data. This secure access can be achieved using a REST interface and a shared access signature key.

[0049] The backup / restore role 216 is a component of the system that allows for the maintenance of recovery data that can restore the client device 290 in the event of a disaster, client device replacement, data loss, or other failure. Backup data can be provided to the system from the device using this role. The restoration of the client device 290 will result in the data stored therein being pushed to the client device 290 to restore the client device 290.

[0050] The monitoring service 230 is a component of the system that provides service status, diagnostics, and troubleshooting capabilities as a combined view of the last mile, outside-in (active), and inside-out (passive) monitoring of system components. The underlying hosting platform provides monitoring of the platform infrastructure (such as data centers and networks) and platform services. Additional diagnostics and troubleshooting can be handled by the monitoring service 230 and performed in the background.

[0051] The analysis service 240 is a component of the system that allows telemetry analysis of the system. Specifically, the analysis service 240 can provide a portal through which administrators can view business and operational analysis capabilities. This can enable administrators to make data-driven decisions about field or business aspects of the service. The analysis service 240 receives data from multiple data sources for post-processing, reporting, and machine learning. These data sources assist in generating the analysis.

[0052] The metadata store 250 is a component of the system that handles metadata for the synchronization process and the files themselves. The metadata store 250 implements replica and project metadata storage, secondary indexes, locking, snapshot isolation, and garbage collection. Secondary indexes support query patterns for various synchronization scenarios, such as range queries. Locking is provided to ensure that one writer at a time can access a specific file or a specific replica where the file resides. These accesses occur while processing a batch of changes or during enumeration of a specific namespace 281. Snapshot isolation consumes committed data and will prevent garbage collection until the various endpoints have consumed the committed data. The metadata store 250 also provides cross-table consistency. Knowledge and project data must be committed together. This ensures that the complete image of a specific replica is known. That is, whether the replica has changed, and the status of the replica.

[0053] The management data store 260 is a component of the system that manages the placement of data within the file store 280 and the corresponding namespace 281, as well as any other data provided by the client device 290 for management purposes. When the file store 280 is shared between various different users and clients, each namespace 281 must be kept separate from the other namespaces 281. The management data store 260 maintains a table for each namespace 281-1, 281-2, 281-N (collectively referred to as 281) managed by the hosting system. Each table represents the configuration of the replicas of a particular tenant and the namespace 281 of each replica stored in the file store 280. This configuration ensures that synchronization represents the correct configuration of the file store 280 and that the metadata also reflects the configuration.

[0054] File storage 280 is a component of the system where the actual data for namespace 281 resides. File storage 280 can store data in containers. Each user has a corresponding container in file storage 280 that corresponds to a sync folder maintained in management data storage 260.

[0055] These user containers can also be shared with multiple different users and devices. Access to the container may occur from multiple endpoints at the same time or almost at the same time. A single container is maintained for universal sharing. In addition, universal sharing corresponds to a synchronization folder on the management data storage 260. Different from traditional file storage 280 in a managed environment, users and devices can directly write to the corresponding container without having to perform these reads and writes through the synchronization role 213. Provide various endpoints (users and devices) with information that allows them to know the actual location on the file storage 280 where the data is located, such as a uniform resource identifier (URI) or a uniform naming convention (UNC). Previous solutions have required the use of an intermediary to access the file storage 280 because the location of the file storage 280 is not accurately known to the device. However, for a specific container or namespace 281, access to the file storage 280 can still be completed by the synchronization role 213 or other core service 210 roles, as traditionally completed. Therefore, depending on the capabilities of the application or device, the file can be accessed by any method. In this way, traditional applications can use the file storage 280 without modification. The data stored in the file storage 280 and in a particular container is stored in a stateless manner. That is, the client manages any transients necessary for any client interaction with the file storage 280. The file storage 280 does not maintain any information about such transients of the data in its own system.

[0056] Before a file namespace 281 can be synchronized with a cloud endpoint, storage must be allocated or provisioned for files, directories, and metadata. This solution provides a single file share or container for each synchronization namespace 281. In this way, multiple namespaces 281 can be hosted on the cloud, but each namespace 281 can remain separate from each other. In one solution, the file share is an Azure file share. However, other versions and types of file shares can be used.

[0057] A file share is a file storage unit that represents the root of a hierarchical namespace of folders and files 281. Shares can be accessed through an application programming interface (API) such as the Azure File REST API, or through protocols such as the ClFS / SMB protocol.

[0058] By mapping the synchronization namespace 281 to the file share, many advantages can be realized. First, the file share allows direct synchronization to the shared namespace 281 root mapping. Other configuration options such as user-to-share mapping or tenant-to-share mapping require that each synchronization namespace 281 be created under the file share root. The second is snapshot isolation. This solution utilizes the file share-level snapshot feature of the hosting service. This supports the ability to create and maintain an efficient copy of the state of the share at a single point in time. This is important for supporting backup and restore, migration, high availability, and other functions that require a consistent view of the namespace 281. The third is security isolation. This solution utilizes the shared access signature (SAS) authorization feature of the host service. This supports the option of controlling access to the share at the root level on a per-namespace 281 basis. Share-level access control can replace or be added to more fine-grained access control at the file or folder level.

[0059] The host system may implement two different schemes for determining when to provision a file share for a particular namespace 281. One scheme is to use explicit provisioning through the management console to create a namespace 281 sync partnership 281 with a particular user's namespace 281. The second scheme is to implicitly provision a namespace 281 when first attempting to sync with the namespace 281.

[0060] Once the sync namespace 281 has been provisioned with file shares, the namespace 281 can be synchronized between the participants and the cloud endpoints. The synchronization solution uses a file synchronization protocol between the two parties (endpoints) of the synchronization partnership. The synchronization process can follow the above description of Figure 1 However, it should be noted that the protocol involves the exchange of metadata about the state of files / folders within the namespace 281 on each endpoint, followed by one or more upload and download sessions in which file and folder metadata and data are transferred and created on each endpoint until the state of the namespace 281 on each endpoint matches. In the event that a file has changed on both sides of the sync partnership since the last sync session, a conflict is detected, which can result in one or both sets of changes being retained.

[0061] The runtime state store 270 is a component of the system that maintains the state of files and the synchronization state of files. The runtime state store 270 enables a specific synchronization role 213 to remain stateless with the file system and file storage 280. When a specific synchronization role 213 needs state to perform a specific task, the runtime state store 270 provides the required state relationship to allow the synchronization process to continue. Because the state is maintained away from the synchronization role 213, any synchronization role 213 can perform any synchronization process.

[0062] Client device 290 is any device that can connect to sync role 213 to synchronize its data with data hosted and stored on file storage 280. Client device 290 may include a server located in a venue, a mobile phone, a notebook computer, a tablet computer, or any other device that interfaces with file storage 280 or another instance of core service 210. In addition, in the case where the device is hosted on another platform, the device may also be a virtual version of the device. Client device 290 may interact with and write to file storage 280 directly, or may access file storage 280 and specific containers contained therein through sync role 213. Each client device 290 also has its own version of sync engine 292. The sync engine is a gateway for client device 290 to initiate a synchronized upload or download with sync role 213. From the client's perspective, the synchronization process is no different from a system in which the only way to access data is through an intermediary.

[0063] File synchronization activities in the present system can be periodic and can be driven by scheduling, or can be on-demand instructions from the client endpoints of the synchronization partnership. File access activities can occur at any time because the client device 290 can directly access the file storage without using an intermediate synchronization role to access cloud-based files.

[0064] Figure 3 is a flow chart illustrating a process for synchronizing files between a client device 290 and a remote file service according to an illustrative embodiment. The synchronization process begins when a client device 290 requests a synchronization session with a core service 210. This is shown in step 310. In some scenarios, the request is generated by the client device 290. In other scenarios, the request is generated by the core service 210. In this scenario, the core service 210 sends a message to the client device 290 instructing the client device 290 to make a synchronization request. The synchronization requests can be on-demand synchronization requests, or they can be periodic synchronization requests. The timing of periodic synchronization requests can be set by an administrator to ensure consistency of data between all synchronization clients. For example, depending on the level of activity in the corresponding files, periodic synchronization requests can be executed once a day or once an hour.

[0065] The synchronization request is received by the core service 210 and provided to the front-end role 211 of the core service 210 to determine whether the request can be processed. This is shown in step 320. Again as previously described, the front-end role 211 does not have direct access to the corresponding file in the file storage, has limited access to any persistent storage of the host system, and can load balance the received requests. In this step of the process, the front-end role 211 implements its gatekeeper mode in protecting the disclosure of client data. The front-end role 211 verifies that the request from the client is the correct request and contains the correct credentials. If the request does not meet the requirements for accessing the underlying data store or service, the front-end role 211 will no longer process these requests. If the request does meet the requirements for access, the front-end role 211 will route the request to the correct service role and shard to support the resource density necessary to maintain stateless synchronization.

[0066] The front-end role 211 analyzes the request and determines whether the particular request is for a long-running task. This is shown in step 330. Long-running tasks are tasks such as enumeration, backup and recovery of the system, etc., which use a large amount or resources or whose execution exceeds a predetermined threshold time amount to complete. These long-running tasks will be sent to the background task queue for processing from there. The front-end role 211 can now try to determine the size of the request by sending a request to the synchronization role 213 for determining the size of the particular request. The synchronization role 213 can read from the metadata storage 250 to determine which files have changed. Based on the number of files that have changed, the front-end role 211 can determine whether the particular synchronization request is a long-running request or a normal request. In some cases, the front end can determine by itself whether the request is for a long-running or short-running task. For example, if the synchronization request is for a specific file or a specific folder, the front-end role 211 can determine that the task is a short-running task. And if the request is for a series of folders or directories, the front-end role 211 can determine that the request is a long-running task. In other cases, the front-end role 211 simply passes the synchronization request to the synchronization role 213 and allows the synchronization role to determine whether the request is short-running or long-running and should be processed by the synchronization role 213 or passed to the background task processing role 214.

[0067] The background task processing role receives the long-running task from the background task queue filled by the front-end role 211. This is shown in step 335. The background task processing role obtains the next task in the queue and determines whether the request or task needs to be sharded. Sharding is a type of database partitioning, which divides a super-large database into smaller, faster, and more manageable parts called data shards. In this case, the file storage 280 has been divided into horizontal partitions or shards. Each shard has the same mode, but retains its own unique data subset. The shard itself is a data storage, running on different instances of the storage node of the file storage 280. In this case, the request will be divided into different parts to reach the appropriate part of the file storage 280 where the underlying data is located. The background task processing role then processes the request to the corresponding file storage 280 to retrieve or update the contained data. The client device 290 will be asynchronously updated by the background task processing role The status of these background tasks. In one scheme, the background processing role 214 processes the request and stores the result in the runtime state storage 270. The front end 211 can retrieve the result of the operation from the runtime state storage 270 so that the response can be provided to the client device 290. In some cases, the front end 211 checks the runtime state for a short period of time to see if the task is completed in a medium amount of time (on the order of 30 seconds), so it can immediately return the result to the client. This is very useful in identifying a heuristic method error for a long-running task, and the task is actually executed very quickly. In other cases, when the front end 211 has waited long enough, it returns a "pending" result to the client, using a URL to represent the pending result. The client 290 can continue to wait for the result by polling the pending result URL. When receiving the pending result URL, the front end 211 can check the runtime state storage 270 to see if the task has reported the result. If one is found, the result of the long-running task is returned to the client 290. The further processing of the long-running task is similar to the processing of the short-running task, and is followed by the following steps 350-370.

[0068] If the request is determined to be a short-running task, the synchronization request is passed to the synchronization role 213 for processing. This is shown in step 340. The front-end role 211 can select a specific synchronization role 213 to receive the request based on multiple factors. When multiple nodes of the synchronization role 213 exist, the front-end role 211 can simply select the synchronization node based on a round-robin scheme. That is, for example, if the first synchronization role 213 node is previously selected, the second synchronization node in the group of nodes will be selected for the next synchronization request. In other schemes, the front-end role 211 can view the load currently experienced on the node and the size of the synchronization request in conjunction with the load balancer, and select a low-usage synchronization role 213 node for processing the request. The front-end role 211 load balances the request and then sends the request to the selected synchronization role 213. In some schemes, due to the size of the synchronization request or the speed at which the request is required to be served in the synchronization, the request can be decomposed by the front-end role 211 or fragmented into several batches. Each of these batches can be sent to a different synchronization role 213 for processing. In some arrangements, the request may instead be sent to the background processing role 214 .

[0069] The sync role 213 receives the sync request from the front-end role 211 and begins processing the sync request. This is shown in step 350. At this point, the sync role 213 needs to determine which files have changed and therefore which files need to be synchronized. The sync role 213 builds or identifies batches of files to be uploaded or downloaded from the file storage 280 based on these changes. Information about which files will be modified by the synchronization process is provided back to the front-end role 211, which provides the information to the client device 290. In some scenarios, the sync role 213 can provide this information directly back to the client device 290. Including in the files to be modified by uploading or downloading, the sync role 213 can provide a uniform resource identifier so that the client device 290 can directly read or write to the file storage 280 as part of the synchronization process. In this way, any sync role 213 can process requests for the synchronization process because it does not need to keep state with files during the process. For upload synchronization, the sync role 213 causes the creation of a temporary folder or area in the file storage 280. The staging folder is a holding area where newer versions of files to be synchronized are temporarily held until the synchronization role 213 can submit the files to the file storage 280 through the synchronization process. In some scenarios, the synchronization role 213 can determine that a particular request sent to it will exceed a predetermined threshold of resource usage. In such a scenario, the synchronization role 213 can redirect the request to the background processing role 214 for processing. In other scenarios, the synchronization role 213 can itself fragment the request and send it to other synchronization roles.

[0070] The client device 290 receives batch information of what files to upload or download from the file storage 280, and transmits the files indicated in the batch. This is shown in step 360. At this step, and depending on the upload or download synchronization, the file system of the client device 290 either uploads the files to a temporary folder in the file storage 280, or downloads the corresponding files from the file storage 280. The upload / download of these files is performed directly with the file storage 280 or through users of different file transfer protocols, and is not performed by the core service 210 or the synchronization role 213. In this way, the specific roles in the core service 210 do not need to maintain the state with the files themselves in this process. Only the file storage 280 is used to maintain the state. When files are uploaded to the file storage 280, they are uploaded to the temporary folder in the file storage 280. Each batch created by the synchronization role 213 can have its own temporary storage area. Once the files are uploaded to the temporary storage area, or the download is completed, the client device 290 sends a message to the synchronization role 213 indicating that the client device 290 has completed the upload / download. In some arrangements, the files in the staging folder and the corresponding versions of the files in file storage 280 are not locked for reading and writing during the upload process. If a file in file storage 280 is changed before synchronization can be completed, such as by accessing the file storage 280 from another device, direct access to the file will not be synchronized or committed at this time, but may be blocked until a later synchronization.

[0071] The synchronization role 213 responds to the indication that the upload / download has been completed by submitting the changes. This is shown in step 370. For downloading, the synchronization role 213 provides a batch of changes to the client, allowing the client to download the file content and apply the changes to its local file storage and local metadata storage. In some schemes, the synchronization role 213 submits the changes by updating the synchronization request to complete, and in some schemes updates the metadata storage 250 to indicate that the specific client has been updated with this information. By uploading, the synchronization role 213 causes the files in the temporary storage area to overwrite or replace the corresponding files in the file storage 280 and update the metadata storage 250. The synchronization role 213 causes the files in the file storage 280 to be renamed as temporary files, and then the files in the temporary storage area are renamed to the file storage 280 as the new version of the file. This allows the files to be updated and the batch to be processed, even if a specific service synchronization node will encounter a failure during the synchronization process, because any synchronization node can pick up the file from the temporary folder and continue the synchronization process.

[0072] Figure 4A component diagram of a computing device according to an embodiment is shown. Computing device 400 can be used to implement one or more computing devices, computer processing or software modules described herein. In one example, computing device 400 can be used to process calculations, execute instructions, receive and send digital signals. In another example, as required by the system of the present embodiment, computing device 400 can be used to process calculations, execute instructions, receive and send digital signals, receive and send search queries and hypertext, and compile computer codes. In addition, computing device 400 can be a distributed computing device, wherein the components of computing device 400 are located on different computing devices connected to each other through a network or other form of connection. In addition, computing device 400 can be a cloud-based computing device.

[0073] Computing device 400 may be any general purpose or special purpose computer now known or that becomes known that is capable of performing the steps and / or performing the functions described herein in software, hardware, firmware, or a combination thereof.

[0074] In its most basic configuration, computing device 400 typically includes at least one central processing unit (CPU) 402 and memory 404. Depending on the exact configuration and type of computing device, memory 404 may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.), or some combination of the two. In addition, computing device 400 may also have additional features / functionality. For example, computing device 400 may include multiple CPUs. The described methods may be performed in any manner by any processing unit in computing device 400. For example, the described processes may be performed in parallel by multiple CPUs.

[0075] The computing device 400 may also include additional storage devices (removable and / or non-removable), including but not limited to magnetic or optical disks or tapes. Such additional storage devices may be stored in the form of Figure 4 406. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Memory 404 and storage 406 are both examples of computer storage media. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 400. Any such computer storage media may be part of computing device 400.

[0076] The computing device 400 may also include a communication device 412 that allows the device to communicate with other devices. The communication device 412 is an example of a communication medium. The communication medium typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism and includes any information transfer medium. The term "modulated data signal" refers to the following signal, which has one or more characteristics thereof set or changed in a manner so that information is encoded in the signal. As an example and not limitation, the communication medium includes wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer-readable medium used herein includes both computer storage media and communication media. The described method can be encoded in any computer-readable medium in any form such as data, computer-executable instructions, etc.

[0077] The computing device 400 may also have input devices 410 such as a keyboard, mouse, pen, voice input device, touch input device, etc. Output devices 408 may also be included such as a display, speaker, printer, etc. All of these devices are well known in the art and need not be discussed in detail.

[0078] Those skilled in the art will recognize that the storage device for storing program instructions can be distributed across a network. For example, a remote computer can store an example of a process described as software. A local or terminal computer can access a remote computer and download part or all of the software to run the program. Alternatively, a local computer can download a fragment of the required software, or perform distributed processing by executing some software instructions at a local terminal and executing some software instructions at a remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to those skilled in the art, all or part of the software instructions can be performed by a dedicated circuit such as a DSP, a programmable logic array, etc.

Claims

1. A method comprising: receiving, from a client device, a synchronization request for synchronizing data between the client device and a file store on a remote hosting platform; dividing the synchronization request into two or more data synchronization tasks for synchronizing data between the client device and the file storage, and assigning the two or more data synchronization tasks to two or more stateless synchronization nodes respectively; transmitting a synchronization state of the data between the client device and the file store from a state store to the two or more stateless synchronization nodes; as well as The data is synchronized based on the synchronization state and the two or more data synchronization tasks via the transmission of the two or more stateless synchronization nodes, which results in a change in data stored in at least one of the client device and the file storage. 2 . The method of claim 1 , wherein the synchronization state comprises a synchronization history of the data between the client and the file store performed by at least one different stateless synchronization node.

3. The method according to claim 1, further comprising: The two or more stateless synchronization nodes are selected from the plurality of stateless synchronization nodes based on current synchronization loads in the plurality of stateless synchronization nodes.

4. The method according to claim 1, further comprising: The two or more stateless synchronization nodes are selected from a plurality of stateless synchronization nodes based on an expected duration of a synchronization task identified by the synchronization request.

5. The method according to claim 1, further comprising: The two or more stateless synchronization nodes are selected from a plurality of stateless synchronization nodes based on a round-robin scheme. 6 . The method of claim 5 , wherein the selecting comprises selecting a stateless synchronization node from the plurality of stateless synchronization nodes that is different from a most recently selected stateless synchronization node.

7. The method of claim 1, wherein the synchronizing comprises resuming the data synchronization between the client device and the file storage based on a point at which a different stateless synchronization node completed the data synchronization between the client device and the file storage. 8 . The method according to claim 1 , wherein the dividing comprises dividing the synchronization request into the two or more data synchronization tasks based on one or more of a size of the data and a speed required by the synchronization request.

9. The method according to claim 1, wherein the division includes executing the synchronization task separately based on two or more shards in the file storage, and dividing the synchronization request into two or more data synchronization tasks.

10. A computing system comprising: a network interface device configured to receive synchronization requests from a client device for synchronizing data between the client device and a file store on a remote hosting platform; at least one processor; as well as a memory storing instructions, wherein the instructions, when executed by the at least one processor, cause the computing system to perform a set of operations, the set of operations comprising: dividing the synchronization request into two or more data synchronization tasks for synchronizing data between the client device and the file storage, and assigning the two or more data synchronization tasks to two or more stateless synchronization nodes, respectively; transferring a synchronization state of the data between the client device and the file store from a runtime state store to the two or more stateless synchronization nodes, and The two or more stateless synchronization nodes are controlled to synchronize the data based on the synchronization state and the two or more data synchronization tasks, which results in a change in data stored in at least one of the client device and the file storage.

11. The computing system of claim 10, wherein the synchronization state comprises a synchronization history of the data between the client and the file store performed by at least one different stateless synchronization node. 12 . The computing system of claim 10 , wherein the processor is further configured to select the two or more stateless synchronization nodes from the plurality of stateless synchronization nodes based on current synchronization loads in the plurality of stateless synchronization nodes. 13 . The computing system of claim 10 , wherein the processor is further configured to select the two or more stateless synchronization nodes from a plurality of stateless synchronization nodes based on an expected duration of a synchronization task identified by the synchronization request. 14 . The computing system of claim 10 , wherein the processor is further configured to select the two or more stateless synchronization nodes from a plurality of stateless synchronization nodes based on a round-robin scheme. 15 . The computing system of claim 14 , wherein the processor selects a stateless synchronization node from the plurality of stateless synchronization nodes that is different from a most recently selected stateless synchronization node.

16. The computing system of claim 10, wherein the processor is configured to control the two or more stateless synchronization nodes to resume the data synchronization between the client device and the file storage based on a point at which different stateless synchronization nodes complete the data synchronization between the client device and the file storage.

17. A non-transitory computer-readable medium comprising: Instructions, when executed by a processor, cause a computer to perform a method, the method comprising: receiving, from a client device, a synchronization request for synchronizing data between the client device and a file store on a remote hosting platform; dividing the synchronization request into two or more data synchronization tasks for synchronizing data between the client device and the file storage, and assigning the two or more data synchronization tasks to two or more stateless synchronization nodes respectively; transmitting a synchronization state of the data between the client device and the file store from a state store to the two or more stateless synchronization nodes; and The data is synchronized based on the synchronization state and the two or more data synchronization tasks via the transmission of the two or more stateless synchronization nodes, which results in a change in data stored in at least one of the client device and the file storage.

18. The non-transitory computer readable medium of claim 17, wherein the method further comprises selecting the two or more stateless synchronization nodes from the plurality of stateless synchronization nodes based on a current synchronization load in the plurality of stateless synchronization nodes.

19. The non-transitory computer-readable medium of claim 17, wherein the method further comprises selecting the two or more stateless synchronization nodes from a plurality of stateless synchronization nodes based on an expected duration of a synchronization task identified by the synchronization request.

20. The non-transitory computer-readable medium of claim 17, wherein the method further comprises selecting the two or more stateless synchronization nodes from a plurality of stateless synchronization nodes based on a round-robin scheme, wherein a stateless synchronization node different from a most recently selected stateless synchronization node is selected from the plurality of stateless synchronization nodes.

Images