Data synchronization method, storage gateway, system and computer-readable storage medium

Abstract

Disclosed are a data synchronization method, storage gateway, system, and computer-readable storage medium. The method specifically includes the following steps: when a file system is updated, the storage gateway records at least one atomic operation log, wherein the at least one atomic operation log is used to record the corresponding operation process of the file system update; the storage gateway constructs an operation dependency graph based on the at least one atomic operation log; and the storage gateway synchronizes the updated data in the file system to the cloud based on the operation dependency graph. This method can improve the speed at which the storage gateway can synchronize and back up updated data in the file system to the cloud in real time.

Description

Technical Field

[0001] The present application relates to the field of cloud storage technology, and in particular to a data synchronization method, storage gateway, system, and computer-readable storage medium. Background Art

[0002] In the information age, traditional data storage methods no longer meet the growing demand for storage systems. Consequently, cloud storage technology has emerged as a hot research topic in this field. In this cloud storage model, users simply deploy a cloud storage gateway (CSG) locally, cache local data to the CSG's file system via the Network File System (NFS) or Common Internet File System (CIFS), and then back it up to the cloud, providing users with massive, secure, and low-cost data storage capabilities.

[0003] Currently, synchronous blocking mechanism or periodic backup mechanism is mainly used to back up local data from CSG's file system to the cloud. However, when the CSG's file system is updated, how to quickly synchronize and back up the updated data in the file system to the cloud is still an urgent problem that needs to be solved. Summary of the Invention

[0004] The present application discloses a data synchronization method, a storage gateway, a system, and a computer-readable storage medium. The method can improve the speed at which the storage gateway synchronizes and backs up updated data in the local file system to the cloud in real time.

[0005] In a first aspect, the present application provides a data synchronization method, which specifically includes the following steps:

[0006] When the file system is updated, the storage gateway records at least one atomic operation log, wherein the at least one atomic operation log is used to record the corresponding operation process when the file system is updated;

[0007] The storage gateway builds an operation dependency graph based on at least one atomic operation log;

[0008] The storage gateway synchronizes updated data in the file system to the cloud based on the operation dependency graph.

[0009] When implementing the method described in the first aspect, when the file system is updated, the storage gateway only needs to record the atomic operation log corresponding to the updated data, and build an operation dependency graph based on the recorded atomic operation log. Therefore, when synchronizing data, the updated data can be synchronized to the cloud concurrently based on the dependency relationship between the nodes in the operation dependency graph, thereby improving the speed at which the storage gateway synchronizes the updated data to the cloud.

[0010] In one possible implementation, when the file system is updated, the storage gateway records at least one atomic operation log, including: when the file system is updated, the storage gateway performs at least one atomic operation, and the at least one atomic operation includes one or more of the following: creating a directory or file, deleting a directory or file, writing a file, and renaming a directory or file; the storage gateway records the at least one atomic operation as at least one atomic operation log, wherein one atomic operation corresponds to one atomic operation log.

[0011] In the above implementation, each time the storage gateway performs an atomic operation, it will record a corresponding atomic operation log. This ensures the uniqueness of each atomic operation log, and also ensures that the operation dependency graph can reflect all the atomic operations performed by the storage gateway when the file system is updated. This enables the storage gateway to assign different working threads to atomic operations that do not have dependencies. The storage gateway can then replay these atomic operation logs concurrently to obtain the updated data in the file system, thereby increasing the speed at which the storage gateway synchronizes updated data to the cloud.

[0012] In one possible implementation, each atomic operation log in at least one atomic operation log records an operation object, and the operation object includes a directory and / or a file. The storage gateway constructs an operation dependency graph based on the at least one atomic operation log, including: the storage gateway constructs an operation dependency graph based on the tree relationship between at least one operation object recorded in the at least one atomic operation log. It can be understood that since the file system uses a tree structure to store data, the updated data is also stored according to the tree structure, that is, the operation objects recorded in the atomic operation log also have a tree relationship, then the storage gateway can construct an operation dependency graph based on the tree relationship between the operation objects recorded in the atomic operation log. Moreover, the operation dependency graph constructed in this way can reflect the association order between different atomic operation logs, that is, the execution order between different atomic operations, which makes it convenient for the storage gateway to replay the updated data in the file system based on the atomic operation log with an association order.

[0013] In one possible implementation, the operation dependency graph includes multiple nodes and at least one directed edge, the multiple nodes include a root node and at least one child node, each child node corresponds to an atomic operation log, and each directed edge indicates a dependency relationship between two atomic operation logs.

[0014] It can be seen that the operation dependency graph is a directed graph, which enables the storage gateway to determine whether there is a dependency relationship between any two atomic operation logs based on the directed edges, thereby facilitating the storage gateway to determine whether a worker thread needs to be allocated for the atomic operation log.

[0015] In one possible implementation, the operation dependency graph includes multiple layers, each layer of the multiple layers includes at least one node, and the multiple layers include a target layer. The storage gateway synchronizes the updated data in the file system to the cloud according to the operation dependency graph, including: the storage gateway traverses multiple nodes in the operation dependency graph in a preset manner, thereby determining the dependency relationship between the atomic operation logs corresponding to the multiple nodes, wherein the preset manner is: after the storage gateway traverses all nodes in the target layer, it traverses all nodes in the next layer of the target layer; the storage gateway synchronizes the updated data in the file system to the cloud according to the dependency relationship between the atomic operation logs corresponding to the multiple nodes.

[0016] In the above implementation, when the storage gateway uses a preset method to traverse all nodes in the operation dependency graph, it can first traverse all nodes in one layer, and then traverse all nodes in the next layer. Compared with other traversal methods (for example, depth-first traversal), the preset method facilitates the storage gateway to determine the nodes in the operation dependency graph that are not pointed to by multiple directed edges, and assign corresponding working threads to the atomic operation logs corresponding to these nodes, so that the storage gateway can replay these atomic operation logs concurrently to obtain updated data in the file system, thereby improving the speed at which the storage gateway synchronizes updated data to the cloud. It can be understood that the atomic operation logs corresponding to the nodes that are not pointed to by multiple directed edges do not directly depend on multiple other atomic operation logs.

[0017] In one possible implementation, at least one child node includes a first child node and a second child node, the first child node corresponds to a first atomic operation log, and the second child node corresponds to a second atomic operation log. The storage gateway synchronizes the updated data in the file system to the cloud based on the dependency relationship between the atomic operation logs corresponding to multiple nodes, including: when there is no dependency relationship between the first atomic operation log and the second atomic operation log, the storage gateway concurrently replays the first atomic operation log and the second atomic operation log to obtain the updated data in the file system, and then synchronizes the updated data in the replayed file system to the cloud; when there is a dependency relationship between the first atomic operation log and the second atomic operation log, and the second atomic operation log depends on the first atomic operation log, the storage gateway replays the first atomic operation log and the second atomic operation log in sequence to obtain the updated data in the file system, and then synchronizes the updated data in the replayed file system to the cloud.

[0018] In the above implementation, when there is no dependency between the first atomic operation log and the second atomic operation log, it means that the first atomic operation log does not need to wait until the second atomic operation log is played back before it is played back, or the second atomic operation log does not need to wait until the first atomic operation log is played back before it is played back, then the first atomic operation log and the second atomic operation log can be executed in different work threads, so the storage gateway can assign different work threads to the first atomic operation log and the second atomic operation log respectively. When the second atomic operation log depends on the first atomic operation log, it means that the second atomic operation log needs to wait until the first atomic operation log is played back before it is played back, so the storage gateway cannot assign different work threads to the first atomic operation log and the second atomic operation log. It can be seen that the above implementation can assign more work threads to different atomic operation logs, thereby increasing the speed at which the storage gateway replays updated data, and further increasing the speed at which the storage gateway synchronizes updated data to the cloud.

[0019] In a possible implementation, the above method further includes: when the file system is updated, the storage gateway updates the metadata in the file system, wherein the storage gateway updates the metadata and records at least one atomic operation log in the same transaction. It is understandable that updating the metadata and recording the atomic operation log must succeed and fail at the same time, that is, when the storage gateway updates the metadata, it also needs to record the corresponding atomic operation log. When the storage gateway does not update the metadata, it does not record the atomic operation log. By ensuring that the updating of the metadata and the recording of the atomic operation log are performed in the same transaction, it is ensured that the storage gateway can accurately replay the updated data in the file system based on the atomic operation log.

[0020] In a possible implementation, the method further includes: when the cloud loses updated data in the file system, the storage gateway synchronizes the updated data in the file system to the cloud again. In the present application, after the storage gateway records the atomic operation log, it also persistently stores the atomic operation log. Therefore, when the cloud loses updated data in the file system, the storage gateway can also rebuild the operation dependency graph based on the atomic operation log, thereby replaying the updated data again, and synchronizing the updated data obtained by replaying again to the cloud again. Through the above implementation, it can be ensured that after the cloud loses part of the data (data updated within a certain time period), it can also quickly obtain this part of the lost data without the need to synchronize all the data in the file system to the cloud again. In other words, the speed at which the storage gateway synchronizes the lost data to the cloud can be improved through the above implementation.

[0021] In one possible implementation, when the application server performs a target operation on the file system, the file system is updated, and the target operation includes at least one of the following: adding data to the file system, deleting data in the file system, and changing data in the file system. It is understandable that after the file system is mounted to the application server, the application server can perform various operations on the file system, such as querying directory information in the file system, changing information about a certain file in the file system, etc. Among the operations performed by the application server on the file system, some operations will cause the data in the file system to be updated, such as target operations; and some operations will not cause the data in the file system to be updated, such as query operations. In this application, the storage gateway will only synchronize the updated data to the cloud when the file system is updated. Therefore, when the application server performs a target operation on the file system, it will trigger the storage gateway to record the atomic operation log, thereby building an operation dependency graph and synchronizing the updated data to the cloud in real time and quickly.

[0022] In a second aspect, the present application provides a storage gateway, which includes a recording unit, a construction unit, and a data synchronization unit.

[0023] The recording unit is used to record at least one atomic operation log when the file system is updated, wherein the at least one atomic operation log is used to record the corresponding operation process when the file system is updated;

[0024] The construction unit is used to construct an operation dependency graph according to at least one atomic operation log;

[0025] The data synchronization unit is used to synchronize the updated data in the file system to the cloud according to the operation dependency graph.

[0026] In one possible implementation, the recording unit is specifically used to: when the file system is updated, perform at least one atomic operation, and the at least one atomic operation includes one or more of the following: create a directory or file, delete a directory or file, write a file, and rename a directory or file; record the at least one atomic operation as at least one atomic operation log, wherein one atomic operation corresponds to one atomic operation log.

[0027] In one possible implementation, each atomic operation log in at least one atomic operation log records an operation object, and the operation object includes a directory and / or a file. The construction unit is specifically used to: construct an operation dependency graph based on the tree relationship between at least one operation object recorded in at least one atomic operation log.

[0028] In one possible implementation, the operation dependency graph includes multiple nodes and at least one directed edge, the multiple nodes include a root node and at least one child node, each child node corresponds to an atomic operation log, and each directed edge indicates a dependency relationship between two atomic operation logs.

[0029] In one possible implementation, the operation dependency graph includes multiple layers, each layer in the multiple layers includes at least one node, and the multiple layers include a target layer. The data synchronization unit is specifically used to: traverse multiple nodes in the operation dependency graph in a preset manner, so as to determine the dependency relationship between the atomic operation logs corresponding to the multiple nodes, wherein the preset method is: after the storage gateway traverses all nodes in the target layer, it traverses all nodes in the next layer of the target layer; according to the dependency relationship between the atomic operation logs corresponding to the multiple nodes, the updated data in the file system is synchronized to the cloud.

[0030] In one possible implementation, at least one child node includes a first child node and a second child node, the first child node corresponds to a first atomic operation log, and the second child node corresponds to a second atomic operation log, and the data synchronization unit is specifically used to: when there is no dependency relationship between the first atomic operation log and the second atomic operation log, concurrently replay the first atomic operation log and the second atomic operation log to obtain updated data in the file system, and then synchronize the updated data in the file system to the cloud; when there is a dependency relationship between the first atomic operation log and the second atomic operation log, and the second atomic operation log depends on the first atomic operation log, replay the first atomic operation log and the second atomic operation log in sequence to obtain updated data in the file system, and then synchronize the updated data in the file system to the cloud.

[0031] In one possible implementation, the storage gateway further includes a metadata management unit, which is used to update the metadata in the file system when the file system is updated, wherein the storage gateway updates the metadata and records at least one atomic operation log in the same transaction.

[0032] In a possible implementation, the data synchronization unit is further configured to synchronize the updated data in the file system to the cloud again when the cloud loses the updated data in the file system.

[0033] In a possible implementation, when the application server performs a target operation on the file system, the file system is updated, and the target operation includes at least one of the following: adding data in the file system, deleting data in the file system, and changing data in the file system.

[0034] In a third aspect, the present application provides a storage gateway, which includes a processor and a memory, and the processor executes the code in the memory to implement the method described in the first aspect or any optional manner of the first aspect.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer instructions are stored, and the computer instructions are used to implement the method described in the first aspect or any optional manner of the first aspect.

[0036] In a fifth aspect, the present application provides a data synchronization system, which includes an application server, a storage gateway, and a cloud. The storage gateway is used to implement the method described in the first aspect or any optional method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a structural diagram of a data synchronization system provided by an embodiment of the present application;

[0039] Figure 2 This is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0040] Figure 3 This is a structural diagram of a tree relationship provided in an embodiment of the present application;

[0041] Figure 4 It is an operation dependency graph provided by an embodiment of the present application;

[0042] Figure 5 This is a schematic diagram of a storage gateway allocating work threads according to an operation dependency graph provided by an embodiment of the present application;

[0043] Figure 6 This is a schematic diagram of the structure of a storage gateway provided in an embodiment of the present application;

[0044] Figure 7 This is a schematic diagram of the structure of another storage gateway provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] In order to facilitate understanding of the technical solution of this application, first, some of the terms involved in this application are explained.

[0047] Cloud storage: is a technology developed on the basis of cloud computing. It is a system that brings together a large number of storage devices in the network to work together through cluster applications, network technology or distributed file systems to provide external data storage and business access functions. Simply put, cloud storage is to put storage resources on the cloud for users to access. Users can access data from the cloud at any time and any place.

[0048] CSG: In order to simplify the use of cloud storage services and enable users to access cloud storage as if they were accessing local disks, one possible implementation method is to deploy CSG locally. CSG provides hybrid cloud services locally and in the cloud. In actual application, CSG supports network file system access protocols (for example, NFS protocol, CIFS protocol), so that local application servers can mount the CSG's file system locally through the network file system access protocol to achieve local data sharing and backup. In addition, CSG also provides data synchronization functions to achieve secondary backup of local data to cloud storage. It is worth noting that CSG can have different names. For example, different standards, different versions of the same standard, different manufacturers, and different application scenarios may have different names for CSG. For example, the term "CSG" can sometimes also be referred to as "cloud storage device", "cloud storage controller", etc.

[0049] Application server: A program that exposes business logic to clients through various protocols. It provides access to business logic for client applications to use.

[0050] The Portable Operating System Interface (POSIX) is a set of interrelated standards developed by the Institute of Electrical and Electronics Engineers (IEEE) to define application programming interfaces (APIs) for running software on various UNIX operating systems. Simply put, the POSIX standard defines the API and operating environment for UNIX-based operating systems, thereby supporting source code portability, allowing programs written for one POSIX-compliant operating system to be compiled and executed on any other POSIX-compliant operating system.

[0051] NFS: A distributed file system protocol that allows different hosts and operating systems to share data over a network. NFS provides the same interface as the CGS file system, allowing local application servers to mount the CSG file system and access it through the POSIX standard interface.

[0052] CFIS: A network file system access protocol that allows programs to access files on remote Internet computers and request services from those computers. The CIFS protocol enables network file sharing between Windows operating system hosts. Similar to NFS, CIFS provides the same interface as the CGS file system. This allows local application servers to mount the CSG file system and access it through the POSIX standard interface.

[0053] Object Storage Service (OBS): This object-based, mass storage service provides users with massive, secure, highly reliable, and low-cost data storage capabilities. It's worth noting that the term "OBS" can have different abbreviations. For example, different standards, different versions of the same standard, different vendors, and different application scenarios may have different abbreviations for "OBS." For example, the term "OBS" can sometimes be abbreviated as "OSS."

[0054] Atomic operations are operations that are not interrupted by the thread scheduling mechanism. Once an atomic operation begins, it continues to run until it completes. Atomic operations are the smallest operation unit cached in memory.

[0055] Metadata, also known as intermediary data or relay data, is data that describes data, mainly information that describes data properties, and is used to support functions such as indicating storage location, historical data, resource search, and file records.

[0056] It is worth noting that the terms used in this application are only used to explain specific embodiments and are not intended to limit the technical solutions of this application.

[0057] Next, the application scenario involved in the embodiment of the present application is introduced: synchronous backup of cloud storage gateway.

[0058] In the information age, as enterprises continue to expand, so too does their data. Traditional data storage methods (e.g., hard disk storage) are complex and restrictive, severely limiting storage capacity and the flexibility of stored data. They are unable to meet the data requirements placed on storage systems. Therefore, cloud storage has emerged. Among existing cloud storage solutions, CSG provides the NFS / CIFS protocol, enabling local application servers to mount the CSG file system locally via the NFS / CIFS protocol. This allows local data to be cached in the CSG, which then backs up the local data to the cloud via a network (e.g., a private network or the internet), providing users with massive, secure, and low-cost data storage capabilities.

[0059] In actual applications, after a local application server mounts the CSG's file system, it can access the CSG's file system. For example, the application server can create new directories or files in the CSG's file system, delete directories or files in the CSG's file system, query a directory or file in the CSG's file system, or modify a directory or file in the CSG's file system. At this point, the data in the CSG's file system may be updated accordingly. In this case, the CSG can synchronize the updated data in the CSG's file system to the cloud using the following two methods.

[0060] One possible implementation method is: CSG adopts a synchronous request blocking method. When the data in the CSG's file system is updated, CSG synchronizes the updated data to the cloud. However, when multiple application servers access the CSG's file system at the same time, the CSG's file system may be updated multiple times in a row. In this case, using the above implementation method requires CSG to synchronize the updated data to the cloud multiple times in a row, that is, each time the CSG's file system is updated, CSG synchronizes the updated data to the cloud once. In addition, CSG can only start the next synchronization after completing one synchronization. Since each synchronization takes a certain amount of time, a lot of time is consumed in waiting during multiple synchronizations, which affects the speed at which CSG synchronizes the updated data in the file system to the cloud.

[0061] Another possible implementation is that the CSG uses a periodic backup method to regularly check whether the data in the CSG's file system has been updated. If the data in the CSG's file system is updated, the CSG synchronizes the updated data to the cloud. Specifically, a resident background process is configured in the CGS, allowing the CSG to periodically perform synchronization and backup tasks. During each cycle, the CSG fully scans all files in the file system. Each time a file is scanned, it determines whether the file has a mapping relationship with an object stored in the cloud. If the file has a mapping relationship with an object stored in the cloud, the CSG determines that the file has not been updated and continues to scan the next file in the CSG's file system. If the file does not have a mapping relationship with any object stored in the cloud, the CSG determines that the file has been updated. In this case, the CSG synchronizes the file or the updated data in the file to the cloud. In addition, if, after all files in the CSG's file system have been scanned, one or more objects in the cloud do not have a mapping relationship with any files in the file system, the CSG determines that the files corresponding to these objects have been updated (deleted). In this case, the CSG synchronizes the updated data to the cloud. It can be seen that CSG cannot synchronize updated data to the cloud well using the above implementation method. The reasons are as follows: First, periodic backup, as the name implies, means that CSG performs a synchronization backup task every once in a while. That is to say, when CSG's file system is updated, CSG cannot synchronize the updated data to the cloud in time. Secondly, every time CSG performs a synchronization backup task, it will perform a full scan of CSG's file system. Since a full scan is to scan and analyze all directories in CSG's file system and all files under all directories, each time CSG performs a synchronization backup task, it will take a lot of time, which will result in low efficiency of CSG in synchronizing data to the cloud. In addition, a full scan requires occupying CSG's processor for a long time, which will seriously affect the performance of CSG, thereby affecting the performance of users accessing the file system.

[0062] To address the aforementioned issues, this application provides a data synchronization method: When the CSG's file system is updated, CSG constructs a corresponding operation dependency graph by recording atomic operation logs. Then, based on the dependencies indicated by the operation dependency graph, it promptly replays the updated data in the file system and synchronizes this updated data to the cloud. Compared to the two methods described above (i.e., the synchronization request blocking method and the periodic backup method), the data synchronization method provided by this application can more efficiently and quickly synchronize updated data in the CSG's file system to the cloud.

[0063] The data synchronization method provided by this application will be described in detail below with reference to the accompanying drawings.

[0064] First, see Figure 1 , Figure 1 It is a structural diagram of a data synchronization system applicable to an embodiment of the present application. Figure 1 The data synchronization system shown includes a local data center 110 and a cloud data center 120 . The local data center includes an application server 111 and a storage gateway 112 .

[0065] In a specific embodiment, the storage gateway 112 is a CSG. The storage gateway 112 supports the NFS / CIFS protocol, and the storage gateway 112 has a file system. The application server can mount the file system of the storage gateway 112 (hereinafter referred to as the file system) through the NFS / CIFS protocol, thereby realizing the function of caching local data to the file system. The storage gateway 112 synchronizes the data in the file system to the cloud data center through a network (for example, a private network or the Internet), thereby realizing the function of storing local data in the cloud. It can be understood that when the application server mounts the file system, the application server is equivalent to the client of the file system, so that the file system can be operated, for example, storing local data in the file system, querying data stored in the file system, changing data stored in the file system, etc. Optionally, the storage gateway 112 can also be other computing devices or storage devices with storage functions, etc., which are not specifically limited here.

[0066] In a specific embodiment, the cloud storage service provided in cloud data center 120 may be OBS, a block storage service, or a file storage service, etc., which is not specifically limited here. Furthermore, in this application, the number of application servers 111 may be one or more, and the number of storage gateways 112 may also be one or more, which is not specifically limited here.

[0067] Next, based on Figure 1 The data synchronization system shown in FIG2 is used to describe the data synchronization method provided by this application in detail. Figure 2 , Figure 2 The following is a flow chart of the data synchronization method provided by the present application, which includes but is not limited to the following steps:

[0068] S101: When a file system is updated, the storage gateway records at least one atomic operation log.

[0069] Among them, at least one atomic operation log is used to record the corresponding operation process when the file system is updated.

[0070] In a specific embodiment, when an application server performs a target operation on a file system, the file system is updated. The target operation includes at least one of the following: adding data to the file system, deleting data from the file system, or modifying data from the file system. It is understood that the application server can access the file system by calling the POSIX interface provided by the file system, thereby performing operations on the file system, such as querying information within a directory or file in the file system, creating a new directory or file in the file system, deleting an existing directory or file in the file system, or modifying information within a directory or file in the file system. At this point, the file system may or may not be updated. For example, when the application server queries information about a file in the file system, the file system is not updated. For another example, when the application server modifies information within a file in the file system, the file system is updated. In general, in the embodiments of the present application, the file system is updated and the storage gateway records the atomic operation log only when the application server performs the target operation on the file system. When the application server does not perform the target operation on the file system, the file system is not updated, the storage gateway does not need to synchronize data with the cloud, and therefore does not need to record the atomic operation log.

[0071] In a specific embodiment, when the file system is updated, the storage gateway performs at least one atomic operation, where the at least one atomic operation includes one or more of the following: creating a directory or file, deleting a directory or file, writing a file, or renaming a directory or file. The storage gateway then records the at least one atomic operation as at least one atomic operation log, where one atomic operation corresponds to one atomic operation log. The specific reason why the storage gateway records an atomic operation as an atomic operation log in this application will be explained in detail in S103 below.

[0072] In a more specific embodiment, the storage gateway records at least one atomic operation as at least one atomic operation log based on the type of the atomic operation and the preset atomic operation log format. The preset atomic operation log format can be predefined by the user based on different atomic operation types. The following shows the formats of atomic operation logs corresponding to several different types of atomic operations:

[0073] (1) The atomic operation is to write a file. The format of the atomic operation log can be defined as: fsync directory / file-name, where the fsync function is used to synchronize the modified file data in the memory to the storage device, and file-name is the name of the modified file, which is stored in the directory directory.

[0074] (2) The atomic operation is to create a directory. The format of the atomic operation log can be defined as: mkdir directory, where the mkdir command is used to create a directory and directory is the name of the newly created directory.

[0075] (3) The atomic operation is to create a file. The format of the atomic operation log can be defined as: mknode directory / file-name, where the mknode command is used to create a file, file-name is the name of the newly created file, and the newly created file is stored in the directory directory.

[0076] (4) The atomic operation is to rename a file or directory. The format of the atomic operation log can be defined as: rename origin-name target-name, where rename is used to rename a directory or file, origin-name is the name of the directory or file before renaming, and target-name is the name of the directory or file after renaming.

[0077] (5) The atomic operation is to delete a directory. The format of the atomic operation log can be defined as: delete directory, where delete is used to perform the delete operation and directory is the name of the directory to be deleted.

[0078] (6) The atomic operation is to delete a file. The format of the atomic operation log can be defined as: delete directory / file-name, where delete is used to perform the delete operation and file-name is the name of the deleted file, which is stored in the directory directory.

[0079] In a specific embodiment, when a file system is updated, the storage gateway updates the metadata in the file system. The storage gateway updates the metadata and records at least one atomic operation log within the same transaction. That is, the storage gateway updates the metadata and records the atomic operation log simultaneously. Specifically, when a file system is updated, the storage gateway updates the metadata stored in the file system and simultaneously records at least one atomic operation log, thereby ensuring that the storage gateway can accurately replay the updated data in the file system based on the at least one atomic operation log.

[0080] S102: The storage gateway constructs an operation dependency graph according to at least one atomic operation log.

[0081] In a specific embodiment, operation objects are recorded in at least one atomic operation log, and the operation objects include directories and / or files. Therefore, the storage gateway can construct an operation dependency graph based on the tree relationship between at least one operation object recorded in at least one atomic operation log.

[0082] In a specific embodiment, since the directories and files in the file system are stored in a tree-like relationship, and the operation objects of the atomic operation are directories and / or files, there is a tree-like relationship between the at least one operation object. The tree-like relationship can be specifically expressed as follows: the file system includes a root directory, which can include at least one subdirectory and at least one file, and each subdirectory can also include at least one subdirectory and at least one file. Figure 3 For example, the file system includes a root directory, which includes subdirectory A, subdirectory B, and file C; subdirectory A includes subdirectory A1, file A2, and file A3; subdirectory B includes file B1 and file B2. Based on the tree relationship, the relationship between directories, the relationship between directories and files, and the relationship between files in the file system can be determined. Figure 3 For example, subdirectories A and B are directories at the same level, file A2 is a file in subdirectory A, and file A3 and file B1 are files in different subdirectories. Therefore, the storage gateway can determine the tree relationships between atomic operation logs based on the tree relationships between the operation objects recorded in the atomic operation logs, thereby constructing an operation dependency graph.

[0083] In a specific embodiment, the operation dependency graph includes multiple nodes and at least one directed edge, the multiple nodes include a root node and at least one child node, each child node corresponds to an atomic operation log, and each directed edge indicates the dependency relationship between two atomic operation logs. Then, the specific process of the storage gateway constructing the operation dependency graph based on at least one atomic operation log is as follows: first, the storage gateway creates a root node. It can be understood that the root node is a virtual node and does not correspond to any atomic operation log in the file system. Then, the storage gateway determines the tree relationship between the child nodes corresponding to each atomic operation log based on the tree relationship of the directories and / or files recorded in each atomic operation log, thereby determining the directed edges between these child nodes to obtain the above-mentioned operation dependency graph.

[0084] For example, the application server sends an operation command to the file system: mkdir dir1 / ; touchdir1 / file1; echo "123" > dir1 / file1; touch dir1 / file2; delete dir1 / file2; mkdirdir2 / ; mv dir1 / file1 dir2 / file3.

[0085] The command "mkdir dir1 / " creates the directory dir1. The command "touch dir1 / file1" creates the file file1 in the directory dir1. The command "echo "123" > dir1 / file1" outputs "123" to the file file1. The command "touch dir1 / file2" creates the file file2 in the directory dir1. The command "delete dir1 / file2" deletes the file dir2. The command "mkdir dir2 / " creates the directory dir2. The command "mv dir1 / file1 dir2 / file3" moves the file file1 to the directory dir2 and renames it to file3.

[0086] After receiving the operation command, the file system executes the corresponding atomic operation and records the corresponding atomic operation log: (1) mkdir dir1; (2) mknode dir1 / file1; (3) fsync dir1 / file1 start=0len=xxx; (4) mknode dir1 / file2; (5) delete dir1 / file2; (6) mkdir dir2; (7) rename dir1 / file1dir2 / file3. Then, based on the tree relationship between the operation objects recorded in the above 7 atomic operation logs, the following is constructed: Figure 4The operation dependency graph shown. Among them, child node A corresponds to atomic operation log (1), child node B corresponds to atomic operation log (6), child node C corresponds to atomic operation log (2), child node D corresponds to atomic operation log (4), child node E corresponds to atomic operation log (3), child node F corresponds to atomic operation log (5), and child node G corresponds to atomic operation log (7). Specifically, the operation object in atomic operation log (1) is directory dir1, the operation objects in atomic operation log (2) and atomic operation log (3) are both file file1 under directory dir1, the operation objects in atomic operation log (4) and atomic operation log (5) are both file file2 under directory dir1, the operation object in atomic operation log (6) is directory dir2, and the operation objects in atomic operation log (7) are file file1 under directory dir1 and directory dir2. Furthermore, since a file can only be created in a directory after it is created, and a file can only be written and renamed after it is created, atomic operation log (2) and atomic operation log (4) depend on atomic operation log (1), atomic operation log (3) depends on atomic operation log (2), atomic operation log (5) depends on atomic operation log (4), and atomic operation log (7) depends on atomic operation log (3) and atomic operation log (6). Based on the above relationship, we can construct the following: Figure 5 The operation dependency graph shown. Figure 5 The two directed edges derived from child node A point to child node C and child node D respectively, the directed edge derived from child node B points to child node G, the directed edge derived from child node C points to child node E, the directed edge derived from child node D points to child node F, and the directed edge derived from child node E points to child node G.

[0087] S103: The storage gateway synchronizes the updated data in the file system to the cloud according to the operation dependency graph.

[0088] In a specific embodiment, an operation dependency graph includes multiple layers, each layer includes at least one node, and the multiple layers include a target layer. The storage gateway synchronizes updated data in the file system to the cloud based on the operation dependency graph. The specific process is as follows: the storage gateway traverses multiple nodes in the operation dependency graph in a preset manner to determine the dependency relationships between the atomic operation logs corresponding to the multiple nodes. The preset manner is: after traversing all nodes in the target layer, the storage gateway traverses all nodes in the layer below the target layer; the storage gateway synchronizes updated data in the file system to the cloud based on the dependency relationships between the atomic operation logs corresponding to the multiple nodes. Optionally, the preset manner is a breadth first search (BFS) algorithm.

[0089] In a more specific embodiment, at least one child node includes a first child node and a second child node, the first child node corresponds to a first atomic operation log, and the second child node corresponds to a second atomic operation log. The storage gateway selects different methods to synchronize the updated data in the file system to the cloud based on whether there is a dependency relationship between the first atomic operation log corresponding to the first child node and the second atomic operation log corresponding to the second child node in the operation dependency graph. The specific process is: when there is no dependency relationship between the first atomic operation log and the second atomic operation log, the storage gateway concurrently replays the first atomic operation log and the second atomic operation log to obtain the updated data in the file system, and then synchronizes the updated data in the replayed file system to the cloud. When there is a dependency relationship between the first atomic operation log and the second atomic operation log, and the second atomic operation log depends on the first atomic operation log, the storage gateway replays the first atomic operation log and the second atomic operation log in sequence to obtain the updated data in the file system, and then synchronizes the updated data in the replayed file system to the cloud.

[0090] In one possible implementation, when there is no dependency between the first atomic operation log and the second atomic operation log, the storage gateway allocates different working threads (hereinafter referred to as threads) to the first atomic operation log and the second atomic operation log respectively, so that the storage gateway can replay the first atomic operation log and the second atomic operation log concurrently.

[0091] In another possible implementation, when there is no dependency between the first atomic operation log and the second atomic operation log, the storage gateway assigns different work coroutines to the first atomic operation log and the second atomic operation log respectively, so that the storage gateway can replay the first atomic operation log and the second atomic operation log concurrently.

[0092] Taking the example of the storage gateway using BFS to traverse all nodes in the operation dependency graph and thus assigning different threads to atomic operation logs that have no dependencies, this paper explains how the storage gateway assigns threads to atomic operation logs. The specific process is as follows: First, starting from the root node in the operation dependency graph, the storage gateway traverses all child nodes adjacent to the root node. Since the child nodes adjacent to the root node are only pointed to by directed edges derived from the root node, the storage gateway determines that there is no dependency between the atomic operation logs corresponding to the child nodes adjacent to the root node. Then, the storage gateway assigns a thread to each atomic operation log corresponding to the child node adjacent to the root node. Then, starting from each of the child nodes adjacent to the root node, all adjacent child nodes are traversed. If the adjacent child nodes are only pointed to by directed edges from the starting point, the storage gateway determines that there is no dependency between the atomic operation logs corresponding to the adjacent child nodes. Then, the storage gateway allocates a thread to each atomic operation log corresponding to the adjacent child node, and so on, until all nodes in the operation dependency graph have been traversed and threads have been allocated to the atomic operation logs corresponding to the child nodes without dependencies. Then, the storage gateway can replay these atomic operation logs by executing the above threads to obtain the updated data in the file system.

[0093] To understand how the storage gateway uses BFS to allocate threads for atomic operation logs, Figure 4 Taking the operation dependency diagram shown as an example, the above process is described in detail: Figure 5 As shown,

[0094] First, the storage gateway starts from the root node and traverses the next layer of child nodes adjacent to the root node (including child node A and child node B). Since child node A is only pointed to by directed edges derived from the root node, and child node B is only pointed to by directed edges derived from the root node, there is no dependency between the atomic operation log corresponding to child node A and the atomic operation log corresponding to child node B. The storage gateway creates thread 1 for the atomic operation log corresponding to child node A and thread 2 for the atomic operation log corresponding to child node B.

[0095] Then, the storage gateway starts from child node A and traverses the next layer of child nodes adjacent to child node A (including child node C and child node D). Since child node C is only pointed to by the directed edge derived from child node A, and child node D is only pointed to by the directed edge derived from child node A, there is no dependency between the atomic operation log corresponding to child node C and the atomic operation log corresponding to child node D. The storage gateway creates thread 3 for the atomic operation log corresponding to child node C and thread 4 for the atomic operation log corresponding to child node D.

[0096] Then, the storage gateway starts from child node B and traverses the child nodes adjacent to child node B (including child node G). Since child node G is pointed to by the directed edge derived from child node B and child node E, the atomic operation log corresponding to child node G needs to be replayed after the atomic operation log corresponding to child node B and the atomic operation log corresponding to child node E are replayed.

[0097] Then, the storage gateway starts from child node C and traverses the child nodes adjacent to child node C (including child node E). Since child node E is only pointed to by the directed edge issued by child node C, the storage gateway can replay the atomic operation log corresponding to child node E after replaying the atomic operation log corresponding to child node C. In other words, the atomic operation log corresponding to child node C and the atomic operation log corresponding to child node E can be replayed successively in the same thread (thread 3).

[0098] Then, the storage gateway starts from child node D and traverses the child nodes adjacent to child node D (including child node F). Since child node F is only pointed to by the directed edge derived from child node D, the storage gateway can replay the atomic operation log corresponding to child node F after replaying the atomic operation log corresponding to child node D. In other words, the atomic operation log corresponding to child node D and the atomic operation corresponding to child node F can be replayed successively in the same thread (thread 4).

[0099] Then, the storage gateway starts from child node G and traverses the child nodes adjacent to child node G to determine that all nodes in the operation dependency graph have been traversed. At this time, the storage gateway ends the traversal and creates thread 5 for the atomic operation log corresponding to child node G. After thread 2 and thread 3 are executed, thread 5 is executed.

[0100] In a specific embodiment, when the cloud loses the updated data in the file system, the storage gateway synchronizes the updated data in the file system to the cloud again. It is understandable that after the cloud obtains the above-mentioned updated data, the user may send an erroneous instruction at a certain moment, thereby deleting the updated data from the cloud, or the cloud may fail at a certain moment, etc. In these cases, the cloud will lose the above-mentioned updated data. Since the storage gateway in this application stores both the atomic operation log and the metadata persistently, the user can send an instruction to the storage gateway so that the storage gateway constructs the operation dependency graph again according to the atomic operation log, and replays the operation dependency graph to obtain the above-mentioned updated data, and then synchronizes the updated data obtained by replaying it to the cloud again, so that the cloud can obtain the above-mentioned updated data again.

[0101] It is worth noting that the aforementioned content S101 describes that the storage gateway records an atomic operation as an atomic operation log. The purpose of this is to ensure that each atomic operation log is unique, thereby increasing the speed at which CSG synchronizes data to OBS. The specific reasons are as follows: If multiple atomic operations are recorded in an atomic operation log, and the multiple atomic operations do not have dependencies, then the operation dependency graph constructed by the storage gateway based on the atomic operation log will not be able to display the relationship between these multiple atomic operations, and it will not be possible to allocate multiple threads to these multiple atomic operation logs that do not have dependencies, thereby reducing the speed at which the storage gateway synchronizes updated data to the cloud. Figure 4 For example, assuming that the storage gateway atomic operations (1) to (5) are all recorded in the same atomic operation log (corresponding to child node A), then the child nodes in the operation dependency graph constructed by the storage gateway only include: child node A, child node B, and child node G. Correspondingly, the storage gateway will allocate 3 threads (including thread 1, thread 2, and thread 5) to replay these atomic operation logs to obtain the updated data in the file system. Compared with Figure 5 When replaying the five threads shown, using three threads will increase the time required to obtain the updated data in the file system, thereby reducing the speed at which the storage gateway synchronizes the updated data in the file system to the cloud. In addition, if multiple atomic operations are recorded in an atomic operation log, and compared to a single atomic operation, multiple atomic operations have different combinations, then more atomic operation logs need to be predefined. Therefore, in this application, the storage gateway records one atomic operation as one atomic operation log.

[0102] In the above scheme, when the file system is updated, the storage gateway only needs to record the atomic operation log corresponding to the updated data, and build an operation dependency graph based on the recorded atomic operation log, so that when the data is synchronized, according to the directed edges between the nodes in the operation dependency graph, the atomic operation log can be replayed concurrently to obtain the updated data in the file system, and then the updated data can be synchronized to the cloud. Compared with the synchronization request blocking mechanism mentioned in the above content, when multiple application servers access the file system at the same time, the storage gateway can use the data synchronization method provided by the present application to replay the atomic operation logs corresponding to the multiple updated data concurrently, thereby speeding up the storage gateway to obtain the multiple updated data. Compared with the periodic backup mechanism mentioned in the above content, the data synchronization method provided by the present application does not require a full scan of the file system, and can also replay the atomic operation log concurrently to obtain the updated data. Therefore, the data synchronization method provided by the present application can increase the speed at which the storage gateway synchronizes the updated data to the cloud.

[0103] The above content elaborates on the method of the embodiment of the present application in detail. In order to better implement the method provided by the embodiment of the present application, the following will introduce the relevant equipment and system provided by the embodiment of the present application for cooperating in implementing the above method.

[0104] like Figure 6 As shown, Figure 6 A schematic structural diagram of a storage gateway provided in an embodiment of the present application is shown. The storage gateway includes a recording unit 210, a construction unit 220 and a data synchronization unit 230.

[0105] The recording unit 210 is used to record at least one atomic operation log when the file system is updated, wherein the at least one atomic operation log is used to record the corresponding operation process when the file system is updated.

[0106] The construction unit 220 is configured to construct an operation dependency graph according to at least one atomic operation log.

[0107] The data synchronization unit 230 is used to synchronize updated data in the file system to the cloud according to the operation dependency graph.

[0108] In a specific embodiment, when the application server performs a target operation on the file system, the file system is updated, and the target operation includes at least one of the following: adding data in the file system, deleting data in the file system, and changing data in the file system.

[0109] In a specific embodiment, the recording unit 210 is specifically used to: when the file system is updated, perform at least one atomic operation, and the at least one atomic operation includes one or more of the following: create a directory or file, delete a directory or file, write a file, and rename a directory or file; record the at least one atomic operation as at least one atomic operation log, wherein one atomic operation corresponds to one atomic operation log.

[0110] In a specific embodiment, each atomic operation log in at least one atomic operation log records an operation object, and the operation object includes a directory and / or a file. The construction unit 220 is specifically used to: construct an operation dependency graph based on the tree relationship between at least one operation object recorded in at least one atomic operation log.

[0111] In a specific embodiment, the operation dependency graph includes multiple nodes and at least one directed edge, the multiple nodes include a root node and at least one child node, each child node corresponds to an atomic operation log, and each directed edge indicates a dependency relationship between two atomic operation logs.

[0112] In a specific embodiment, the operation dependency graph includes multiple layers, each layer in the multiple layers includes at least one node, and the multiple layers include a target layer. The data synchronization unit 230 is specifically used to: traverse multiple nodes in the operation dependency graph in a preset manner, so as to determine the dependency relationship between the atomic operation logs corresponding to the multiple nodes, wherein the preset manner is: after the storage gateway traverses all nodes in the target layer, it traverses all nodes in the next layer of the target layer; according to the dependency relationship between the atomic operation logs corresponding to the multiple nodes, the updated data in the file system is synchronized to the cloud.

[0113] In a specific embodiment, at least one child node includes a first child node and a second child node, the first child node corresponds to a first atomic operation log, and the second child node corresponds to a second atomic operation log, and the data synchronization unit 230 is specifically used to: when there is no dependency relationship between the first atomic operation log and the second atomic operation log, the first atomic operation log and the second atomic operation log are replayed concurrently to obtain updated data in the file system, and then the updated data in the replayed file system is synchronized to the cloud; when there is a dependency relationship between the first atomic operation log and the second atomic operation log, and the second atomic operation log depends on the first atomic operation log, the first atomic operation log and the second atomic operation log are replayed in sequence to obtain updated data in the file system, and then the updated data in the replayed file system is synchronized to the cloud.

[0114] In a specific embodiment, the data synchronization unit 230 is further configured to synchronize the updated data in the file system to the cloud again when the cloud loses the updated data in the file system.

[0115] In a specific embodiment, the storage gateway also includes a metadata management unit 240, which is used to update the metadata in the file system when the file system is updated, wherein the storage gateway updates the metadata and records at least one atomic operation log in the same transaction.

[0116] In a specific implementation, as an embodiment, the storage gateway also includes a receiving unit 250 and a cache management unit 260, wherein the receiving unit 250 provides a network system file system protocol (for example, NFS protocol, CIFS protocol) with external POSIX semantics, so that the application server can mount the file system and access the file system. The receiving unit 250 is specifically used to receive target operations and other related steps sent by the application server. The cache management unit 260 is used to store updated data in the file system. Specifically, after the storage gateway obtains updated data in the file system based on the atomic operation log playback, it can also store the updated data in the file system obtained by these playbacks in the cache management unit 260, so that the storage gateway can directly synchronize the updated data in the file system stored in the cache management unit 260 to the cloud to speed up data synchronization.

[0117] The storage gateway of the embodiment of the present application is merely illustrated by the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, that is, the internal structure of the storage gateway can be divided into different functional modules to complete all or part of the functions described above. In addition, the storage gateway provided in the above embodiment and the storage gateway in the above method embodiment are based on the same concept. The specific implementation process is detailed in the above method embodiment and will not be repeated here.

[0118] like Figure 7 As shown, Figure 7 3 shows a schematic diagram of the structure of another storage gateway provided by an embodiment of the present application, which includes a processor 310, a communication interface 320 and a memory 330. The processor 310, the communication interface 320 and the memory 330 are coupled via a bus 340.

[0119] The processor 310 can be a central processing unit (CPU), a general-purpose processor, a DSP, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device (PLD), a CPLD, a transistor logic device, a hardware component, or any combination thereof. The processor 310 can implement or execute the various exemplary methods described in conjunction with the embodiments of the present application. Specifically, the processor 310 reads the program code stored in the memory 330 and cooperates with the communication interface 320 to execute some or all of steps S101-S103.

[0120] Communication interface 320 can be a wired or wireless interface for communicating with other modules or devices. Wired interfaces can be Ethernet, controller area network, local interconnect network (LIN), or FlexRay interfaces. Wireless interfaces can be cellular or wireless LAN interfaces. Specifically, communication interface 220 can connect to other devices (e.g., application servers, cloud computing) to enable the storage gateway to synchronize local data with the cloud.

[0121] The memory 330 may include a volatile memory, such as a random access memory (RAM); the memory 330 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD); the memory 330 may also include a combination of the above types of memory. The memory 330 may store program code and program data. The program code is composed of Figure 6 The program data consists of the code of some or all units in the storage gateway shown, for example, the code of the recording unit 210, the code of the construction unit 220, the code of the data synchronization unit 230, the code of the metadata management unit 240, the code of the receiving unit 250 and the code of the cache management unit 260. Figure 6 The data generated by the storage gateway in the process of running the program shown, such as atomic operations, metadata, atomic operation logs, operation dependency graphs, etc.

[0122] The bus 340 can be a controller area network (CAN) or other internal bus. The bus 340 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0123] The storage gateway in the embodiment of the present application is used to execute the method executed by the storage gateway in the above method embodiment, which belongs to the same concept as the above method embodiment. Its specific implementation process is detailed in the above method embodiment and will not be repeated here.

[0124] The present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computing device (e.g., Figure 6 or Figure 7When running on the storage gateway shown in the figure, the computing device is caused to execute the method executed by the storage gateway in the above method embodiment.

[0125] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The above-mentioned computer program product includes one or more computer instructions. When the above-mentioned computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The above-mentioned computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The above-mentioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a storage disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as an SSD), etc. In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed devices can also be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the indirect coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0127] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present application.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0129] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium may include, for example: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0130] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: The storage gateway detects that the application server performs a target operation on the file system, where the target operation is used to update data in the file system; The storage gateway performs at least one atomic operation according to the target operation, where the at least one atomic operation includes one or more of the following: creating a directory or file, deleting a directory or file, renaming a directory or file, and writing a file; The storage gateway records the at least one atomic operation as at least one atomic operation log according to a type of the at least one atomic operation, wherein the at least one atomic operation log is used to record an operation process corresponding to the file system update, and operations included in the atomic operation corresponding to each atomic operation log need to be executed continuously; The storage gateway updates metadata in the file system, wherein the storage gateway updates the metadata and records the at least one atomic operation log in the same transaction; The storage gateway constructs an operation dependency graph according to the at least one atomic operation log; The storage gateway synchronizes the updated data in the file system to the cloud according to the operation dependency graph.

2. The method according to claim 1, characterized in that One atomic operation corresponds to one atomic operation log.

3. The method according to claim 1 or 2, characterized in that Each atomic operation log in the at least one atomic operation log records an operation object, where the operation object includes a directory and / or a file. The storage gateway constructs an operation dependency graph according to the at least one atomic operation log, including: The storage gateway constructs the operation dependency graph according to the tree relationship between at least one operation object recorded in the at least one atomic operation log.

4. The method according to any one of claims 1 to 3, characterized in that The operation dependency graph includes multiple nodes and at least one directed edge. The multiple nodes include a root node and at least one child node. Each child node corresponds to an atomic operation log. Each directed edge indicates a dependency relationship between two atomic operation logs.

5. The method according to claim 4, characterized in that The operation dependency graph includes multiple layers, each layer includes at least one node, and the multiple layers include a target layer. The storage gateway synchronizes updated data in the file system to the cloud according to the operation dependency graph, including: The storage gateway traverses multiple nodes in the operation dependency graph in a preset manner to determine dependency relationships between atomic operation logs corresponding to the multiple nodes, wherein the preset manner is: after traversing all nodes in the target layer, the storage gateway traverses all nodes in a layer below the target layer; The storage gateway synchronizes the updated data in the file system to the cloud according to the dependency relationship between the atomic operation logs corresponding to the multiple nodes.

6. The method according to claim 5, characterized in that The at least one child node includes a first child node and a second child node, the first child node corresponds to a first atomic operation log, and the second child node corresponds to a second atomic operation log, and the storage gateway synchronizes updated data in the file system to the cloud according to a dependency relationship between the atomic operation logs corresponding to the multiple nodes, including: When there is no dependency relationship between the first atomic operation log and the second atomic operation log, the storage gateway concurrently synchronizes the first atomic operation log and the second atomic operation log to the cloud; When there is a dependency relationship between the first atomic operation log and the second atomic operation log, and the second atomic operation log depends on the first atomic operation log, the storage gateway replays the first atomic operation log and the second atomic operation log in sequence to obtain the updated data in the file system, and then synchronizes the updated data in the file system obtained by replay to the cloud.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When the cloud loses the updated data in the file system, the storage gateway synchronizes the updated data in the file system to the cloud again.

8. The method according to any one of claims 1 to 7, characterized in that The target operation includes at least one of the following: adding data in the file system, deleting data in the file system, and changing data in the file system.

9. A storage gateway, characterized in that: The storage gateway includes a recording unit, a construction unit and a data synchronization unit. The recording unit is configured to detect that an application server performs a target operation on a file system, the target operation being used to update data in the file system; and perform at least one atomic operation according to the target operation, the at least one atomic operation comprising one or more of the following: creating a directory or a file, deleting a directory or a file, renaming a directory or a file, and writing a file; Recording the at least one atomic operation as at least one atomic operation log according to the type of the at least one atomic operation, wherein the at least one atomic operation log is used to record the operation process corresponding to the file system update, and the operations included in the atomic operation corresponding to each atomic operation log need to be executed continuously; updating metadata in the file system, wherein the storage gateway updates the metadata and records the at least one atomic operation log in the same transaction; The construction unit is configured to construct an operation dependency graph according to the at least one atomic operation log; The data synchronization unit is used to synchronize the updated data in the file system to the cloud according to the operation dependency graph.

10. The gateway according to claim 9, characterized in that One atomic operation corresponds to one atomic operation log.

11. The gateway according to claim 9 or 10, characterized in that: Each atomic operation log in the at least one atomic operation log records an operation object, where the operation object includes a directory and / or a file. The construction unit is specifically configured to: The operation dependency graph is constructed according to the tree relationship between at least one operation object recorded in the at least one atomic operation log.

12. The gateway according to any one of claims 9 to 11, characterized in that: The operation dependency graph includes multiple nodes and at least one directed edge. The multiple nodes include a root node and at least one child node. Each child node corresponds to an atomic operation log. Each directed edge indicates a dependency relationship between two atomic operation logs.

13. The gateway according to claim 12, characterized in that The operation dependency graph includes multiple layers, each of the multiple layers includes at least one node, the multiple layers include a target layer, and the data synchronization unit is specifically configured to: Traversing multiple nodes in the operation dependency graph in a preset manner to determine dependency relationships between atomic operation logs corresponding to the multiple nodes, wherein the preset manner is: after traversing all nodes in the target layer, the storage gateway traverses all nodes in a layer below the target layer; According to the dependency relationship between the atomic operation logs corresponding to the multiple nodes, the updated data in the file system is synchronized to the cloud.

14. The gateway according to claim 13, wherein: The at least one child node includes a first child node and a second child node, the first child node corresponds to a first atomic operation log, and the second child node corresponds to a second atomic operation log, and the data synchronization unit is specifically configured to: When there is no dependency relationship between the first atomic operation log and the second atomic operation log, the first atomic operation log and the second atomic operation log are concurrently replayed to obtain updated data in the file system, and then the updated data in the file system obtained by replaying is synchronized to the cloud; When there is a dependency relationship between the first atomic operation log and the second atomic operation log, and the second atomic operation log depends on the first atomic operation log, the first atomic operation log and the second atomic operation log are replayed in sequence to obtain the updated data in the file system, and then the updated data in the file system obtained by replay is synchronized to the cloud.

15. The gateway according to any one of claims 9 to 14, characterized in that: The data synchronization unit is further configured to synchronize the updated data in the file system to the cloud again when the cloud loses the updated data in the file system.

16. The gateway according to any one of claims 9 to 15, characterized in that: The target operation includes at least one of the following: adding data in the file system, deleting data in the file system, and changing data in the file system.

17. A storage gateway, characterized in that: The storage gateway includes a processor and a memory, and the processor executes code in the memory to implement the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that Computer instructions are stored, and the computer instructions are used to implement the method according to any one of claims 1 to 8.

19. A data synchronization system, characterized in that: The data synchronization system includes an application server, a storage gateway and a cloud, wherein the storage gateway is used to implement the method according to any one of claims 1 to 8.

Images