File synchronization method, computing device and disaster recovery file system

Abstract

The embodiment of the invention provides a file synchronization method, a computing device and a disaster recovery file system, which can roll back a file operation which is not backed up by a first computing device and then synchronize a file operation from a second computing device, so that the influence of the file operation which is not backed up on a file synchronization result can be removed, and inconsistency of main and standby file data is prevented. The method comprises the steps that a first computing device is switched from a main end to a standby end, after a file operation log from a second computing device is received, metadata operation after a consistency point in a local log is rolled back in an inverted order, and a write operation result is synchronized as last write data; and sequentially executing file operations after the consistency point in the file operation log from the second computing device to realize file synchronization.

Description

Technical Field

[0001] This application relates to the field of storage, and more particularly to a file synchronization method, a computing device, and a disaster recovery file system. Background Technology

[0002] In a disaster recovery file system, file synchronization is the method of synchronizing files between the primary and backup ends.

[0003] One file synchronization method is roughly as follows: After the first server (i.e., the master server) fails, the first server and the second server (i.e., the backup server) switch over. After the second server uses the file operation log to replay the log, the second server can act as the new master server and provide file read and write functions to users. When the first server recovers from the failure state to the normal state, the first server performs file synchronization based on the file operation log from the second server.

[0004] In practical applications, the first server may have some file operations that are not synchronized with the second server. These file operations may affect file operations from the second server, resulting in inconsistencies between the primary and backup file data. Summary of the Invention

[0005] This application provides a file synchronization method, a computing device, and a disaster recovery file system, which can synchronize write operation results or rollback metadata operations, thereby removing the impact of file operation results not backed up by the first computing device on the file synchronization results and preventing inconsistency between primary and backup file data.

[0006] A first aspect provides a file synchronization method, comprising: after a first computing device switches from a master to a backup, receiving file operation logs from a second computing device; determining a consistency point based on a local log and the file operation logs from the second computing device; when file operations after the consistency point in the local log include metadata operations, the first computing device rolls back the metadata operations after the consistency point in the local log in reverse order; when file operations after the consistency point in the local log include write operations, the first computing device synchronizes with the second computing device the last write data corresponding to the write operation in the second computing device, and then sequentially executes the file operations after the consistency point in the file operation logs from the second computing device. Wherein, when the first computing device is the backup, the second computing device is the master. The local log is the file operation log of the first computing device, and the consistency point is the last file operation in the same file operation sequence in the file operation logs from the second computing device and the local log.

[0007] In this way, the first computing device can roll back the results of unbacked-up metadata operations to the consistency point, and then synchronize the metadata operations received by the second computing device and update the results of the unbacked-up write operations to the results of the last write operation, thereby achieving file synchronization, overcoming the impact of unbacked-up file operations in the first computing device on the file synchronization results, and preventing inconsistency between primary and backup file data.

[0008] In conjunction with the first aspect, in a first possible implementation, synchronizing the last written data corresponding to the write operation in the second computing device between the first computing device and the second computing device includes: after the first computing device sends write operation information to the second computing device, it receives the last written data corresponding to the write operation information sent by the second computing device, and then updates the data of the write operation with the last written data corresponding to the write operation information, thereby providing a feasible method for synchronizing write operation results. The write operation information includes the address of the write operation and the size of the file to be written.

[0009] In conjunction with the above possible implementations in the first aspect, in the second possible implementation, the first computing device rolls back the metadata operations after the consistency point in the local log in reverse order, which includes: the first computing device selecting the metadata operations after the consistency point in the local log in reverse order; when the metadata operation is a file attribute modification operation, the first computing device searches for the file attribute value before the modification in the local log according to the modification operation, and restores the modified file attribute value to the file attribute value before the modification; when the metadata operation is a file management operation, the first computing device restores the file of the management operation.

[0010] In conjunction with the second possible implementation of the first aspect, in the third possible implementation, file attributes include one or more of the following: file time, file permissions, or file size.

[0011] In conjunction with the second possible implementation of the first aspect, in the fourth possible implementation, the file recovery management operation by the first computing device includes: when the metadata operation is a creation operation, the first computing device deletes the file created by the operation; when the metadata operation is a linking operation, the first computing device deletes the file linked by the operation; when the metadata operation is a deletion operation, the first computing device searches for the deleted file in the file system of the first computing device according to the deletion operation and recovers the deleted file; when the metadata operation is a delinking operation, searches for the delinked file in the file system of the first computing device according to the delinking operation and recovers the delinked file; when the metadata operation is a rename operation, the first computing device searches for the file information of the rename operation in the local log, and the file information of the rename operation includes the source file identifier and the target file identifier; when the file information of the rename operation includes the target file existence identifier, the source file identifier, and the target file identifier, the first computing device renames the file with the target file identifier to the source file identifier and recovers the target file in the file system of the first computing device; when the file information of the rename operation includes the source file identifier and the target file identifier but does not include the target file existence identifier, the first computing device renames the file with the target file identifier to the source file identifier.

[0012] In conjunction with the possible implementations mentioned above in the first aspect, in the fifth possible implementation, the first computing device sequentially executes file operations from the second computing device after a consistency point in the file operation log, including: for file operations from the second computing device after a consistency point in the file operation log, the first computing device sequentially selects file operations to be processed; when the file operation to be processed is a write operation, the first computing device updates the data of the file operation to be processed in the first computing device to the data of the file operation to be processed in the second computing device; when the file operation to be processed is a metadata operation, the first computing device executes the file operation to be processed. The file operation log of the second computing device and the data of the file operations to be processed in the second computing device can be simultaneously sent by the second computing device to the first computing device.

[0013] A second aspect provides a computing device that can serve as a first computing device. This computing device includes a transceiver module, a rollback module, and a synchronization module. The transceiver module receives file operation logs from a second computing device after the first computing device switches from a master to a backup. The rollback module determines a consistency point based on a local log and the file operation logs from the second computing device. When file operations after the consistency point in the local log include metadata operations, the first computing device rolls back the metadata operations after the consistency point in the local log in reverse order. When file operations after the consistency point in the local log include write operations, the first computing device synchronizes with the second computing device the last write data corresponding to the write operation in the second computing device. The synchronization module sequentially executes file operations after the consistency point in the file operation logs from the second computing device.

[0014] In conjunction with the second aspect, in the first possible implementation, the rollback module is specifically used to send write operation information to the second computing device; receive the last written data corresponding to the write operation information sent by the second computing device; and update the data of the write operation to the last written data corresponding to the write operation information.

[0015] In conjunction with the above possible implementations of the second aspect, in the second possible implementation, the rollback module is specifically used to select metadata operations after the consistency point from the local log in reverse order; when the metadata operation is a file attribute modification operation, it searches for the file attribute value before the modification in the local log according to the modification operation, and restores the modified file attribute value to the file attribute value before the modification; when the metadata operation is a file management operation, it restores the file of the management operation.

[0016] In conjunction with the second possible implementation of the second aspect, in the third possible implementation, the file attributes include at least one of file time, file permissions, or file size.

[0017] In conjunction with the second possible implementation of the second aspect, in the fourth possible implementation, the rollback module is specifically used to: delete the file created when the metadata operation is a creation operation; delete the file linked when the metadata operation is a linking operation; when the metadata operation is a deletion operation, locate the deleted file in the file system of the first computing device according to the deletion operation and restore the deleted file; when the metadata operation is a delinking operation, locate the delinked file in the file system of the first computing device according to the delinking operation and restore the delinked file; when the metadata operation is a rename operation, search for the file information of the rename operation in the local log; when the file information of the rename operation includes the target file existence identifier, the source file identifier, and the target file identifier, rename the file with the target file identifier to the source file identifier and restore the target file in the file system of the first computing device; when the file information of the rename operation includes the source file identifier and the target file identifier but does not include the target file existence identifier, rename the file with the target file identifier to the source file identifier.

[0018] In conjunction with the above-mentioned possible implementations of the second aspect, in the fifth possible implementation, for file operations after the consistency point in the file operation log of the second computing device, the synchronization module is specifically used to sequentially select the file operations to be processed from them; when the file operation to be processed is a write operation, the data of the file operation to be processed in the first computing device is updated to the data of the file operation to be processed in the second computing device; when the file operation to be processed is a metadata operation, the file operation to be processed is executed.

[0019] For the explanations of terms, the specific steps for each module, and the beneficial effects in the second part, please refer to the corresponding descriptions in the first part.

[0020] The third aspect provides a disaster recovery file system, which includes the computing device in the second aspect or any possible implementation of the second aspect.

[0021] A fourth aspect provides a computing device cluster, characterized in that it includes at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform a file synchronization method as described in the first aspect or any possible implementation thereof.

[0022] The fifth aspect provides a computer-readable storage medium comprising computer program instructions; when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices executes a file synchronization method of the first aspect or any possible implementation thereof.

[0023] A sixth aspect provides a computer program product comprising instructions; when the instructions are executed by a cluster of computing devices, the cluster of computing devices causes the cluster of computing devices to perform a file synchronization method in the first aspect or any possible implementation thereof. Attached Figure Description

[0024] Figure 1A This is a schematic diagram of a high-availability file system supporting disaster recovery in an embodiment of this application;

[0025] Figure 1B This is a schematic diagram of an elastic file service system that supports disaster recovery in an embodiment of this application.

[0026] Figures 2A to 2C This is a diagram illustrating a file synchronization failure.

[0027] Figure 3 This is a flowchart of a file synchronization method in an embodiment of this application;

[0028] Figures 4A to 4D This is a schematic diagram of the file synchronization method in an embodiment of this application;

[0029] Figure 5 This is a structural diagram of a computing device in an embodiment of this application;

[0030] Figure 6 This is a structural diagram of a disaster recovery file system in an embodiment of this application;

[0031] Figure 7 This is a structural diagram of a computing device in an embodiment of this application;

[0032] Figure 8 This is a structural diagram of a computing device cluster in an embodiment of this application;

[0033] Figure 9 This is another structural diagram of the computing device cluster in the embodiments of this application. Detailed Implementation

[0034] First, let's introduce some of the terms used in this application:

[0035] A file system is a structured way of storing and organizing data files. All data in a computer consists of 0s and 1s; the series of 0s and 1s stored on the hardware medium are completely indistinguishable and unmanageable to us. Therefore, we use the concept of "files" to organize this data. Data used for the same purpose is grouped into different types of files according to the structural requirements of different applications. Different file extensions are usually used to indicate different types, and each file is given a name that is easy to understand and remember. When there are many files, we group them according to a certain method, and each group of files is placed in the same directory (or folder). Furthermore, in addition to files, directories can also have subdirectories (called subdirectories or subfolders), and all files and directories form a tree structure. This tree structure has a specific name: a file system. There are many types of file systems, the most common being Windows' FAT / FAT32 / NTFS, and Linux's EXT2 / EXT3 / EXT4 / XFS / BtrFS, etc. To facilitate searching, directories are listed level by level from the root node down to the file itself. The names of these directories, subdirectories, and files are concatenated using special characters (e.g., "\" in Windows / DOS, " / " in Unix-like systems). This string of characters is called a file path, such as " / etc / systemd / system.conf" in Linux or "C:\Windows\System32\taskmgr.exe" in Windows. A path is a unique identifier for accessing a specific file. For example, D:\data\file.exe in Windows is a file path, representing the file.exe file located in the data directory on the D partition.

[0036] File systems are built on block devices. They not only record file paths but also which blocks make up a file and which blocks contain directory / subdirectory information. Different file systems have different organizational structures. For ease of management, block devices like hard drives are typically divided into multiple logical block devices, also known as hard drive partitions. Conversely, since the capacity and performance of a single medium are limited, multiple physical block devices can be combined into a single logical block device using techniques such as various levels of RAID and JBOD. File systems can also be built on these logical block devices. In any case, application server applications do not need to know the specific location of the file they want to access on the underlying block device; they only need to send the file's filename / ID to the file system, which then retrieves the file path based on the filename / ID.

[0037] Common file access protocols include Network File System (NFS), Common Internet File System (CIFS), and Server Message Block (SMB), etc., and this application does not impose any restrictions on them.

[0038] A directory tree in a file system is a multi-level structure used to manage files. Within the directory tree, each file or directory contains metadata information. When a user accesses a file by pathname, the file system first performs path decoding to locate the target file and checks if the user has the correct permissions; secondly, the file system performs metadata operations, atomically updating the target file's metadata.

[0039] The file semantic layer is an abstraction layer in the file system that separates the names, attributes, and other information of files and directories from their actual storage locations. This allows for functions such as file renaming, moving, and permission management without changing the actual location of the files on the storage medium. File semantic layer replication refers to copying file operations from the primary file system to the secondary file system synchronously or asynchronously. By replaying these file operations, cross-file system replication is achieved, ensuring high availability of the file system.

[0040] File operations include metadata operations and data operations. Metadata is information describing data attributes, used to record storage location, historical information, resource information, and file records, etc. Metadata operations include creation operations (e.g., create), deletion operations (e.g., delete), opening directory operations (e.g., opendir), closing directory operations (e.g., closedir), reading directory operations (e.g., readdir), renaming operations (e.g., rename), linking operations (e.g., link), and unlinking operations (e.g., unlink). Data operations are file operations performed on data in the storage medium, such as write and read operations.

[0041] The `create` operation is used to create directories or files. The `delete` operation is used to delete empty directories or delete files. The `open` operation is used to open a directory. The `close` operation is used to close a directory after reading it to free up internal tablespace. The `read` operation is used to retrieve the next directory entry in an opened directory. The `rename` operation is used to change the name of a directory. The `link` operation is used to create a link from a file to the name specified by a path; the same file can appear in multiple directories. For a file that is unlinked, if it only appears in one directory, the unlink operation removes it from the file. If it appears in multiple directories, the unlink operation removes the link to the specified pathname, without affecting links to other pathnames.

[0042] In storage systems, write-ahead logging (WAL) is frequently used to execute write requests. WAL is a technique used to ensure data integrity; simply put, before actually executing a write request, the entire write operation is recorded—this record is the log. Subsequent modifications to the data are also saved in the log. The log is persisted to disk first, and only after successful log saving does the data storage operation proceed. During the storage process, if an error occurs at any step, the correct result can be obtained by replaying the operation log on disk.

[0043] It's important to note that file semantic layer copying differs from block copying. Block copying is reentrant, meaning that data can be repeatedly flushed to the same location on the same disk, with the data from the last flush becoming the final version. However, at the file semantic layer, many metadata operations on files are not reentrant.

[0044] The file synchronization method in this application is applied to a disaster recovery file system, which can be a high-availability file system that supports disaster recovery or an elastic file service system that supports disaster recovery. Figure 1A This demonstrates a highly available file system that supports disaster recovery. (See also...) Figure 1A In one embodiment, the client device can access the high-availability file system via the Internet. The client device can be a desktop computer, mobile phone, tablet computer, vehicle head unit, IoT device, wearable smart device, etc. The high-availability file system includes multiple computing devices, such as a first computing device and a second computing device. A computing device is a device with both computing and storage capabilities, such as a server or desktop computer. Both the first and second computing devices include file operation logs and hard disks. The hard disk can be, but is not limited to, a solid-state drive, a disk drive, or a disk array.

[0045] In a highly available file system, when the first computing device acts as the master, the second computing device acts as the backup. The first computing device provides the file system to users, who can perform file operations on the files within the file system, such as metadata or data operations. The first computing device records file operations in a file operation log and synchronizes the file operation records and data with the second computing device. Under normal conditions, the file operation logs of the first and second computing devices are identical.

[0046] When the first computing device fails, it switches from being the primary to the backup, and the second computing device switches from the backup to the primary. The second computing device can execute the file operation log, thereby providing the user with the current state of the file system. From the user's perspective, the file system provided by the first and second computing devices is the same file system.

[0047] It should be understood that Figure 1A The first and second computing devices shown can be located in the same data center or in different data centers. In a high-availability file system, there can be one or more computing devices serving as backups.

[0048] Figure 1B This paper illustrates an elastic file service system that supports disaster recovery. The elastic file service system can provide elastic file services. Elastic file service (SFS) provides high-performance file storage (such as network attached storage, NAS) that can be scaled on demand, offering shared access to multiple compute instances in the cloud. Compute instances can be, but are not limited to, elastic cloud servers (ECS), containers, or bare metal servers. Elastic cloud servers can also be called virtual machines or elastic instances. A virtual machine (VM) refers to a complete computer system with full hardware system functionality simulated by software, running in a completely isolated environment. A container isolates application processes and their dependencies in an independent running environment.

[0049] See Figure 1B In one embodiment, the client device connects to a virtual private cloud (VPN) via a network. The VPN has multiple computing instances, which may be located in the same region or different regions. The computing instances can connect to an elastic file service system via the network. The elastic file service system includes multiple computing devices, such as a first computing device and a second computing device. Both the first and second computing devices include file operation logs and hard disks. When the first computing device acts as the primary device, the second computing device acts as the backup device. The process of primary / backup switching between the first and second computing devices can be found in [reference needed]. Figure 1A The corresponding description in the illustrated embodiment.

[0050] The following section describes the file synchronization method in a primary / standby switchover scenario, using a first computing device and a second computing device as examples. Taking the first computing device as the primary and the second as the standby, when the first computing device is in normal operation, it provides file services to the user. When the user operates the file system of the first computing device through a client device, the first computing device records the file operations and data in a file operation log and synchronizes the log and data to the second computing device. The second computing device then replays these operation logs to keep the primary and standby file systems synchronized.

[0051] When the file system of the first computing device fails, it is degraded from the primary to the backup, and the second computing device is promoted from the backup to the primary. The second computing device then takes over the file read / write functionality from the first. During the primary / backup switchover, the primary device may have file operations that were not synchronized to the second computing device (i.e., unbacked-up file operations). These file operations may affect the replication of the file semantic layer. For example, after the file system of the first computing device fails, some file operations may not be synchronized to the second computing device. This can lead to a discrepancy between the data received by the second computing device and the data received by the first computing device (the old primary) when the second computing device begins receiving services. After the second computing device has been running for a period of time, the first computing device may have file operation information that is not present in the second computing device, but it has not synchronized some file operations received by the second computing device after the failure. When the first computing device recovers, the file operations that were not synchronized to the second computing device will affect the file operation results received by the second computing device after the failure, resulting in inconsistency between the primary and backup file data. This will be described in more detail below. Figures 2A-2C The file synchronization failure was indicated.

[0052] See Figure 2A The first computing device and the second computing device receive the same file operations, which include: create A(100) and create B(101), where the logical number of create A(100) is 1 and the logical number of create B(101) is 2. After executing the above file operations, A(100) and B(101) are created, where A is the filename of file 100 and B is the filename of file 101.

[0053] See Figure 2B When the first computing device fails, the first computing device switches from the master end to the backup end. The file operations that the first computing device does not back up include: rename B→A and create C(102). After rename B→A, delete A(100), name file 101 as A, and then execute create C(102) to obtain A(101) and C(102).

[0054] After the second computing device switches from the backup end to the master end, it receives file operations input by the user. The file operations input by the user include: rename A→B and mkdir A(102). When rename A→B, B(101) is deleted, file 100 is renamed to B, and then mkdir A(102) is executed to obtain B(100) and A(102).

[0055] See Figure 2C When the first computing device returns to normal, the first computing device and the second computing device synchronously rename A→B and mkdir A(102). When rename A→B is executed, file 101 is named B (i.e. B(101)). Executing mkdir A(102) will create A(102). At this time, A(102) and C(102) conflict, and the creation of A(102) fails.

[0056] The file operation results of the second computing device include B(100) and A(102), and the file operation results of the first computing device include B(101) and C(102). Therefore, a file synchronization error occurred. The applicant found that the file synchronization error was caused by the fact that some metadata operations were not reentrant. The rename B→A received by the first computing device after the failure affected the file operation result of the rename A→B received by the second computing device, and the create C(102) received by the first computing device after the failure affected the file operation result of the mkdir A(102) received by the second computing device.

[0057] In view of this, this application provides a file synchronization method. This application can also directly update the write operation results in the file operation log to the last written data and roll back the metadata operations, thus eliminating the impact of file operations not backed up by the first computing device on the file synchronization results. See also... Figure 3 One embodiment of the file synchronization method in this application includes:

[0058] S301, The first computing device switches from the master end to the backup end.

[0059] S302, the second computing device switches from the backup end to the master end.

[0060] In this embodiment, the first computing device acts as the primary terminal before the failure, and the second computing device acts as the backup terminal. The first and second computing devices synchronize file operations. After the first computing device fails, it switches from primary to backup, and the second computing device switches from backup to primary. During the primary / backup switchover, the first computing device may receive file operations from user equipment. These file operations are not synchronized between the first and second computing devices and are stored in a local log. These file operations can be considered abnormal file operations, and their metadata is non-reentrant. The write operation address or write data size may differ from the write operation information received by the second computing device. Therefore, both the metadata and the write operations may affect the file operations received by the second computing device after the primary / backup switchover.

[0061] S303, The user equipment sends a file operation to the second computing device.

[0062] After the second computing device switches from the backup end to the master end, the file operations input by the user are file operations that the second computing device needs to synchronize with the first computing device.

[0063] S304. The second computing device saves file operations from the user equipment in the file operation log of the second computing device.

[0064] S305, The second computing device sends the file operation log of the second computing device to the first computing device.

[0065] File operation logs include metadata operations and write operations. Since read operations are unrelated to modifying metadata and data, file operation logs may exclude read operations. Metadata operations include, but are not limited to, modifications to file attributes and file management operations. File attributes include, but are not limited to, one or more of file time, file permissions, or file size. File management operations include, but are not limited to, creation, linking, deletion, unlinking, renaming, or truncation operations.

[0066] S306. The first computing device determines the consistency point based on the local log and the file operation log of the second computing device.

[0067] The local log is the file operation log of the first computing device. The first computing device determines the same file operation sequence in the file operation log from the second computing device and the local log; the consistency point is the last file operation in that file operation sequence. When the consistency point is not the last file operation in the local log, it indicates that the first computing device has unbacked-up file operations, and these unbacked-up file operations need to be rolled back. Unbacked-up file operations are file operations after the consistency point in the local log; these can be considered file operations that the first computing device has not synchronized with the second computing device. When the consistency point is the last file operation in the local log, it indicates that the first computing device does not have any unbacked-up file operations.

[0068] S307. When file operations after a consistency point in the local log include metadata operations, the first computing device rolls back the metadata operations after the consistency point in the local log in reverse order.

[0069] In an optional embodiment, S307 includes: the first computing device selecting metadata operations after the consistency point from the local log in reverse order; when the metadata operation is a file attribute modification operation, the first computing device searches for the file attribute value before the modification in the local log according to the modification operation, and restores the modified file attribute value to the file attribute value before the modification; when the metadata operation is a file management operation, the first computing device restores the file of the management operation. Rolling back the metadata operation can restore the file system metadata to the consistency point, solving the impact of unbacked-up file operations on metadata.

[0070] S308. When file operations after a consistency point in the local log include write operations, the first computing device and the second computing device synchronize the last written data corresponding to the write operation in the second computing device.

[0071] Optionally, S308 includes: the first computing device sending write operation information to the second computing device, the write operation information including a write operation address and a write data size, the second computing device determining the write data based on the write operation information, the first computing device receiving the last write data sent by the second computing device based on the write operation information (i.e., the last write data corresponding to the write operation information), and updating the write operation data to the last write data.

[0072] It should be noted that when file operations after the consistency point in the local log include metadata operations and write operations, file operations can be selected in reverse order. When the selected file operation is a metadata operation, the metadata operation is rolled back. When the selected file operation is a write operation, the first computing device and the second computing device synchronize the last written data corresponding to the write operation in the second computing device.

[0073] S309. The first computing device sequentially executes the file operations following the consistency point in the file operation log of the second computing device.

[0074] Specifically, the first computing device can synchronize file operations after the consistency point in the file operation log from the second computing device to the local log, and then execute the file operations sequentially.

[0075] Optionally, S309 includes: for file operations from the file operation log of the second computing device after a consistency point, the first computing device sequentially selects file operations to be processed from them; when the file operation to be processed is a write operation, the first computing device updates the data of the file operation to be processed in the first computing device to the data of the file operation to be processed in the second computing device; when the file operation to be processed is a metadata operation, the first computing device executes the file operation to be processed. The file operation log of the second computing device and the data of the file operations to be processed in the second computing device can be simultaneously sent by the second computing device to the first computing device.

[0076] In this embodiment, for file operations that are not backed up by the first computing device, the write operation results in the above file operations are synchronized to the latest results, and the metadata operations in the above file operations are first rolled back to the consistency point. This can remove the impact of file operations received by the first computing device after the primary-backup switch on the file synchronization results. Then, the file operations received by the second computing device after the primary-backup switch are executed, so that the file system of the first computing device is synchronized with the file system of the second computing device, and the inconsistency of primary and backup file data is prevented.

[0077] Secondly, the local log can save file operations after the consistency point in the file operation log of the second computing device, thereby recording file operations before and after the failure without having to use multiple logs to record file operations before and after the failure, which can save the overhead of creating failure logs.

[0078] The method for managing rollback files in this application is described in detail below. In an optional embodiment, the files for which the first computing device restores management operations include:

[0079] When the metadata operation is a creation operation, the first computing device searches for the parent directory and file identifier of the file corresponding to the creation operation in the local log, and deletes the file corresponding to the creation operation based on the parent directory and file identifier.

[0080] When the metadata operation is a link operation, the first computing device searches for the parent directory and file identifier of the file corresponding to the link operation in the local log, and deletes the file corresponding to the link operation based on the parent directory and file identifier.

[0081] When the metadata operation is a deletion operation, the first computing device searches for the file to be deleted in the first computing device's file system according to the deletion operation and restores the deleted file.

[0082] When the metadata operation is a delinking operation, the first computing device searches for the delinking file in the file system of the first computing device according to the delinking operation and restores the delinking file.

[0083] When the metadata operation is a rename operation, the first computing device searches for the file information of the rename operation in the local log. When the file information of the rename operation includes the target file existence identifier, the source file identifier, and the target file identifier, the file with the target file identifier (i.e., the source file) is named the source file identifier, the target file identifier is associated with the target file in the file system, and the target file is restored in the file system of the first computing device. When the file information of the rename operation includes the source file identifier and the target file identifier but does not include the target file existence identifier, the file with the target file identifier (i.e., the source file) is named the source file identifier.

[0084] It should be noted that when performing a creation or linking operation before rollback, the first computing device saves the file parent directory and file identifier of the above operation in the local log.

[0085] When performing a delete or unlink operation before rollback, the first computing device delays the deletion of the file being deleted or unlinked. Delayed deletion means deleting the file after the primary and backup are synchronized; at this time, the file is still stored in the file system, which can be understood as hiding the file from the user.

[0086] When performing a renaming operation before rollback, if the file corresponding to the renaming operation includes the target file, the target file is delayed in being deleted. The first computing device saves the file information of the renaming operation (including the source file identifier, the target file identifier, and the target file existence identifier) ​​in its local log. The target file existence identifier is used to indicate that the target file is stored in the file system. If the file corresponding to the renaming operation does not include the target file, the first computing device saves the file information of the renaming operation (including the source file identifier and the target file identifier) ​​in its local log. For example, in the rename operation A→B, A is the source file identifier, and B is the target file identifier. Both the source file identifier and the target file identifier can be, but are not limited to, filenames.

[0087] This optional embodiment describes a rollback method for some metadata operations. When file operations after the consistency point also include other types of metadata operations, this application can also roll back other types of metadata operations, without specific limitations. For file operations after the consistency point in the local log, after the first computing device rolls back the above file operations in reverse order, the file system metadata can be restored to the consistency point, eliminating the results of file operations that the first computing device did not back up.

[0088] To facilitate understanding, a specific example is provided below to illustrate the file synchronization method of this application. (See attached document.) Figures 4A to 4D In this example, see Figure 4A The file operations received by both the first and second computing devices include: create A(100) and create B(101). The logical number of create A(100) is 1, and the file identified by A is 100. The logical number of create B(101) is 2, and the file identified by B is 101. After executing the above file operations, A(100) and B(101) can be created.

[0089] See Figure 4B When the first computing device fails, the file operations that the first computing device does not back up include: rename B→A and create C(102). After rename B→A, file 100 is deleted with a delay. File 101 is renamed to A. The existence identifier of the target file corresponding to file A, file B and file 100 is recorded in the file operation log. Executing the above file operations can obtain A(101) and C(102). The first computing device switches from the master end to the backup end.

[0090] The second computing device switches from the backup end to the master end. The second computing device receives file operations input by the user, including: rename A→B and mkdir A(102). After rename A→B, file 101 is deleted after a delay, file 100 is renamed to B, and file names A and B are recorded in the file operation log. Then, executing mkdir A(102) will yield B(100) and A(102).

[0091] See Figure 4C After the first computing device recovers from the fault state to the normal state, it rolls back create C(102) and rename B→A in reverse order. That is, it first deletes C(102), and then changes the file name of file 101 from A to B according to the target file existence identifier corresponding to file 100. It associates A with file 100 to obtain B(101) and A(100).

[0092] See Figure 4DAfter the first computing device receives and executes the rename A→B and mkdir A(102) commands from the second computing device, it deletes file 101, renames file 100 to B, and creates directory A(102). It can be seen that both the first and second computing devices obtain file synchronization results of B(100) and A(102), thus completing file synchronization.

[0093] It should be noted that when the target file identifier corresponds to the target file and the source file identifier corresponds to the source file, the handling of the target file differs when the master and slave perform the same renaming operation. The master will delay deleting the target file, effectively hiding it, while the slave will delete the target file. For example, if the source file 100 is named A and the target file 101 is named B, when the master executes `rename A→B`, it will delay deleting the target file 101; when the slave executes `rename A→B`, it will delete the target file 101.

[0094] See Figure 5 In one embodiment, this application provides a computing device 500 including a transceiver module 501, a rollback module 502, and a synchronization module 503. After the computing device 500 switches from a master to a backup, the transceiver module 501 is used to receive file operation logs from a second computing device; the rollback module 502 is used to determine a consistency point based on the local log and the file operation logs from the second computing device; when the file operations after the consistency point in the local log include metadata operations, the module rolls back the metadata operations after the consistency point in the local log in reverse order; when the file operations after the consistency point in the local log include write operations, the module synchronizes the last write data corresponding to the write operation in the second computing device with the second computing device; the synchronization module 503 is used to sequentially execute the file operations after the consistency point in the file operation logs from the second computing device.

[0095] The transceiver module 501, rollback module 502, and synchronization module 503 can all be implemented in software or hardware. For example, the implementation of synchronization module 503 will be described below. Similarly, the implementation of transceiver module 501 and rollback module 502 can refer to the implementation of synchronization module 503.

[0096] As an example of a software functional unit, the synchronization module 503 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance may be one or more. For example, the synchronization module 503 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0097] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0098] As an example of a hardware functional unit, the synchronization module 503 may include at least one computing device, such as a server. Alternatively, the synchronization module 503 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0099] The multiple computing devices included in the synchronization module 503 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the synchronization module 503 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the synchronization module 503 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0100] It should be noted that, in other embodiments, the transceiver module 501 can be used to perform... Figure 3 The rollback module 502 can be used to execute any step in the file synchronization method shown in the embodiment. Figure 3 The synchronization module 503 can be used to execute any step in the file synchronization method shown in the embodiment. Figure 3 In the file synchronization method shown in the embodiment, the steps implemented by the transceiver module 501, the rollback module 502, and the synchronization module 503 can be specified as needed. Different steps in the file synchronization method are implemented by the transceiver module 501, the rollback module 502, and the synchronization module 503 to realize all the functions of the first computing device or the second computing device.

[0101] In other embodiments, the rollback module 502 is specifically used to send write operation information to the second computing device; receive the last write data corresponding to the write operation information sent by the second computing device; and update the write operation data to the last write data corresponding to the write operation information.

[0102] In other embodiments, the rollback module 502 is specifically used to select metadata operations after the consistency point from the local log in reverse order; when the metadata operation is a file attribute modification operation, it searches for the file attribute value before the modification in the local log according to the modification operation, and restores the modified file attribute value to the file attribute value before the modification; when the metadata operation is a file management operation, it restores the file of the management operation.

[0103] In other embodiments, the rollback module 502 is specifically configured to: delete the file created when the metadata operation is a creation operation; delete the file linked when the metadata operation is a linking operation; search for the deleted file in the file system of the first computing device according to the deletion operation and restore the deleted file when the metadata operation is a deletion operation; search for the unlinked file in the file system of the first computing device according to the unlinking operation and restore the unlinked file when the metadata operation is a rename operation; search for the file information of the rename operation in the local log when the file information of the rename operation includes a target file existence identifier, a source file identifier, and a target file identifier; rename the file with the target file identifier to the source file identifier and restore the target file in the file system of the first computing device when the file information of the rename operation includes the source file identifier and the target file identifier but does not include the target file existence identifier; and rename the file with the target file identifier to the source file identifier.

[0104] In other embodiments, for file operations after a consistency point in the file operation log of the second computing device, the synchronization module 503 is specifically used to sequentially select file operations to be processed from them; when the file operation to be processed is a write operation, the data of the file operation to be processed in the first computing device is updated to the data of the file operation to be processed in the second computing device; when the file operation to be processed is a metadata operation, the file operation to be processed is executed.

[0105] In an alternative embodiment, when computing device 500 implements Figure 3 In the embodiment shown, when the second computing device functions, the transceiver module 501 is used to send file operation logs to the first computing device. When the transceiver module 501 receives write operation information sent by the first computing device, the synchronization module 503 is used to synchronize the last written data corresponding to the write operation information in the second computing device with the first computing device.

[0106] like Figure 6As shown, this application also provides a disaster recovery file system 600, which includes a first computing device 601 and a second computing device 602. The first computing device 601 is used to receive file operation logs from the second computing device 602 after switching from a primary end to a backup end, and to determine a consistency point based on the file operation logs of the first computing device 601 and the second computing device 602. When the file operations after the consistency point in the local log include metadata operations, the first computing device 601 rolls back the metadata operations after the consistency point in the local log in reverse order. When the file operations after the consistency point in the local log include write operations, the first computing device 601 and the second computing device 602 synchronize the last written data corresponding to the write operation in the second computing device 602. The first computing device 601 is also used to sequentially execute the file operations after the consistency point in the file operation logs of the second computing device 602.

[0107] In an optional embodiment, the first computing device 601 is specifically configured to send write operation information to the second computing device 602; receive the last written data corresponding to the write operation information sent by the second computing device 602; and update the data of the write operation to the last written data corresponding to the write operation information.

[0108] Both the first computing device 601 and the second computing device 602 can be implemented in software or in hardware. For example, the implementation of the first computing device 601 will be described below. Similarly, the implementation of the second computing device 602 can refer to the implementation of the first computing device 601.

[0109] As an example of a software functional unit, the first computing device 601 may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Further, the aforementioned computing device may be one or more. For example, the first computing device 601 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0110] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0111] As an example of a hardware functional unit, the first computing device 601 may include at least one computing device, such as a server. Alternatively, the first computing device 601 may also be a device implemented using an ASIC or a PLD. The aforementioned PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0112] The multiple computing devices included in the first computing device 601 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first computing device 601 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first computing device 601 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0113] This application also provides a computing device 700. For example... Figure 7 As shown, the computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0114] The 702 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 704 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 704 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 706, processor 704, communication interface 708).

[0115] Processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0116] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0117] The memory 706 stores executable program code, which the processor 704 executes to implement the functions of the aforementioned transceiver module 501, rollback module 502, and synchronization module 503, thereby realizing the file synchronization method. In other words, the memory 706 stores instructions for executing the file synchronization method.

[0118] The communication interface 703 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0119] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0120] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 of one or more computing devices 700 in the computing device cluster may store the same instructions for performing file synchronization methods.

[0121] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the file synchronization method. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for executing the file synchronization method.

[0122] It should be noted that the memory 706 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the computing device 500. That is, the instructions stored in the memory 706 of different computing devices 700 can implement the functions of the transceiver module 501, the rollback module 502, or the synchronization module 503.

[0123] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, the two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 706 in computing device 700A stores instructions for performing the functions of the first computing device 601. Simultaneously, the memory 706 in computing device 700B stores instructions for performing the functions of the second computing device 602.

[0124] It should be understood that Figure 9 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0125] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster. The difference is that the memory 706 of one or more computing devices 700 in the computing device cluster can store the same instructions for executing file synchronization methods.

[0126] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the file synchronization method. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for executing the file synchronization method.

[0127] The memory 706 in different computing devices 700 within the computing device cluster can store different instructions for executing some functions of the disaster recovery file system 600. That is, the instructions stored in the memory 706 of different computing devices 700 can implement the functions of the first computing device 601 or the second computing device 602.

[0128] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform a file synchronization method.

[0129] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a file synchronization method.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A file synchronization method, characterized in that, The method is applied to a disaster recovery file system, which includes a first computing device and a second computing device. When the first computing device acts as the primary endpoint, the second computing device acts as the backup endpoint. The method includes: After the first computing device switches from the master to the backup, it receives file operation logs from the second computing device. The first computing device determines a consistency point based on a local log and a file operation log from a second computing device. The consistency point is the last file operation in the same file operation sequence in the file operation log from the second computing device and the local log. When file operations after a consistency point in the local log include metadata operations, the first computing device rolls back the metadata operations after the consistency point in the local log in reverse order. When file operations after a consistency point in the local log include write operations, the first computing device and the second computing device synchronize the last written data corresponding to the write operation in the second computing device. The first computing device sequentially executes file operations following the consistency point in the file operation log from the second computing device.

2. The method according to claim 1, characterized in that, The last written data corresponding to the write operation in the second computing device, synchronized between the first computing device and the second computing device, includes: The first computing device sends write operation information to the second computing device; The first computing device receives the last written data corresponding to the write operation information sent by the second computing device; The first computing device updates the data of the write operation with the last written data corresponding to the write operation information.

3. The method according to claim 1 or 2, characterized in that, The first computing device performs metadata operations in reverse order after the consistency point in the local log, including: The first computing device performs metadata operations by selecting the consistency point from the local log in reverse order. When the metadata operation is a file attribute modification operation, the first computing device searches for the file attribute value before modification in the local log according to the modification operation, and restores the modified file attribute value to the file attribute value before modification. When the metadata operation is a file management operation, the first computing device restores the file of the management operation.

4. The method according to claim 3, characterized in that, The file attributes include at least one of file time, file permissions, or file size.

5. The method according to claim 3, characterized in that, The file for the first computing device to resume the management operation includes: When the metadata operation is a creation operation, the first computing device deletes the file created in the creation operation. When the metadata operation is a link operation, the first computing device deletes the file involved in the link operation. When the metadata operation is a deletion operation, the first computing device searches for the file to be deleted in the file system of the first computing device according to the deletion operation, and restores the file to be deleted. When the metadata operation is a delinking operation, the first computing device searches for the file in the file system of the first computing device according to the delinking operation and restores the file in the delinking operation. When the metadata operation is a rename operation, the first computing device searches for the file information of the rename operation in the local log; When the file information of the renaming operation includes a target file existence identifier, a source file identifier, and a target file identifier, the first computing device renames the file with the target file identifier to the source file identifier and restores the target file in the file system of the first computing device; When the file information of the renaming operation includes a source file identifier and a target file identifier, but the file information of the renaming operation does not include a target file existence identifier, the first computing device renames the file with the target file identifier to the source file identifier.

6. The method according to any one of claims 1 to 5, characterized in that, The first computing device sequentially executes file operations following the consistency point in the file operation log from the second computing device, including: For file operations following the consistency point in the file operation log from the second computing device, the first computing device sequentially selects file operations to be processed from them. When the file operation to be processed is a write operation, the first computing device updates the data of the file operation to be processed in the first computing device to the data of the file operation to be processed in the second computing device; When the file operation to be processed is a metadata operation, the first computing device executes the file operation to be processed.

7. A computing device, characterized in that, The computing device is used as a first computing device, the computing device comprising: The transceiver module is used to receive file operation logs from the second computing device after switching from the master end to the backup end; The rollback module is used to determine a consistency point based on the local log and the file operation log from the second computing device. The consistency point is the last file operation in the same file operation sequence in the file operation log from the second computing device and the local log. When the file operations after the consistency point in the local log include metadata operations, the metadata operations after the consistency point in the local log are rolled back in reverse order. When the file operations after the consistency point in the local log include write operations, the last written data corresponding to the write operation in the second computing device is synchronized with the second computing device. A synchronization module is used to sequentially execute file operations following the consistency point in the file operation log from the second computing device.

8. The apparatus according to claim 7, characterized in that, The rollback module is specifically used to send write operation information to the second computing device; receive the last write data corresponding to the write operation information sent by the second computing device; and update the write operation data to the last write data corresponding to the write operation information.

9. The apparatus according to claim 7 or 8, characterized in that, The rollback module is specifically used to perform metadata operations after selecting a consistency point in reverse order from the local log; when the metadata operation is a file attribute modification operation, it searches for the file attribute value before the modification in the local log according to the modification operation, and restores the modified file attribute value to the file attribute value before the modification; when the metadata operation is a file management operation, it restores the file of the management operation.

10. The apparatus according to claim 9, characterized in that, The file attributes include at least one of file time, file permissions, or file size.

11. The apparatus according to claim 9, characterized in that, The rollback module is specifically used to delete the file created during the metadata operation and to delete the file linked during the metadata operation when the metadata operation is a creation operation. When the metadata operation is a deletion operation, the file to be deleted is located in the file system of the first computing device according to the deletion operation, and the file to be deleted is restored. When the metadata operation is a delinking operation, the file of the delinking operation is located in the file system of the first computing device according to the delinking operation, and the file of the delinking operation is restored. When the metadata operation is a renaming operation, the file information of the renaming operation is searched in the local log; when the file information of the renaming operation includes a target file existence identifier, a source file identifier, and a target file identifier, the file with the target file identifier is renamed to the source file identifier, and the target file is restored in the file system of the first computing device; when the file information of the renaming operation includes a source file identifier and a target file identifier, but the file information of the renaming operation does not include a target file existence identifier, the file with the target file identifier is renamed to the source file identifier.

12. The apparatus according to any one of claims 7 to 11, characterized in that, For file operations following the consistency point in the file operation log from the second computing device, the synchronization module is specifically used to sequentially select file operations to be processed from them; when the file operation to be processed is a write operation, the data of the file operation to be processed in the first computing device is updated to the data of the file operation to be processed in the second computing device; when the file operation to be processed is a metadata operation, the file operation to be processed is executed.

13. A disaster recovery file system, characterized in that, The computing device includes any one of claims 7 to 12.

14. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 6.

15. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1 to 6.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 6.

Images