Dynamic file mirroring method and system based on serialized operation log

Abstract

The invention discloses a dynamic file mirroring method and system based on a serialized operation log. The method comprises the steps that S1, a rule module of a source end sends a rule capture strategy and a rule mirroring strategy to an operation capture module and a file mirroring module respectively; s2, calling a file mirroring module to perform comparison and mirroring of specified file directories from the source end to the standby end according to a rule mirroring strategy; s3, an operation capture module captures IO operation of an application layer in real time according to a rule capture strategy, serializes the IO operation and stores the IO operation in an operation cache module; step S4, when the mirror image comparison task is completed, recording the time point of mirror image completion, and transmitting the serialized IO operation log to the standby end by the operation cache module; and S5, the standby end uses an operation application module to operate and apply the serialized IO operation logs before the mirroring completion time point to a file system of the standby end according to a sequence, so that the data of the source end and the standby end at the mirroring completion time point are consistent.

Description

Technical Field

[0001] The present invention relates to the technical field of computer disaster recovery and backup, and in particular to a dynamic file mirroring method and system based on serialized operation logs. Background Art

[0002] In the file system, file comparison and mirroring are basic technologies and methods based on file system backup. In the prior art, in order to ensure data consistency, it is generally necessary to freeze or snapshot the file system on the production side, and then compare and mirror it to achieve full or incremental backup.

[0003] However, the method of taking snapshots first and then synchronizing the images in the prior art has the following disadvantages: the snapshot function depends on the snapshot support of the file system on the production side. For file systems that do not support snapshots, such as the Linux ext series file systems, an additional snapshot module needs to be loaded to support it. Another disadvantage of the snapshot system is that the snapshot system needs to support the application system, and the application system needs to provide snapshot components. For example, database applications such as Oracle and Sql Server need to install corresponding snapshot components to support the snapshot function of the file system.

[0004] The Chinese patent application number CN201410439971.6 describes a 'method and device for capturing and transmitting a file system serialization operation log', which implements a dynamic file mirroring method based on a file system serialization operation log. This method does not rely on file system snapshot technology, but uses a file system serialization operation log to fill the inconsistency between source data and backup data caused by dynamic changes of files during the mirroring process. Therefore, the present invention intends to solve the problem of inconsistency between source data and backup data caused by dynamic changes of files during the mirroring process based on a file system serialization operation log. Summary of the invention

[0005] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a dynamic file mirroring method and system based on serialized operation logs, so as to achieve the purpose of filling the inconsistency between source data and backup data caused by dynamic changes of files in the mirroring process by replaying the file system serialized operation logs without relying on file system snapshot technology.

[0006] To achieve the above object, the present invention proposes a dynamic file mirroring method based on serialized operation logs, comprising: Step S1, the rule management module at the source end sends the rule capture strategy and the rule mirror strategy to the operation capture module and the file mirror module respectively according to the configured file backup task rule; Step S2, calling the file mirroring module to compare and mirror the specified file directory from the source end to the backup end according to the rule mirroring strategy; Step S3, the operation capture module captures the IO operations of the application layer in real time according to the rule capture strategy, and stores the IO operations in the operation cache module after serialization; Step S4, when the mirror comparison task is completed, the time point when the mirror is completed is recorded, and the operation cache module transmits the captured serialized IO operation log to the standby end; Step S5, the standby end uses the operation application module to apply the serialized IO operation log before the time point when the mirroring is completed to the standby end's file system in sequence, so as to achieve consistency of data between the source end and the standby end at the time point when the mirroring is completed.

[0007] Preferably, after step S5, the following steps are also included: When the IO operation log before the time point of mirroring completion is applied on the standby side and the data on the source side and the standby side are consistent, the backup task enters the real-time replication stage. After the time point of mirroring completion, the operation capture module continues to capture the serialized IO operation log in real time by transmitting it to the standby side, and the standby side operation application module continues to apply the serialized IO operation log to the standby side file system in sequence.

[0008] Preferably, in step S3, the operation capture module monitors and captures the I / O operation information of the files in the specified directory in real time according to the rule capture strategy, and generates a serialized I / O operation log for transmission to the operation cache module of the application layer.

[0009] Preferably, in step S2, the file mirroring module scans all files in the designated file directory and subdirectories according to the regular mirroring strategy, and if inconsistent file blocks are found, the file mirroring module at the backup end synchronizes the file blocks through the network transmission module.

[0010] Preferably, the file mirroring module scans the file list on the specified directories of the file systems of the source and backup ends according to the rule mirroring strategy and calculates the MD5 values ​​in blocks, finds inconsistent file blocks in the file by comparing the MD5 values, and synchronizes the file blocks through the network transmission module.

[0011] Preferably, in step S4, the time standard of recording is based on the source end.

[0012] Preferably, in step S5, when a rename IO event is analyzed, a rename file mapping table is established to record the file name mapping relationship before and after the rename, and the files before and after the rename are searched for each record in the rename file mapping table on the backup side, and the rename situation is processed according to the search result.

[0013] Preferably, the renaming situation processing according to the search result specifically includes: if the backup end only finds the file before the rename, the file before the rename is renamed to obtain the renamed file; if the backup end only finds the renamed file but does not find the file before the rename, the IO operation of the file name before the rename is replayed on the renamed file; if the backup end can find neither the file before the rename nor the file after the rename, it is necessary to find the renamed file name from the source end and re-mirror the file to the backup end.

[0014] Preferably, in step S5, during the process of replaying the renaming operation log, if the renamed file is not found on the standby end, the renamed file name is found from the source end, and the file is re-mirrored to the standby end; if the renamed file is found on the standby end, the standby end sends a file list composed of the renamed file name and the file summary to the source end, and the source end then synchronizes the files in the file list with inconsistent file summaries to the standby end. To achieve the above object, the present invention further provides a dynamic file mirroring system based on serialized operation logs, including a source end and a backup end, wherein the source end includes: The rule management module is used to obtain the configured file backup task rules, send the rule capture strategy and the rule mirror strategy to the operation capture module and the file mirror module respectively according to the configured file backup task rules, and is responsible for tracking the running status of the file backup task and scheduling the running process; An operation capture module is used to receive the rule capture strategy transmitted by the rule module, monitor and capture the file IO operations of the specified directory according to the rule capture strategy, and is responsible for serializing the captured IO operations and transmitting them to the operation cache module of the application layer; A first file mirroring module is used to receive the rule mirroring strategy, compare and mirror the specified file directory from the source end to the backup end according to the rule mirroring strategy, and record the time point of mirroring completion when the comparison mirroring task is completed; An operation cache module is used to receive and save the serialized IO operation log transmitted by the operation capture module in real time, and transmit the captured serialized IO operation log to the standby end through the network transmission module after the first file mirroring module completes the comparison of the mirroring task and records the time point of mirroring completion; A first network transmission module, used to provide communication between the source end and the backup end; The standby end includes: A second network transmission module, used to provide communication between the source end and the backup end; The second file mirroring module is used to compare and mirror the specified file directory from the source end to the backup end; The operation application module is used to receive the serialized IO operation log transmitted by the source-side operation cache module, and replay the serialized IO operation log in sequence on the backup-side file system to obtain a result consistent with the source-side file IO operation.

[0015] Compared with the prior art, the present invention provides a dynamic file mirroring method and system based on serialized operation logs, which captures file change operations in the process of comparing the file directory with the mirror, and caches them on the source side after serialization, transmits them to the backup side after the mirroring is completed, and then applies the operations in sequence to the file system on the backup side, so that the data on the source side and the backup side are consistent after the operation application is completed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the steps of a dynamic file mirroring method based on serialized operation logs of the present invention. ; Figure 2 A system architecture diagram of a dynamic file mirroring system based on serialized operation logs of the present invention; Figure 3 The flowchart of the dynamic file mirroring based on the serialized operation log according to the specific embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0018] Figure 1 The following is a flow chart of the steps of a dynamic file mirroring method based on serialized operation logs of the present invention. Figure 1 As shown, the present invention provides a dynamic file mirroring method based on serialized operation logs, comprising: Step S1, the rule management module at the source end sends the rule capture strategy and the rule mirror strategy to the operation capture module and the file mirror module respectively according to the configured file backup task rules.

[0019] In a specific embodiment of the present invention, the file backup task rules are used to specify the backed up files and directories, target disaster recovery address, disaster recovery path and work machine (the work machine here refers to the source end) address, as well as to set whether the transmission is encrypted, compressed, bandwidth controlled and other processing strategies. The rule capture strategy refers to the capture strategy in the kernel operation capture module, which is mainly used to specify which IO events under the file directory need to be captured. It is mainly composed of a rule identification ID (also called a capture strategy ID, used to uniquely identify the capture strategy rule), an included directory and an excluded path. The included directory refers to the directory to be protected, such as " / home", " / Data / log" under Linux; the excluded path refers to the sub-path excluded in the protected path, such as " / home / chenyq", which means protecting " / home", " / Data / log", but not protecting " / home / chenyq". The rule mirroring strategy refers to the policy information of the mirroring task in the file mirroring module, which mainly includes: the rule identification ID, the mapping relationship between the source host, the target host, the source directory and the backup directory, and some parameters used in the mirroring, such as whether to encrypt and compress, whether to synchronize file attributes, etc. The rule capture strategy and the rule mirroring capture strategy are subsets of the configuration information of the file backup task rules in different modules.

[0020] When the rule management module starts the mirroring task from the source end to the backup end, before the mirroring task starts, the rule management module first sends the captured rule capture strategy to the operation capture module, so that the operation capture module starts to capture the file IO operations generated by the application layer program according to the rule capture strategy. The rule management module also sends the rule mirroring strategy to the file mirroring module, so that the file mirroring module compares and mirrors the files according to the rule mirroring strategy.

[0021] Step S2, calling the file mirroring module to compare and mirror the specified file directory from the source end to the backup end according to the rule mirroring strategy. Specifically, the file mirroring module starts to compare the file directory specified by the mirror according to the rule mirroring strategy. The file mirroring module scans all files in the specified file directory and subdirectories. If inconsistent file blocks are found, the source end and the backup end synchronize the file blocks through the network transmission module. In a specific embodiment of the present invention, the file mirroring module is deployed on the source end and the backup end respectively. The file mirroring module scans the file list on the specified directory of the file system of the source end and the backup end according to the rule mirroring strategy and calculates the checksum (MD5) value in blocks, compares the checksum and checksum values ​​to find inconsistent file blocks in the file, and synchronizes the file blocks through the network transmission module.

[0022] Step S3, the operation capture module captures the IO operation of the application layer in real time according to the rule capture strategy, and stores the IO operation in the operation cache module after serialization. Specifically, the operation capture module monitors and captures the I / O operation information of the files in the specified directory in real time according to the rule capture strategy, and generates a serialized I / O operation log to transmit to the operation cache module of the application layer. In this way, during the file mirroring process of step S2, the content of the file may still change in real time due to the IO operation of the application program, and these changes will be captured by the operation capture module to generate a serialized IO operation log, which will be sent to the operation cache module of the application layer.

[0023] In a specific embodiment of the present invention, the serialized operation log includes: the time when the I / O operation occurs, the process initiated, the specific file for which the operation is performed, the specific operation location of the file, and the content of the operation. As for how the operation capture module performs capture and how to generate the serialized operation log, the specific implementation steps have been disclosed in the Chinese patent with patent number CN201410439971.6, which will not be repeated here.

[0024] Step S4, when the mirror comparison task is completed, the time point when the mirror is completed is recorded, and the operation cache module transmits the captured serialized IO operation log to the standby end through the network transmission module.

[0025] In a specific embodiment of the present invention, in order to eliminate the time inconsistency between the source and backup data, the recorded time standard is based on the source. That is to say, the IO operations that occurred to the files on the source during the period from the start of mirroring to the completion of mirroring have been recorded and stored in the operation cache module on the source. These serialized operation logs indicate the direct cause of the data difference between the source and backup ends.

[0026] Step S5, the standby end uses the operation application module to apply the serialized IO operation log before the time point when the mirroring is completed to the standby end's file system in sequence, so as to achieve consistency of data between the source end and the standby end at the time point when the mirroring is completed.

[0027] Since the serialized IO operation log records the direct cause of the data difference between the source and the backup, the serialized IO operation log before the time point of mirroring completion is applied to the backup file system in sequence, that is, the serialized IO operation log is replayed on the backup file system. After the operation application is completed, the data on the source and backup sides at the time point of mirroring completion can be consistent.

[0028] However, during the file mirroring process, the file may be renamed. For the files renamed during the file mirroring process, the corresponding files may not be found during the backup end replay process. Therefore, the rename operation during the file mirroring process needs to be specially processed. In order to ensure the consistency of data between the source and backup ends, the rename operation is relatively special during the replay of the serialized IO operation log in the backup end file system and requires special processing. The particularity of the rename operation is that after the file is renamed, the IO operation log of the original file name may not be able to find the corresponding file during the backup end replay process. To avoid this problem, in a specific embodiment of the present invention, the following two solutions are adopted: 1. After the standby end analyzes the rename IO event, it establishes a rename file mapping table to record the mapping relationship between the file names before and after the rename. According to the rename file mapping table, the standby end searches for the files before and after the rename for each record. The following situations are handled according to the search results: 1. If the backup end only finds the file before the rename, it means that the file after the rename has not been synchronized during the mirror scanning process (whether it has been synchronized depends on the relationship between the time point when the file is scanned during the mirror scanning process and the time point when the rename operation occurs). In this case, rename the file before the rename to obtain the file after the rename.

[0029] 2. If the backup end only finds the files after the rename but not the files before the rename, it means that the files after the rename are synchronized but the files before the rename are not synchronized. The reason is that during mirroring, there is a sequence between the time when the files are scanned and synchronized and the time when the rename operation, including the IO operation, is performed. At this time, the IO operation for the file name before the rename is replayed for the file after the rename. This situation is illustrated as follows: If the source operation sequence is: 1) Write operation on A (can be multiple operations) 2) A -> B rename operation 3) Write operation on B (can be multiple operations).

[0030] If only the renamed file B can be found on the standby side, the above operation becomes: 1) For the Write operation of A, the object cannot be found and cannot be cached. 2) Find the rename event of A->B and convert all the Write operations on A in 1 to operations on B 3) The write operation on B is performed in sequence.

[0031] 3. If the backup end can find neither the file before nor after the rename, it is necessary to find the name of the file after the rename from the source end and re-image the file to the backup end. In this case, the target of the rename has been scanned and the rename event occurs again. In this case, it is necessary to find the name of the file after the rename from the source end and re-image the file to the backup end.

[0032] 2. When replaying the rename operation log, if the renamed file is not found on the standby end, the renamed file name is found from the source end and the file is re-mirrored to the standby end; if the renamed file is found on the standby end, the standby end sends a file list consisting of the renamed file name and file digest (such as MD5) to the source end, and the source end synchronizes the files in the file list with inconsistent file digests to the standby end.

[0033] The difference between the above two methods is: Method 2 is to find inconsistent files and re-image them from the source end to the backup end regardless of whether the renamed file is found on the backup end. Method 1 is to re-image the file from the source end only when the file cannot be found before or after rename. Otherwise, it is sufficient to replay the IO after rename. For the above two methods, Method 2 is simple to implement and not prone to errors; Method 1 is complex to implement and has many factors to consider, but it is highly efficient for large files during implementation.

[0034] Preferably, in step S5, the present invention further comprises the following steps: After the IO operation log before the time point of mirroring completion is applied on the standby side and the data on the source and standby sides are consistent, the backup task enters the real-time replication stage. In the real-time replication stage, that is, after the time point of mirroring completion, the serialized IO operation log that is still being captured by the operation capture module on the source side is transmitted to the standby side through the network, and the operation application module on the standby side continues to apply the IO operation log to the file system on the standby side in sequence.

[0035] Figure 2 This is a system architecture diagram of a dynamic file mirroring system based on serialized operation logs of the present invention. Figure 2 As shown, the present invention provides a dynamic file mirroring system based on serialized operation logs, including: a source end 20 and a backup end 21.

[0036] The source end 20 specifically includes: The rule management module 201 is used to obtain the configured file backup task rules, send the rule capture strategy and the rule mirror strategy to the operation capture module and the file mirror module respectively according to the configured file backup task rules, and is responsible for tracking the running status of the file backup task and scheduling the running process.

[0037] In a specific embodiment of the present invention, the rule management module 201 receives the file backup task rules issued by the user, and sends the rule capture strategy and the rule mirror strategy to the operation capture module and the file mirror module respectively, and is responsible for tracking the running status of the task and scheduling the running process. The file backup task rules are used to specify the backup files and directories, target disaster recovery address, disaster recovery path and work machine (the work machine here refers to the source end) address, as well as setting whether the transmission is encrypted, compressed, bandwidth controlled and other processing strategies. The rule capture strategy refers to the capture strategy in the kernel operation capture module, which is mainly used to specify which IO events under the file directory need to be captured. It is mainly composed of a rule identification ID (also called a capture strategy ID, used to uniquely identify the capture strategy rule), an included directory and an excluded path. The included directory refers to the directory to be protected, such as " / home", " / Data / log" under Linux; the excluded path refers to the sub-path excluded in the protected path, such as " / home / chenyq", which means protecting " / home", " / Data / log", but not protecting " / home / chenyq". The rule mirroring strategy refers to the policy information of the mirroring task in the file mirroring module, which mainly includes: the rule identification ID, the mapping relationship between the source host, the target host, the source directory and the backup directory, and some parameters used in the mirroring, such as whether to encrypt and compress, whether to synchronize file attributes, etc. The rule capture strategy and the rule mirroring capture strategy are subsets of the configuration information of the file backup task rules in different modules.

[0038] The operation capture module 202 is used to receive the rule capture strategy transmitted by the rule module 201, and monitor and capture the file IO operations of the specified directory according to the rule capture strategy, and is responsible for serializing the captured IO operations and transmitting them to the operation cache module of the application layer. Specifically, the operation capture module 202 monitors and captures the I / O operation information of the files in the specified directory in real time according to the rule capture strategy, and generates a serialized I / O operation log and transmits it to the operation cache module of the application layer. In this way, during the file mirroring process, the content of the file may still change in real time due to the IO operation of the application program, and these changes will be captured by the operation capture module to generate a serialized IO operation log, which will be sent to the operation cache module 204 of the application layer.

[0039] In a specific embodiment of the present invention, the serialized operation log includes: the time when the I / O operation occurs, the initiated process, the specific file for which the operation is performed, the specific operation location of the file, and the content of the operation.

[0040] The file mirroring module 203 is used to receive the rule mirroring strategy, compare and mirror the specified file directory from the source end to the backup end according to the rule mirroring strategy, and record the time point when the mirroring task is completed. Specifically, the file mirroring module 203 compares the file directory specified by the mirroring according to the rule mirroring strategy, and the file mirroring module 203 scans all files in the specified file directory and subdirectories. If inconsistent file blocks are found, the file mirroring module at the source end and the file mirroring module at the backup end synchronize the file blocks through the network transmission module. That is to say, the file mirroring module will be deployed on the source end and the backup end respectively. The file mirroring module scans the file list on the specified directory of the file system of the source end and the backup end according to the rule mirroring strategy and calculates the checksum (MD5) value in blocks, compares the checksum and checksum values ​​to find inconsistent file blocks in the file, and synchronizes the file blocks through the network transmission module.

[0041] The operation cache module 204 runs at the application layer and is used to receive the serialized IO operation log transmitted by the operation capture module 202 of the kernel layer in real time and save it in the memory or disk. After the file mirroring module 203 compares the mirroring task completion and records the time point of mirroring completion, the captured serialized IO operation log is transmitted to the backup end through the network transmission module.

[0042] In a specific embodiment of the present invention, in order to eliminate the time inconsistency between the source and backup data, the recorded time standard is based on the source. That is to say, the IO operations that occurred to the files on the source during the period from the start of mirroring to the completion of mirroring have been recorded and stored in the operation cache module on the source. These serialized operation logs indicate the direct cause of the data difference between the source and backup ends.

[0043] The network transmission module 205 is used to provide communication between the source end and the backup end. Specifically, the network transmission module 205 provides reliable connection-oriented data transmission between the two endpoints, and provides functions such as automatic reconnection, breakpoint continuation, encryption compression, etc. on the basis of encapsulating TCP connection. In a specific embodiment of the present invention, data transmission between the file mirroring module of the source end and the backup end, and from the operation cache module to the operation application module is built on this basis, that is, the network transmission module will be deployed on the source end and the backup end respectively.

[0044] The standby end 21 specifically includes: The network transmission module 210 is used to provide communication between the source end and the backup end, and is used for data transmission between the file mirroring modules of the source end and the backup end, and between the operation cache module and the operation application module.

[0045] The file mirroring module 211 is used to compare and mirror the specified file directory from the source end to the backup end.

[0046] The operation application module 212 is used to receive the serialized IO operation log transmitted by the source-side operation cache module 204, and replay the serialized IO operation log in sequence on the backup-side file system to obtain a result consistent with the source-side file IO operation.

[0047] Since the serialized IO operation log records the direct cause of the data difference between the source and the backup, the serialized IO operation log before the time point of mirroring completion is applied to the backup file system in sequence, that is, the serialized IO operation log is replayed on the backup file system. After the operation application is completed, the data on the source and backup sides at the time point of mirroring completion can be consistent.

[0048] However, during the file mirroring process, the file may be renamed. For the files renamed during the file mirroring process, the corresponding files may not be found during the backup end replay process. Therefore, the rename operation during the file mirroring process needs to be specially processed. In order to ensure the consistency of data between the source and backup ends, the rename operation is relatively special during the replay of the serialized IO operation log in the backup end file system and requires special processing. The particularity of the rename operation is that after the file is renamed, the IO operation log of the original file name may not be able to find the corresponding file during the backup end replay process. To avoid this problem, in a specific embodiment of the present invention, the following two solutions are adopted: 1. After the standby end analyzes the rename IO event, it establishes a rename file mapping table to record the mapping relationship between the file names before and after the rename. According to the rename file mapping table, the standby end searches for the files before and after the rename for each record. The following situations are handled according to the search results: 1. If the backup end only finds the file before the rename, it means that the file after the rename has not been synchronized during the mirror scanning process (whether it has been synchronized depends on the relationship between the time point when the file is scanned during the mirror scanning process and the time point when the rename operation occurs). In this case, rename the file before the rename to obtain the file after the rename.

[0049] 2. If the backup end only finds the files after the rename but not the files before the rename, it means that the files after the rename are synchronized but the files before the rename are not synchronized. The reason is that during mirroring, there is a sequence between the time when the files are scanned and synchronized and the time when the rename operation, including the IO operation, is performed. At this time, the IO operation for the file name before the rename is replayed for the file after the rename. This situation is illustrated as follows: If the source operation sequence is: 1) Write operation on A (can be multiple operations) 2) A -> B rename operation 3) Write operation on B (can be multiple operations).

[0050] If only the renamed file B can be found on the standby side, the above operation becomes: 1) For the Write operation of A, the object cannot be found and cannot be cached. 2) Find the rename event of A->B and convert all the Write operations on A in 1 to operations on B 3) The write operation on B is performed in sequence.

[0051] 3. If the backup end can find neither the file before nor after the rename, it is necessary to find the name of the file after the rename from the source end and re-image the file to the backup end. In this case, the target of the rename has been scanned and the rename event occurs again. In this case, it is necessary to find the name of the file after the rename from the source end and re-image the file to the backup end.

[0052] 2. When replaying the rename operation log, if the renamed file is not found on the standby end, the renamed file name is found from the source end and the file is re-mirrored to the standby end; if the renamed file is found on the standby end, the standby end sends a file list consisting of the renamed file name and file digest (such as MD5) to the source end, and the source end synchronizes the files in the file list with inconsistent file digests to the standby end.

[0053] The difference between the above two methods is: Method 2 is to find inconsistent files and re-image them from the source end to the backup end regardless of whether the renamed file is found on the backup end. Method 1 is to re-image the file from the source end only when the file cannot be found before or after rename. Otherwise, it is sufficient to replay the IO after rename. For the above two methods, Method 2 is simple to implement and not prone to errors; Method 1 is complex to implement and has many factors to consider, but it is highly efficient for large files during implementation.

[0054] After the IO operation log before the time point of mirroring completion is applied on the standby side and the data on the source and standby sides are consistent, the backup task enters the real-time replication stage. In the real-time replication stage, that is, after the time point of mirroring completion, the serialized IO operation log that is still being captured by the operation capture module on the source side is transmitted to the standby side through the network, and the operation application module on the standby side continues to apply the IO operation log to the file system on the standby side in sequence. Example

[0055] like Figure 3 As shown, in this embodiment, a rule management module, an operation capture module, an operation cache module, a file mirroring module, and a network transmission module are deployed on the source end (production end); an operation application module, a file mirroring module, and a network transmission module are deployed on the standby end. The dynamic file mirroring process based on the serialized operation log is as follows: Step 1: The user configures the rule strategy of the file backup task through the configuration interface, which includes the host, directory, time window, compression flow control, etc. of the backup task. The rule strategy is sent and recorded to the rule management module, which automatically manages and schedules the file backup task according to the configured rule strategy, and displays the running status of the backup task to the user in the form of status information and logs.

[0056] Step 2: The rule management module starts the mirroring task from the source end to the backup end. Before the mirroring task starts, the rule module sends the rule capture strategy to the operation capture module. The operation capture module starts to capture the file IO operations generated by the application layer program according to the rule capture strategy, that is, according to the main information "backup monitoring directory" in the rule capture strategy, it captures the matched file IO operation logs, and serializes the IO operations to ensure that the IO order is consistent with the actual IO order, and sends them to the operation cache module; Step 3: The rule management module sends the rule mirroring strategy to the file mirroring module, and the file mirroring module starts to compare the file directory specified by the mirroring; Step 4, the file mirroring module completes the mirroring synchronization function of the specified directory of the file system. The file mirroring module on the source end contacts the file mirroring module on the backup end according to the strategy and compares the mirroring of the specified directory and subdirectory. The file list is scanned on the specified directory of the file system on the source end and the backup end, and the checksum (MD5) value is calculated in blocks. The checksum value is compared to find inconsistent blocks in the file and synchronize the file block. Specifically, the file mirroring module scans all files in the specified directory and subdirectory, calculates the checksum (MD5) value for the file block, compares the checksum values ​​of the corresponding file blocks on the source end and the backup end, and finds inconsistent file blocks. The source end and the backup end synchronize the file blocks through the network transmission module. When all file blocks of a file are compared and synchronized, the mirroring comparison of the file is completed. In this embodiment, the comparison mirroring of different files can be completed in parallel by multiple threads to speed up the mirroring speed. During the comparison mirroring process, changes caused by new writes to the file are not considered in the comparison process (this is captured by the operation capture module) Step 5, the task of comparing the mirror is completed, and the time point of the mirror completion is recorded. In order to eliminate the time inconsistency between the source and backup data, the recorded time standard is based on the source. The IO operations of the source files from the beginning of the mirror to the completion of the mirror have been recorded and stored in the operation cache module. These serialized operation logs indicate the direct cause of the difference between the source and backup data.

[0057] Step 6: The operation cache module on the source side transmits the captured serialized IO operation log to the standby side through the network; Step 7: The operation application module on the backup side applies all IO operation logs before the time point when the mirroring is completed to the file system on the backup side in sequence, i.e., replays them. After the operation application is completed, the data on the source side and the backup side at the time point when the mirroring is completed can be consistent.

[0058] To ensure data consistency between the source and backup ends, during the replay of the serialized IO operation log in the backup file system, the rename operation is relatively special and requires special processing. The particularity of the rename operation is that after the file is renamed, the IO operation log of the original file name may not be able to find the corresponding file during the replay process on the backup end. To avoid this problem, there are two solutions: 1. Establish a rename file mapping table to map all file names before rename in the serialized IO operation log to the problem name after rename; 2. Record the problem of not finding the original file name during the replay process and re-image the renamed file.

[0059] Step 8 After the IO operation log before the time point of mirroring completion is applied on the standby side and the data on the source and standby sides are consistent, it means that the incremental data generated in the mirroring stage has been applied and balanced. At this time, the backup task enters the real-time replication stage. In this stage, the IO operation log after the time point of mirroring completion and the operation capture module on the source side continue to capture the IO operation log and transmit it to the standby side through the network. The operation application module on the standby side continues to apply the IO operation log to the standby side's file system in sequence, that is, the incremental stage of real-time replication is to continuously replay the serialized IO operation log transmitted from the source side on the standby side, and under ideal network conditions, the data difference between the source and standby sides is within milliseconds.

[0060] In summary, the present invention provides a dynamic file mirroring method and system based on serialized operation logs, which captures file change operations in the process of comparing the file directory with the mirror, and caches them on the source end after serialization, transmits them to the backup end after the mirroring is completed, and then applies the operations to the backup end file system in sequence on the backup end, so that the data on the source end and the backup end are consistent after the operation application is completed.

[0061] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Any person skilled in the art may modify and alter the above embodiments without violating the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be as set forth in the claims.

Claims

1. A dynamic file mirroring method based on serialized operation logs, comprising: Step S1, the rule management module at the source end sends the rule capture strategy and the rule mirror strategy to the operation capture module and the file mirror module respectively according to the configured file backup task rule; Step S2, calling the file mirroring module to compare and mirror the specified file directory from the source end to the backup end according to the rule mirroring strategy; Step S3, the operation capture module captures the IO operations of the application layer in real time according to the rule capture strategy, and stores the IO operations in the operation cache module after serialization; Step S4, when the mirror comparison task is completed, the time point when the mirror is completed is recorded, and the operation cache module transmits the captured serialized IO operation log to the standby end; Step S5, the standby end uses the operation application module to apply the serialized IO operation log before the time point when the mirroring is completed to the standby end's file system in sequence, so as to achieve consistency of data between the source end and the standby end at the time point when the mirroring is completed.

2. A dynamic file mirroring method based on serialized operation logs as claimed in claim 1, characterized in that: After step S5, the following steps are also included: When the IO operation log before the time point of mirroring completion is applied on the standby side and the data on the source side and the standby side are consistent, the backup task enters the real-time replication stage. After the time point of mirroring completion, the operation capture module continues to capture the serialized IO operation log in real time by transmitting it to the standby side, and the standby side operation application module continues to apply the serialized IO operation log to the standby side file system in sequence.

3. A dynamic file mirroring method based on serialized operation logs as described in claim 1, characterized in that: In step S3, the operation capture module monitors and captures the I / O operation information of the files in the specified directory in real time according to the rule capture strategy, and generates a serialized I / O operation log to transmit to the operation cache module of the application layer.

4. A dynamic file mirroring method based on serialized operation logs as claimed in claim 1, characterized in that: In step S2, the file mirroring module scans all files in the specified file directory and subdirectories according to the regular mirroring strategy, and if inconsistent file blocks are found, the file mirroring module at the backup end synchronizes the file blocks through the network transmission module.

5. A dynamic file mirroring method based on serialized operation logs as claimed in claim 4, characterized in that: The file mirroring module scans the file list on the specified directory of the file system of the source end and the backup end according to the rule mirroring strategy and calculates the MD5 value in blocks, finds out inconsistent file blocks in the file by comparing the MD5 values, and synchronizes the file blocks through the network transmission module.

6. A dynamic file mirroring method based on serialized operation logs as claimed in claim 1, characterized in that: In step S4, the time standard of the recording is based on the source end.

7. A dynamic file mirroring method based on serialized operation logs as claimed in claim 1, characterized in that: In step S5, when a rename IO event is analyzed, a rename file mapping table is established to record the file name mapping relationship before and after the rename. According to the rename file mapping table, the files before and after the rename are searched on the backup side for each record therein, and the rename situation is processed according to the search result.

8. A dynamic file mirroring method based on serialized operation logs as claimed in claim 7, characterized in that: The renaming situation processing according to the search result specifically includes: if the standby end only finds the file before the rename, the file before the rename is renamed to obtain the renamed file; if the standby end only finds the renamed file but does not find the file before the rename, the IO operation of the file name before the rename is replayed on the renamed file; if the standby end can neither find the file before the rename nor the file after the rename, it is necessary to find the renamed file name from the source end and re-mirror the file to the standby end.

9. A dynamic file mirroring method based on serialized operation logs as claimed in claim 1, characterized in that: In step S5, during the process of replaying the renaming operation log, if the renamed file is not found on the backup end, the renamed file name is found from the source end, and the file is re-mirrored to the backup end; if the renamed file is found on the backup end, the backup end will send a file list composed of the renamed file name and the file summary to the source end, and the source end will then synchronize the files in the file list with inconsistent file summaries to the backup end.

10. A dynamic file mirroring system based on serialized operation logs, comprising a source end and a backup end, wherein the source end comprises: The rule management module is used to obtain the configured file backup task rules, send the rule capture strategy and the rule mirror strategy to the operation capture module and the file mirror module respectively according to the configured file backup task rules, and is responsible for tracking the running status of the file backup task and scheduling the running process; An operation capture module is used to receive the rule capture strategy transmitted by the rule module, monitor and capture the file IO operations of the specified directory according to the rule capture strategy, and is responsible for serializing the captured IO operations and transmitting them to the operation cache module of the application layer; A first file mirroring module is used to receive the rule mirroring strategy, compare and mirror the specified file directory from the source end to the backup end according to the rule mirroring strategy, and record the time point of mirroring completion when the comparison mirroring task is completed; An operation cache module is used to receive and save the serialized IO operation log transmitted by the operation capture module in real time, and transmit the captured serialized IO operation log to the standby end through the network transmission module after the first file mirroring module completes the comparison of the mirroring task and records the time point of mirroring completion; A first network transmission module, used to provide communication between the source end and the backup end; The standby end includes: A second network transmission module, used to provide communication between the source end and the backup end; The second file mirroring module is used to compare and mirror the specified file directory from the source end to the backup end; The operation application module is used to receive the serialized IO operation log transmitted by the source-side operation cache module, and replay the serialized IO operation log in sequence on the backup-side file system to obtain a result consistent with the source-side file IO operation.

Images