A data intelligent capture method and system for volume-level continuous data protection

Abstract

The present invention proposes a data intelligent capture method and system for volume-level continuous data protection, which can effectively solve the problems in traditional CDP technology, including: a volume-level driver module, an intelligent capture and replication module for system volume-level data, which is used to transmit the captured change data to the backup end and apply it. According to different data change log file information, different methods will be used to transmit the corresponding changed data to the backup end, and the changed data will be applied. The data capture mode switching module and the application layer module monitor the disk IO status, network status, cache space size and other information of the protected server, and flexibly switch the data capture mode of the driver module to solve the cache space overflow problem, and make a good balance between cache space usage and recoverable points so that the protection plan can work normally.

Description

Technical Field

[0001] The present invention belongs to the fields of computer data disaster recovery and backup, virtualization and cloud computing, and in particular relates to an intelligent data capture method based on whole-machine protection. Background Art

[0002] In traditional continuous data protection (CDP) solutions, every change in the system data needs to be captured in real time and transmitted to the backup end immediately. This approach is suitable in many scenarios, especially when the amount of system data changes not very frequently.

[0003] In the CDP protection solution, two critical steps are generally required. The first step is data synchronization, which is to completely synchronize the protected data on the source side to the backup side. The second step is to copy the real-time changed data to the backup side after the synchronization is completed. During the synchronization period, even if these changed data can be transmitted to the backup side in a timely manner, they cannot be applied immediately because the synchronization has not been completed. These changed data can only be cached, either on the protected server side or on the backup side.

[0004] However, in some special scenarios, especially when the data changes generated on the server side are very frequent and the amount of original data is large, this disadvantage will be revealed. For example, the amount of original data on a production server is very large, and the frequency of data changes is also very high, which often makes the entire protection unable to be carried out.

[0005] For example, the data synchronization step alone takes dozens of hours or even days. During this period, the amount of data that changes often reaches hundreds of GB or even TB. This makes it difficult for both the production server and the backup end to find such a large cache space to store these changed data, thus causing the protection plan to fail. Summary of the Invention

[0006] Therefore, to address these issues with traditional CDP protection solutions, the present invention proposes a data intelligent capture method and system for volume-level continuous data protection, which can effectively solve the problems in traditional CDP technology, including:

[0007] The volume-level driver module intelligently captures system volume-level data. There are two primary methods for capturing data changes: the traditional data capture method, which records every data change in a log file. This is referred to as the traditional method in this paper. The other capture method only records metadata about data changes, such as volume information, starting location, and length, without recording the specific data content. This is referred to as the metadata method in this paper. The data information record files generated by either method are referred to as data change log files in this paper. The volume-level driver module of this paper can freely switch between the traditional data capture method and the metadata method.

[0008] The synchronization module is used to synchronize the data of the protected end to the backup end.

[0009] The replication module is used to transfer the captured change data to the backup end and apply it.

[0010] According to different data change log file information, different methods will be used to transmit the corresponding changed data to the backup end, and the changed data will be applied.

[0011] The data capture mode switching module and the application layer module flexibly switch the data capture mode of the driver module by monitoring the disk IO status, network status, cache space size and other information of the protected server to solve the cache space overflow problem and make a good balance between cache space usage and recoverable points so that the protection plan can work normally. DETAILED DESCRIPTION

[0012] The following describes the embodiments of the present invention through specific examples. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through other different specific examples, and the details in this specification may be modified and altered based on different perspectives and applications without departing from the spirit of the present invention.

[0013] The present invention provides a data intelligent capture method and system for volume-level continuous data protection, comprising:

[0014] The volume-level driver module intelligently captures system volume-level data. There are two primary methods for capturing data changes: the traditional data capture method, which records every data change in a log file. This is referred to as the traditional method in this paper. The other capture method only records metadata about data changes, such as volume information, starting location, and length, without recording the specific data content. This is referred to as the metadata method in this paper. The data information record files generated by either method are referred to as data change log files in this paper. The volume-level driver module of this paper can freely switch between the traditional data capture method and the metadata method.

[0015] Traditional data capture methods retain the content of every data change, thus occupying a large cache space. However, because this method preserves every data change, it allows for more recovery time points. Metadata capture methods only retain the location information of each change—the starting position and size of the data change—without storing the specific data. Therefore, this method's data change log occupies very little space, only a few hundredths or even a few thousandths of the space used by traditional capture methods. However, when using metadata mode, the recovery time points are significantly reduced.

[0016] The synchronization module is used to synchronize the data of the protected end to the backup end.

[0017] The replication module is used to transfer the captured change data to the backup end and apply it.

[0018] According to different data change log file information, different methods will be used to transmit the corresponding changed data to the backup end, and the changed data will be applied.

[0019] The data capture mode switching module and the application layer module flexibly switch the data capture mode of the driver module by monitoring the disk IO status, network status, cache space size and other information of the protected server to solve the cache space overflow problem and make a good balance between cache space usage and recoverable points so that the protection plan can work normally.

[0020] In order to better describe the present invention, specific steps are provided below for illustration.

[0021] Step S1: Notify the volume-level driver to start capturing data in a metadata capture mode and start synchronizing the data to the backup end.

[0022] In step S2, after the volume-level driver starts capturing data, it captures metadata of data changes (volume starting position and length) in real time, writes this information to the data change log file in real time, and generates a new data log file according to a preset period (for example, two seconds).

[0023] In step S3, after data synchronization begins, the application layer obtains data file information from the driver. At this time, since it is still in the data synchronization stage, the data corresponding to the data log is not sent to the backup end. Instead, the data log file is obtained from the volume-level driver, and is parsed in the order of generation, and the merging operation is continuously performed. The merging process is described in detail below.

[0024] For example, the application layer obtains three data change log files: 1.log, 2.log, and 3.log. Assume that the change data in 1.log is: Volume 1, 0-8192, 1024-4096, 0-1024; the change data in 2.log is: Volume 1, 4096-8192, 1024-8192; and the change data in 3.log is: Volume 1, 0-1024, 4096-8192. The total change data size is 28672. Because these changes do not need to be sent and applied immediately, they are merged when they are sent. That is, repeated data changes are only recorded once. Therefore, the merged data change information is: Volume 1, 0-8192.

[0025] Step S4: If the data synchronization is completed, the application layer notifies the volume-level driver to switch to the traditional data capture mode and generate a bookmark. That is, the data capture mode between the completion of the data synchronization and the bookmark is the traditional capture mode.

[0026] Step S5: Start generating the merged data change log, read the corresponding data from the volume according to the corresponding volume information recorded in the log, send it to the backup end, and apply the data.

[0027] Step S6: If the metadata data change log is sent, start sending the traditional data log. The data change log generated by the traditional capture mode is directly sent to the backup end and applied.

[0028] Step S7: After receiving the bookmark information, the backup end generates a snapshot on the backup end.

[0029] In step S8, the application layer detects the current IO status and remaining disk cache space of the protected system, and evaluates whether it is necessary to switch the data capture mode based on the current network conditions. If it is found that it is necessary to switch to the traditional data capture mode, the volume-level driver is notified to switch to the traditional capture mode. If it is found that the network processing is insufficient or there are too many IOs, resulting in a continuous increase in cache space, when it grows to a preset threshold (for example, 80%), the volume-level driver is notified to switch to the metadata capture mode.

[0030] In summary, the data intelligent capture method and system for volume-level continuous data protection of the present invention effectively solve the problems of excessive cache requirements and failure to perform normal protection in traditional CDP protection solutions.

[0031] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Any skilled artisan may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be as set forth in the appended claims.

Claims

1. A data intelligent capture method for volume-level continuous data protection, characterized in that: The following steps are involved: Step S1: Notify the volume-level driver to start capturing data in metadata capture mode and start synchronizing the data to the backup end; Step S2: The volume-level driver starts capturing data; Capture metadata of data changes in real time, write this information to data change log files in real time, and generate new data log files according to the preset cycle; Step S3: After data synchronization starts, the application layer obtains data file information from the driver and continuously performs merging operations; Step S4: if data synchronization is completed, switch to the traditional data capture mode and generate a bookmark; Step S5: Start generating a merged data change log, read the corresponding data from the volume according to the corresponding volume information recorded in the log, send it to the backup end, and apply the data; Step S6: If the metadata data change log is sent, start sending the traditional data log. The data change log generated by the traditional capture mode is directly sent to the backup end and applied; Step S7: After receiving the bookmark information, the backup end generates a snapshot on the backup end; In step S8, the application layer detects the current IO status and remaining disk cache space of the protected system, and evaluates whether it is necessary to switch the data capture mode based on the current network conditions. If it is found that it is necessary to switch to the traditional data capture mode, the volume-level driver is notified to switch to the traditional capture mode. If it is found that the network processing is insufficient or there are too many IOs, resulting in a continuous increase in cache space, when it grows to a preset threshold, the volume-level driver is notified to switch to the metadata capture mode.

2. A data intelligent capture system for volume-level continuous data protection and a data intelligent capture method for volume-level continuous data protection according to claim 1, characterized in that: include: Volume-level driver module, used for intelligent capture of system volume-level data; Synchronization module, used to synchronize the data of the protected end to the backup end; The replication module is used to transfer the captured change data to the backup end and apply it; The data capture mode switching module and the application layer module flexibly switch the data capture mode of the driver module by monitoring the disk IO status, network status and cache space size information of the protected server to solve the cache space overflow problem and make a good balance between cache space usage and recoverable points so that the protection plan can work normally.