A cloud storage data full-amount disaster recovery method and system

Abstract

The present application relates to a kind of cloud storage data full quantity disaster recovery method and system, under the premise that disaster recovery data has consistency time point, reduce the influence on production system IO from space occupation, IO delay, IO quantity etc., and can reduce RPO to a certain extent, especially for the scene that source volume has a large amount of data Append during backup, effect is particularly remarkable, for example, database, file etc.Such as scene, can help industry customer to provide data disaster recovery service for it after cloud.

Description

Technical Field

[0001] This invention relates to the field of data disaster recovery, and in particular to a method and system for full disaster recovery of cloud storage data. Background Technology

[0002] With the rapid development and widespread application of information technology, various application systems face increasing business continuity challenges. In response, many users build their own disaster recovery centers locally or remotely, but this often presents problems such as high construction costs, fixed models, decentralized operation and maintenance, and outdated systems. The rapid development of cloud computing technology provides users with elastically scalable, on-demand infrastructure and platform services, making it increasingly popular. China Telecom Cloud provides disaster recovery services, offering cross-AZ or cross-Region level protection for users' cloud-based or on-premises production systems against business interruption risks caused by accidents or anomalies. This allows users to use China Telecom Cloud with greater peace of mind, which is not only a common user demand but also an important requirement for information security level protection. While cloud-based storage technologies can support China Telecom Cloud host backup, they cannot support on-premises scenarios. Continuous protection based on block-level data is a commonly used technology for real-time backup data acquisition and can serve as a foundational technology for disaster recovery of cloud or on-premises hosts.

[0003] A key constraint of disaster recovery services is ensuring that the data state during a switch to the target region is consistent with the source data at a specific point in time. This point-in-time consistency can be achieved using snapshot technology. Traditional backup systems create a snapshot of the source volume before backing up the data, then initiate the backup, copying data from the snapshot volume, and finally destroying the snapshot when the backup is complete. Snapshots typically have three implementation methods: image-separated snapshots, copy-on-write (COW) snapshots, and copy-on-write (ROW) snapshots. Since snapshots created by public cloud disaster recovery services are only used temporarily during the backup period, prioritizing the read / write performance of the production host's source volume is crucial; therefore, copy-on-write (COW) snapshots are more suitable.

[0004] For COW snapshots, after a snapshot is created, when the source volume data changes, the original data of the source volume needs to be processed first, resulting in additional IO overhead. For example, when the source volume generates one write IO, the COW snapshot needs to generate one read IO of the source volume to obtain the original data, then generate one write IO of the snapshot volume to save the original data, and only then can the actual write IO of the source volume be executed. That is, one IO is amplified into three IOs.

[0005] In summary, while cloud disaster recovery systems based on snapshot mechanisms for continuous block-level data protection can provide consistent point-in-time data, they increase the I / O load on the source system, specifically in the following four aspects:

[0006] 1. Space usage: Snapshot volumes require additional storage space;

[0007] 2. Increased latency: For a source volume write IO that requires snapshot processing, the synchronous processing mechanism of the snapshot mechanism and the IO amplification effect of the snapshot will increase the latency of the source volume write IO.

[0008] 3. Performance degradation: During the data backup time window, the snapshot mechanism needs to continuously monitor and process the address space of the entire source volume, which requires a large number of I / O operations and affects the overall performance of the source volume.

[0009] 4. Increased RPO: When monitoring the entire source volume address space, more data needs to be recorded in the snapshot volume. When the address space of this data is not contiguous, it will reduce the read I / O performance of the snapshot volume and make it easier for sequential reads to be forced into random reads. Slower reading of target data means that it will take more time to back up the data in the snapshot volume. Before the backup is completed, the production system will not be able to recover to the current state in the event of an accident, which increases the RPO (Recovery Point Object, one of the important indicators of disaster recovery system). Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology by proposing a cloud storage data full disaster recovery method and system. Under the premise of ensuring that the disaster recovery data has a consistent time point, it reduces the impact on production system IO in terms of space occupation, IO latency, and IO quantity, and can reduce RPO to a certain extent. It is particularly effective for scenarios where a large amount of data appends to the source volume during the backup period, such as databases and archives. It can help industry customers provide data disaster recovery services after migrating to the cloud.

[0011] The objective of this invention can be achieved through the following technical solutions:

[0012] A method for full disaster recovery of cloud storage data includes:

[0013] Step 100: Start backup, sequentially copy the block data of the source volume TargetVol to the disaster recovery server, and have the disaster recovery service write the block data to the mirror volume.

[0014] Step 200: During the sequential copying of the TargetVol block data, the Change Block Tracking (CBT) module continuously monitors the data changes and creates a location marker table, which marks the location of the changed data.

[0015] Step 300: After completing the sequential copy of the TargetVol block data, create a snapshot and obtain the location marker table;

[0016] Step 400: Merge data according to the location information in the location marking table, read data of the corresponding location from the snapshot volume, and send it to the disaster recovery server to update the data of the corresponding location in the image volume;

[0017] Step 500: End backup.

[0018] Furthermore, the CBT module continuously monitors data changes, including:

[0019] The CBT module continuously monitors data changes in the copied area based on a step window; and...

[0020] The monitoring range of the CBT module gradually increases as the data backup progresses within the data backup time window, and data changes occurring outside the monitoring range are not marked.

[0021] Furthermore, the CBT module continuously monitors data changes, including:

[0022] The CBT module only marks the location information of the changed data in the location mark table, without storing the data itself.

[0023] Furthermore, before ending the backup, the snapshot is destroyed when all positions in the location marker table have been processed.

[0024] Furthermore, after the snapshot is established, new data is concurrently read from the snapshot volume according to the location recorded by CBT and updated to the disaster recovery server to update the data at the corresponding location in the mirror volume.

[0025] This invention also provides a cloud storage data full disaster recovery system, including a production server and a disaster recovery server.

[0026] The production server includes:

[0027] An agent program is used to directly and sequentially copy block data from the source volume TargetVol to the disaster recovery server without creating a snapshot when the backup starts.

[0028] The Change Block Tracking (CBT) module is used to continuously monitor data changes and create a location marker table during the sequential copying of TargetVol data, and to mark the location of the changed data in the location marker table.

[0029] The snapshot module is used to create snapshots after the sequential copying of TargetVol is completed;

[0030] A snapshot volume device is used to obtain the location marker table, merge data according to the location information in the location marker table, read data from the snapshot volume at the corresponding location, and send it to the disaster recovery server.

[0031] The disaster recovery server includes:

[0032] The data writing module is used to write sequentially copied data received from the source volume to the mirror volume;

[0033] The data update module is used to update the data at the corresponding location in the mirror volume after receiving the data at the corresponding location in the snapshot volume.

[0034] Furthermore, the CBT module is also used for:

[0035] The CBT module continuously monitors data changes based on a step window, and

[0036] The CBT module gradually increases the monitoring range within the data backup time window as the data backup progresses, and does not mark data changes that occur outside the monitoring range.

[0037] Furthermore, the CBT module is also used for:

[0038] When the CBT module is marked, it only stores the location information of the changed data, and does not need to store the data itself.

[0039] Furthermore, the production server also includes:

[0040] The snapshot destruction module is used to destroy snapshots when all positions in the location marker table have been processed.

[0041] The snapshot volume device is also used to concurrently read new data from the snapshot volume according to the location of the CBT record after the snapshot is established, and send it to the disaster recovery server.

[0042] The present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the above-described cloud storage data full disaster recovery method.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. By applying CBT to full backups, the number of data blocks that need to be processed for consistency during full backups is reduced, the amount of additional I / O is reduced, the backup time is shortened, and the RPO is lowered;

[0045] 2. Snapshots are only created in the later stages of the backup process, which reduces the space occupied by snapshots during full backups;

[0046] 3. Data consistency is achieved based on snapshots. Because the improved new CBT tag has fewer changed data blocks during full backup, the snapshot duration is shorter, the snapshot volume needs to cache less data, and the IO amplification effect caused by snapshots has a reduced impact on the performance of the production system.

[0047] 4. The original backup algorithm reads all data sequentially from the snapshot volume. Because the snapshot internally maps IO addresses, it is actually a random read. The full backup algorithm proposed in this invention first reads sequentially from the source volume, and then reads concurrently randomly from the snapshot volume, which improves read performance. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of this invention or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of COW snapshot I / O;

[0050] Figure 2 A block diagram of a traditional block data disaster recovery system;

[0051] Figure 3 This is a diagram illustrating data changes during a traditional backup process.

[0052] Figure 4 A block diagram of the improved block data disaster recovery system;

[0053] Figure 5 This is a diagram showing the effect comparison before and after the CBT improvement;

[0054] Figure 6 This is a diagram illustrating the data changes during the improved backup process;

[0055] Figure 7 This is a diagram comparing the data backup process before and after the improvement. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it.

[0057] Example 1

[0058] This invention provides a method for full disaster recovery of cloud storage data. During the monitoring of the I / O of the target volume, the data monitoring range is dynamically adjusted through step window control. Only data blocks that change within the monitoring range are marked, while data changes that occur outside the monitoring range do not need to be marked.

[0059] The full disaster recovery method described in this embodiment includes the following specific steps:

[0060] Step 100: Start backup, sequentially copy the block data of the source volume TargetVol to the disaster recovery server, and have the disaster recovery service write the block data to the mirror volume.

[0061] Step 200: During the sequential copying of the TargetVol block data, the Change Block Tracking (CBT) module continuously monitors the data changes and creates a location marker table, which marks the location of the changed data.

[0062] Step 300: After completing the sequential copy of the TargetVol block data, create a snapshot and obtain the location marker table;

[0063] Step 400: Merge data according to the location information in the location marking table, read data of the corresponding location from the snapshot volume, and send it to the disaster recovery server to update the data of the corresponding location in the image volume;

[0064] Step 500: End backup.

[0065] Optionally, the CBT module continuously monitors the data changes in the copied area based on a step window; and the monitoring range of the CBT module gradually increases as the data backup progresses within the data backup time window, without marking data changes that occur outside the monitoring range.

[0066] Optionally, the CBT module only marks the location information of the changed data in the location mark table, without storing the data itself.

[0067] Optionally, before ending the backup, the snapshot is destroyed when all positions in the location marker table have been processed.

[0068] Optionally, after the snapshot is established, new data is concurrently read from the snapshot volume according to the location recorded by CBT and updated to the disaster recovery server to update the data at the corresponding location in the mirror volume.

[0069] Optionally, the stored data includes cloud storage data and local storage data.

[0070] Traditional disaster recovery systems perform full backups by first creating a snapshot, and then the disaster recovery program copies data from the snapshot volume to the full mirror volume in block-level order. During the data copying process, the first data change in all locations on the source volume is recorded in the snapshot volume to ensure that the data copied by the disaster recovery program from the snapshot volume has a consistent time point T0. A diagram illustrating the data changes during the backup process is shown below. Figure 3 As shown in the diagram, a block is a BLOCK. When the data in a BLOCK changes after a snapshot is successfully created, the data will be copied to the snapshot volume, resulting in storage space occupation. At the same time, additional read IO and write IO occur when data is written to the snapshot volume, affecting the performance of production storage.

[0071] Example 2

[0072] Figure 2 This is a diagram of a traditional block data disaster recovery system. TargetVol is the production storage, serving as the source volume for the disaster recovery system. Snapshot is the snapshot module, responsible for providing consistent point-in-time data for the source volume, while also storing the original data at locations where changes have occurred in the source volume. Snap device is the snapshot volume device, providing the agent with a read interface to the TargetVol snapshot volume. Internally, it maps read I / O; if the data is unchanged, it reads the data directly from the corresponding location in the source volume; if the data has changed, it reads the data from the snapshot space. The agent is the disaster recovery system's proxy program, responsible for sending data from the snapshot volume to the disaster recovery server. The server is responsible for writing the received data to the local mirror volume. Before starting the backup, the backup system's agent needs to create a snapshot volume on the source volume, and then complete a consistent backup of the TargetVol data at a specific point in time by copying the snapshot volume.

[0073] Figure 4 This is a block diagram of the block data disaster recovery system provided in Embodiment 2 of the present invention. It can execute the block data disaster recovery method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. The present invention applies CBT technology to full backup and makes adaptive improvements to CBT technology. CBT, or Changed Block Tracing technology, monitors the IO of the target volume, marks the data blocks that have changed in the target, and records the location of the data changes. It can be used by the backup system to identify the data blocks that have changed during the sequential copying of all data blocks, so as to perform targeted consistency state processing.

[0074] like Figure 4 As shown, the disaster recovery system includes a production server and a disaster recovery server.

[0075] The production server includes:

[0076] An agent program is used to directly and sequentially copy block data from the source volume TargetVol to the disaster recovery server without creating a snapshot when a backup is initiated.

[0077] The Change Block Tracking (CBT) module is used to continuously monitor data changes and create a location marker table during the sequential copying of TargetVol data, and to mark the location of the changed data in the location marker table.

[0078] The snapshot module is used to create snapshots after the sequential copying of TargetVol is completed;

[0079] A snapshot volume device is used to obtain the location marker table, merge data according to the location information in the location marker table, read data from the snapshot volume at the corresponding location, and send it to the disaster recovery server.

[0080] The disaster recovery server includes:

[0081] The data writing module is used to write sequentially copied data received from the source volume to the mirror volume;

[0082] The data update module is used to update the data at the corresponding location in the mirror volume after receiving the data at the corresponding location in the snapshot volume.

[0083] Optionally, the CBT module is further configured to: continuously monitor data changes based on a step window, and gradually increase the monitoring range within the data backup time window as the data backup progress increases, without marking data changes occurring outside the monitoring range.

[0084] Optionally, the CBT module may only store the location information of the changed data when marking it, without storing the data itself.

[0085] Optionally, the production server further includes: a snapshot destruction module, used to destroy snapshots when all positions in the location marking table have been processed;

[0086] Optionally, the snapshot volume device is also configured to concurrently read new data from the snapshot volume based on the location of the CBT record after the snapshot is created, and send it to the disaster recovery server.

[0087] Optionally, the stored data includes cloud storage data and local storage data.

[0088] The novel CBT proposed in this invention dynamically adjusts the data monitoring range through step window control, reducing unnecessary data block marking. Figure 5This is a diagram comparing the effects of CBT before and after the improvement. The target volume is divided into 7 data blocks (A, B, C, D, E, F), and multiple data changes occurred between time T0 and T6. Traditional CBT requires monitoring all blocks of the entire volume throughout the entire data backup time window (T0-T6), marking changes in 6 blocks, such as... Figure 5 As shown in the right half, the improved CBT does not fix the monitoring scope to all blocks of the entire volume. Instead, it gradually increases the monitoring scope as the data backup progresses within the data backup time window. Data changes occurring outside the monitoring scope do not need to be marked.

[0089] Figure 5 The middle triangle indicates the current backup location, and the leftmost block A and the block indicated by the triangle represent the monitoring range of the improved CBT. From Figure 5 As can be seen, the monitoring range gradually increases with the backup progress. Under the same data change conditions, the improved CBT only marks 3 data blocks. Changes in locations not yet copied during the backup process do not need to be recorded (D3, E4). When copying data to locations that have changed during subsequent backups, the latest data can be copied directly. Therefore, the improved CBT reduces the number of marked data blocks compared to the original, thereby reducing the number of data blocks that need to be processed for consistency. With a given IO performance, this is equivalent to shortening the overall backup process time and reducing RPO (lower RPO is better in disaster recovery systems).

[0090] Furthermore, this invention also improves the full backup algorithm by separating the full data copy from the full data consistency processing.

[0091] If at some point Tn, the rate of change of the data is expressed using λ n The degree to which data is written sequentially is indicated by γ. n This indicates that the data copying speed is expressed in v. n This indicates that the improved CBT records the number of Blocks. CBT (t) The time T for copying the CBT record's block. copy_cbt_block (t) Storage space occupied by snapshot volume cache Block snapshot (t), Additional IO Amplification Amount for Snapshots num_of_extra The calculation methods for (t) are as follows:

[0092]

[0093]

[0094]

[0095] IO num_of_extra (t)=(1 Read +1 Write )·Count(Block snapshot (t)) (4)

[0096] For example, if the source volume data is 1TB, the block size is set to 4KB, and the proportion of blocks whose data changes during a full backup remains stable at 5%, i.e., λ = 0.05, then using a traditional data backup algorithm, the snapshot volume needs to record more than 13.42 million additional blocks, resulting in an additional overhead of more than 27 million I / Os (one write I / O corresponds to one additional read I / O), and an additional 52GB of storage space (caching the original data before the changes). If the optimized backup algorithm is used, the snapshot volume only needs to record the data changes that occurred during the period of copying the changed block data. Under stable production operations, with a constant data change rate, the duration of the snapshot mode is only 5% of that of the traditional algorithm. The corresponding space occupied by the snapshot data is reduced to 670,000 blocks, and the storage I / O overhead is reduced to 1.34 million I / Os.

[0097] Example 3

[0098] Embodiment 3 of the present invention also provides a storage medium containing computer-executable instructions, on which a computer program is stored. When the program is executed by a processor, it implements a cloud storage data full disaster recovery method provided in other embodiments of the present invention.

[0099] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for full disaster recovery of cloud storage data, characterized in that, Step 100: Start the backup. When the backup starts, there is no need to create a snapshot. The block data of the source volume TargetVol is directly copied sequentially to the disaster recovery server, and the disaster recovery service area writes the block data to the mirror volume. Step 200: During the sequential copying of the TargetVol block data, the Change Block Tracking (CBT) module continuously monitors the data changes and creates a location marker table, which marks the location of the changed data. The CBT module continuously monitors data changes, including: the CBT module continuously monitors data changes in the copied area based on a step window; and the monitoring range of the CBT module gradually increases as the data backup progress increases within the data backup time window, and data changes occurring outside the monitoring range are not marked. Step 300: After completing the sequential copy of the TargetVol block data, create a snapshot and obtain the location marker table; Step 400: Merge data according to the location information in the location marking table, read data of the corresponding location from the snapshot volume, and send it to the disaster recovery server to update the data of the corresponding location in the image volume; Step 500: End backup.

2. The cloud storage data full disaster recovery method according to claim 1, characterized in that, Before ending the backup, the snapshot is destroyed when all positions in the location marker table have been processed.

3. The cloud storage data full disaster recovery method according to claim 1, characterized in that, After the snapshot is established, new data is read concurrently from the snapshot volume according to the location recorded by CBT and updated to the disaster recovery server to update the data at the corresponding location in the mirror volume.

4. A cloud storage data full disaster recovery system, comprising a production server and a disaster recovery server. The production server includes: An agent program is used to directly and sequentially copy block data from the source volume TargetVol to the disaster recovery server without creating a snapshot when the backup starts. The Change Block Tracking (CBT) module is used to continuously monitor data changes and create a location marker table during the sequential copying of TargetVol data, and to mark the location of changed data in the location marker table; the CBT module is also used to: continuously monitor data changes based on a step window, and gradually increase the monitoring range within the data backup time window as the data backup progress increases, and not mark data changes that occur outside the monitoring range; The snapshot module is used to create snapshots after the sequential copying of TargetVol is completed; A snapshot volume device is used to obtain the location marker table, merge data according to the location information in the location marker table, read data from the snapshot volume at the corresponding location, and send it to the disaster recovery server. The disaster recovery server includes: The data writing module is used to write sequentially copied data received from the source volume to the mirror volume; The data update module is used to update the data at the corresponding location in the mirror volume after receiving the data at the corresponding location in the snapshot volume.

5. The cloud storage data full disaster recovery system according to claim 4, characterized in that, The production server also includes: The snapshot destruction module is used to destroy snapshots when all positions in the location marker table have been processed. The snapshot volume device is also used to concurrently read new data from the snapshot volume according to the location of the CBT record after the snapshot is established, and send it to the disaster recovery server.

6. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the cloud storage data full disaster recovery method as described in any one of claims 1-3.

Images