A continuous data protection method based on multi-cloud architecture

Abstract

This invention belongs to the field of data protection technology, specifically referring to a continuous data protection method based on a multi-cloud architecture. The technical solution is a continuous data protection method based on a multi-cloud architecture, comprising the following steps: constructing a distributed continuous data protection system across multiple cloud nodes, each cloud node including a data capture module, a log storage module, a metadata management module, and a cross-cloud synchronization module; when the virtual machine disk is in continuous data protection mode, the data capture module intercepts write I / O operations of the virtual machine disk at the virtualization layer, extracts the modified disk data to generate a data change block, and simultaneously generates corresponding metadata records; improving system reliability by synchronizing data logs across multiple cloud nodes, so that data can still be recovered from other cloud nodes when a single cloud node fails.

Description

Technical Field

[0001] This invention belongs to the field of data protection technology, specifically referring to a continuous data protection method based on a multi-cloud architecture. Background Technology

[0002] With the development of cloud computing technology, more and more enterprise business systems are being deployed in virtualization platforms and cloud computing environments. To ensure that business systems can recover quickly in the event of misoperation, software failure, or disaster, continuous data protection technology is widely used in the field of data protection.

[0003] Continuous data protection technology records every write operation in real time, enabling the system to restore the data state to any point in time, thereby avoiding the data loss problem caused by traditional periodic backups.

[0004] In existing technologies, most continuous data protection systems are typically deployed in a single data center or cloud environment, with their data logs and metadata usually stored within the same storage system. When a data center-level failure or cloud platform failure occurs, the continuous data protection system itself may not be able to provide reliable data recovery capabilities, thereby impacting business continuity.

[0005] Furthermore, in existing technologies, data recovery typically relies on a single node reading log data, lacking a cross-node recovery scheduling mechanism. In multi-node environments, it is difficult to fully utilize the network and storage resources of different nodes, thus affecting data recovery efficiency. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a continuous data protection method based on a multi-cloud architecture.

[0007] The technical solution adopted by this invention is as follows: This invention provides a continuous data protection method based on a multi-cloud architecture, comprising the following steps: S1. Construct a distributed continuous data protection system across multiple cloud nodes. Each cloud node includes a data capture module, a log storage module, a metadata management module, and a cross-cloud synchronization module. S2. When the virtual machine disk is in continuous data protection mode, the data capture module intercepts the write I / O operation of the virtual machine disk at the virtualization layer, extracts the modified disk data to generate a data change block, and generates the corresponding metadata record. S3. The data change block is written to the data log volume of the local cloud node in an append-only manner, and the metadata record is written to the metadata log volume; The metadata record includes a timestamp, write start address, data length, and storage offset address of the data change block in the data log volume; S4. The system control node assigns an incrementing log sequence number to each data change block according to the writing order, and maintains a unified log sequence numbering system across multiple cloud nodes. S5. The cross-cloud synchronization module copies the data change blocks in the data log volume according to the log sequence number order, and sends the data change blocks to the remote cloud node through the network transmission module. The cross-cloud synchronization operation is triggered when a new data change block is added to the data log volume. S6. Construct a unified timeline index structure among multiple cloud nodes based on the log sequence number. Each index node includes: timestamp, log sequence number, data log volume number, and data offset address. S7. When a new metadata record is generated, the corresponding index node is appended to the end of the time axis index structure in timestamp order. S8. When a data recovery request for a specified recovery time point is received, locate the corresponding metadata record set according to the time axis index structure; S9. Read the data change block from the data log volume of the corresponding cloud node according to the metadata record set, and perform the data replay operation in chronological order. The data replay operation includes writing the data change block to the corresponding storage location of the reference data copy in a block overwrite manner according to the write start address and data length in the metadata record, thereby restoring the virtual machine disk data at the specified time point.

[0008] Furthermore, the multi-cloud architecture includes at least two of the following: public cloud platform, private cloud platform, and edge cloud platform.

[0009] Furthermore, the data capture module is deployed in the virtualization layer or storage virtualization layer to intercept write I / O operations before they are submitted to the underlying storage device; the data change block is the disk block or disk sector data modified during the write I / O operation.

[0010] Furthermore, the timeline index structure adopts a linked list structure or a tree index structure sorted by timestamp.

[0011] Furthermore, when the data log volume capacity reaches a preset capacity threshold, the system automatically creates a new data log volume and updates the log sequence number; the capacity threshold is a preset storage capacity between 200MB and 2GB.

[0012] Furthermore, the cross-cloud synchronization module employs an asynchronous replication mechanism for data synchronization; during the cross-cloud synchronization process, a data verification algorithm is used to verify the integrity of the synchronized data.

[0013] This invention also discloses a continuous data protection system based on a multi-cloud architecture, comprising: The data capture module is used to intercept virtual machine disk write I / O operations and generate data change blocks; The log storage module is used to store data change blocks; The metadata management module is used to generate and record metadata records; The cross-cloud synchronization module is used to synchronize data logs across multiple cloud nodes according to the log sequence number. The recovery scheduling module is used to select data reading nodes during data recovery; The data recovery module is used to perform data replay operations based on metadata records to recover data at a specified point in time.

[0014] A unified control node is used to assign log sequence numbers and manage data protection policies for multiple cloud nodes.

[0015] Furthermore, the log storage module includes a data log volume and a metadata log volume.

[0016] Furthermore, the recovery scheduling module calculates a recovery priority score based on network latency, storage node load, and data integrity status; and selects the cloud node with the highest recovery priority score as the data reading node.

[0017] Furthermore, the data change block is compressed before being written to the data log volume, and the system is suitable for KVM, Xen, or VMware virtualization environments.

[0018] The beneficial effects achieved by the present invention using the above structure are as follows: (1) Improve system reliability. By synchronizing data logs between multiple cloud nodes, data can still be recovered from other cloud nodes when a single cloud node fails.

[0019] (2) Supports cross-cloud disaster recovery. Data logs are stored in copies across multiple cloud nodes, enabling data recovery across different cloud environments.

[0020] (3) Ensure log order consistency. Through a unified log sequence numbering mechanism, ensure log order consistency in a multi-cloud environment.

[0021] (4) Improve recovery efficiency. The recovery scheduling module selects the best recovery node based on network latency and node load to improve recovery speed.

[0022] (5) Supports recovery at any point in time. By utilizing the time axis index structure and data replay mechanism, the data state at any historical point in time can be restored. Detailed Implementation

[0023] The following are specific embodiments of the present invention, which further describe the technical solution of the present invention. However, the scope of protection of the present invention is not limited to these embodiments. All changes or equivalent substitutions that do not depart from the concept of the present invention are included within the scope of protection of the present invention.

[0024] Example 1 This embodiment provides a continuous data protection system based on a multi-cloud architecture, which includes multiple cloud nodes and at least one control node.

[0025] Multiple cloud nodes can be deployed on different cloud platforms, including public cloud platforms, private cloud platforms, hybrid cloud platforms, and edge cloud platforms. Each cloud node includes the following functional modules: data capture module, log storage module, metadata management module, and cross-cloud synchronization module.

[0026] Specifically, the data capture module is deployed in the virtualization layer or storage virtualization layer to intercept write I / O operations on the virtual machine disk, intercept I / O requests at the hypervisor layer, intercept write operations at the storage virtualization driver layer, and obtain write data through the block device filter driver.

[0027] The log storage module is used to store data change blocks, and it includes a data log volume and a metadata log volume.

[0028] Data log volumes are used to store the actual data change blocks, while metadata log volumes are used to store the corresponding metadata records.

[0029] The control node is used to assign log sequence numbers to data change blocks, manage data protection policies for multiple cloud nodes, and coordinate cross-cloud data synchronization tasks. The control node can be deployed on any cloud node or as an independent management node.

[0030] Example 2 This example illustrates the specific process of data capture and logging.

[0031] When a user enables continuous data protection mode for a virtual machine disk, the system begins to monitor the write I / O operations of the virtual machine disk in real time.

[0032] When an application writes data to the virtual machine disk, the write request first reaches the virtualization layer. The data capture module intercepts the write I / O operation at the virtualization layer and obtains the write start address, write data content, write data length, and current timestamp. Subsequently, the system extracts the modified disk data into data change blocks.

[0033] Data change blocks can be divided into fixed-size disk blocks, such as 4KB, 8KB, and 16KB. Then the corresponding metadata record is generated.

[0034] Each metadata record includes at least a timestamp, the write start address, the data length, and the storage offset address of the data change block in the data log volume.

[0035] After generating the data change block and metadata record, the system performs the following operations: the data change block is written to the data log volume in append mode, and the metadata record is written to the metadata log volume.

[0036] Append-only writing can avoid frequent random writes and improve log writing efficiency.

[0037] Example 3 Log sequence number allocation mechanism To ensure consistent log order in a multi-cloud environment, this embodiment employs a unified log sequence numbering mechanism.

[0038] Once a data change block is generated, the system sends the metadata information of that data change block to the control node.

[0039] The control node assigns an incrementing log sequence number to each data change block in the order it is written.

[0040] Log sequence numbers can be generated in the following ways: incrementing integer numbering, timestamp plus counter combination numbering, and distributed ID generation algorithm.

[0041] In a preferred embodiment, an incrementing integer number is used as the log sequence number.

[0042] After the log sequence number is generated, it will be recorded in both the metadata log volume and the timeline index structure.

[0043] By using a unified log sequence number, the consistency of data change order across different cloud nodes can be guaranteed.

[0044] Example 4 Cross-cloud log synchronization process When a new data change block is added to the data log volume, the system will trigger the cross-cloud synchronization module to perform a synchronization task.

[0045] The cross-cloud synchronization module reads the data change blocks in the data log volume in the order of log sequence number.

[0046] Subsequently, the data change block is sent to the remote cloud node via the network transmission module.

[0047] In one implementation, the network transmission module can use the TCP protocol.

[0048] After receiving the data, the remote cloud node writes the data change block to the local data log volume and records the corresponding metadata.

[0049] To ensure the integrity of synchronized data, data verification operations can be performed during data transmission: CRC check, MD5 check, SHA check. When the verification results are consistent, the remote node confirms successful synchronization.

[0050] This mechanism allows data log copies to be stored across multiple cloud nodes, thereby improving the system's data reliability.

[0051] Example 5 Time axis index structure maintenance The timeline index structure is used to record the temporal order of all data change blocks.

[0052] In one implementation, the timeline index structure can be a linked list.

[0053] Each index node includes: timestamp, log sequence number, data log volume number, and data offset address.

[0054] When a new metadata record is generated, the system generates a new index node based on the record and appends the node to the end of the timeline index structure in timestamp order.

[0055] This index structure allows the system to quickly locate data change records at any given time point.

[0056] Example 6 Data recovery and scheduling mechanisms When a user initiates a data recovery request, a recovery time point needs to be specified. The system first searches for all data change records before that time point based on the timeline index structure. Then, the recovery scheduling module selects recovery nodes from multiple cloud nodes. The recovery scheduling module calculates the recovery priority score based on the following parameters: network latency, which represents the network transmission latency from the client to each cloud node; node storage load, which represents the current IO load of the storage system of each cloud node; and data integrity status, which represents whether the data logs of each node are complete.

[0057] Scoring formula: Score = w1 network latency + w2 storage load + w3 data integrity; Among them, w1, w2, and w3 are weight parameters, and the system selects the cloud node with the highest score as the data reading node.

[0058] Subsequently, the system reads the data change blocks from the data log volume of that node and performs the data replay operation in chronological order.

[0059] During data replay: Based on the write start address and data length in the metadata record, the system writes the data change block to the corresponding storage location of the reference data copy in a block overwrite manner.

[0060] By applying data change blocks one by one, the historical state of the virtual machine disk can be gradually restored.

[0061] Once all data change blocks have been replayed, the virtual machine disk data at the specified point in time can be obtained.

[0062] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for continuity data protection based on multi-cloud architecture, characterized in that: Includes the following steps: S1. Construct a distributed continuous data protection system across multiple cloud nodes. Each cloud node includes a data capture module, a log storage module, a metadata management module, and a cross-cloud synchronization module. S2. When the virtual machine disk is in continuous data protection mode, the data capture module intercepts the write I / O operation of the virtual machine disk at the virtualization layer, extracts the modified disk data to generate a data change block, and generates the corresponding metadata record. S3. The data change block is written to the data log volume of the local cloud node in an append-only manner, and the metadata record is written to the metadata log volume; The metadata record includes a timestamp, write start address, data length, and storage offset address of the data change block in the data log volume; S4. The system control node assigns an incrementing log sequence number to each data change block according to the writing order, and maintains a unified log sequence numbering system across multiple cloud nodes. S5. The cross-cloud synchronization module copies the data change blocks in the data log volume according to the log sequence number order, and sends the data change blocks to the remote cloud node through the network transmission module. The cross-cloud synchronization operation is triggered when a new data change block is added to the data log volume. S6. Construct a unified timeline index structure among multiple cloud nodes based on the log sequence number. Each index node includes: timestamp, log sequence number, data log volume number, and data offset address. S7. When a new metadata record is generated, the corresponding index node is appended to the end of the time axis index structure in timestamp order. S8. When a data recovery request for a specified recovery time point is received, locate the corresponding metadata record set according to the time axis index structure; S9. Read the data change block from the data log volume of the corresponding cloud node according to the metadata record set, and perform the data replay operation in chronological order. The data replay operation includes writing the data change block to the corresponding storage location of the reference data copy in a block overwrite manner according to the write start address and data length in the metadata record, thereby restoring the virtual machine disk data at the specified time point.

2. The continuous data protection method based on a multi-cloud architecture according to claim 1, characterized in that: The multi-cloud architecture includes at least two of the following: public cloud platform, private cloud platform, and edge cloud platform.

3. The continuous data protection method based on a multi-cloud architecture according to claim 2, characterized in that: The data capture module is deployed in the virtualization layer or storage virtualization layer and is used to intercept write I / O operations before they are submitted to the underlying storage device; the data change block is the disk block or disk sector data that is modified during the write I / O operation.

4. The continuous data protection method based on a multi-cloud architecture according to claim 3, characterized in that: The timeline index structure adopts a linked list structure or a tree index structure sorted by timestamp.

5. A continuous data protection method based on a multi-cloud architecture according to claim 4, characterized in that: When the data log volume capacity reaches the preset capacity threshold, the system automatically creates a new data log volume and updates the log sequence number; the capacity threshold is a preset storage capacity between 200MB and 2GB.

6. The continuous data protection method based on a multi-cloud architecture according to claim 5, characterized in that: The cross-cloud synchronization module uses an asynchronous replication mechanism for data synchronization; during the cross-cloud synchronization process, a data verification algorithm is used to verify the integrity of the synchronized data.

7. A continuous data protection system based on a multi-cloud architecture, comprising the continuous data protection method based on a multi-cloud architecture according to claim 6, characterized in that, include: The data capture module is used to intercept virtual machine disk write I / O operations and generate data change blocks; The log storage module is used to store data change blocks; The metadata management module is used to generate and record metadata records; The cross-cloud synchronization module is used to synchronize data logs across multiple cloud nodes according to the log sequence number. The recovery scheduling module is used to select data reading nodes during data recovery; The data recovery module is used to perform data replay operations based on metadata records to recover data at a specified point in time. A unified control node is used to assign log sequence numbers and manage data protection policies for multiple cloud nodes.

8. A continuous data protection system based on a multi-cloud architecture according to claim 7, characterized in that: The log storage module includes a data log volume and a metadata log volume.

9. A continuous data protection system based on a multi-cloud architecture according to claim 8, characterized in that: The recovery scheduling module calculates a recovery priority score based on network latency, storage node load, and data integrity status; and selects the cloud node with the highest recovery priority score as the data reading node.

10. A continuous data protection system based on a multi-cloud architecture according to claim 9, characterized in that: The data change blocks are compressed before being written to the data log volume. The system is suitable for KVM, Xen, or VMware virtualization environments.