Data protection method and system based on CDP

Abstract

The invention relates to the technical field of data protection, and provides a CDP-based data protection method and system, and the method comprises the steps: incremental snapshot generation: a data source agent monitors the data change on a main data source, and generates an incremental snapshot; real-time copying: the transmission engine transmits the incremental snapshot to a backup server in real time; snapshot management: the backup server manages the incoming incremental snapshot data, including operations of storage, indexing, compression and the like; and recovery operation: the user can select proper recovery options through the management console, such as a specific time point, a specific application program or a specific file level to carry out recovery operation. By adopting the method, a user can realize minimum data loss and fastest-speed recovery of key data.

Description

Technical Field

[0001] The present invention belongs to the technical field of data protection, and in particular, relates to a data protection method and system based on CDP. Background Art

[0002] In existing data protection methods, data is usually backed up or snapshotted before data transmission. However, this method cannot handle failures that occur during data transmission. Therefore, when a failure occurs, the entire data block needs to be retransmitted or complex data recovery operations need to be performed, which consumes a lot of time and resources and may cause data loss or damage.

[0003] Traditional data backup methods often require regular manual data backup, which carries the risk of data loss or damage. Therefore, a more reliable and efficient data protection method is needed. Summary of the invention

[0004] The present invention provides a CDP-based data protection method and system to solve the problems existing in the prior art.

[0005] In a first aspect, the present invention provides a data protection method based on CDP, comprising: Incremental snapshot generation: The data source agent monitors data changes on the primary data source and generates incremental snapshots; Real-time replication: The transfer engine transfers incremental snapshots to the backup server in real time; Snapshot management: The backup server manages the incoming incremental snapshot data, including storage, indexing, and compression operations; Recovery operations: Users can select appropriate recovery options through the management console, such as recovery operations at a specific point in time, specific applications, or specific file levels.

[0006] Furthermore, the data source agent includes: the data source agent is a middleware tool for data integration and data exchange, which acts as a bridge between the data source and the data target, and is responsible for extracting, converting and loading data from different data sources and transmitting it to the target system.

[0007] Further, the monitoring of data changes on the primary data source and generating incremental snapshots includes: the data source agent uses polling query, subscription / publishing mode or trigger to monitor data changes on the primary data source; once the data source agent detects data changes on the primary data source, it will capture relevant change information; based on the captured data change information, such as updated records, newly added records or deleted records, the data source agent can generate incremental snapshots according to business needs. The incremental snapshot can be the latest data set of addition, modification and deletion operations, or a snapshot containing only newly added data.

[0008] Furthermore, the transmission engine transmits the incremental snapshot to the backup server in real time, including: the data source agent transmits the incremental snapshot to the target location of the backup server in real time through the selected transmission engine, such as a network transmission protocol (such as FTP, SFTP, HTTP, HTTPS), a message queue system (such as RabbitMQ, Kafka) or a distributed file system (such as Hadoop HDFS), etc. The target location can be a local file system path of the server, a database table, a topic of a message queue, etc.

[0009] Furthermore, the operations of storage, indexing and compression include: Storage management: The backup server uses appropriate storage media (hard disk, tape, etc.) and storage technology (RAID, distributed storage, etc.) to effectively store incremental snapshot data. For large-scale data backup, the backup server may need to support horizontal expansion and high availability to meet storage requirements; Index management: The backup server creates indexes to track and manage incremental snapshot data. The index can provide the ability to quickly find and access specific data based on information such as file name, path, and timestamp; Compression and optimization: In order to save storage space and reduce network transmission volume, incremental snapshot data is compressed, and the compression algorithm is selected according to the data type and nature. In addition, data optimization is also performed, such as data deduplication and data block segmentation, to improve storage efficiency and backup speed; Security management: Ensuring the security and confidentiality of incremental snapshot data through encryption technology, access control, and identity authentication; Data integrity verification: Data integrity verification mechanism is implemented through checksum, hash algorithm or digital signature.

[0010] In a second aspect, the present invention provides a data protection system based on CDP, comprising: Data source agent: deployed on the primary data source, responsible for collecting data changes and generating incremental snapshots; Transfer engine: responsible for transferring incremental snapshot data to the backup storage device in real time; Backup server: used to save incremental snapshot data and provide multiple recovery options; Management Console: The user interface used to configure and manage the CDP system.

[0011] Furthermore, the system also includes a CDP server and multiple clients, the CDP server is used to store data, and the clients are used to collect data and copy the data to the CDP server in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the implementation of the CDP-based data protection method provided by the present invention;

[0013] Figure 2 It is a module diagram of the CDP-based data protection system provided by the present invention. Implementation

[0014] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0015] Figure 1 The following is a flow chart of the implementation of the CDP-based data protection method provided by the present invention, which specifically includes the following steps: Incremental snapshot generation: The data source agent monitors data changes on the primary data source and generates incremental snapshots; Real-time replication: The transfer engine transfers incremental snapshots to the backup server in real time; Snapshot management: The backup server manages the incoming incremental snapshot data, including storage, indexing, and compression operations; Recovery operations: Users can select appropriate recovery options through the management console, such as recovery operations at a specific point in time, specific applications, or specific file levels.

[0016] In the embodiment of the present invention, the specific process of generating an incremental snapshot can be divided into the following steps: The data source agent establishes a connection with the primary data source and continuously monitors data changes on the primary data source based on the block level or file level. Once the data source agent detects changes on the primary data source, it captures the details of these changes. For block-level CDP, the agent records the location, size, and content of the changed data blocks; for file-level CDP, the agent records the metadata information of the files that are created, modified, or deleted; Based on the captured data change information, the data source agent generates incremental snapshots. Incremental snapshots only contain changed data blocks or files, not the entire data set. The agent records these changes for subsequent data recovery operations.

[0017] In the embodiment of the present invention, real-time replication transmits incremental snapshots to the backup server in real time through a transmission engine. The transmission engine mainly functions as follows: Capable of capturing incremental changes in source data and transferring these changes to the backup server, avoiding the need to retransmit the entire data set each time, improving backup efficiency and reducing network bandwidth usage; It can monitor the changes of source data in real time and transmit these changes to the backup server in time to ensure the synchronization of backup data with source data; Support data encryption and security verification mechanism during transmission to ensure the security of data during transmission; It uses efficient data transmission algorithms and optimization strategies to automatically adjust transmission speed and priority according to network conditions, bandwidth limitations, system load and other factors to achieve the best transmission effect.

[0018] In the embodiment of the present invention, snapshot management is specifically divided into the following points: Storage management: The backup server uses appropriate storage media (hard disk, tape, etc.) and storage technology (RAID, distributed storage, etc.) to effectively store incremental snapshot data. For large-scale data backup, the backup server may need to support horizontal expansion and high availability to meet storage requirements; Index management: The backup server creates indexes to track and manage incremental snapshot data. The index can provide the ability to quickly find and access specific data based on information such as file name, path, and timestamp; Compression and optimization: In order to save storage space and reduce network transmission volume, incremental snapshot data is compressed, and the compression algorithm is selected according to the data type and nature. In addition, data optimization is also performed, such as data deduplication and data block segmentation, to improve storage efficiency and backup speed; Security management: Ensuring the security and confidentiality of incremental snapshot data through encryption technology, access control, and identity authentication; Data integrity verification: Data integrity verification mechanism is implemented through checksum, hash algorithm or digital signature.

[0019] In the embodiment of the present invention, the user can select a suitable recovery option through the management console to perform the operation. Depending on the specific situation, one of the following recovery options can be selected: Restore to a specific point in time: Users can choose to restore to a specific point in time, which will roll back the system to the state at that point in time. This is very useful for dealing with unexpected erroneous changes or data corruption issues; Specific application recovery: If only specific applications need to be recovered, users can select this option. This will ensure that only the specified application is returned to the previous state, while other parts remain unchanged; Specific file level recovery: Users can choose to recover specific files or folders. This is very useful for accidental deletion or damage of a single file or folder; After you select a restore option, the system performs the corresponding actions and restores the target to the selected state or point in time.

[0020] Figure 2 It is a module diagram of the CDP-based data protection system provided by the present invention, which specifically includes: Data source agent: deployed on the primary data source, responsible for collecting data changes and generating incremental snapshots; Transfer engine: responsible for transferring incremental snapshot data to the backup storage device in real time; Backup server: used to save incremental snapshot data and provide multiple recovery options; Management Console: The user interface used to configure and manage the CDP system.

[0021] In an embodiment of the present invention, a data source agent is a component deployed on a primary data source, and its main function is to collect data changes and generate incremental snapshots. The data source agent maintains a real-time connection with the primary data source and monitors data changes. When data in the primary data source changes, the data source agent captures and records these changes. The data source agent can also build incremental snapshots that include all data changes that have occurred since the last snapshot.

[0022] In an embodiment of the present invention, a transmission engine is a component responsible for transmitting incremental snapshot data to a backup storage device in real time. Its main function is to ensure the security and reliability of data and transmit incremental snapshot data from a primary data source to a backup storage device. The transmission engine communicates with a data source agent, obtains incremental snapshot data, and transmits it to a backup storage device. The transmission engine typically uses some transmission protocols and technologies, such as network transmission protocols (such as TCP / IP), data compression and encryption, etc., to provide secure and efficient data transmission services. It can adjust the transmission frequency according to configuration settings to meet different business needs.

[0023] In an embodiment of the present invention, a backup server is a device or system for storing incremental snapshot data and providing multiple recovery options. The backup server has the following features and functions: Data storage: The backup server saves incremental snapshot data by using efficient storage technologies such as hard disk arrays (RAID), network storage (NAS) or tape libraries. These storage devices provide large capacity and high reliability to ensure the security and persistence of backup data; Data compression and deduplication: The backup server has data compression and deduplication functions. Data compression can reduce the storage space of backup data, while data deduplication can exclude duplicate data blocks to reduce the storage and transmission of redundant data; Multiple recovery options: The backup server provides multiple recovery options to meet different data recovery needs. For example, you can perform a full recovery to restore the backup data to a specific point in time; you can also perform a partial recovery to restore only a specific data fragment such as a file, directory or database; Security and encryption: The backup server has security and encryption functions to protect the confidentiality and integrity of the backup data. Data encryption technology, such as the AES encryption algorithm, can be used to encrypt the backup data to prevent unauthorized access and tampering; Automation and Scheduling: The backup server supports automated backup and scheduling functions, and can perform backup operations according to a predetermined schedule or trigger event. This ensures the timeliness and continuity of backup data.

[0024] In the embodiment of the present invention, the management console is a user interface for configuring and managing the CDP system. In the management console, the user can perform the following operations: Configure CDP policy: Users can set the backup policy of the CDP system through the management console, including parameters such as backup frequency, retention time, data compression and encryption. Users can specify corresponding backup policies for different data sets according to different needs and priorities; Monitor backup status: The management console provides real-time monitoring of the status and progress of backup tasks. Users can view completed backup tasks, understand whether the backup is successful, and obtain detailed information such as the version number and timestamp of the backup data; Manage backup storage: Through the management console, users can manage the backup storage space of the CDP system. They can configure storage capacity, add or delete storage devices, and set the storage location and distribution of backup data; Data recovery and restoration: The management console provides data recovery and restoration functions. Users can select specific backup points as needed and restore data to a specified time point or target location. In addition, the management console also supports partial recovery, where users can choose to restore a file, folder or specific data object; Monitoring and reporting: The management console can monitor the performance and health of the CDP system and generate various reports. Users can view indicators such as system resource utilization, backup success rate, backup duration, etc. to evaluate the efficiency and reliability of the system.

[0025] Through the management console, users can easily configure and manage the CDP system to ensure real-time backup and rapid recovery of data. At the same time, the management console also provides functions such as operation log recording and permission management to enhance the security and controllability of the system.

[0026] In the embodiment of the present invention, a CDP server and multiple clients are also included. The CDP server is used to store data, and the clients are used to collect data and copy the data to the CDP server in real time.

[0027] The present invention provides a CDP-based data storage system, which realizes two-way data backup and can restore data at any time. When the client data is damaged, the corresponding version of data can be obtained from the CDP backup center, thereby realizing effective data backup and making it easier to restore data. At the same time, when restoring data from the CDP backup center, data security protection is also performed, thereby realizing safe storage of data.

[0028] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data protection method based on CDP, characterized in that: The method comprises the following steps: Incremental snapshot generation: The data source agent monitors data changes on the primary data source and generates incremental snapshots; Real-time replication: The transfer engine transfers incremental snapshots to the backup server in real time; Snapshot management: The backup server manages the incoming incremental snapshot data, including storage, indexing, and compression operations; Recovery operations: Users can select appropriate recovery options through the management console, such as recovery operations at a specific point in time, specific applications, or specific file levels.

2. The CDP-based data protection method according to claim 1, characterized in that: The data source agent includes: the data source agent is a middleware tool for data integration and data exchange, which acts as a bridge between the data source and the data target, and is responsible for extracting, converting and loading data from different data sources and transmitting it to the target system.

3. The CDP-based data protection method according to claim 1, characterized in that: The monitoring of data changes on the primary data source and generating incremental snapshots includes: Monitoring data changes: The data source agent can use polling queries, subscription / publishing mode or triggers to monitor data changes on the primary data source; Generate incremental snapshots: Based on the captured data change information, such as updated records, new records, or deleted records, the data source agent can generate incremental snapshots according to business needs. Incremental snapshots can be the latest data set of addition, modification, and deletion operations, or a snapshot containing only newly added data.

4. The CDP-based data protection method according to claim 1, characterized in that: The transmission engine transmits the incremental snapshot to the backup server in real time, including: selecting a transmission engine suitable for real-time transmission when creating a backup task, and configuring a target location for receiving the incremental snapshot on the backup server. Once the incremental snapshot changes, the data source agent transmits the incremental snapshot to the configured target location in real time through the selected transmission engine.

5. The CDP-based data protection method according to claim 1, characterized in that: The storage, indexing and compression operations include: using a highly available storage system, such as a RAID array or a distributed storage system, to ensure data reliability and persistence; building indexes for incremental snapshot data based on different identifiers or keywords, such as timestamps, file names, directory structures, etc., so that data can be quickly retrieved and restored when needed; compressing the received incremental snapshot data using compression algorithms such as Gzip, Zip, LZMA, etc.; implementing a data integrity verification mechanism through checksums, hash algorithms or digital signatures, etc.; the backup server can perform some data processing operations after receiving the incremental snapshot data, such as data format conversion, data cleaning, encryption, etc.

6. A data protection system based on CDP, characterized in that: The system includes the following modules: Data source agent: deployed on the primary data source, responsible for collecting data changes and generating incremental snapshots; Transfer engine: responsible for transferring incremental snapshot data to the backup storage device in real time; Backup server: used to save incremental snapshot data and provide multiple recovery options; Management Console: The user interface used to configure and manage the CDP system.

7. The CDP-based data protection system according to claim 6, characterized in that: It also includes a CDP server and multiple clients, the CDP server is used to store data, and the clients are used to collect data and copy the data to the CDP server in real time.

Images