A network-based scalable continuous data protection method

Abstract

The application discloses a network-based scalable continuous data protection method, comprising the following steps: a client node intercepts a local write operation request and sends the request to a server system through a network; a monitoring module of the server receives the request and forwards the request to a data storage module, the module appends data to one or more physical block devices in a received order; when a current device storage space is exhausted, the monitoring module automatically switches to a next available device to continue appending data, thereby realizing dynamic expansion of the storage capacity. The method solves the problem that a traditional CDP system cannot realize recovery at an arbitrary time point due to the limitation of fixed storage space through a seamless expansion mechanism without interrupting service.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data storage and disaster recovery, and particularly relates to a network-based scalable continuous data protection method. BACKGROUND

[0002] Continuous data protection (CDP) technology is an important backup technology that realizes data recovery at any time point by capturing and recording data write operations in real time. However, in the implementation process of the existing CDP technology, the backup medium usually adopts a fixed local storage space, the capacity of which is preset and cannot be dynamically expanded. When the backup medium is full, the system will be forced to overwrite early data or stop backup, which directly destroys the core ability of CDP technology to recover to any historical moment, and seriously limits the long-term application of the technology in large-capacity data scenarios. Therefore, how to break through the physical limitation of local storage space and realize seamless dynamic expansion of backup capacity has become a key technical problem to be solved in the field. SUMMARY

[0003] (I) Invention purpose

[0004] In order to overcome the above shortcomings, the purpose of the present application is to provide a network-based scalable continuous data protection method to solve the technical problem that the existing CDP technology cannot realize real recovery at any time point due to the fixed capacity of the backup medium and the inability to dynamically expand.

[0005] (II) Technical solution

[0006] In order to achieve the above purpose, the technical solution provided by the present application is as follows:

[0007] A network-based scalable continuous data protection method, comprising the following steps:

[0008] S1, a client node intercepts a write operation request initiated to its local storage device;

[0009] S2, the client node sends the write operation request to a server system through a network;

[0010] S3, a monitoring module in the server system receives the write operation request and forwards it to a data saving module;

[0011] S4, the data saving module appends the received write operation data to one or more physical block devices in the order of its reception;

[0012] S5, when the storage space of the currently written physical block device is exhausted, the monitoring module automatically switches and appends the subsequent write operation data to the next available physical block device to realize dynamic expansion of storage capacity.

[0013] The present application intercepts the write operation by the client node and forwards it to the server, coordinates the data writing into the physical block device by the monitoring module in sequence, and automatically switches to the next device when the storage space is exhausted. The structure realizes the dynamic expansion of the storage capacity of the CDP system without interrupting the service, and fundamentally solves the problem that the traditional CDP system cannot realize the real arbitrary time point recovery due to the limitation of the fixed storage space.

[0014] In some embodiments, in step S2, the write operation request sent by the client node encapsulates metadata, and the metadata includes: a client identifier for identifying the data source, a timestamp for identifying the time when the write operation occurs, and a data length of the write operation data.

[0015] By encapsulating the metadata including the client identifier, the timestamp and the data length in the write operation request, each piece of write operation data has traceability and context information. This design enhances the fineness of data management and provides a reliable metadata basis for subsequent data recovery, audit and multi-client isolation.

[0016] In some embodiments, the monitoring module maintains a write sequence table, and the write sequence table is used to record the write sequence of the plurality of physical block devices; the operation of "switching to the next available physical block device" in step S5 is performed based on the sequence indicated by the write sequence table.

[0017] The monitoring module maintains the write sequence table, which clearly specifies the write sequence of the plurality of physical block devices, thereby maintaining the global sequence consistency of data writing when switching the storage device. This mechanism effectively avoids data out-of-order or loss caused by device switching, and guarantees the consistency and reliability of the CDP timeline.

[0018] In some embodiments, the data preservation module includes one or more data preservation servers, and each data preservation server manages one or more physical block devices.

[0019] The data preservation module adopts a distributed server architecture, and each server is responsible for managing one or more physical block devices, which not only improves the processing capacity and throughput of the system, but also enhances the scalability and reliability of the system, facilitating the flexible deployment of hardware resources according to actual storage requirements.

[0020] In some embodiments, an independent CDP receiving process is created for each physical block device, and the CDP receiving process is responsible for sequentially writing the write operation data received by it into the corresponding physical block device.

[0021] An independent CDP receiving process is set up for each physical block device, which realizes parallel processing and device-level isolation of write operations, significantly improves the data writing efficiency and system concurrency capability, and at the same time reduces the impact of single point failure on the overall system.

[0022] In some embodiments, the client node runs a kernel module, which is configured to provide a virtual block device interface to an upper layer operating system to receive write operation requests, to a lower layer management local storage devices to perform actual data read / write, and to send write operation requests to the server system through a network interface.

[0023] The client node implements the virtual block device interface through the kernel module, transparently intercepts and forwards the write operation to the server without modifying the upper layer application or operating system, which is easy to deploy and has no user awareness, improving the compatibility and ease of use of the system.

[0024] In some embodiments, the server system supports receiving and processing write operation requests from multiple different client nodes; when recovering data, the data from different client nodes is distinguished and recovered separately according to the client identifier carried in the write operation data.

[0025] The server supports concurrent writing of multiple clients and relies on the client identifier to achieve logical isolation of data, which can accurately distinguish and restore the data of each client when recovering, significantly improving the practicality and security of the system in a multi-tenant scenario.

[0026] In some embodiments, the write order table also records a state identifier for each physical block device, which indicates that the physical block device is currently in an "active", "full", or "faulty" state; wherein the monitoring module is specifically configured to: according to the order specified by the write order table and the state identifier, skip the physical block device in the "full" or "faulty" state, and use the physical block device in the "active" state as the next available physical block device.

[0027] By introducing a state identifier for the physical block device and combining it with the write order table, the system can intelligently skip full or faulty devices and automatically select available devices for continuous writing. This mechanism ensures data order while achieving fault tolerance and load balancing, enhancing the robustness and automation level of the system.

[0028] In some embodiments, when the monitoring module switches the write operation data to a new physical block device, the CDP receiving process corresponding to the new physical block device first writes a special metadata record to the block device;

[0029] The metadata record at least contains a switching timestamp and the identifier of the previous physical block device;

[0030] Wherein, during data recovery, the server system determines the data retrieval boundary across physical block devices according to the special metadata record.

[0031] When switching devices, a special metadata record containing a timestamp and the identifier of the previous device is written, providing a clear boundary marker for cross-device data retrieval. This greatly improves the efficiency and accuracy of locating data at the target time point during data recovery, and is especially suitable for large-scale distributed storage environments.

[0032] In some embodiments,

[0033] The server system also includes an index service module;

[0034] The index service module listens to the write operation data forwarded by the monitoring module and extracts the client identifier and timestamp contained therein to generate and maintain a global index.

[0035] The global index records the mapping relationship between write operation data for each client identifier and the physical block device location;

[0036] During data recovery, the server system first queries the global index to determine the physical block device where the data at the target time point is located, and then reads the data from the determined physical block device.

[0037] By establishing an independent index service module, building and maintaining a global index, the physical location of data can be quickly located by client and timestamp. This effectively solves the problem of decreased retrieval performance caused by the increase in the number of devices, and ensures that recovery efficiency is no longer affected by expansion, resulting in a significant optimization of the overall system performance. Attached Figure Description

[0038] Figure 1 This is a system architecture diagram of the network-based scalable continuous data protection method of the present invention;

[0039] Figure 2 This is a flowchart of the server-side process in the network-based scalable continuous data protection method of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0041] This invention provides a network-based, scalable, continuous data protection method, with reference to... Figure 1 The system architecture shown consists of two main parts: client machines and a scalable continuous data protection server system. Its implementation steps are as follows:

[0042] First, on the client node, a continuous data protection kernel module intercepts write operation requests destined for its local storage device. This kernel module provides a virtual block device interface to the upper-layer operating system to receive write operation requests; downwards, it performs two operations in parallel through a logical block device (such as / dev / cdpdev): on the one hand, it manages the local storage device (such as / dev / sda) to perform the actual data read and write (normal operation); on the other hand, it sends the write operation request to the server system via TCP protocol in an append-only manner through a network interface device (such as / sys / class / net / eth0). This kernel module intercepts and processes write operations transparently without modifying the operating system or application, thus achieving simplicity of deployment and system compatibility.

[0043] The client node sends a write operation request, encapsulated with metadata, to the server system over the network. This metadata includes a client identifier to distinguish different client origins, a timestamp indicating when the write operation occurred, and the length of the write operation data. This metadata ensures that each write operation has traceable contextual information, laying the foundation for subsequent data recovery and multi-client management.

[0044] The monitoring module in the server system (listening at 0.0.0.0:8999) receives the write operation request and forwards it to the server system via TCP protocol. The monitoring module operates independently and is responsible for coordinating the allocation and sequential consistency of write operations. The monitoring module maintains a write sequence table, which records the write order of multiple physical block devices. Each record contains a sequence number (Seq) and the corresponding data storage server IP address and listening port (e.g., Seq:1 corresponds to 172.16.100.100:3000, Seq:2 corresponds to 172.16.100.100:3001, etc.).

[0045] The data storage module comprises one or more data storage servers (e.g., 172.16.100.100 and 172.16.100.101), each managing one or more physical block devices (e.g., / dev / sda, / dev / sdb, etc.). The data storage module appends received write operation data to these physical block devices in the order it is received. Specifically, an independent CDP receiving process (e.g., listening on ports 3000, 3001, 3002, etc.) is created for each physical block device. Each CDP receiving process is responsible for appending its received write operation data to the corresponding physical block device in a sequential write manner, thereby achieving device-level isolation and parallel processing, improving system throughput and fault tolerance.

[0046] When the storage space of the currently written physical block device is exhausted, the monitoring module can automatically detect this status change and immediately and seamlessly switch subsequent write operations to the next available physical block device for appending, thereby achieving dynamic, online expansion of storage capacity. Specifically, the monitoring module determines the storage exhaustion status by monitoring the remaining storage space of each physical block device in real time or receiving a "device full" signal from the data storage module. Once the switching condition is triggered, the monitoring module will pause writing to the current device and select the next available device according to a preset strategy. All newly arrived write operations will be immediately redirected to the new device for sequential appending. The entire process requires no manual intervention and will not interrupt the ongoing backup service, thus ensuring the continuity and availability of the CDP system.

[0047] Building upon this, the monitoring module maintains a write order table, stored as a list or database, recording the logical write order of all connected physical block devices. Each entry corresponds to a physical block device and includes its unique identifier, IP address (if applicable), current status identifier (e.g., "active," "full," or "faulty"), and its sequence number in the write chain. The status identifier is dynamically updated by the monitoring module based on the real-time device status: "active" indicates the device is normal and writable; "full" indicates the device space is exhausted; and "faulty" indicates the device is offline or experiencing read / write errors. Switching operations are strictly performed according to the device order recorded in this table. When selecting the next device, the monitoring module starts from the next sequence number after the current device, checking the status of subsequent devices sequentially, automatically skipping inactive devices marked "full" or "faulty," until the first "active" device is found as the write target. This mechanism not only ensures strict global sequential consistency of write operations across multiple physical devices, avoiding data out-of-order issues that may result from device switching, but also significantly improves the system's tolerance to device failures and the utilization efficiency of storage resources, achieving true high availability.

[0048] Furthermore, the server system is designed to simultaneously receive and process concurrent write operation requests from multiple different client nodes across the network. Each write operation request packet encapsulates a unique client identifier (such as a client ID or MAC address). During the data recovery phase, the server system utilizes this client identifier to accurately distinguish write operation records belonging to different clients from a mixed data stream. The recovery process can target specific clients, extracting and replaying only their corresponding write operation sequences, thereby achieving logical data isolation and separate recovery. This not only enables a single backup system to provide independent CDP (multi-tenant support) services to multiple clients but also completely eliminates the risk of data interference or leakage between different clients, enhancing the system's security and usability.

[0049] Specifically, when the monitoring module performs a switchover operation, directing the write operation data stream to a new physical block device, the CDP receiving process corresponding to that new device first writes a special metadata record to the beginning of that block device before writing any actual user data. This record has a fixed, identifiable format, and its content includes at least two key fields: first, a precise switchover timestamp (usually taken from the monitoring module's system time), used to record the time when the switchover event occurred; and second, a unique identifier of the previous physical block device (such as device ID or logical unit number), used to indicate which device the data originated from. This special record physically forms a clear boundary marker. During subsequent data recovery, the recovery engine can scan these special records on each physical block device to quickly construct a complete, cross-device data flow timeline. When it is necessary to recover to a specific point in time, the system can quickly locate the range of physical devices where the data at that point in time is located, greatly narrowing the search scope and improving the efficiency and accuracy of the recovery operation, especially when processing massive amounts of historical data spanning multiple physical devices, where the advantages are particularly evident.

[0050] Preferably, the server-side system can also deploy a separate index service module, which runs as a background process or microservice. Its core function is to continuously monitor and parse all write operation data streams forwarded by the monitoring module. For each write operation, the index service module extracts the embedded key metadata, primarily the client identifier and the write operation timestamp. It then uses this information to generate and maintain a global index. This global index is typically stored in memory or high-speed persistent storage using an efficient data structure (such as a B+ tree or hash table), establishing a direct mapping from "client identifier + timestamp" to "physical block device identifier + device offset." In this way, when a user initiates a data recovery request, specifying the target client and a specific time point, the recovery system no longer needs to sequentially scan all physical block devices; instead, it first queries this global index. The index service module can return the specific physical block device location of the target data within milliseconds, allowing the recovery process to directly access the specified location on that device to read the data. This "index first, locate later" mechanism makes data retrieval efficiency almost unaffected by the number of underlying physical devices and the total amount of data, making it particularly suitable for large-scale, long-term distributed backup environments, and greatly optimizing system performance.

[0051] In summary, this invention achieves seamless expansion and multi-client support for the continuous data protection system through client-side interception and forwarding, server-side coordination, and dynamic switching of multiple devices, effectively solving the problem that traditional CDP systems cannot achieve true recovery at any point in time due to storage space limitations.

[0052] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A network-based scalable continuous data protection method, characterized by, The method comprises the following steps: S1, a client node intercepts a write operation request initiated to its local storage device; S2, the client node sends the write operation request to a server system through a network; S3, a monitoring module in the server system receives the write operation request and forwards it to a data saving module; S4, the data saving module appends the received write operation data to one or more physical block devices in the order of reception; S5, when the storage space of the currently written physical block device is exhausted, the monitoring module automatically switches and appends the subsequent write operation data to the next available physical block device to realize dynamic expansion of storage capacity; The server system also sets an index service module; The index service module listens to the write operation data forwarded by the monitoring module and extracts the client identifier and timestamp contained therein to generate and maintain a global index; The global index records the mapping relationship between the write operation data of each client identifier and the physical block device location; During data recovery, the server system first queries the global index to determine the physical block device where the data at the target time point is located, and then reads the data from the determined physical block device.

2. The method of claim 1, wherein, In step S2, the write operation request sent by the client node encapsulates metadata, which includes a client identifier for identifying the data source, a timestamp for identifying the time when the write operation occurs, and the data length of the write operation data.

3. The method according to claim 1 or 2, characterized in that, The monitoring module maintains a write order table, which records the write order of multiple physical block devices; the operation of "switching to the next available physical block device" in step S5 is performed based on the order indicated by the write order table.

4. The method of claim 1, wherein, The data saving module includes one or more data saving servers, and each data saving server manages one or more physical block devices.

5. The method of claim 4, wherein, A separate CDP receiving process is created for each physical block device, which is responsible for sequentially writing the write operation data received by it to the corresponding physical block device.

6. The method of claim 1, wherein, The client node runs a kernel module, which provides a virtual block device interface to the upper layer operating system to receive write operation requests, manages the local storage device to perform actual data read and write, and sends write operation requests to the server system through a network interface.

7. The method of claim 1, wherein, The server system supports receiving and processing write operation requests from multiple different client nodes; during data recovery, the data from different client nodes is distinguished and recovered separately according to the client identifier carried in the write operation data.

8. The method of claim 3, wherein, The write order table also records a state identifier for each physical block device, which indicates that the physical block device is currently "active", "full" or "faulty"; The monitoring module is specifically used for: According to the order specified by the write order table and the state identifier, the physical block devices in the "full" or "faulty" state are skipped, and the physical block devices in the "active" state are used as the next available physical block device.

9. The method of claim 8, wherein, when the monitoring module switches the write operation data to a new physical block device, a CDP receiving process corresponding to the new physical block device first writes a special metadata record to the block device; the metadata record at least contains a switching timestamp and an identification of a previous physical block device; wherein, in data recovery, the server system determines a data retrieval boundary across physical block devices according to the special metadata record.

Images