Expandable continuous data protection method based on network

Abstract

The invention discloses a network-based expandable continuous data protection method, which comprises the following steps that: a client node intercepts a local write operation request and sends the local write operation request to a server system through a network; a monitoring module of the server side receives and forwards the request to a data storage module, and the data storage module additionally writes data into one or more physical block devices according to a receiving sequence; and when the storage space of the current equipment is used up, the monitoring module automatically switches to the next available equipment to continue additional writing, so that the dynamic expansion of the storage capacity is realized. According to the method, through a seamless capacity expansion mechanism, the problem that a traditional CDP system cannot be recovered at any time point due to limitation of a fixed storage space can be solved without interrupting service.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data storage and disaster recovery, and particularly relates to a network-based scalable continuous data protection method. BACKGROUND

[0002] Continuous data protection (CDP) technology is an important backup technology that realizes data recovery at any time point by capturing and recording data write operations in real time. However, in the implementation process of the existing CDP technology, the backup medium usually adopts a fixed local storage space, the capacity of which is preset and cannot be dynamically expanded. When the backup medium is full, the system will be forced to overwrite early data or stop backup, which directly destroys the core ability of CDP technology to recover to any historical moment, and seriously limits the long-term application of the technology in large-capacity data scenarios. Therefore, how to break through the physical limitation of local storage space and realize seamless dynamic expansion of backup capacity has become a key technical problem to be solved in the field. SUMMARY

[0003] (I) Invention purpose In order to overcome the above shortcomings, the purpose of the present application is to provide a network-based scalable continuous data protection method to solve the technical problem that the existing CDP technology cannot realize real recovery at any time point due to the fixed capacity of the backup medium and the inability to dynamically expand.

[0004] (II) Technical solutions In order to achieve the above purpose, the technical solutions provided by the present application are as follows: A network-based scalable continuous data protection method, comprising the following steps: S1, a client node intercepts a write operation request initiated to its local storage device; S2, the client node sends the write operation request to a server system through a network; S3, a monitoring module in the server system receives the write operation request and forwards it to a data saving module; S4, the data saving module appends the received write operation data to one or more physical block devices in the order of its reception; S5, when the storage space of the currently written physical block device is exhausted, the monitoring module automatically switches and appends the subsequent write operation data to the next available physical block device to realize dynamic expansion of storage capacity.

[0005] The application intercepts the write operation by the client node and forwards it to the server, coordinates the data writing into the physical block device by the monitoring module, and automatically switches to the next device when the storage space is exhausted. The structure realizes the dynamic expansion of the storage capacity of the CDP system without interrupting the service, and fundamentally solves the problem that the traditional CDP system cannot realize the real arbitrary time point recovery due to the limitation of the fixed storage space.

[0006] In some embodiments, in step S2, the write operation request sent by the client node encapsulates metadata, and the metadata includes: a client identifier for identifying the data source, a timestamp for identifying the time when the write operation occurs, and a data length of the write operation data.

[0007] By encapsulating the metadata including the client identifier, the timestamp and the data length in the write operation request, each piece of write operation data has traceability and context information. This design enhances the fineness of data management and provides a reliable metadata basis for subsequent data recovery, audit and multi-client isolation.

[0008] In some embodiments, the monitoring module maintains a write order table, and the write order table is used to record the write order of the plurality of physical block devices; the operation of "switching to the next available physical block device" in step S5 is performed based on the order indicated by the write order table.

[0009] The monitoring module maintains the write order table, which clearly specifies the write order of the plurality of physical block devices, thereby maintaining the global sequence consistency of data writing when switching the storage device. This mechanism effectively avoids data out-of-order or loss caused by device switching, and guarantees the consistency and reliability of the CDP timeline.

[0010] In some embodiments, the data preservation module includes one or more data preservation servers, and each data preservation server manages one or more physical block devices.

[0011] The data preservation module adopts a distributed server architecture, and each server is responsible for managing one or more physical block devices, which not only improves the processing capacity and throughput of the system, but also enhances the scalability and reliability of the system, facilitating the flexible deployment of hardware resources according to actual storage requirements.

[0012] In some embodiments, an independent CDP receiving process is created for each physical block device, and the CDP receiving process is responsible for sequentially writing the write operation data received by it into the corresponding physical block device.

[0013] An independent CDP receiving process is set up for each physical block device, which realizes parallel processing and device-level isolation of write operations, significantly improves data writing efficiency and system concurrency capability, and reduces the impact of single point failure on the overall system.

[0014] In some embodiments, the client node runs a kernel module, which is configured to provide a virtual block device interface to an upper layer operating system to receive write operation requests, to a lower layer management local storage devices to perform actual data read / write, and to send write operation requests to the server system through a network interface.

[0015] The client node implements the virtual block device interface through the kernel module, transparently intercepts and forwards the write operation to the server without modifying the upper layer application or operating system, which is easy to deploy and has no user awareness, improving the compatibility and ease of use of the system.

[0016] In some embodiments, the server system supports receiving and processing write operation requests from multiple different client nodes; when recovering data, the data from different client nodes is distinguished and recovered separately according to the client identifier carried in the write operation data.

[0017] The server supports concurrent writing of multiple clients and relies on the client identifier to achieve logical isolation of data, which can accurately distinguish and restore the data of each client when recovering, significantly improving the practicality and security of the system in a multi-tenant scenario.

[0018] In some embodiments, the write order table also records a state identifier for each physical block device, which indicates that the physical block device is currently in an "active", "full" or "faulty" state; wherein the monitoring module is specifically configured to: according to the order specified by the write order table and the state identifier, skip the physical block device in the "full" or "faulty" state, and use the physical block device in the "active" state as the next available physical block device.

[0019] By introducing a state identifier for the physical block device and combining it with the write order table, the system can intelligently skip full or faulty devices and automatically select available devices for continuous writing. This mechanism ensures data order while achieving fault tolerance and load balancing, enhancing the robustness and automation level of the system.

[0020] In some embodiments, when the monitoring module switches the write operation data to a new physical block device, the CDP receiving process corresponding to the new physical block device first writes a special metadata record to the block device; The metadata record at least contains a switching timestamp and the identifier of the previous physical block device; Wherein, during data recovery, the server system determines the data retrieval boundary across physical block devices according to the special metadata record.

[0021] The special metadata record containing the timestamp and the previous device identification is written at the device switching time, clear boundary marks are provided for cross-device data retrieval, the efficiency and accuracy of locating the target time point data during data recovery are greatly improved, and the method is especially suitable for large-scale distributed storage environment.

[0022] In some embodiments, The server system also sets an index service module; The index service module listens to the write operation data forwarded by the monitoring module, extracts the client identification and timestamp contained therein, and generates and maintains a global index therefrom; The global index records the mapping relationship between the write operation data of each client identification and the physical block device location; Wherein, during data recovery, the server system first queries the global index to determine the physical block device where the data at the target time point is located, and then reads the data from the determined physical block device.

[0023] By setting up an independent index service module, a global index is constructed and maintained, the physical location of data is quickly located according to the client and the timestamp, the problem of decreased retrieval performance caused by the increase in the number of devices is effectively solved, the recovery efficiency is no longer affected by the expansion, and the overall performance of the system is significantly optimized. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the system architecture diagram of the network-based scalable continuous data protection method of the present application; Figure 2 is the flow chart of the server system side in the network-based scalable continuous data protection method of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer and more explicit, the present application will be further described in detail below with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0026] The network-based scalable continuous data protection method provided by the present application, with reference to Figure 1 The system architecture, is composed of a client machine and a scalable continuous data protection server system, and the implementation steps are as follows: First, at the client node, by running a continuous data protection kernel module, intercept the write operation request initiated to its local storage device. The kernel module provides a virtual block device interface to the upper layer operating system to receive write operation requests; down through the logical block device (such as / dev / cdpdev) and parallelly execute two operations: on the one hand, manage the local storage device (such as / dev / sda) to perform actual data read and write (normal operation), on the other hand, through the network device (such as / sys / class / net / eth0) in the form of append write operation, send the write operation request to the server system through the TCP protocol. The kernel module intercepts and processes the write operation in a transparent manner, without modifying the operating system or application, thereby achieving the simplicity of deployment and compatibility of the system.

[0027] The client node sends the write operation request encapsulated with metadata to the server system through the network. The metadata includes the client identification for distinguishing different client sources, the timestamp for identifying the time when the write operation occurs, and the write operation data length. These metadata make each write operation have traceable context information, laying a foundation for subsequent data recovery and multi-client management.

[0028] The monitoring module (listening on 0.0.0.0:8999) in the server system receives the write operation request and forwards it to the server system through the TCP protocol. The monitoring module runs independently and is responsible for coordinating the allocation and sequence consistency of the write operation. The monitoring module maintains a write sequence table, which records the write sequence of multiple physical block devices, and each record contains a sequence number (Seq) and the corresponding data saving server IP address and listening port (such as Seq:1 corresponds to 172.16.100.100:3000, Seq:2 corresponds to 172.16.100.100:3001, etc.).

[0029] The data saving module includes one or more data saving servers (such as 172.16.100.100 and 172.16.100.101), and each server manages one or more physical block devices (such as / dev / sda, / dev / sdb, etc.). The data saving module appends the received write operation data to these physical block devices in the order of their reception. In particular, a separate CDP receiving process (such as listening ports 3000, 3001, 3002, etc.) is created for each physical block device, and each CDP receiving process is responsible for appending the write operation data it receives to the corresponding physical block device in a sequential write manner, thereby achieving device-level isolation and parallel processing, improving system throughput and fault tolerance.

[0030] When the storage space of the current physical block device for writing is exhausted, the monitoring module can automatically detect the state change and immediately seamlessly switch the subsequently arrived write operation data to the next available physical block device for additional writing, thereby realizing dynamic and online expansion of the storage capacity. Specifically, the monitoring module judges the storage exhaustion state by monitoring the remaining storage space of each physical block device in real time or receiving a "device full" signal from the data saving module. Once the switching condition is triggered, the monitoring module suspends writing to the current device and selects the next available device according to a preset strategy. All newly arrived write operation data will be immediately diverted to the new device for sequential addition, and the entire process does not require manual intervention and does not interrupt the ongoing backup service, thereby ensuring the continuity and availability of the CDP system.

[0031] On this basis, the monitoring module internally maintains a write sequence table in the form of a list or a database, which records the logical write sequence of all accessed physical block devices. Each table entry corresponds to a physical block device and contains its unique identifier, IP address (if applicable), current state identifier (such as "active", "full", or "faulty"), and its sequence number in the write chain. The state identifier is dynamically updated by the monitoring module according to the real-time status of the device: "active" indicates that the device is normal and writable; "full" indicates that the device space is exhausted; and "faulty" indicates that the device is offline or has read / write abnormalities. The switching operation is strictly performed according to the device sequence recorded in this table. When selecting the next device, the monitoring module starts from the next sequence number of the current device and sequentially checks the status of the subsequent devices, automatically skipping those inactive devices marked as "full" or "faulty", until the first device with a status of "active" is found as the write target. This mechanism not only ensures that the write operation maintains strict global sequence consistency among multiple physical devices, avoiding data out-of-order caused by device switching, but also significantly improves the system's tolerance to device failures and the utilization efficiency of storage resources, achieving true high availability.

[0032] Further, the server system is designed to simultaneously receive and process concurrent write operation requests from multiple different client nodes in the network. Each write operation request data packet encapsulates a unique client identifier (such as a client ID or MAC address). During data recovery, the server system uses this client identifier to accurately distinguish the write operation records belonging to different clients from the mixed data stream. The recovery process can extract and replay only the write operation sequence corresponding to a specific client, thereby achieving logical isolation and separate recovery of data. This not only enables a set of backup system to provide independent CDP services for multiple clients (multi-tenant support), but also completely eliminates the risk of data interference or leakage between different clients, enhancing the security and practicality of the system.

[0033] In particular, when the monitoring module performs a switchover operation to direct the write operation data stream to a new physical block device, the CDP receiving process corresponding to the new device will first write a special metadata record to the beginning of the block device before writing any actual user data. This record has a fixed, identifiable format, and its content contains at least two key fields: one is the precise switchover timestamp (usually taken from the system time of the monitoring module), used to record the time when the switchover event occurred; the other is the unique identifier of the previous physical block device (such as the device ID or logical unit number), used to indicate which device the data comes from. This special record physically forms an explicit boundary marker. When data recovery is performed in the future, the recovery engine can scan these special records on each physical block device to quickly build a complete, cross-device data stream timeline. When recovery to a specific time point is needed, the system can quickly locate the physical device range where the data at that time point is located, greatly reducing the search range and improving the efficiency and accuracy of the recovery operation, especially when dealing with massive historical data spanning multiple physical devices, the advantage is particularly obvious.

[0034] Preferably, the server system can also deploy a separate index service module, which runs in the form of a background process or microservice. Its core function is to continuously monitor and parse all write operation data streams forwarded by the monitoring module. For each write operation, the index service module extracts the embedded key metadata, mainly the client identifier and write operation timestamp. Subsequently, it uses this information to generate and maintain a global index. This global index is usually stored in memory or high-speed persistent storage using efficient data structures (such as B+ trees or hash tables), which establishes a direct mapping relationship between "client identifier + timestamp" and "physical block device identifier + device internal offset". In this way, when the user initiates a data recovery request, specifying the target client and a certain time point, the recovery system no longer needs to sequentially scan all physical block devices, but first queries this global index. The index service module can return the specific physical block device location where the target data is located within milliseconds, and the recovery process then directly accesses the specified location on the device to read the data. This "index first, then locate" mechanism makes the data retrieval efficiency almost independent of the number of underlying physical devices and the total amount of data, and is particularly suitable for large-scale, long-running distributed backup environments, greatly optimizing system performance.

[0035] In summary, the present application realizes seamless expansion and multi-client support of the continuous data protection system through client interception and forwarding, server coordination and multi-device dynamic switchover, effectively solving the problem that traditional CDP systems cannot achieve true arbitrary time point recovery due to storage space limitations.

[0036] It should be understood that the foregoing detailed description of the application, rather than limiting the application, is intended to explain and describe the current implementation of the application. Therefore, any modification, equivalent replacement or improvement made without departing from the spirit and scope of the application should be included in the protection scope of the application. In addition, the appended claims of the application are intended to cover all changes and modifications falling within the scope and boundary of the appended claims, or the equivalent form of such scope and boundary.

Claims

1. A network-based scalable continuous data protection method, characterized by, The following steps are involved: S1. The client node intercepts the write operation request initiated to its local storage device; S2. The client node sends the write operation request to the server system through the network; S3. The monitoring module in the server system receives the write operation request and forwards it to the data storage module; S4, the data storage module writes the received write operation data to one or more physical block devices in the order in which they are received; S5. When the storage space of the physical block device currently being written is exhausted, the monitoring module automatically switches and appends the subsequent write operation data to the next available physical block device to achieve dynamic expansion of storage capacity.

2. The method of claim 1, wherein, In step S2, the write operation request sent by the client node is encapsulated with metadata, and the metadata includes: a client identifier for identifying the source of data, a timestamp for identifying the time when the write operation occurs, and the data length of the write operation data.

3. The method according to claim 1 or 2, characterized in that, The monitoring module maintains a write sequence table, which is used to record the write sequence of multiple physical block devices; the operation of "switching to the next available physical block device" in step S5 is performed based on the order indicated by the write sequence table.

4. The method of claim 1, wherein, The data storage module includes one or more data storage servers, and each of the data storage servers manages one or more physical block devices.

5. The method of claim 4, wherein, An independent CDP receiving process is created for each of the physical block devices. The CDP receiving process is responsible for sequentially writing the received write operation data into its corresponding physical block device.

6. The method of claim 1, wherein, The client node runs a kernel module, which is used to: provide a virtual block device interface to the upper layer operating system to receive write operation requests; manage the local storage device to the lower layer to perform actual data reading and writing; and send write operation requests to the server system through the network interface.

7. The method of claim 1, wherein, The server system supports receiving and processing write operation requests from multiple different client nodes; when restoring data, the server system distinguishes and restores data from different client nodes according to the client identifiers carried in the write operation data.

8. The method of claim 3, wherein, The write order table also records a status identifier for each physical block device, where the status identifier is used to indicate whether the physical block device is currently in an "active", "full" or "failure" state; The monitoring module is specifically used for: According to the order specified by the write order table and the state identifier, the physical block device with the state of "full" or "failure" is skipped, and the physical block device with the state of "active" is used as the next available physical block device.

9. The method according to claim 8, characterized in that When the monitoring module switches the write operation data to a new physical block device, the CDP receiving process corresponding to the new physical block device first writes a special metadata record to the block device; The metadata record at least includes a switching timestamp and an identifier of a previous physical block device; During data recovery, the server system determines the data retrieval boundary across physical block devices based on the special metadata record.

10. The method according to claim 4, characterized in that The service end system further sets an index service module; The index service module listens to the write operation data forwarded by the monitoring module, and extracts the client identification and time stamp contained in the write operation data, to generate and maintain a global index; The global index records the mapping relationship between the write operation data of each client identification and the physical block device location; During data recovery, the service end system first queries the global index to determine the physical block device where the data at the target time point is located, and then reads the data from the determined physical block device.

Images