A remote backup method and system suitable for narrow bandwidth environments

Abstract

This invention discloses a remote backup method and system suitable for narrow bandwidth environments, relating to the field of data backup and recovery technology. The method includes the following steps: deploying a cache server to build a centralized relay architecture; receiving backup instructions and determining the backup type; if it is an incremental backup, entering incremental caching mode; if it is the first or full backup, entering full pass-through mode; performing sparse file, bitmap, and metadata collaborative capture for incremental backup; performing pass-through forwarding mode for full backup; and using QoS-based intelligent bandwidth utilization and transmission priority scheduling. This invention accurately captures incremental data through a dual-file mechanism of sparse files and bitmaps, avoiding storage resource waste. Combined with a dual QoS strategy of snapshot pre-transmission and dynamic priority scheduling, it maximizes narrow bandwidth resource utilization, balances fairness in multi-client backup with core business priority, and efficiently adapts to narrow bandwidth, high latency cross-data center backup scenarios.

Description

Technical Field

[0001] This invention relates to the field of data backup and recovery technology, specifically to a remote backup method and system suitable for narrow bandwidth environments. Background Technology

[0002] With the rapid development of information technology, data has become one of the core assets of enterprises. Data security and disaster recovery capabilities have become an important part of information system construction. Remote backup, as a key means of ensuring data security, can effectively deal with the risk of data loss caused by local disasters (such as fire, earthquake, ransomware attack, etc.), especially in data protection scenarios across regions and data centers. However, in practical applications, the communication network between the source client and the remote backup server is mostly a narrow bandwidth, high latency Internet channel or VPN link. Such network environments have limited bandwidth resources and high transmission latency, which puts forward stringent requirements on the efficiency and stability of remote backup.

[0003] Currently, the typical implementation of remote backup solutions on the market is as follows: A Copy-on-Write (COW) snapshot operation is performed in the block-level backup driver on the source client. The relevant data generated by the COW snapshot is stored in local files on the source end. Then, a scheduled task extracts the data from this snapshot, compresses and deduplicates the data before transmission, and finally transmits the processed data to the remote backup server to complete the backup. Although existing solutions are widely used, they still have the following significant limitations in narrow bandwidth and high-latency network environments:

[0004] The source-side business is severely affected: COW snapshot operations will significantly increase the IO pressure on the source side, leading to a decrease in source-side IO performance. At the same time, the local storage of COW data requires a large amount of storage space on the source side, and the compression and deduplication processing before data transmission will also consume a large amount of CPU resources on the source side. Multiple resource occupations directly affect the normal operation of the source-side core business.

[0005] Low utilization of narrow bandwidth resources: The existing solution adopts a timed transmission mode, which is not optimized for narrow bandwidth characteristics and cannot dynamically adjust the transmission strategy according to the network status, which can easily lead to bandwidth idleness or sudden congestion.

[0006] Insufficient fairness in multi-client backup: When multiple clients back up simultaneously, a starvation phenomenon occurs, meaning that some clients' backup tasks are delayed for a long time because they cannot compete for bandwidth, while other clients occupy the channel for a long time, resulting in poor overall efficiency and fairness of backup tasks. Summary of the Invention

[0007] This invention provides a remote backup method and system suitable for narrow bandwidth environments, which can effectively solve the problems of severe impact on source-end services, low utilization of narrow bandwidth resources, and insufficient fairness of multi-client backup in the existing solutions mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a remote backup method suitable for narrow bandwidth environments, comprising the following steps:

[0009] S1. Deploy cache servers and build a centralized relay architecture;

[0010] S2. Receive backup command, determine backup type. If it is incremental backup, enter incremental cache mode and execute S3. If it is first or full backup, enter full pass mode and execute S4.

[0011] S3. Perform incremental backups by co-capturing sparse files, bitmaps, and metadata.

[0012] S4. Perform a full backup in pass-through forwarding mode;

[0013] S5. QoS-based intelligent bandwidth utilization and transmission priority scheduling;

[0014] S6. Server-side backup complete, and cache server resources cleaned up.

[0015] According to the above technical solution, step S1 specifically includes the following steps:

[0016] S11. Cache server deployment and resource configuration;

[0017] S12. Initialize the built-in modules of the cache server;

[0018] S13. Establish the connection between the source client and the cache server;

[0019] S14. Establish a remote link between the cache server and the remote backup server;

[0020] In step S11, a physical or virtual cache server is deployed inside the source data center, and its storage, CPU, and memory are configured according to the number of backup clients and the amount of business data.

[0021] S12 includes a data receiving module, a storage management module, a narrowband transmission module, and an intelligent QoS management module;

[0022] In step S13, a lightweight block-level driver module needs to be installed on the backup source client. The driver module copies all write operations of the source disk to the cache server in a mirror manner.

[0023] In step S14, a secure transmission protocol and a breakpoint resume mechanism are configured.

[0024] According to the above technical solution, step S2 specifically includes the following steps:

[0025] S21. Receive backup trigger command;

[0026] S22. Parse and determine the backup type;

[0027] In step S21, the data receiving module of the cache server listens for backup instructions sent by the management terminal.

[0028] In step S22, if it is an incremental backup, the system enters the incremental caching mode, sends a signal to the source client to prepare to start the snapshot, and jumps to step S3. If it is a full backup, the system enters the full pass-through mode, the source client prepares to read the full data, and jumps to step S4.

[0029] According to the above technical solution, step S3 specifically includes the following steps:

[0030] S31. Create and initialize the sparse file;

[0031] S32. Create and initialize bitmap files and metadata indexes;

[0032] S33. Write data and dynamically update the bitmap and metadata hash table;

[0033] S34. Receive the snapshot end command and complete the snapshot capture;

[0034] S35, Notification transmission service;

[0035] In step S31, a sparse file with the same size as the source client's disk is created in the file system of the cache server;

[0036] The sparse file is used to record the location information of the incremental data written in this snapshot. When the source client has a data write operation, the data receiving module of the cache server will write the data to the corresponding position in the sparse file.

[0037] In step S32, a bitmap file corresponding to the sparse file is created in the cache server. The size of the bitmap file is related to the number of data blocks divided in the sparse file. At the same time, a hash table with the data logical offset address as the key is created.

[0038] Bitmap files are used to record whether each data block has changed. A bit value of 0 indicates no change, and a bit value of 1 indicates a change.

[0039] The metadata management structure uses a hash-type data structure to record the location and length information of each IO operation;

[0040] In S33, the source client performs a write operation, the block-level driver captures the I / O, and sends its content, logical offset address and data length to the cache server. The cache server writes the data to the corresponding offset position in the sparse file, sets the value of the corresponding bit in the bitmap file to 1, and inserts or updates the physical offset address and data length corresponding to the logical offset address in the metadata hash table.

[0041] S34 is that when the data receiving module detects the snapshot end signal, it stops writing data to the current sparse file, marks its status as read-only, uses it as the snapshot data file for this incremental backup, and simultaneously creates a new blank sparse file and metadata hash table, and guides the subsequent IO data generated by the client to write to the new sparse file.

[0042] S35 is when the cache server completes the snapshot instruction processing, sparse file, bitmap file and metadata hash table maintenance operations, it notifies the narrowband transmission module that there is new incremental backup data to be transmitted.

[0043] According to the above technical solution, step S4 specifically includes the following steps:

[0044] S41. Capture of the entire source data stream;

[0045] S42, Data stream memory transfer and processing;

[0046] S43. The compressed and deduplicated data stream is transmitted directly.

[0047] In step S41, the source client reads the data from the entire disk and sends it to the cache server in the form of a data stream.

[0048] In step S42, after the data receiving module of the cache server receives the data stream, it does not write it to the disk, but sends it directly into the processing pipeline. The narrow bandwidth transmission module performs compression and deduplication calculations on the data stream in memory.

[0049] In step S43, the processed data is directly forwarded to the remote backup server by the narrow bandwidth transmission module.

[0050] According to the above technical solution, S5 is designed to maximize the utilization of bandwidth resources by employing a dual strategy of snapshot pre-transmission and dynamic priority scheduling, while ensuring the fairness of multi-client backup and the backup priority of core services, in response to the narrow bandwidth characteristics.

[0051] According to the above technical solution, step S5 specifically includes the following steps:

[0052] S51, Data popularity analysis and metadata tagging;

[0053] S52. Real-time monitoring and idle status determination of bandwidth;

[0054] S53, Perform pre-transmission of low-frequency data blocks;

[0055] S54, Multi-client transmission priority queue management;

[0056] S55, starvation detection mechanism;

[0057] S56, bandwidth allocation strategy;

[0058] S57, Manual priority intervention;

[0059] S51 is that during an incremental snapshot, the intelligent QoS management module of the cache server divides the data on the disk into multiple data blocks. When the source client performs read and write operations on the data blocks, the data receiving module of the cache server will notify the intelligent QoS management module to update the access counter of the corresponding data block. Based on the access frequency, each data block is labeled with a heat tag, and the QoS level is matched according to the heat tag.

[0060] S52, the intelligent QoS management module monitors the network bandwidth utilization of the narrow bandwidth transmission module in real time;

[0061] In S53, the pre-transmission task selects data blocks marked as cold data and whose bitmaps are marked as valid from the currently completed snapshots. The narrow bandwidth transmission module reads, compresses, and deduplicates these cold data blocks and then transmits them to the backup server. The backup server temporarily stores them as pre-transmission cache data.

[0062] S54 maintains a transmission queue for each client's backup task. The queue supports multiple priority levels and allows manual setting of service priorities.

[0063] S55 monitors the waiting time and number of transmission opportunities for each client task;

[0064] S56, in which high-priority tasks are given priority in acquiring bandwidth, supports bandwidth reservation;

[0065] The S57 allows users to manually increase the priority of a client task through the management interface.

[0066] According to the above technical solution, step S6 specifically includes the following steps:

[0067] S61. The remote backup server completes data reception and snapshot creation;

[0068] S62. Send a backup completion confirmation message to the cache server;

[0069] S63, The cache server performs resource cleanup;

[0070] In step S61, after receiving the full and incremental data transmitted by the cache server, the backup server generates a backup snapshot and records the snapshot metadata.

[0071] In step S62, after receiving the backup completion confirmation information, the intelligent QoS management module updates the backup status record and marks the current backup task as completed.

[0072] In step S63, the storage management module cleans up the snapshot data files, bitmap files, and metadata files on the cache server, frees up disk space, and releases the memory resources occupied by the storage management module and the intelligent QoS management module during the backup process.

[0073] A remote backup system suitable for narrow bandwidth environments includes a source client, a cache server, and a remote backup server;

[0074] The source client includes a block-level driver module, the cache server integrates a data receiving module, a storage management module, a narrow bandwidth transmission module, and an intelligent QoS management module, and the remote backup server includes a data storage and snapshot module.

[0075] According to the above technical solution, the block-level driver module, as the core component for source data acquisition and forwarding, is embedded in the underlying operating system. It intercepts and copies the write IO operations of the source disk in real time, without performing the overhead operations of COW snapshot and compression deduplication, and only undertakes the functions of data forwarding and command response.

[0076] The data receiving module receives backup data streams from the client, receives and parses instructions sent by the backup management terminal, and when data arrives, notifies the storage management module to store it and notifies the intelligent QoS management module to update the data block access count.

[0077] The storage management module is responsible for creating, maintaining, and updating sparse files, bitmap files, and metadata indexes; recording the status changes of each data block during incremental backup; and managing storage resources on the cache server.

[0078] The narrow bandwidth transmission module performs compression and deduplication calculations on the data to be transmitted in memory, and transmits the processed data to the remote backup server according to the QoS policy.

[0079] The intelligent QoS management module executes QoS policies to optimize bandwidth utilization and ensure fairness and priority in multi-client backup.

[0080] The data storage and snapshot module receives full and incremental data transmitted from the cache server, generates backup snapshots, records snapshot metadata, sends a backup completion confirmation message to the intelligent QoS management module of the cache server, and triggers resource cleanup.

[0081] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0082] 1. By deploying a dedicated cache server within the data center where the source system is located, high-overhead operations such as snapshot processing, data caching, compression, and deduplication, which were originally concentrated on the source client, are offloaded to the cache server. This significantly reduces the performance impact on the source system. At the same time, combined with innovative designs such as sparse file snapshot management, intelligent QoS transmission strategies, and data pre-transmission mechanisms, the backup efficiency and fairness in narrow bandwidth environments are significantly improved, and the scalability and stability of the system are enhanced. This effectively solves the core problems of high local resource consumption, poor network adaptability, and untimely backup in existing technologies. It is particularly suitable for scenarios involving cross-data center, low-bandwidth links, and concurrent backup of multiple clients, and has significant technological progress and practical value.

[0083] By employing a dual-file mechanism of sparse files and bitmaps, incremental data is accurately captured, avoiding waste of storage resources. Combined with a dual QoS strategy of snapshot pre-transmission and dynamic priority scheduling, the utilization rate of narrow bandwidth resources is maximized, balancing the fairness of multi-client backup with the priority of core business, and efficiently adapting to cross-data center backup scenarios with narrow bandwidth and high latency.

[0084] 2. By introducing a caching server and completely offloading resource-intensive operations such as copy-on-write, data caching, compression, and deduplication from the source client, the core contradiction between traditional backup solutions and production operations competing for computing, storage, and IO resources is fundamentally resolved. The source only needs to run a lightweight driver for data mirroring and forwarding, greatly relieving the pressure on its CPU, memory, and local storage. This effectively ensures the continuity and high-performance operation of core business applications and significantly reduces the impact on the source production system. By adopting an intelligent QoS transmission strategy combined with a snapshot pre-transmission mechanism, based on data access heat analysis, low-frequency static data blocks are transmitted in advance during bandwidth idle periods, and whether to repeat transmission is determined according to data changes. This maximizes the utilization of bandwidth resources and avoids bandwidth idleness and sudden congestion, thereby maximizing the utilization of narrow bandwidth resources.

[0085] By introducing a dynamic priority scheduling mechanism and starvation detection and automatic priority promotion functions, the system avoids the starvation phenomenon of some tasks occupying bandwidth for a long time and others lagging behind when multiple clients are backing up concurrently. This ensures the fairness of all backup tasks and is suitable for scenarios involving cross-data center, low-bandwidth links, and multiple clients with concurrent backups. It can effectively cope with backup needs of different scales and complexities, and has good scalability, allowing for flexible adjustment and expansion as business grows and data volume increases. Attached Figure Description

[0086] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0087] In the attached diagram:

[0088] Figure 1 This is a flowchart of the steps of the remote backup method of the present invention;

[0089] Figure 2 This is a block diagram of the remote backup system of the present invention;

[0090] Figure 3 This is a schematic diagram of the sparse file and bitmap mechanism of the present invention;

[0091] Figure 4 This is a schematic diagram of the snapshot pretransmission mechanism of the present invention;

[0092] Figure 5 This is a schematic diagram of the dynamic priority scheduling mechanism of the present invention. Detailed Implementation

[0093] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0094] Example: Figure 1 As shown, the present invention provides a technical solution, a remote backup method suitable for narrow bandwidth environments, comprising the following steps:

[0095] S1. Deploy cache servers and build a centralized relay architecture;

[0096] S2. Receive backup command, determine backup type. If it is incremental backup, enter incremental cache mode and execute S3. If it is first or full backup, enter full pass mode and execute S4.

[0097] S3. Perform incremental backups by co-capturing sparse files, bitmaps, and metadata.

[0098] S4. Perform a full backup in pass-through forwarding mode;

[0099] S5. QoS-based intelligent bandwidth utilization and transmission priority scheduling;

[0100] S6. Server-side backup complete, and cache server resources cleaned up.

[0101] Based on the above technical solution, S1, by constructing system hardware and network infrastructure, achieves centralized management and relay of backup traffic, specifically including the following steps:

[0102] S11. Cache server deployment and resource configuration;

[0103] S12. Initialize the built-in modules of the cache server;

[0104] S13. Establish the connection between the source client and the cache server;

[0105] S14. Establish a remote link between the cache server and the remote backup server;

[0106] S11: Deploy a physical cache server within the source data center. Configure its storage, CPU, and memory according to the number of backup clients and the amount of business data. Specific configuration requirements are as follows:

[0107] Sufficient storage: for temporary storage of snapshot data for incremental backups;

[0108] Multi-core CPU: Used to perform data compression and deduplication calculations;

[0109] Base memory: Used to ensure the efficient operation of various system services;

[0110] S12 includes a data receiving module, a storage management module, a narrowband transmission module, and an intelligent QoS management module;

[0111] The data receiving module configures the client communication port and data transmission protocol; the storage management module creates a sparse file storage pool and a bitmap file index library; the narrowband transmission module configures VPN / Internet channel parameters and optimizes the transmission protocol; and the intelligent QoS management module loads the priority rule library, initializes the bandwidth monitoring threshold, and initializes the data block access frequency statistics.

[0112] S13, the required backup source client installs a lightweight block-level driver module, the driver module copies all write operations of the source disk to the cache server in a mirror manner;

[0113] S14 configures a secure transmission protocol and a breakpoint resume mechanism to ensure reliable transmission from the cache server to the remote backup server in the network environment.

[0114] Based on the above technical solution, in step S2, the cache server receives the backup instruction, determines the backup type, and if it is an incremental backup, it enters the incremental caching mode and executes step S3; if it is the first or full backup, it enters the full pass-through mode and executes step S4. Specifically, this includes the following steps:

[0115] S21. Receive backup trigger command;

[0116] S22. Parse and determine the backup type;

[0117] S21, listen for backup commands sent by the management terminal through the data receiving module of the cache server;

[0118] S22. If it is an incremental backup, enter incremental caching mode, send a signal to the source client to prepare to start the snapshot, and jump to step S3. If it is a full backup, enter full pass-through mode, the source client prepares to read the full data, and jumps to step S4.

[0119] like Figure 3 As shown, based on the above technical solution, S3, through a three-layer collaborative mechanism of sparse files, bitmaps, and metadata hash tables, accurately captures the effective data of incremental backups, relieving the pressure of source snapshot generation and data storage, and providing accurate data positioning for subsequent QoS transmission. Specifically, it includes the following steps:

[0120] S31. Create and initialize the sparse file;

[0121] S32. Create and initialize bitmap files and metadata indexes;

[0122] S33. Write data and dynamically update the bitmap and metadata hash table;

[0123] S34. Receive the snapshot end command and complete the snapshot capture;

[0124] S35, Notification transmission service;

[0125] S31. Create a sparse file in the file system of the cache server with the same capacity as the disk of the source client. The file is stored on demand. In the initial state, the file content is empty and only occupies a small amount of disk space to record metadata. The corresponding physical storage space is occupied only when the client IO writes, so as to avoid pre-occupying the full disk resources.

[0126] The sparse file is used to record the location information of the incremental data written in this snapshot. When the source client has a data write operation, the data receiving module of the cache server will write the data to the corresponding position in the sparse file.

[0127] S32, create a bitmap file corresponding to the sparse file in the cache server. The size of the bitmap file is related to the number of data blocks in the sparse file. Each data block corresponds to a bit in the bitmap file. Initialize the value of each bit to 0. At the same time, create a hash table with the data logical offset address as the key to accurately record the actual physical offset address and data length written to the sparse file in this IO.

[0128] The bitmap file is used to record whether each data block has changed. A bit value of 0 indicates no change, and a bit value of 1 indicates a change. When the source client has a data write operation, in addition to writing the data to the sparse file, the cache server will update the bitmap file and set the status of the corresponding data block in the bitmap file to 1 according to the location where the data was written, indicating that the data block has changed.

[0129] The metadata management structure, namely the hash table, uses a hash-type data structure to record the location and length information of each I / O operation. The hash table uses the unique identifier of the data block as the key, specifically the logical block address, and the location and length information of the I / O operation corresponding to that data block as the value. During the initialization phase, the hash table is empty. As the source client I / O operation proceeds, the information in the hash table is continuously updated and improved. Through this hash structure, detailed information of each I / O operation can be recorded more accurately. In conjunction with the bitmap file, it provides a more comprehensive and accurate data location basis for subsequent data processing and transmission.

[0130] Bitmap files mark whether data blocks have changed, while hash metadata structures record the specific location and length of each changed data block in detail. The combination of the two can manage incremental backup data more efficiently.

[0131] S33, the source client performs a write operation, the block-level driver captures the I / O, and sends its content, logical offset address and data length to the cache server. The cache server writes the data to the corresponding offset position in the sparse file, sets the value of the corresponding bit in the bitmap file to 1, and inserts the physical offset address and data length corresponding to the logical offset address into the metadata hash table.

[0132] S34 is when the data receiving module detects the snapshot end signal, it stops writing data to the current sparse file, marks its status as read-only and unmodifiable, and uses it as the snapshot data file for this incremental backup. At the same time, it creates a new blank sparse file and metadata hash table, and guides the subsequent IO data generated by the client to write to the new sparse file. This realizes an isolation mechanism where one snapshot corresponds to one dedicated sparse file and metadata hash table, thus forming a snapshot chain arranged in chronological order.

[0133] S35 is when the cache server completes the snapshot instruction processing, sparse file, bitmap file, and metadata hash table maintenance operations, it notifies the narrowband transmission module that there is new incremental backup data to be transmitted.

[0134] Based on the above technical solution, S4 aims to avoid data being written to disk on the cache server, acting only as a relay node for real-time forwarding. This reduces unnecessary data storage and processing steps, improves backup efficiency, reduces cache server resource consumption, and ensures that data can be directly and quickly transmitted to the remote backup server. Specifically, it includes the following steps:

[0135] S41. Capture of the entire source data stream;

[0136] S42, Data stream memory transfer and processing;

[0137] S43. The compressed and deduplicated data stream is transmitted directly.

[0138] S41, the source client reads the data from the entire disk and sends it to the cache server in the form of a data stream;

[0139] S42, after the data receiving module of the cache server receives the data stream, it does not write it to the disk, but sends it directly to the processing pipeline. The narrow bandwidth transmission module performs compression and deduplication calculations on the data stream in memory.

[0140] S43, the processed data is directly forwarded to the remote backup server by the narrow bandwidth transmission module.

[0141] Based on the above technical solutions, S5 is designed for narrow bandwidth characteristics. It maximizes the use of bandwidth resources and avoids bandwidth idleness and congestion through a dual strategy of snapshot pretransmission and dynamic priority scheduling, while ensuring the fairness of multi-client backup and the backup priority of core services.

[0142] Snapshot pretransmission is a predictive data transmission mechanism based on data access heat analysis. In a narrow bandwidth environment, the intelligent QoS management module uses the bandwidth idle period to select data blocks with low access frequency, stable data status and valid bitmap markings from the currently completed incremental snapshots. Stable data status refers to low-frequency static blocks. When network bandwidth is detected to be idle, these blocks are compressed and deduplicated before being transmitted to the remote backup server for temporary storage. If the data block does not change in subsequent snapshots, it is considered to have been backed up and does not need to be transmitted again. If it changes, the new version will overwrite the pretransmitted data.

[0143] Dynamic priority scheduling is a fairness and priority guarantee mechanism for multi-client concurrent backup scenarios. This mechanism maintains an independent transmission queue for each client's backup task and dynamically adjusts the scheduling order of each task according to the preset business priority and real-time transmission status. The system continuously monitors the waiting time and bandwidth acquisition of each task. When it detects that some tasks have not obtained transmission opportunities for a long time, i.e., the risk of starvation, it automatically raises their priority. It also supports manual intervention to ensure that critical business data is backed up first.

[0144] like Figure 4-5 As shown, based on the above technical solution, S5 specifically includes the following steps:

[0145] S51, Data popularity analysis and metadata tagging;

[0146] S52. Real-time monitoring and idle status determination of bandwidth;

[0147] S53, Perform pre-transmission of low-frequency data blocks;

[0148] S54, Multi-client transmission priority queue management;

[0149] S55, starvation detection mechanism;

[0150] S56, bandwidth allocation strategy;

[0151] S57, Manual priority intervention;

[0152] S51 involves the cache server's intelligent QoS management module dividing the data on the disk into multiple data blocks during an incremental snapshot. The size of the data blocks can be set according to actual conditions. An access counter is maintained for each data block to record the access frequency of the data block during the snapshot period. When the source client performs read and write operations on the data block, the cache server's data receiving module will notify the intelligent QoS management module to update the access counter of the corresponding data block. Based on the access frequency, each data block is labeled with a heat tag, and the QoS level is matched according to the heat tag. Specifically, there are high-frequency change blocks, mid-frequency stable blocks, and low-frequency static blocks. Among them, high-frequency change blocks are not pre-transmitted, mid-frequency stable blocks can be selectively pre-transmitted, and low-frequency static blocks can be pre-transmitted.

[0153] S52, the intelligent QoS management module monitors the network bandwidth utilization of the narrow bandwidth transmission module in real time. When it detects that the current bandwidth utilization is lower than the set threshold and there is no high-priority task being transmitted, it triggers a pre-transmission task.

[0154] S53, the pre-transmission task selects data blocks marked as cold data and whose bitmaps are marked as valid from the currently completed snapshots. The narrow bandwidth transmission module reads, compresses, and deduplicates these cold data blocks before transmitting them to the backup server. The backup server temporarily stores them as pre-transmission cache data. In the next snapshot cycle, if these pre-transmission data blocks change again, they are recaptured and transmitted according to the normal process. If they have not changed, they are directly referenced when the snapshot is officially transmitted.

[0155] S54 maintains a transmission queue for each client's backup task. The queue supports multiple priority levels and allows manual setting of service priorities.

[0156] S55 monitors the waiting time and number of transmission opportunities for each client task. If a task fails to obtain bandwidth resources for N consecutive scheduling cycles, it is determined to be potentially starved and its priority is automatically increased by one level until it obtains a transmission opportunity.

[0157] S56 prioritizes high-priority tasks by acquiring bandwidth, supports bandwidth reservation, and ensures the minimum transmission rate for critical services.

[0158] S57 allows users to manually increase the priority of a client task through the management interface and supports an emergency backup task queue-jumping mechanism.

[0159] Based on the above technical solution, S6 completes the final storage of data on the remote backup server, ensuring data security and integrity, while releasing resources occupied on the cache server to prepare for subsequent backup tasks. Specifically, it includes the following steps:

[0160] S61. The remote backup server completes data reception and snapshot creation;

[0161] S62. Send a backup completion confirmation message to the cache server;

[0162] S63, The cache server performs resource cleanup;

[0163] S61, after receiving the full and incremental data transmitted by the cache server, the backup server generates a backup snapshot and records the snapshot metadata;

[0164] S62, after receiving the backup completion confirmation information, the intelligent QoS management module updates the backup status record and marks the current backup task as completed;

[0165] S63, the storage management module cleans up snapshot data files, bitmap files and metadata files on the cache server, freeing up disk space and releasing the memory resources occupied by the storage management module and the intelligent QoS management module during the backup process.

[0166] like Figure 2 As shown, a remote backup system suitable for narrow bandwidth environments includes a source client, a cache server, and a remote backup server.

[0167] The source client includes a block-level driver module, the cache server integrates a data receiving module, a storage management module, a narrow bandwidth transmission module, and an intelligent QoS management module, and the remote backup server includes a data storage and snapshot module.

[0168] Based on the above technical solution, the block-level driver module, as the core component for source data acquisition and forwarding, is embedded in the underlying operating system. It intercepts and copies the write I / O operations of the source disk in real time, without performing the overhead operations of COW snapshot and compression deduplication. It only undertakes the functions of data forwarding and command response, thereby reducing the consumption of source resources.

[0169] The data receiving module receives backup data streams from the client, receives and parses instructions sent by the backup management terminal, and notifies the storage management module to store the data when it arrives, and notifies the intelligent QoS management module to update the data block access count.

[0170] The storage management module is responsible for creating, maintaining, and updating sparse files, bitmap files, and metadata indexes; recording the status changes of each data block during incremental backups; managing storage resources on the cache server; and ensuring effective data storage and timely cleanup.

[0171] The narrow bandwidth transmission module performs compression and deduplication calculations on the data to be transmitted in memory, and then transmits the processed data to the remote backup server according to the QoS policy.

[0172] The intelligent QoS management module executes QoS policies to optimize bandwidth utilization and ensure fairness and priority for multi-client backup.

[0173] The data storage and snapshot module receives full and incremental data, including pre-transmission data, from the cache server, generates backup snapshots, records snapshot metadata, sends a backup completion confirmation message to the cache server's intelligent QoS management module, and triggers resource cleanup.

[0174] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote backup method suitable for narrow bandwidth environments, characterized in that: Includes the following steps: S1. Deploy cache servers and build a centralized relay architecture; S2. Receive backup command, determine backup type. If it is incremental backup, enter incremental cache mode and execute S3. If it is first or full backup, enter full pass mode and execute S4. S3. Perform incremental backups by co-capturing sparse files, bitmaps, and metadata. S4. Perform a full backup in pass-through forwarding mode; S5. QoS-based intelligent bandwidth utilization and transmission priority scheduling; S6. Server-side backup complete, and cache server resources cleaned up.

2. The remote backup method suitable for narrow bandwidth environments according to claim 1, characterized in that: S1 specifically includes the following steps: S11. Cache server deployment and resource configuration; S12. Initialize the built-in modules of the cache server; S13. Establish the connection between the source client and the cache server; S14. Establish a remote link between the cache server and the remote backup server; In step S11, a physical or virtual cache server is deployed inside the source data center, and its storage, CPU, and memory are configured according to the number of backup clients and the amount of business data. S12 includes a data receiving module, a storage management module, a narrowband transmission module, and an intelligent QoS management module; In step S13, a lightweight block-level driver module needs to be installed on the backup source client. The driver module copies all write operations of the source disk to the cache server in a mirror manner. In step S14, a secure transmission protocol and a breakpoint resume mechanism are configured.

3. The remote backup method suitable for narrow bandwidth environments according to claim 1, characterized in that: S2 specifically includes the following steps: S21. Receive backup trigger command; S22. Parse and determine the backup type; In step S21, the data receiving module of the cache server listens for backup instructions sent by the management terminal. In step S22, if it is an incremental backup, the system enters the incremental caching mode, sends a signal to the source client to prepare to start the snapshot, and jumps to step S3. If it is a full backup, the system enters the full pass-through mode, the source client prepares to read the full data, and jumps to step S4.

4. The remote backup method suitable for narrow bandwidth environments according to claim 3, characterized in that: S3 specifically includes the following steps: S31. Create and initialize the sparse file; S32. Create and initialize bitmap files and metadata indexes; S33. Write data and dynamically update the bitmap and metadata hash table; S34. Receive the snapshot end command and complete the snapshot capture; S35, Notification transmission service; In step S31, a sparse file with the same size as the source client's disk is created in the file system of the cache server; The sparse file is used to record the location information of the incremental data written in this snapshot. When the source client has a data write operation, the data receiving module of the cache server will write the data to the corresponding position in the sparse file. In step S32, a bitmap file corresponding to the sparse file is created in the cache server. The size of the bitmap file is related to the number of data blocks divided in the sparse file. At the same time, a hash table with the data logical offset address as the key is created. Bitmap files are used to record whether each data block has changed. A bit value of 0 indicates no change, and a bit value of 1 indicates a change. The metadata management structure uses a hash-type data structure to record the location and length information of each IO operation; In S33, the source client performs a write operation, the block-level driver captures the I / O, and sends its content, logical offset address and data length to the cache server. The cache server writes the data to the corresponding offset position in the sparse file, sets the value of the corresponding bit in the bitmap file to 1, and inserts or updates the physical offset address and data length corresponding to the logical offset address in the metadata hash table. S34 is that when the data receiving module detects the snapshot end signal, it stops writing data to the current sparse file, marks its status as read-only, uses it as the snapshot data file for this incremental backup, and simultaneously creates a new blank sparse file and metadata hash table, and guides the subsequent IO data generated by the client to write to the new sparse file. S35 is when the cache server completes the snapshot instruction processing, sparse file, bitmap file and metadata hash table maintenance operations, it notifies the narrowband transmission module that there is new incremental backup data to be transmitted.

5. A remote backup method suitable for narrow bandwidth environments according to claim 3, characterized in that: S4 specifically includes the following steps: S41. Capture of the entire source data stream; S42, Data stream memory transfer and processing; S43. The compressed and deduplicated data stream is transmitted directly. In step S41, the source client reads the data from the entire disk and sends it to the cache server in the form of a data stream. In step S42, after the data receiving module of the cache server receives the data stream, it does not write it to the disk, but sends it directly into the processing pipeline. The narrow bandwidth transmission module performs compression and deduplication calculations on the data stream in memory. In step S43, the processed data is directly forwarded to the remote backup server by the narrow bandwidth transmission module.

6. The remote backup method suitable for narrow bandwidth environments according to claim 1, characterized in that: The S5 is designed to address narrow bandwidth characteristics by employing a dual strategy of snapshot pre-transmission and dynamic priority scheduling to maximize bandwidth resource utilization while ensuring fairness in multi-client backups and prioritizing the backup of core services.

7. A remote backup method suitable for narrow bandwidth environments according to claim 6, characterized in that: S5 specifically includes the following steps: S51, Data popularity analysis and metadata tagging; S52. Real-time monitoring and idle status determination of bandwidth; S53, Perform pre-transmission of low-frequency data blocks; S54, Multi-client transmission priority queue management; S55, starvation detection mechanism; S56, bandwidth allocation strategy; S57, Manual priority intervention; S51 is that during an incremental snapshot, the intelligent QoS management module of the cache server divides the data on the disk into multiple data blocks. When the source client performs read and write operations on the data blocks, the data receiving module of the cache server will notify the intelligent QoS management module to update the access counter of the corresponding data block. Based on the access frequency, each data block is labeled with a heat tag, and the QoS level is matched according to the heat tag. S52, the intelligent QoS management module monitors the network bandwidth utilization of the narrow bandwidth transmission module in real time; In S53, the pre-transmission task selects data blocks marked as cold data and whose bitmaps are marked as valid from the currently completed snapshots. The narrow bandwidth transmission module reads, compresses, and deduplicates these cold data blocks and then transmits them to the backup server. The backup server temporarily stores them as pre-transmission cache data. S54 maintains a transmission queue for each client's backup task. The queue supports multiple priority levels and allows manual setting of service priorities. S55 monitors the waiting time and number of transmission opportunities for each client task; S56, in which high-priority tasks are given priority in acquiring bandwidth, supports bandwidth reservation; The S57 allows users to manually increase the priority of a client task through the management interface.

8. The remote backup method suitable for narrow bandwidth environments according to claim 1, characterized in that: S6 specifically includes the following steps: S61. The remote backup server completes data reception and snapshot creation; S62. Send a backup completion confirmation message to the cache server; S63, The cache server performs resource cleanup; In step S61, after receiving the full and incremental data transmitted by the cache server, the backup server generates a backup snapshot and records the snapshot metadata. In step S62, after receiving the backup completion confirmation information, the intelligent QoS management module updates the backup status record and marks the current backup task as completed. In step S63, the storage management module cleans up the snapshot data files, bitmap files, and metadata files on the cache server, frees up disk space, and releases the memory resources occupied by the storage management module and the intelligent QoS management module during the backup process.

9. A remote backup system suitable for narrow bandwidth environments, characterized in that: Includes source client, cache server and remote backup server; The source client includes a block-level driver module, the cache server integrates a data receiving module, a storage management module, a narrow bandwidth transmission module, and an intelligent QoS management module, and the remote backup server includes a data storage and snapshot module.

10. A remote backup system suitable for narrow bandwidth environments according to claim 9, characterized in that: The block-level driver module, as the core component for source data acquisition and forwarding, is embedded in the underlying operating system. It intercepts and copies the write I / O operations of the source disk in real time, without performing overhead operations such as COW snapshots and compression deduplication. It only undertakes the functions of data forwarding and command response. The data receiving module receives backup data streams from the client, receives and parses instructions sent by the backup management terminal, and when data arrives, notifies the storage management module to store it and notifies the intelligent QoS management module to update the data block access count. The storage management module is responsible for creating, maintaining, and updating sparse files, bitmap files, and metadata indexes; recording the status changes of each data block during incremental backup; and managing storage resources on the cache server. The narrow bandwidth transmission module performs compression and deduplication calculations on the data to be transmitted in memory, and transmits the processed data to the remote backup server according to the QoS policy. The intelligent QoS management module executes QoS policies to optimize bandwidth utilization and ensure fairness and priority in multi-client backup. The data storage and snapshot module receives full and incremental data transmitted from the cache server, generates backup snapshots, records snapshot metadata, sends a backup completion confirmation message to the intelligent QoS management module of the cache server, and triggers resource cleanup.

Images