Permanent incremental backup method based on S3 bucket-level virtual snapshot

Abstract

The invention discloses a permanent incremental backup method based on an S3 bucket-level virtual snapshot, and relates to the technical field of data storage and backup, the method comprises the following steps: initializing a backup system and starting a backup agent program, an index management service and an S3 object storage service; complete backup is executed, and a VID and S3 bucket-level virtual snapshoot are generated; efficiently managing a mapping relation between a file path and a snapshot identifier through a B + tree and an inverted index by utilizing a MongoDB index service; incremental backup is executed, and file changes are accurately recognized through Hash comparison and modification time comparison; a storage position is optimized by adopting an incremental storage and deduplication storage mechanism and an IncDSI optimization algorithm; executing backup chain integration and cleaning; and performing data recovery based on the VID of the file and the directory. According to the method, the backup efficiency is remarkably improved, the storage overhead is reduced, the data recovery process is optimized, and the method is suitable for data backup and management of a large-scale file system and has important application value and prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and backup, and particularly to a permanent incremental backup method based on S3 bucket-level virtual snapshots. Background Art

[0002] With the rapid development of information technology, the amount of data has shown explosive growth in various industries. As the core infrastructure for data storage and management, file systems carry a large amount of unstructured data, such as documents, pictures, videos, etc. These data have become the basic information of modern society, driving the continuous development of storage technology. However, with the sharp increase in data scale, traditional backup methods face many challenges in dealing with large-scale data. Especially in file systems, how to perform incremental backups efficiently and accurately has become the bottleneck of current backup technology.

[0003] In existing backup technologies, with the sharp increase in data volume, traditional incremental backup methods are difficult to meet the backup requirements of current file systems. Especially for the backup of a large number of unstructured small files, traditional incremental backups are often inefficient and difficult to ensure data integrity and recovery speed. In addition, traditional backup systems usually lack effective storage optimization mechanisms, easily resulting in storage redundancy. As the backup chain continues to expand, the storage space utilization rate is low, and the data recovery process becomes complex and time-consuming.

[0004] To solve this problem, the industry has begun to explore new backup models. Among them, permanent incremental backup has attracted much attention due to its high efficiency and storage space savings. Permanent incremental backup continuously tracks changes in the file system and only backs up newly added or modified data, thus avoiding duplicate backups and unnecessary storage overhead. However, it is not easy to implement permanent incremental backup for complex and diverse file systems. It requires a new backup method and system that can flexibly adapt to the characteristics of different file systems, efficiently track file changes, and ensure the long-term accessibility and recovery efficiency of backup data. Summary of the Invention

[0005] The purpose of the present invention is to provide a permanent incremental backup method based on S3 bucket-level virtual snapshots, which solves the limitations of existing incremental backup technologies such as storage redundancy, low recovery efficiency, and complex backup chain management. By introducing S3 bucket-level virtual snapshots and the IncDSI optimization algorithm, the storage efficiency during the backup process is significantly improved, data redundancy is reduced, and at the same time, the speed and accuracy of data recovery are greatly enhanced, ensuring the efficient management and rapid recovery of backup data in large-scale file systems.

[0006] To achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A permanent incremental backup method based on S3 bucket-level virtual snapshots, comprising the following steps:

[0008] Step S1: Initialize the backup system, start the backup agent program, the index management service, and the S3 object storage, and enable the multi-version control mechanism of S3 to manage S3 bucket-level virtual snapshots;

[0009] Step S2: Perform a full backup. The backup agent program concurrently reads the file system data using a multi-channel streaming API, uploads the data to the S3 storage bucket, generates a unique version identifier VID, and simultaneously creates an S3 bucket-level virtual snapshot at time T1;

[0010] Step S3: Record the file path, snapshot identifier SnapshotID, and version identifier VID in the MongoDB index service. Establish a mapping relationship between the file path and the snapshot ID using a B+ tree index, and manage the file version history using an inverted index mechanism to achieve the index management of S3 bucket-level virtual snapshots;

[0011] Step S4: Perform an incremental backup. The backup agent program identifies file changes through hash comparison and modification time comparison, stores the changed files, generates a new version identifier VID, and updates the S3 bucket-level virtual snapshot;

[0012] Step S5: Store the file data to be backed up and its metadata. The storage uses an incremental storage and deduplication storage mechanism, and uses the IncDSI optimization algorithm to adjust the index position to avoid storage conflicts and ensure the data consistency of the S3 bucket-level virtual snapshot;

[0013] Step S6: When the incremental backup chain exceeds a preset threshold, perform backup chain integration. Use transaction logs and timestamp partition storage for data merging, and use the LRU algorithm to clean up S3 bucket-level virtual snapshots that have not been accessed for a long time;

[0014] Step S7: Perform data recovery based on the directory and file VID. Retrieve the snapshot data using the snapshot mapping relationship, and use the data block reorganization technology to extract the required data blocks to recover the data corresponding to the S3 bucket-level virtual snapshot.

[0015] As a preferred solution of the present invention, in step S2, the backup agent program identifies the files to be backed up through hash comparison and modification time comparison, and generates the version identifier VID of the files, where:

[0016] Hash comparison: Calculate the hash value of the file content and compare it with the hash value stored in the MongoDB index service. If they match, only update the index relationship. If they do not match, perform data storage and update the index information;

[0017] Modification time comparison: By comparing the latest modification times of files, mark the files to be backed up to ensure the accuracy of incremental backups.

[0018] As a preferred embodiment of the present invention, in step S3, the MongoDB index service uses a B+ tree structure and an inverted index mechanism for data management, where:

[0019] The B+ tree index structure is used to store the mapping relationship between file paths, snapshot identifiers SnapshotID, and version identifiers VID to achieve efficient retrieval;

[0020] The inverted index mechanism improves the efficiency of querying file versions by snapshot identifiers by maintaining the mapping relationship from file versions to snapshot IDs;

[0021] The incremental index update algorithm performs old index merging, data deduplication, and index optimization during index data updates, reducing redundant storage and improving query performance, and optimizing storage and query efficiency during the incremental backup process.

[0022] As a preferred embodiment of the present invention, the incremental index update algorithm includes:

[0023] Step S3.1: Calculate the index position of backup data based on the linear regression model of file feature vectors, predict the optimal storage position of incremental backup data in the index, and reduce index redundancy;

[0024] Step S3.2: Calculate the predicted storage position through the linear regression model, and the formula is as follows:

[0025] M(k) = |A × G(k) + b|;

[0026] Where: M(k) is the predicted position of backup data, indicating the storage position of incremental backup data in the index; A is the parameter matrix of the linear regression model, indicating the weighted influence of different data features on the index position; G(k) is the feature vector of backup data, including file size, modification time, and access frequency information; b is the bias term used to adjust the predicted position;

[0027] Step S3.3: Based on the dynamic index adjustment method of the optimization objective function, when the deviation between the predicted position and the actual storage position exceeds the preset threshold, perform dynamic index adjustment to optimize the storage method of incremental backup data, and the formula is:

[0028] L(v n+1 ) = λ1l1(v n+1 ) + (1 - λ1)l2(v n+1 ) + λ2||v n+1 || 2 ;

[0029] Where: L(vn+1 ) is the feature vector of the newly added backup data, representing the storage characteristics of the incremental backup data; λ1 and λ2 are adjustment parameters, which are respectively used to control the balance between the accuracy of index storage and the update efficiency; l1(v n+1 ) is used to optimize the accuracy of the index position; l2(v n+1 ) is used to optimize the storage efficiency of index update; ||v n+1 || 2 represents the data norm, which controls the stability of the data during the index optimization process;

[0030] Step S3.4: Based on the conflict detection and index rearrangement method of similarity calculation, when a conflict occurs in the predicted storage location of the incremental backup data, the IncDSI optimization algorithm is used to re-adjust the index position to ensure the uniqueness of the incremental backup data. The formula is:

[0031]

[0032] Where: is the query vector of the newly added incremental backup data, representing the query characteristics of the current data; is the query vector of the existing incremental backup data; v j is the feature vector of the existing incremental backup data;

[0033] Step S3.5: Adopt the B+ tree index and inverted index optimization methods to improve the data query efficiency based on the snapshot ID, and ensure that the backup data can be quickly retrieved and restored.

[0034] As a preferred solution of the present invention, in step S6, the backup chain integration process adopts batch transaction processing to perform data optimization in the storage system, where:

[0035] The transaction log mechanism records the data changes of the backup chain to ensure data consistency;

[0036] The backup chain merge operation migrates the latest version data to the new backup chain set to reduce storage fragmentation;

[0037] The timestamp partition storage stores the data according to the snapshot generation time, and cleans the snapshot data based on the retention period and retention number of the backup policy.

[0038] As a preferred solution of the present invention, the S3 bucket-level virtual snapshot supports accurate data backtracking based on the snapshot identifier, where:

[0039] Restore the file version based on the snapshot time point, retrieve the snapshot mapping relationship through the MongoDB index service, and restore the data at the specified time point;

[0040] The data block reorganization extracts the required data blocks, constructs a complete file, and performs data consistency verification.

[0041] As a preferred solution of the present invention, the backup agent program dynamically adjusts the backup frequency according to the free space of the storage system, the length of the backup chain, and the number of snapshots, optimizes the storage resource configuration, improves the backup efficiency, and reduces the storage overhead.

[0042] As a preferred solution of the present invention, the management of the incremental backup adopts a time-sequential management method:

[0043] Each incremental backup T1, T2, Tn, T(N+1) generates a complete backup snapshot, and integrates the incremental data with the backup snapshot of the previous moment to form a new snapshot; among them, the data of each generated complete backup snapshot supports fast mounting to achieve fast recovery;

[0044] Whenever the backup chain reaches a preset threshold, the backup chain integration is executed, the old backup is merged with the current incremental snapshot, and the index information is migrated to the new backup chain set to ensure the continuity of the incremental backup chain and the efficient management of data.

[0045] As a preferred solution of the present invention, the backup system includes:

[0046] A network interface for data transmission between the backup agent program and the S3 storage service. The network interface uses the S3 RESTful API and HTTP / HTTPS protocols for remote communication and supports multi-channel streaming data transmission;

[0047] A memory for storing file system data and incremental backup data. The memory includes a local cache storage area and a remote S3 storage bucket. The local cache storage area is used to temporarily store incremental backup data and perform data deduplication, compression, and sharding. The remote S3 storage bucket adopts a multi-version storage strategy and combines a timestamp partition storage strategy to manage snapshot data;

[0048] A processor for executing the calculation tasks of the backup agent program. The processor adopts an asynchronous parallel processing architecture and executes index management, data transmission scheduling, and incremental backup task scheduling. Among them, the index management optimizes the storage of snapshot metadata through B+ tree index and inverted index. The data transmission scheduling adjusts the data backup strategy according to the data modification frequency, access popularity, and storage status. The incremental backup task scheduling combines the IncDSI optimization algorithm to adjust the storage location of incremental data.

[0049] As a preferred embodiment of the present invention, the network interface, the memory, and the processor are interconnected via a data bus to ensure smooth execution of backup operations and data consistency. The processor adjusts the backup frequency and storage resource configuration according to the data status in the memory and the transmission status of the network interface to optimize the backup efficiency.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows: By introducing multi-version control of S3 and index management of MongoDB, the present invention effectively solves problems such as storage redundancy, low backup efficiency, and slow data recovery in traditional backup technologies. The multi-version control mechanism of S3 object storage and the management of bucket-level virtual snapshots are adopted to ensure that each backup generates a unique version identifier (VID) and forms a complete backup snapshot, improving the reliability of backup data and the convenience of management. The backup agent program concurrently reads file system data through a multi-channel streaming API and uploads it to the S3 storage bucket, reducing the backup time and improving the efficiency of large-scale data backup. Precise identification of file changes is achieved through the hash comparison and modification time comparison mechanisms, avoiding unnecessary duplicate data backups and improving the utilization rate of storage space. The B+ tree index and inverted index mechanism of MongoDB are used to efficiently manage backup data, ensuring a clear mapping relationship between file paths and snapshot identifiers, facilitating data recovery. The combination of incremental storage and deduplication storage mechanisms reduces redundant storage, and the index position is adjusted through the IncDSI optimization algorithm to avoid storage conflicts, greatly improving the utilization rate of storage space. When the length of the incremental backup chain exceeds a preset threshold, data merging is performed using transaction logs and timestamp partitioning storage, and snapshot data is cleared based on the retention period and retention count of the backup policy, further optimizing the storage structure. Data recovery based on the snapshot ID quickly locates the data through the snapshot mapping relationship and extracts the required data blocks using the data block reorganization technology, achieving accurate and fast data recovery. The present invention significantly improves the efficiency and accuracy of the backup process, optimizes the performance of the storage and recovery processes, and enhances the management and recovery efficiency of large-scale file system backup data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts.

[0052] Among them:

[0053] Figure 1 is the method flow chart of the present invention;

[0054] Figure 2 is the architecture schematic diagram of the embodiment of the present invention. Detailed implementation manners

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention fall within the scope of protection of the present invention.

[0056] As Figure 1 shown, this is an embodiment of the present invention, and this embodiment provides a permanent incremental backup method based on S3 bucket-level virtual snapshots, including the following steps:

[0057] Step S1: Initialize the backup system

[0058] Start the backup agent program, the index management service, and the S3 object storage, and enable the multi-version control mechanism of S3 to manage S3 bucket-level virtual snapshots;

[0059] In this embodiment, the backup system is first initialized. The backup system consists of the following modules:

[0060] Backup agent program: Responsible for executing data backup tasks, reading data from the file system, and uploading it to the S3 storage bucket.

[0061] MongoDB index service: Used to manage the index information of backup data, including file paths, version identifiers (VIDs), snapshot identifiers (SnapshotIDs), etc.

[0062] S3 object storage service: Used to store backup data and support the multi-version control mechanism to ensure that different versions of backup data can be managed.

[0063] After starting the backup agent program, the program connects to the S3 storage bucket through the configured S3 RESTful API and establishes a connection with the MongoDB index service. At this time, the system will start the multi-version control mechanism (Versioning) and ensure that a unique identifier can be generated for each file version. This stage is the system initialization stage, preparing for subsequent backup operations.

[0064] Step S2: Perform a full backup

[0065] The backup agent program uses the multi-channel streaming API to concurrently read data from the file system, upload the data to the S3 storage bucket, generate a unique version identifier VID, and create an S3 bucket-level virtual snapshot at time T1;

[0066] Specifically, the backup agent identifies files to be backed up through hash comparison and modification time comparison, and generates a version identifier (VID) for the files, where:

[0067] Hash comparison: By calculating the hash value of the file content and comparing it with the hash value stored in the MongoDB index service, if they match, only the index relationship is updated; if they do not match, data storage is performed and index information is updated.

[0068] Modification time comparison: By comparing the latest modification time of the file, the files to be backed up are marked to ensure the accuracy of incremental backup.

[0069] During the upload process in this embodiment, the backup agent also temporarily saves the file version identifier (VID) returned by the S3 interface. This identifier is used to ensure that the backed-up files can be associated with the corresponding S3 storage versions.

[0070] Step S3: Create index records

[0071] In the MongoDB index service, record the file path, snapshot identifier (SnapshotID), and version identifier (VID), establish a mapping relationship between the file path and the snapshot ID using a B+ tree index, and manage the file version history using an inverted index mechanism to achieve the index management of S3 bucket-level virtual snapshots.

[0072] Specifically, the MongoDB index service uses a B+ tree structure and an inverted index mechanism for data management, where:

[0073] The B+ tree index structure is used to store the mapping relationship between the file path, snapshot identifier (SnapshotID), and version identifier (VID) to achieve efficient retrieval.

[0074] The inverted index mechanism improves the efficiency of querying the file version by the snapshot ID by maintaining the mapping relationship from the file version to the snapshot ID.

[0075] The incremental index update algorithm performs old index merging, data deduplication, and index optimization when the index data is updated, reduces redundant storage, and improves query performance, optimizing the storage and query efficiency during the incremental backup process.

[0076] In this embodiment, the incremental index update algorithm includes:

[0077] Step S3.1: Calculate the index position of the backup data based on the linear regression model of the file feature vector, predict the optimal storage position of the incremental backup data in the index, and reduce index redundancy.

[0078] Step S3.2: Calculate the predicted storage position through the linear regression model, and the formula is as follows:

[0079] M(k) = |A × G(k) + b|;

[0080] Where: M(k) is the predicted position of the backup data, representing the storage position of the incremental backup data in the index; A is the parameter matrix of the linear regression model, representing the weighted influence of different data features on the index position; G(k) is the feature vector of the backup data, including file size, modification time, and access frequency information; b is the bias term, used to adjust the predicted position.

[0081] Step S3.3: Based on the dynamic index adjustment method of the optimization objective function, when the deviation between the predicted position and the actual storage position exceeds the preset threshold, perform dynamic index adjustment to optimize the storage method of the incremental backup data. The formula is:

[0082] L(v n+1 ) = λ1l1(v n+1 ) + (1 - λ1)l2(v n+1 ) + λ2||v n+1 || 2 ;

[0083] Where: L(v n+1 ) is the feature vector of the newly added backup data, representing the storage characteristics of the incremental backup data; λ1 and λ2 are adjustment parameters, used to control the balance between the accuracy of index storage and the update efficiency respectively; l1(v n+1 ) is used to optimize the accuracy of the index position; l2(v n+1 ) is used to optimize the storage efficiency of index update; ||v n+1 || 2 is the data norm, used to control the stability of data during the index optimization process.

[0084] Step S3.4: Based on the conflict detection and index rearrangement method of similarity calculation, when a conflict occurs in the predicted storage position of the incremental backup data, use the IncDSI optimization algorithm to re-adjust the index position to ensure the uniqueness of the incremental backup data. The formula is:

[0085]

[0086] Where: is the query vector of the newly added incremental backup data, representing the query features of the current data; is the query vector of the existing incremental backup data; v j is the feature vector of the existing incremental backup data.

[0087] The IncDSI optimization algorithm in this embodiment optimizes the storage location allocation during the incremental backup process, improves the storage efficiency, reduces redundant storage, and enhances the data recovery speed. This algorithm intelligently adjusts the storage location by combining file data characteristics, linear regression prediction, and data similarity calculation to ensure the efficiency and data consistency of incremental backup.

[0088] Technical principle of the algorithm: Feature vector calculation: First, feature extraction is performed on the data to be backed up. The features of each file or data block include, but are not limited to, file size, modification time, modification frequency, and access pattern, etc. These feature data constitute the feature vector, which is the basis for predicting the storage location. The establishment of the feature vector can accurately reflect the storage requirements and priorities of the file or data block. Linear regression model for predicting the storage location: Based on the feature vector, the IncDSI optimization algorithm uses a linear regression model to predict the optimal storage location of the incremental data. This regression model automatically adjusts the storage strategy according to the storage pattern of historical data and predicts the best location of each data block in the backup chain. The parameters of the regression model have been iteratively optimized multiple times to ensure the accurate prediction of the storage location. Similarity calculation and conflict detection: The algorithm further calculates the similarity between data blocks. If it is found that the data blocks have a high similarity, they are stored in adjacent locations to avoid storage conflicts and effectively utilize the storage space. This process combines the deduplication storage mechanism to avoid duplicate storage and improve the space utilization rate. Optimization and adjustment of the storage location: Dynamic storage location adjustment: According to the prediction results of the linear regression model, the algorithm intelligently allocates the most suitable storage location for the incremental backup data. The adjusted storage location can effectively avoid conflicts between data blocks and optimize the use of the storage space. Avoidance of redundant storage: The similarity calculation and data deduplication mechanism effectively reduce redundant storage, making the incremental backup data storage more compact and reducing unnecessary data backups. For data that has not changed, the system avoids duplicate storage, thereby further optimizing the storage space. Optimization of incremental data storage: For files or data blocks with different characteristics, the system can adjust the storage location of the incremental data according to the prediction results. For important and frequently modified data, the algorithm stores it in a high-priority location to ensure the access efficiency of the data; for data with fewer changes, the system stores it in a relatively low-priority location to reduce the occupation of the system's storage resources.

[0089] Performance optimization and technical advantages:

[0090] Improve storage efficiency: By intelligently adjusting the storage location and reducing redundant data, the IncDSI optimization algorithm can significantly improve the utilization rate of backup storage space. Improve backup and recovery speed: By optimizing the storage and index location of incremental backup data, the system can more quickly locate and recover the required data blocks, especially suitable for large-scale file systems. Reduce backup time and storage overhead: The algorithm reduces the time and storage space required for backup through accurate prediction and storage location optimization, especially when dealing with a large amount of data, the effect is particularly obvious. Flexibly respond to different backup scenarios: The present invention provides a highly flexible storage optimization mechanism that can dynamically adjust the storage strategy according to the characteristics of backup data to ensure the high efficiency of the system.

[0091] Comparison with the prior art:

[0092] Existing incremental backup technologies mostly rely on the modification time of files or hash comparison to determine whether a file has changed, resulting in problems such as large computational overhead and wasted storage space. However, the present invention introduces the IncDSI optimization algorithm, combines the calculation of file feature vectors and similarity analysis, optimizes the storage location and index update during the incremental backup process, reduces redundant storage, and improves the backup efficiency. Compared with traditional methods, this algorithm not only improves the utilization rate of storage space, but also significantly optimizes the efficiency of the backup and recovery process, especially in the backup and recovery of large-scale file systems

[0093] Step S3.5: Adopt the B+ tree index and inverted index optimization methods to improve the data query efficiency based on the snapshot ID, and ensure that the backup data can be quickly retrieved and recovered.

[0094] In this example, through the efficient indexing service of MongoDB, the present invention can achieve precise management of each backup file, ensuring that each backup version can be accurately retrieved and recovered.

[0095] Step S4: Perform incremental backup

[0096] Perform incremental backup. The backup agent identifies file change situations through hash comparison and modification time comparison, stores the changed files, generates a new version identifier VID at the same time, and updates the S3 bucket-level virtual snapshot;

[0097] In this example, the backup agent performs the incremental backup operation. The key to incremental backup is to only back up the changed data. The specific operations are as follows:

[0098] The backup agent reads the metadata of the file to obtain the latest modification time information.

[0099] Compare the latest modification time with the records in the MongoDB indexing service to determine whether the file has changed. If the modification time of the file has changed, mark the file as needing backup and generate a new version identifier (VID).

[0100] Upload the changed file data and metadata to the S3 bucket and generate a new S3 bucket-level virtual snapshot.

[0101] Meanwhile, the backup agent writes information such as the changed file path, modification time, version identifier (VID), etc. to the MongoDB indexing service to update the index relationship.

[0102] Through hash comparison and modification time comparison, the system can accurately identify which files need to be incrementally backed up, ensuring that each backup only contains the changed data, greatly reducing the waste of storage space.

[0103] Step S5: Data storage and deduplication

[0104] Store the file data to be backed up and its metadata. The storage adopts an incremental storage and deduplication storage mechanism, and uses the IncDSI optimization algorithm to adjust the index position to avoid storage conflicts and ensure the data consistency of the S3 bucket-level virtual snapshot;

[0105] In this example, the backup agent performs deduplication and incremental storage operations on the backup data. The combination of incremental backup and deduplication storage mechanism can effectively optimize the storage space utilization rate. The specific operations are as follows:

[0106] Incremental storage: Each incremental backup only saves the changed data and does not repeat the data that has been backed up before.

[0107] Deduplication storage mechanism: By deduplicating data blocks, redundant data storage is avoided. For example, if a file has not changed during the incremental backup, the system will not back up the file again.

[0108] IncDSI optimization algorithm: This algorithm dynamically adjusts the index position to avoid storage conflicts and ensure the uniqueness and consistency of the incremental backup data.

[0109] Through these mechanisms, while ensuring the accuracy of the backup, the present invention significantly improves the utilization efficiency of the storage space.

[0110] Step S6: Backup chain integration

[0111] When the incremental backup chain exceeds the preset threshold, perform backup chain integration, use transaction logs and timestamp partition storage for data merging, and clean up snapshot data based on the retention period and retention count of the backup policy;

[0112] Specifically, the backup chain integration process adopts batch transaction processing to perform data optimization in the storage system, where:

[0113] The transaction log mechanism records the data changes of the backup chain to ensure data consistency;

[0114] The backup chain merging operation migrates the latest version of the data to the new backup chain set to reduce storage fragmentation;

[0115] The timestamp partition storage stores the data according to the snapshot generation time, and clears the snapshot data based on the retention period and the number of retained snapshots of the backup policy.

[0116] Step S7: Data recovery

[0117] Based on the directory and file VID, data recovery is performed. The snapshot mapping relationship is retrieved to retrieve the snapshot data, and the data block reorganization technology is used to extract the required data blocks to restore the data corresponding to the S3 bucket-level virtual snapshot.

[0118] Specifically, the S3 bucket-level virtual snapshot supports accurate data backtracking based on the snapshot identifier, where:

[0119] Restore the file version based on the snapshot time point, retrieve the snapshot mapping relationship through the MongoDB index service, and restore the data at the specified time point;

[0120] The data block reorganization extracts the required data blocks, constructs a complete file, and performs data consistency verification.

[0121] In the data recovery process of this embodiment, the present invention can efficiently extract the required data blocks, ensure the accuracy and integrity of file recovery, and greatly improve the recovery efficiency.

[0122] In this embodiment, the backup agent program dynamically adjusts the backup frequency according to the free space of the storage system, the length of the backup chain, and the number of snapshots, and optimizes the storage resource configuration to improve the backup efficiency and reduce the storage overhead.

[0123] Among them, the management of incremental backup adopts a time sequence management method:

[0124] Each incremental backup T1, T2, Tn, T(N+1) generates a complete backup snapshot, and integrates the incremental data with the backup snapshot at the previous moment to form a new snapshot; among them, the data of each generated complete backup snapshot supports fast mounting to achieve fast recovery;

[0125] Whenever the backup chain reaches the preset threshold, the backup chain integration is performed, the old backup is merged with the current incremental snapshot, and the index information is migrated to the new backup chain set to ensure the continuity of the incremental backup chain and the efficient management of data.

[0126] Specifically, the backup system includes:

[0127] A network interface for data transmission between the backup agent program and the S3 storage service. The network interface uses the S3 RESTful API and HTTP / HTTPS protocols for remote communication and supports multi-channel streaming data transmission;

[0128] A memory for storing file system data and incremental backup data. The memory includes a local cache storage area and a remote S3 storage bucket. The local cache storage area is used to temporarily store incremental backup data and perform data deduplication, compression, and fragmentation. The remote S3 storage bucket adopts a multi-version storage policy and manages snapshot data in combination with a timestamp partitioning storage policy;

[0129] A processor for executing the computing tasks of the backup agent program. The processor adopts an asynchronous parallel processing architecture and executes index management, data transmission scheduling, and incremental backup task scheduling. Among them, index management optimizes the storage of snapshot metadata through B+ tree index and inverted index. Data transmission scheduling adjusts the data backup policy according to data modification frequency, access popularity, and storage status. Incremental backup task scheduling adjusts the storage location of incremental data in combination with the IncDSI optimization algorithm.

[0130] Among them, the network interface, memory, and processor are interconnected through a data bus to ensure the smooth execution of backup operations and data consistency. The processor adjusts the backup frequency and storage resource configuration according to the data status in the memory and the transmission status of the network interface to optimize the backup efficiency.

[0131] As Figure 2 shown, it describes the processes of full backup and incremental backup in the incremental backup method based on S3 bucket-level virtual snapshots, and how they interact with the storage system through a multi-channel method. Specifically, the figure is divided into two main parts: the upper part is about the processes of incremental backup and full backup, and the lower part is the storage and data access layer.

[0132] Full backup part:

[0133] Step T1 (Full backup): The data system first performs a full backup. All data to be backed up (such as {key1, vid1}, {key2, vid2}, etc.) is stored in the S3 bucket, and a unique version identifier (VID) is generated. The data at this time is saved as a full backup snapshot, labeled as T1.

[0134] After backup: The data after full backup is stored as a permanent snapshot and can be used for subsequent incremental backup or data recovery. T1 is the initial full backup snapshot.

[0135] Incremental backup part:

[0136] Step T2 (Incremental Backup 1): After the first full backup (T1), incremental backup starts. The incremental backup accurately identifies the changed parts of files through hash comparison and modification time comparison. At this time, only the changed files (such as {key1, vid2}, {key2, vid2}) are backed up and stored in the S3 bucket, generating a new version identifier (vid2). The incremental backup snapshot at this time is marked as T2.

[0137] Steps T3 to TN (Incremental Backup N): Over time, the incremental backup continues (from T3 to TN). Each incremental backup generates a new version of the snapshot, and only the parts that have changed relative to the previous snapshot (such as T1, T2, etc.) are backed up.

[0138] Backup Chain Integration:

[0139] When the incremental backup chain reaches a preset threshold, the backup system integrates the backup chain (as shown in the figure). The integrated data is merged into a new snapshot (T(N + 1)). This process uses a transaction log and a timestamp partitioning strategy to reduce storage fragmentation and improve data retrieval efficiency.

[0140] Data Recovery Process:

[0141] During data recovery, based on the directory and file VID, the backup system retrieves the required snapshot data through the index mapping relationship, and then restores the data at a specific point in time through data block reorganization technology to ensure the accuracy and consistency of the recovery process.

[0142] Lower Part - Storage System Interaction:

[0143] The lower part of the figure shows the interaction between the file system, the backup management system, and the S3 storage service:

[0144] Multi-Channel Concurrent Reading: The backup system efficiently reads data from the file system concurrently through a multi-channel streaming API.

[0145] Data Storage and Deduplication Mechanism: The backup system not only stores data in S3 but also uses a deduplication storage mechanism to avoid data redundancy.

[0146] S3 Storage and Backup Data: The data is uploaded to the S3 storage bucket and supports version control to ensure the integrity and consistency of the backup data.

[0147] By only storing the changed parts through incremental backup, the use of storage space is significantly reduced, while the backup efficiency is improved. Through the recovery mechanism based on the directory and file VID, data can be quickly restored from different time points. With the deduplication storage mechanism and the IncDSI optimization algorithm, storage conflicts and redundancies are effectively reduced, enhancing the storage efficiency during the backup process.

[0148] Through the above embodiments, the present invention adopts the efficient index management of MongoDB and the multi-version control feature of S3 object storage, and can achieve efficient and reliable data backup and recovery. Through incremental backup, deduplication storage, backup chain integration and fast data recovery, the present invention effectively improves the backup efficiency, reduces the storage overhead, and ensures the data consistency and rapidity of recovery.

[0149] In summary, the present invention provides a permanent incremental backup method based on S3 bucket-level virtual snapshots, which utilizes the multi-version control of S3 and the efficient index technology of MongoDB to effectively solve the problems of storage redundancy, low recovery efficiency and complex backup chain management in traditional backup technologies. Through innovative means such as incremental backup, deduplication storage and backup chain integration, the present invention not only improves the data backup efficiency and reduces the storage overhead, but also ensures the rapidity and accuracy of data recovery. The present invention has broad application prospects, is particularly suitable for data backup and management of large-scale file systems, and can provide an efficient and reliable backup solution for cloud storage and distributed file systems. Finally, the present invention provides a solid technical foundation for the long-term preservation and rapid recovery of file data, and has significant innovative value and practical application significance.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0151] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. And the scope of the preferred embodiments of the present application includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in an opposite order according to the involved functions, rather than in the order shown or discussed.

[0152] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed in the present application, and these should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A permanent incremental backup method based on S3 bucket-level virtual snapshots, characterized in that, It includes the following steps: Step S1: Initialize the backup system, start the backup agent program, index management service, and S3 object storage, and enable the multi-version control mechanism of S3 to manage S3 bucket-level virtual snapshots; Step S2: Perform a full backup. The backup agent program concurrently reads the file system data using the multi-channel streaming API, uploads the data to the S3 storage bucket, generates a unique version identifier VID, and creates an S3 bucket-level virtual snapshot at time T1; Step S3: Record the file path, snapshot identifier SnapshotID, and version identifier VID in the MongoDB index service. Establish a mapping relationship between the file path and the snapshot ID using a B+ tree index, and manage the file version history using the inverted index mechanism to achieve the index management of S3 bucket-level virtual snapshots; Step S4: Perform an incremental backup. The backup agent program identifies file changes through hash comparison and modification time comparison, stores the changed files, generates a new version identifier VID, and updates the S3 bucket-level virtual snapshot; Step S5: Store the file data to be backed up and its metadata. The storage uses an incremental storage and deduplication storage mechanism, and adjusts the index position using the IncDSI optimization algorithm to avoid storage conflicts and ensure the data consistency of S3 bucket-level virtual snapshots; Step S6: When the incremental backup chain exceeds the preset threshold, perform backup chain integration. Use transaction logs and timestamp partitioning storage for data merging, and clean up snapshot data based on the retention period and retention count of the backup policy; Step S7: Perform data recovery based on the directory and file VID snapshot ID. Retrieve the snapshot data using the snapshot mapping relationship, and use the data block recombination technology to extract the required data blocks to recover the data corresponding to the S3 bucket-level virtual snapshot.

2. The permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 1, wherein, In Step S2, the backup agent program identifies the files to be backed up through hash comparison and modification time comparison, and generates the version identifier VID of the files, where: Hash comparison: Calculate the hash value of the file content and compare it with the hash value stored in the MongoDB index service. If they match, only update the index relationship. If they do not match, perform data storage and update the index information; Modification time comparison: Mark the files to be backed up by comparing the latest modification time of the files to ensure the accuracy of the incremental backup.

3. A permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 1, characterized in that, In Step S3, the MongoDB index service uses a B+ tree structure and an inverted index mechanism for data management, where: The B+ tree index structure is used to store the mapping relationship between the file path, snapshot identifier SnapshotID, and version identifier VID to achieve efficient retrieval; The inverted index mechanism improves the efficiency of querying the file version by the snapshot ID by maintaining the mapping relationship from the file version to the snapshot ID. The incremental index update algorithm performs old index merging, data deduplication, and index optimization when the index data is updated, reduces redundant storage, and improves the query performance, optimizing the storage and query efficiency during the incremental backup process.

4. A permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 3, characterized in that, The incremental index update algorithm includes: Step S3.1: Calculate the index position of the backup data based on the linear regression model of the file feature vector, predict the optimal storage position of the incremental backup data in the index, and reduce index redundancy; Step S3.2: Calculate the predicted storage position through the linear regression model, and the formula is as follows: M(k) = |A × G(k) + b|; Where: M(k) is the predicted position of the backup data, indicating the storage position of the incremental backup data in the index; A is the parameter matrix of the linear regression model, indicating the weighted influence of different data features on the index position; G(k) is the feature vector of the backup data, including file size, modification time, and access frequency information; b is the bias term used to adjust the predicted position; Step S3.3: Based on the dynamic index adjustment method of the optimization objective function, when the deviation between the predicted position and the actual storage position exceeds the preset threshold, perform dynamic index adjustment to optimize the storage method of the incremental backup data, and the formula is: L(v n+1 ) = λ1l1(v n+1 )+(1 - λ1)l2(v n+1 )+λ2||v n+1 || 2 ; Among them: L(v n+1 ) is the feature vector of the newly added backup data, representing the storage characteristics of the incremental backup data; λ1 and λ2 are adjustment parameters, respectively used to control the balance between the accuracy of index storage and the update efficiency; l1(v n+1 ) is used to optimize the accuracy of the index position; l2(v n+1 ) is used to optimize the storage efficiency of index updates; ||v n+1 || 2 represents the data norm, controlling the stability of the data during the index optimization process; Step S3.4: Based on the conflict detection and index rearrangement method of similarity calculation, when a conflict occurs in the predicted storage position of the incremental backup data, use the IncDSI optimization algorithm to readjust the index position to ensure the uniqueness of the incremental backup data, and the formula is: Wherein: is the query vector for the newly added incremental backup data, representing the query features of the current data; is the query vector for the existing incremental backup data; v j is the feature vector of the existing incremental backup data; Step S3.5: Adopt the B+ tree index and inverted index optimization method to improve the data query efficiency based on the snapshot ID, and ensure that the backup data can be quickly retrieved and restored.

5. A permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 1, characterized in that, In Step S6, the backup chain integration process adopts batch transaction processing to perform data optimization in the storage system, where: The transaction log mechanism records the backup chain data changes to ensure data consistency; The backup chain merge operation migrates the latest version of the data to the new backup chain set to reduce storage fragmentation; The timestamp partition storage stores the data based on the snapshot generation time, and cleans the snapshot data based on the retention period and retention count of the backup policy.

6. A permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 1, characterized in that, The S3 bucket-level virtual snapshot supports accurate data backtracking based on the snapshot identifier, where: Restore the file version based on the snapshot time point, retrieve the snapshot mapping relationship through the MongoDB index service, and restore the data at the specified time point; The data block reorganization extracts the required data blocks, constructs a complete file, and performs data consistency verification.

7. A permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 1, characterized in that The backup agent program dynamically adjusts the backup frequency according to the free space, backup chain length, and snapshot quantity of the storage system, optimizes the storage resource configuration, improves the backup efficiency, and reduces the storage overhead.

8. A permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 1, characterized in that The management of the incremental backup adopts a time-series management method: Each incremental backup T1, T2, Tn, T(N+1) generates a complete backup snapshot, and integrates the incremental data with the backup snapshot at the previous moment to form a new snapshot; among them, the data of each generated complete backup snapshot supports fast mounting to achieve fast recovery; Whenever the backup chain reaches the preset threshold, perform backup chain integration, merge the old backup with the current incremental snapshot, and migrate the index information to the new backup chain set to ensure the continuity of the incremental backup chain and the efficient management of data.

9. A permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 1, characterized in that The backup system includes: A network interface for data transmission between the backup agent and the S3 storage service. The network interface uses the S3 RESTful API and the HTTP / HTTPS protocol for remote communication and supports multi-channel streaming data transmission; A memory for storing file system data and incremental backup data. The memory includes a local cache storage area and a remote S3 storage bucket. The local cache storage area is used to temporarily store incremental backup data and perform data deduplication, compression, and fragmentation. The remote S3 storage bucket adopts a multi-version storage strategy and manages snapshot data in combination with a timestamp partitioning storage strategy; A processor for executing the computing tasks of the backup agent. The processor adopts an asynchronous parallel processing architecture and executes index management, data transmission scheduling, and incremental backup task scheduling. Among them, index management optimizes the storage of snapshot metadata through B+ tree indexing and inverted indexing. Data transmission scheduling adjusts the data backup strategy according to data modification frequency, access popularity, and storage status. Incremental backup task scheduling adjusts the storage location of incremental data in combination with the IncDSI optimization algorithm; 10. A permanent incremental backup method based on S3 bucket-level virtual snapshots according to claim 9, characterized in that, The network interface, memory, and processor are interconnected through a data bus to ensure the smooth execution of backup operations and data consistency. The processor adjusts the backup frequency and storage resource configuration according to the data status in the memory and the transmission status of the network interface to optimize the backup efficiency.

Images