Security increment synchronization method for continuous data protection

Abstract

The invention discloses a security increment synchronization method for continuous data protection, which belongs to the technical field of cloud storage and data protection, and comprises the following steps: S1, when a client initiates a write operation for updating a local file, comparing a client version number with a cloud version number; s2, generating incremental content, and distributing a unique block ID for each formula block according to the file formula; s3, encrypting the data blocks to obtain encrypted data blocks, and disrupting the sequence of the encrypted data blocks; s4, the client uploads the encrypted data blocks and the block IDs corresponding to the encrypted data blocks, and updates a deletion formula of the cloud; and S5, the cloud end manages the encrypted data block uploaded in the step S4 through a key value storage system, updates the version number of the cloud end, and notifies the client to update the corresponding version number at the same time. According to the method, the advantages of Rsync and Dsync are combined, an informed redundancy detection mechanism is provided, byte-level redundancy is ensured to be eliminated, calculation and bandwidth overhead is remarkably reduced, and an encryption protection mechanism of a data and synchronization mode is introduced.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud storage and data protection, and in particular to a secure incremental synchronization method for continuous data protection. Background Art

[0002] Continuous Data Protection (CDP) is a real-time data protection technology designed to provide continuous data backup and recovery capabilities. Unlike traditional backup methods, CDP continuously monitors and records client data changes, providing real-time protection for files and data. When client data changes, the CDP system immediately captures these changes and synchronizes them to the cloud or a remote server, ensuring that every update is backed up promptly.

[0003] In addition to real-time synchronization, CDP must also provide reliable security to prevent the leakage of sensitive information. Specifically, CDP must ensure the confidentiality of synchronized data while hiding data access patterns to prevent potential attackers from inferring user behavior. To this end, Oblivious Random Access Machine (ORAM) technology has been proposed to effectively mitigate the risk of access pattern leakage.

[0004] Oblivious Random Access Machine (ORAM) is a technology for protecting data privacy. Existing work (see Aviv A J, Choi SG, Mayberry T, et al. ObliviSync: Practical oblivious file backup and synchronization. Network and Distributed System Security Symposium. 2017) suggests that it prevents external observers from inferring data content by hiding data access patterns. It pseudo-randomly accesses stored data, avoiding exposure to access patterns and thus improving security. However, ORAM suffers from two issues: 1) Scalability. While ORAM-based synchronization methods can protect synchronization patterns, they also suffer from poor scalability. As data volume grows and the storage array gradually fills up, the client needs to continuously select free slots to synchronize data blocks, resulting in inefficient synchronization. This process significantly increases bandwidth overhead, and the computational overhead of decrypting and re-encrypting data blocks also increases significantly with increasing data volume. 2) Bandwidth overhead. In the best-case scenario, the bandwidth overhead of ORAM-based synchronization methods is k times that of uploading and modifying data blocks.

[0005] Therefore, due to scalability and high bandwidth overhead, the ORAM algorithm struggles to meet the real-time requirements of CDP. To address this challenge, there are various incremental synchronization methods that can be used with CDP to improve synchronization efficiency and reduce bandwidth overhead, such as Rsync and block-based incremental synchronization, and the secure synchronization method Rsyncrypto.

[0006] Rsync is a sliding window-based incremental synchronization method that improves synchronization efficiency by avoiding the retransmission of byte-level redundancy. When a client modifies a file and synchronizes it with an earlier version stored in the cloud, the cloud first divides the original file into fixed-size blocks and sends the fingerprints of these blocks (such as cryptographic hashes) to the client. The client calculates the fingerprint of each sliding window in the modified file and compares it with the fingerprint provided by the cloud. If there is a match, the client skips the block; if there is a mismatch, only the modified part (delta) is uploaded. However, Rsync has a large computational overhead because it needs to generate a large number of fingerprints for each byte-level sliding window.

[0007] Block-based incremental synchronization (Shilane P, Huang M, Wallace G, et al. WAN-optimized replication of backup datasets using stream-informed deltacompression. 10th USENIX Conference on File and Storage Technologies. 2012) significantly reduces the computational overhead of Rsync by detecting modified content at a coarse-grained data block level, avoiding the generation of large fingerprints within a byte-level sliding window. Specifically, the cloud partitions the original file into variable-sized data blocks using content-defined chunking (CDC), effectively alleviating the boundary shift issues associated with fixed-size chunking. The cloud then sends the fingerprints of these data blocks to the client. When processing the modified file, the client uses the same chunking method and compares the generated block fingerprints with the fingerprints provided by the cloud. This allows the client to identify modified data blocks and upload them to the cloud. However, block-based incremental synchronization can only detect redundancy at the block level and may incur higher bandwidth overhead compared to Rsync.

[0008] Although the above two algorithms can effectively improve bandwidth efficiency, they have the problem of insufficient security. These two synchronization methods do not perform encryption and cannot guarantee the security of user data. Rsynccrypto enhances encryption security by extending the cipher block chaining (CBC) mode. By encrypting the data block with the previous encrypted block after XOR, it ensures that data modifications will not affect the encryption status of other data blocks. Although this algorithm can protect the data content, it fails to effectively protect the synchronization mode, that is, it fails to properly control how and which data changes are synchronized to the cloud. This means that the system may still leak which data blocks have been updated in the original file. In addition, the encryption method used is relatively weak and only supports applying differences on encrypted blocks, which limits the security and privacy protection of encrypted synchronization. Summary of the Invention

[0009] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a secure incremental synchronization method for continuous data protection.

[0010] The object of the present invention is achieved through the following technical solutions:

[0011] The present invention discloses a secure incremental synchronization method for continuous data protection, comprising the following steps:

[0012] S1. A file storage system includes a cloud and multiple clients, wherein the client is used to segment updated files, and the cloud is used to store files. When a client initiates a write operation to update a local file, the client version number is first compared with the cloud version number. If the local file version is found to be outdated, the client version is restored. If the local file version is the latest version, the client version is synchronized.

[0013] S2. During the synchronization operation, the client generates incremental content based on the updated file and assigns a unique block ID to each recipe block based on the file recipe. The client appends the block ID of the recipe block to the file recipe. The block ID is eight bytes long, with the first four bytes being the cloud version number received by the client during this synchronization, and the last four bytes being a random number generated using a random initialization vector (IV).

[0014] S3. The client adds a one-byte prefix to each data block and adds eight bytes to the end of each recipe block. The client encrypts each data block using a semantically secure encryption method to obtain an encrypted data block and scrambles the order of the encrypted data blocks using a sixteen-byte random initialization vector.

[0015] S4. The client uploads the encrypted data block and its corresponding block ID to the cloud and updates the deletion recipe in the cloud.

[0016] S5. The cloud manages the encrypted data blocks uploaded in step S4 through a key-value storage system. Each block ID is mapped to the corresponding encrypted data block. After receiving the encrypted data block, the cloud adds it to the block storage and updates the cloud version number. At the same time, the cloud notifies the client to perform the corresponding version number update.

[0017] Further, the restoration in step S1 specifically includes the following steps:

[0018] S11. The client submits the current client version number Dv[Client] to the cloud. The cloud stores it in the query block and returns the encrypted data blocks whose block ID prefix ranges are within [Dv[Client]+1, Dv[Cloud]], where Dv[Cloud] represents the cloud version number.

[0019] S12. The client receives and decrypts the encrypted data blocks based on the version order, and splices the recipe blocks in the order of block ID to restore the complete file recipe. The version order is from Dv[Client]+1 to Dv[Cloud].

[0020] S13. For each entry in the file recipe, the client checks the version number Cv in the corresponding block ID. If Cv < i, where i represents the version, the client obtains the corresponding block from the local file. If Cv = i, the client obtains the corresponding base block and performs delta decompression, or directly restores the complete block.

[0021] S14. Repeat step S13 until all versions of the file blocks are restored. Finally, the client updates its version number to Dv[Cloud].

[0022] Preferably, the generation of incremental content in step S2 specifically includes the following steps:

[0023] S21. If it is the first update of the file, the source file and the generated file recipe are chunked using a chunking method based on the data block content. After generating the corresponding metadata, it is directly uploaded to the cloud. If it is not the first update, the recipe file corresponding to the file is first read.

[0024] S22. After reading the recipe file, the block IDs of the previously generated recipe blocks are put into the deletion recipe through this file.

[0025] S23. After updating and deleting the recipe, the system divides the new file into blocks to generate data blocks, then identifies new data blocks and duplicate data blocks in the data blocks, and marks the data blocks during the identification process. After identification, the new data blocks are uploaded and the duplicate data blocks are removed; then, a file recipe for the new file is generated, and the file recipe for the new file is divided into blocks to generate file recipe blocks. Finally, the new data blocks and the file recipe blocks are shuffled to confuse the positional relationship between the data blocks;

[0026] S24. Update the deletion recipe based on the mark in step S23. If the block in the old file is not marked, add the block ID of the block to the deletion recipe.

[0027] S25. Process the file recipe of the update file and cut the file recipe of the update file into blocks using a block-based method based on the data block content. After each block is processed, generate the corresponding metadata and put it into the sorting queue to shuffle the order. After generating a recipe file block, add the block ID of the block to the end of the recipe file.

[0028] Preferably, the identification of duplicate data blocks in step S23 is based on the locality of file modification, i.e., file modifications are concentrated in the first region, and changes in other regions except the first region are within a preset range. Specifically, the system maintains a variable offset that records the logical position of potential identical blocks when processing each data block. The initial value of offset is 0, indicating the beginning of the old file, and the value recorded by offset is the block position in the old file, not the block position in the updated file.

[0029] The system also maintains a variable havesimilarchunk to identify whether the data block currently being processed by the new file has a potential identical block. Its initial value is true. When the system is about to generate a data block from the remaining unblocked content, it first checks havesimilarchunk. If its initial value is true, it further reads the information of the data block M corresponding to the offset and uses the length of the data block M as the length of the data block to be generated.

[0030] Calculate the hash value of data block M, then generate a hash value for the data block to be generated based on the length of data block M, and compare it with the hash value of data block M. If the two hash values are the same, the data block to be generated is a duplicate data block, and the system sets the initial value of havesimilarchunk to true and offset to M+1; otherwise, the data block to be generated is a new data block; if the initial value of havesimilarchunk is false before generating the data block, a new data block is generated using the CDC block technology;

[0031] The hash value is used to determine whether the data block to be generated matches the data block in any old file. If the data block to be generated matches any old data block, the data block is a duplicate data block; the system sets the initial value of havesimilarchunk to true and offset to M+1; otherwise, the data block to be generated is a new data block; the system sets the initial value of havesimilarchunk to false.

[0032] Preferably, after identifying the new data block in step S23, the system performs fine-grained redundancy reduction through incremental compression, specifically including: if the data block to be generated is considered to be a new data block, and the initial value of havesimilarchunk is true at this time, the data block corresponding to the offset is used as the basic data block, and the new data block is incrementally compressed to generate an incremental block; if the size of the incremental block is less than 1 / n of the size before compression, the incremental block is retained, and the initial value of havesimilarchunk is set to true, and the offset is set to M+1; otherwise, the system does not perform incremental compression, and the initial value of havesimilarchunk is set to false.

[0033] Preferably, the marking method described in step S23 includes: before the block cutting starts, setting a block ID to number mapping as a mark through the recipe file of the old file, initializing the mark and setting it to 0; if the block is a duplicate data block, setting the mark to 2; if the block is a basic block of the incremental compression block, setting the mark to 1.

[0034] Preferably, the updating of the cloud deletion recipe in step S4 specifically includes the following steps:

[0035] S41. When the cloud starts up, it determines the capacity of the deletion recipe based on the number of encrypted data blocks uploaded for the first time. The deletion recipe stores the block IDs corresponding to the data blocks to be deleted. The length of the block IDs is 5N, where N represents the product of the average number of blocks and the length of the block ID. For each subsequent upload, the cloud calculates the average number of uploaded blocks.

[0036] S42. After the client processes the file block, it determines the length of the deletion formula. If it exceeds 80% of the capacity, the cloud update will perform a deletion operation, upload the deletion formula in plain text, and update the total length of the deletion formula based on the length 5N. If it does not exceed, the ciphertext is uploaded and the cloud does not perform the deletion operation.

[0037] The beneficial effects of the present invention are:

[0038] 1) This invention combines the advantages of Rsync and Dsync, proposing an "informed redundancy detection" mechanism. This eliminates byte-level redundancy while significantly reducing computational and bandwidth overhead. Existing Rsync-based synchronization methods typically require byte-level comparisons and hash calculations, which incur significant computational overhead. In contrast, this invention significantly reduces this overhead through innovative mechanisms, improving synchronization efficiency and ensuring efficient data transmission.

[0039] 2) This invention introduces an encryption mechanism for data and synchronization patterns, addressing the shortcomings of traditional synchronization solutions in protecting the synchronization pattern. Existing encryption synchronization methods typically use weak encryption methods and fail to effectively protect the synchronization pattern. However, this invention significantly improves the security of the synchronization process by encrypting and protecting the data content and synchronization pattern, effectively preventing potential attackers from leaking information by analyzing the synchronization pattern.

[0040] 3) This invention was successfully integrated into an existing cloud-edge fusion synchronization system and validated its effectiveness and security in real-world environments. Existing research has largely focused on single synchronization modes or storage optimization, lacking synchronization solutions tailored to cloud-edge fusion environments. Through this integration, this method not only improves synchronization performance but also enhances data transmission security in cloud-edge fusion environments, resolving data security risks inherent in traditional synchronization solutions and providing a new technical path for future research in this area.

[0041] 4) This invention utilizes a multi-level synchronization optimization scheme, combining ORAM-based synchronization mode protection, incremental synchronization schemes, and redundancy detection technology. Traditional ORAM methods, due to their high overhead, cannot meet the requirements for efficient synchronization. However, this invention significantly reduces storage and bandwidth consumption during synchronization by employing write-only ORAM and intelligent redundancy detection technology, while effectively protecting the synchronization mode. Furthermore, by combining byte-level redundancy elimination with performance optimization strategies, this invention further improves synchronization efficiency and security, ensuring the security and consistency of data content. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic flow chart of the steps of a secure incremental synchronization method for continuous data protection according to an embodiment of the present invention;

[0043] Figure 2 Schematic diagram showing the file writing speed results of the present invention compared with the other four existing methods under four data sets according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0045] The present invention discloses a secure incremental synchronization method for continuous data protection, which aims to solve the problems of insufficient security, low scalability and low bandwidth efficiency in the existing technology. The method is mainly used in the field of cloud storage. While ensuring data security, it significantly improves data storage efficiency, enhances bandwidth efficiency during upload, and effectively saves storage overhead. In terms of specific implementation, the method supports data reading and writing and file update operations by deploying a client on the cloud storage user side, and deploys a server on the cloud to store the latest version of the file for easy data recovery. The flow chart of its steps is as follows: Figure 1 As shown, the specific steps include:

[0046] S1. A file storage system includes a cloud and multiple clients, wherein the client is used to segment updated files, and the cloud is used to store files. When a client initiates a write operation to update a local file, the client version number is first compared with the cloud version number. If the local file version is found to be outdated, the client version is restored. If the local file version is the latest version, the client version is synchronized.

[0047] S2. During the synchronization operation, the client generates incremental content based on the updated file and assigns a unique block ID to each recipe block based on the file recipe. The client appends the recipe block ID to the file recipe for subsequent garbage collection (GC) and tracking of these recipe blocks. The block ID is eight bytes long, with the first four bytes representing the cloud version number received by the client during this synchronization, and the last four bytes representing a random number generated using a random initialization vector (IV). This generation method ensures the randomness and uniqueness of the block ID, improving the security and unpredictability of data blocks.

[0048] S3. The client adds a one-byte prefix to each data block to distinguish it from the recipe block. It also adds an eight-byte block ID at the end of each recipe block to identify the block ID of the next recipe block. The client encrypts each data block using a semantically secure encryption method to obtain an encrypted data block. It then uses a sixteen-byte random initialization vector to scramble the order of the encrypted data blocks, thereby breaking the correlation between the logical order of the data blocks and the upload order.

[0049] S4. The client uploads the encrypted data block and its corresponding block ID to the cloud and updates the deletion recipe in the cloud to facilitate garbage collection (GC) and pattern hiding.

[0050] S5. The cloud manages the encrypted data blocks uploaded in step S4 through a key-value storage system. Each block ID is mapped to the corresponding encrypted data block. After receiving the encrypted data block, the cloud adds it to the block storage and updates the cloud version number. Meanwhile, the cloud notifies the client to perform the corresponding version number update.

[0051] Exemplarily, the restoration described in step S1 specifically includes the following steps:

[0052] S11. The client submits the current client version number Dv[Client] to the cloud. The cloud stores it in the query block and returns the encrypted data blocks whose block ID prefix ranges are within [Dv[Client] + 1, Dv[Cloud]], where Dv[Cloud] represents the cloud version number.

[0053] S12. The client receives and decrypts the encrypted data blocks based on the version order, and splices the recipe blocks in the order of block ID to restore the complete file recipe. The version order is from Dv[Client] + 1 to Dv[Cloud].

[0054] S13. For each entry in the file recipe, the client checks the version number Cv in the corresponding block ID. If Cv < i (indicating that this data block already existed in version i - 1), where i represents the version, the client obtains the corresponding block from the local file. If Cv = i (indicating that this data block is a differential block obtained by delta compression based on the base block of version i - 1 or a new complete block of version i), the client obtains the corresponding base block and performs delta decompression, or directly restores the complete block.

[0055] S14. Repeat step S13 until all versions of the file blocks are restored. Finally, the client updates its version number to Dv[Cloud].

[0056] Exemplarily, the generation of incremental content described in step S2 specifically includes the following steps:

[0057] S21. If it is the first update of the file, the source file and the generated file recipe are chunked using a chunking method based on the data block content. After generating the corresponding metadata, it is directly uploaded to the cloud. If it is not the first update, the recipe file corresponding to this file is first read.

[0058] S22. After reading the recipe file, the block IDs of the previously generated recipe blocks are put into the deletion recipe through this file, so that the cloud can update the file version and save storage space.

[0059] S23. After updating and deleting the recipe, the system divides the new file into blocks to generate data blocks, then identifies new data blocks (i.e., increments) and duplicate data blocks (i.e., redundancies) in the data blocks, and marks the data blocks during the identification process. After identification, the new data blocks are uploaded and the duplicate data blocks are removed. Then, a file recipe for the new file is generated, and the file recipe for the new file is divided into blocks to generate file recipe blocks. Finally, the new data blocks and the file recipe blocks are shuffled to confuse the positional relationship between the data blocks.

[0060] S24. Update the deletion recipe based on the mark in step S23. If the block in the old file is not marked, add the block ID of the block to the deletion recipe so that the cloud can perform data updates later, saving storage space.

[0061] S25. Process the file recipe of the updated file and cut the file recipe of the updated file into blocks using a block-based method based on the data block content. After each block is processed, the corresponding metadata is generated and placed into a sorting queue to shuffle the order. After a recipe file block is generated, the block ID of the block is added to the end of the recipe file so that the recipe can be updated and deleted during the next update.

[0062] Exemplarily, the duplicate data block identification described in step S23 is based on the local characteristics of file modification, that is, the file modification is concentrated in the first area, and the changes in other areas except the first area are within a preset range, and the preset range is a range pre-set according to the specific working environment and is not limited to a specific numerical range; specifically, the system maintains a variable offset that records the logical position of potential identical (similar) blocks when processing each data block; the initial value of offset is 0, indicating the file beginning of the old file, and the value recorded by offset is the block position in the old file, not the block position in the updated file; the system also maintains a variable havesimilarchunk for identifying whether there are potential identical (similar) blocks in the data block currently being processed by the new file, and its initial value is true; when the system is about to generate a data block from the remaining content of the unblocked (new file), it first checks havesimilarchunk. If its initial value is true, it further reads the information of the data block M corresponding to offset, and uses the length of the data block M as the length of the data block to be generated. degree; in this embodiment, the hash value of the data block M is calculated by the SHA-256 algorithm, and then a hash value is generated for the data block to be generated based on the length of the data block M, and compared with the hash value of the data block M. If the two hash values are the same, the data block to be generated is a duplicate data block, and the system sets the initial value of havesimilarchunk to true and the offset to M+1; otherwise, the data block to be generated is a new data block; if the initial value of havesimilarchunk is false before the data block is generated, a new data block is generated by the CDC block technology; in this embodiment, the new data block is generated by the FastCDC technology; the hash value is used to determine whether the data block to be generated matches the data block in any old file. If the data block to be generated matches any old data block, the data block is a duplicate data block; the system sets the initial value of havesimilarchunk to true and the offset to M+1; otherwise, the data block to be generated is a new data block; the system sets the initial value of havesimilarchunk to false.

[0063] Exemplarily, after identifying the new data block in step S23, the system performs fine-grained redundancy reduction through incremental compression, specifically including: if the data block to be generated is considered to be a new data block, and the initial value of havesimilarchunk is true at this time, the data block corresponding to the offset is used as the basic data block, and the new data block is incrementally compressed to generate an incremental block; if the size of the incremental block is less than 1 / n of the size before compression, in this embodiment, the value of n in the incremental block size judgment condition is 3; the incremental block is retained, and the initial value of havesimilarchunk is set to true, and the offset is set to M+1; otherwise, the system does not perform incremental compression, and the initial value of havesimilarchunk is set to false.

[0064] For example, the marking method described in step S23 includes: before the block division begins, a block ID to number mapping is set as a marker in the recipe file of the old file, and the marker is initialized to 0; if the block is a duplicate data block, the marker is set to 2; if the block is a base block of an incrementally compressed block, the marker is set to 1. This marking method can update the information of the base block at the same time as the block to be deleted, facilitating file reconstruction.

[0065] Exemplarily, the updating of the cloud deletion recipe described in step S4 specifically includes the following steps:

[0066] S41. When the cloud starts up, it determines the capacity of the deletion recipe based on the number of encrypted data blocks uploaded for the first time. The deletion recipe stores the block IDs corresponding to the data blocks to be deleted. The length of the block IDs is 5N, where N represents the product of the average number of blocks and the length of the block ID. For each subsequent upload, the cloud calculates the average number of uploaded blocks.

[0067] S42. After the client processes the file block, it determines the length of the deletion recipe. If it exceeds 80% of the capacity, the cloud will perform a deletion operation, upload the deletion recipe in plain text, and update the total length of the deletion recipe based on the length 5N. If it does not exceed the capacity, the ciphertext is uploaded and the cloud does not perform the deletion operation. This operation is to hide the synchronization mode and improve security.

[0068] Exemplarily, the present invention is compared and analyzed, specifically including: 1) a synchronization method based on content-defined blocking. First, content-defined blocking is performed, and then block-based incremental synchronization is performed. In this embodiment, the Plain method is implemented using C++, and to ensure a fair comparison, the content-defined blocking in Plain is configured, and the minimum, average and maximum block sizes are 2KiB, 4KiB and 8KiB, respectively, which is the same as the method of the present invention. Through this method, the performance of the synchronization scheme based on traditional content-defined blocking in incremental synchronization can be analyzed. 2) Rsyncrypto, this method implements encrypted synchronization and encrypts file data using deterministic IV in CBC mode. Encryption is performed using rsyncrypto v1.13, and the rsync protocol is implemented through librsync v2.3.4 to perform incremental synchronization on the encrypted data. This scheme can evaluate the impact of the encrypted synchronization method on synchronization efficiency and security, and compare it with the secure incremental synchronization method of the present invention. 3) A synchronization method based on write-only ORAM (blind synchronization). The open source implementation of Oblivisync is used for evaluation. It should be noted that Oblivisync relies on Dropbox for synchronization. In order to adapt Oblivisync to the test environment, a CDP directory was set up in the cloud through Samba and mounted to the client through the CIFS protocol. This setting ensures that the content written in the client CDP directory can be automatically synchronized to the cloud. According to the default configuration, a 1024-slot storage array (each slot is 1MiB) is used, and three slots are randomly selected in each round to store block data. With this configuration, the performance of the ORAM-based blind synchronization method in the cloud environment can be evaluated. 4) base, this method uses FastCDC (Fast Content-Defined Blocking) technology to block file data and uses existing function-based methods (such as Finesse) to identify basic blocks for Delta compression. This method is a comparative benchmark that demonstrates the basic way of compressing and synchronizing data blocks in traditional incremental synchronization technology to evaluate its performance in synchronization efficiency and storage optimization.

[0069] For example, the evaluation is performed on four real-world datasets, each containing 20 versions: (i) MdKuber[6], containing Kubernetes change logs from v19.0 to v19.16, stored in markdown format, with an average size of 375.8 KiB per version; (ii) HtmlCv

[10] , containing OpenCV API documentation files, stored in html format, with an average size of 2.0 MiB per version; (iii) TarDiffu[3], containing Diffuser source code from v0.23.1 to v0.30.3, stored in tarball format, with an average size of 21.9 MiB per version; (iv) TarLinux[8], containing Linux source code from v2.6.11 to v2.6.14, stored in tarball format, with an average size of 207.4 MiB per version. Among them, MdKuber and HtmlCv simulate the incremental synchronization of small files in user editing mode, and TarDiffu and TarLinux simulate the incremental synchronization during software development. The results of the file writing speed of the present invention compared with the other four existing methods under four data sets are shown in the figure below. Figure 2 As shown, where * represents the results of the present invention, it can be seen that this method performs best among all the compared methods. On the TarDiffu dataset, this method improves the synchronization speed by up to 2.5 times and 4.2 times compared with the Plain and Base methods, respectively. The reason is that this method only performs content-defined segmentation in a few cases and avoids feature calculation for each non-repeated block, which significantly reduces the computational overhead compared to the Base method. In addition, thanks to the use of the informed redundancy detection (same block) mechanism, this method is 19.1 times faster than the RsyncCrypto method. When byte-level redundancy is detected, informed redundancy detection helps to reduce the performance overhead of Rsync.

[0070] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A secure incremental synchronization method for continuous data protection, characterized in that: It includes the following steps: S1. The file storage system includes a cloud and multiple clients. The clients are used to chunk the updated files, and the cloud is used to store the files. When a client initiates a write operation to update the local file, first compare the client version number with the cloud version number. If it is found that the version of the local file is outdated, recovery is performed. If the version of the local file is the latest version, synchronization is performed; S2. In the synchronization operation, the client generates incremental content based on the updated file. At the same time, according to the file recipe, a unique block ID is assigned to each recipe block. In the file recipe, the client appends the block ID of the recipe block. The total length of the block ID is eight bytes. The first four bytes are the cloud version number received by the client during this synchronization, and the last four bytes are a random number generated using a random initialization vector IV; S3. The client adds a one-byte prefix to each data block and adds eight bytes at the end of each recipe block. The client uses a semantically secure encryption method to encrypt each data block to obtain an encrypted data block, and uses a sixteen-byte random initialization vector to scramble the order of the encrypted data blocks; S4. The client uploads the encrypted data blocks and their corresponding block IDs to the cloud and updates the deletion recipe in the cloud; S5. The cloud manages the encrypted data blocks uploaded in step S4 through a key-value storage system. Each block ID is mapped to the corresponding encrypted data block. After receiving the encrypted data block, the cloud adds it to the block storage and updates the cloud version number. At the same time, the cloud notifies the client to perform the corresponding version number update.

2. A secure incremental synchronization method for continuous data protection according to claim 1, characterized in that: The recovery described in step S1 specifically includes the following steps: S11. The client submits the current client version number Dv[Client] to the cloud. The cloud stores it in the query block and returns the encrypted data blocks whose block ID prefix ranges from [Dv[Client]+1, Dv[Cloud]], where Dv[Cloud] represents the cloud version number; S12. The client receives and decrypts the encrypted data blocks based on the version order, and splices the recipe blocks in the order of the block IDs to restore the complete file recipe. The version order is from Dv[Client]+1 to Dv[Cloud]; S13. For each entry in the file recipe, the client checks the version number Cv in the corresponding block ID. If Cv < i, where i represents the version, the client obtains the corresponding block from the local file. If Cv = i, the client obtains the corresponding base block and performs delta decompression, or directly restores the complete block; S14. Repeat step S13 until all versions of the file blocks are restored. Finally, the client updates its version number to Dv[Cloud].

3. A secure incremental synchronization method for continuous data protection according to claim 2, characterized in that: The generation of incremental content described in step S2 specifically includes the following steps: S21. If it is the first time to update the file, use a chunking method based on the data block content to chunk the source file and the generated file recipe, generate the corresponding metadata, and directly upload it to the cloud. If it is not the first time to update, first read the recipe file corresponding to the file; S22. After reading the recipe file, the block ID of the previously generated recipe block is placed into the deletion recipe through the file; S23. After updating and deleting the recipe, the system divides the new file into blocks to generate data blocks, then identifies new data blocks and duplicate data blocks in the data blocks, and marks the data blocks during the identification process. After identification, the new data blocks are uploaded and the duplicate data blocks are removed; then, a file recipe for the new file is generated, and the file recipe for the new file is divided into blocks to generate file recipe blocks. Finally, the new data blocks and the file recipe blocks are shuffled to confuse the positional relationship between the data blocks; S24. Update the deletion recipe based on the mark in step S23. If the block in the old file is not marked, add the block ID of the block to the deletion recipe. S25. Process the file recipe of the update file and cut the file recipe of the update file into blocks using a block-based method based on the data block content. After each block is processed, generate the corresponding metadata and put it into the sorting queue to shuffle the order. After generating a recipe file block, add the block ID of the block to the end of the recipe file.

4. A secure incremental synchronization method for continuous data protection according to claim 3, characterized in that: The identification of duplicate data blocks in step S23 is based on the locality of file modifications, i.e., file modifications are concentrated in the first region, and changes in other regions outside the first region are within a preset range. Specifically, when processing each data block, the system maintains a variable offset that records the logical location of potentially identical blocks. The initial value of offset is 0, indicating the beginning of the old file, and the value recorded by offset is the block position in the old file, not the block position in the updated file. The system also maintains a variable havesimilarchunk to identify whether the data block currently being processed by the new file has a potential identical block. Its initial value is true. When the system is about to generate a data block from the remaining unblocked content, it first checks havesimilarchunk. If its initial value is true, it further reads the information of the data block M corresponding to the offset and uses the length of the data block M as the length of the data block to be generated. Calculate the hash value of data block M, then generate a hash value for the data block to be generated based on the length of data block M, and compare it with the hash value of data block M. If the two hash values are the same, the data block to be generated is a duplicate data block, and the system sets the initial value of havesimilarchunk to true and offset to M+1; otherwise, the data block to be generated is a new data block; if the initial value of havesimilarchunk is false before generating the data block, a new data block is generated using the CDC block technology; The hash value is used to determine whether the data block to be generated matches the data block in any old file. If the data block to be generated matches any old data block, the data block is a duplicate data block; the system sets the initial value of havesimilarchunk to true and offset to M+1; otherwise, the data block to be generated is a new data block; the system sets the initial value of havesimilarchunk to false.

5. The secure incremental synchronization method for continuous data protection according to claim 4, characterized in that: After identifying the new data block in step S23, the system performs fine-grained redundancy reduction through incremental compression, specifically including: if the data block to be generated is considered to be a new data block, and the initial value of havesimilarchunk is true at this time, the data block corresponding to the offset is used as the basic data block, and the new data block is incrementally compressed to generate an incremental block; if the size of the incremental block is less than 1 / n of the size before compression, the incremental block is retained, and the initial value of havesimilarchunk is set to true, and the offset is set to M+1; otherwise, the system does not perform incremental compression, and the initial value of havesimilarchunk is set to false.

6. A secure incremental synchronization method for continuous data protection according to claim 5, characterized in that: The marking method described in step S23 includes: before the block cutting starts, setting a block ID to number mapping as a mark through the recipe file of the old file, initializing the mark and setting it to 0; if the block is a duplicate data block, setting the mark to 2; if the block is a basic block of the incremental compression block, setting the mark to 1.

7. A secure incremental synchronization method for continuous data protection according to claim 6, characterized in that: The updating of the cloud deletion recipe described in step S4 specifically includes the following steps: S41. When the cloud starts up, it determines the capacity of the deletion recipe based on the number of encrypted data blocks uploaded for the first time. The deletion recipe stores the block IDs corresponding to the data blocks to be deleted. The length of the block IDs is 5N, where N represents the product of the average number of blocks and the length of the block ID. For each subsequent upload, the cloud calculates the average number of uploaded blocks. S42. After the client processes the file block, it determines the length of the deletion formula. If it exceeds 80% of the capacity, the cloud update will perform a deletion operation, upload the deletion formula in plain text, and update the total length of the deletion formula based on the length 5N. If it does not exceed, the ciphertext is uploaded and the cloud does not perform the deletion operation.

Images