A source-side deduplication method and system supporting compression and encryption with variable-length block partitioning.

Abstract

This invention discloses a source-side deduplication method and system supporting compression and encryption with variable-length block segmentation. The method includes: compressing a new file while simultaneously calculating a window fingerprint, segmenting the file into blocks to obtain compressed data blocks; calculating a strong hash value for each compressed data block and using it as a deduplication fingerprint; comparing the deduplication fingerprint with a deduplication fingerprint database on a server for deduplication; generating a random key, encrypting the compressed data blocks with the random key, and uploading it to the server for storage; generating a deduplication key using the deduplication fingerprint and the compressed data blocks; encrypting the random key using the deduplication key and a user key respectively, and saving it in metadata for download; and deduplicating existing identical files or identical compressed data blocks. This invention combines variable-length block segmentation, compression, encryption, and deduplication to achieve the same deduplication effect as plaintext while ensuring user data security.

Description

Technical Field

[0001] This invention relates to the field of data security technology, specifically a source-side deduplication method and system that supports compression and encryption and uses variable-length blocks. Background Technology

[0002] In the context of rapid digital development, user data security and protection has become one of the most pressing core issues in the information field. Enterprises collect, process, and store massive amounts of user personal data and other sensitive content daily, making this data highly vulnerable to attackers. At the same time, this massive data volume brings extremely high storage pressure, and the industry urgently needs to introduce efficient data optimization technologies such as encryption, compression, and deduplication.

[0003] Currently, strict ciphertext decryption is theoretically impossible. However, given the need for data security and decryption, a proposal has been made to moderately relax security requirements, allowing the ciphertext to reveal information about its equality with the original text. This provides a theoretical basis for ciphertext decryption. The earliest proposed scheme is CE (Convergent Encryption), with the following algorithm:

[0004] e = Encrypt(hash(d), d);

[0005] Encrypt(key, d) is a symmetric encryption algorithm that encrypts data d using key as the secret key, and hash(d) is a hash function.

[0006] Since the ciphertext encrypted by different users using the original hash value is always the same, it lacks semantic security. Attackers can deduce some plaintext information by using certain patterns or statistical characteristics in the ciphertext. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a source-side deduplication method and system that supports compression and encryption through variable-length blocks. This method improves deduplication efficiency by employing variable-length blocks and compression while ensuring data security.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0009] The present invention provides a source-end deduplication method for variable-length block compression and encryption, comprising:

[0010] Calculate the strong hash value of the user-uploaded file and compare it with the deduplication fingerprint database. If the corresponding strong hash value exists in the deduplication fingerprint database, perform deduplication.

[0011] If the corresponding strong hash value does not exist in the fingerprint database, perform the following operations:

[0012] While compressing the file data, the window fingerprint is calculated, and the file is divided into blocks to obtain compressed data blocks;

[0013] Calculate the strong hash value of the compressed data block and use the calculated strong hash value as the deduplication fingerprint;

[0014] The duplicate fingerprint is compared with the duplicate fingerprint database in the server. If the duplicate fingerprint exists in the database, the duplicate fingerprint is deduplicated.

[0015] Generate a random key, use the random key to encrypt the compressed data block, and upload it to the server for storage;

[0016] A deduplication key is generated using deduplication fingerprints and compressed data blocks;

[0017] The random key is encrypted using the deduplication key and the user key, and then saved to the metadata for download.

[0018] A further improvement of the present invention is that the fingerprint is calculated using a 64-byte sliding window.

[0019] A further improvement of the present invention is that the deduplication process specifically includes:

[0020] Read the metadata, use the file's strong hash value or deleting fingerprint to decrypt the encrypted random key in the metadata, and obtain the random key;

[0021] The random key is encrypted using the user's key and saved to the metadata for download.

[0022] A further improvement of this invention is that the deduplication key for compressed data blocks is calculated as follows:

[0023] ddk=sha256(cid+fp);

[0024] Where: ddk is the deduplication key, sha is the secure hash algorithm, cid is the strong hash value of the compressed data block, and fp is the window fingerprint.

[0025] The present invention provides a variable-length block-based source-end deduplication system supporting compression and encryption, comprising:

[0026] The file deduplication judgment module is used to calculate the strong hash value of the user-uploaded file and compare it with the deduplication fingerprint database to determine whether to perform deduplication processing.

[0027] The data compression module is used to compress the file data while calculating the window fingerprint, dividing it into blocks, and obtaining compressed data blocks.

[0028] The data block deduplication judgment module is used to calculate the strong hash value of the compressed data block and use the calculated strong hash value as the deduplication fingerprint. The deduplication fingerprint is compared with the deduplication fingerprint database in the server to determine whether to perform deduplication processing.

[0029] The data encryption module is used to generate a random key, encrypt the compressed data block with the random key, and upload it to the server for storage.

[0030] The key encryption module is used to generate a deduplication key using the deduplication fingerprint and compressed data block, encrypt a random key using the deduplication key and the user key respectively, and save it to the metadata for download.

[0031] The deduplication module is used to perform deduplication when a strong hash value or deduplication fingerprint of a corresponding file exists in the deduplication fingerprint database.

[0032] A further improvement of the present invention is that the deduplication module performs the following operations:

[0033] Read the metadata, use the file's strong hash value or deleting fingerprint to decrypt the encrypted random key in the metadata, and obtain the random key;

[0034] The random key is encrypted using the user's key and saved to the metadata for download.

[0035] A further improvement of this invention is that the deduplication key for compressed data blocks is calculated as follows:

[0036] ddk=sha256(cid+fp);

[0037] Where: ddk is the deduplication key, sha is the secure hash algorithm, cid is the strong hash value of the compressed data block, and fp is the window fingerprint.

[0038] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described variable-length block source-end deduplication method that supports compression and encryption.

[0039] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned variable-length block-based source-end deduplication method supporting compression and encryption.

[0040] The beneficial effects of this invention are as follows: The data key (random key) in this invention is randomly generated, and there is no correlation between the encrypted data and the plaintext, satisfying semantic security. This prevents attackers from deducing the plaintext information through statistical characteristics. Compared to the existing method of directly dividing plaintext into blocks and then compressing the data, this invention uses a method of compressing while calculating window fingerprints, which can better constrain the size of the final data blocks. While ensuring data security, this invention combines variable-length blocks, compression, encryption, and deduplication, improving deduplication efficiency. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method in an embodiment of the present invention;

[0042] Figure 2 This is a flowchart of the file upload process in an embodiment of the present invention;

[0043] Figure 3 This is a flowchart of the file download process in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0045] like Figure 1 As shown in this embodiment, a source-side deduplication method supporting compression and encryption with variable-length blocks includes:

[0046] Calculate the strong hash value (Cid) of the file uploaded by the user and compare it with the deduplication fingerprint database. If the strong hash value of the corresponding file exists in the deduplication fingerprint database, perform deduplication.

[0047] If the strong hash value of the corresponding file does not exist in the deleting fingerprint database, perform the following operations:

[0048] While compressing the file data, the window fingerprint is calculated, and the file is divided into blocks to obtain compressed data blocks.

[0049] Calculate the strong hash value of the compressed data block and use the calculated strong hash value as the deduplication fingerprint. Compare the deduplication fingerprint with the deduplication fingerprint database in the server. If a corresponding deduplication fingerprint exists in the database, perform deduplication processing.

[0050] Generate a random key, use the random key to encrypt the compressed data block, and upload it to the server for storage.

[0051] A deduplication key is generated using the deduplication fingerprint and compressed data block. The deduplication key and the user key are used to encrypt a random key and save it to the metadata for download.

[0052] In this embodiment, when other users upload the same file, the random key ciphertext (i.e., the encrypted random key) is extracted from the metadata stored on the server. The deduplication key is calculated locally based on the file content. The random key is decrypted using the deduplication key. The random key is then encrypted using the user key and saved to the metadata for download. At this time, there is no need to actually save the data, thus achieving deduplication.

[0053] like Figure 2 As shown, in this embodiment, when there is no strong hash value for the corresponding file in the deduplication fingerprint database, the specific process of file upload includes:

[0054] Step 1.1: Compress the data on the client side. When the compressed data size reaches the minimum block size, proceed to the next step.

[0055] Step 1.2: Use a 64-byte sliding window to calculate the window fingerprint fp sequentially until the fingerprint value meets the boundary conditions or the sliding window slides beyond the maximum length;

[0056] Step 1.3: Using the current sliding window as the end position, compress the data to obtain a compressed data block.

[0057] Step 1.4: Calculate the strong hash value (hash value) cid=sha256(block) of the compressed data block, and calculate the deduplication key ddk=sha256(cid+fp);

[0058] Step 1.5: Send the cid to the server and request to check if the data for the cid already exists. If it exists, return the metadata of the cid and jump to step 1.8. If it does not exist, it means that it is a new compressed data block and proceed to the next step.

[0059] Step 1.6: Generate a random key dek, use the random key dek to encrypt and compress the data block to obtain the eblock (encrypted block), and send the eblock to the server for storage;

[0060] Step 1.7: Encrypt the random key dek using the deduplication key to obtain ddek, and then encrypt the random key dek using the user key (the user's own key) uek to obtain udek. Proceed to step 1.9.

[0061] Step 1.8: Parse the received metadata, decrypt ddek in the metadata with the deduplication key to obtain the random key dek, and encrypt the random key dek with the user key uek to obtain udek;

[0062] Step 1.9: Send ddek and udek to the server for storage, then return to step 1.1 to continue processing the remaining data until all data has been processed.

[0063] Combination Figure 3 As shown, the file download process includes:

[0064] Step 2.1: The user requests to read the file metadata from the server through the client, and reads the list of compressed data blocks and the corresponding udek from it;

[0065] Step 2.2: Send a request to the server to read the encrypted block data eblock of the file in sequence;

[0066] Step 2.3: Decrypt eblock using udek to obtain compressed data block block, decompress compressed data block block to obtain original data block;

[0067] Step 2.4, repeat steps 2.2 to 2.3 until all compressed data blocks have been decrypted and decompressed;

[0068] Step 2.5: Merge all original data blocks and return the generated file to the user.

[0069] In this invention, the encrypted data stored on the server is stored using a random key, achieving semantic security. No features of the original text can be analyzed or statistically derived from the encrypted data, ensuring a perfect separation of permissions between the data owner and the storage owner, preventing tampering, unauthorized sharing, or attacks.

[0070] The above method is implemented based on a variable-length block-based source-side deduplication system supporting compression and encryption, as described in this embodiment. The system includes:

[0071] The file deduplication judgment module is used to calculate the strong hash value of the user-uploaded file and compare it with the deduplication fingerprint database to determine whether to perform deduplication processing.

[0072] The data compression module is used to compress the file data while calculating the window fingerprint, dividing it into blocks, and obtaining compressed data blocks.

[0073] The data block deduplication judgment module is used to calculate the strong hash value of the compressed data block and use the calculated strong hash value as the deduplication fingerprint. The deduplication fingerprint is compared with the deduplication fingerprint database in the server to determine whether to perform deduplication processing.

[0074] The data encryption module is used to generate a random key, encrypt the compressed data block with the random key, and upload it to the server for storage.

[0075] The key encryption module is used to generate a deduplication key using the deduplication fingerprint and compressed data block, encrypt a random key using the deduplication key and the user key respectively, and save it to the metadata for download.

[0076] The deduplication module is used to perform deduplication when a strong hash value or deduplication fingerprint of a corresponding file exists in the deduplication fingerprint database.

[0077] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0078] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A source-end deduplication method supporting compression and encryption with variable-length block partitioning, characterized in that: include: Calculate the strong hash value of the user-uploaded file and compare it with the deduplication fingerprint database. If the corresponding strong hash value exists in the deduplication fingerprint database, perform deduplication. If the corresponding strong hash value does not exist in the fingerprint database, perform the following operations: Step 1.1: Compress the data on the client side. When the compressed data size reaches the minimum block size, proceed to the next step. Step 1.2: Use a 64-byte sliding window to calculate the window fingerprint fp sequentially until the fingerprint value meets the boundary conditions or the sliding window slides beyond the maximum length; Step 1.3: Using the current sliding window as the end position, compress the data to obtain a compressed data block. Step 1.4: Calculate the strong hash value of the compressed data block block, cid = sha256(block), and calculate the deduplication key ddk = sha256(cid + fp), where: ddk is the deduplication key, sha is the secure hash algorithm, cid is the strong hash value of the compressed data block, and fp is the window fingerprint; Step 1.5: Send the cid to the server and request to check if the data for the cid already exists. If it exists, return the metadata of the cid and jump to step 1.

8. If it does not exist, it means that it is a new compressed data block and proceed to the next step. Step 1.6: Generate a random key dek, use the random key dek to encrypt and compress the data block to obtain eblock, and send eblock to the server for storage; Step 1.7: Encrypt the random key dek using the deduplication key to obtain ddek, then encrypt the random key dek using the user key uek to obtain udek, and jump to step 1.9; Step 1.8: Parse the received metadata, decrypt ddek in the metadata with the deduplication key to obtain the random key dek, and encrypt the random key dek with the user key uek to obtain udek; Step 1.9: Send ddek and udek to the server for storage, then return to Step 1.1 to continue processing the remaining data until all data has been processed. The specific steps for deduplication include: Read the metadata, use the file's strong hash value or deleting fingerprint to decrypt the encrypted random key in the metadata, and obtain the random key; The random key is encrypted using the user's key and saved to the metadata for download.

2. A source-end deduplication system supporting compression and encryption based on the method of claim 1, characterized in that: include: The file deduplication judgment module is used to calculate the strong hash value of the user-uploaded file and compare it with the deduplication fingerprint database to determine whether to perform deduplication processing. The data compression module is used to compress the file data while calculating the window fingerprint, dividing it into blocks, and obtaining compressed data blocks. The data block deduplication judgment module is used to calculate the strong hash value of the compressed data block and use the calculated strong hash value as the deduplication fingerprint. The deduplication fingerprint is compared with the deduplication fingerprint database in the server to determine whether to perform deduplication processing. The data encryption module is used to generate a random key, encrypt the compressed data block with the random key, and upload it to the server for storage. The key encryption module is used to generate a deduplication key using the deduplication fingerprint and compressed data block, encrypt a random key using the deduplication key and the user key respectively, and save it to the metadata for download. The deduplication module is used to perform deduplication when a strong hash value or deduplication fingerprint of a corresponding file exists in the deduplication fingerprint database. The deduplication module performs the following operations: Read the metadata, use the file's strong hash value or deleting fingerprint to decrypt the encrypted random key in the metadata, and obtain the random key; The random key is encrypted using the user's key and saved to the metadata for download.

3. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the source-end deduplication method with support for compression and encryption as described in claim 1.

4. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the source-end deduplication method with support for compression and encryption as described in claim 1.

Images