A data uploading method based on multi-scene safe data deduplication

Abstract

The application discloses a data uploading method based on multi-scene safe repeated data deletion, and comprises the following steps: S1, a data uploading model comprising a cloud storage provider (CSP), a user running client (C) and an index service (IS) is constructed; S2, multiple data scenes that need to be uploaded are defined, and data transmission modes under each scene are defined; S3, popularity checking and popularity over-processing are performed before data uploading, and whether a block is popular is determined; and S4, data uploading is performed according to whether the block is popular or not. In the application, information of unpopular data segments is not leaked, popular data segments are effectively identified, and repeated data deletion of block-level encrypted data is realized in a very safe mode.

Description

Technical Field

[0001] This invention relates to data transmission under deduplication, and in particular to a data upload method for secure deduplication in multiple scenarios. Background Technology

[0002] Cloud storage providers are constantly seeking technologies designed to reduce redundant data and maximize space savings. We focus on deduplication, one of the most popular technologies and already adopted by many major providers.

[0003] The underlying principle of deduplication is to store duplicate data only once. Thanks to this mechanism, space savings can reach 70%, and even more in backup applications. On the other hand, while achieving low cost, users also need to ensure data confidentiality through encryption. However, deduplication and encryption are two conflicting technologies, and it is currently difficult to simultaneously encrypt and delete deduplicated data. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data upload method for secure deduplication based on multiple scenarios. Information about unwanted data segments is not leaked, and popular data segments are identified very effectively, achieving block-level encrypted data deduplication in a very secure manner.

[0005] The objective of this invention is achieved through the following technical solution: a data upload method for secure deduplication in multiple scenarios, comprising the following steps:

[0006] S1. Construct a data upload model that includes cloud storage provider CSP, user-run client C, and index service IS;

[0007] S2. Define the data scenarios that need to be uploaded, and the data transmission method for each scenario;

[0008] S3. Before uploading data, perform a popularity check and excessive popularity processing to determine if the block is popular;

[0009] S4. Upload data based on the popularity of the block.

[0010] The beneficial effects of this invention are: information about unwanted data segments is not leaked, and popular data segments are identified very effectively. Block-level deduplication of encrypted data is achieved in a very secure manner, saving more storage space compared to file-level deduplication. Attached Figure Description

[0011] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0012] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0013] like Figure 1 As shown, a data upload method based on secure deduplication in multiple scenarios includes the following steps:

[0014] S1. Construct a data upload model that includes cloud storage provider CSP, user-run client C, and index service IS;

[0015] The data upload in step S1 includes:

[0016] Users want to store their data on an untrusted cloud storage provider (CSP) while leveraging source-based block-level deduplication and protecting the confidentiality of their data; the cloud storage provider (CSP) is storing user data correctly while attempting to disclose its contents.

[0017] Before uploading data, C runs a secure lookup protocol to check if the data is popular. The cloud storage provider CSP is responsible for generating a perfect hash function (PHF) on the popular blocks and storing the resulting collision-free hash table.

[0018] A trusted third party called the Index Service (IS) is introduced to help client C discover the actual number of copies of an unpopular block; IS only stores information on unpopular blocks, and once a block becomes popular, all the corresponding information is removed from its database.

[0019] The cloud storage provider (CSP) stores a hash table of popular block IDs, constructed using a perfect hash function (PHF). Each element of the hash table is defined by (PHF(h(CE(bi)), h(CE(bi))), where h(CE(bi))) is the keyless secure hash value of the convergent cryptographic block. Before any operation, given the current set of popular blocks, the CSP creates a corresponding secure PHF, which is only updated when the CSP needs to store new popular blocks.

[0020] S2. Define the data scenarios that need to be uploaded, and the data transmission method for each scenario;

[0021] Step S2 includes:

[0022] Define the scenario of undesirable data upload as Scenario 1. In this scenario, if C finds that the data is not yet undesirable, it will upload the data to CSP and update IS.

[0023] Define the data upload scenario with excessive popularity as Scenario 2. In this scenario, if C finds that the popularity of the data is t−1, where t is the popularity threshold, then perform the operation of uploading the newly popular data, IS deletes all information related to the data, and CSP deletes all previously stored encrypted copies.

[0024] Define the popular data upload scenario as Scenario 3. In this scenario, C only uploads metadata. C has already detected that the requested data is popular, so it can perform duplicate data deletion.

[0025] S3. Before uploading data, perform a popularity check and excessive popularity processing to determine if the block is popular;

[0026] Step S3 includes a popularity check step, which includes:

[0027] S301. Before uploading file F, C divides file F into blocks F = {bi}, encrypts each block using convergent encryption CE, and calculates its id;

[0028] S302. Client C retrieves PHF from CSP and evaluates it on {h(CE(bi))}. The result of this operation is a set of indices I = {PHF(h(CE(bi)))}, where each index represents the position of a potentially popular block ID stored in the hash table of CSP.

[0029] S303. Perform popularity checks using the obtained index without revealing the block content to the CSP;

[0030] Given a set of indexes, the client retrieves the corresponding block ID stored in the hash table and compares it with its own block ID. Any block bi that makes h(CE(bi)) equal to the popular block ID retrieved from the CSP is considered popular and will therefore be deduplicated. The index will not show any usable information on the block.

[0031] Step S3 also includes a popularity over-processing step:

[0032] If the popularity check shows that a block is unpopular, C needs to check whether it will trigger a popularity shift:

[0033] Once a block is uploaded by t users, it will start to become popular. In order for C to be aware of the change in popularity and perform a transition, C will send an update to IS whenever the popularity check returns a negative result for a given block ID.

[0034] IS stores a list of block IDs and owners, which correspond to each encrypted copy of this unpopular block; when the number of data owners for a particular block reaches t, the popularity conversion protocol is triggered, and IS returns the list of block IDs to C;

[0035] To complete this transition phase, the CSP store converges the encrypted copy, removes the corresponding encrypted copy, and updates the PHF. From this point on, the block will be considered popular and will therefore be deduplicated.

[0036] This process is completely transparent to other users who are not welcome to upload the same block. During the upload phase, users also retain encrypted information about the aggregate encryption key, which allows users to decrypt the block when it becomes popular.

[0037] S4. Upload data based on the popularity of the block.

[0038] The content requested by the user client will vary depending on the state of the block:

[0039] If the block is unpopular, C will upload the block and encrypt it using a random key;

[0040] If the block is popular, C only uploads the block ID, so that CSP can update its data structure.

[0041] In embodiments of this application, to avoid managing the storage of encryption keys, C relies on the CSP to store random encryption keys and convergent encryption keys, both of which are encrypted using a secret key known only to the client.

[0042] In the embodiments of this application, we analyze the security of the proposed scheme, with the CSP considered the primary adversary. The CSP performs all operations correctly, but it may attempt to discover the original content, unwanted data. We do not consider scenarios where the CSP operates in a Byzantine-style manner. We assume the CSP cannot collude with IS because the component is trusted. Since the goal of a malicious CSP is to discover the content of unwanted blocks, we analyze in detail whether (and how) the confidentiality of unwanted data is guaranteed at all stages of the protocol. However, if a user wishes to maintain the confidentiality of a file even when it becomes popular, they might encrypt the file using a standard encryption solution and upload it to the cloud without following the protocol steps.

[0043] By definition, unwanted blocks are encrypted using semantically secure symmetric encryption. Therefore, the confidentiality of unwanted data segments is guaranteed due to the security of the underlying encryption mechanism.

[0044] Security during popularity checks. Information exchanged during popularity checks must not reveal any information that could expose the identity of an unpopular block held by a user. Due to the one-way nature of the secure PHF, the identity of unpopular blocks is protected: client-generated queries do not include the actual unpopular block ID, but rather an integer i calculated by evaluating the secure PHF on the block ID. Due to the one-way nature of the underlying secure hash function (SHA-3), simple guessing by exploring the results of the secure hash function embedded in the PHF is not feasible. Furthermore, when the PHF evaluates an unpopular block ID, there will inevitably be a collision between this unpopular block ID and a popular block ID stored on the CSP. These collisions serve as the primary countermeasure against the disclosure of unpopular block IDs sent to the CSP during the lookup process. By a reasonable assumption, we can also consider the output of the underlying secure hash function (SHA-3) to be random. In the case of a collision between an unpopular block ID and a popular block ID stored on the CSP, due to the randomness of the underlying secure hash function, the PHF output based on this hash function is uniformly distributed between 0 and m. In this conflict scenario, the probability that the CSP guesses the unwanted block ID used by the client as PHF input is therefore:

[0045]

[0046] in P It is a collection of popular block IDs stored in the CSP. P ¯ is the remainder of the block ID field, including all possible non-popular block IDs, and α is the load factor of the PHF, such that .

[0047] Assuming the cardinality of the entire domain is much larger than the cardinality of the popular block ID set (which would happen if the popular block IDs were the result of a secure hash function), we can declare that the number of collisions per index is large enough to prevent malicious CSPs from inferring the actual block IDs used as PHF inputs. Collisions can effectively hide the identity of unwanted blocks from untrusted cloud providers while keeping the lookup protocol highly efficient and lightweight for users;

[0048] Security against potential protocol vulnerabilities. Now consider some other attacks that might be implemented by the CSP. For each attack, we propose simple but effective countermeasures that are easy to implement and do not significantly increase computational and network overhead. First, we believe that the CSP might pre-build a PHF based on some specific data that has not yet been uploaded by users (e.g., derived from a dictionary). In this case, clients would detect that the block they are requesting is popular, even though it has never been actually uploaded by any user; such a block would then be stored with a lower level of protection. As a countermeasure to this type of attack, we suggest that IS append a signature to each popular block ID during popularity transformation. Thus, IS would sign the popular block ID before it is stored by the CSP, allowing clients to verify the authenticity of these blocks when running popularity checks. Such a countermeasure would have minimal impact on system performance. Another attack we consider relates to file verification attacks, where convergence encryption is also vulnerable. In practice, during popularity checks, the CSP might compare the index sequence sent by the client with the sequence generated for a given popular file f. If the two sequences match, it is possible that the client is actually uploading f. To hide this information from the CSP, the client might add some random indexes to the index list sent during popularity checks. Because the index list contains this noise, the CSP's identification of the target file will be blocked. This countermeasure also prevents the CSP from running a "learn the remaining information" attack. Furthermore, the overhead of this countermeasure is negligible in terms of bandwidth and computation.

[0049] Users might force a surge in popularity by repeatedly uploading random or targeted blocks. As a countermeasure, a popularity threshold might be set. t = t + u ,in u This is the expected maximum number of malicious users. The threshold can be dynamically updated at any time during the system's lifecycle. In fact, this parameter is transparent to both users and CSPs, so IndexService can update it based on security needs. Users could also launch a DoS attack by deleting random blocks stored in the cloud. This could happen during popularity transitions: the client is asked to attach a list of block IDs, which may not be an actual encrypted copy of the block being uploaded. It is recommended that the index service sign the list of block IDs to be deleted so that the cloud can verify the authenticity of the request. This signing does not significantly increase overhead.

[0050] In summary, in the embodiments of this application, the solution allows for the reduction of storage size by deduplicating popular data, relies on a symmetric encryption algorithm, which is very effective even when dealing with large amounts of data; and implements deduplication at the block level, saving more storage space compared to file-level deduplication; and does not require any coordination or initialization between users; and does not incur any storage overhead for unwanted data blocks.

[0051] Although exemplary embodiments of the invention have been described for illustrative purposes, those skilled in the art will understand that various modifications, additions, and substitutions in form and detail may be made without departing from the scope and spirit of the invention disclosed in the appended claims, and all such modifications and substitutions should fall within the scope of protection of the appended claims. Furthermore, the various steps in the method claimed by this invention can be combined in any combination. Therefore, the description of the embodiments disclosed in this invention is not intended to limit the scope of the invention, but rather to describe the invention. Accordingly, the scope of the invention is not limited by the above embodiments, but is defined by the claims or their equivalents.

Claims

1. A data upload method for secure deduplication in multiple scenarios, characterized in that: Includes the following steps: S1. Construct a data upload model that includes cloud storage provider CSP, user-run client C, and index service IS; Users want to store their data on an untrusted cloud storage provider (CSP) while leveraging source-based block-level deduplication and protecting the confidentiality of their data; the cloud storage provider (CSP) is storing user data correctly while attempting to disclose its contents. Before uploading data, C runs a secure lookup protocol to check if the data is popular. The cloud storage provider CSP is responsible for generating a perfect hash function (PHF) on the popular blocks and storing the resulting collision-free hash table. A trusted third party called the Index Service IS was introduced to help client C discover the actual number of replicas of an unwanted block. IS only stores information on unpopular blocks; once a block becomes popular, all the corresponding information will be deleted from its database. S2. Define the data scenarios that need to be uploaded, and the data transmission method for each scenario; Define the scenario of undesirable data upload as Scenario 1. In this scenario, if C finds that the data is not yet undesirable, it will upload the data to CSP and update IS. Define the data upload scenario with excessive popularity as Scenario 2. In this scenario, if C finds that the popularity of the data is t−1, where t is the popularity threshold, then perform the operation of uploading the newly popular data, IS deletes all information related to the data, and CSP deletes all previously stored encrypted copies. Define the popular data upload scenario as Scenario 3. In this scenario, C only uploads metadata. C has already detected that the requested data is popular, so it can perform duplicate data deletion. S3. Before uploading data, perform a popularity check and excessive popularity processing to determine if the block is popular; S4. Upload data based on the popularity of the block; The content requested by the user client will vary depending on the state of the block: If the block is unpopular, C will upload the block and encrypt it using a random key; If the block is popular, C only uploads the block ID, so that CSP can update its data structure.

2. The data upload method based on multi-scenario secure deduplication according to claim 1, characterized in that: The cloud storage provider (CSP) stores a hash table of popular block IDs, which is constructed using a perfect hash function (PHF). Each element of the hash table is defined by (PHF(h(CE(bi))), h(CE(bi))), where h(CE(bi)) is the keyless secure hash value of the convergent cryptographic block. Before any operation, given the current set of popular blocks, the CSP creates a corresponding secure PHF, which is only updated when the CSP needs to store new popular blocks.

3. The data upload method for secure deduplication based on multiple scenarios according to claim 1, characterized in that: Step S3 includes a popularity check step, which includes: S301. Before uploading file F, C divides file F into blocks F = {bi}, encrypts each block using convergent encryption CE, and calculates its id; S302. Client C retrieves PHF from CSP and evaluates it on {h(CE(bi))}. The result of this operation is a set of indices I = {PHF(h(CE(bi)))}, where each index represents the position of a potentially popular block ID stored in the hash table of CSP. S303. Perform popularity checks using the obtained index without revealing the block content to the CSP; Given a set of indexes, the client retrieves the corresponding block ID stored in the hash table and compares it with its own block ID. Any block bi that makes h(CE(bi)) equal to the popular block ID retrieved from the CSP is considered popular and will therefore be deduplicated. The index will not show any usable information on the block.

4. The data upload method for secure deduplication based on multiple scenarios according to claim 1, characterized in that: Step S3 also includes a popularity over-processing step: If the popularity check shows that a block is unpopular, C needs to check whether it will trigger a popularity shift: Once a block is uploaded by t users, it will start to become popular. In order for C to be aware of the change in popularity and perform a transition, C will send an update to IS whenever the popularity check returns a negative result for a given block ID. IS stores a list of block IDs and owners, which correspond to each encrypted copy of this unclaimed block; When the number of data owners for a specific block reaches t, the popularity conversion protocol is triggered, and is returns the list of block IDs to C; To complete this transition phase, the CSP store converges the encrypted copy, removes the corresponding encrypted copy, and updates the PHF. From this point on, the block will be considered popular and will therefore be deduplicated. This process is completely transparent to other users who are not welcome to upload the same block. During the upload phase, users also retain encrypted information about the aggregate encryption key, which allows users to decrypt the block when it becomes popular.

5. The data upload method for secure deduplication based on multiple scenarios according to claim 1, characterized in that: To avoid managing the storage of encryption keys, C relies on a CSP to store both random and convergent encryption keys, both of which are encrypted using a secret key known only to the client.

Images