SM3-based cloud data deduplication method and system

Abstract

The invention discloses a cloud data deduplication method and system based on SM3. The method comprises the following steps that: a data uploader divides a file to be uploaded into blocks, performs SM3 hash calculation on each data block, and generates a short hash value in combination with a disturbance salt value; constructing a Merkel tree by using the short hash values of all the data blocks, and generating a file-level short hash value Hf; the uploader sends the file, the Hf and the identity of the uploader to a cloud service provider, and the cloud carries out re-judgment query on the Hf through an extensible dynamic bloom filter; if yes, recording the access authority of the uploader and avoiding repeated storage; and if not, storing the file, recording Hf and uploader information, and updating the Bloom filter. The system supports dynamic bitmap capacity expansion, the short hash collision resistance is improved in combination with a disturbance mechanism, the storage and query overhead is remarkably reduced, and the data deduplication efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing data storage, and relates to a cloud data deduplication method and system based on SM3. The method realizes efficient deduplication through SM3 short hash generation and dynamic Bloom filter, and is suitable for data deduplication scenarios in cloud services. Background Art

[0002] In cloud storage platforms, due to the massive amount of data generated, data deduplication technology has become a key storage optimization solution. Traditional deduplication methods typically use fixed hashing algorithms (such as SHA-256) to calculate hashes on data blocks before comparison. However, as data volumes continue to increase, traditional hashing methods are gradually experiencing performance and storage bottlenecks. Therefore, to improve deduplication efficiency and reduce storage costs, short hashing technology has become an effective solution.

[0003] First, from the perspective of hash security, while fixed hash algorithms (such as SHA-256) offer strong collision protection, storage and querying are the primary performance bottlenecks in actual cloud data deduplication applications. To reduce storage overhead and improve query efficiency, short hashing technology has become a viable solution. Short hashing significantly reduces hash value storage requirements by truncating a portion of the output of a traditional hash algorithm (for example, the first 64 or 128 bits of SHA-256). During deduplication operations, short hashing reduces the storage space and bandwidth required for each hash value comparison, thereby accelerating the deduplication process. Assuming that each data block's hash value is stored as 64 bits (8 bytes), compared to a 256-bit (32-byte) hash value, storage requirements are reduced by approximately 75%. This optimization effectively reduces storage space consumption and improves system scalability when processing large-scale data. While short hashing significantly improves deduplication efficiency, the reduced output space also significantly increases the probability of collisions. The output space of short hashes (such as 64-bit hashes) is significantly smaller than that of 256-bit hashes. According to the birthday problem theory, the probability of collisions is greatly increased. Therefore, while short hashes can improve computational efficiency and reduce storage requirements, they also introduce the potential risk of collisions. In certain demanding scenarios, collisions can lead to false positives or missed positives, thus affecting the accuracy of data deduplication. Therefore, there is an urgent need to improve short hash algorithms to enhance the security and robustness of the system.

[0004] Secondly, in terms of storage space scalability, traditional deduplication methods face the limitations of data structures such as Bloom filters. Bloom filters, a commonly used deduplication acceleration tool, are prone to "bit explosion" in big data scenarios, leading to a rapid expansion of storage space. This phenomenon not only increases storage costs but also increases the error rate, affecting the reliability of deduplication queries. Furthermore, Bloom filters are unable to dynamically adapt to changes in storage space when processing large amounts of data, which limits the scalability of deduplication systems and makes it difficult to adapt to the continued growth of data storage needs. With the rapid growth of data volumes, the bottlenecks of traditional methods in storage space and query performance have become increasingly significant, and a new mechanism is urgently needed to improve the storage efficiency and scalability of deduplication systems.

[0005] In summary, traditional deduplication methods face numerous challenges in terms of hash security and storage scalability. There is an urgent need for more secure and flexible algorithms and data structures to adapt to the challenges of large-scale cloud storage environments. Addressing these issues will not only improve the efficiency of data deduplication systems but also enhance their resilience against malicious attacks, thereby promoting the development of cloud storage technology.

[0006] For the convenience of describing the present invention, some related technologies related to the present invention are introduced below.

[0007] A Bloom filter is a probabilistic data structure with high space efficiency and fast query speed. It is used to quickly determine whether an element is likely to exist in a set. Its core consists of a bitmap array and several hash functions, which support the rapid determination of set membership.

[0008] The basic operating principle of a Bloom filter is as follows: when a new element is inserted, its index position is calculated using multiple hash functions, and the corresponding bitmap position is marked as 1. When querying the existence of an element, the system uses the same hash function to check whether all corresponding bits are 1. If so, the element is likely present; if any bit is 0, it is definitely absent. This structure does not retain the original element data, relying solely on bitmap markings, resulting in low storage costs and high query efficiency.

[0009] Bloom filters have two significant characteristics: one is that there will be no false negatives (i.e., they will not mistakenly judge existing elements as non-existent), and the other is that there may be false positives (misjudging non-existent elements as existing). The false positive rate p can be controlled by adjusting the size of the bit array and the number of hash functions, according to the formula Where m is the size of the bitmap array, q is the number of elements, and k is the number of hash functions. In engineering practice, 3-7 hash functions are usually selected to strike a balance between space and accuracy. Summary of the Invention

[0010] In response to the technical problems existing in the prior art, the purpose of the present invention is to provide a cloud data deduplication method and system based on SM3. By combining the domestic cryptographic algorithm SM3 with technologies such as Bloom filter, a cloud data deduplication mechanism with high security, low conflict rate and scalability is constructed, which is suitable for scenarios such as data centers, cloud platforms and distributed file systems.

[0011] The technical solutions of the present invention are as follows:

[0012] A cloud data deduplication method based on SM3 short hashing, comprising the following steps:

[0013] 1) The first uploader of file F generates a unique short hash value (Hf) for the file F to be uploaded. The first uploader uploads the file F and the short hash value Hf to the cloud service provider at the same time;

[0014] 2) The cloud service provider stores the file F and records the short hash value Hf, the uploader's information (such as uploader ID, file metadata, etc.), and the access rights to the file F;

[0015] 3) When a subsequent uploader attempts to upload file F′, the short hash value Hf′ of F′ is first calculated and sent to the cloud service provider;

[0016] 4) The cloud service provider queries the scalable dynamic Bloom filter to determine whether Hf′ already exists. If the query result is "hit", it means that F′ has been uploaded by other uploaders, and the subsequent uploader is granted access to the file F′. If the query result is "miss", the cloud system stores the complete file F′ and the short hash value Hf, and records its information as the first uploader of the file F′.

[0017] Furthermore, the method for generating a storage file includes: encoding the original file using a redundant code.

[0018] Furthermore, the short hash value Hf is generated by the following steps:

[0019] 1) Divide the file F into fixed-size blocks (e.g., m KB) and obtain a set of data blocks: F → {B1, B2, ..., B n}, each B i The size is m KB;

[0020] 2) For each data block B i Execute the SM3 algorithm to get the hash value H i =SM3(B i );

[0021] 3) For each data block B i , combined with H i Generate disturbance salt value S i;

[0022] 4) For each data block B i , according to H i and S i Generate short hash value Hf i ;

[0023] 5) The short hash values ​​of all data blocks [Hf1, Hf2, ..., Hf n ] as a leaf node, construct the root node R of the binary Merkel tree, and the short hash value of R is the short hash value Hf of F.

[0024] Furthermore, data block B is generated by the following steps: i The disturbance salt value S i :

[0025] 1) From H i Select 8 bytes as the perturbation offset seed where h j ∈[0,255] is the jth perturbation offset seed;

[0026] 2) Define the sampling segment length as l, the jth perturbation offset seed h j The calculation formula for the offset position is offset j =(h j +jc)mod(|B i |-l), where c is the perturbation constant introduced for perturbation enhancement (which can be set to a specific prime number, such as 17), and j is the index (j∈[0,7]);

[0027] 3) Read 64-byte fragments at each offset position, and splice a total of 8 64-byte fragments into the perturbation fragment: fragment i =B i [offset0:offset0+63]+…+B i [offset7:offset7+63];

[0028] 4) Disturbance fragment of splicing i Perform SM3 hashing operation and take the first 4 bytes as the disturbance salt value S i =SM3(fragment i )[0:4].

[0029] Furthermore, data block B is generated by the following steps: i The short hash value Hf i :

[0030] 1) From H i Select 8 bytes to form the main hash substring

[0031] H si =H i [3]||H i [7]||H i

[11] ||H i

[15] ||H i

[19] ||H i

[23] ||H i

[27] ||H i

[31] ; The sampling points of the selected 8 bytes are distributed as unequally as possible, and the data contribution of each part is as uniform as possible to counter the risk of local confrontation in the summary; the current selection of 8 bytes is based on: equidistant distribution, odd-even mixing, no use of consecutive bytes, and including the tail summary byte. Other selection methods can also be used. The principle is to obtain a short hash value with high discrimination, low collision rate, and security and unpredictability as much as possible; it can also be other than 8 bytes. The currently selected position can be an example.

[0032] 2) S i Repeat twice to form a 64-bit salt value S ext_i =S i ||S i ;

[0033] 3) Perform XOR to get the short hash value Furthermore, the short hash value of the root node R of the binary Merkel tree is constructed by the following steps:

[0034] 1) For each short hash value Hf i After completing to 128 bits, we get I i =Hf i ||Hf i ;

[0035] 2) Will I i As a leaf node, Merkel construction is performed from the bottom up. The two child nodes I i and I j The parent node is N i,j =SM3(I i ||I j ), if the number of nodes is odd, you can copy the last node or insert a fixed "empty block to fill" to make up for it;

[0036] 3) Construct a 256-bit Merkel tree root node by using the short hash value of the 128-bit complement of all data blocks

[0037] R 256 =SM3(…(SM3(I1||I2)||SM3(I3||I4))…)∈{0,1} 256 ;

[0038] 4) From R 256 Extract 8 bytes for splicing:

[0039] R s =R[3]||R[7]||R

[11] ||R

[13] ||R

[17] ||R

[19] ||R

[23] ||R

[29] ; Similarly, the 8-byte positions selected here are the same as those above, and are also used to obtain a short hash value with high discrimination, low collision rate, and security and unpredictability. Of course, it can be more than 8 bytes, and the current selected positions are just an example.

[0040] 5) From R 256 Dynamically select the 8-byte disturbance offset value Offset j =R[j]+R[31-j]mod 32,j=

[0041] 0,1,…7;

[0042] 6) From R 256 Extract 8 bytes according to the disturbance offset value to form the disturbance fragment

[0043] S R =R[Offset0||Offset1||…||Offset7];

[0044] 7) The short hash value of the root node R is calculated to be Furthermore, the scalable dynamic Bloom filter is queried to determine whether the short hash value Hf exists through the following steps:

[0045] 1) Convert Hf to a 64-bit unsigned integer Where Hf[i] represents the i-th

[0046] bytes;

[0047] 2) Set K = 7, that is, the Bloom filter uses 7 hash functions, for each hash function h k (k=0,1,…,6), calculate the position of Hf in the bitmap array m total Is the total size of the current bitmap array: m total =m0+b num ×m1, where m0 is the initial bitmap size, b num is the current number of extended blocks, m1 is the extended bitmap size;

[0048] 3) For each position pos i (i=0,1,…,6), if pos i<m0, check if it exists in the base bitmap array L0, that is, whether L0[pos i is 1. If pos i ≥m0, check if it exists in the extended bitmap array L1, that is

[0049] whether it is 1;

[0050] 4) If all positions pos i (i = 0, 1,..., 6) already exist in L0 or L1, it is determined as "hit";

[0051] 5) For all positions pos i (i = 0, 1,..., 6), if any pos i does not exist in L0 or L1, it is determined as "missed", and the operation of writing data into the Bloom filter is performed.

[0052] Furthermore, a base bitmap array L0 with size m0 is constructed through the expected number of elements to be stored q (such as 1 million) and the target false positive rate p (such as p = 0.01). When the base bitmap array L0 is initialized, all bits are set to 0.

[0053] Furthermore, the extended bitmap array L1 consists of multiple bitmap blocks, and the size of each block m1 = 1.5m0.

[0054] Furthermore, data is written into the Bloom filter through the following steps:

[0055] 1) For all positions pos i (i = 0, 1,..., 6), if pos i <m0, set L0[pos i = 1. If pos i ≥m0, set

[0056]

[0057] 2) Count the number of newly added 1s △, and set the value of the set bit count m set to m set +△, where the initial value of m set is 0;

[0058] 3) Calculate the current filling rate of the bitmap array, that is, judge where τ = 0.75. If it holds, trigger the dynamic expansion of the bitmap array.

[0059] Furthermore, the dynamic expansion of the extended bitmap array L1 is performed through the following steps:

[0060] 1) In L1, add an extended bitmap block of size m1,

[0061] 2) Traverse L0 and randomly select 50% of the elements in L0 to be migrated. For each position pos to be migrated in L0, recalculate the hash to obtain the short hash value Hf(x) of the original element x at that position. Select the same hash function h used when inserting the original element x. k , calculate the position in the new extension block

[0062] Set L0[pos]=0, set

[0063]

[0064] 3) Current number of extension blocks b num Set to b num +1.

[0065] A cloud data deduplication system based on SM3, comprising:

[0066] The data uploader is used to divide the file F to be uploaded into multiple data blocks {B1, B2, ..., B n}, for each data block B i Execute the SM3 algorithm to generate a 256-bit hash value H i , from H i Select the first 8 bytes as the perturbation offset seed, calculate the fragment offset in the data block, splice multiple fragments and perform SM3 hash operation, and extract the first 4 bytes as the perturbation salt value S i , from H i Dynamically select 8 bytes to form the main hash substring H si , S i Expanded to 64-bit salt value, with H si XOR generates the data block short hash value Hf i , based on all Hf i Build a Merkel tree, generate a file-level short hash value Hf, and upload the file F and its short hash value Hf to the cloud service provider.

[0067] A cloud service provider is used to construct an extensible dynamic Bloom filter comprising a basic bitmap array (L0) and an extensible bitmap array (L1), uses seven hash functions to calculate the bitmap position of a file F, and determines whether the file is a duplicate based on the query result of the extensible dynamic Bloom filter; if the file is determined to be a duplicate, access rights are granted to subsequent uploaders to avoid duplicate storage; if the file is not a duplicate, the uploaded original file F and its short hash value Hf are stored, uploader information (such as user ID, file metadata) and access rights are recorded, and the Bloom filter is updated; when the Bloom filter fill rate exceeds a threshold (such as 75%), the bitmap array is automatically expanded and the data is migrated.

[0068] Compared with the prior art, the present invention has the following positive effects:

[0069] First, the present invention adopts SM3 short hash technology to significantly reduce the storage space of hash values ​​(only 64 bits are required). At the same time, the Merkel tree structure is used to ensure the efficiency of file integrity verification, and the dynamic Bloom filter is used to reduce query overhead, thereby improving deduplication efficiency. Secondly, the present invention enhances the anti-collision capability of short hashes through perturbation salt value and dynamic offset technology, thereby reducing the risk of misjudgment. Finally, the dynamic Bloom filter of the present invention supports on-demand expansion, avoiding the "bit explosion" problem of traditional Bloom filters, balancing storage space and misjudgment rate, and adapting to data growth. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a system model diagram.

[0071] Figure 2 It is the query and write flow chart of scalable dynamic Bloom filter. DETAILED DESCRIPTION

[0072] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0073] Figure 1 The system architecture diagram of the present invention is shown, which includes the following two main entities:

[0074] (1) Data uploader: responsible for the division of file data blocks, hash calculation, disturbance processing and short hash value generation.

[0075] (2) Cloud service provider: responsible for file storage, duplication detection, scalable dynamic Bloom filter management and permission distribution.

[0076] Specifically, the technical solution adopted by the present invention is as follows:

[0077] The entire deduplication process consists of four stages: file preprocessing and short hash generation, deduplication determination, storage and access rights management, and dynamic Bloom filter update and expansion. The main tasks of each stage are as follows:

[0078] In the file preprocessing and short hash generation phase, the data uploader first uses a redundant code to encode the original file into a storage file F, and then divides the file F into m equal data blocks. Then, the short hash value Hf of the file is calculated, and the uploader's identity, file F, and short hash value Hf are uploaded to the cloud service provider. Specifically:

[0079] 1) The data uploader divides the file F into blocks of fixed size (e.g., m KB) and obtains a set of data blocks:

[0080] F→{B1,B2,...,B n}, each data block B i The size is m KB;

[0081] 2) For each data block B i Execute the SM3 algorithm to get the hash value H i =SM3(B i );

[0082] 3) For each data block B i :From the hash value H i Select the first 8 bytes as the perturbation offset seed

[0083] where h j ∈[0,255] is the value of the jth byte; the length of the sampling segment is defined as l, and the calculation formula for the offset position of the jth hash byte is offset j =(h j +jc)mod(|B i |-l), where h j is the jth hash byte, c is the perturbation constant for introducing perturbation enhancement (can be set to a specific prime number, such as 17), and j is the byte index (j∈[0,7]); a 64-byte fragment is read at each offset position, and a total of 8 64-byte fragments are spliced ​​into a perturbation fragment: fragment i =B i [offset0:offset0+63]+…+B i [offset7:offset7+63]; perform SM3 hashing on the concatenated disturbance fragments and take the first 4 bytes as the disturbance salt value

[0084] S i =SM3(fragment i )[0:4];

[0085] 4) For each data block B i , from the hash value H i Select 8 bytes to form the main hash substring

[0086] H si =H i [3]||H i [7]||H i

[11] ||H i

[15] ||H i

[19] ||H i

[23] ||H i

[27] ||H i

[31] , the disturbance salt value S i Repeat twice to form a 64-bit salt value S ext_i =S i ||S i , perform XOR to get the short hash value

[0087] 5) For each short hash value Hf i After completing to 128 bits, we get I i =Hf i ||Hf i ;

[0088] 6) Will I i As a leaf node, Merkel construction is performed from the bottom up. The two child nodes I i and I j The parent node is

[0089] N i,j =SM3(I i ||I j ), if the number of nodes is odd, you can copy the last node or insert a fixed "empty block to fill" to make up for it;

[0090] 7) For all data blocks B i The short hash value I i , construct a 256-bit Merkel tree root node

[0091] R 256 =SM3(…(SM3(I1||I2)||SM3(I3||I4))…)∈{0,1} 256 ;

[0092] 8) From R 256 Extract a fixed 8 bytes for splicing:

[0093] R s=R[3]||R[7]||R

[11] ||R

[13] ||R

[17] ||R

[19] ||R

[23] ||R

[29] ;

[0094] 9) From R 256 Dynamically select the 8-byte disturbance offset value Offset j =R[j]+R[31-j]mod 32,j=

[0095] 0,1,…7;

[0096] 10) From R 256 Extract 8 bytes according to the disturbance offset value to form the disturbance fragment

[0097] S R =R[Offset0||Offset1||…||Offset7];

[0098] 11) The short hash value of the root node R is calculated to be Hf is the short hash value of file F;

[0099] 12) The data uploader uploads the uploader identity, file F, and short hash value Hf to the cloud service provider.

[0100] In the deduplication determination stage, Figure 2 As shown, the cloud service provider's cloud system uses an extensible dynamic Bloom filter for deduplication determination. The extensible dynamic Bloom filter consists of a basic bit group array L0 and an extended bitmap array L1. The basic bitmap array L0 of size m0 is constructed by the expected number of elements q (e.g., 1 million) and the target error rate p (e.g., p=0.01). When the basic bitmap array L0 is initialized, all bits are set to 0. The extended bitmap array L1 consists of multiple bitmap blocks, and the size of each block is m1 = 1.5m0. Specifically:

[0101] 1) Convert Hf to a 64-bit unsigned integer Where Hf[i] represents the i-th byte of Hf;

[0102] 2) Set K = 7, that is, the Bloom filter uses 7 hash functions, for each hash function h k (k=0,1,…,6), calculate the position of Hf in the bitmap array m total Is the total size of the current bitmap array: m total =m0+b num ×m1, where m0 is the initial bitmap size, b num is the current number of extended blocks, m1 is the extended bitmap size;

[0103] 3) For each position pos k (k = 0, 1, …, 6), if pos k < m0, check whether it exists in the base bitmap L0, that is, whether L0[pos k is 1. If pos k ≥ m0, check whether it exists in the extended bitmap L1, that is whether it is 1;

[0104] 4) If all pos k (k = 0, 1, …, 6) have existed in L0 or L1, it is determined as "hit";

[0105] 5) For all positions pos k (k = 0, 1, 6), if any pos k does not exist in L0 or L1, it is determined as "missed".

[0106] In the storage and access permission management phase, as Figure 2 shown, the cloud system of the cloud server provider performs corresponding storage and permission allocation according to the result of the deduplication determination. When the deduplication determination is "hit", the cloud system records the binding of the uploader identity and the file F and grants permissions; when the determination is "missed", it stores the file F, the short hash value Hf and the binding relationship, and grants permissions.

[0107] In the dynamic Bloom filter update and expansion phase, as shown in 2, when the determination is "missed", the cloud system of the cloud service provider writes to the dynamic Bloom filter and triggers the dynamic expansion of the bitmap array when capacity expansion is triggered. Specifically:

[0108] 1) For all positions pos k (k = 0, 1, …, 6), if pos k < m0, set L0[pos k = 1. If pos k ≥ m0, set

[0109] 2) Count the number of newly set 1s △, and set the value of the set bit count m set to m set + △, where the initial value of m set is 0;

[0110] 3) Calculate the current filling rate of the bitmap array, that is, judge where τ = 0.75. If it holds, trigger the dynamic expansion of the bitmap array.[[ID=s57]]

[0111] If the dynamic expansion of the bitmap array is triggered, then:

[0112] 1) In L1, add an extended bitmap block of size m1,

[0113] 2) Traverse L0 and randomly select 50% of the elements in L0 to be migrated. For each position pos to be migrated in L0, recalculate the hash to obtain the short hash value Hf(x) of the original element x at that position, and select the same hash function h used when inserting the original element x. k , calculate the position in the new extension block

[0114] Set L0[pos]=0, set

[0115]

[0116] 3) Current number of extension blocks b num Set to b num +1.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art should understand that any modifications, equivalent replacements or improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention, and the scope of protection shall be based on what is defined in the claims.

Claims

1. A cloud data deduplication method based on SM3, comprising the following steps: 1) The data uploader generates a unique short hash value Hf for the file F to be uploaded, and then uploads the file F and its short hash value Hf to the cloud service provider; 2) The cloud service provider determines whether the short hash value Hf already exists. If so, the cloud service provider grants the data uploader access to the file F. If the file F does not exist, the file F is stored, and the short hash value Hf, the information of the data uploader, and the access rights of the file F are recorded.

2. The method according to claim 1, characterized in that The method for generating the short hash value Hf is: 21) Divide the file F into fixed-size blocks and obtain a set of data blocks {B1, B2, ..., B n }; 22) For each data block B i Execute the SM3 algorithm to get the hash value H i =SM3(B i ); 23) For each data block B i , combined with its corresponding hash value H i Generate disturbance salt value S i ; 24) For each data block B i , according to the hash value H i and disturbance salt value S i Generate short hash value Hf i ; 25) The short hash value of each data block [Hf1, Hf2, ..., Hf n ] as a leaf node, construct the root node R of the binary Merkel tree, and use the short hash value of the root node R as the short hash value Hf.

3. The method according to claim 2, characterized in that Generate data block B i The disturbance salt value S i The method is: 31) From the hash value H i Select n bytes as the perturbation offset seed; where h j is the jth perturbation offset seed, j∈[0,n-1]; 32) Define the sampling segment length as l and calculate h j offset j =(h j +jc)mod(|B i |-l), c is the disturbance constant introduced for disturbance enhancement; 33) Read 64-byte fragments at each offset and splice them to get the perturbation fragment i ; 34) Perturbation fragment i Perform SM3 hashing operation and take the first 4 bytes as the disturbance salt value S i .

4. The method according to claim 2 or 3, characterized in that Generate data block B i The short hash value Hf i The method is: from H i Select n bytes from the main hash string H si ; S i Repeat twice to form a 64-bit salt value S ext_i =S i ||S i ; Perform XOR to get the short hash value 5. The method according to claim 4, characterized in that The method for calculating the short hash value of the root node R is: 51) For each short hash value Hf i After completing to 128 bits, the completed short hash value I is obtained i =Hf i ||Hf i ; 52) Each completed short hash value I i As a leaf node, construct a binary Merkel tree and obtain the root node R of the 256-bit binary Merkel tree. 256 ; 53) From the root node R 256 Extract n bytes and concatenate them to get R s ; 54) From the root node R 256 Dynamically select n bytes of disturbance offset value Offset j '=R[j]+R[31- j]mod32, j=0,1,…n-1; 55) According to the disturbance offset value Offset j 'From the root node R 256 Extract n bytes to form the disturbance segment S R ; 56) Calculate the short hash value of the root node R 6. The method according to claim 5, characterized in that n=8; main hash substring H si =H i [3]||H i [7]||H i [11]||H i [15]||H i [19]||H i [23]||H i [27]||H i [31]; R s =R[3]||R[7]||R[11]||R[13]||R[17]||R[19]||R[23]||R[29]。 7. The method according to claim 1, characterized in that The cloud service provider queries the scalable dynamic Bloom filter through steps 71) to 75) to determine whether the short hash value Hf already exists. The method is as follows: 71) Convert the short hash value Hf into an unsigned integer Wherein, Hf[i] represents the i-th byte of the short hash value Hf; 72) Assume that the Bloom filter uses K hash functions. For each hash function h k , calculate the position of the short hash value Hf in the bitmap array m total Is the total size of the current bitmap array: m total =m0+b num ×m1, where m0 is the initial bitmap size, b num is the current number of extension blocks, m1 is the extended bitmap size; 73) For each position pos k , if pos k < m0, check whether it exists in the base bitmap L0. If pos k ≥ m0, Check whether it exists in the extended bitmap array L1; 74) If all positions pos k If both exist in L0 or L1, it is considered "hit"; 75) For all positions pos k If any pos k If it does not exist in L0 or L1, it is determined as a "miss" and the short hash value Hf is written into the Bloom filter.

8. The method according to claim 7, characterized in that Construct a basic bitmap array L0 of size m0 through the expected number of stored elements q and the target error rate p. When the basic bitmap array L0 is initialized, all bits are set to 0; The extended bitmap array L1 is composed of a plurality of bitmap blocks, and the size of each bitmap block is m1=1.5m0.

9. The method according to claim 7, characterized in that The extended bitmap array L1 is dynamically expanded through steps 91) to 93): 91) In the extended bitmap array L1, a new extended bitmap block of size m1 is added. 92) Traverse the base bitmap array L0, randomly select 50% of the elements in the base bitmap array L0 to be migrated, and for each position pos to be migrated in L0, recalculate the short hash value Hf(x) of the original element x corresponding to the position pos, and select the same hash function h used when the original element x was inserted. k , calculate the position in the new extension block Set L0[pos]=0, set 93) Set the current extension block number b num Set to b num +1.

10. A cloud data deduplication system based on SM3, characterized in that: Including data uploaders and cloud service providers; The data uploader is used to generate a unique short hash value Hf for the file F to be uploaded, and then upload the file F and its short hash value Hf to the cloud service provider; The cloud service provider determines whether the short hash value Hf already exists, and if so, grants the data uploader access to the file F; If the file F does not exist, the file F is stored, and the short hash value Hf, the information of the data uploader, and the access rights of the file F are recorded.

Images