Data deduplication method and system based on storage system

Abstract

The invention discloses a data deduplication method based on a storage system. The data deduplication method comprises the following steps: S1, dividing data in the storage system into a plurality of data blocks; s2, calculating a unique data fingerprint for each data block; s3, constructing a fingerprint index structure, storing a fingerprint set of existing data blocks, searching each newly generated fingerprint Fi in the fingerprint index structure, if Fi exists in the fingerprint set, indicating that the corresponding data block is repeated, and skipping to step S4; if Fi does not exist in the fingerprint set, indicating that the corresponding data block is a new data block, and adding Fi into the fingerprint set; and S4, retaining a data block copy, and marking the position of the duplicated data block in the index. According to the scheme, the repeated data are identified and deleted, so that the storage space is saved, and the utilization rate of a storage system is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer data storage, and in particular to a data deduplication method and system based on a storage system. Background Art

[0002] With the rapid development of information technology, storage systems face the challenge of high data duplication when processing large-scale data. This problem leads to waste of storage space and degradation of system performance, affecting the efficiency and performance of storage systems. Traditional data deduplication methods have problems such as low efficiency and high computational complexity, and cannot effectively address the challenges brought by data duplication in storage systems. Summary of the invention

[0003] The present invention mainly solves the technical problems of low efficiency and high computational complexity in the prior art, and provides a data deduplication method and system based on a storage system with high computational efficiency, which can effectively improve the performance and efficiency of the storage system.

[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a data deduplication method based on a storage system, comprising the following steps:

[0005] S1, dividing the data in the storage system into several data blocks;

[0006] S2. Calculate a unique data fingerprint for each data block;

[0007] S3, build a fingerprint index structure to store the fingerprint set of existing data blocks, and for each newly generated fingerprint F i , search in the fingerprint index structure, if F i exists in the fingerprint set, it indicates that the corresponding data block is duplicated, and jump to step S4; if F i does not exist in the fingerprint set, it means that the corresponding data block is a new data block, and F i Add to the fingerprint collection;

[0008] S4: Keep a copy of the data block and mark the location of the duplicate data block in the index. The remaining duplicate data blocks will be deleted to save storage space.

[0009] Preferably, the step S1 is specifically as follows:

[0010] S101, initialization: calculate the initial hash value H0 of the first w bytes:

[0011]

[0012] In the formula, D is the original data, w is the window size, M is the modulus, and B is the base;

[0013] S102, sliding window calculation: for each position i, calculate the hash value H of the current position i :

[0014] H i =((H i-1 -D[i-1]·B w-1 )·B+D[i+w-1])mod M

[0015] Where, 1≤i≤Nw, N is the length of the original data D;

[0016] S103, determine the boundary: if H i If modT=0, a data block is divided at position i, and T is the threshold value. T can generally be set to 10.

[0017] This method can dynamically adjust the size of the data block so that similar data content is divided into the same data block, thereby improving the deduplication effect.

[0018] Preferably, in step S2, the data fingerprint is calculated using SHA-256 or MD5 hash function. The above two hash functions can ensure the uniqueness of the fingerprint while having a fast calculation speed.

[0019] Preferably, in step S3, the fingerprint index structure is a hash table or a Bloom filter.

[0020] Preferably, the data deduplication method based on the storage system further includes fingerprint updating and maintenance steps, specifically:

[0021] Regularly clean up expired or useless fingerprint information, and dynamically adjust the fingerprint index structure and storage strategy according to data changes. Specifically:

[0022] Regularly monitor and count the access frequency of data blocks; for data blocks with access frequency higher than the upper access threshold, store their fingerprints in the high priority storage area; for data blocks with access frequency lower than the lower access threshold, store their fingerprints in the low priority storage area; for data blocks exceeding the access period threshold, delete their fingerprints to save storage space; the upper access threshold, the lower access threshold and the access period threshold are all empirical parameters and are set manually;

[0023] When deduplication is performed, if the amount of data to be processed is less than or equal to the priority control threshold, the newly generated fingerprint F i The newly generated fingerprint F is compared with the fingerprint sets in the high priority storage area and the low priority storage area; if the amount of data to be processed is greater than the priority control threshold, the newly generated fingerprint F iOnly compare with the fingerprint set in the high-priority storage area, and store it in the high-priority temporary storage area if there is no duplication (if there is duplication, delete the data block as mentioned above and mark the duplicate position in the index); if no new data enters after all new data is processed, compare the fingerprints in the high-priority temporary storage area with the fingerprint set in the low-priority storage area one by one, and store the fingerprints in the high-priority storage area if there is no duplication; if there is duplication, delete the corresponding data block and mark the position of the duplicate data block in the index. This can improve the efficiency of data deduplication.

[0024] It also includes data block size and distribution change management: according to the changes in data block size and distribution, dynamically adjust the parameters of the hash table or Bloom filter, such as the size of the hash table, the bit array length of the Bloom filter, and the number of hash functions.

[0025] Ensure that the fingerprint index structure can efficiently handle the size and distribution of the current data blocks to avoid performance degradation due to changes in data characteristics.

[0026] Also includes data lifecycle management:

[0027] 1) Dynamically adjust the retention strategy of fingerprint information based on the life cycle characteristics of the data.

[0028] 2) For data with a short life cycle, clean up its fingerprint information in time to release storage space.

[0029] 3) For data with a long life cycle, update its fingerprint information regularly to ensure the accuracy and effectiveness of data deduplication.

[0030] A data deduplication system based on a storage system, running the data deduplication method based on a storage system as described above, comprises:

[0031] Data block division module: divides the data in the storage system into several data blocks;

[0032] Data fingerprint calculation module: calculates a unique data fingerprint for each data block;

[0033] Fingerprint matching module: builds a fingerprint index structure to store the fingerprint set of existing data blocks and compares each newly generated fingerprint F i , search in the fingerprint index structure, if F i exists in the fingerprint set, it indicates that the corresponding data block is repeated; if F i does not exist in the fingerprint set, it means that the corresponding data block is a new data block, and F i Add to the fingerprint collection;

[0034] Duplicate data processing module: For duplicate data blocks, a copy of the data block is retained and the location of the duplicate data block is marked in the index;

[0035] Fingerprint update and maintenance module: regularly clean up expired or useless fingerprint information, and dynamically adjust the fingerprint index structure and storage strategy according to data changes.

[0036] The substantial effect brought by the present invention is that through the steps of data block division, data fingerprint calculation, fingerprint comparison and duplicate data processing, duplicate data can be identified and deleted, thereby saving storage space and improving the utilization rate of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a data deduplication method based on a storage system of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0039] Embodiment: A data deduplication method based on a storage system in this embodiment, such as Figure 1 As shown, the following steps are included:

[0040] S1, dividing the data in the storage system into several data blocks;

[0041] S2. Calculate a unique data fingerprint for each data block;

[0042] S3, build a fingerprint index structure to store the fingerprint set of existing data blocks, and for each newly generated fingerprint F i , search in the fingerprint index structure, if F i exists in the fingerprint set, it indicates that the corresponding data block is duplicated, and jump to step S4; if F i does not exist in the fingerprint set, it means that the corresponding data block is a new data block, and F i Add to the fingerprint collection;

[0043] S4. Only one copy of the data block is retained, the remaining duplicate data blocks are deleted, and the location of the duplicate data blocks is marked in the index.

[0044] After searching and comparing the fingerprints of all data blocks, all duplicate data blocks are deleted, and the utilization rate of the entire storage system is effectively improved.

[0045] The whole system structure is as follows:

[0046] 1. Data block partitioning module

[0047] First, the data in the storage system is divided into multiple data blocks according to a fixed size or specific rules. The data blocks can be divided in bytes or bits to ensure that each data block is of the same size or complies with specific rules.

[0048] The fixed size partitioning method is as follows:

[0049] Fixed size partitioning: The data is partitioned according to a predetermined fixed size S bytes. For example, if the data block size S is set to 4KB, the original data D is partitioned into several 4KB data blocks.

[0050] D={B1,B2,…,B n}

[0051] Among them, B i Represents the i-th data block.

[0052] Content-based partitioning is partitioning based on specific rules of data content (such as content features or boundary symbols). This method can dynamically adjust the size of data blocks so that similar data content is partitioned into the same data block, thereby improving the deduplication effect.

[0053] The specific operations are as follows:

[0054] Assume that the original data is D and the length is N. Set the window size to w and select a modulus M and base B.

[0055] ①. Initialization: Calculate the initial hash value H0 of the first w bytes:

[0056]

[0057] ② Sliding window calculation: For each position i (from 1 to Nw), calculate the hash value H of the current position i :

[0058] H i =((H i-1 -D[i-1]·B w-1 )·B+D[i+w-1])mod M

[0059] ③. Determine the boundary: If H i If mod T=0, a data block is divided at position i, and T is a preset critical value.

[0060] For example, there is a piece of data D = [data in bytes], the window size w = 4, the modulus M = 101, the base B = 256, and the critical value T = 10.

[0061] ①. Calculate the initial hash value H0:

[0062] H0=(D[0]·256 3 +D[1]·256 2 +D[2]·256 1 +D[3]·256 0 )mod 101

[0063] ②、Sliding window calculates hash value H i :

[0064] H1=((H0-D[0]·256 3 )·256+D[4])mod 101

[0065] ③、Check condition H i mod 10=0, if the condition is met, the division will be performed.

[0066] 2. Data fingerprint calculation module

[0067] Calculate a unique data fingerprint for each data block, using a hash function such as SHA-256, MD5, etc. The fingerprint calculation process is as follows:

[0068] Select a suitable hash function H to ensure the calculation speed and uniqueness of the fingerprint.

[0069] For each data block B i Perform hash operation to generate the corresponding fingerprint F i :

[0070] F i =H(B i )

[0071] Among them, B i is the i-th data block, F i is the corresponding data fingerprint.

[0072] 3. Fingerprint matching module

[0073] The calculated data fingerprint F i With the existing fingerprint set {F′1,F′2,…,F′ m} to compare and determine whether there are duplicate data blocks. The specific comparison process is as follows:

[0074] (1) Build an efficient fingerprint index structure (such as a hash table or Bloom filter) to store and retrieve existing fingerprints.

[0075] (2) For each newly generated fingerprint F i , search in the fingerprint index structure:

[0076] If F i exists in the fingerprint set, it means that data block B iis repetitive;

[0077] If F i does not exist in the fingerprint set, it means that data block B i is a new data block, and F i Added to the fingerprint collection.

[0078] 4. Duplicate data processing

[0079] For the detected duplicate data block B i , the following treatment methods can be adopted:

[0080] Keep a copy of the data: Only keep one copy of the data block and mark the location of the duplicate data block in the index. The location and fingerprint of each data block are recorded through the index table I:

[0081] I={(F i ,P i )}

[0082] Among them, P i Indicates data block B i The location in the storage system.

[0083] 5. Fingerprint update and maintenance

[0084] In order to ensure the continuous and efficient operation of the data deduplication process, the fingerprint index structure needs to be regularly updated and maintained, including:

[0085] Regular cleanup: Clean up expired or useless fingerprint information to prevent the fingerprint index structure from over-expanding and affecting search efficiency.

[0086] Dynamic adjustment: Dynamically adjust the fingerprint index structure and storage strategy according to data changes to adapt to different storage requirements and workloads.

[0087] The specific process is as follows:

[0088] Regularly monitor and count the access frequency of data blocks; for data blocks with access frequency higher than the upper access threshold, store their fingerprints in the high priority storage area; for data blocks with access frequency lower than the lower access threshold, store their fingerprints in the low priority storage area; for data blocks exceeding the access period threshold, delete their fingerprints to save storage space; the upper access threshold, the lower access threshold and the access period threshold are all empirical parameters and are set manually;

[0089] When deduplication is performed, if the amount of data to be processed is less than or equal to the priority control threshold, the newly generated fingerprint F i The newly generated fingerprint F is compared with the fingerprint sets in the high priority storage area and the low priority storage area; if the amount of data to be processed is greater than the priority control threshold, the newly generated fingerprint Fi Only compare with the fingerprint set in the high-priority storage area, and store it in the high-priority temporary storage area if there is no duplication (if there is duplication, delete the data block as mentioned above and mark the duplicate position in the index); if no new data enters after all new data is processed, compare the fingerprints in the high-priority temporary storage area with the fingerprint set in the low-priority storage area one by one, and store the fingerprints in the high-priority storage area if there is no duplication; if there is duplication, delete the corresponding data block and mark the position of the duplicate data block in the index. This can improve the efficiency of data deduplication.

[0090] It also includes data block size and distribution change management: according to the changes in data block size and distribution, dynamically adjust the parameters of the hash table or Bloom filter, such as the size of the hash table, the bit array length of the Bloom filter, and the number of hash functions.

[0091] Ensure that the fingerprint index structure can efficiently handle the size and distribution of the current data blocks to avoid performance degradation due to changes in data characteristics.

[0092] Also includes data lifecycle management:

[0093] 1) Dynamically adjust the retention strategy of fingerprint information based on the life cycle characteristics of the data.

[0094] 2) For data with a short life cycle, clean up its fingerprint information in time to release storage space.

[0095] 3) For data with a long life cycle, update its fingerprint information regularly to ensure the accuracy and effectiveness of data deduplication.

[0096] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0097] Although the terms such as data fingerprint and hash value are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A data deduplication method based on a storage system, characterized in that: The following steps are involved: S1, dividing the data in the storage system into several data blocks; S2. Calculate a unique data fingerprint for each data block; S3, build a fingerprint index structure to store the fingerprint set of existing data blocks, and for each newly generated fingerprint F i , search in the fingerprint index structure, if F i exists in the fingerprint set, it indicates that the corresponding data block is duplicated, and jump to step S4; if F i does not exist in the fingerprint set, it means that the corresponding data block is a new data block, and F i Add to the fingerprint collection; S4. Keep a copy of the data block and mark the location of the duplicate data block in the index.

2. A data deduplication method based on a storage system according to claim 1, characterized in that: The step S1 is specifically as follows: S101, initialization: calculate the initial hash value H0 of the first w bytes: In the formula, D is the original data, w is the window size, M is the modulus, and B is the base; S102, sliding window calculation: for each position i, calculate the hash value H of the current position i : H i =((H i-1 -D[i-1]·B w-1 )·B+D[i+w-1])mod M Where, 1≤i≤Nw, N is the length of the original data D; S103, determine the boundary: if H i If modT=0, a data block is divided at position i, and T is the critical value.

3. The data deduplication method based on a storage system according to claim 1, characterized in that: In step S2, the data fingerprint is calculated using the SHA-256 or MD5 hash function.

4. The data deduplication method based on a storage system according to claim 1, characterized in that: In step S3, the fingerprint index structure is a hash table or a Bloom filter.

5. The data deduplication method based on a storage system according to claim 1, characterized in that: It also includes fingerprint update and maintenance steps, specifically: Regularly clean up expired or useless fingerprint information, and dynamically adjust the fingerprint index structure and storage strategy according to data changes. Specifically: Regularly monitor and count the access frequency of data blocks; for data blocks with access frequency higher than the upper access threshold, store their fingerprints in a high-priority storage area; for data blocks with access frequency lower than the lower access threshold, store their fingerprints in a low-priority storage area; for data blocks that exceed the access period threshold, delete their fingerprints; When deduplication is performed, if the amount of data to be processed is less than or equal to the priority control threshold, the newly generated fingerprint F i The newly generated fingerprint F is compared with the fingerprint sets in the high priority storage area and the low priority storage area; if the amount of data to be processed is greater than the priority control threshold, the newly generated fingerprint F i Only compare with the fingerprint set in the high-priority storage area, and store it in the high-priority temporary storage area if there is no duplication. If no new data enters after all new data is processed, compare the fingerprints in the high-priority temporary storage area with the fingerprint set in the low-priority storage area one by one. If there is no duplication, store the fingerprint in the high-priority storage area. If there is duplication, delete the corresponding data block and mark the position of the duplicate data block in the index.

6. A data deduplication system based on a storage system, running the data deduplication method based on a storage system according to claim 1, characterized in that: include: Data block division module: divides the data in the storage system into several data blocks; Data fingerprint calculation module: calculates a unique data fingerprint for each data block; Fingerprint matching module: builds a fingerprint index structure to store the fingerprint set of existing data blocks and compares each newly generated fingerprint F i , search in the fingerprint index structure, if F i exists in the fingerprint set, it indicates that the corresponding data block is repeated; if F i does not exist in the fingerprint set, it means that the corresponding data block is a new data block, and F i Add to the fingerprint collection; Duplicate data processing module: For duplicate data blocks, a copy of the data block is retained and the location of the duplicate data block is marked in the index; Fingerprint update and maintenance module: regularly clean up expired or useless fingerprint information, and dynamically adjust the fingerprint index structure and storage strategy according to data changes.

Images