A data deduplication method and system based on a storage system

Abstract

The application discloses a data deduplication method based on a storage system, which comprises the following steps: S1, dividing data in the storage system into a plurality of data blocks; S2, calculating a unique data fingerprint for each data block; S3, constructing a fingerprint index structure, storing a fingerprint set of the existing data blocks, and adding each newly generated fingerprint F i to the fingerprint set; S4, retaining a data block copy and marking the position of the duplicate data block in the index. i If Fi exists in the fingerprint set, it indicates that the corresponding data block is duplicate, and the step S4 is jumped to; if Fi does not exist in the fingerprint set, it indicates that the corresponding data block is a new data block, and Fi is added to the fingerprint set; S4, retaining a data block copy and marking the position of the duplicate data block in the index. i The scheme realizes the identification and deletion of the duplicate data, thereby saving the storage space and improving the utilization rate of the storage system.

Description

Technical Field

[0001] This invention relates to the field of computer data storage, and in particular to a data deduplication method and system based on a storage system. Background Technology

[0002] With the rapid development of information technology, storage systems face the challenge of high data redundancy when processing large-scale data. This problem leads to wasted storage space and decreased system performance, affecting the efficiency and performance of storage systems. Traditional data deduplication methods suffer from inefficiency and high computational complexity, and cannot effectively address the challenges posed by data redundancy in storage systems. Summary of the Invention

[0003] This invention primarily addresses the technical problems of low efficiency and high computational complexity in existing technologies, providing a computationally efficient data deduplication method and system based on a storage system, which can effectively improve the performance and efficiency of the storage system.

[0004] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a data deduplication method based on a storage system, comprising the following steps:

[0005] S1. Divide the data in the storage system into several data blocks;

[0006] S2. Calculate a unique data fingerprint for each data block;

[0007] S3. Construct a fingerprint index structure to store the fingerprint set of existing data blocks, and for each newly generated fingerprint F i Search in the fingerprint index structure; if F i If it exists in the fingerprint set, it indicates that the corresponding data block is duplicated, and proceed to step S4; if F i If it does not exist in the fingerprint set, it indicates that the corresponding data block is a new data block, and F will be used. i Add to the fingerprint collection;

[0008] S4. Keep a copy of the data block and mark the location of the duplicate data block in the index. The remaining duplicate data blocks will be deleted, thus saving storage space.

[0009] Preferably, step S1 specifically includes:

[0010] S101, Initialization: Calculate the initial hash value H0 for the first w bytes:

[0011]

[0012] In the formula, D is the original data, w is the window size, M is the modulus, and B is the base.

[0013] S102, Sliding window calculation: For each position i, calculate the hash value H of the current position. i :

[0014] H i =((H) i-1 -D[i-1]·B w-1 )·B+D[i+w-1])mod M

[0015] In the formula, 1≤i≤Nw, and N is the length of the original data D;

[0016] S103. Determine the boundary: If H i If modT = 0, then a data block is created at position i, where T is the threshold value. T can generally be set to 10.

[0017] This method can dynamically adjust the size of data blocks, so that similar data content is divided into the same data blocks, thereby improving the deduplication effect.

[0018] Preferably, in step S2, the data fingerprint is calculated using either the SHA-256 or MD5 hash function. These two hash functions can ensure the uniqueness of the fingerprint while maintaining high computation speed.

[0019] Preferably, in step S3, the fingerprint index structure is a hash table or a Bloom filter.

[0020] As a preferred option, the data deduplication method based on the storage system also includes fingerprint update and maintenance steps, specifically:

[0021] Regularly clean up expired or useless fingerprint information, and dynamically adjust the fingerprint index structure and storage strategy according to data changes. Specifically:

[0022] Regularly monitor and analyze the access frequency of data blocks; for data blocks with an access frequency higher than the upper access threshold, store their fingerprints in the high-priority storage area; for data blocks with an access frequency lower than the lower access threshold, store their fingerprints in the low-priority storage area; for data blocks that exceed the access expiration threshold, delete their fingerprints to save storage space; the upper access threshold, lower access threshold, and access expiration threshold are all empirical parameters and are set manually.

[0023] During deduplication, if the amount of data to be processed is less than or equal to the priority control threshold, then the newly generated fingerprint F will be... i The fingerprints are compared with those in both the high-priority and low-priority storage areas; if the amount of data to be processed exceeds the priority control threshold, the newly generated fingerprint F is then... iThe fingerprints are compared only with the fingerprint set in the high-priority storage area. If no duplicates are found, the fingerprint is stored in the high-priority temporary storage area (if duplicates are found, the data block is deleted and the duplicate position is marked in the index). After all new data has been processed and no new data has entered, the fingerprints in the high-priority temporary storage area are compared one by one with the fingerprint set in the low-priority storage area. If no duplicates are found, the fingerprint is stored in the high-priority storage area; if duplicates are found, the corresponding data block is deleted and the position of the duplicate data block is marked in the index. This improves the efficiency of data deduplication.

[0024] In addition, it includes data block size and distribution change management: dynamically adjust the parameters of the hash table or Bloom filter based on changes in data block size and distribution, such as the size of the hash table, the bit array length of the Bloom filter, and the number of hash functions.

[0025] Ensure that the fingerprint index structure can efficiently handle the size and distribution of the current data blocks, avoiding performance degradation caused by changes in data characteristics.

[0026] It also includes data lifecycle management:

[0027] 1) Dynamically adjust the fingerprint information retention strategy based on the data lifecycle characteristics.

[0028] 2) For data with a short lifespan, clean up its fingerprint information in a timely manner to free up storage space.

[0029] 3) For data with a long lifecycle, update its fingerprint information regularly to ensure the accuracy and effectiveness of data deduplication.

[0030] A storage-based data deduplication system, running the storage-based data deduplication method described above, includes:

[0031] Data block partitioning module: Divides the data in the storage system into several data blocks;

[0032] Data fingerprint calculation module: Calculates a unique data fingerprint for each data block;

[0033] Fingerprint matching module: Constructs a fingerprint index structure, stores the fingerprint set of existing data blocks, and performs fingerprint matching for each newly generated fingerprint F. i Search in the fingerprint index structure; if F i If it exists in the fingerprint set, it indicates that the corresponding data block is duplicated; if F i If it does not exist in the fingerprint set, it indicates that the corresponding data block is a new data block, and F will be used. i Add to the fingerprint collection;

[0034] The deduplication module: For duplicate data blocks, retain a copy of the data block and mark the location of the duplicate data block in the index;

[0035] Fingerprint update and maintenance module: Regularly cleans up expired or useless fingerprint information and dynamically adjusts the fingerprint index structure and storage strategy according to data changes.

[0036] The substantial effect of this invention is that, through steps such as data block partitioning, data fingerprint calculation, fingerprint comparison, and duplicate data processing, it achieves the identification and deletion of duplicate data, thereby saving storage space and improving the utilization rate of the storage system. Attached Figure Description

[0037] Figure 1 This is a flowchart of a data deduplication method based on a storage system according to the present invention. Detailed Implementation

[0038] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0039] Example: This example illustrates a data deduplication method based on a storage system, such as... Figure 1 As shown, it includes the following steps:

[0040] S1. Divide the data in the storage system into several data blocks;

[0041] S2. Calculate a unique data fingerprint for each data block;

[0042] S3. Construct a fingerprint index structure to store the fingerprint set of existing data blocks, and for each newly generated fingerprint F i Search in the fingerprint index structure; if F i If it exists in the fingerprint set, it indicates that the corresponding data block is duplicated, and proceed to step S4; if F i If it does not exist in the fingerprint set, it indicates that the corresponding data block is a new data block, and F will be used. i Add to the fingerprint collection;

[0043] S4. Keep only one copy of the data block, delete the remaining duplicate data blocks, and mark the position of the duplicate data blocks in the index.

[0044] After searching and comparing the fingerprints of all data blocks, all duplicate data blocks were deleted, and the utilization rate of the entire storage system was effectively improved.

[0045] The overall system structure is as follows:

[0046] 1. Data block partitioning module

[0047] First, the data in the storage system is divided into multiple data blocks of fixed size or according to specific rules. The data blocks can be divided in units of bytes or bits, ensuring that each data block is the same size or conforms to specific rules.

[0048] The fixed-size division method is as follows:

[0049] Fixed-size partitioning: Data is divided into blocks of a predetermined fixed size S bytes. For example, if the data block size S is set to 4KB, the original data D is divided into several 4KB data blocks.

[0050] D = {B1, B2, ..., B} n}

[0051] Among them, B i This represents the i-th data block.

[0052] Content-based partitioning divides data based on specific rules governing the data content (such as content characteristics or boundary symbols). This method can dynamically adjust the size of data blocks, ensuring that similar data content is grouped into the same blocks, thereby improving deduplication efficiency.

[0053] The specific steps are as follows:

[0054] Assume the original data is D, with a length of N. Set the window size to w, and select a modulus M and a base B.

[0055] ① Initialization: Calculate the initial hash value H0 for the first w bytes:

[0056]

[0057] ② Sliding window calculation: For each position i (from 1 to Nw), calculate the hash value H of the current position. i :

[0058] H i =((H) i-1 -D[i-1]·B w-1 )·B+D[i+w-1])mod M

[0059] ③ Determine the boundary: If H i If mod T = 0, then a data block is created at position i, where T is a preset threshold value.

[0060] For example, there is a data segment D = [data in bytes], window size w = 4, modulus M = 101, cardinality B = 256, and critical value T = 10.

[0061] ① Calculate the initial hash value H0:

[0062] H0=(D[0]·256 3 +D[1]·256 2 +D[2]·256 1 +D[3]·256 0 )mod 101

[0063] ② Calculate the hash value H using a sliding window. i :

[0064] H1=((H0-D[0]·256 3 )·256+D[4])mod 101

[0065] ③ Inspection condition H i If mod 10 = 0, then partitioning is performed.

[0066] 2. Data fingerprint calculation module

[0067] A unique data fingerprint is calculated for each data block using a hash function such as SHA-256 or MD5. The fingerprint calculation process is as follows:

[0068] Choose a suitable hash function H to ensure computation speed and fingerprint uniqueness.

[0069] For each data block B i Perform a hash operation to generate the corresponding fingerprint F. i :

[0070] F i =H(B i )

[0071] Among them, B i For the i-th data block, F i This is the corresponding data fingerprint.

[0072] 3. Fingerprint comparison module

[0073] The calculated data fingerprint F i Compared with the existing fingerprint set {F′1,F′2,…,F′ m The comparison is performed to determine if duplicate data blocks exist. The specific comparison process is as follows:

[0074] (1) Construct an efficient fingerprint index structure (such as a hash table or Bloom filter) to store and retrieve existing fingerprints.

[0075] (2) For each newly generated fingerprint F i Search within the fingerprint index structure:

[0076] If F i If it exists in the fingerprint set, it indicates that data block B... iIt is a repetition;

[0077] If F i If it does not exist in the fingerprint set, it indicates that data block B... i It is a new data block, and will F i Add to the fingerprint collection.

[0078] 4. Duplicate Data Processing

[0079] For the detected duplicate data block B i The following methods can be adopted:

[0080] Maintain a single copy of the data: Only one copy of the data block is kept, and the location of duplicate data blocks is marked in the index. The location and fingerprint of each data block are recorded in index table I.

[0081] I = {(F i ,P i )}

[0082] Among them, P i Indicates data block B i Location in the storage system.

[0083] 5. Fingerprint update and maintenance

[0084] To ensure the continuous and efficient operation of the data deduplication process, the fingerprint index structure needs to be updated and maintained regularly. This includes:

[0085] Regular cleanup: Clean up expired or useless fingerprint information to prevent the fingerprint index structure from becoming too large and affecting search efficiency.

[0086] Dynamic adjustment: The fingerprint index structure and storage strategy are dynamically adjusted according to changes in data to adapt to different storage needs and workloads.

[0087] The specific process is as follows:

[0088] Regularly monitor and analyze the access frequency of data blocks; for data blocks with an access frequency higher than the upper access threshold, store their fingerprints in the high-priority storage area; for data blocks with an access frequency lower than the lower access threshold, store their fingerprints in the low-priority storage area; for data blocks that exceed the access expiration threshold, delete their fingerprints to save storage space; the upper access threshold, lower access threshold, and access expiration threshold are all empirical parameters and are set manually.

[0089] During deduplication, if the amount of data to be processed is less than or equal to the priority control threshold, then the newly generated fingerprint F will be... i The fingerprints are compared with those in both the high-priority and low-priority storage areas; if the amount of data to be processed exceeds the priority control threshold, the newly generated fingerprint F is then...i The fingerprints are compared only with the fingerprint set in the high-priority storage area. If no duplicates are found, the fingerprint is stored in the high-priority temporary storage area (if duplicates are found, the data block is deleted and the duplicate position is marked in the index). After all new data has been processed and no new data has entered, the fingerprints in the high-priority temporary storage area are compared one by one with the fingerprint set in the low-priority storage area. If no duplicates are found, the fingerprint is stored in the high-priority storage area; if duplicates are found, the corresponding data block is deleted and the position of the duplicate data block is marked in the index. This improves the efficiency of data deduplication.

[0090] In addition, it includes data block size and distribution change management: dynamically adjust the parameters of the hash table or Bloom filter based on changes in data block size and distribution, such as the size of the hash table, the bit array length of the Bloom filter, and the number of hash functions.

[0091] Ensure that the fingerprint index structure can efficiently handle the size and distribution of the current data blocks, avoiding performance degradation caused by changes in data characteristics.

[0092] It also includes data lifecycle management:

[0093] 1) Dynamically adjust the fingerprint information retention strategy based on the data lifecycle characteristics.

[0094] 2) For data with a short lifespan, clean up its fingerprint information in a timely manner to free up storage space.

[0095] 3) For data with a long lifecycle, update its fingerprint information regularly to ensure the accuracy and effectiveness of data deduplication.

[0096] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0097] Although this document uses terms such as data fingerprint and hash value frequently, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A data deduplication method based on a storage system, characterized in that, Includes the following steps: S1. Divide the data in the storage system into several data blocks; S2. Calculate a unique data fingerprint for each data block; S3. Construct a fingerprint index structure to store the fingerprint set of existing data blocks, and for each newly generated fingerprint F i Search in the fingerprint index structure; if F i If it exists in the fingerprint set, it indicates that the corresponding data block is duplicated, and we proceed to step S4; if F i If it does not exist in the fingerprint set, it indicates that the corresponding data block is a new data block, and F will be... i Add to the fingerprint collection; S4. Keep a copy of the data block and mark the location of the duplicate data block in the index; The data deduplication method based on the storage system also includes fingerprint update and maintenance steps, specifically: Regularly clean up expired or useless fingerprint information, and dynamically adjust the fingerprint index structure and storage strategy according to data changes. Specifically: Regularly monitor and analyze the access frequency of data blocks; for data blocks with an access frequency higher than the upper access threshold, store their fingerprints in the high-priority storage area; for data blocks with an access frequency lower than the lower access threshold, store their fingerprints in the low-priority storage area; for data blocks that have exceeded the access expiration threshold, delete their fingerprints. During deduplication, if the amount of data to be processed is less than or equal to the priority control threshold, the newly generated fingerprint F will be... i The fingerprints are compared with those in both the high-priority and low-priority storage areas; if the amount of data to be processed exceeds the priority control threshold, the newly generated fingerprint F is then... i The fingerprints are compared only with the fingerprint set in the high-priority storage area. If there are no duplicates, they are stored in the high-priority temporary storage area. After all new data has been processed and no new data has entered, the fingerprints in the high-priority temporary storage area are compared one by one with the fingerprint set in the low-priority storage area. If there are no duplicates, the fingerprints are stored in the high-priority storage area. If there are duplicates, the corresponding data block is deleted and the position of the duplicate data block is marked in the index.

2. The data deduplication method based on a storage system according to claim 1, characterized in that, Step S1 specifically involves: S101. Initialization: Calculate the initial hash value H0 for the first w bytes: In the formula, D is the original data, w is the window size, M is the modulus, and B is the base. S102, Sliding window calculation: For each position i, calculate the hash value H of the current position. i : H i =((H i-1 -D[i-1]·B w-1 )·B+D[i+w-1])mod M In the formula, 1≤i≤Nw, and N is the length of the original data D; S103. Determine the boundary: If H i If modT = 0, then a data block is created at position i, where T is the critical value.

3. The data deduplication method based on a storage system according to claim 1, characterized in that, In step S2, the data fingerprint is calculated using the SHA-256 or MD5 hash function.

4. The data deduplication method based on a storage system according to claim 1, characterized in that, In step S3, the fingerprint index structure is a hash table or a Bloom filter.

5. A data deduplication system based on a storage system, running the data deduplication method based on a storage system as described in claim 1, characterized in that, include: Data block partitioning module: Divides the data in the storage system into several data blocks; Data fingerprint calculation module: Calculates a unique data fingerprint for each data block; Fingerprint matching module: Constructs a fingerprint index structure, stores the fingerprint set of existing data blocks, and performs fingerprint matching for each newly generated fingerprint F. i Search in the fingerprint index structure; if F i If it exists in the fingerprint set, it indicates that the corresponding data block is duplicated; if F i If it does not exist in the fingerprint set, it indicates that the corresponding data block is a new data block, and F will be... i Add to the fingerprint collection; The deduplication module: For duplicate data blocks, retain a copy of the data block and mark the location of the duplicate data block in the index; Fingerprint update and maintenance module: Regularly cleans up expired or useless fingerprint information and dynamically adjusts the fingerprint index structure and storage strategy according to data changes.

Images