A data deduplication method and filter

Abstract

The present application provides a method and filter for deduplication, which includes the following steps: performing hash calculation on a data block to obtain fingerprint information; calculating a vector and a candidate bucket; querying whether the vector exists in a structure database; if not, inserting the vector into the candidate bucket; if so, querying whether the fingerprint information exists in a disk database; if so, deleting a copy of the vector; if not, inserting the vector into the candidate bucket. The present application uses the first empty slot index as an empty slot indicator position, which can quickly determine whether a slot is empty when inserting a vector, thereby improving insertion efficiency; the candidate bucket of the present application is associated with a slot array, a BitSet group, and a CountingSet group. When the amount of data to be processed gradually increases, a query is performed in the BitSet group or the CountingSet group, providing stable high query efficiency and significantly reducing memory overhead; and the present application timely expands the slot array by judging space utilization, thereby avoiding relocation operations and improving insertion performance.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and processing, and in particular to a method and filter for deleting duplicate data. Background Art

[0002] Currently, the total amount of data in the network is growing exponentially, which not only consumes more and more network bandwidth to transmit data, but also takes up a huge amount of data storage space. To reduce the total cost of ownership of computer data storage systems and computer networks, enterprises have begun to adopt data deduplication technology.

[0003] Data deduplication has become a mainstream and crucial technology in computer systems. It works by identifying duplicate data in a data stream, retaining only one copy of the duplicate data, and deleting the remaining redundant data. Pointers are used to reference the deduplicated data locations, thus saving significant amounts of data storage space and network bandwidth.

[0004] In order to meet the scalable performance requirements of massive data backup, it is crucial to use efficient data structures or algorithms in memory to accelerate index queries, avoid disk access, and achieve fast data deduplication.

[0005] However, existing technologies for accelerating index queries have many problems when working in deduplication systems: for example, they cannot tell whether a slot is empty when inserting a vector; the query efficiency is low and unstable; a large number of relocation operations lead to a decrease in insertion performance; and the memory overhead is too large. Summary of the Invention

[0006] The present invention provides a deduplication method and filter to solve the problems of existing accelerated index query technology working in a deduplication system, such as the inability of an insertion vector to distinguish whether a slot is empty, low query efficiency and instability, a large number of relocation operations leading to reduced insertion performance, and excessive memory overhead.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is to provide a method for deduplicating data, comprising the steps of: cutting a file object to be processed into multiple data blocks, performing hash calculation on each of the data blocks, and obtaining fingerprint information corresponding to the data block; calculating a vector of the fingerprint information to be queried and its corresponding candidate bucket; querying whether the vector exists in a structure database associated with the candidate bucket; if the vector does not exist in the structure database, inserting the vector into the corresponding candidate bucket; if the vector exists in the structure database, querying whether the fingerprint information corresponding to the vector exists in a disk database; if the fingerprint information exists in the disk database, deleting a copy of the vector of the fingerprint information; if the fingerprint information does not exist in the disk database, inserting the vector of the fingerprint information into the corresponding candidate bucket; wherein the structure database includes a slot array, a BitSet group, and a CountingSet group.

[0008] In some embodiments, the query whether the vector exists in the structure database associated with the candidate bucket includes the steps of: calculating the space utilization of the storage bucket based on the number of the storage bucket and the vectors stored therein; wherein the storage bucket includes multiple candidate buckets; if the space utilization is not greater than the first set threshold or the second set threshold, querying whether the vector exists in the slot array associated with the candidate bucket; if the space utilization is greater than the second set threshold and not greater than the third set threshold, constructing the BitSet group associated with the candidate bucket based on the slot array, and querying whether the vector exists in the corresponding BitSet group; if the space utilization is greater than the third set threshold, constructing the CountingSet group associated with the candidate bucket based on the slot array and the BitSet group, and querying whether the vector exists in the corresponding CountingSet group.

[0009] In some embodiments, the method of inserting the vector into the corresponding candidate bucket includes the following steps: the number of the candidate buckets is two, and one of the candidate buckets is selected as the target insertion bucket; if the BitSet group and the CountingSet group associated with the target insertion bucket are not constructed, the vector is inserted into the slot position indicated by the first empty slot index of the slot array associated with the target insertion bucket, and the first empty slot index is recalculated; if the BitSet group associated with the target insertion bucket has been constructed, and the value of the corresponding position of the BitSet group is 1, the vector is inserted into the slot position indicated by the first empty slot index, and the first empty slot index is recalculated; if the value of the corresponding position of the BitSet group is 0, the value of the corresponding position of the BitSet group is set to 1; if the CountingSet group associated with the target insertion bucket has been constructed, the value of the corresponding counter in the CountingSet group is increased by one.

[0010] In some embodiments, the method of selecting one of the candidate buckets as the target insertion bucket includes the following steps: if the two CountingSet groups respectively associated with the two candidate buckets have been constructed, then comparing the values ​​of the corresponding counters of the two CountingSet groups, and taking the candidate bucket of the CountingSet group with the smaller value of the corresponding counter as the target insertion bucket; if the CountingSet group corresponding to only one of the two candidate buckets has been constructed, then taking the candidate bucket of the constructed CountingSet group as the target insertion bucket; if the two CountingSet groups associated with the two candidate buckets have not been constructed, and the CountingSet groups associated with the two candidate buckets have not been constructed, then If the two BitSet groups associated with the candidate bucket have been constructed, the candidate bucket of the BitSet group with a value of 0 at the corresponding position is used as the target insertion bucket; if the two CountingSet groups associated with the two candidate buckets have not been constructed, and the BitSet group corresponding to only one candidate bucket has been constructed, the candidate bucket with the constructed BitSet group is used as the target insertion bucket; if the two CountingSet groups and the two BitSet groups associated with the two candidate buckets have not been constructed, the number of empty slots of the two slot arrays associated with the two candidate buckets is compared, and the candidate bucket of the slot array with the larger number of empty slots is used as the target insertion bucket.

[0011] In some embodiments, the method of inserting the vector into the corresponding candidate bucket also includes the steps of: under the premise that the two CountingSet groups and the two BitSet groups associated with the two candidate buckets are not constructed, and the number of empty slots in the two slot arrays associated with the two candidate buckets is 0: if the space utilization is less than the first set threshold, randomly selecting one of the candidate buckets, creating a new slot array for the selected candidate bucket, copying the contents of the original slot array of the selected candidate bucket to the corresponding position of the newly created slot array, and deleting the original slot array; wherein, the number of slots in the newly created slot array is twice the number of slots in the original slot array.

[0012] In some embodiments, the method of inserting the vector into the corresponding candidate bucket also includes the steps of: on the premise that the two CountingSet groups and the two BitSet groups associated with the two candidate buckets are not constructed, and the number of empty slots in the two slot arrays associated with the two candidate buckets is 0: if the space utilization is not less than the first set threshold and not greater than the second set threshold, then creating multiple corresponding slot arrays for the storage bucket, and the number of slots in each newly created slot array is equal to the number of slots in each of the initial slot arrays.

[0013] In some embodiments, the method for deleting a copy of the vector of the fingerprint information includes the following steps: if the BitSet group and the CountingSet group associated with the candidate bucket are not constructed, then the vector at the slot position indicated by the first non-empty slot index of the slot array associated with the candidate bucket is moved to the slot position where the copy of the vector is located, the copy of the vector is deleted, and the first non-empty slot index is recalculated; if the BitSet group associated with the candidate bucket has been constructed, then query whether there is a copy of the vector in the slot array associated with the candidate bucket; if so, delete the copy of the vector and set the value of the corresponding position of the BitSet group to 1; if not, set the value of the corresponding position of the BitSet group to 0; if the CountingSet group associated with the candidate bucket has been constructed, then reduce the value of the corresponding counter in the CountingSet group by one.

[0014] In some embodiments, the method for constructing the BitSet group associated with the candidate bucket includes the steps of: reading the vector in the slot array associated with the candidate bucket, assigning a position to the BitSet group, and determining whether the value of the corresponding position in the BitSet group is 1; if it is not 1, setting the value of the corresponding position in the BitSet group to 1, deleting the corresponding vector in the slot array, and continuing to read the remaining vectors in the slot array; if it is 1, continuing to read the remaining vectors in the slot array, and repeating the above steps until all vectors in the slot array are read.

[0015] In some embodiments, the method for constructing the CountingSet group associated with the candidate bucket includes the following steps: if the BitSet group associated with the candidate bucket has not been constructed, reading all vectors in the slot array associated with the candidate bucket, assigning a counter to the CountingSet group, adding one to the value of each corresponding counter in the CountingSet group according to the vector, and deleting the slot array; if the BitSet group associated with the candidate bucket has been constructed, reading the value of each position in the BitSet group, assigning a counter to the CountingSet group, if the value of the corresponding position in the BitSet group is 1, adding one to the value of the corresponding counter in the CountingSet group, then deleting the BitSet group, continuing to read all vectors in the slot array associated with the candidate bucket, adding one to the value of each corresponding counter in the CountingSet group according to the vector, and deleting the slot array.

[0016] The present invention also provides a filter, including a storage bucket, wherein the storage bucket includes multiple candidate buckets, the candidate buckets are used to store vectors, the candidate buckets have slot arrays, BitSet groups and CountingSet groups associated therewith, and the candidate buckets implement the above-mentioned deduplication method based on the slot arrays, BitSet groups and CountingSet groups.

[0017] The beneficial effects of the present invention are as follows: the present invention discloses a deduplication method and filter, the method comprising the steps of: cutting a file object to be processed into multiple data blocks, performing a hash calculation on each data block, and obtaining fingerprint information corresponding to the data block; calculating a vector of the fingerprint information to be queried and its corresponding candidate bucket; querying whether the vector exists in a structure database associated with the candidate bucket; if the vector does not exist in the structure database, inserting the vector into the corresponding candidate bucket; if the vector exists in the structure database, querying whether the fingerprint information corresponding to the vector exists in a disk database; if the fingerprint information exists in the disk database, deleting a copy of the fingerprint information vector; if the fingerprint information does not exist in the disk database, inserting the fingerprint information vector into the corresponding candidate bucket; wherein the structure database includes a slot array, a BitSet group, and a CountingSet group. The filter comprises a storage bucket, the storage bucket includes multiple candidate buckets, the candidate bucket is used to store the vector, the candidate bucket has an associated slot array, a BitSet group, and a CountingSet group, and the candidate bucket implements the above-mentioned deduplication method based on the slot array, the BitSet group, and the CountingSet group. The present application sets the first empty slot index as the empty slot indication position, so that when inserting a vector, it can quickly determine whether the slot is empty, thereby improving the efficiency of vector insertion; the candidate bucket in the present application is associated with three data structures (slot array, BitSet group and CountingSet group). When the amount of processed data gradually increases, the vector can be queried in the BitSet group or the CountingSet group, which provides stable high query efficiency without affecting the false positive rate, while significantly reducing memory overhead; and the present application expands the slot array in a timely manner by judging the space utilization, effectively avoiding relocation operations and improving insertion performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of the data deduplication method provided by this application;

[0019] Figure 2 It is a schematic diagram of the composition of the filter provided by this application;

[0020] Figure 3 This is a schematic diagram of the structure of the slot array in the deduplication method provided by this application;

[0021] Figure 4 This is a schematic diagram of the structure of the BitSet group in the deduplication method provided by this application;

[0022] Figure 5 This is a schematic diagram of the structure of the CountingSet group in the deduplication method provided by this application;

[0023] Figure 6 Schematic diagram of the first empty slot index in the data deduplication method provided by this application;

[0024] Figure 7 This is a schematic diagram of the insertion vector in the deduplication method provided by this application. Figure 1 ;

[0025] Figure 8 This is a schematic diagram of the insertion vector in the deduplication method provided by this application. Figure 2 ;

[0026] Figure 9 This is a schematic diagram of the insertion vector in the deduplication method provided by this application. Figure 3 ;

[0027] Figure 10 This is a schematic diagram of the insertion vector in the deduplication method provided by this application. Figure 4 ;

[0028] Figure 11 This is a schematic diagram of deleting vector copies in the deduplication method provided by this application. Figure 1 ;

[0029] Figure 12 This is a schematic diagram of deleting vector copies in the deduplication method provided by this application. Figure 2 ;

[0030] Figure 13 This is a schematic diagram of deleting vector copies in the deduplication method provided by this application. Figure 3 ;

[0031] Figure 14 This is a schematic diagram of deleting vector copies in the deduplication method provided by this application. Figure 4 ;

[0032] Figure 15 This is a schematic diagram of deleting vector copies in the deduplication method provided by this application. Figure 5 ;

[0033] Figure 16 This is a schematic diagram of deleting vector copies in the deduplication method provided by this application. Figure 6 ;

[0034] Figure 17 This is a schematic diagram of deleting vector copies in the deduplication method provided by this application. Figure 7 ;

[0035] Figure 18 This is a schematic diagram of deleting vector copies in the deduplication method provided by this application. Figure 8 . DETAILED DESCRIPTION

[0036] To facilitate understanding of the present invention, the present invention will be described in more detail below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present invention.

[0037] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification are intended solely for the purpose of describing specific embodiments and are not intended to limit the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0038] Figure 1 The following is a flowchart of the method for deduplication provided by the present application, including the following steps:

[0039] S1: Cut the file object to be processed into multiple data blocks, perform hash calculation on each data block, and obtain the fingerprint information corresponding to the data block.

[0040] S2: Calculate and obtain the vector of the fingerprint information to be queried and its corresponding candidate bucket.

[0041] S3: Query whether the vector exists in the structure database associated with the candidate bucket.

[0042] S4: If the vector does not exist in the structure database, insert the vector into the corresponding candidate bucket.

[0043] S5: If the vector exists in the structure database, query whether the fingerprint information corresponding to the vector exists in the disk database.

[0044] S6: If the fingerprint information exists in the disk database, the copy of the fingerprint information vector is deleted.

[0045] S7: If the fingerprint information does not exist in the disk database, the vector of the fingerprint information is inserted into the corresponding candidate bucket.

[0046] The structure database includes a slot array, a BitSet group, and a CountingSet group.

[0047] The present application sets the first empty slot index as the empty slot indication position, so that when inserting a vector, it can quickly determine whether the slot is empty, thereby improving the efficiency of vector insertion; the candidate bucket in the present application is associated with three data structures (slot array, BitSet group and CountingSet group). When the amount of processed data gradually increases, the vector can be queried in the BitSet group or the CountingSet group, which provides stable high query efficiency without affecting the false positive rate, while significantly reducing memory overhead; and the present application expands the slot array in a timely manner by judging the space utilization, effectively avoiding relocation operations and improving insertion performance.

[0048] The following combination Figures 1 to 17 , this application is described in detail with specific embodiments.

[0049] like Figure 1 As shown, the deduplication method provided in the embodiment of the present application is described in detail as follows:

[0050] S1: Cut the file object to be processed into multiple data blocks, perform hash calculation on each data block, and obtain the fingerprint information corresponding to the data block.

[0051] Hashing is a method for mapping data of arbitrary length to fixed-length data. It is commonly used in scenarios such as data integrity verification, password storage and comparison, data sharding, and load balancing. A hash function is the core algorithm for implementing hashing. It maps input data to a hash value or hash code, typically a fixed-length integer (such as 32 or 64 bits) in hexadecimal. A hash value, also known as a hash value, digest, or fingerprint, is the process of converting binary data of arbitrary length into a unique value of fixed length using a hash algorithm.

[0052] Common hash algorithms include MD5, SHA-1, and SHA-256. SHA-256 is currently the most widely used hash algorithm and is suitable for encrypting sensitive data. Hash algorithms have the following characteristics: the input can be a string of any length, and the output is a fixed length. However, the fixed-length output of a hash algorithm poses a collision problem: different input values ​​can produce the same output value, meaning that different data blocks can produce the same fingerprint information.

[0053] S2: Calculate and obtain the vector of the fingerprint information to be queried and its corresponding candidate bucket.

[0054] like Figure 2As shown, the filter of the present application includes a storage bucket 1, which includes multiple candidate buckets 2. Each candidate bucket 2 has a structure database associated with it (slot array 3, BitSet group 4 and CountingSet group 5). When querying fingerprint information, it is first necessary to calculate its corresponding vector, as well as the two candidate buckets 2 corresponding to the fingerprint information and their positions. Then, it is necessary to check whether the corresponding vector exists in the structure database associated with the two candidate buckets 2, so as to determine whether the data block to which the fingerprint information belongs is new data and further process it.

[0055] S3: Query whether the vector exists in the structure database associated with the candidate bucket.

[0056] In this embodiment, the structure database includes a slot array, a BitSet group, and a CountingSet group.

[0057] The structure of the slot array is as follows Figure 3 As shown, CF0 is the initial slot array, CF1 is the newly created slot array, each slot array includes multiple slots, each slot stores a corresponding vector, B i Represents one of the candidate buckets.

[0058] The structure of the BitSet group is as follows Figure 4 As shown, the BitSet group can accurately represent the existence status of all vectors, and the length of the BitSet group is 2 r , r represents the length of each vector. Each position in the BitSet corresponds to a vector (such as 000, 001, 010, and so on), and the value stored in each position can only be 0 or 1. 0 means the vector corresponding to that position does not exist, and 1 means it does. For example, if the value in the position corresponding to the vector 000 is 1, it means the vector 000 exists in the BitSet, while if the value in the position corresponding to the vector 111 is 0, it means the vector 111 does not exist in the BitSet. This allows us to check whether the corresponding vector of the fingerprint information exists in the BitSet.

[0059] The structure of the CountingSet group is as follows Figure 5 As shown, the CountingSet group is used to count the frequency of vector occurrences. Each CountingSet group includes multiple counters, each counter corresponds to a vector (such as 000, 001, 010, etc.); the length of the CountingSet group is 1×2 r, r represents the length of each vector, and 1 represents the length of each counter. The value of each counter in the CountingSet group represents the number of times its corresponding vector appears. For example, the counter corresponding to the vector 000 has a value of 1, indicating that the vector 000 appears once in the CountingSet group. The counter corresponding to the vector 011 has a value of 3, indicating that the vector 011 appears three times in the CountingSet group. The counter corresponding to the vector 111 has a value of 0, indicating that the vector 111 has not yet appeared in the CountingSet group. Based on the above information, you can query whether the corresponding vector of the fingerprint information exists in the CountingSet group.

[0060] Furthermore, the space utilization of the bucket is calculated based on the number of the bucket and the vectors stored therein.

[0061] Specifically, the space utilization is equal to the number of all vectors stored in the bucket divided by the number of slots allocated to the vectors and / or the number of positions in the BitSet group and / or the number of counters. The space utilization includes three preset thresholds: a first set threshold α, a second set threshold β, and a third set threshold γ, where the first set threshold α < the second set threshold β < the third set threshold γ.

[0062] Furthermore, if the space utilization is not greater than the first set threshold α or the second set threshold β, it is queried whether the vector exists in the slot array associated with the candidate bucket.

[0063] Combine Figure 3 And the above related content, if the query vector is 011, according to Figure 3 From the contents of the third slot, the fifth slot, and the seventh slot of the slot array CF0, it can be seen that the vector 011 exists in the slot array associated with the candidate bucket, so the query result is a false positive. A false positive means that the vector exists in the structure database associated with the candidate bucket, but may not exist in the disk database. Therefore, it is necessary to further search for the corresponding fingerprint information in the disk database; if the vector to be queried is 010, according to the content of the first slot of the slot array CF1, it can be seen that the vector 010 exists in the slot array associated with the candidate bucket, so the query result is a false positive; if the vector to be queried is 111, according to the contents of the slots of the two slot arrays CF0 and CF1, it can be seen that the vector 111 does not exist in the slot array associated with the candidate bucket, so the query result is negative.

[0064] Furthermore, if the space utilization is greater than the second set threshold β and not greater than the third set threshold γ, a BitSet group associated with the candidate bucket is constructed according to the slot array, and a query is made as to whether the vector exists in the corresponding BitSet group.

[0065] Specific, combined Figure 3 and Figure 4 , the method for constructing a BitSet group associated with a candidate bucket comprises the steps of:

[0066] like Figure 3 As shown, first read the vector in the slot array associated with the candidate bucket, assign a position to the BitSet group, and then determine whether the value of the corresponding position in the BitSet group is 1. If it is not 1 (that is, 0), set the value of the corresponding position in the BitSet group to 1, delete the corresponding vector in the slot array, and continue reading the remaining vectors in the slot array. If it is 1, continue reading the remaining vectors in the slot array and repeat the above steps until all vectors in the slot array are read.

[0067] For example, first read Figure 3 The vector 001 in the first slot of the CF0 slot array in the BitSet group has just been allocated, so the value in each position is 0 (that is, the initial state), so the corresponding position of the vector 001 in the BitSet group (that is, Figure 4 The value at the second position of the BitSet group in the is set to 1, and then Figure 3 Delete the vector 001 in the first slot of the CF0 slot array. Continue to read the vector 001 in the second slot of the CF0 slot array. At this time, the value of the corresponding position of the vector 001 in the BitSet group has been set to 1, so it is not processed; and continue to read the vector 011 in the third slot of the CF0 slot array. Set the corresponding position of the vector 011 in the BitSet group (i.e. Figure 4 The value at the fourth position of the BitSet group in the byte array is set to 1, and then Figure 3 The vector 011 in the third slot of the slot array CF0 is deleted. The vector 100 in the fourth slot of the slot array CF0 is read and processed, and so on until the last vector 101 in the slot array CF1 is read and processed. At this time, the BitSet group associated with the candidate bucket is completed, as shown in the following example. Figure 4 As shown in , the corresponding content of the slot array also changes.

[0068] Furthermore, it is queried whether the vector exists in the corresponding BitSet group that has been constructed.

[0069] Specifically, if the query vector is 011, according to Figure 4The value 1 in the fourth position of the BitSet group indicates that vector 011 exists in the BitSet group associated with the candidate bucket, so the query result is a false positive, and it is necessary to further search for the fingerprint information corresponding to vector 011 in the disk database; if the vector to be queried is 111, according to Figure 4 The value 0 in the eighth position of the BitSet group indicates that vector 111 does not exist in the BitSet group associated with the candidate bucket, so the query result is negative.

[0070] Furthermore, if the space utilization is greater than a third set threshold γ, a CountingSet group associated with the candidate bucket is constructed according to the slot array and the BitSet group, and a query is made as to whether the vector exists in the corresponding CountingSet group.

[0071] Specific, combined Figures 3 to 5 , the method for constructing a CountingSet group associated with a candidate bucket comprises the steps of:

[0072] If the BitSet group associated with the candidate bucket is not constructed, read all vectors in the slot array associated with the candidate bucket, assign counters to the CountingSet group, increase the value of each corresponding counter in the CountingSet group by one according to the vector, and delete the slot array.

[0073] For example, the BitSet group is not constructed, read Figure 3 For all vectors in the two slot arrays CF0 and CF1, the counters of the CountingSet group have just been assigned, and the value of each counter is 0. Then the value of the counter corresponding to each vector is increased by 1. If there are two identical vectors in the slot array (such as vector 001), the value of the counter is increased by 1 respectively. Finally, the value of the counter corresponding to vector 001 in the CountingSet group is 2. Figure 5 As shown in , the slot array is finally deleted, and the construction of the CountingSet group associated with the candidate bucket is completed.

[0074] If the BitSet group associated with the candidate bucket has been constructed, read the value of each position in the BitSet group, assign a counter to the CountingSet group, and if the value of the corresponding position in the BitSet group is 1, add one to the value of the corresponding counter in the CountingSet group, then delete the BitSet group, continue to read all vectors in the slot array associated with the candidate bucket, add one to the value of each corresponding counter in the CountingSet group according to the vector, and delete the slot array.

[0075] For example, the BitSet group has been constructed and read Figure 4The value of each position in the BitSet group is added by one to the vectors corresponding to the positions with a value of 1 (i.e., vectors 000, 001, 010, 011, 100, 101, 110) and the corresponding counters in the CountingSet group. Then, the BitSet group is deleted (e.g., Figure 5 Continue reading Figure 4 For all vectors of the slot array CF0 (i.e. vectors 011, 001, 011), add one to the corresponding counter values ​​of these three vectors in the CountingSet group, and finally delete the slot array (e.g. Figure 5 As shown), at this time, the construction of the CountingSet group associated with the candidate bucket is completed.

[0076] Furthermore, it is necessary to query whether the vector exists in the corresponding CountingSet group that has been constructed.

[0077] Specifically, if the query vector is 011, according to Figure 5 The value of the fourth counter of the CountingSet group is 3, which means that vector 011 exists in the CountingSet group associated with the candidate bucket. The query result is a false positive, and it is necessary to further search for the fingerprint information corresponding to vector 011 in the disk database. If the vector to be queried is 111, according to Figure 5 The value of the eighth counter of the CountingSet group is 0, which means that vector 111 does not exist in the CountingSet group associated with the candidate bucket, so the query result is negative.

[0078] S4: If the vector does not exist in the structure database, insert the vector into the corresponding candidate bucket.

[0079] According to the query results in S3, vectors that do not exist in the structure database (i.e., vectors with negative query results) are inserted into the corresponding candidate buckets.

[0080] There are two candidate buckets, one of which is selected as the target insertion bucket, and the vector is inserted into the target insertion bucket.

[0081] Specifically, if two CountingSet groups associated with two candidate buckets have been constructed, the values ​​of the corresponding counters of the vector to be inserted in the two CountingSet groups are compared, and the candidate bucket of the CountingSet group with the smaller value of the corresponding counter is used as the target insertion bucket, and the value of the corresponding counter in the CountingSet group with the smaller value is increased by one, that is, the vector to be inserted is inserted into the target insertion bucket.

[0082] If only one of the two candidate buckets has a corresponding CountingSet group already constructed, the candidate bucket with the constructed CountingSet group is inserted into the bucket as the target, and the value of the corresponding counter in the CountingSet group is increased by one.

[0083] If the two CountingSet groups associated with the two candidate buckets have not been constructed, and the two BitSet groups associated with the two candidate buckets have been constructed, the candidate bucket of the BitSet group whose value at the corresponding position of the vector to be inserted is 0 is used as the target insertion bucket, and the value at the corresponding position of the BitSet group is set to 1.

[0084] If the two CountingSet groups associated with the two candidate buckets are not built, and only one candidate bucket has a BitSet group built, the candidate bucket with the built BitSet group is used as the target bucket for insertion. If the value of the corresponding position of the vector to be inserted in the BitSet group is 1, the vector is inserted into the bucket with the same value. Figure 6 The target is inserted into the slot position indicated by the first empty slot index lastCelllndex of the slot array CF0 associated with bucket B1, and the first empty slot index lastCelllndex is recalculated. The slot position indicated by the first empty slot index lastCelllndex is the position of the first empty slot in the slot array from the left to the right. If the value of the corresponding position of the vector to be inserted into the BitSet group is 0, the value of the corresponding position of the BitSet group is set to 1.

[0085] If the two CountingSet groups and the two BitSet groups associated with the two candidate buckets are not constructed, the number of empty slots in the two slot arrays associated with the two candidate buckets is compared, and the candidate bucket with the slot array with the larger number of empty slots is inserted into the bucket as the target; then the vector is inserted into the bucket associated with Figure 6 The target shown is inserted into the slot position indicated by the first empty slot index lastCelllndex of the slot array CF0 associated with the bucket B1, and the first empty slot index is recalculated.

[0086] Under the premise that the two CountingSet groups and the two BitSet groups associated with the two candidate buckets are not constructed, and the number of empty slots in the two slot arrays associated with the two candidate buckets is 0:

[0087] like Figure 7 、 Figure 8As shown, if the space utilization is less than the first set threshold α, a candidate bucket is randomly selected, a new slot array is created for the selected candidate bucket, the contents of the original slot array of the selected candidate bucket are copied to the corresponding positions of the newly created slot array, and the original slot array is deleted; wherein the number of slots in the newly created slot array is twice the number of slots in the original slot array, Figure 7 is the original slot array, Figure 8 is a newly created slot array. The vector to be inserted is inserted into the first empty slot in the newly created slot array from left to right, where b6 is the vector to be inserted.

[0088] like Figure 9 、 Figure 10 As shown, if the space utilization is not less than the first set threshold α and not greater than the second set threshold β, multiple corresponding slot arrays are created for the bucket, with the number of slots in each newly created slot array equal to the number of slots in each original slot array. Here, CF0 is the original slot array for the bucket, and CF1 is the newly created slot array for the bucket. Each slot array in CF1 has four slots, which is equal to the number of slots in each original slot array. The vector to be inserted is then inserted into the newly created slot array, with f9 being the vector to be inserted.

[0089] S5: If the vector exists in the structure database, query whether the fingerprint information corresponding to the vector exists in the disk database.

[0090] According to S3, if the vector exists in the structure database, the query result is a false positive, and it is necessary to further query the corresponding fingerprint information in the disk database to determine whether the data block to which the fingerprint information belongs is new data; if the fingerprint information exists in the disk database, the data block to which it belongs is not new data; if the fingerprint information does not exist in the disk database, the data block to which it belongs is new data.

[0091] S6: If the fingerprint information exists in the disk database, the copy of the fingerprint information vector is deleted.

[0092] If the fingerprint information exists in the disk database, it means that the data block to which it belongs is not new data, so the copy of the vector corresponding to the fingerprint information needs to be deleted to save space.

[0093] Specifically, the method for deleting the copy of the fingerprint information vector includes the following steps:

[0094] like Figure 11 and Figure 12As shown, if the BitSet group and the CountingSet group associated with the candidate bucket are not constructed, the vector 101 at the slot position indicated by the first non-empty slot index of the slot array associated with the candidate bucket (that is, the previous one of the slot position indicated by the first empty slot index lastCelllndex) is moved to the slot position where the copy 110 of the vector is located, the copy 110 of the vector is deleted, and the first non-empty slot index is recalculated. After the processing is completed, Figure 12 shown.

[0095] like Figures 13 to 16 As shown in , if the BitSet group associated with the candidate bucket has been constructed, then query whether there is a copy of the vector in the slot array associated with the candidate bucket. Figure 13 As shown in , there is a copy of the vector such as 011, then delete the copy of the vector 011, and set the value of the corresponding position of the BitSet group to 1, and move the vector 111 at the slot position indicated by the first non-empty slot index in the slot array to the original position of the copy of the vector 011 (that is, the position of the third slot). After the processing is completed, Figure 14 As shown. Figure 15 If there is no copy of the vector in the slot array shown in , such as 101, the value of the corresponding position of the BitSet group is set to 0. After the processing is completed, Figure 16 shown.

[0096] like Figure 17 and Figure 18 As shown, if the CountingSet group associated with the candidate bucket has been constructed, combined with Figure 17 , assuming that the number of copies of the vector to be deleted is 110, then the value of the corresponding counter in the CountingSet group is reduced by one (that is, the value of the seventh counter in the CountingSet group is changed from 2 to 1). After the processing is completed, Figure 18 shown.

[0097] S7: If the fingerprint information does not exist in the disk database, the vector of the fingerprint information is inserted into the corresponding candidate bucket.

[0098] If the fingerprint information does not exist in the disk database, it means that the data block to which it belongs is new data. The vector corresponding to the fingerprint information needs to be inserted into the corresponding candidate bucket. The specific insertion method can be found in S4 and will not be repeated here.

[0099] In summary, the deduplication method of the present application sets the first empty slot index lastCelllndex as the empty slot indication position, so that when inserting a vector, it can quickly determine whether the slot is empty, thereby improving the efficiency of vector insertion; the candidate bucket in the present application is associated with three data structures (slot array, BitSet group and CountingSet group). When the amount of processed data gradually increases, the vector can be queried in the BitSet group or the CountingSet group, which provides stable high query efficiency without affecting the false positive rate, while significantly reducing memory overhead; and the present application expands the slot array in a timely manner by judging the space utilization, effectively avoiding relocation operations and improving insertion performance.

[0100] Corresponding to the data deduplication method of the above embodiment, the embodiment of the present application further provides a filter. For ease of illustration, only the part related to the embodiment of the present application is shown.

[0101] like Figure 2 As shown, the filter includes a storage bucket 1, which includes multiple candidate buckets 2. The candidate buckets 2 are used to store vectors. Each candidate bucket 2 has a slot array 3, a BitSet group 4, and a CountingSet group 5 associated with it. The candidate bucket 2 implements the above-mentioned deduplication method based on the slot array 3, the BitSet group 4, and the CountingSet group 5.

[0102] It should be noted that since the above-mentioned filter is based on the same concept as the embodiment of the method of the present application, its specific functions, the process of each component to realize its respective functions and the technical effects brought about can be specifically referred to the description of the embodiment shown in the aforementioned deduplication method, and will not be repeated here.

[0103] It can be seen that the present invention discloses a deduplication method and filter, which includes the following steps: cutting a file object to be processed into multiple data blocks, performing hash calculation on each data block, and obtaining fingerprint information corresponding to the data block; calculating a vector of the fingerprint information to be queried and its corresponding candidate bucket; querying whether the vector exists in a structure database associated with the candidate bucket; if the vector does not exist in the structure database, inserting the vector into the corresponding candidate bucket; if the vector exists in the structure database, querying whether the fingerprint information corresponding to the vector exists in a disk database; if the fingerprint information exists in the disk database, deleting a copy of the fingerprint information vector; if the fingerprint information does not exist in the disk database, inserting the fingerprint information vector into the corresponding candidate bucket; wherein the structure database includes a slot array, a BitSet group, and a CountingSet group. The filter includes a storage bucket, the storage bucket includes multiple candidate buckets, the candidate bucket is used to store the vector, the candidate bucket has an associated slot array, a BitSet group, and a CountingSet group, and the candidate bucket implements the above-mentioned deduplication method based on the slot array, the BitSet group, and the CountingSet group. The present application sets the first empty slot index as the empty slot indication position, so that when inserting a vector, it can quickly determine whether the slot is empty, thereby improving the efficiency of vector insertion; the candidate bucket in the present application is associated with three data structures (slot array, BitSet group and CountingSet group). When the amount of processed data gradually increases, the vector can be queried in the BitSet group or the CountingSet group, which provides stable high query efficiency without affecting the false positive rate, while significantly reducing memory overhead; and the present application expands the slot array in a timely manner by judging the space utilization, effectively avoiding relocation operations and improving insertion performance.

[0104] The above are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for deduplicating data, characterized in that: Including steps: Cut the file object to be processed into multiple data blocks, perform hash calculation on each data block, and obtain fingerprint information corresponding to the data block; Calculate and obtain the vector of the fingerprint information to be queried and its corresponding candidate bucket; Querying whether the vector exists in a structure database associated with the candidate bucket comprises the steps of: Calculating the space utilization of the storage bucket according to the number of the storage bucket and the vectors stored therein; wherein the storage bucket includes a plurality of the candidate buckets; If the space utilization is not greater than the first set threshold or the second set threshold, querying whether the vector exists in the slot array associated with the candidate bucket; If the space utilization is greater than the second set threshold and not greater than the third set threshold, constructing a BitSet group associated with the candidate bucket according to the slot array, and querying whether the vector exists in the corresponding BitSet group; If the space utilization is greater than the third set threshold, constructing a CountingSet group associated with the candidate bucket according to the slot array and the BitSet group, and querying whether the vector exists in the corresponding CountingSet group; If the vector does not exist in the structure database, inserting the vector into the corresponding candidate bucket includes the steps of: There are two candidate buckets, and one of the candidate buckets is selected as the target insertion bucket; If neither the BitSet group nor the CountingSet group associated with the target insertion bucket is constructed, inserting the vector into the slot position indicated by the first empty slot index of the slot array associated with the target insertion bucket, and recalculating the first empty slot index; If the BitSet group associated with the target insertion bucket has been constructed and the value of the corresponding position of the BitSet group is 1, inserting the vector into the slot position indicated by the first empty slot index and recalculating the first empty slot index; if the value of the corresponding position of the BitSet group is 0, setting the value of the corresponding position of the BitSet group to 1; If the CountingSet group associated with the target insertion bucket has been constructed, increment the value of the corresponding counter in the CountingSet group by one; If the vector exists in the structure database, query whether the fingerprint information corresponding to the vector exists in the disk database; If the fingerprint information exists in the disk database, the copy of the vector of the fingerprint information is deleted; if the fingerprint information does not exist in the disk database, the vector of the fingerprint information is inserted into the corresponding candidate bucket; The structure database includes a slot array, a BitSet group and a CountingSet group.

2. The data deduplication method according to claim 1, wherein: The method of selecting one of the candidate buckets as a target insertion bucket comprises the steps of: If the two CountingSet groups respectively associated with the two candidate buckets have been constructed, comparing the values ​​of the corresponding counters of the two CountingSet groups, and inserting the candidate bucket of the CountingSet group with the smaller value of the corresponding counter as the target bucket; If the CountingSet group corresponding to only one of the two candidate buckets has been constructed, the candidate bucket with the constructed CountingSet group is used as the target insertion bucket; If the two CountingSet groups associated with the two candidate buckets are not constructed, and the two BitSet groups associated with the two candidate buckets are constructed, the candidate bucket of the BitSet group with a value of 0 at the corresponding position is used as the target insertion bucket; If the two CountingSet groups associated with the two candidate buckets are not constructed, and the BitSet group corresponding to only one candidate bucket has been constructed, the candidate bucket with the constructed BitSet group is used as the target insertion bucket; If the two CountingSet groups and the two BitSet groups associated with the two candidate buckets are not constructed, the numbers of empty slots in the two slot arrays associated with the two candidate buckets are compared, and the candidate bucket of the slot array with the larger number of empty slots is inserted as the target bucket.

3. The data deduplication method according to claim 2, wherein: The method of inserting the vector into the corresponding candidate bucket further comprises the steps of: Under the premise that the two CountingSet groups and the two BitSet groups associated with the two candidate buckets are not constructed, and the number of empty slots in the two slot arrays associated with the two candidate buckets is 0: If the space utilization is less than the first set threshold, a candidate bucket is randomly selected, a new slot array is created for the selected candidate bucket, the contents of the original slot array of the selected candidate bucket are copied to the corresponding positions of the newly created slot array, and the original slot array is deleted; wherein the number of slots in the newly created slot array is twice the number of slots in the original slot array.

4. The data deduplication method according to claim 2, wherein: The method of inserting the vector into the corresponding candidate bucket further comprises the steps of: Under the premise that the two CountingSet groups and the two BitSet groups associated with the two candidate buckets are not constructed, and the number of empty slots in the two slot arrays associated with the two candidate buckets is 0: If the space utilization is not less than the first set threshold and not greater than the second set threshold, multiple corresponding slot arrays are newly created for the storage bucket, and the number of slots in each newly created slot array is equal to the number of slots in the initial slot array.

5. The data deduplication method according to claim 1, wherein: The method for deleting the copy of the vector of the fingerprint information comprises the steps of: If neither the BitSet group nor the CountingSet group associated with the candidate bucket is constructed, moving the vector at the slot position indicated by the first non-empty slot index of the slot array associated with the candidate bucket to the slot position where the copy of the vector is located, deleting the copy of the vector, and recalculating the first non-empty slot index; If the BitSet group associated with the candidate bucket has been constructed, querying whether there is a copy of the vector in the slot array associated with the candidate bucket; If it exists, delete the copy of the vector and set the value of the corresponding position of the BitSet group to 1; If it does not exist, the value of the corresponding position of the BitSet group is set to 0; If the CountingSet group associated with the candidate bucket has been constructed, the value of the corresponding counter in the CountingSet group is reduced by one.

6. The data deduplication method according to claim 1, wherein: The method for constructing the BitSet group associated with the candidate bucket comprises the steps of: Read the vector in the slot array associated with the candidate bucket, assign a position to the BitSet group, and determine whether the value of the corresponding position in the BitSet group is 1; If it is not 1, the value of the corresponding position in the BitSet group is set to 1, and the corresponding vector in the slot array is deleted, and the remaining vectors in the slot array are continued to be read; If it is 1, continue to read the remaining vectors in the slot array and repeat the above steps until all vectors in the slot array are read.

7. The data deduplication method according to claim 1, wherein: The method for constructing the CountingSet group associated with the candidate bucket comprises the steps of: If the BitSet group associated with the candidate bucket is not constructed, read all vectors in the slot array associated with the candidate bucket, assign a counter to the CountingSet group, increase the value of each corresponding counter in the CountingSet group by one according to the vector, and delete the slot array; If the BitSet group associated with the candidate bucket has been constructed, read the value of each position in the BitSet group, assign a counter to the CountingSet group, if the value of the corresponding position in the BitSet group is 1, add one to the value of the corresponding counter in the CountingSet group, then delete the BitSet group, continue to read all vectors in the slot array associated with the candidate bucket, add one to the value of each corresponding counter in the CountingSet group according to the vector, and delete the slot array.

8. A filter, characterized in that: The method includes a storage bucket, wherein the storage bucket includes multiple candidate buckets, the candidate buckets are used to store vectors, the candidate buckets have slot arrays, BitSet groups and CountingSet groups associated therewith, and the candidate buckets implement the deduplication method according to any one of claims 1 to 7 based on the slot arrays, BitSet groups and CountingSet groups.

Images