Method and device for identifying duplicated data based on hierarchical hash lookup

Abstract

The invention provides a method and device for identifying duplicated data based on hierarchical hash lookup, and the method comprises the steps: executing two independent hash operations in parallel for input data to obtain two lookup addresses, and constructing a lookup structure consisting of a plurality of levels of mutually independent hash tables and a residual table. The method comprises the following steps: firstly, carrying out double-hash search on data in a first-level hash table, and cascading the data to a next-level hash table if a conflict occurs until all hierarchical tables are accessed; and for the data which still conflict, sequentially searching in the residual table. According to the method, the probability of Hash conflicts is effectively reduced by adopting the multi-stage independent Hash table and a double-Hash lookup mechanism; more importantly, most of the search delays are limited in a predictable limited clock period by the hierarchical pipeline design, so that the search delay under the worst condition is obviously optimized, and an extremely high system throughput rate is realized in combination with multi-pass parallel.

Description

Method and apparatus for duplicate data identification based on hierarchical hash search Technical Field

[0001] This invention relates to the field of data storage and processing, and in particular to a method and apparatus for identifying duplicate data based on hierarchical hash lookup, which falls under the technical categories of data deduplication, cache management, or high-speed network storage. Background Technology

[0002] With the rapid development of cutting-edge applications such as large-scale artificial intelligence model training and massive data backup, the global data volume is experiencing explosive growth, resulting in significant data redundancy. This redundancy not only severely impacts the write performance of storage media but also leads to a huge waste of valuable storage resources. Therefore, there is an urgent need to design efficient data deduplication algorithms to accurately identify and eliminate redundant data, thereby reducing storage requirements and improving storage resource utilization. The efficiency of duplicate data identification directly affects the efficiency of deduplication.

[0003] In existing technologies, duplicate data identification commonly employs hash-based lookup methods. However, traditional single hash tables or open addressing methods suffer from high hash collision probabilities when processing massive amounts of data, leading to a sharp decline in lookup efficiency. To resolve collisions, commonly used optimized lookup structures such as chained structures or cuckoo hashes lengthen the lookup path, and in the worst-case scenario, the lookup latency becomes unpredictable, severely limiting the system's high-speed processing capabilities and real-time performance. Therefore, there is an urgent need for a duplicate data identification scheme that can effectively reduce collisions and ensure stable lookup latency. Summary of the Invention

[0004] The present invention aims to solve the technical problems of high probability of hash collision and difficulty in determining the worst-case search delay in the prior art, and provides a method and apparatus for identifying duplicate data based on hierarchical hash search.

[0005] The present invention specifically adopts the following technical solution: a method for identifying duplicate data based on hierarchical hash lookup, characterized by the following steps: (1) initializing several levels of mutually independent hash tables and a residual table, each level of hash table having a different table depth; (2) performing two independent hash operations on the current input data to obtain a first hash value and a second hash value respectively; (3) performing a double hash lookup on the current input data in the first level hash table, and marking the current input data based on the lookup result; wherein the double hash lookup includes: 1) performing a first lookup based on the first hash value as the address; 2) if a data conflict occurs in the first lookup, then performing a second lookup based on the second hash value as the address; (4) if a conflict still occurs in the current level hash table after the double hash lookup, then the current input data is used as the conflicting data, accessing the next level hash table, and repeating step (3); (5) if a data conflict still occurs after all hierarchical hash tables have been accessed, then the current input data is sent to the residual table for sequential lookup to determine whether the data is duplicated.

[0006] This invention effectively reduces the probability of hash collisions by employing multi-level independent hash tables and a double-hash lookup mechanism. More importantly, the hierarchical pipeline design limits the vast majority of lookup latency to a predictable finite number of clock cycles, thereby significantly optimizing the lookup latency in the worst case and achieving extremely high system throughput by combining multi-channel parallelism.

[0007] The specific beneficial effects of the present invention are as follows: (1) Effectively reduce the probability of hash collisions: By combining double hash lookup and hierarchical hash table cascade structure, and by reasonably setting the depth of each level of hash table, under the condition of 32k input data, the number of data that collide after the hierarchical hash table lookup can be reduced to less than 10, thereby significantly reducing the probability of hash collisions.

[0008] (2) Optimize the worst-case search latency: The search process for most data is limited to the number of preset hierarchical tables, and the worst-case search latency is optimized to a predictable upper limit, ensuring the system's high-speed real-time processing capability.

[0009] (3) High reliability: As the final solution for conflicting data, the residual table ensures that even in extreme conflict situations, duplicate data can still be accurately identified while non-duplicate data is stored.

[0010] (4) Hardware pipeline acceleration and high throughput: The hierarchical hash lookup mode proposed in this invention is perfectly matched with the hardware pipeline structure. The hierarchical hash lookup modules are cascaded in a pipeline structure to further reduce lookup latency. Combined with multi-channel parallel recognition, this device fully leverages the advantages of hardware parallelism to achieve extremely high throughput and system real-time performance. Attached Figure Description

[0011] Figure 1. Schematic diagram of the duplicate data identification scheme based on hierarchical hash search; Figure 2. Flowchart of the duplicate data identification scheme based on hierarchical hash search; Figure 3. Device diagram of the duplicate data identification scheme based on hierarchical hash search. Detailed Implementation

[0012] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0013] Example 1 This example provides a method for identifying duplicate data based on hierarchical hash lookup, as shown in Figures 1 and 2, including the following main steps: (1) Hash generation and structure initialization 1) Initialize N-level independent hash tables and a residual table.

[0014] 2) Receive the current input data.

[0015] 3) Perform two independent hash operations on the current input data to obtain the first hash value and the second hash value, which are used as the lookup address. The data bit width of the two hash values ​​depends on the depth of the first-level hash table.

[0016] (2) Double hash lookup starts from the first-level hash table and proceeds sequentially. Taking the current hash table as an example: 1) First lookup: Use the first hash value as the lookup address to find the storage location corresponding to the current hash table. There are three possible outcomes based on the lookup result: Case 1: The lookup location is empty. Mark the current input data as non-duplicate data and write it to the current lookup location to end the lookup.

[0017] Case 2: If the search position is not empty and the stored data is the same as the current input data, mark the current input data as duplicate data and end the search.

[0018] Scenario 3: If the search location is not empty and the stored data is different from the current input data, it is considered a data conflict.

[0019] 2) Second search: For data that collides after the first search, use the second hash value as the search address to find the storage location corresponding to the current hash table. If the result of the second search matches either case one or case two above, then perform the corresponding operation and end the search; if data collision still occurs, proceed to step (3).

[0020] (3) Cascading processing of conflicting data: The data marked as conflicting data is used as input to enter the next level hash table, and the double hash lookup process in step (2) is repeated. If the data is still conflicting after the search, the search continues to the next level hash table. If the data is still conflicting after all the hierarchical hash tables have been accessed, step (4) is executed.

[0021] (4) Sequential search of the residual table: For data that still results in a collision after searching through all hierarchical hash tables, a sequential search is performed starting from the first address space of the residual table. The corresponding operation is performed based on the search result: Case 1: The position is empty. The current input data is marked as non-duplicate data and written to the current search position, ending the search.

[0022] Case 2: If the search position is not empty, mark the current input data as duplicate data, and the stored data is the same as the current input data, then end the search.

[0023] Scenario 3: If the search location is not empty, but the stored data is different from the current input data, then sequentially access the next address of the residual table until all the stored data in the residual table has been accessed. If there is still a conflict, mark the current input data as non-duplicate data and write it to the end of the residual table to end the search.

[0024] 2. Storage Space Assessment: When performing duplicate data identification, the size of all hierarchical hash tables and residual tables needs to be considered. For hierarchical hash tables, assuming the number of data to be deduplicated is... The data bit width is The data flag is The total depth of all hierarchical hash tables is Then the total size of all hierarchical hash tables is:

[0025] The size of the residual table is related to the probability that data collisions will still occur after the data to be deduplicated has passed through all hierarchical hash tables. Assume the number of data items to be deduplicated in a single-level hash table is... The depth of the hash table is In a single-level hash table, the probability of a position being selected during a lookup approximates a Poisson distribution, i.e.:

[0026] Selecting a location multiple times is considered a data conflict. Therefore, the probability of a conflict occurring after a single lookup in a single-level hash table can be expressed as:

[0027] In this invention, a four-level hash table is used as an example, and the depth of each level of the hash table is as follows: , , , Therefore, the probability of a data collision occurring after two lookups in each level of the hash table is as follows:

[0028] Therefore, theoretically, the size of the residual table is:

[0029] Therefore, the storage space required for data deduplication in this invention is:

[0030] Based on theoretical analysis and practical testing, taking a four-level hash table as an example, when the amount of fingerprint data to be queried is at or below 32k, the depth ratio of the four-level hash table is configured as 8:4:2:1, where the depth of the first-level hash table is 32k, which can reduce the depth of the residual table to below 10, thereby ensuring high processing efficiency.

[0031] This embodiment of the method for duplicate data identification based on hierarchical hash lookup employs a double-hash lookup mechanism. For the current input data, two independent hash operations are performed to obtain a first hash value and a second hash value as independent lookup addresses. In the first-level hash table, the first hash value is used for the initial lookup. If a data collision occurs, the second hash value is used for a second lookup, which can reduce the probability of data collisions in the hash table by half. A hierarchical cascaded structure is adopted, establishing several levels of independent hash tables. After a data collision occurs in the first-level table, it is cascaded to the next-level table for duplicate lookup. A residual table fallback mechanism is used. For data that still collides after the hierarchical hash table lookup, it is sent to the residual table, where a sequential lookup is used for final storage and retrieval. The depth of the residual table determines the worst-case lookup latency.

[0032] Example 2 This example provides a device for duplicate data identification based on hierarchical hash lookup, used to implement the aforementioned duplicate data identification method based on hierarchical hash lookup. To meet the requirements of high-speed data processing, this device is designed with a high-parallelism hardware acceleration architecture, achieving parallel identification of multiple data streams by increasing the number of channels, thereby achieving extremely high throughput.

[0033] As shown in Figure 3, this device specifically includes the following modules: (1) Input module: configured for inputting the data stream to be identified. In this embodiment, the input module supports multi-channel parallel input.

[0034] (2) Preprocessing module: It is configured to quickly compare the data input in parallel from multiple channels at the same time before double hash lookup. Once duplicates are detected in the same batch of input data, the duplicate data is sent directly to the output module to reduce unnecessary hash calculation and lookup delay.

[0035] (3) Hash value calculation module: configured to receive input data, execute two independent hash algorithms in parallel, and generate the first hash value and the second hash value.

[0036] (4) Hierarchical hash table lookup module: This is the core of this device. It includes N independent hash table lookup sub-modules (corresponding to the N-level hash tables in the data table cache module). Each sub-module uses the first hash value and the second hash value as the lookup address and performs lookup logic on the corresponding hash table.

[0037] The hierarchical hash lookup mode proposed in this invention is naturally suited to hardware pipeline architectures. The N hash table lookup sub-modules are cascaded in a strict pipeline structure. After the current level module completes the lookup of the current input data, it can send conflicting data to the next level module for lookup while continuing to process the next data, thereby maximizing the advantages of hardware parallelism and ensuring that, even in the worst case, the lookup latency is still limited to within the pipeline depth, greatly optimizing the lookup latency in the worst-case scenario.

[0038] (5) Residual table lookup module: configured to receive conflict data output from the hierarchical hash table lookup module, responsible for reading and writing operations on the residual table to implement sequential search logic, and output the identified duplicate data.

[0039] (6) Data table caching module: This module contains storage entities for hash tables at all levels and residual tables, configured to cache non-repeating data fingerprints and status information. These storage units are preferably implemented using on-chip high-speed storage to match the high-speed read and write requirements of the lookup module.

[0040] (7) Output module: configured to receive the identification results of the preprocessing module, the hierarchical hash table lookup module and the residual table lookup module, and output the final marked duplicate data to the outside world.

[0041] The device for duplicate data identification based on hierarchical hash lookup in this embodiment is preferably implemented using a hardware acceleration architecture such as a field-programmable gate array (FPGA). The hierarchical cascaded structure adopts a pipelined design, transforming the multi-level lookup process into a fixed serial processing depth, which can maximize the advantages of hardware parallelism; at the same time, the device supports multi-channel parallel identification, and further achieves high throughput by increasing the number of channels.

Claims

1. A method for identifying duplicate data based on hierarchical hash lookup, characterized in that, The steps include: (1) initializing several levels of independent hash tables and a residual table, with each level of hash table having a different table depth; (2) performing two independent hash operations on the current input data to obtain the first hash value and the second hash value respectively; (3) Perform a double hash lookup on the current input data in the first-level hash table, and mark the current input data based on the lookup result; The double hash lookup includes: 1) performing a first lookup based on the first hash value as the address; 2) If a data conflict occurs during the first lookup, a second lookup is performed based on the second hash value as the address; (4) If a conflict still occurs after the current level hash table is double-hashed, the current input data is used as the conflicting data, the next level hash table is accessed, and step (3) is repeated; (5) If a data conflict still occurs after all the hierarchical hash tables have been accessed, the current input data is sent to the residual table for sequential lookup to determine whether the data is duplicated.

2. The method according to claim 1, characterized in that: The two independent hash operations use different hash functions, and the bit width of the two resulting hash values ​​is determined by the depth of the first-level hash table.

3. The method according to claim 1, characterized in that: The step of marking input data based on search results specifically includes: (1) when the first search position is empty, marking the current input data as non-duplicate data and writing it into the current hash table to find the location, and ending the search; (2) when the first search position is not empty and the stored data is the same as the current input data, marking the current input data as duplicate data, and ending the search; (3) when the first search position is not empty and the stored data is different from the current input data, it is considered that a data conflict has occurred.

4. The method according to claim 1, characterized in that: The residual table accesses its storage space using a sequential search method. If the stored data is the same as the current input data, the current input data is marked as duplicate data, and the search ends; otherwise, the current input data is marked as non-duplicate data and written to the end of the residual table.

5. The method according to claim 1, characterized in that: The depth of the residual table is adaptively configured based on the collision probability of the hierarchical hash table.

6. The method according to claim 5, characterized in that: Configure a four-level hash table, with the table depth of each level configured in a ratio of 8:4:2:1, and the residual table depth set to 10.

7. A device for identifying duplicate data based on hierarchical hash lookup, characterized in that, The apparatus for implementing the method according to any one of claims 1 to 6 comprises: (1) an input module configured to input a data stream to be identified; (2) a hash value calculation module configured to receive input data, execute two independent hash algorithms in parallel, and generate a first hash value and a second hash value; (3) a hierarchical hash table lookup module configured to perform double hash lookup on the several levels of hash tables, wherein the lookup of each level of hash table adopts a pipelined cascade structure; (4) a residual table lookup module configured to receive data that has conflicted after the hierarchical hash table lookup and perform sequential lookup; (5) a data table cache module configured to store the hash tables at each level and the residual table; and (6) an output module configured to output the identified duplicate data to the outside world.

8. The apparatus according to claim 7, characterized in that: The device is integrated on an FPGA. The hash value calculation module and the hierarchical hash table lookup module utilize the parallel characteristics of the FPGA to realize parallel calculation and parallel read / write operations of data.

9. The apparatus according to claim 7, characterized in that: The input module supports multi-channel parallel input, and the device also includes a preprocessing module configured to pre-filter duplicate data in the multi-channel parallel input before hash lookup.

Images