Method and device for identifying duplicated data based on hierarchical hash lookup

By employing hierarchical hash lookup and a double hash mechanism, combined with residual tables and hardware pipelines, the problems of hash collisions and latency are resolved, enabling efficient duplicate data identification and improving storage resource utilization and system real-time performance.

CN121958264APending Publication Date: 2026-05-01HEFEI UNIV OF TECH
0 Cites 0 Cited by

Patent Information

Application Number
CN202511866173.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-11
Publication Date
2026-05-01

AI Technical Summary

Technical Problem

In existing technologies, the probability of hash collisions is high and the worst-case lookup latency is difficult to determine, resulting in low efficiency in duplicate data identification and affecting storage resource utilization and system real-time performance.

Method used

A hierarchical hash lookup method is adopted, which reduces the probability of hash collisions and optimizes lookup latency by using multi-level independent hash tables and a double hash lookup mechanism, combined with a residual table. High throughput is achieved by utilizing hardware pipelines and multi-channel parallelism.

Benefits of technology

It significantly reduces the probability of hash collisions, optimizes lookup latency in the worst case, ensures high-speed real-time processing capabilities, and improves storage resource utilization and system throughput.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention provides a method and device for identifying duplicated data based on hierarchical hash lookup, and the method comprises the steps: executing two independent hash operations in parallel for input data to obtain two lookup addresses, and constructing a lookup structure consisting of a plurality of levels of mutually independent hash tables and a residual table. The method comprises the following steps: firstly, carrying out double-hash search on data in a first-level hash table, and cascading the data to a next-level hash table if a conflict occurs until all hierarchical tables are accessed; and for the data which still conflict, sequentially searching in the residual table. According to the method, the probability of Hash conflicts is effectively reduced by adopting the multi-stage independent Hash table and a double-Hash lookup mechanism; more importantly, most of the search delays are limited in a predictable limited clock period by the hierarchical pipeline design, so that the search delay under the worst condition is obviously optimized, and an extremely high system throughput rate is realized in combination with multi-pass parallel.
Need to check novelty before this filing date? Find Prior Art