A deduplication method, device, equipment and readable storage medium

Abstract

The invention discloses a deduplication method, which converts the comparison of deduplication data into a two-level comparison, and stores a part of the eigenvalues ​​written in the IO, that is, the first part of the eigenvalues ​​in the cache, and the written The other part of the eigenvalues ​​of the IO, that is, the second part of the eigenvalues ​​is stored in the disk to avoid excessive occupation of the cache space; when comparing, the first part of the eigenvalues ​​is first compared, and only the first part of the eigenvalues ​​is hit Only when the first part of the eigenvalue comparison is not hit, then the second part of the eigenvalue comparison is discarded and the data is written directly. This two-level comparison The deduplication performance can be improved by caching pre-screening, and unnecessary memory overhead can be reduced by caching and post-allocation. The overall performance of the system is high. The invention also discloses a deduplication device, equipment and a readable storage medium, which have corresponding technical effects.

Description

technical field [0001] The present invention relates to the technical field of all-flash storage, in particular to a deduplication method, device, equipment and readable storage medium. Background technique [0002] At present, the all-flash storage system has gradually become the mainstream storage device of major operators and financial institutions. In order to increase the total data storage capacity under the premise of constant capacity, major manufacturers have supported the deduplication (deduplication) feature. Deduplication is a technology to save storage space. Usually, there are many duplicate data in the data storage pool. Deduplication is a technology to find and process these duplicate data. Simply put, deduplication is to delete Only one copy of N duplicate data is kept, and the address pointer of N-1 data points to the only one. Deduplication saves customer costs, but introduces a lot of overhead that does not exist in traditional storage for the system, su...

AI Technical Summary

Problems solved by technology

This can effectively reduce the writing of duplicate data to the disk and greatly improve the effective utilization of the storage system. However, frequent HASH reading of the disk will greatly increase the delay of writing data and reduce the overall performance of the storage system.

To solve the above problems, there is a way to cache all HASH data, but the current memory size cannot support enough HASH data for all caches

Images