Method for using hashes to find candidates for deduplication

Abstract

FIELD: computer technology.SUBSTANCE: method and system for using information to generate an optimized data identifier for deduplication are proposed. The system comprises the steps of: calculating a first check code for a data block using first and second cyclic redundancy code calculation algorithms, wherein the second checksum calculation algorithm differs from the first; forming a combined identifier of the data block by concatenating or bitwise combining said first and second check codes, obtaining an intermediate value of the full length; truncating said combined identifier to a predetermined target length; comparing said final optimized identifier with a set of previously stored identifiers to determine a duplicate of said data block; using said final optimized identifier for the operation of deduplicating said data block or indexing said data block.EFFECT: ability to use smaller hashes to search for deduplication candidates, which allows for saving disk space without affecting the performance of the deduplication process under high load factors.9 cl, 4 dwg, 1 tbl

Description

[0001] Technical field

[0002] The invention relates to the field of computing, data processing, storage and transmission, namely to a method for generating a hash signature of an optimal size, which increases the efficiency of deduplication by reducing the load on computing and stored cloud resources with an optimal probability of collisions.

[0003] Technology Level

[0004] A patent application, US 20190334727 A1, published October 31, addresses the issue of data privacy in an untrusted environment with shared deduplication, such as in multi-tenant cloud services, without sacrificing storage efficiency. This is particularly relevant in multi-tenant or cloud environments where traditional cryptographically strong hashes (e.g., SHA-256) create unacceptably large metadata overhead. A secure deduplication method and system with reduced hash values ​​is proposed. For each data segment and a specific deduplication domain with a unique cryptographic key, a keyed hash value is generated using a keyed hash function (preferably a universal hash function, UHF), which receives both the segment itself and the domain's secret key as input. This key hash value, which has a compact size in the range of 128-160 bits, is then used for the deduplication operation by comparing with the entries in the index table.This achieves a dual technical benefit: firstly, security is ensured, since without knowledge of the domain key it is impossible to calculate the correct hash and carry out a brute-force attack; secondly, the volume of metadata required for indexing is significantly reduced, which increases the overall efficiency and scalability of the storage system.

[0005] The main difference in our proposed invention is that it addresses a fundamentally different problem, as it is based on a different architectural premise and threat model, which considers the environment of a trusted and controlled perimeter (trusted domain). This means that a different architectural approach is used, which allows the proposed invention to focus on solving the problem of efficient, in terms of the use of computing resources, and high-speed deduplication, which ensures a qualitatively different technical result: maximum acceleration of the deduplication process and a reduction in the computing load on trusted system components, as well as associated savings in network traffic.

[0006] A patent application, US 20190340262 A1, published November 7, addresses the high computational and resource load associated with duplicate search in deduplication systems, which requires frequent comparison of hash values ​​across large volumes of data, slowing down write operations. A method and system for accelerated block-level deduplication is proposed, based on two-stage hash reduction. For a new data block, the hashes of its parts (chunks) are first calculated, then these hashes are sequentially truncated in two stages—first to a size of less than 20 (preferably 10) bytes, and then to a size of less than 5 (preferably 1) bytes, forming a compact string of reduced hashes. This string is compared with a hash table, which stores similarly truncated hashes of previously written blocks.The key criterion for deduplicating an entire block is detecting a sequential match between two consecutive truncated hashes of the new block and the first and last hashes of the stored block in the table. This achieves the primary technical result: significantly accelerating write operations by dramatically reducing the amount of data involved in comparison (down to comparing single-byte values) and reducing memory requirements for the hash table, while maintaining acceptable reliability in detecting duplicates of entire blocks based on matching checkpoints at their beginning and end.

[0007] The main difference in our proposed invention is the optimization of the preceding and most resource-intensive stage of the initial calculation of the hash values ​​themselves. The proposed invention eliminates the need for a complete hash recalculation for each new data block, which provides a qualitatively different saving of computing resources – not due to faster comparison of already calculated hashes, but by drastically reducing the operations required to obtain them. This approach also leads to a concomitant technical result – significant savings in network traffic, since synchronization between nodes requires transmitting only minimal incremental updates rather than complete hash sets.

[0008] Publication US 20240134528 A1, published April 25, addresses the performance degradation and computational overhead associated with false positives (collisions) of non-cryptographic hash functions used for the initial, fast duplicate search. Instead of costly byte-by-byte comparisons of the entire block contents for each hash match, the patent proposes a method and system for intelligent early filtering of mismatched blocks based on a comparison of their metadata. After a preliminary match between the candidate and target block using a fast, non-cryptographic hash (e.g., MurmurHash, xxHash), the system verifies one or more secondary deterministic attributes—the checksum of the original or compressed data, the compression method, and / or the size of the compressed block.If this metadata doesn't match, the deduplication attempt immediately terminates, saving resources on a full data comparison. If the metadata does match, the process can proceed to a more thorough, yet still optimized, check (for example, comparing parts of compressed data). This technical result significantly increases the overall efficiency and speed of the deduplication process by minimizing the number of resource-intensive full block comparisons, while maintaining high reliability in detecting true duplicates and filtering out false hash matches.

[0009] The main difference in our proposed invention is its use of a different optimization approach, aimed not at accelerating hash comparisons or filtering collisions, but at radically reducing computational costs at the initial calculation stage. This is achieved by calculating not full hashes for each new data block, but only an incremental portion of the existing hash, which, when combined, forms a combined hash. An additional and important technical result is significant savings in network traffic, since synchronization between nodes, clients, or servers requires transmitting only compact incremental updates, rather than full hash sets.

[0010] US Patent 11650976 B2, published May 16, 2023, addresses the inefficient use of memory and computing resources in deduplication systems. This problem arises from the need to maintain a single shared hash table (fingerprint) for all data, a significant portion of which may be unique and will never be deduplicated. This issue is particularly acute in resource-constrained storage systems (e.g., flash drives) and in the context of background management operations (garbage collection, wear leveling). A system and method for selectively managing fingerprint tables is proposed. These tables classify data into two categories: data with a high deduplication probability and data with a low probability. Their hashes (fingerprints) are then stored separately in two different tables: a priority (first) table and a secondary (second) table.Classification is based on dynamically collected data attributes, such as access frequency, age (time), success rate of previous deduplication, and device status metrics (health, read voltage). Table management (adding and deleting records) is tied to internal system maintenance events. This results in significant memory savings and increased system performance by reducing the size and accelerating searches in the primary, priority fingerprint table, which concentrates only potentially duplicate data, while unique data is kept uncluttered, stored in a compact secondary table or not indexed at all.The claimed invention focuses on optimizing the storage and retrieval of data identifiers (hashes), i.e., metadata organization. The key idea here is the intelligent partitioning of a single metadata space based on predicted usefulness, which saves memory and accelerates retrieval in the context of background operations specific to flash memory (garbage collection, calibration). This optimization occurs "at the top level," after the hashes have been calculated.

[0011] The main difference in our proposed invention is that it is aimed at optimizing the initial, most resource-intensive stage—hash calculation. This eliminates the problem of redundant calculations in scenarios with high data locality, where the same fragments are processed repeatedly. The core idea is incremental hash calculation, eliminating duplicate calculations.

[0012] So, we have a problem: there's a hash table that needs to be searched for candidates, where the key is the block hash and the value is the block address. Internally, SDS (Software Defined Storage) already uses some hashes (crc32) for integrity checking. However, simply using crc32 doesn't make sense—it produces too many collisions. Instead, we add another hash (xxh32, adler32, etc.) to crc32. This, on the one hand, provides sufficient hash variability (fewer collisions), and on the other, saves disk space and CPU compared to calculating a new "good" hash.

[0013] The technical result of the invention is to provide the ability to use smaller hashes to search for candidates for deduplication, which, on the one hand, saves disk space, and on the other hand, does not affect the performance of the deduplication process under a large load factor.

[0014] The specified technical result is achieved by implementing a method for generating an optimized data identifier for deduplication, performed by a computing device, comprising the steps of:

[0015] a) calculating a first check code for a data block using a first cyclic redundancy code calculation algorithm;

[0016] b) calculating a second check code for said data block using a second checksum calculation algorithm different from the first;

[0017] c) forming a combined data block identifier by concatenating or bitwise combining said first and second control codes, obtaining an intermediate value of full length;

[0018] d) truncating said combined identifier to a predetermined target length that is less than its full length to obtain a final optimized identifier, wherein the target length is chosen to balance between the probability of collision and the amount of metadata;

[0019] e) comparing said final optimized identifier with a set of previously stored identifiers to determine whether said data block is a duplicate;

[0020] f) use said final optimized identifier for the operation of deduplicating said data block or indexing said data block.

[0021] An additional feature is that the target length selection contains steps in which:

[0022] define the expected volume of data to be stored;

[0023] define the deduplication block size;

[0024] define the time cost of checking a deduplication candidate data block against the original data block, where the time cost includes: a) reading the data block from disk; b) exchanging data blocks over the network to aggregate the deduplication candidate data block and the original data block on one host for comparison; c) comparing the data blocks with each other; and / or d) the cost of comparing the data in the deduplication candidate data block and the data in the original data block with each other;

[0025] Calculate the number of collisions with hashes of different lengths for the estimated data volume;

[0026] calculate the time lost on data blocks with collision hashes = time to check a candidate data block for deduplication * the number of data blocks with collisions;

[0027] Calculate the total hash size = hash size (bytes) * data size / block size;

[0028] Based on the above values, a table is formed with columns that indicate the values: “hash size”, “number of collisions”, “time spent on collision blocks”, “space allocated for storing hashes”;

[0029] select an acceptable ratio between the time spent on collision data blocks and the storage space spent on storing the hashes of data blocks;

[0030] Calculate the available time budget for collision resolution as the difference between the rewrite time and the optimal operation time, which takes into account the time operations without collisions, and based on this, select the optimal hash size from the mentioned table by iterating through the hash sizes and selecting the minimum size at which the predicted loss time due to collisions does not exceed the specified one.

[0031] An additional feature is that the first cyclic redundancy code calculation algorithm is, in one of the special cases, the CRC32 algorithm, and the second checksum calculation algorithm is, in one of the special cases, the Adler32 algorithm.

[0032] An additional feature is that the first control code calculated in step (a) is additionally used for the integrity check operation of said data block when it is read or transmitted, eliminating the need for a separate calculation of the integrity code.

[0033] An additional feature is that the combined identifier in step (c) has a length of 64 bits, and the predetermined target length in step (d) is at least 54 bits.

[0034] The specified technical result is also achieved with the software and hardware implementation of the proposed method in a computing system for processing data for generating an optimized data identifier for generating an optimized data identifier for deduplication, comprising: - a storage device; - a processor configured to execute stored instructions from the storage device for implementing the method according to any one of paragraphs 1-6 of the formula.

[0035] An additional feature is that the processor is configured to use as: a first CRC32 control code for a data block; a second Adler-32 control code for said data block with optimization based on the data obtained during the CRC32 calculation; used to form a 64-bit combined identifier from said first and second control codes; used to truncate said 64-bit combined identifier to a 54-bit final identifier; and used said 54-bit final identifier for deduplication or indexing of said data block.

[0036] An additional feature is that the processor is further configured to store said 54-bit final identifier in said memory device, providing a memory space saving of at least 16% compared to storing 64-bit identifiers for the same data blocks.

[0037] An additional feature is that the first CRC32 check code is also used by the processor to check the integrity of the said data block when it is transmitted over the communication network, saving network traffic that would be spent on transmitting a separate integrity code.

[0038] The specified technical result is also achieved with the software and hardware implementation of the proposed method by means of a machine-readable storage medium, on which computer-executable instructions are stored, which, when executed by the processor, instruct the processor to implement the method for generating an optimized data identifier for deduplication according to any of paragraphs 1-6 of the formula.

[0039] The conducted analysis of the state of the art allows us to determine that the proposed technical solution is new and has an inventive level, and the possibility of its use in industry determines its industrial applicability.

[0040] These and other aspects will become apparent and will be explained with reference to the drawings and the embodiment described hereinafter.

[0041] The invention is explained by the following graphic materials:

[0042] Fig. 1 shows the component diagram of the computing system for generating an optimized data identifier for deduplication.

[0043] Fig. 2 shows the deduplication diagram.

[0044] Figure 3 shows the diagram of the deduplication stages in a Software Define Storage (SDS) system.

[0045] Fig. 4 shows the diagram of the proposed solution for generating an optimized data identifier for deduplication based on a combined hash.

[0046] Implementation of the invention

[0047] Fig. 1 shows an example of a general view of a computing system - a host (201), implementing the claimed method for generating an optimized data identifier for deduplication. Communication network (100) data processing system (200). The host (201) can be a part of a computer system, for example, a module, a storage node, a part of a computing cluster, a distributed node - processing all the data necessary for implementing the invention. Shown are interconnected means: host (node) (201), processor (101), input-output interfaces (102), ROM storage device (103), control data bus (system bus) (104), RAM memory (random access memory, RAM, physical memory) (105), input-output means (106), network interfaces (network card, network adapter, network interface, network device) (107).

[0048] SDS (Software Define Storage) system - software-defined storage system provides high availability and speed of access to information critical to applications.

[0049] Deduplication in data storage is a type of compression that uses an algorithm to remove identical parts of data—duplicate copies of repeating data. It works by identifying and storing unique fragments of information during analysis, and comparing new and old fragments as the analysis progresses. When a duplicate fragment is identified, it is replaced with a reference to a unique entry, and the space occupied by the duplicate is reclaimed.

[0050] One of the problems is how to find duplicate chunks in a large data set. The obvious solution—comparing all data with all other data—is very resource-intensive. To optimize this, the data is divided into blocks and the hashes of the blocks are pre-compared.

[0051] The effectiveness of deduplication in SDS is measured by the deduplication ratio, which is the ratio of the original data size to the reduced size.

[0052] Typical deduplication in SDS (Software Define Storage), as shown in Fig. 1 and 2, consists of the following steps:

[0053] • building block hash index -> block address.

[0054] • crawl the index and search for candidates for deduplication.

[0055] • comparison of deduplication candidates with each other.

[0056] • removing duplicates.

[0057] The main issue when building deduplication systems is the choice of which hash function / functions to use to calculate the hash.

[0058] • Use of cryptographically strong hashes (SHA256, keyed hash functions, etc.).

[0059] • Using hashes to check integrity (CRC32 etc.).

[0060] • Using fast hashes with separate counting (XXH etc.).

[0061] The disadvantages of existing solutions are:

[0062] • Using cryptographically strong hashes is excessively expensive computationally.

[0063] • Using hashes for integrity checking, as well as other low-resolution hashes (<= 32-bit), results in too many collisions. When deduplicating 4K blocks, we can obtain unique values ​​for no more than 16 TB of data; beyond that, collisions are guaranteed.

[0064] • Using fast hashes with separate calculation is acceptable, but they must be calculated separately, given that some hashes (CRC32) already exist.

[0065] The main difference of the proposed method for using hashes to find deduplication candidates, as shown in Fig. 4, is the use of a combined hash in the processing stages. This involves calculating a first check code for a data block using a first cyclic redundancy check algorithm (CRC32) and calculating a second check code for said data block using, for example, a second Adler32 checksum algorithm, thereby obtaining a combined hash.

[0066] The Adler-32 checksum is obtained by calculating two 16-bit checksums A and B and concatenating their bits into a 32-bit integer (B in the high-order bits, A in the low-order bits). A is equal to the sum of all bytes in the string plus one, and B is the sum of all individual values ​​of A at each step. At the beginning of the Adler-32 function, A is initialized to one and B to zero. Then, for each data byte[i], the calculations are performed: A = (A + byte[i]) mod 65521, B = (B + A) mod 65521. The sums are taken modulo 65521 (the largest prime number less than 2^16 = 65536). The bytes are written in network order, B occupies the 2 most significant bytes. After all bytes have been processed, the final 32-bit checksum value is formed as (B << 16) | A.

[0067] Form a combined data block identifier by concatenating or bitwise combining said first and second control codes, obtaining an intermediate value of the full length.

[0068] Truncate the specified combined identifier to an optimal target length that is less than its full length to obtain the final optimized identifier, where the target length is chosen to balance the probability of collision and the amount of metadata.

[0069] Compare said final optimized identifier with a set of previously stored identifiers to determine a duplicate of said data block, and use said final optimized identifier for a deduplication operation of said data block or indexing of said data block.

[0070] The methodology for determining the target length to balance between the probability of collision and the volume of metadata is as follows:

[0071] 1) Determine the expected volume of data to be stored.

[0072] 2) Determine the size of the deduplication block.

[0073] 3) Determine the overhead (time) of checking a candidate block against the original block, which includes:

[0074] a. Reading data block from disk;

[0075] b. exchanging blocks over the network to aggregate the candidate block and the original block on one host for comparison;

[0076] c. comparison of blocks with each other;

[0077] d. Other possible costs for comparing the candidate block data and the original block data.

[0078] 4) Calculate the number of collisions with hashes of different lengths for the expected volume of data, for example, using the well-known method see. https: / / matt.might.net / articles / counting-hash-collisions / .

[0079] 5) Calculate the time lost on blocks with collision hashes = time to verify a candidate block * the number of blocks with collisions.

[0080] 6) Calculate the total size of hashes = hash size (bytes) * (data size / block size).

[0081] 7) Create a table with the following columns: “Hash size”, “Number of collisions”, “Time spent on collision blocks”, “Space allocated for storing hashes”.

[0082] Below is an example table for 2 PB of data, a 4 KB deduplication block, and a block processing overhead of 15 µs:

[0083] Table 1

[0084] 8) Select an acceptable ratio between the time spent on collision blocks and the storage volume spent on storing block hashes, wherein the selection of an acceptable ratio may occur based on pre-set values ​​or automatically, for example, including, but not limited to, based on the calculation of the average write speed measurement in the cluster, the characteristics of the index storage (speed of adding and traversing a block), the time of comparing block data and the time of deduplication.

[0085] For our example with a 2 PB (2 × 10¹⁵ bytes) cluster with a deduplication block size of 4096 bytes (4 KB). The processing time of one block (read, send over the network, compare data in case of a collision) is 15 µs (15 × 10⁻⁶ s). The target (acceptable) time of a full cluster rewrite is set to 4,398,247 seconds. The ideal deduplication time (excluding collisions) is 4,398,047 seconds. Thus, the available time budget (acceptable ratio time) for collision resolution can be calculated as the difference between the rewrite time and the ideal operation time, which takes into account temporary operations without collisions. In total, we are left with: 4,398,247 - 4,398,047 = 200 sec. This means that the system selects the optimal hash size of 54 bits, according to the example in Table 1.Thus, the system automatically cycles through possible hash sizes (for example, from 32 to 64 bits) and selects the minimum at which the predicted collision loss time does not exceed the available budget, which corresponds to 54 bits.

[0086] The proposed combined hash solution allows for:

[0087] saving disk space by about 16% compared to a 64-bit hash, with a slight increase in the number of collisions of only 0.008% compared to a 128-bit hash;

[0088] saving traffic, which would inevitably increase if a 64-bit hash were used instead of a 32-bit one;

[0089] CRC32 is already used to check data integrity, calculating adler32 hash is twice as fast as calculating xxh64, one of the fastest hashes.

[0090] The method according to the proposed invention may also take the form of at least software code, instructions, or software applications embodied on tangible media such as hard drives, RAM, or any other machine-readable medium. The instructions may be provided as downloadable or firmware and / or implemented in whole or in part in hardware components such as ASICs, FPGAs, DSPs, GPUs, or any other similar devices. The instructions may be configured for execution by one or more processors of a method for using hashes to find deduplication candidates. Examples of software instructions may include machine language codes, such as codes generated by a compiler, as well as high-level language codes that can be executed by a computer using an interpreter and the like.

[0091] The information provided in the description is exemplary and does not limit the scope of the present method defined by the claims. One skilled in the art will recognize that, in developing any practical embodiment of the present method, many embodiment-specific decisions must be made to achieve specific goals, and these specific goals will vary from embodiment to embodiment.

Claims

1. A method for generating an optimized data identifier for deduplication, performed by a computing device, comprising the steps of: (a) computing a first check code for a data block using a first cyclic redundancy check algorithm; (b) calculating a second check code for said data block using a second checksum calculation algorithm different from the first; (c) forming a combined data block identifier by concatenating or bit-combining said first and second control codes to obtain an intermediate value of full length; (d) truncating said combined identifier to a predetermined target length that is less than its full length to obtain a final optimized identifier, wherein the target length is selected to balance between the likelihood of collision and the amount of metadata; (e) comparing said final optimized identifier with a set of previously stored identifiers to determine whether said data block is a duplicate; (f) using said final optimized identifier for the operation of deduplicating said data block or indexing said data block.

2. The method according to paragraph 1, characterized in that said selection of the target length includes the steps of: determine the expected volume of data to be stored; determine the size of the deduplication block; determine the time cost of checking a deduplication candidate data block against the original data block, where the time cost includes: a) reading the data block from disk; b) exchanging data blocks over the network to aggregate the deduplication candidate data block and the original data block on a single host for comparison; c) comparing the data blocks with each other; and / or d) the cost of comparing the data in the deduplication candidate data block and the data in the original data block with each other; calculate the number of collisions with hashes of different lengths for the expected volume of data; calculate the time lost on data blocks with collision hashes = time to check a candidate data block for deduplication * number of data blocks with collisions; calculate the total size of hashes = hash size (bytes) * data size / block size; form a table containing the values: “hash size”, “number of collisions”, “time spent on collision blocks”, “space allocated for storing hashes”; select an acceptable ratio between the time spent on collision data blocks and the storage volume spent on storing the hashes of data blocks; The available time budget for resolving collisions is calculated as the difference between the rewrite time and the optimal operation time, which takes into account the time operations without collisions, and on this basis the optimal hash size is selected from the mentioned table by iterating through the hash sizes and selecting the minimum size at which the predicted loss time due to collisions does not exceed the specified one.

3. The method according to claim 1, characterized in that the first algorithm for calculating the cyclic redundancy code is the CRC32 algorithm, and the second algorithm for calculating the checksum is the Adler32 algorithm.

4. The method according to claim 1, characterized in that the first control code calculated in step (a) is additionally used for the operation of checking the integrity of said data block when it is read or transmitted over a communication network.

5. The method of claim 1, wherein the combined identifier in step (c) has a length of 64 bits, and the predetermined target length in step (d) is at least 54 bits.

6. A system for generating an optimized data identifier for deduplication, comprising: a storage device; a processor configured to execute stored instructions from the storage device to implement the method according to paragraph 1.

7. The system according to claim 6, characterized in that the processor is configured to use: a first CRC32 control code for a data block; a second Adler-32 control code for said data block with optimization based on the data obtained during the calculation of the CRC32; and also used to form a 64-bit combined identifier from said first and second control codes; using truncation of the 64-bit combined identifier to a 54-bit final identifier; and using the 54-bit final identifier for deduplication or indexing of said data block.

8. The system according to claim 7, characterized in that the processor is configured to store a 54-bit final identifier in a storage device.

9. The system according to claim 7, characterized in that the processor is configured to additionally check the integrity of the data block during its transmission over the communication network based on the first CRC32 control code.

Images