A method, device and storage medium for deduplication query

Abstract

The application discloses a deduplication query method, comprising the following steps: receiving a write request, and determining that the data corresponding to the current received write request is target data; calculating a fingerprint value of the target data to obtain a target fingerprint value, and querying the target fingerprint value in a cuckoo filter; if the target fingerprint value is found in the cuckoo filter, adding a current logical address of the target data to metadata mapping of a physical address of the target data, otherwise, storing the target data, adding the target fingerprint value to the cuckoo filter, and adding the current logical address of the target data to the metadata mapping of the physical address of the target data. The application can reduce the frequency of deduplication metadata query in a B+ tree on a disk, and improve the performance of the whole storage system. The application also provides a deduplication query device, a deduplication query equipment and a computer readable storage medium, which have the above beneficial effects.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and more specifically, to a deduplication query method, apparatus, device, and storage medium. Background Technology

[0002] In all-flash storage systems, SSDs are significantly more expensive than traditional hard disk drives, making deduplication and compression crucial features. Deduplication requires metadata support, and as the system's supported storage capacity increases, the metadata grows linearly. With block mapping in 8KB units, if the data volume reaches several petabytes (PB), the metadata will be at the terabyte (TB) level.

[0003] Clearly, the metadata of the TB category has caused considerable trouble for data retrieval. Therefore, how to effectively manage metadata to achieve fast data reading and writing is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a deduplication query method, deduplication query device, deduplication query equipment, and computer-readable storage medium that can improve the disk application performance of write caching.

[0005] To achieve the above objectives, this application provides a deduplication query method, the specific technical solution of which is as follows:

[0006] Receive write requests and determine that the data corresponding to the currently received write request is the target data;

[0007] Calculate the fingerprint value of the target data to obtain the target fingerprint value, and then query the target fingerprint value in the Cuckoo Filter;

[0008] If the target fingerprint value is found in the cuckoo filter, then a metadata mapping from the current logical address of the target data to the physical address of the target data is added; otherwise, the target data is stored, the target fingerprint value is added to the cuckoo filter, and a metadata mapping from the current logical address of the target data to the physical address of the target data is added.

[0009] Optionally, if the target fingerprint value is found in the cuckoo filter, the method further includes:

[0010] Check if there is a metadata mapping from the fingerprint value to the physical address to which the target fingerprint value belongs. If it exists, determine that the physical address in the metadata mapping from the fingerprint value to the physical address to which the target fingerprint value belongs is the physical address of the target data, and execute the step of adding the current logical address of the target data to the metadata mapping of the physical address of the target data. Otherwise, execute the step of storing the target data.

[0011] Accordingly, after storing the target data, the target fingerprint value is added to the metadata mapping of the physical address of the target data.

[0012] Optionally, before querying the target fingerprint value in the cuckoo filter, the method further includes:

[0013] Set up the array structure and linked list structure corresponding to the cuckoo filter; wherein, the same fingerprint value is stored in two positions in the array structure, and the fingerprint value is a secondary hash value made by hashing a portion of the bits corresponding to the physical address.

[0014] Optionally, when inserting the hash value of the data at the physical address into the array structure, it further includes:

[0015] The insertion position of the hash value in the array structure is calculated using the hash algorithm contained in the cuckoo filter.

[0016] Optionally, when calculating the insertion position of the hash value in the array structure using the hash algorithm included in the cuckoo filter, if an element exists at the insertion position, the method further includes:

[0017] A portion of the hash value is obtained using a reverse hash lookup function;

[0018] Determine whether the aforementioned bits are the same;

[0019] If they are the same, perform a precise HP lookup on the hash value;

[0020] If they are different, the hash value is inserted into the linked list structure corresponding to the array structure.

[0021] Optional, also includes:

[0022] The mapping relationship between the physical address and the corresponding logical address is managed using the controller on the solid-state drive;

[0023] Specifically, after the storage pool is created, the physical addresses are integrated and remapped; the logical addresses are recoded using the pool information and storage offset information of the storage pool.

[0024] Optionally, determine that the data corresponding to the currently received write request is the target data, including:

[0025] Determine that the data in the data block corresponding to the currently received write request is the target data;

[0026] Accordingly, calculating the fingerprint value of the target data to obtain the target fingerprint value includes:

[0027] The target fingerprint value is obtained by calculating the fingerprint value of the target data using a preset hash algorithm.

[0028] This application also provides a deduplication query device, including:

[0029] The receiving module is used to receive write requests and determine that the data corresponding to the currently received write request is the target data;

[0030] The query module is used to calculate the fingerprint value of the target data to obtain the target fingerprint value, and query the target fingerprint value in the Cuckoo Filter;

[0031] The execution module is configured to, if the target fingerprint value is found in the cuckoo filter, add a metadata mapping from the logical address in the write request corresponding to the target data to the physical address of the target data; otherwise, store the target data, add the target fingerprint value to the cuckoo filter, and add a metadata mapping from the logical address in the write request corresponding to the target data to the physical address of the target data.

[0032] Optional, also includes:

[0033] The mapping query module is used to query whether there is a metadata mapping from fingerprint value to physical address to which the target fingerprint value belongs if the target fingerprint value is found in the cuckoo filter. If it exists, the physical address in the metadata mapping from fingerprint value to physical address to which the target fingerprint value belongs is determined to be the physical address of the target data, and the step of adding the current logical address of the target data to the metadata mapping of the physical address of the target data is executed. Otherwise, the step of storing the target data is executed.

[0034] Optional, also includes:

[0035] The filter setting module is used to set the array structure and linked list structure corresponding to the cuckoo filter; wherein, the same fingerprint value is stored in two positions in the array structure, and the fingerprint value is a secondary hash value made by hashing a portion of the bits corresponding to the physical address.

[0036] Optional, also includes:

[0037] The position calculation module is used to calculate the insertion position of the hash value in the array structure using the hash algorithm contained in the cuckoo filter.

[0038] Optional, also includes:

[0039] The insertion module is used to obtain a portion of the hash value using a hash reverse lookup function when an element exists at the insertion position; determine whether the portion of the bits are the same; if they are the same, perform a precise HP lookup on the hash value; if they are different, insert the hash value into the linked list structure corresponding to the array structure.

[0040] Optional, also includes:

[0041] The mapping relationship management module is used to manage the mapping relationship between the physical address and the corresponding logical address using the controller on the solid-state drive; wherein, after the storage pool is created, the physical address is integrated and remapped; and the logical address is recoded using the pool information and storage offset position information of the storage pool.

[0042] Optionally, the receiving module includes:

[0043] The data determination unit is used to determine that the data in the data block corresponding to the currently received write request is the target data;

[0044] Accordingly, the query module includes:

[0045] The hash calculation unit is used to calculate the fingerprint value of the target data using a preset hash algorithm to obtain the target fingerprint value.

[0046] This application discloses a deduplication query method, comprising: receiving a write request and determining that the data corresponding to the currently received write request is target data; calculating the fingerprint value of the target data to obtain a target fingerprint value, and querying the target fingerprint value in a cuckoo filter; if the target fingerprint value is found in the cuckoo filter, adding a metadata mapping from the current logical address of the target data to the physical address of the target data; otherwise, storing the target data, adding the target fingerprint value to the cuckoo filter, and adding a metadata mapping from the current logical address of the target data to the physical address of the target data.

[0047] Upon receiving a write request, this application calculates the corresponding target fingerprint value and queries the target fingerprint value in the Cuckoo Filter. Once found, the current logical address of the target data is directly added. If the target fingerprint value is not found, the target data is stored and metadata mapping of the target fingerprint value and the physical address of the target data is added. This eliminates the need to query the B+ tree and allows direct querying of the corresponding metadata mapping using the Cuckoo Filter, reducing the frequency of deleting metadata and querying the B+ tree on the disk, thus improving the performance of the entire storage system.

[0048] This application also provides a deduplication query device, a deduplication query equipment, and a computer-readable storage medium, which have the above-mentioned beneficial effects, and will not be elaborated here. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a deduplication query method provided in an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the structure of a deduplication query device provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the structure of a deduplication query device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] See Figure 1 , Figure 1 This is a flowchart illustrating a deduplication query method provided in an embodiment of the present invention. The specific scheme includes:

[0055] S101: Receive a write request and determine the target data corresponding to the currently received write request;

[0056] This step aims to receive write requests, which are typically I / O requests, i.e., write I / O requests. Of course, other forms of write requests are also possible.

[0057] After receiving a write request, the target data corresponding to the write request can be further determined, that is, the target data to be operated on by the write request, or the data to be written by the write request.

[0058] There are no restrictions on how to determine the target data. You can first determine the data block corresponding to the currently received write request, and then determine the corresponding target data from the data block. Alternatively, you can determine the target data based on information such as the address contained in the write request.

[0059] S102: Calculate the fingerprint value of the target data to obtain the target fingerprint value, and query the target fingerprint value in the Cuckoo Filter;

[0060] This step aims to calculate the fingerprint value of the target data. There is no limitation on how the fingerprint value is calculated. A preset hash algorithm can be used to calculate the fingerprint value of the target data. The hash algorithm can be the original hash algorithm or any variant algorithm evolved from the hash algorithm. As long as it can calculate the fingerprint value of the target data, it can be applied to this step.

[0061] The target fingerprint value then needs to be queried in the Cuckoo Filter. The Cuckoo Filter uses an array structure plus a linked list structure to store fingerprint values, with the same fingerprint value stored in two positions within the array. The fingerprint value stored in the array is a rehashed version of a portion of the hash corresponding to the physical address.

[0062] The Cuckoo Filter, by its conventional design, uses only two hash tables, ensuring that a lookup can be completed in just two memory accesses. Compared to the Bloom Filter, which uses K hash functions and requires K memory accesses, the Cuckoo Filter significantly outperforms the Bloom Filter when dealing with large datasets that cannot be fully loaded into memory. Deduplication metadata queries do not require absolute accuracy; misjudgments only affect the deduplication rate, not data consistency. Therefore, the Cuckoo Filter is well-suited for deduplication metadata queries.

[0063] When calculating the insertion position of the hash value in the array structure using the hash algorithm included in the Cuckoo Filter, if an element already exists at the insertion position, a partial bit of the hash value can be obtained through a hash reverse lookup function; then, it is determined whether the partial bits are the same. If they are the same, a precise HP lookup is performed on the hash value; otherwise, the hash value is inserted into the linked list structure corresponding to the array structure.

[0064] Specifically, when performing this step, the Cuckoo Filter can be used to reduce the query precision of the hash value query interface queryHpinfo, changing it from an exact query to a fuzzy query, thereby improving query efficiency.

[0065] It is easy to understand that this embodiment assumes that the fingerprint value of the data can be stored in the cuckoo filter before this step. Then, step S103 is executed based on whether the target fingerprint value exists in the cuckoo filter.

[0066] S103: If the target fingerprint value is found in the cuckoo filter, then add the metadata mapping from the current logical address of the target data to the physical address of the target data; otherwise, store the target data, add the target fingerprint value to the cuckoo filter, and add the metadata mapping from the current logical address of the target data to the physical address of the target data.

[0067] This step involves different processing depending on whether the target fingerprint value is being retrieved. In other words, different processing methods are applied depending on whether the target fingerprint value can be retrieved.

[0068] If the target fingerprint value can be found, the metadata mapping from the current logical address of the target data to its physical address can be directly added. This reduces the frequency of B+ tree queries, thereby improving system performance.

[0069] If the target fingerprint value cannot be found, it indicates that the Cuckoo Filter did not previously store the target data. The target data can be stored first, and the target fingerprint value added to the Cuckoo Filter. A metadata mapping from the current logical address of the target data to its physical address should also be added. This facilitates subsequent target fingerprint queries; that is, a target fingerprint value is created for the target data even if one was not found initially. Therefore, the next time a target fingerprint value is queried, the process of finding the found target fingerprint value can be followed.

[0070] In other words, this step actually involves two completely different execution processes. Simply put, regardless of whether the target fingerprint value is found, it is necessary to add metadata mapping from the current logical address to the physical address of the target data. The only difference is that if the target fingerprint value can be found, there is no need to store the target data or add the target fingerprint value to the Cuckoo Filter.

[0071] Furthermore, before adding the metadata mapping from the current logical address of the target data to its physical address, we can first obtain the logical address of the write request corresponding to the target data and determine that this logical address is the current logical address of the target data. In other words, the logical address of the write request is used as the current logical address of the target data. The physical address of the target data can be directly obtained and will not be discussed here.

[0072] For metadata mapping, the controller on the solid-state drive can be used to manage the metadata mapping between the physical address and the corresponding logical address. Of course, other storage units can also be used to store the metadata mapping. It should be noted that storing and managing the metadata mapping can be performed by two separate entities, or it can be directly managed by a single entity, such as the controller on the solid-state drive.

[0073] Furthermore, after the storage pool is created, physical addresses can be consolidated and remapped, and logical addresses can be re-encoded using the pool information and storage offset information. No specific limitations are placed on how the logical addresses are re-encoded; this can be configured and set by those skilled in the art based on the relationship between physical and logical addresses.

[0074] In this embodiment, upon receiving a write request, the corresponding target fingerprint value is calculated, and the target fingerprint value is queried in the Cuckoo Filter. Once the target fingerprint value is found, the current logical address of the target data is directly added. If the target fingerprint value is not found, the target data is stored, and metadata mapping of the target fingerprint value and the physical address of the target data is added. Therefore, there is no need to query the B+ tree. The corresponding metadata mapping can be directly queried using the Cuckoo Filter, reducing the frequency of deleting metadata and querying the B+ tree on the disk, thus improving the performance of the entire storage system.

[0075] Based on the above embodiments, as a preferred embodiment, if a target fingerprint value is found in the Cuckoo Filter, it can also be queried whether there is a metadata mapping from the fingerprint value to the physical address to which the target fingerprint value belongs. If it exists, the physical address in the metadata mapping from the fingerprint value to the physical address to which the target fingerprint value belongs is determined to be the physical address of the target data, and a metadata mapping from the current logical address of the target data to the physical address of the target data is added. Otherwise, the target data is stored.

[0076] There are no restrictions on how to query the metadata mapping here. You can query the HP metadata tree to see if there is a metadata mapping from the target fingerprint value to the physical address. The HP metadata tree is a mapping from H to P, where H is a 64-bit fingerprint value and P is a physical address where the data is stored. Furthermore, this HP metadata tree can specifically be an HP metadata B+ tree.

[0077] The corresponding process in this embodiment is as follows:

[0078] The first step is to receive the write request and determine that the data corresponding to the currently received write request is the target data.

[0079] The second step is to calculate the fingerprint value of the target data to obtain the target fingerprint value, and then query the target fingerprint value in the cuckoo filter. If the target fingerprint value is found in the cuckoo filter, proceed to the third step; if the target fingerprint value is not found in the cuckoo filter, proceed to the seventh step.

[0080] The third step is to add metadata mapping from the current logical address of the target data to the physical address of the target data;

[0081] Step 4: Check if there is a metadata mapping from the target fingerprint value to the physical address; if it exists, proceed to step 5; otherwise, proceed to step 6.

[0082] Specifically, when querying whether a metadata mapping exists, one can check the HP metadata tree to see if there is a metadata mapping from the target fingerprint value to the physical address. The HP metadata tree is not limited here; one feasible approach is to use an HP metadata B+ tree.

[0083] The fifth step is to determine the physical address in the metadata mapping from the fingerprint value to the physical address of the target fingerprint value as the physical address of the target data, and to add the metadata mapping from the current logical address of the target data to the physical address of the target data.

[0084] Step 6: Store the target data.

[0085] Step 7: Store the target data, add the target fingerprint value to the cuckoo filter, and add metadata mapping from the current logical address of the target data to the physical address of the target data.

[0086] As can be seen, this embodiment adds a query process for the metadata mapping from the fingerprint value to the physical address of the target fingerprint value, based on the previous embodiment. This can more quickly determine the physical address of the target data, thereby quickly executing the process of establishing the metadata mapping from the logical address to the physical address of the target data, which is beneficial for the establishment of the target fingerprint value and subsequent query.

[0087] See Figure 2 , Figure 2 A schematic diagram of a deduplication query device provided in an embodiment of the present invention includes:

[0088] The receiving module is used to receive write requests and determine the target data corresponding to the currently received write request;

[0089] The query module is used to calculate the fingerprint value of the target data to obtain the target fingerprint value, and query the target fingerprint value in the Cuckoo Filter;

[0090] The execution module is configured to, if the target fingerprint value is found in the cuckoo filter, add a metadata mapping from the logical address in the write request corresponding to the target data to the physical address of the target data; otherwise, store the target data, add the target fingerprint value to the cuckoo filter, and add a metadata mapping from the logical address in the write request corresponding to the target data to the physical address of the target data.

[0091] Based on the above embodiments, as a preferred embodiment, it further includes:

[0092] The mapping query module is used to query whether there is a metadata mapping from fingerprint value to physical address to which the target fingerprint value belongs if the target fingerprint value is found in the cuckoo filter. If it exists, the physical address in the metadata mapping from fingerprint value to physical address to which the target fingerprint value belongs is determined to be the physical address of the target data, and the step of adding the current logical address of the target data to the metadata mapping of the physical address of the target data is executed. Otherwise, the step of storing the target data is executed.

[0093] Based on the above embodiments, as a preferred embodiment, it further includes:

[0094] The filter setting module is used to set the array structure and linked list structure corresponding to the cuckoo filter; wherein, the same fingerprint value is stored in two positions in the array structure, and the fingerprint value is a secondary hash value made by hashing a portion of the bits corresponding to the physical address.

[0095] Based on the above embodiments, as a preferred embodiment, it further includes:

[0096] The position calculation module is used to calculate the insertion position of the hash value in the array structure using the hash algorithm contained in the cuckoo filter.

[0097] Based on the above embodiments, as a preferred embodiment, it further includes:

[0098] The insertion module is used to obtain a portion of the hash value using a hash reverse lookup function when an element exists at the insertion position; determine whether the portion of the bits are the same; if they are the same, perform a precise HP lookup on the hash value; if they are different, insert the hash value into the linked list structure corresponding to the array structure.

[0099] Based on the above embodiments, as a preferred embodiment, it further includes:

[0100] The mapping relationship management module is used to manage the mapping relationship between the physical address and the corresponding logical address using the controller on the solid-state drive; wherein, after the storage pool is created, the physical address is integrated and remapped; and the logical address is recoded using the pool information and storage offset position information of the storage pool.

[0101] Based on the above embodiments, as a preferred embodiment, the receiving module includes:

[0102] The data determination unit is used to determine that the data in the data block corresponding to the currently received write request is the target data;

[0103] Accordingly, the query module includes:

[0104] The hash calculation unit is used to calculate the fingerprint value of the target data using a preset hash algorithm to obtain the target fingerprint value.

[0105] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can implement the steps of the methods provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0106] This application also provides a deduplication query device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps of the method provided in the above embodiments. Of course, the deduplication query device may also include various network interfaces, power supplies, and other components. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of the structure of a deduplication query device provided in an embodiment of this application. The deduplication query device in this embodiment may include: a processor 2101 and a memory 2102.

[0107] Optionally, the deduplication query device may also include a communication interface 2103, an input unit 2104, a display 2105, and a communication bus 2106.

[0108] The processor 2101, memory 2102, communication interface 2103, input unit 2104, and display 2105 all communicate with each other through the communication bus 2106.

[0109] In this embodiment of the application, the processor 2101 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, an off-the-shelf programmable gate array, or other programmable logic devices.

[0110] The processor can call programs stored in memory 2102. Specifically, the processor can execute the operations performed by the deduplication query device in the above embodiment.

[0111] The memory 2102 is used to store one or more programs, which may include program code, including computer operation instructions. In this embodiment, the memory stores at least a program for implementing the following functions:

[0112] Receive write requests and determine that the data corresponding to the currently received write request is the target data;

[0113] Calculate the fingerprint value of the target data to obtain the target fingerprint value, and then query the target fingerprint value in the Cuckoo Filter;

[0114] If the target fingerprint value is found in the cuckoo filter, then a metadata mapping from the current logical address of the target data to the physical address of the target data is added; otherwise, the target data is stored, the target fingerprint value is added to the cuckoo filter, and a metadata mapping from the current logical address of the target data to the physical address of the target data is added.

[0115] In one possible implementation, the memory 2102 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during the use of the computer.

[0116] In addition, memory 2102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0117] The communication interface 2103 can be an interface for a communication module, such as the interface for a GSM module.

[0118] This application may also include a display 2105 and an input unit 2104, etc.

[0119] Figure 3 The structure of the deduplication query device shown does not constitute a limitation on the deduplication query device in the embodiments of this application. In practical applications, the deduplication query device may include devices such as... Figure 3 More or fewer components as shown, or combinations of certain components.

[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. As the system provided in the embodiments corresponds to the method provided in the embodiments, the description is relatively simple; relevant parts can be found in the method section.

[0121] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0122] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A deduplication query method, characterized in that, Applications in all-flash storage systems include: Receive write requests and determine that the data corresponding to the currently received write request is the target data; Calculate the fingerprint value of the target data to obtain the target fingerprint value, and then query the target fingerprint value in the Cuckoo Filter; If the target fingerprint value is found in the cuckoo filter, then the metadata mapping from the current logical address of the target data to the physical address of the target data is added; otherwise, the target data is stored, the target fingerprint value is added to the cuckoo filter, and the metadata mapping from the current logical address of the target data to the physical address of the target data is added; the logical address of the write request is used as the current logical address of the target data. If the target fingerprint value is found in the cuckoo filter, the following is also included: Check if there is a metadata mapping from the target fingerprint value to its physical address; If it exists, determine the physical address in the metadata mapping from fingerprint value to physical address to which the target fingerprint value belongs as the physical address of the target data, and execute the step of adding the current logical address of the target data to the metadata mapping of the physical address of the target data; otherwise, execute the step of storing the target data. Accordingly, after storing the target data, the target fingerprint value is added to the metadata mapping of the physical address of the target data; Before querying the target fingerprint value in the cuckoo filter, the method further includes: setting the array structure and linked list structure corresponding to the cuckoo filter; wherein, the same fingerprint value is stored in two positions in the array structure, and the fingerprint value is a secondary hash value made on a portion of the hash corresponding to the physical address; the cuckoo filter has two hash tables; When inserting the hash value of the data at the physical address into the array structure, it also includes: The insertion position of the hash value in the array structure is calculated using the hash algorithm contained in the cuckoo filter; When calculating the insertion position of the hash value in the array structure using the hash algorithm included in the cuckoo filter, if an element exists at the insertion position, the method further includes: A portion of the hash value is obtained using a reverse hash lookup function; Determine whether the aforementioned bits are the same; If they are the same, perform a precise HP lookup on the hash value; If they are different, the hash value is inserted into the linked list structure corresponding to the array structure.

2. The deduplication query method according to claim 1, characterized in that, Also includes: The mapping relationship between the physical address and the corresponding logical address is managed using the controller on the solid-state drive; Specifically, after the storage pool is created, the physical addresses are integrated and remapped; the logical addresses are recoded using the pool information and storage offset information of the storage pool.

3. The deduplication query method according to claim 1, characterized in that, Determine that the data corresponding to the currently received write request is the target data, including: Determine that the data in the data block corresponding to the currently received write request is the target data; Accordingly, calculating the fingerprint value of the target data to obtain the target fingerprint value includes: The target fingerprint value is obtained by calculating the fingerprint value of the target data using a preset hash algorithm.

4. A deduplication query device, characterized in that, Applications in all-flash storage systems include: The receiving module is used to receive write requests and determine that the data corresponding to the currently received write request is the target data; The query module is used to calculate the fingerprint value of the target data to obtain the target fingerprint value, and query the target fingerprint value in the Cuckoo Filter; An execution module is configured to, if the target fingerprint value is found in the cuckoo filter, add a metadata mapping from the logical address in the write request corresponding to the target data to the physical address of the target data; otherwise, store the target data, add the target fingerprint value to the cuckoo filter, and add a metadata mapping from the logical address in the write request corresponding to the target data to the physical address of the target data. The mapping query module is used to query whether there is a metadata mapping from fingerprint value to physical address to which the target fingerprint value belongs if the target fingerprint value is found in the cuckoo filter. If it exists, the physical address in the metadata mapping from fingerprint value to physical address to which the target fingerprint value belongs is determined to be the physical address of the target data, and the step of adding the current logical address of the target data to the metadata mapping of the physical address of the target data is executed. Otherwise, the step of storing the target data is executed. The filter setting module is used to set the array structure and linked list structure corresponding to the cuckoo filter; wherein, the same fingerprint value is stored in two positions in the array structure, and the fingerprint value is a secondary hash value made by hashing a portion of the bits corresponding to the physical address; the cuckoo filter has two hash tables; The position calculation module is used to calculate the insertion position of the hash value in the array structure using the hash algorithm contained in the cuckoo filter; The insertion module is used to obtain a portion of the hash value using a hash reverse lookup function when an element exists at the insertion position; determine whether the portion of the bits are the same; if they are the same, perform a precise HP lookup on the hash value; if they are different, insert the hash value into the linked list structure corresponding to the array structure.

5. A deduplication query device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the deduplication query method as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the deduplication query method as described in any one of claims 1 to 3.

Images