Storage device, fingerprint index construction method, query method and equipment

Abstract

The invention discloses a storage device, a fingerprint index construction method, a query method and equipment, in the application, the storage device stores a fingerprint index, the fingerprint index comprises a first index layer and a second index layer, the first index layer is used for storing first fields of fingerprints in a first fingerprint set, and the second index layer is used for storing second fields of fingerprints in a second fingerprint set; the second index layer is used for storing the second fields of the fingerprints in the first fingerprint set. The first index layer comprises a plurality of index entries, and one index entry in the first index layer represents one value of the first field; one index entry of the first index layer corresponds to one group of index entries of the second index layer; one index entry of the second index layer represents one value of the second field. The structure of the fingerprint index is simpler, the storage space occupied by the fingerprint index can be effectively reduced, and the construction difficulty of the fingerprint index is reduced. Due to the corresponding relation between the index layers in the fingerprint index, jumping between the index layers is simpler, and the query efficiency of the fingerprint index is improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a storage device, a fingerprint index construction method, a query method, and an apparatus. Background Technology

[0002] For storage systems used to store backup data or those with frequently updated data, deduplication is often required when writing new data. Deduplication is performed based on a fingerprint table: First, for any data shard within the data, the lookup key in the fingerprint table is a key-value pair whose value is the fingerprint of that data shard. If the key-value pair is found, it means that the data shard is already stored in the storage system, and it can be deleted during data deduplication. If the key-value pair is not found, it means that the data shard is not stored in the storage system, and it does not need to be deleted during data deduplication.

[0003] Typically, a fingerprint table records multiple key-value pairs, the number of which is the same as the number of data shards stored in the storage system. The larger the number of data shards in the storage system, the larger the fingerprint table will be, and the more storage space it will occupy. Summary of the Invention

[0004] This application provides a storage device, a fingerprint index construction method, a query method, and an apparatus to reduce the storage space occupied by the fingerprint index.

[0005] In a first aspect, embodiments of this application also provide a storage device for storing a fingerprint index. The fingerprint index includes multiple index layers. The structure of the fingerprint index is described below using two adjacent layers as an example. For ease of distinction, these two adjacent index layers are referred to as the first index layer and the second index layer, respectively. The first index layer is used to store the first field of the fingerprint in the first fingerprint set (such as field A in the embodiments of this application), and the second index layer is used to store the second field of the fingerprint in the first fingerprint set (such as field B in the embodiments of this application).

[0006] The first index layer comprises multiple index entries, each representing a value of the first field. Each index entry in the first index layer corresponds to a set of index entries in the second index layer; each index entry in the second index layer represents a value of the second field. All index entries within the same index layer have the same length.

[0007] In this storage device, the fingerprint index has a multi-layered structure. Each index entry in the fingerprint index represents the value of a specific field in the fingerprint, and there is a correspondence between adjacent layers. This eliminates the need to store a large number of fingerprints, effectively reducing the storage space occupied by the fingerprint index. The fingerprint index structure is simpler, reducing the difficulty of constructing it. Due to the correspondence between index layers, determining an index entry in the upper layer allows for locating a set of index entries in the lower layer, simplifying navigation between index layers and improving the query efficiency of the fingerprint index.

[0008] In one possible implementation, the second index layer is also used to store information related to fingerprints in the first fingerprint set. The information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0009] In this storage device, the second index layer stores relevant information about fingerprints in the first fingerprint set, making the fingerprint index applicable to different query scenarios.

[0010] In one possible implementation, the fingerprint index further includes a third index layer. The third index layer is used to store the third field (such as field C in the embodiments of this application) of the fingerprints in the first fingerprint set and related information of the fingerprints. An index item in the second index layer corresponds to a set of index items in the third index layer. An index item in the third index layer represents a value of the third field. The related information of the fingerprint includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0011] In this storage device, the third index layer stores relevant information about fingerprints in the first fingerprint set, making the fingerprint index suitable for different query scenarios.

[0012] In one possible implementation, the third field includes a first subfield and a second subfield. The third index layer includes at least one set of buckets. An index entry in the second index layer corresponds to a set of buckets in the third index layer. Each bucket in the set of buckets includes one or more index entries. Each bucket represents a value of the first subfield, and an index entry in each bucket represents a value of the second subfield.

[0013] In this storage device, in the third index layer, index items with the same value in the first sub-field are clustered into a bucket, that is, the bucket is used to represent the value of the first sub-field. The bucket setting makes it possible to locate a bucket in the group when jumping to a group of buckets in the third index layer by using the specific value of the first sub-field, which further reduces the number of index items that need to be jumped to when jumping from the second index layer to the third index layer.

[0014] In one possible implementation, multiple index entries in the first index layer include a first index entry, and a set of index entries in the second index layer corresponding to the first index entry includes a second index entry. The second index entry stores relevant information about the first fingerprint in the first fingerprint set. The value of the first field in the first fingerprint is consistent with the value represented by the first index entry, and the value of the second field in the first fingerprint is consistent with the value represented by the second index entry.

[0015] In this storage device, a first index entry and its corresponding second index entry can represent a first fingerprint in a first fingerprint set. Therefore, in this fingerprint index, corresponding index entries in each index layer can represent a fingerprint in the first fingerprint set.

[0016] In one possible implementation, multiple index entries in the first index layer include a third index entry, a set of index entries in the second index layer corresponding to the third index entry includes a fourth index entry, a set of index entries in the third index layer corresponding to the fourth index entry includes a fifth index entry, and the fifth index entry stores relevant information about the second fingerprint in the first fingerprint set. The value of the first field in the second fingerprint is consistent with the value represented by the third index entry, the value of the second field in the second fingerprint is consistent with the value represented by the fourth index entry, and the value of the third field in the second fingerprint is consistent with the value represented by the fifth index entry.

[0017] In this storage device, the third index entry and its corresponding fourth and fifth index entries can represent the second fingerprint in the first fingerprint set. Within this fingerprint index, corresponding index entries in each index layer can represent the fingerprints in the first fingerprint set.

[0018] In one possible implementation, the first fingerprint set includes fingerprints obtained by sampling from a second fingerprint set based on a sampling ratio, the second fingerprint set including fingerprints of data in a storage system or storage node, and the number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

[0019] In this storage device, the first fingerprint set includes only a portion of the fingerprints in the second fingerprint set, thereby reducing the number of fingerprints recorded in the fingerprint index and further reducing the storage space occupied by the fingerprint index.

[0020] In one possible implementation, the upper and lower limits of fingerprints in the first fingerprint set can be determined based on the sampling ratio. For example, the fingerprint values ​​in the first fingerprint set are greater than a first value (the first value is the lower limit) and less than a second value (the second value is the upper limit). The upper and / or lower limits of fingerprints in the first fingerprint set are related to the sampling ratio. For example, the second value is determined based on the sampling ratio. Or, for another example, the first value is determined based on the sampling ratio.

[0021] In this storage device, the sampling ratio can determine the upper and lower limits of fingerprints in the first fingerprint set. When the sampling ratio is changed, only the upper or lower limit of fingerprints in the first fingerprint set needs to be changed. When only the upper or lower limit of fingerprints in the first fingerprint set is changed, the changes in the fingerprints included in the first fingerprint set are small.

[0022] In one possible implementation, the fingerprint index is distributed across N partitions of the storage device, where each partition is a segment of storage space and N is a positive integer.

[0023] In this storage device, the fingerprint index can be stored in partitions, and the storage method of the fingerprint index is relatively flexible.

[0024] In one possible implementation, the storage device comprises N nodes, each node including a partition. A node can be a memory or a computing device.

[0025] In this storage device, fingerprint indexes can be stored across nodes, effectively expanding the applicable storage methods for fingerprint indexes and making their deployment more flexible.

[0026] Secondly, this application also provides a fingerprint index construction method, the related beneficial effects of which can be found in the relevant description of the first aspect and will not be repeated here. The method is used to construct a fingerprint index for a first fingerprint set, the fingerprint of the first fingerprint set including a first field and a second field. The method can be executed by a construction device. Here, we take the construction of two index layers in the fingerprint index as an example for illustration. In the method: the construction device constructs the first index layer of the fingerprint index and the second index layer of the fingerprint index.

[0027] When constructing the first index layer of the fingerprint index, the construction device sets up multiple index entries in the first index layer. Each index entry in the first index layer represents a value of the first field. That is, the construction device can use the first index entry layer to store the first field. Each index entry represents a value of the first field.

[0028] When constructing the second index layer of the fingerprint index, the construction device sets up multiple sets of index entries in the second index layer. Each index entry in the first index layer corresponds to one set of index entries in the second index layer, and each index entry in the set of index entries in the second index layer represents a value of the second field. That is, the construction device can use the second index layer to store the second field. Each index entry represents a value of the second field.

[0029] Using the above method, the construction device can quickly construct the fingerprint index, and the structure of the fingerprint index is relatively simple.

[0030] In one possible implementation, the index entries of the second index layer store information related to fingerprints in the first fingerprint set. The information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0031] In one possible implementation, the fingerprints in the first fingerprint index set also include a third field, and the construction device can construct a third index layer of the fingerprint index.

[0032] When constructing the third index of the fingerprint index, the construction device sets up multiple sets of index entries in the third index layer. One index entry in the second index layer corresponds to one set of index entries in the third index layer. One index entry in the set of index entries in the third index layer represents a value of the third field. The index entries in the third index layer store the relevant information of the fingerprints in the first fingerprint set. The relevant information of the fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0033] In one possible implementation, the third field includes a first subfield and a second subfield. When the construction device sets multiple sets of index entries in the third index layer, it sets at least one set of buckets in the third index layer. An index entry in the second index layer corresponds to a set of buckets in the third index layer. Each bucket in the set of buckets includes one or more index entries. Each bucket represents a value of the first subfield, and an index entry in each bucket represents a value of the second subfield.

[0034] In one possible implementation, multiple index entries in the first index layer include a first index entry, and a set of index entries in the second index layer corresponding to the first index entry includes a second index entry; the second index entry stores relevant information about the first fingerprint in the first fingerprint set, the value of the first field in the first fingerprint is consistent with the value represented by the first index entry, and the value of the second field in the first fingerprint is consistent with the value represented by the second index entry.

[0035] In one possible implementation, multiple index entries in the first index layer include a third index entry, a set of index entries in the second index layer corresponding to the third index entry includes a fourth index entry, a set of index entries in the third index layer corresponding to the fourth index entry includes a fifth index entry, and the fifth index entry stores relevant information about the second fingerprint in the first fingerprint set. The value of the first field in the second fingerprint is consistent with the value represented by the third index entry, the value of the second field in the second fingerprint is consistent with the value represented by the fourth index entry, and the value of the third field in the second fingerprint is consistent with the value represented by the fifth index entry.

[0036] In one possible implementation, the construction module samples fingerprints from the second fingerprint set based on a sampling ratio. The first fingerprint set includes the sampled fingerprints, and the second fingerprint set includes fingerprints from data in the storage system or storage node. The number of fingerprints in the first fingerprint set is less than the number of fingerprints in the second fingerprint set.

[0037] In one possible implementation, the upper and lower limits of fingerprints in the first fingerprint set can be determined based on the sampling ratio. For example, the fingerprint values ​​in the first fingerprint set are greater than a first value (the first value is the lower limit) and less than a second value (the second value is the upper limit). The upper and / or lower limits of fingerprints in the first fingerprint set are related to the sampling ratio. For example, the second value is determined based on the sampling ratio. Or, for another example, the first value is determined based on the sampling ratio.

[0038] In one possible implementation, the fingerprint index is distributed across N partitions of the storage device, where each partition is a segment of storage space and N is a positive integer.

[0039] In one possible implementation, the storage device comprises N nodes, each node including a partition.

[0040] Thirdly, this application also provides a fingerprint query method, the related beneficial effects of which are described above and will not be repeated here. The method is used to query a target fingerprint based on a fingerprint index in a first fingerprint set. The fingerprint index is used to store the first fingerprint set, where the fingerprints in the first fingerprint set are fingerprints of data in a data set.

[0041] The fingerprint index includes a first index layer and a second index layer. The first index layer is used to store the first field of the fingerprints in the first fingerprint set, and the second index layer is used to store the second field of the fingerprints in the first fingerprint set. One index entry in the first index layer corresponds to a set of index entries in the second index layer.

[0042] There are many scenarios for querying a target fingerprint in a first fingerprint set. For example, determining whether a dataset includes target data corresponding to the target fingerprint. Another example is determining whether a particular fingerprint is already stored in the first fingerprint set. This application does not limit the scenarios to which this fingerprint query method is applicable. The fingerprint query method provided in this application is applicable to any scenario that requires determining whether a target fingerprint exists in the first fingerprint set. This method is executed by a query device, in which:

[0043] The query device determines whether a first index entry exists among multiple index entries in the first index layer, and the value of the first index entry is consistent with the value of the first field of the target fingerprint.

[0044] If a first index term exists among multiple index terms in the first index layer, determine whether a second index term exists among a set of index terms in the second index layer corresponding to the first index term. The value represented by the second index term is consistent with the value of the second field in the target fingerprint.

[0045] If a second index term exists in a set of index terms of the second index layer corresponding to the first index term, it indicates that the target fingerprint exists in the first fingerprint set.

[0046] Using the above method, when performing a query based on the fingerprint index, the query device can easily and conveniently jump from the first index layer to the second index layer based on the correspondence between the first index layer and the second index layer, and locate a set of index items in the second index layer. The fingerprint query method is simpler and more efficient.

[0047] In one possible implementation, the querying device obtains relevant information about the target fingerprint from the second index entry. The relevant information about the target fingerprint includes some or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the region where the target data is located.

[0048] Using the above method, after locating the second index item, the query device can quickly obtain relevant information about the target fingerprint from the second index item.

[0049] In one possible implementation, the fingerprint index also includes a third index layer, which stores the third field of the fingerprints in the first fingerprint set. One of the index entries in the third index layer represents a value of the third field.

[0050] The query device determines whether a third index entry exists within a set of index entries in the third-level index corresponding to the second index entry. The value represented by the third index entry is consistent with the value of the third field in the target fingerprint. If a third index entry exists within a set of index entries in the third-level index corresponding to the second index entry, the query device retrieves relevant information about the target fingerprint from the third index entry. This relevant information includes some or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the region where the target data is located.

[0051] Using the above method, after the query device jumps from the second index layer to the third index layer, it can locate the third index item in the third index layer and then obtain relevant information about the target fingerprint from the second index item.

[0052] In one possible implementation, the third index layer is also used to store information related to fingerprints in the first fingerprint set. The information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0053] In one possible implementation, the third field includes a first subfield and a second subfield. The third index layer includes at least one set of buckets. An index entry in the second index layer corresponds to a set of buckets in the third index layer. Each set of buckets includes one or more buckets. Each bucket includes one or more index entries. Each bucket represents a value of the first subfield. An index entry in each bucket represents a value of the second subfield.

[0054] When the query device determines whether a third index term exists in a set of index terms in the third-level index corresponding to the second index term, it determines the target bucket in a set of buckets in the third-level index corresponding to the second index term. The value represented by the target bucket is consistent with the value of the first sub-field in the target fingerprint.

[0055] If a third index term exists in a set of index terms in the third-level index corresponding to the second index term, the value represented by the third index term is consistent with the value of the second sub-field in the target fingerprint.

[0056] Using the above method, multiple index layers are aggregated together in the third index layer by buckets. When the query device jumps to a group of buckets in the third index layer, it can quickly narrow the search scope to a target bucket in that group of buckets by using the value of the first sub-field of the target fingerprint.

[0057] In one possible implementation, if the first index entry, the second index entry, or the third index entry does not exist, the query device can determine that the target fingerprint does not exist in the first fingerprint set.

[0058] Using the above method, if no corresponding index entry is found at any index layer, it can be determined that the target fingerprint does not exist in the first fingerprint set, and the query process can be stopped.

[0059] In one possible implementation, the first fingerprint set includes fingerprints obtained by sampling from a second fingerprint set based on a sampling ratio, the second fingerprint set including fingerprints of data in a storage system or storage node, and the number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

[0060] In one possible implementation, the upper and lower limits of fingerprints in the first fingerprint set can be determined based on the sampling ratio. For example, the fingerprint values ​​in the first fingerprint set are greater than a first value (the first value is the lower limit) and less than a second value (the second value is the upper limit). The upper and / or lower limits of fingerprints in the first fingerprint set are related to the sampling ratio. For example, the second value is determined based on the sampling ratio. Or, for another example, the first value is determined based on the sampling ratio.

[0061] In one possible implementation, the fingerprint index is distributed across N partitions of the storage device, where each partition is a segment of storage space and N is a positive integer.

[0062] In one possible implementation, the storage device comprises N nodes, each node including a partition.

[0063] Fourthly, this application also provides a fingerprint index, the relevant details of which can be found in the foregoing description and will not be repeated here. The fingerprint index includes a first index layer and a second index layer. The first index layer is used to store the first field of fingerprints in the first fingerprint set, and the second index layer is used to store the second field of fingerprints in the first fingerprint set.

[0064] The first index layer includes multiple index entries. One index entry in the first index layer represents a value of the first field. One index entry in the first index layer corresponds to a set of index entries in the second index layer. One index entry in the second index layer represents a value of the second field.

[0065] In one possible implementation, the second index layer is also used to store information related to fingerprints in the first fingerprint set. The information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0066] In one possible implementation, the fingerprint index further includes a third index layer, which stores the third field of the fingerprints in the first fingerprint set and related information about the fingerprints. One index entry in the second index layer corresponds to a set of index entries in the third index layer. One index entry in the third index layer represents a value of the third field. The related information about the fingerprint includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0067] In one possible implementation, the third field includes a first subfield and a second subfield. The third index layer includes at least one set of buckets. An index entry in the second index layer corresponds to a set of buckets in the third index layer. Each bucket in the set of buckets includes one or more index entries. Each bucket represents a value of the first subfield, and an index entry in each bucket represents a value of the second subfield.

[0068] In one possible implementation, multiple index entries in the first index layer include a first index entry, and a set of index entries in the second index layer corresponding to the first index entry includes a second index entry. The second index entry stores relevant information about the first fingerprint in the first fingerprint set. The value of the first field in the first fingerprint is consistent with the value represented by the first index entry, and the value of the second field in the first fingerprint is consistent with the value represented by the second index entry.

[0069] In one possible implementation, multiple index entries in the first index layer include a third index entry, a set of index entries in the second index layer corresponding to the third index entry includes a fourth index entry, a set of index entries in the third index layer corresponding to the fourth index entry includes a fifth index entry, and the fifth index entry stores relevant information about the second fingerprint in the first fingerprint set. The value of the first field in the second fingerprint is consistent with the value represented by the third index entry, the value of the second field in the second fingerprint is consistent with the value represented by the fourth index entry, and the value of the third field in the second fingerprint is consistent with the value represented by the fifth index entry.

[0070] In one possible implementation, the first fingerprint set includes fingerprints obtained by sampling from a second fingerprint set based on a sampling ratio, the second fingerprint set including fingerprints of data in a storage system or storage node, and the number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

[0071] In one possible implementation, the fingerprint values ​​in the first fingerprint set are greater than a first value and less than a second value, which is determined based on the sampling ratio.

[0072] In one possible implementation, the fingerprint values ​​in the first fingerprint set are greater than a first value and less than a second value, where the first value is determined based on the sampling ratio.

[0073] In one possible implementation, the fingerprint index is distributed across N partitions of the storage device, where each partition is a segment of storage space and N is a positive integer.

[0074] In one possible implementation, the storage device comprises N nodes, each node including a partition.

[0075] Fifthly, embodiments of this application also provide a construction apparatus that has the function of implementing the behavior in the method example of the second aspect described above. The beneficial effects can be found in the description of the second aspect and will not be repeated here. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the construction apparatus includes a first construction module, a second construction module, and optionally, a third construction module, a sampling module, and a storage module. These modules can perform the corresponding functions in the method example of the second aspect described above; see the detailed description in the method example for details, which will not be repeated here.

[0076] Sixthly, this application also provides a query device that performs the actions described in the method examples of the third aspect. The beneficial effects are described in the third aspect and will not be repeated here. The functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. In one possible design, the device structure includes a first query module, a second query module, and optionally, a third query module. These modules can perform the corresponding functions in the method examples of the third aspect, as detailed in the method examples, and will not be repeated here.

[0077] Seventhly, this application also provides a computing device, including a processor and a memory, and may further include a communication interface. The processor executes computer program instructions in the memory to perform the method provided by the second aspect or any possible implementation thereof. Alternatively, the processor executes computer program instructions in the memory to perform the method provided by the third aspect or any possible implementation thereof. The memory is coupled to the processor and stores computer program instructions and data necessary for determining the fingerprint construction process or fingerprint query process. The communication interface is used for communicating with other devices, such as acquiring a target fingerprint.

[0078] Eighthly, this application provides a computing device system including at least one computing device. Each computing device includes a memory and a processor. The processor of at least one computing device is configured to access computer program instructions in the memory to execute the method provided in the second aspect or any possible implementation thereof, or the processor of at least one computing device is configured to access computer program instructions in the memory to execute the method provided in the third aspect or any possible implementation thereof.

[0079] Ninthly, this application provides a computer-readable storage medium that, when executed by a computing device, allows the computing device to perform the method provided in the second aspect or any possible implementation thereof, or to perform the method provided in the third aspect or any possible implementation thereof. The storage medium stores computer program instructions. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0080] Tenthly, this application provides a computer device program product, which includes computer program instructions. When executed by a computing device, the computing device performs the method provided in the second aspect or any possible implementation thereof, or performs the method provided in the third aspect or any possible implementation thereof. The computer program product can be a software installation package. When it is necessary to use the method provided in the second aspect or any possible implementation thereof, or when it is necessary to use the method provided in the third aspect or any possible implementation thereof, the computer program product can be downloaded and executed on the computing device.

[0081] In the eleventh aspect, this application also provides a computer chip connected to a memory, the chip being used to read and execute computer program instructions stored in the memory, to execute the methods in the second aspect and various possible implementations of the second aspect, or to execute the methods in the third aspect and various possible implementations of the third aspect.

[0082] The technical effects that can be achieved in the second to eleventh aspects mentioned above are described in the description of the technical effects that can be achieved by the corresponding design schemes in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0083] Figure 1 A schematic diagram of the structure of a fingerprint index provided in this application;

[0084] Figures 2A-2B This is a schematic diagram of the structure of a fingerprint index provided in an embodiment of this application;

[0085] Figure 3 A schematic diagram of a fingerprint index construction method provided in this application;

[0086] Figures 4-5 A schematic diagram of a fingerprint index construction process is provided for this application;

[0087] Figure 6 A schematic diagram of a fingerprint query method provided in this application;

[0088] Figures 7-8 A schematic diagram of a fingerprint query process provided in this application;

[0089] Figure 9 A schematic diagram of a fingerprint index construction method provided in this application;

[0090] Figures 10A to 10C A schematic diagram of a fingerprint index structure provided in this application;

[0091] Figure 11 A schematic diagram of a fingerprint query method provided in this application;

[0092] Figures 12A-12B A schematic diagram of fingerprint index deployment provided in this application;

[0093] Figures 13A-13B A schematic diagram illustrating the splitting of a fingerprint index provided in this application;

[0094] Figure 14 A schematic diagram of a construction device provided in this application;

[0095] Figure 15 A schematic diagram of the structure of a query device provided in this application;

[0096] Figures 16-17 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0097] Before describing the storage device, fingerprint index construction method, query method, and equipment provided in the embodiments of this application, the concepts involved in the embodiments of this application will be explained first:

[0098] (1) Data fingerprint.

[0099] A data fingerprint is information formed based on the data itself and can be used to represent the data. A data fingerprint can be understood as a "summary" extracted from the data. Data fingerprints can be used to compare data; when two sets of data have the same data fingerprint, it means that the two sets of data are very likely the same.

[0100] This application does not limit the method of generating data fingerprints. Generally, data fingerprints can be generated by calling a data fingerprint algorithm. Data fingerprint algorithms include, but are not limited to: MD5, xxhash, sha1, sha256, sha3, and blake3. This application does not limit the specific type of data fingerprint algorithm.

[0101] When generating a fingerprint of data, the data can first be segmented into multiple parts; for ease of explanation, one of these multiple parts is called a data slice. After obtaining multiple data slices, a fingerprint for each data slice is generated. The fingerprint of the data includes the fingerprints of all the data slices.

[0102] The granularity of data segmentation can be fixed, meaning each resulting data slice is the same size. Alternatively, the granularity can be variable, meaning each resulting data slice is not necessarily the same size. In this type of segmentation, a range of values ​​for the granularity can be set, thus limiting the range of sizes for the resulting data slices. Each data slice formed during data segmentation must fall within this range.

[0103] (2) Deleting domains.

[0104] For any data shard, the deduplication field refers to the storage area (also simply called a region) allocated for that data shard and required to be written to it. Typically, during data deduplication, deduplication fields are allocated to each data shard included in the data. For any given data shard, it needs to be written to its allocated deduplication field. If a data shard identical to the original already exists within the deduplication field, then when writing to the deduplication field, no write operation is required. That is, the data shard does not need to occupy storage space within the deduplication field; only the storage address of the data shard already stored within the deduplication field needs to be recorded. This storage address is the storage address of the data shard to be written to the deduplication field. If no identical data fragment is stored in the deduplication field of a data fragment, then when writing to the deduplication field, a write operation needs to be performed on that data fragment. This requires the data fragment to occupy storage space within the deduplication field to record its storage address. Therefore, the deduplication field of a data fragment limits its deduplication scope. In other words, when deduplicating a data fragment, the data already stored in its deduplication field is used as the data for comparison.

[0105] The specific form of the deduplication field in a data shard depends on the data writing scenario and the granularity of storage space partitioning within that scenario. For example, when writing data to a node and performing deduplication on that data, deduplication fields can be allocated within that node for each data shard included in the data. These fields can be the node's memory or its storage pool. A storage pool can be understood as a segment of storage space logically partitioned within the node. Similarly, when writing data to a storage system and performing deduplication on that data, deduplication fields can be allocated within the storage system for each data shard included in the data. These fields can be nodes within the storage system or their memory. The specific table format of the deduplication field in a data shard is quite flexible, and this application does not limit the specific form of the deduplication field.

[0106] First, a fingerprint index provided in the embodiments of this application will be introduced.

[0107] 1) The structure of the fingerprint index.

[0108] A fingerprint index is an index built upon a first set of fingerprints. Information related to a fingerprint is obtained by querying the fingerprint within this index. The "information related to the fingerprint" obtained is the "data" that the fingerprint index targets. The "data" that the fingerprint index targets varies depending on the application scenario. For example, in a scenario where the data corresponding to a fingerprint needs to be retrieved, the "data" that the fingerprint index targets is the data corresponding to the fingerprint itself; that is, querying the fingerprint within the index will retrieve the data corresponding to that fingerprint. In a scenario where the storage location of data needs to be determined, the "data" that the fingerprint index targets is the storage address of the data; that is, querying the fingerprint within the index will retrieve the storage address of the data. In a scenario where the node where the data is located needs to be identified, the "data" that the fingerprint index targets is the node where the data is located; that is, querying the fingerprint within the index will reveal information about the node where the data is located (such as the node's identifier, physical location, etc.). In scenarios where it is necessary to determine the region where data is located, the "data" targeted by the fingerprint index is the region where the data is located. In other words, by using the fingerprint to query the fingerprint index, information about the region where the data is located (such as the region identifier) ​​can be obtained. This scenario is common when querying the deduplication domain of data, that is, the region mentioned above is the deduplication domain.

[0109] In addition, there is a special scenario where it is necessary to determine whether certain data should be stored or whether certain data exists. In this scenario, a fingerprint is used to query the fingerprint index. If the fingerprint is found, it means that the data corresponding to the fingerprint exists or has been stored. Otherwise, the data corresponding to the fingerprint does not exist or has not been stored.

[0110] For ease of explanation, "information related to the fingerprint" will be referred to as fingerprint-related information or fingerprint-related information. There is a correspondence between fingerprints and their related information. This correspondence can exist in the form of key-value pairs, where the fingerprint is the "key" and the correspondence between the fingerprint and its related information is the "value" corresponding to that "key". This embodiment only illustrates the correspondence between fingerprints and their related information in key-value pair form; however, other forms of correspondence are not limited in this embodiment.

[0111] The fingerprint in the first dataset, as a string, can be split into multiple segments. That is, the fingerprint in the first dataset includes multiple fields, and the possible values ​​of these fields form the basis for constructing the fingerprint index. It's important to note that the field values ​​are related to the fingerprint type. For example, if the fingerprint is a binary character fingerprint, the field values ​​could be segments containing 0 and / 1. Or, if the fingerprint is a hexadecimal character fingerprint, the field values ​​could be segments containing some or all of the characters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F.

[0112] The structure of the fingerprint index is explained below.

[0113] like Figure 1 The above is a schematic diagram of a fingerprint index structure provided in an embodiment of this application. The fingerprint index has a multi-layer structure and includes multiple index layers. The multiple index layers in the fingerprint index are arranged layer by layer, with each layer in the multi-layer structure being an index layer. There is a vertical relationship between the multiple index layers of the fingerprint index, that is, there is an adjacent index layer above any index layer except the topmost index layer (this adjacent index layer can be called the upper index layer), and there is an adjacent index layer below any index layer except the bottommost index layer (this adjacent index layer can be called the lower index layer).

[0114] Each index layer corresponds to a field in the fingerprint. An index layer is used to store a field of the fingerprint, and the index layer covers all possible values ​​of that field.

[0115] Each index layer includes multiple index entries, and each index entry in the index layer represents a value of the field corresponding to that index layer.

[0116] In this fingerprint index, any index layer other than the lowest index layer corresponds to a set of index entries in the next lower index layer. Each set of index entries includes one or more index entries, and each index entry in this set represents a value of the field corresponding to its respective index layer.

[0117] In other words, any index entry in an index layer points to a set of index entries in the next lower index layer. From the perspective of the correspondence between index entries in the upper and lower layers, any index entry in the set of index entries being pointed to represents a value of the field corresponding to the field in the next lower index layer when the field corresponding to the current index layer is in a certain value state.

[0118] Based on the correspondence between the upper and lower index layers, starting from any index item in the top-level index layer, an index item is selected in each index layer, corresponding to the index item selected in the previous index layer, until an index item is selected from the bottom-level index layer. That is, starting from any index item in the top-level index layer, a set of index items in the second-level index layer that the index item points to is determined; an index item is selected from the determined set of index items in the second-level index layer, and then a set of index items in the third-level index layer that the index item points to is determined; an index item is selected from the determined set of index items in the third-level index layer, and then a set of index items in the fourth-level index layer that the index item points to is determined; an index item is selected from the determined set of index items in the fourth-level index layer, and so on, until the bottom-level index layer is reached, where an index item is selected from the set of index items in the bottom-level index layer.

[0119] Based on this method of selecting index terms, the index terms selected in the lowest index layer represent a fingerprint. The values ​​of each field of this fingerprint are consistent with the values ​​represented by the index terms selected in each index layer.

[0120] To explain the statement "the index entry selected in the lowest index layer represents a fingerprint" from another perspective: Due to the correspondence between the upper and lower index layers, starting from the lowest index layer and working upwards, each index layer always identifies an index entry that corresponds to the index entry identified in the next lower index layer. In this way, there is a correspondence between the index entries identified in each index layer; that is, an index entry in any index layer always has a corresponding index entry in any index layer above it. Therefore, the index entry in the lowest index layer can represent a fingerprint, and this fingerprint is consistent with the value of the field represented by the index entry identified in each index layer.

[0121] Optionally, the index entries in the lowest index layer include fingerprint-related information. The fingerprint is the fingerprint represented by the index entry. In this embodiment, in any index layer other than the lowest index layer, each index entry has the same length, meaning each index entry occupies the same storage space. For the lowest index layer, it may contain fingerprint-related information, and the length of each index entry is related to the length of the fingerprint-related information. Therefore, the length of each index entry in the lowest index layer can be the same or different. The case where the lengths of index entries in the lowest index layer are different will be explained below.

[0122] As described in the structural description of the fingerprint index, for any index layer, the index entries in that layer represent a value of the corresponding field. Here, we will explain one way in which an index entry represents a value of the corresponding field, as mentioned in the embodiments of this application.

[0123] In any index layer other than the lowest level, each index entry records the storage address of a set of index entries in the next lower index layer. For example, each index entry can record the starting address of each index entry in a set of index entries in the next lower index layer. Furthermore, since each index entry in this index layer has the same length, each index entry can record the starting address of the first index entry in a set of index entries in the next lower index layer.

[0124] For any index entry in this index layer, in the next index layer corresponding to that index entry, the index entries in that group are arranged in order of the value represented by each index entry. For example, the index entries in that group are arranged from largest to smallest according to the value represented by each index entry. Or, the index entries in that group are arranged from smallest to largest according to the value represented by each index entry.

[0125] Since the index entries in this group are arranged in a certain order, the position of each index entry in the group can represent the value represented by that index entry.

[0126] For example, this group of index entries includes eight index entries, each representing one of eight possible values ​​for the corresponding field in the index layer. These eight values ​​might be 0, 1, 2, 3, 4, 5, 6, and 7. The index entries in this group are arranged in ascending order of their represented values. Therefore, the index entry at the beginning of the group represents a value of 0, the second index entry represents a value of 1, the third index entry represents a value of 2, the fourth index entry represents a value of 3, the fifth index entry represents a value of 4, the sixth index entry represents a value of 5, the seventh index entry represents a value of 6, and the eighth index entry represents a value of 7.

[0127] For the lowest level index layer, the set of index entries corresponding to any index entry in the upper index layer can be arranged in the following two ways:

[0128] Method 1: The index items in this group are arranged in order of the value they represent. For example, the index items in this group are arranged from largest to smallest. Another example is the arrangement of the index items in this group from largest to smallest.

[0129] In this approach, each index entry in the group may not contain the value of its corresponding field; that is, the storage space containing the index entry does not contain the value it represents. The position of any index entry within the group is related to the value it represents.

[0130] Method 2: Each index entry in this group contains the value it represents; that is, the value is written into the storage space where the index entry is located. This method does not restrict the order of the index entries in this group. For example, the index entries in this group can be arranged in order according to the size of the value they represent, or they can be arranged randomly.

[0131] For the top-level index layer, multiple index entries are arranged in order of the value represented by each entry. Each index entry may not contain the value of its corresponding field; that is, the storage space containing the index entry does not contain the value it represents. The position of the index entry within the multiple index entries indicates the value of its corresponding field. Thus, the position of any index entry within the multiple index entries is related to the value it represents.

[0132] Both methods have their advantages. Method 1 is suitable for scenarios where the fingerprint index has ample storage space. In the lowest index layer, a set of index entries corresponding to any index entry in the upper index layer is allowed to contain index entries representing all possible values ​​of the corresponding field in that lowest index layer. The fingerprint index can record a large number of fingerprints. In scenarios where fingerprints are queried using the fingerprint index, this method not only ensures the accuracy of the query results but also guarantees the query efficiency of the fingerprint index (i.e., achieving fast queries). Method 2 is suitable for scenarios where the storage space of the lowest index layer in the fingerprint index is limited, meaning that the number of index entries contained in that lowest index layer is required to be fixed. In this scenario, the fingerprint index can record a smaller number of fingerprints, but the fingerprints recorded in the fingerprint index are always being updated. Therefore, in Method 2, each index entry can be replaced, that is, some index entries that have been written for a long time can be deleted in a timely manner to free up storage space to write new index entries. The index entries in the lowest index layer of the fingerprint index can change.

[0133] As explained above regarding "a method of representing the value of the corresponding field by an index term," when an index term is located in a certain index layer, the storage address of a set of index terms corresponding to that index term in the next lower index layer can be obtained from the located index term. Then, based on this storage address, the corresponding set of index terms in the next lower index layer can be located. Within this set of index terms, the value of the field corresponding to the next lower index layer in the known fingerprint can be used to locate one of the index terms in that set. This process is repeated until the index term in the lowest index layer is located. The specific method of searching using fingerprint indexing under this method of representing the value of the corresponding field by an index term will be explained below and will not be detailed here.

[0134] The following section uses two-layer and three-layer fingerprint indexes as examples to further introduce the structure of the fingerprint index, so as to gain a clearer understanding of its structure.

[0135] The fingerprint in the first fingerprint set includes two fields, namely field A and field B.

[0136] like Figure 2A As shown, the fingerprint index comprises two index layers, referred to as the first index layer and the second index layer, respectively. The first index layer corresponds to field A and is used to store field A; it includes multiple index entries, each representing a value of field A. The second index layer corresponds to field B and is used to store field B. One index entry in the first index layer corresponds to a set of index entries in the second index layer, where each index entry represents a value of B.

[0137] Assume that field A and field B can both take the values ​​of integers from 0 to 7. The first index layer consists of eight index entries, each representing one value of field A. These eight entries represent the eight integers from 0 to 7. Each index entry in the first index layer points to a set of index entries in the second index layer. Each set of index entries in the second index layer contains multiple index entries. Each set of index entries covers all possible values ​​of field B. Since field B can take the values ​​of integers from 0 to 7, each set of index entries contains eight index entries. The second index layer consists of eight sets of index entries, for a total of 64 index entries.

[0138] Given a fingerprint, determine the values ​​of fields A and B within that fingerprint. Here, we assume field A is 0 and field B is 7. Starting from the first index layer in this two-layer structure, perform a search operation layer by layer: search for the first index entry in all index entries of the first index layer that represents a value of 0 for field A, then jump to a set of index entries in the corresponding second index layer. In the second index layer, search for the second index entry in the jumped-to set of index entries that represents a value of 7 for field B.

[0139] If a second index entry is found in the second index layer, it means that the known fingerprint has been found in the fingerprint index, indicating that the data corresponding to the known fingerprint exists. If the second index entry searched in the second index layer also includes fingerprint-related information, then the fingerprint-related information in the second index entry is the information related to the known fingerprint.

[0140] If no index entry is found in either layer of the two-layer structure, it means that the known fingerprint is not recorded in the fingerprint index.

[0141] The fingerprint in the first fingerprint set includes three fields: field A, field B, and field C.

[0142] like Figure 2B As shown, the fingerprint index comprises three index layers, referred to as the first index layer, the second index layer, and the third index layer. The first index layer corresponds to field A and is used to store field A; it includes multiple index entries, each representing a value of field A. The second index layer corresponds to field B and is used to store field B. One index entry in the first index layer corresponds to a set of index entries in the second index layer, where one index entry represents a value of B. The third index layer corresponds to field C and is used to store field C. One index entry in the second index layer corresponds to a set of index entries in the third index layer, where one index entry represents a value of C.

[0143] Assume that field A can take the values ​​of integers from 0 to 7, field B can take the values ​​of integers from 0 to 7, and field C can take the values ​​of integers from 0 to 7. The correspondence between the first and second index layers, the number of index entries included, and... Figure 2A Similar to the previous explanation, details will not be repeated here. For the third index layer, any index entry in the second index layer points to a set of index entries in the third index layer. Each set of index entries in the third index layer includes multiple index entries. Each set of index entries covers all possible values ​​of field C. Since the possible values ​​of field C are integers from 0 to 7, each set of index entries includes 8 index entries. The third index layer includes a total of 64 sets of index entries, totaling 512 index entries.

[0144] Given a fingerprint, determine the values ​​of fields A, B, and C within that fingerprint. Here, we assume field A is 0, field B is 7, and field C is 7. Starting from the first index layer in this three-layer structure, perform the search operation layer by layer: the method for determining index items in the first and second index layers is similar to... Figure 2A The method for determining index items in the first and second index layers is similar. After determining the second index item in the second index layer, it jumps to a set of index items in the third index layer corresponding to that second index item. In the third index layer, it searches for the third index item in the jumped-to set of index items where the value of the characterizing field C is 7.

[0145] If a third index entry is found in the third index layer, it means that the known fingerprint has been found in the fingerprint index, indicating that the data corresponding to the known fingerprint exists. If the third index entry also includes fingerprint-related information, then the fingerprint-related information in the third index entry is the information needed for the known fingerprint.

[0146] If no index entry is found in any of the three layers of the structure, it means that the known fingerprint is not recorded in the fingerprint index.

[0147] The above description of querying known fingerprints in the first fingerprint set based on fingerprint index is only a simplified description. The following text will take querying the target fingerprint as an example to explain the query method based on fingerprint index in detail.

[0148] It's important to note that the description of the fingerprint index structure assumes that any field in the fingerprint can be equal to all possible values. For example, if a field in the fingerprint is a two-bit binary field, then the possible values ​​for this field are 00, 01, 11, and 10, a total of four possible values. However, in practical applications, the value of a field in a fingerprint depends on the specific fingerprints included in the first fingerprint set. Taking a two-bit binary field in any fingerprint as an example, since the specific fingerprints included in the first fingerprint set are different, the possible values ​​for this field may be some of the four possible values, or, when the values ​​of other fields in the fingerprint are fixed, the possible values ​​for this field may be some of the four possible values. Therefore, the index layer corresponding to this field in the fingerprint index only needs to cover this partial value.

[0149] In other words, the fingerprints recorded in the fingerprint index constructed for the first fingerprint set are the fingerprints in the first fingerprint set. Each index layer in the fingerprint index corresponds to a field in the fingerprint, and each index layer covers all possible values ​​of that field in each fingerprint in the first fingerprint set.

[0150] Since the index entries represent one possible value for the corresponding field, the following further clarifies the statement that "each index layer covers all possible values ​​for this field in each fingerprint within the first fingerprint set":

[0151] The two overlay methods provided in the embodiments of this application will be described in turn below:

[0152] This section introduces two numerical values. One value represents the number of possible values ​​for this field; for ease of explanation, this value will be referred to as the first value. The other value represents the number of possible values ​​for this field in each fingerprint within the first fingerprint set; for ease of explanation, this value will be referred to as the second value. The first value represents the number of possible values ​​for this field without limiting the first fingerprint set. The second value represents the number of possible values ​​for this field with limiting the first fingerprint set. The first value is greater than or equal to the second value.

[0153] Coverage Method 1: Invalid index entries are allowed in each index layer. An invalid index entry in any index layer will not have a corresponding set of index entries in the next index layer.

[0154] Since the position of an index entry within a set of index entries in a fingerprint index represents the value it represents, this set of index entries can contain multiple index entries, the number of which equals the first value. That is, this set of index entries reserves multiple index entries representing all possible values. For all possible values ​​of this field in each fingerprint of the first fingerprint set, the index entry representing this value records the storage address of a set of index entries in the next lower index layer. For values ​​that are not present in this field of each fingerprint of the first fingerprint set, the index entry representing this value either does not record any storage address or records an invalid identifier to indicate that the value does not exist or that the index entry is invalid. Using this method, the number of index entries with recorded storage addresses (i.e., valid index entries) in the index layer equals the second value.

[0155] Based on this configuration, the fingerprint index contains two types of index entries: valid index entries and invalid index entries. Valid index entries record a storage address and correspond to a set of index entries in the next lower index layer. In this embodiment, a valid index entry is one that can represent a value and corresponds to a set of index entries in the next lower index layer. Invalid index entries do not record a storage address, and there is no corresponding set of index entries in the next lower index layer.

[0156] For index entries in the lowest-level index layer, if the index entries in the lowest-level index layer are arranged in Method 1, the number of index entries in the lowest-level index layer is the same as the first value. There are also valid or invalid index entries in the lowest-level index layer. Invalid index entries include an invalid identifier to indicate that the value does not exist. Valid index entries include a valid identifier and / or fingerprint-related information. The valid identifier indicates that the value exists or that the index entry is valid. If the index entries are arranged in Method 2, then all index entries included in this lowest-level index layer are valid index entries. That is, each index entry in the lowest-level index layer includes a valid identifier and / or fingerprint-related information.

[0157] Coverage Method Two: Invalid index entries are allowed in the lowest-level index layer. These invalid index entries indicate that the fingerprint represented by the invalid entry does not exist. The invalid index entry in this lowest-level index layer may include an invalid identifier; furthermore, the invalid index entry does not include fingerprint-related information.

[0158] Since the position of an index entry within a set of index entries in a fingerprint index represents the value it represents, this set of index entries can contain multiple index entries, the number of which is equal to a first value. In other words, this set of index entries reserves multiple index entries representing all possible values. Each of these multiple index entries corresponds to a set of index entries in the next lower index layer; that is, one index entry corresponds to a set of index entries in the next lower index layer. Each of these multiple index entries records the storage address of the set of index entries in its corresponding next lower index layer. Index entries in each index layer are arranged in this manner.

[0159] For index entries in the lowest-level index layer, if the index entries in the lowest-level index layer are arranged in Method 1, there may be valid or invalid index entries in the lowest-level index layer. If the fingerprint index is applicable to scenarios where data is stored or whether a dataset contains data, an invalid index entry includes an invalid identifier to indicate that the fingerprint represented by the invalid index entry does not exist. If the fingerprint index is applicable to scenarios where fingerprint-related information is queried, an invalid index entry does not include fingerprint-related information. That is, if no fingerprint-related information is obtained from the index entry, the index entry can be determined to be invalid. Furthermore, the invalid index entry may also include an invalid identifier to indicate that the fingerprint represented by the invalid index entry does not exist. A valid index entry includes a valid identifier and / or fingerprint-related information. The valid identifier indicates that the fingerprint value represented by the valid index entry exists or that the index entry is valid. For an explanation of how an index entry in the lowest-level index layer represents a fingerprint, please refer to the foregoing content, which will not be repeated here.

[0160] If the index entries are arranged in Method 2, then all index entries in the lowest index layer are valid index entries. That is, each index entry in the lowest index layer includes information related to a valid identifier and / or fingerprint.

[0161] like Figure 1 The fingerprint index shown has a multi-layered structure, which simplifies its construction and improves query efficiency. In this fingerprint index, each index entry represents the value of a specific field within the fingerprint. The lowest index layer only needs to store fingerprint-related information. Compared to fingerprint tables that directly record key-value pairs (where the key is the fingerprint and the value is the corresponding data), fingerprint indexes do not require storing a large number of fingerprints, thus requiring less storage space.

[0162] 2) The process of building the fingerprint index.

[0163] The following explains the process of constructing a fingerprint index for the fingerprints in the first fingerprint set. For example... Figure 3 As shown in the figure, a fingerprint index construction method is provided in an embodiment of this application. The method includes the following steps:

[0164] Step 300: The construction device samples the fingerprints in the second fingerprint set to obtain the first fingerprint set.

[0165] When building a fingerprint index, the building device can build the fingerprint index only for a portion of the fingerprints, which can effectively reduce the size of the fingerprint index and reduce the storage space occupied by the fingerprint index.

[0166] For example, the second fingerprint set includes fingerprints of all data in the dataset, with each fingerprint representing a data shard within the dataset. This dataset can be data from a database, nodes, a node cluster, a hard drive, or a group of hard drives, etc. When constructing the fingerprint index, the constructing device can use the second fingerprint set as the first fingerprint set, or it can execute step 300 to extract a portion of fingerprints from the second fingerprint set, with this portion constituting the first fingerprint set.

[0167] The embodiments of this application do not limit the execution method of step 300. The fingerprints in the second fingerprint set can be sampled by a model taking method, or other methods can be used. The fingerprint sampling method will be further described below, and will not be detailed here.

[0168] Step 301: The construction device splits the fingerprints in the first fingerprint set into N fields. The number of fields determines the number of layers of the fingerprint index. This embodiment does not limit the number of fields, and N is a positive integer.

[0169] A fingerprint is typically a string. With a fixed fingerprint generation algorithm, the length of the fingerprint generated from the data using that algorithm is also fixed. Therefore, a fingerprint is a fixed-length string. Different fingerprint generation algorithms may result in different string types; for example, the fingerprint can be a hexadecimal string or a decimal string.

[0170] During step 301, the constructing device can directly split the fingerprint into multiple segments, each segment being a field. Alternatively, the fingerprint can be converted first, such as converting it into a binary string; then the converted fingerprint can be split into multiple segments, each segment being a field.

[0171] Step 302: The construction device constructs each index layer, starting from the topmost index layer. For any index layer, which corresponds to a field in the fingerprint, constructing the index layer includes: setting one or more sets of index entries in the index layer, each set of index entries including one or more index entries, where an index entry in the index layer represents a value of the field. An index entry in the index layer above this index layer corresponds to a set of index entries in this index layer.

[0172] This index layer includes multiple index entries that cover all possible values ​​of the field corresponding to this index layer, with each index entry representing one value. The number of groups of index entries in this index layer is equal to the number of index entries in the previous index layer. The topmost index layer of the fingerprint index consists of a set of index entries.

[0173] Based on the description of each index layer covering all possible values ​​of the field in each fingerprint of the first fingerprint set, valid index entries, and invalid index entries, the method of constructing this index layer is explained:

[0174] If the first overlay method is used, the construction process is as follows:

[0175] When constructing this index layer, the construction device sets one or more sets of index entries in the index layer. Each set of index entries includes one or more index entries, and each index entry records the storage address of a set of index entries in the next lower index layer. The position of the index entry in the set of index entries represents the value represented by the index entry.

[0176] Specifically, when the construction device sets any set of index entries in the index layer, it sets multiple index entries within that set. The number of index entries is equal to a first value, which is equal to the number of possible values ​​for the corresponding field in the index layer. The position of each index entry within the multiple index entries represents one possible value for that field. In other words, the possible values ​​for the corresponding field in the index layer are ordered sequentially according to their magnitude, and the order position of a specific value is the position of the index entry representing that specific value within the multiple index entries.

[0177] For all possible values ​​of this field in each fingerprint within the first fingerprint set, the storage address of a set of index entries in the next lower index layer is recorded in the index entry representing that value. This index entry is the valid index entry.

[0178] For any value in the first fingerprint set that is not present in this field, the index entry representing that value either does not record any storage address or records an invalid identifier to indicate that the value does not exist or that the index entry is invalid. This index entry is thus considered an invalid index entry.

[0179] In this way, the number of index entries (i.e., valid index entries) with recorded storage addresses in this index layer is equal to the second value. The second value is the number of possible values ​​for this field given the first fingerprint set.

[0180] For the lowest-level index layer, if a set of index entries in this lowest-level index layer is arranged according to Method 1, the method for constructing a set of index entries in the lowest-level index layer is similar to the methods mentioned above, except that: there are also valid or invalid index entries in the lowest-level index layer. Invalid index entries may include invalid identifiers to indicate that the value does not exist, or the invalid index entry may include fingerprint-related information to indicate that the fingerprint represented by the invalid index entry does not exist. Valid index entries include valid identifiers and / or fingerprint-related information; the valid identifier indicates that the value exists or that the index entry is valid. If a set of index entries in this lowest-level index layer is arranged according to Method 2, when constructing a set of index entries in the lowest-level index layer corresponding to the index entries in the upper-level index layer, multiple valid index entries are set in this set of index entries, that is, multiple valid index entries are set. Each index entry includes the value it represents, and optionally, also includes an index entry with valid identifiers and / or fingerprint-related information.

[0181] If the second overlay method is used, the construction process is as follows:

[0182] When constructing this index layer, the construction device sets one or more sets of index entries in the index layer. Each set of index entries includes one or more index entries, and each index entry records the storage address of a set of index entries in the next lower index layer. The position of the index entry in the set of index entries represents the value represented by the index entry.

[0183] Specifically, when the construction device sets any set of index entries in the index layer, it sets multiple index entries within that set. The number of index entries is equal to a first value, which is equal to the number of possible values ​​for the corresponding field in the index layer. The position of each index entry within the multiple index entries represents one possible value for that field. In other words, the possible values ​​for the corresponding field in the index layer are ordered sequentially according to their magnitude, and the order position of a specific value is the position of the index entry representing that specific value within the multiple index entries. Each of these multiple index entries records the storage address of a set of index entries in the next lower index layer. Index entries in each index layer are arranged in this manner.

[0184] The construction process of the lowest level index layer is similar to that of the first method of covering the index layer. For details, please refer to the previous explanation, which will not be repeated here.

[0185] With this construction method, there is a correspondence between the index items of each index layer. For example, starting from any index item in the lowest index layer and moving upwards, a corresponding index item can always be found in each index layer.

[0186] Therefore, at the lowest level of the index layer, one index entry can represent one fingerprint. The values ​​of each field of this fingerprint are consistent with the values ​​represented by the lowest-level index entry and the values ​​represented by the corresponding index entries in each of the higher index layers.

[0187] Here, the phrase "the multiple index entries included in this index layer cover all possible values ​​of the field corresponding to this index layer" is explained: All possible values ​​refer to all possible values ​​of the field in each fingerprint within the first fingerprint set. Introducing the correspondence between index entries of each index layer, for any set of index entries in an index layer, all possible values ​​of the field represented by that set of index entries are all possible values ​​of that field in fingerprints within the first fingerprint set that satisfy the following condition: The fingerprint includes one or more upper-level fields, and the value of the upper-level field in the fingerprint is equal to the value represented by the corresponding index entry in the upper-level index layer. The upper-level field is the field corresponding to the index layer above the index layer containing this set of index entries; the index layer corresponding to one upper-level field is called the upper-level index layer.

[0188] After steps 301 and 302, based on the current first fingerprint set, the index entries that can be introduced into the fingerprint index have been arranged. For fingerprint indexes used in scenarios that require obtaining fingerprint-related information, the construction device also needs to add fingerprint-related information to each index entry in the lowest-level index layer; this fingerprint is the fingerprint represented by that index entry. For fingerprint indexes used in scenarios that require determining whether data is stored or whether data already exists, it is not necessary to add fingerprint-related information to each index entry in the lowest-level index layer. Of course, fingerprint indexes that include fingerprint-related information in each index entry of the lowest-level index layer are also applicable to scenarios that require determining whether data is stored or whether data already exists.

[0189] In this embodiment, changes to fingerprints in the first fingerprint set are permitted, such as adding new fingerprints, deleting existing fingerprints, or modifying fingerprints in the first fingerprint set. When fingerprints in the first fingerprint set change, the fingerprint index needs to be updated.

[0190] When a new fingerprint is added to the first fingerprint set, the construction device adds an index entry representing that fingerprint to the fingerprint index. The values ​​of each field in the new fingerprint (referred to as the new fingerprint) are determined. Starting from the topmost index layer of the fingerprint index, the construction device performs the following judgment operation in each index layer: Within the search range of each index layer, it determines whether an index entry representing the value of a specified field in the new fingerprint exists (e.g., whether the index entry representing that value within the search range records a storage address). This specified field is the field corresponding to that index layer. If it exists (e.g., the index entry representing that value within the search range records a storage address), the device jumps to a set of index entries in the next index layer corresponding to that index entry. If the value does not exist (e.g., the index entry representing the value within the search range does not record a storage address), an index entry representing the value of the specified field in the new fingerprint is added to the index layer. If a storage address is recorded in the index entry representing the value, this storage address becomes the storage address of a set of index entries in the next index layer. A new set of index entries is added to the next index layer, but no index entries are written to this set for the time being. If none of these multiple index entries record any information, the search jumps to this set of index entries. The search range of each index layer is the set of index entries that the previous index layer jumps to when performing the judgment operation. For the top-level index layer, the search range is all index entries of that top-level index layer. The judgment operation is performed layer by layer until the bottom-level index layer is reached.

[0191] In the lowest index layer, after an index entry is added to the lowest index layer, if the fingerprint index is used to query fingerprint-related information, the construction device adds the fingerprint-related information of the new fingerprint to the added index entry.

[0192] It should be noted that the above-mentioned method of adding index entries to fingerprint abbreviations is based on the aforementioned overlay method one. If overlay method two is used, when a new fingerprint is added to the first fingerprint set, the construction device only needs to add an index entry for that fingerprint to the bottom-level index layer. For example, when the bottom-level index entries are arranged in method one, the construction device can mark the index entries representing the values ​​of specified fields in the new fingerprint as valid index entries. For instance, it can store relevant information about the new fingerprint in these index entries (fingerprint index entries are suitable for scenarios involving querying fingerprint-related information), or add valid identifiers to these index entries (fingerprint index entries are suitable for scenarios involving querying whether data is stored or whether data exists). Similarly, when the bottom-level index entries are arranged in method two, the construction device can add index entries to the bottom-level index layer. These index entries include the values ​​of specified fields in the new fingerprint. Furthermore, these index entries can also add relevant information about the new fingerprint (fingerprint index entries are suitable for scenarios involving querying fingerprint-related information).

[0193] When an existing fingerprint is deleted from the first fingerprint set, the construction device deletes the index entry representing the deleted fingerprint in the lowest index layer of the fingerprint index. If the lowest index entries are arranged in mode one, the construction device can mark the index entry representing the deleted fingerprint in that lowest index layer as an invalid index entry, such as deleting the fingerprint's related information from that index entry (fingerprint index entries are suitable for scenarios involving querying fingerprint-related information), or adding an invalid identifier to that index entry (fingerprint index entries are suitable for scenarios involving querying whether data is stored or whether data exists). If the lowest index entries are arranged in mode one, the construction device can delete the index entry representing the deleted fingerprint in that lowest index layer.

[0194] When a fingerprint in the first fingerprint set is modified, since this modification may cause changes to the fingerprint itself, the fingerprint before the modification is referred to as the old fingerprint, and the fingerprint after the modification is referred to as the new fingerprint. When updating the fingerprint index, the construction device deletes the index entry representing the old fingerprint in the lowest index layer. The construction device then adds an index entry representing the new fingerprint to the fingerprint index. The method for adding the index entry representing the new fingerprint and deleting the index entry representing the old fingerprint can be found in the foregoing description and will not be repeated here.

[0195] In addition, the length of fingerprint-related information can be dynamically adjusted within the fingerprint index. That is, the length of the relevant information recorded in the fingerprint index can be inconsistent.

[0196] For example, fingerprint-related information is the node identifier where the data resides. Since there are many ways to set node identifiers, there may be situations where node identifiers within the same node cluster have the same length, while node identifiers in different node clusters have different lengths. In this embodiment, fingerprint-related information is recorded in the index entry. Differences in the length of the fingerprint-related information do not affect the structure of the fingerprint index. Correspondingly, even if a new index entry needs to be added to the fingerprint index, and the length of the fingerprint-related information in the new index entry differs from the length of the fingerprint-related information in existing index entries, the position of the new index entry can be uniquely determined within the fingerprint index based on the values ​​of each field in the fingerprint, thus allowing the addition of the new index entry.

[0197] After the fingerprint index is built, if a new index entry containing longer fingerprint-related information needs to be added to the fingerprint index, the existing index entries do not need to be changed. Only the corresponding index entry needs to be added to the fingerprint index. The method for adding index entries is explained above and will not be repeated here. Conversely, if a new index entry containing shorter fingerprint-related information needs to be added to the fingerprint index, the existing index entries do not need to be changed to increase the length of the fingerprint-related information. Only the corresponding index entry needs to be added to the fingerprint index. The method for adding index entries is explained above and will not be repeated here.

[0198] Because the length of fingerprint-related information can change, the length of index entries in the lowest level of the fingerprint index may differ.

[0199] Of course, the fingerprint index also allows for variations in fingerprint length. For example, new fingerprints can be added to the first fingerprint set, and the length of these new fingerprints may differ from the lengths of the fingerprints already recorded in the fingerprint index. For instance, if some fingerprints need to be added to the first fingerprint set, each fingerprint identifying a data fragment, a character segment can be added to the data fragment fingerprint after it has been generated. This makes the length of these data fragment fingerprints longer than the length of the fingerprints already recorded in the fingerprint index. As another example, if some fingerprints need to be added to the first fingerprint set, each fingerprint identifying a data fragment, a character segment can be deleted from the data fragment fingerprint after it has been generated. This makes the length of these data fragment fingerprints shorter than the length of the fingerprints already recorded in the fingerprint index.

[0200] ① After the fingerprint index is constructed, the length of the fingerprint increases.

[0201] In other words, the fingerprint length represented by subsequent index entries in the fingerprint index increases. For ease of explanation, after the fingerprint index is built, the field corresponding to the lowest index layer is called field P, and the added field after the fingerprint length increases is called field Q. Therefore, after the fingerprint index is built and the fingerprint length increases, the building device can add a new index entry to the fingerprint index. The fingerprint represented by this new index entry is the fingerprint with the increased length. The method of adding index entries can be found in the previous explanation. The difference is that when adding a new index entry in the lowest index layer, the fields corresponding to this new index entry include both field Q and field P. That is, the new index entry added in the lowest index layer can represent a value of field Q and a value of field P.

[0202] ② After the fingerprint index is constructed, the fingerprint length is reduced.

[0203] In other words, the fingerprint length segments represented by the index entries that need to be recorded later in this fingerprint index are, for ease of explanation, referred to as field P at the bottom layer of the index after its construction. After the fingerprint length is reduced, field P is reduced to field K. Therefore, after the fingerprint index is constructed and the fingerprint length is reduced, the construction device can add new index entries to the fingerprint index. These new index entries represent the fingerprint with the reduced length. The method for adding index entries can be found in the previous explanation. The difference is that when adding a new index entry at the bottom layer, the field corresponding to this new index entry is field K. That is, the new index entry added at the bottom layer can represent one value of field K.

[0204] Considering the possibility of fingerprint length reduction, the arrangement of index entries in the lowest-level index layer of a fingerprint index suitable for this scenario can adopt Method Two. That is, the index entries include the values ​​of the corresponding fields; optionally, the index entries can also include fingerprint-related information. Since the values ​​included in the index entries may differ, the length of the index entries in the lowest-level index layer of the fingerprint index may also differ.

[0205] The following describes how to construct... Figure 2A Taking the fingerprint index shown in 2B as an example, we will further introduce the construction process of the fingerprint index in order to gain a clearer understanding of the construction method of the fingerprint index.

[0206] I. The fingerprint index includes a two-layer structure:

[0207] like Figure 4 As shown in the figure, a fingerprint index construction method is provided in an embodiment of this application. The method includes the following steps:

[0208] Step 400: The constructing device samples fingerprints from the second fingerprint set to obtain the first fingerprint set. For details on the execution method, please refer to the relevant explanation of step 300; it will not be repeated here.

[0209] Step 401: The constructing device splits the fingerprints in the first fingerprint set into two fields. These two fields include field A and field B.

[0210] Step 402: The device constructs the first index layer of the fingerprint index: multiple index entries are set in the first index layer, and one index entry in the first index layer represents a value of field A.

[0211] Assuming that the specific value of field A of the fingerprint in the first fingerprint set may be an integer from 0 to 9, the construction device sets 10 index entries in the first index layer, which represent 10 integers from 0 to 9 respectively.

[0212] The construction method of the first index layer is explained by combining the description of each index layer covering all possible values ​​of the field in each fingerprint in the first fingerprint set, valid index entries, and invalid index entries:

[0213] If the first overlay method is used, the construction process is as follows:

[0214] When constructing the first index layer, the construction device sets up multiple index entries within it. The number of index entries is equal to a first value, which is equal to the number of possible values ​​for field A. The position of each index entry within the multiple index entries represents one possible value for the field. In other words, the possible values ​​of the field in this index layer are ordered sequentially according to their magnitude, and the position of a specific value is the position of the index entry representing that specific value within the multiple index entries.

[0215] For all possible values ​​of field A in each fingerprint within the first fingerprint set, the storage address of a set of index entries in the next lower index layer is recorded in the index entry representing that value. This index entry is the valid index entry.

[0216] For any value in field A that is not present in any of the fingerprints in the first fingerprint set, the index entry representing that value either does not record any storage address or records an invalid identifier to indicate that the value does not exist or that the index entry is invalid. This index entry is called an invalid index entry.

[0217] Taking the example that field A in the first fingerprint set could potentially take the value of an integer from 0 to 9, and assuming that the first fingerprint set is not limited, field A could potentially take the value of an integer from 0 to 15. That is, the first value is equal to 16, and the second value is equal to 10. When constructing the first index layer, the construction device sets 16 index entries in the first index layer. The position of any of the 16 index entries indicates the specific value of field A. For example, the index entry at the beginning represents a value of 0, and the index entry at the end represents a value of 15.

[0218] For all possible values ​​of field A in each fingerprint within the first fingerprint set (e.g., integers from 0 to 9), the storage address of a set of index entries in the second index layer corresponding to that value is recorded in the index entry representing that value. This index entry is the valid index entry.

[0219] For any value in field A that is not present in the first fingerprint set (such as an integer from 10 to 15), the index entry representing that value either does not record any storage address or records an invalid identifier to indicate that the value does not exist or that the index entry is invalid. This index entry is thus considered an invalid index entry.

[0220] The construction process using the second covering method is similar to that using the first covering method, with the difference being that: there is no concept of invalid index entries in the first index layer. The first index layer sets up multiple index entries, the number of which is equal to a first value, which is equal to the number of possible values ​​for field A. Each index entry in the first index layer corresponds to a set of index entries in the second index layer, and each index entry records the storage address of a set of index entries in its corresponding next-level index layer.

[0221] Step 403: Construct the second index layer of the fingerprint index: Set up multiple sets of index entries in the second index layer. One index entry in the first index layer corresponds to one set of index entries in the construction device of the second index layer. One index entry in the set of index entries in the second index layer represents a value of field B.

[0222] Assuming field B may take the values ​​of integers from 0 to 9, the construction device sets up 10 sets of index entries in the second index layer, with one index entry in the first index layer corresponding to one of these sets. Each set of index entries includes 10 index entries, each representing one of the 10 integers from 0 to 9.

[0223] If the fingerprint index is used to query fingerprint-related information, the construction device stores fingerprint-related information in each index entry of the second index layer. Given the correspondence between the first and second index layers, an index entry in the second index layer represents a fingerprint, and the information stored in the index entry is the information related to the fingerprint it represents.

[0224] The second index layer is the lowest index layer. The construction method of the second index layer is similar to that of the lowest index layer in step 302. For details, please refer to the above content, which will not be repeated here.

[0225] II. The fingerprint index consists of a three-layer structure:

[0226] like Figure 5 As shown in the figure, a fingerprint index construction method is provided in an embodiment of this application. The method includes the following steps:

[0227] Step 500: The constructing device samples fingerprints from the second fingerprint set to obtain the first fingerprint set. For details on the execution method, please refer to the relevant explanation of step 300; it will not be repeated here.

[0228] Step 501: The constructing device splits the fingerprints in the first fingerprint set into three fields. These two fields include field A, field B, and field C.

[0229] Step 502: The device constructs the first index layer of the fingerprint index. For details, please refer to the relevant explanation in step 402, which will not be repeated here.

[0230] Step 503: The construction device constructs the second index layer of the fingerprint index. The construction method of any group of index entries in the second index layer is similar to the construction method of multiple index entries in the first index layer in step 502. For details, please refer to the foregoing description and will not be repeated here.

[0231] Step 504: The construction device constructs the third index layer of the fingerprint index. Multiple sets of index entries are set in the third index layer. One index entry in the second index layer corresponds to one set of index entries in the third index layer, and one index entry in the set of index entries in the third index layer represents a value of field C.

[0232] Assuming field C can take the values ​​of integers from 0 to 9, 100 sets of index entries are set in the third index layer, and one index entry in the second index layer corresponds to one of these sets of index entries. Each set of index entries consists of 10 index entries, each representing one of the 10 integers from 0 to 9.

[0233] If the fingerprint index is used to query fingerprint-related information, the construction device stores fingerprint-related information in each index entry of the third index layer. Given the correspondence between the first, second, and third index layers, an index entry in the third index layer represents a fingerprint, and the information stored in the index entry is the information related to the fingerprint it represents.

[0234] The third index layer is the lowest level index layer. The construction method of the third index layer is similar to that of the second index layer in step 402, the difference being that the corresponding fields are different. For details, please refer to the previous explanation, which will not be repeated here.

[0235] It should be noted that the above description of the fingerprint index construction method is based on the example of a construction device executing the fingerprint index construction method. This application embodiment does not limit the relationship between the construction device and the device containing the first fingerprint set. The device containing the first fingerprint set can also be understood as a device that stores the data corresponding to the fingerprints in the first fingerprint set. The construction device can be deployed on this device, or it can be a device preceding this device used for constructing the fingerprint index. After constructing the fingerprint index, the construction device can send the fingerprint index to the device, which receives and stores it.

[0236] Given the simple structure of the fingerprint index, its construction is relatively easy, and its updating is more convenient. Furthermore, this fingerprint index supports variations in the length of fingerprint-related information, making it more scalable.

[0237] 3) Fingerprint-based query method.

[0238] For example Figure 1 Using the fingerprint index structure shown as an example, we will explain the query method based on the fingerprint index. Figure 6 The diagram illustrates a fingerprint-based query method according to an embodiment of this application. In this method, the query retrieves a target fingerprint from a first fingerprint set based on the fingerprint index.

[0239] Step 600: The query device obtains the target fingerprint to be queried and determines the values ​​of each field in the target fingerprint.

[0240] There are many scenarios in which a query device queries a target fingerprint in a first fingerprint set. For example, the query device receives a fingerprint query instruction, which instructs the query to search for the target fingerprint in the first fingerprint set, and the fingerprint query instruction carries the target fingerprint. Another example is that the query device receives a data query instruction, which instructs the query to search for target data in a dataset. That is, it determines whether the target data exists in the dataset. After receiving the data query instruction, the query device generates the fingerprint of the target data (i.e., the target fingerprint). Yet another example is that the query device receives a data write request, which requests the writing of target data. After receiving the data write request, the query device retrieves the target data from the data write request and generates the fingerprint of the target data.

[0241] Step 601: The query device starts from the topmost index layer of the fingerprint index and performs the following first search operation layer by layer until it reaches the lowest index layer. The first search operation performed in any index layer is as follows: It searches for an index entry representing the value of the first target field in the target fingerprint within the search range of that index layer. This first target field is the field corresponding to that index layer. If the index entry is found, it jumps to a set of index entries in the next index layer corresponding to that index entry. If no index entry is found, the fingerprint index-based query method can be terminated.

[0242] The search scope at this index layer is the set of index entries that the previous index layer jumps to when performing the first search operation. For the top-level index layer, the search scope is all the index entries of that top-level index layer.

[0243] The method by which the query device executes step 601 is explained in conjunction with the aforementioned index terms that characterize a way of taking a value for the corresponding field, as well as valid and invalid index terms.

[0244] If the first covering method is used, the query process is as follows:

[0245] The first search operation performed by the query device in any index layer is as follows: The query device searches for a valid index entry representing the value of the first target field in the target fingerprint within the search range of the index layer. If a valid index entry is found, the device jumps to the storage address of a set of index entries in the next index layer recorded in the found valid index entry. If no valid index entry is found, the fingerprint-based query method is terminated. Therefore, in step 602, the query device searches for valid index entries representing the value of each field in the target fingerprint layer by layer. The index entry found at the lowest index layer is considered a valid index entry, indicating that the target fingerprint has been found. Otherwise, the target fingerprint has not been found.

[0246] If the second covering method is used, the query process is as follows:

[0247] The first search operation performed by the query device in any index layer is as follows: The query device searches for an index entry representing the value of the first target field in the target fingerprint within the search range of the index layer. Based on the storage address of a set of index entries in the corresponding next index layer recorded in the searched index entry, it jumps to that storage address. This continues until the bottom index layer is reached. Therefore, in step 602, the query device searches for index entries representing the value of each field in the target fingerprint layer by layer. The index entry finally found in the bottom index layer is a valid index entry, indicating that the target fingerprint has been found. Otherwise, the target fingerprint has not been found.

[0248] Step 602: The fingerprint represented by the index item searched by the query device within the search range of the lowest index layer is the target fingerprint.

[0249] When this fingerprint index is applied to scenarios requiring the acquisition of fingerprint-related information, the query device retrieves the fingerprint-related information of the target fingerprint from the index entries searched within the search scope of the lowest index layer. If step 601 is executed and no corresponding index entry is found in a certain layer, the fingerprint index-based query method is terminated, indicating that the fingerprint index does not record the target fingerprint, or that the fingerprint-related information of the target fingerprint does not exist.

[0250] When this fingerprint index is applied to a scenario where fingerprint-related information needs to be obtained, the query device searches the index entries from the search scope of the lowest index layer to obtain the fingerprint-related information of the target fingerprint.

[0251] This fingerprint index is used in scenarios where it is necessary to determine whether data exists or has been stored. If the query device finds an index item within the search range of the lowest index layer, it indicates that the data corresponding to the target fingerprint exists or has been stored.

[0252] Because fingerprint indexes have a hierarchical structure and corresponding relationships between each index layer, queries based on this index can progressively locate a smaller range of index terms as the search scope. The correspondence between index layers not only simplifies the query process but also allows for rapid location of the search scope during the layer-by-layer search. Compared to queries based on fingerprint tables, queries using fingerprint indexes can gradually narrow down the search scope through layer-by-layer searching. In other words, queries based on fingerprint tables use the entire set of fingerprints as the search scope, while queries using fingerprint indexes essentially eliminate fingerprints from the first fingerprint set whose sub-segments differ from the target fingerprint's sub-segments.

[0253] The following is based on such Figure 2A Using the fingerprint index shown in 2B as an example, we will further introduce the query process based on fingerprint indexes to provide a clearer understanding of the fingerprint index-based query method.

[0254] I. The fingerprint index has a two-layer structure.

[0255] like Figure 7 The diagram shown is a schematic of a fingerprint-based query method provided in an embodiment of this application.

[0256] Step 700: The query device obtains the target fingerprint to be queried and determines the values ​​of field A and field B in the target fingerprint.

[0257] Step 701: The query device determines the first index term from multiple index terms in the first index layer of the fingerprint index. The value represented by the first index term is consistent with the value of field A in the target fingerprint.

[0258] The method by which the query device executes step 701 is explained in conjunction with the aforementioned index terms that characterize a way of taking a value for the corresponding field, as well as valid and invalid index terms.

[0259] If the first covering method is used, the query process is as follows:

[0260] The query device performs the following in the first index layer: the query device searches for a valid index entry representing the value of field A in the target fingerprint in the first index layer.

[0261] The query device can obtain the starting address P of the first index layer in the fingerprint index. The query device calculates the starting address of the index entry representing the value of field A in the target fingerprint. The sorting position of the value of field A in the target fingerprint among all possible values ​​of field A is S. That is, there are S-1 possible values ​​of field A before the value of field A in the target fingerprint, and there are S-1 index entries before the index entry representing that value. Therefore, the starting address p of the index entry representing that value satisfies: p = P + (S-1)*D, where D is the length of the index entry in the first index layer. In other words, the starting address of the index entry representing that value is the starting address P of the first index layer in the fingerprint index entry plus an offset, which is equal to the total length of the index entries before the index entry representing that value, i.e., the offset is (S-1)*D.

[0262] After calculating the starting address of the index entry representing the value, the query device retrieves the index entry at that starting address and determines whether it records the storage address of a corresponding set of index entries in the second index layer. If the index entry records the storage address of a corresponding set of index entries in the second index layer, then the index entry is the first index entry. If the index entry does not record the storage address of a corresponding set of index entries in the second index layer, then the index entry is an invalid index entry and not the first index entry. That is, the first index entry does not exist in the first index layer.

[0263] After finding the first index item, the query device obtains the storage address of a set of index items in the second index layer corresponding to the first index item from the first index item and executes step 702.

[0264] If the second covering method is used, the query process is as follows:

[0265] The query device performs the following operations in the first index layer: Based on the value of field A in the target fingerprint, the query device searches for an index entry representing that value in the first index layer. The method of searching for this index entry is similar to the method of searching for a valid index entry using a covering approach in a one-time query process. The only difference is that this query process will definitely find an index entry representing that value, and the storage address of a set of index entries in the second index layer corresponding to that index entry is recorded.

[0266] Step 702: Jump to a set of index entries in the second index layer corresponding to the first index entry. The query device determines the second index entry in the set of index entries in the second index layer corresponding to the first index entry. The value represented by the second index entry is consistent with the value of field B in the target fingerprint.

[0267] When the fingerprint index is applied to a scenario where fingerprint-related information needs to be obtained, the query device retrieves the fingerprint-related information of the target fingerprint from the second index item.

[0268] When this fingerprint index is used in scenarios where it is necessary to determine whether data exists or has been stored, the query device finds the second index item from the second index layer, indicating that the data corresponding to the target fingerprint exists or has been stored.

[0269] The method by which the query device executes step 702 is explained in conjunction with the aforementioned index terms that characterize a way of taking a value for the corresponding field, as well as valid and invalid index terms.

[0270] The second index layer is the lowest index layer. If the index entries in the second index layer are arranged in method one, the way the query device determines the second index entry from a set of index entries in the second index layer corresponding to the first index entry is similar to the way the query device determines the first index entry from multiple index entries in the first index layer in step 701. For details, please refer to the foregoing explanation; it will not be repeated here. When the query device finds a valid index entry in the second index layer that represents the value of the target fingerprint field B, it can obtain fingerprint-related information about the target fingerprint from it. If the index entries in the second index layer are arranged in method two, the query device can traverse each index entry in a set of index entries in the second index layer corresponding to the first index entry until it finds a valid index entry containing the value of the target fingerprint field B. If no valid index entry containing the value of the target fingerprint field B is found, it means that the target fingerprint has not been found.

[0271] II. The fingerprint index consists of a three-layer structure.

[0272] like Figure 8 The diagram shown is a schematic of a fingerprint-based query method provided in an embodiment of this application.

[0273] Step 800: The query device obtains the target fingerprint to be queried and determines the values ​​of field A, field B and field C in the target fingerprint.

[0274] Step 801: The query device determines the first index entry from multiple index entries in the first index layer of the fingerprint index. For details, please refer to the relevant explanation in step 701, which will not be repeated here.

[0275] Step 802: The query device determines the second index term from a set of index terms in the second index layer corresponding to the first index term. The value represented by the first index term is consistent with the value of field B in the target fingerprint. The way the query device determines the second index term from a set of index terms in the second index layer corresponding to the first index term is similar to the way the first index term is determined from multiple index terms in step 701. For details, please refer to the foregoing explanation, which will not be repeated here.

[0276] Step 803: Jump to a set of index entries in the third index layer corresponding to the second index entry. The query device determines the third index entry in the set of index entries in the third index layer corresponding to the second index entry. The value represented by the third index entry is consistent with the value of field C in the target fingerprint.

[0277] When the fingerprint index is applied to a scenario where fingerprint-related information needs to be obtained, the query device retrieves the fingerprint-related information of the target fingerprint from the third index item.

[0278] This fingerprint index is used in scenarios where it is necessary to determine whether data exists or has been stored. If the query device finds the third index item from the third index layer, it indicates that the data corresponding to the target fingerprint exists or the data corresponding to the target fingerprint has been stored.

[0279] The third index layer is the lowest index layer. The way the query device determines the third index item from a set of index items in the third index layer corresponding to the second index item is similar to the way it determines the second index item from a set of index items in the second index layer corresponding to the first index item in step 703. For details, please refer to the foregoing explanation, which will not be repeated here.

[0280] Derived from, for example Figure 1 The fingerprint index shown in this application embodiment provides another fingerprint index. This fingerprint index will be described in detail below.

[0281] 1. The process of building a fingerprint index.

[0282] like Figure 9 As shown in the figure, a fingerprint index construction method is provided in an embodiment of this application. The method includes the following steps:

[0283] Step 901: The construction device splits the fingerprints in the first fingerprint set into N fields. The execution method of step 901 is the same as that of step 301, as detailed in the foregoing description, and will not be repeated here.

[0284] Step 902: Each field corresponds to an index layer. The construction device constructs all index layers except the bottommost index layer. For any index layer other than the bottommost index layer, the index layer corresponds to a field in the fingerprint. Constructing the index layer includes: setting one or more sets of index entries in the index layer, each set of index entries including one or more index entries, where an index entry in the index layer represents a value of the field. An index entry in the index layer above this index layer corresponds to a set of index entries in this index layer. The execution method of step 901 is similar to that of step 302, except that the bottommost index layer is not constructed in step 902. For details, please refer to the foregoing description, which will not be repeated here.

[0285] It should be noted that the lowest-level index layer sets up one or more buckets, each bucket group includes one or more buckets, and each bucket includes one or more index entries. An index entry in the index layer above this lowest-level index layer corresponds to a bucket group in the lowest-level index layer. The index entry in the index layer above this lowest-level index layer records the storage address of the corresponding bucket group in the lowest-level index layer.

[0286] Step 903: The construction device constructs the lowest-level index layer. For the lowest-level index layer, the fields corresponding to this lowest-level index layer are further split into two subfields. To easily distinguish these two subfields, they are referred to as the first subfield and the second subfield, respectively.

[0287] Constructing the lowest-level index layer includes: setting up one or more buckets in this lowest-level index layer, each bucket containing multiple buckets, each bucket storing a first subfield, each bucket covering all possible values ​​of the first subfield, and each bucket representing one value of the first subfield. Each bucket contains multiple index entries, and each bucket stores a second subfield, the multiple index entries in this bucket covering all possible values ​​of the second subfield, and each index entry representing one value of the second subfield.

[0288] For any index entry in a bucket, the index entry represents a value of the second subfield given a certain value for the first subfield, where the value of the first subfield is the value represented by the bucket. A bucket can be understood as a storage space of a specified size where index entries can be written. The size of the bucket can be preset, and each bucket can have the same or different size.

[0289] Similarly, following this construction, there is a correspondence between index entries in each index layer. Starting from any index entry in the lowest index layer and moving upwards, a corresponding index entry can always be found in each index layer. In the lowest index layer, each index entry represents a fingerprint. The values ​​of each field of this fingerprint are consistent with the values ​​represented by that index entry in the lowest layer and the values ​​represented by the corresponding index entries in each of the index layers above it.

[0290] In the lowest-level index layer, index entries are "clustered" at the bucket level, grouping some entries into a single bucket. This design simplifies the fingerprint index structure. Furthermore, when querying based on this fingerprint index, the search scope is effectively narrowed down at the lowest level. For example, without clustering in the construction of the lowest-level index layer, a query based on this fingerprint index will reach a set of index entries corresponding to the upper-level index entries. However, with clustering in the construction of the lowest-level index layer, the search scope will reach a set of buckets corresponding to the upper-level index entries, and the value of the first subfield in the known fingerprint can quickly further narrow the search scope to one bucket within that set.

[0291] For fingerprint indexes used in scenarios requiring the acquisition of fingerprint-related information, the construction device also needs to add fingerprint-related information to each index entry in the lowest-level index layer; this fingerprint is the fingerprint represented by that index entry. For fingerprint indexes used in scenarios requiring the determination of whether data is stored or whether data already exists, it is not necessary to add fingerprint-related information to each index entry in the lowest-level index layer.

[0292] The method for constructing the lowest-level index layer is explained by considering that each index layer covers all possible values, valid index entries, and invalid index entries for that field in each fingerprint within the first fingerprint set:

[0293] If the first overlay method is used, the construction process is as follows:

[0294] When the construction device sets any set of buckets in the lowest index layer, it sets multiple buckets within that set. The number of buckets is equal to a third value, which is equal to the number of possible values ​​for the first sub-field. This third value represents the number of possible values ​​for the first sub-field when the first fingerprint set is not limited. The position of each bucket within these multiple buckets represents one possible value for the first sub-field. In other words, the possible values ​​for the first sub-field are ordered in ascending order of value, and the order position of a specific value is the position of the bucket representing that specific value within the multiple buckets.

[0295] For all possible values ​​of the first subfield in each fingerprint within the first fingerprint set, multiple index entries are set in the bucket representing each value. This bucket is the valid bucket.

[0296] If the multiple index entries within the valid bucket are arranged in Method 1: the construction device sets multiple index entries in the bucket, the number of index entries being equal to a fourth value, which is equal to the number of possible values ​​for the second subfield. The fourth value is the number of possible values ​​for the second subfield when the first fingerprint set is not limited. The position of each index entry within the multiple index entries represents one possible value for the second subfield. In other words, the possible values ​​of the second subfield are ordered sequentially according to their magnitude, and the order position of a specific value is the position of the index entry representing that specific value within the multiple index entries.

[0297] For each fingerprint in the first fingerprint set, the index entry representing all possible values ​​of the second subfield is a valid index entry. This index entry stores information related to the valid identifier and / or fingerprint.

[0298] For any value in the first fingerprint set that is not present in the second subfield, the index entry representing that value either does not record fingerprint-related information (applicable to scenarios involving querying fingerprint-related information) or records an invalid identifier to indicate that the value does not exist or that the index entry is invalid. This index entry is thus considered an invalid index entry.

[0299] If the multiple index entries within the valid bucket are arranged in Method 2: the construction device writes multiple valid index entries into the valid bucket, each valid index entry including the represented value, and optionally, also including relevant information of a valid identifier and / or fingerprint.

[0300] For values ​​that are not present in the first subfield of each fingerprint in the first fingerprint set, the bucket representing that value is an invalid bucket. In this bucket, no information may be recorded, or an invalid identifier may be recorded.

[0301] If the second overlay method is used, the construction process is as follows:

[0302] When the construction device sets any set of buckets at the lowest index layer, it sets multiple buckets within that set. The number of buckets is equal to a third value, which is equal to the number of possible values ​​for the first sub-field. This third value represents the number of possible values ​​for the first sub-field when the first fingerprint set is not limited. The position of each bucket within the multiple buckets represents one possible value for the first sub-field. In other words, the possible values ​​for the first sub-field are ordered in ascending order of value, and the order position of a specific value is the position of the bucket representing that specific value within the multiple buckets.

[0303] If the multiple index entries within a bucket are arranged in Method 1: the construction device sets multiple index entries in each bucket, and the number of index entries in each bucket is equal to a fourth value. This fourth value is equal to the number of possible values ​​for the second subfield. The fourth value is the number of possible values ​​for the second subfield when the first fingerprint set is not limited. The position of each index entry among the multiple index entries represents one possible value for the second subfield. In other words, the possible values ​​of the second subfield are ordered in ascending order of value, and the order position of a specific value is the position of the index entry representing that specific value among the multiple index entries.

[0304] For each fingerprint in the first fingerprint set, the index entry representing that value is a valid index entry. A valid index entry is one that represents a fingerprint, and stores a valid identifier and / or fingerprint-related information in that index entry.

[0305] For any value in the second subfield that is not present in any of the fingerprints in the first fingerprint set, the index entry representing that value either does not record any storage address or records an invalid identifier to indicate that the value does not exist or that the index entry is invalid. This index entry is thus considered invalid. A valid index entry is one that does not represent a fingerprint.

[0306] If the multiple index entries within the bucket are arranged in Method 2: the construction device writes multiple valid index entries into the bucket, each valid index entry including the represented value, and optionally, also including relevant information about a valid identifier and / or fingerprint.

[0307] In this embodiment, the size of each bucket in the bottommost index layer can be the same. The length of the index entries within each bucket can be the same or different, and the length of the index entries is related to the fingerprint-related information they contain.

[0308] According to such Figure 9 The construction method shown can construct fingerprint indexes with multiple layers of indexes, taking the construction of two-layer and three-layer fingerprint indexes as examples. Its construction method is similar to... Figure 4 as well as Figure 5 The construction methods shown are similar, the difference lies in the construction method of the lowest level index layer.

[0309] For a two-layer fingerprint index, the bottom layer is the second index layer. When constructing the second index layer, the corresponding field B is further split into two subfields. To easily distinguish these two subfields, they are referred to as subfield B1 and subfield B2, respectively.

[0310] Constructing a second index layer includes setting one or more sets of buckets in this second index layer. One set of buckets stores subfield B1, and each set of buckets comprises multiple buckets. Each bucket in each set represents a value of subfield B1. Each bucket includes multiple index entries, and each index entry represents a value of subfield B2.

[0311] For a three-layer fingerprint index, the bottom layer is the second index layer. When constructing the second index layer, the corresponding field B is further split into two subfields. To easily distinguish these two subfields, they are referred to as subfield B1 and subfield B2, respectively.

[0312] Constructing a third index layer includes setting one or more sets of buckets in this third index layer. One set of buckets stores subfield B1, and each set of buckets contains multiple buckets. Each bucket in each set represents one value of subfield B1. Each bucket contains multiple index entries, and each index entry represents one value of subfield B2.

[0313] Combining valid and invalid index entries, the construction method for the second index layer in a two-layer structure and the third index layer in a three-layer structure is similar to that for the lowest index layer. For details, please refer to the previous explanation, which will not be repeated here.

[0314] 2. The specific structure of the fingerprint index.

[0315] like Figure 10A The diagram shown is a schematic representation of a fingerprint index provided in an embodiment of this application.

[0316] The fingerprint index has a multi-layered structure, comprising multiple index layers. Each index layer corresponds to a field in the fingerprint, and each index layer is used to store the corresponding field.

[0317] Each index layer, except for the lowest index layer, consists of multiple index entries. Each index entry in a given index layer represents a value of the field corresponding to that index layer. Each index entry in a given index layer corresponds to a set of index entries in the next lower index layer.

[0318] The lowest-level index layer comprises multiple buckets, each containing multiple sub-buckets, and each sub-bucket contains multiple index entries. Each bucket corresponds to the first sub-field in the target field, which is the field corresponding to the lowest-level index layer. Each bucket also corresponds to one index entry in the index layer above it. Each bucket stores the first sub-field, covering all possible values ​​of that first sub-field, with one sub-bucket representing one possible value of the first sub-field. Each bucket stores the second sub-field, with multiple index entries in each bucket covering all possible values ​​of the second sub-field in the target field, and each index entry representing one possible value of the second sub-field.

[0319] Each index entry in the lowest index layer includes fingerprint-related information, which is the fingerprint represented by that index entry.

[0320] Combining the descriptions of index layers covering all possible values ​​of a field in each fingerprint within the first fingerprint set, index entries representing a single value of the corresponding field, and valid and invalid index entries, the lowest-level index layer sets up one or more buckets, each bucket containing one or more index entries. An index entry in the index layer above this lowest-level index layer corresponds to a bucket in the lowest-level index layer. The index entry in the index layer above this lowest-level index layer records the storage address of the corresponding bucket in the lowest-level index layer.

[0321] If the first covering method is used, the structure of the lowest index layer in the fingerprint index is as follows:

[0322] In this lowest-level index layer, each bucket group comprises multiple buckets. The number of buckets in each group is a third value, equal to the number of possible values ​​for the first sub-field. This third value represents the number of possible values ​​for the first sub-field when the first fingerprint set is not limited. The position of each bucket within the multiple buckets represents one possible value for the first sub-field. In other words, the possible values ​​of the first sub-field are ordered in ascending order of value, and the order position of a specific value represents the position of the bucket representing that specific value within the multiple buckets.

[0323] For each fingerprint in the first fingerprint set, for all possible values ​​of the first subfield, multiple index entries are set in the bucket representing that value. This bucket is a valid bucket.

[0324] The multiple index entries within this effective bucket are arranged in Method 1: the bucket contains multiple index entries, the number of which equals a fourth value, which equals the number of possible values ​​for the second subfield. The fourth value represents the number of possible values ​​for the second subfield when the first fingerprint set is not limited. The position of each index entry within the multiple index entries represents one possible value for the second subfield. In other words, the possible values ​​of the second subfield are ordered in ascending order of value, and the order position of a specific value is the position of the index entry representing that specific value within the multiple index entries.

[0325] For each fingerprint in the first fingerprint set, the index entry representing all possible values ​​of the second subfield is a valid index entry. This index entry includes relevant information about the valid identifier and / or fingerprint.

[0326] For any value in the second subfield that is not present in any of the fingerprints in the first fingerprint set, the index entry representing that value either does not record any storage address or records an invalid identifier to indicate that the value does not exist or that the index entry is invalid. This index entry is thus considered an invalid index entry.

[0327] If the multiple index entries within the valid bucket are arranged in Method 2: the valid bucket includes multiple valid index entries, each containing the value it represents, and optionally, also including relevant information about a valid identifier and / or fingerprint.

[0328] If the second covering method is used, the structure of the lowest index layer in the fingerprint index is as follows:

[0329] The structure of the lowest index layer in the fingerprint index using the second covering method is similar to that of the lowest index layer in the fingerprint index using the first covering method. The difference is that in the lowest index layer of the fingerprint index using the second covering method, there is no concept of invalid buckets, and each bucket contains multiple index entries.

[0330] Each index entry represents a fingerprint. If the fingerprint represented by the index entry does not exist in the first fingerprint set, then the index entry is invalid. If the fingerprint index is used for scenarios that query fingerprint-related information, i.e., the fingerprint index entry is required to record fingerprint-related information, the invalid index entry does not contain fingerprint-related information. Furthermore, it may include an invalid identifier to indicate that the represented fingerprint does not exist. If the fingerprint index is used for scenarios that use the fingerprint index to determine whether to store data or whether the data set contains data, the invalid index entry includes an invalid identifier to indicate that the represented fingerprint does not exist.

[0331] For multiple index entries within each bucket in the lowest index layer, the arrangement of the index entries is similar to that of the lowest index layer in a fingerprint index using the covering method.

[0332] From the fingerprint index construction process and its specific structure, we know that parameters affecting the specifications of the fingerprint index also include bucket size and number of buckets. The number of buckets and the number of index entries within each bucket are related to how the target field is partitioned. For example, if the first subfield formed after partitioning the target field is longer, then the first subfield has more possible values. Conversely, if the second subfield is shorter, then the second subfield has fewer possible values. Under this partitioning method, there are more buckets and fewer index entries within each bucket, and the bucket size can be preset.

[0333] The following section uses two-layer and three-layer fingerprint indexes as examples to further introduce the structure of the fingerprint index, so as to gain a clearer understanding of its structure.

[0334] The fingerprint in the first fingerprint set includes two fields, namely field A and field B.

[0335] like Figure 10B As shown, this fingerprint index comprises two index layers, referred to as the first index layer and the second index layer, respectively. For information on the fields corresponding to the first and second index layers and the meaning of the index entries, please refer to [link to relevant documentation]. Figure 2A The relevant explanations will not be repeated here.

[0336] Figure 2A The fingerprint indexes shown are different. Figure 10B In the fingerprint index shown, field B corresponding to the second index layer is further split into two subfields, subfield B1 and subfield B2. The second index layer includes multiple buckets. One index entry in the first index layer corresponds to one bucket in the second index layer. Since the first index layer includes 10 index entries, the second index layer includes 10 buckets.

[0337] There are multiple buckets within each bucket group, each storing one possible value for subfield B1. Each bucket in each group represents one possible value for subfield B1. Therefore, the number of buckets in each group is the same as the number of possible values ​​for subfield B1. Each bucket also contains multiple index entries, each representing one possible value for subfield B2. Therefore, the number of index entries in each bucket is the same as the number of possible values ​​for subfield B2.

[0338] The fingerprint in the first fingerprint set includes three fields: field A, field B, and field C.

[0339] like Figure 10CAs shown, this fingerprint index comprises three index layers, referred to as the first index layer, the second index layer, and the third index layer, respectively. For information on the fields corresponding to the first, second, and third index layers and the meaning of the index entries, please refer to [link to relevant documentation]. Figure 2B The relevant explanations will not be repeated here.

[0340] Figure 2A The fingerprint indexes shown are different. Figure 10C In the fingerprint index shown, field C corresponding to the third index layer is further split into two subfields, C1 and C2. The third index layer includes multiple buckets. One index entry in the second index layer corresponds to one bucket in the third index layer. Since the second index layer includes 64 index entries, it also includes 64 buckets.

[0341] There are multiple buckets within each bucket group, each storing one possible value for subfield C1. Each bucket in each group represents one possible value for subfield C1. Therefore, the number of buckets in each group is the same as the number of possible values ​​for subfield C1. Each bucket also contains multiple index entries, each representing one possible value for subfield C2. Therefore, the number of index entries in each bucket is the same as the number of possible values ​​for subfield C2.

[0342] Combining valid and invalid index entries, the structure of the second index layer in a two-layer structure and the third index layer in a three-layer structure are similar to the bottom index layer. For details, please refer to the previous explanation, which will not be repeated here.

[0343] 3. Fingerprint-based query method.

[0344] For example Figure 10A Using the fingerprint index structure shown as an example, we will explain the query method based on the fingerprint index. Figure 11 The diagram shown is a schematic representation of a fingerprint index provided in an embodiment of this application.

[0345] Step 1100: The query device obtains the target fingerprint to be queried and determines the values ​​of each field in the target fingerprint.

[0346] Step 1101: The query device starts from the topmost index layer of the fingerprint index and performs the first search operation in each index layer until it reaches the index layer above the bottommost index layer (i.e., the penultimate index layer). The explanation of the first search operation can be found in the relevant explanation of step 601 and will not be repeated here.

[0347] Step 1102: The query device performs a second search operation in the index layer above the lowest index layer: it searches for an index entry representing the value of the second target field in the target fingerprint within the search range of this index layer. The second target field is the field corresponding to this index layer. If the index entry is found, it jumps to a set of buckets in the lowest index layer corresponding to that index entry. If no index entry is found, the fingerprint-based query method is terminated.

[0348] The index entry representing the value of the second target field in the target fingerprint records the storage address of a set of buckets in the lowest index layer. The query device can jump to a set of buckets in the lowest index layer corresponding to the index entry based on the storage address.

[0349] Step 1103: The query device performs a third search operation in the lowest-level index layer: within the search range of this index layer, it searches for target buckets that represent the value of the first subfield in the target fingerprint, and within these target buckets, it searches for target index entries that represent the value of the second subfield in the target fingerprint. The search range of this index layer is the set of buckets that were jumped to in step 1102.

[0350] The query device can obtain the starting address BP of the group of buckets. The query device calculates the starting address of the bucket representing the value of the first sub-field in the target fingerprint. The sorting position of the value of the first sub-field in the target fingerprint among all possible values ​​of the first sub-field is BS. That is, among all possible values ​​of the first sub-field, there are BS-1 values ​​before the value of the first sub-field in the target fingerprint, and there are S-1 buckets before the bucket representing this value. Therefore, the starting address bp of the target bucket representing this value satisfies: bp = BP + (BS-1)*d, where d is the size of the bucket. In other words, the starting address of the target bucket representing this value is the starting address BP of the group of buckets in the fingerprint buckets plus an offset, which is equal to the total size of the buckets before the bucket representing this value, i.e., the offset is (BS-1)*d.

[0351] After determining the target bucket, the query device searches within the target bucket for target index entries that represent the value of the second sub-field in the target fingerprint. If multiple index entries within the target bucket are arranged in Mode 1, the query device searches among these multiple index entries for target index entries that represent the value of the second sub-field in the target fingerprint in a manner similar to the execution method of step 701. For details, please refer to the relevant description in 701, which will not be repeated here.

[0352] If multiple index entries within the target bucket are arranged in Method 2: the query device can traverse each index entry in the target bucket until a valid index entry containing the value of the second subfield B in the target fingerprint is found. If no valid index entry containing the value of the second subfield B in the target fingerprint is found, it means that the target fingerprint has not been found.

[0353] Step 1104: If a target index entry is found in the target bucket, it means that the target fingerprint has been found in the fingerprint index, and the query device can obtain the fingerprint-related information of the target fingerprint from the target index entry. If no target index entry is found in the target bucket, it means that the target fingerprint has not been found in the fingerprint index.

[0354] The following is based on such Figure 10C Using the fingerprint index shown in 10B as an example, this paper further introduces the query process based on fingerprint indexes to provide a clearer understanding of the fingerprint index-based query method.

[0355] I. The fingerprint index has a two-layer structure.

[0356] Based on such Figure 10B The fingerprint index query method shown is the same as Figure 7 The fingerprint-based query method shown is similar, the difference being: when based on, for example... Figure 10B When querying the fingerprint index, the query device, after determining the first index entry, jumps to a set of buckets in the second index layer corresponding to the first index entry. Within this set of buckets, a target bucket is determined, whose value matches the value of subfield B1 in the target fingerprint. After determining the target bucket, a second index entry is determined from multiple index entries within that target bucket, and this second index entry's value matches the value of subfield B2 in the target fingerprint.

[0357] II. The fingerprint index consists of a three-layer structure.

[0358] Based on such Figure 10B The fingerprint index query method shown is the same as Figure 8 The fingerprint-based query method shown is similar, the difference being: when based on, for example... Figure 10B When querying the fingerprint index, the query device, after determining the second index entry, jumps to a set of buckets in the third index layer corresponding to the second index entry. Within this set of buckets, a target bucket is determined, whose value matches the value of subfield C1 in the target fingerprint. After determining the target bucket, a third index entry is determined from multiple index entries within that target bucket, and this third index entry's value matches the value of subfield C2 in the target fingerprint.

[0359] The third index layer is the lowest index layer. The way the query device determines the third index item from a set of index items in the third index layer corresponding to the second index item is similar to the execution method of step 1103. For details, please refer to the above description, which will not be repeated here.

[0360] The following are two deployment methods for fingerprint indexes:

[0361] Deployment Method 1: The fingerprint index is deployed in a distributed storage system. The fingerprint index is stored in the data management node of this distributed storage system.

[0362] like Figure 12A The diagram shown is a system architecture schematic of a distributed storage system provided in an embodiment of this application. The distributed storage system includes a server cluster. The server cluster includes one or more servers 130 ( Figure 12A The diagram shows three servers 130 (but is not limited to two servers 130), and these servers 130 can communicate with each other. A server 130 is a device that has both computing and storage capabilities, such as a server or desktop computer. A server is a node in this distributed storage system.

[0363] In terms of hardware, such as Figure 12A As shown, server 130 includes at least processor 132, memory 133, network interface card 134, and hard disk 135. Processor 132, memory 133, network interface card 134, and hard disk 105 are connected via a bus.

[0364] Network interface card 134 can be used to communicate with other servers 130. Network interface card 134 can also be used to communicate with hard disk 135 to expand the capacity of server 130.

[0365] Processor 132 can be a central processing unit (CPU) or other specific integrated circuits. Processor 132 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0366] Memory 133 refers to the internal memory that directly exchanges data with processor 132. Memory 133 can be dynamic random access memory (DRAM). Besides DRAM, memory 133 can also be other random access memories, such as static random access memory (SRAM). Alternatively, memory 133 can also be read-only memory (ROM). For example, read-only memory can be programmable read-only memory (PROM) or erasable programmable read-only memory (EPROM). Memory 133 can also be flash memory or other types of memory. Memory 133 stores the computer program instructions required by processor 132 to perform data operations.

[0367] Hard disk 135 is used to provide storage resources, such as storing data. It can be a disk or other types of storage media, such as a solid-state drive or a shingled magnetic recording hard disk.

[0368] exist Figure 12A The fingerprint index deployed in the distributed storage system shown can be a "global" fingerprint index, which records the fingerprints of data in all servers (such as server hard drives) within the storage system. Although not shown, the storage system may also include a server for managing data. The structure of this server is similar to that of server 130, as described above, and will not be repeated here. This server can serve as the data management node in the distributed storage system, and the global fingerprint index is stored on this server.

[0369] For example, the fingerprint information included in the index entries of this global fingerprint index represents the node where the data resides. In this distributed storage system, to query the node where a specific data resides, the server storing the global fingerprint index can use the fingerprint of the data to query the global fingerprint index. If the target index entry is found in the lowest level of the global fingerprint index, then the node where the data resides is retrieved from that target index entry. If the target index entry is not found in the lowest level of the global fingerprint index, then it means that the data is not stored in this distributed storage system.

[0370] In the foregoing description, the server used for data management is considered a separate server. In practical applications, one or more servers 130 in the distributed storage system can undertake the data management function. That is, the global fingerprint index can be shared by one or more servers 130, and all one or more servers 130 can access the global fingerprint index. For example, shared storage spaces (which can be located in the memory of server 130) are deployed in one or more servers 130, and the global fingerprint index is stored in these shared storage spaces. One or more servers 130 can access these shared storage spaces and thus read the global fingerprint index. Another example is that the global fingerprint index is distributed across one or more servers 130, and the global fingerprint index includes multiple fingerprint indexes. One of the one or more servers 130 stores one fingerprint index, and the structure of each fingerprint index is as follows... Figure 1 or Figure 10A As shown in the diagram. Any server 130 can query the fingerprint index to determine the node where the data corresponding to the fingerprint to be queried is located.

[0371] It should be noted that the above explanation uses the example of a global fingerprint index recording fingerprints of data from all servers (such as server hard drives) in the storage system. In practical applications, this global fingerprint index can record only the fingerprints of a subset of data from all servers in the storage system. That is, when constructing the global fingerprint index, the fingerprints of the data in the storage system are sampled, extracting only the fingerprints of a portion of the data, and the global fingerprint index is constructed only from the fingerprints of that subset. This sampling method reduces the number of fingerprints recorded in the global fingerprint index, thus reducing the storage space it occupies.

[0372] Deployment Method 2: The fingerprint index is deployed in a distributed storage system. The fingerprint index is stored on each node of this distributed storage system.

[0373] like Figure 12B The diagram shown is a schematic of the system architecture of a distributed storage system provided in an embodiment of this application. In this distributed storage system, the computing and storage functions required by the distributed storage system are deployed on different nodes. The node with computing function is called a computing node, and the node with storage function is called a storage node.

[0374] exist Figure 12B The distributed storage system comprises a compute node cluster and a storage node cluster. The compute node cluster includes one or more compute nodes (140...). Figure 12BThe diagram shows two compute nodes 140 (but is not limited to two compute nodes 140), and the compute nodes 140 can communicate with each other. A compute node 140 is a computing device, such as a server, desktop computer, or controller of a storage array.

[0375] Compute node 140 has computing capabilities and can receive and process data requests sent from outside the storage system. If the data request is for accessing data or for processing data, compute node 140 can send a data access request to storage node 150 to request access to the data from the storage node or to retrieve the data from the storage node.

[0376] In terms of hardware, such as Figure 12B As shown, the compute node 140 includes at least a processor 142, memory 143, and a network interface card (NIC) 144. The specific form of the processor 142 is similar to that of the processor 132, as described above, and will not be repeated here. It is used to process data requests and trigger the NIC 144 to request data access from the storage node.

[0377] Memory 143 is used to store computer program instructions required by the processor 142 to process data requests. It can also temporarily store data. When the data stored in memory 143 reaches a certain threshold, the computing node 140 can send a data access request to the storage node 150 via network interface card 144 to persistently store the data. Network interface card 144 is used to communicate with storage node 150.

[0378] Any compute node 140 can access any storage node 150 in the storage node cluster via the network. The storage node cluster includes multiple storage nodes 150. Figure 12B The diagram shows three storage nodes 150, but is not limited to three storage nodes 150. Each storage node 150 includes one or more control units 151, a network interface card (NIC) 154, and multiple hard disks 155. The NIC 154 is used to communicate with the compute node 140. The hard disks 155 are used to store data and can be disks or other types of storage media, such as solid-state drives (SSDs) or shingled magnetic recording hard disks (SMARDs).

[0379] The functions of the control unit 151 (equivalent to the processor in storage node 150), network card 154 and multiple hard disks 155 are similar to those of the processor 142, network card 144 and hard disk 135 mentioned above. For details, please refer to the above description and it will not be repeated here.

[0380] exist Figure 12B In the distributed storage system shown, a fingerprint index can be deployed in each storage node. The fingerprint index records the fingerprint of the data within that storage node (such as the hard drive of the storage node).

[0381] For example, the fingerprint-related information included in the index entries of this fingerprint index within each storage node serves as the data's storage address. When a storage node needs to query certain data, it can use the data's fingerprint to query the fingerprint index. If the target index entry is found in the lowest level of the fingerprint index, the storage address of the data is retrieved from that target index entry, and thus the data is obtained. If the target index entry is not found in the lowest level of the fingerprint index, it means that the data is not stored in that storage node.

[0382] For example, the fingerprint index within each storage node does not include fingerprint-related information in its index entries. When a storage node needs to query whether it stores certain data, it can use the fingerprint of that data to query the fingerprint index. If the target index entry is found in the lowest level of the fingerprint index, then it is determined that the data is stored in that node, and the data can be retrieved. If the target index entry is not found in the lowest level of the fingerprint index, then it means that the data is not stored in that storage node.

[0383] For a dataset, the larger the dataset (that is, the larger the number of data fragments it contains), the more fingerprints the data fragments will have. If all the data fingerprints in the dataset are recorded in the fingerprint index, the fingerprint index will be quite large, and storing the fingerprint index will occupy a lot of storage space.

[0384] To reduce the fingerprint storage space occupied by the fingerprint index, a fingerprint sampling method can be used. That is, the index entries (i.e., the index entries at the lowest level) only need to represent the fingerprints of a portion of the data fragments in the dataset. Furthermore, a partitioned storage method can be used when saving the fingerprint index. This means the fingerprint index is distributed and stored across multiple partitions, where each partition can be understood as a segment of storage space. These partitions can reside on the same storage device or on different storage devices. The storage device can be a memory or a computing device with storage capabilities.

[0385] Currently, sampling-based saving is commonly used to save fingerprint tables, which contain multiple key-value pairs. The key in each pair represents a fingerprint, and the value can be the data corresponding to that fingerprint or the storage address of that data. Sampling-based saving can use modulo 0 to determine which key-value pairs are retained in the fingerprint table. For example, with a modulo of 5, the fingerprint table will retain key-value pairs whose fingerprint values ​​are multiples of 5. That is, it can save key-value pairs with fingerprint values ​​of 0, 5, 10, 15, 20, 25, 30, 35, and so on. These key-value pairs can be distributed across multiple partitions.

[0386] This sampling method has certain drawbacks. For a fingerprint table that has already been sampled and saved, changing the sampling ratio (i.e., changing the modulus) will cause a large range of fluctuations in the key-value pairs within the fingerprint table. It may be necessary to add new key-value pairs to the fingerprint table and delete some existing key-value pairs.

[0387] For example, assuming the sampling modulus of the fingerprint table that has been sampled and saved is 5, then the fingerprint table that has been sampled and saved currently contains key-value pairs with fingerprint values ​​of 0, 5, 10, 15, 20, 25, 30, 35...

[0388] If the sampling ratio of the fingerprint table is changed to a modulus of 7, then the fingerprint table needs to retain key-value pairs whose fingerprint values ​​are multiples of 7. That is, it can store key-value pairs with fingerprint values ​​of 0, 7, 14, 21, 28, 35, 42, 49...

[0389] As can be seen, before and after the sampling ratio change, only a small portion of the key-value pairs that need to be retained in the fingerprint table are the same. Most of the key-value pairs are different, so when changing the sampling ratio, it is necessary to delete most of the key-value pairs already stored in the fingerprint table.

[0390] If the fingerprint table uses a partitioned storage method, then when the sampling ratio is changed, the key-value pairs included in most of the partitions of the fingerprint table need to be adjusted.

[0391] Taking the aforementioned sampling ratio change from modulo 5 to modulo 7 as an example, assuming the fingerprint table is distributed across 8 partitions, the sequence number of the key-value pairs stored in each partition before and after the sampling ratio change is shown in Table 1.

[0392] Table 1

[0393] Sampling ratio Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 Partition 7 Partition 8 Modulo 0: Modulus is 5 0 5 10 15 20 25 30 35 Modulus 0: Modulus is 7 0 7 14 21 28 35 42 49

[0394] As can be seen from Table 1, changing the sampling ratio will cause changes in the key-value pairs recorded in these 7 partitions, resulting in a loss of deduplication rate.

[0395] This application provides a method for sampling and saving fingerprint indexes. In this method, the sampling ratio can be represented by the value range of the fingerprints. Specifically, fingerprints from data segments whose fingerprint values ​​fall within the specified range are extracted from all current data segments. The proportion of extracted fingerprints from data segments whose values ​​fall within the specified range to the total number of fingerprints in all current data segments is equal to the sampling ratio. The method includes the following steps:

[0396] Step 1: Obtain the sampling ratio to determine the fingerprint value range. For example, if the sampling ratio is 16% and there are 100 fingerprints in the current data slice, then the fingerprint value range can be 0-16. Similarly, if the sampling ratio is 12% and there are 100 fingerprints in the current data slice, then the fingerprint value range can be 0-12. Assume the fingerprint value range is: the fingerprint value is greater than a first value and less than a second value. The first and second values ​​are the lower and upper limits of the fingerprint. The lower limit of the upper limit of the fingerprint is related to the sampling ratio. Changes in the sampling ratio will change the upper and / or lower limits of the fingerprint.

[0397] Step 2: Extract fingerprints from all current data fragments (such as those included in the aforementioned second fingerprint set) whose fingerprint values ​​fall within the specified range. The extracted fingerprint value is greater than the first value and less than the second value.

[0398] Step 3: Construct a fingerprint index for the fingerprints of the extracted data fragments (the fingerprints of the extracted data fragments can be fingerprints included in the first fingerprint set mentioned above).

[0399] Step 4: Distribute and store the fingerprint index across multiple partitions.

[0400] Regarding step 4, this application embodiment does not limit the method of fingerprint index partition storage. The following will use... Figure 1 Taking the fingerprint index shown as an example, two partitioned storage methods are listed:

[0401] Method 1: When storing the fingerprint index in partitions, it can be split into multiple first fingerprint sub-indexes. The structure of each first fingerprint sub-index is similar to that of the fingerprint index, meaning each first fingerprint sub-index has a multi-level structure. Each first fingerprint sub-index includes multiple index layers, and each index layer stores a field from the fingerprint. Each index entry in each index layer represents a value of the stored field. An index entry in one index layer corresponds to a set of index entries in the next index layer. Each partition stores one first fingerprint sub-index.

[0402] like Figure 13A The diagram shows a fingerprint index and a first fingerprint sub-index. The first fingerprint sub-index has the same structure as the fingerprint index, but the difference is that the first fingerprint sub-index records only a portion of the fingerprints recorded in the fingerprint index.

[0403] This application does not limit the method of splitting the fingerprint index into multiple first fingerprint sub-indexes. For example, for any first fingerprint sub-index, the first fingerprint sub-index includes a portion of the index entries of the topmost index layer in the fingerprint index. Furthermore, the first fingerprint sub-index also includes the corresponding index entries of the portion of the index entries in other index layers of the fingerprint index (i.e., all index layers except the topmost index layer).

[0404] For example, for any first fingerprint sub-index, the first fingerprint sub-index includes a portion of the index entries from the lowest index layer of the fingerprint index. Furthermore, the first fingerprint sub-index also includes the corresponding index entries of the portion of the index entries in other index layers of the fingerprint index (i.e., all index layers except the lowest index layer).

[0405] In this approach, the first fingerprint sub-index of each partition records a subset of fingerprints from the first fingerprint set. That is, the fingerprint represented by the index entry in the first fingerprint sub-index is a subset of the fingerprints in the first fingerprint set.

[0406] When initially constructing each first fingerprint sub-index, the construction device can first determine the fingerprints that need to be recorded in the first fingerprint sub-index entries of each partition. That is, the construction device can split the first fingerprint set into multiple fingerprint subsets. The embodiments of this application do not limit the method by which the construction device splits the first fingerprint set into multiple fingerprint subsets.

[0407] For example, the construction device divides the first fingerprint set into multiple fingerprint subsets based on the fourth field of the first fingerprint set. The fingerprints included in each fingerprint subset have the same value for the fourth field. The fingerprints in different fingerprint subsets have different values ​​for the fourth field. The first fingerprint index entry within each partition is used to record the fingerprints in each fingerprint subset.

[0408] The fourth field may be different from the first, second, or third field, or it may be the same as the first, second, or third field, or it may be partially the same as the first, second, or third field (e.g., the fourth field may have overlapping character segments with the first field).

[0409] For example, the constructing device divides the first fingerprint set into multiple fingerprint subsets based on the hash values ​​of each fingerprint in the first fingerprint set. Each fingerprint subset contains fingerprints with the same hash value. The fingerprints in different fingerprint subsets have different hash values.

[0410] After determining the plurality of fingerprint subsets, the construction device constructs a first fingerprint sub-index for any given fingerprint subset. The method by which the construction device constructs the first fingerprint sub-index based on the fingerprint subset is similar to the method of constructing the fingerprint index, the only difference being that: in the aforementioned description, the fingerprint index is constructed based on a first fingerprint set, while here it is constructed based on a fingerprint subset, thus the range of fingerprints targeted is different.

[0411] After constructing each first fingerprint sub-index, the construction device stores each first fingerprint sub-index in a partition.

[0412] When a query device needs to query a target fingerprint, it can first determine the first fingerprint sub-index that may record the target fingerprint, that is, determine the partition where the first fingerprint sub-index is located. After determining the partition,

[0413] For example, when the building device splits the first fingerprint set into multiple fingerprint subsets based on the fourth field in the first fingerprint set, the query device can determine the fingerprint subset to which the target fingerprint belongs based on the fourth field, and then determine the partition where the first fingerprint sub-index is located.

[0414] The constructing device splits the first fingerprint set into multiple fingerprint subsets based on the hash values ​​of each fingerprint in the first fingerprint set. The querying device can determine the fingerprint subset to which the target fingerprint belongs based on the hash value, and thus determine the partition where the first fingerprint sub-index is located.

[0415] Of course, in practical applications, the query device may not be certain about the first fingerprint sub-index that might record the target fingerprint, that is, it may not be certain about the partition where the first fingerprint sub-index is located. The query device can simultaneously query the first fingerprint sub-index in each partition.

[0416] Method 2: When storing the fingerprint index in partitions, it can be split into multiple second fingerprint sub-indexes according to the multi-layered structure of the fingerprint index. Each second fingerprint sub-index includes a portion of the fingerprint index's layers. Each partition stores one second fingerprint sub-index. That is, each second fingerprint sub-index contains only a portion of the fingerprint index's layers.

[0417] like Figure 13B The diagram illustrates the fingerprint index and the second fingerprint sub-index. Each second fingerprint sub-index contains only a portion of the layers within the fingerprint index. When two adjacent index layers in the fingerprint index reside in different second fingerprint sub-indexes, the second fingerprint sub-index containing the upper index layer of those two adjacent index layers can point to the second fingerprint sub-index containing the lower index layer of those two adjacent index layers. Furthermore, any index entry in the upper index layer can point to a set of index entries within the lower index layer of the second fingerprint sub-index containing the lower index layer.

[0418] In this approach, there is a correspondence between the second fingerprint sub-indexes of each partition (i.e., a correspondence between index layers).

[0419] When the query device needs to query a target fingerprint, it can start the query from the second fingerprint sub-index containing the top-level index layer, and jump between the various second fingerprint sub-indexes. The query process is similar to the aforementioned method of querying using fingerprint indexes, the only difference being that the various index layers may be located in different partitions. For details, please refer to the aforementioned explanation; it will not be repeated here.

[0420] For example, if the range of fingerprint values ​​is 0 to 16, then the fingerprint index needs to record fingerprints with values ​​of 0, 1, 2, 3, ..., 16.

[0421] If we want to change the sampling ratio (that is, the range of fingerprint values), assuming the range of fingerprint values ​​becomes 0 to 12, that is, only the upper limit of the fingerprint value is changed, then the fingerprint index needs to record fingerprints with values ​​of 0, 1, 2, 3..., 12.

[0422] It is evident that, before and after the change in sampling ratio, the proportion of identical index entries among the index entries to be retained in the fingerprint index remains relatively large; that is, most index entries are the same, with only a small portion differing. The change in sampling ratio will not cause significant changes to the fingerprints recorded in the fingerprint index.

[0423] If the fingerprint index is stored in a partitioned manner, then when the sampling ratio is changed, the index entries included in a small portion of the partitions in which the fingerprint index is distributed need to be adjusted.

[0424] The fingerprint index is used for partitioned storage in Method 1. Taking the aforementioned sampling ratio change from the range of values ​​(0-16) to the range of values ​​(0-12) as an example, assuming that the fingerprint table is distributed in 8 partitions, the sequence number of the key-value pairs stored in each partition before and after the sampling ratio change is shown in Table 2.

[0425] Table 2

[0426] Sampling ratio Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 Partition 7 Partition 8 Value range: 0~16 0、8 1、9 2、10 3、11 4、12 5、13 6、14 7、15 Value range: 0~12 0、8 1、9 2、10 3、11 4 5 6 7

[0427] As can be seen from Table 2, changing the sampling ratio will cause changes in the index entries recorded in this small portion of the partitions, with almost no loss in the deduplication rate.

[0428] In this application embodiment, the fingerprint index can be stored in a partitioned manner, meaning the fingerprint index can be distributed and stored across multiple partitions. In this application embodiment, the partitioned fingerprint index supports scaling up or down the number of partitions.

[0429] ① Increase the number of partitions.

[0430] Expanding the number of partitions refers to increasing the number of partitions used to distribute the fingerprint index. An increase in the number of partitions indicates an increase in the storage space that the fingerprint index can occupy. When the number of partitions increases, new index entries that need to be written to the fingerprint index can be stored in the newly added partitions, or some existing index entries in the fingerprint index can be migrated to the newly added partitions.

[0431] If the fingerprint index is stored using Partition 1, when the number of partitions is increased, new index entries that need to be written to the fingerprint index can be stored in the newly added partition. These new index entries refer to the index entries to be added in the lowest-level index layer, as well as the corresponding index entries in other index layers. Alternatively, some existing index entries in the fingerprint index can be migrated to the newly added partition. These existing index entries are some of the index entries in the lowest-level index layer of the fingerprint index, as well as the corresponding index entries in other index layers. In other words, a first fingerprint sub-index is stored in the newly added partition.

[0432] If the fingerprint index is stored in partitioned mode 2, when the number of partitions is increased, the second fingerprint sub-index can be further split into multiple third fingerprint sub-indexes, and the third fingerprint sub-indexes can be stored in the increased partitions. This application does not limit the splitting method of the second fingerprint sub-index; for example, the second fingerprint sub-index can be split at the layer level.

[0433] ② Reduction of the number of partitions.

[0434] Reducing the number of partitions refers to decreasing the number of partitions distributed across the fingerprint index, which means reducing the storage space that the fingerprint index can occupy. When it is necessary to reduce the number of partitions, some partitions need to be deleted while others are retained. This can be achieved by migrating index entries stored in the partitions to be deleted to the retained partitions, or by directly deleting the index entries in the partitions to be deleted. In this embodiment, the method of migrating index entries stored in the partitions to be deleted to the retained partitions is not limited. For example, index entries written to the fingerprint index in the order they were written can be stored in the retained partitions, meaning that index entries written in the partitions to be deleted within that period can be migrated to the retained partitions, and index entries written before that period can be deleted from the retained partitions.

[0435] If the fingerprint index is stored using Partition 1, reducing the number of partitions can be achieved by directly deleting the first fingerprint sub-index of one or more partitions. Alternatively, the order in which index entries were written to the lowest level of each first fingerprint sub-index can be considered, retaining the index entries written to the lowest level of each first fingerprint sub-index within the most recent period. Then, the first fingerprint sub-indexes in the partitions to be deleted are merged into the first fingerprint sub-indexes of the retained partitions. This reduces the number of first fingerprint sub-indexes.

[0436] If the fingerprint index is stored in mode 2 partitioning, when reducing the number of partitions, the order in which index entries are written to the lowest level of the fingerprint index can be considered. Index entries written to the lowest level within the most recent period can be retained. Then, the second fingerprint sub-indexes in the partitions that need to be deleted can be merged into the second fingerprint sub-indexes of the retained partitions. This reduces the number of second fingerprint sub-indexes.

[0437] It is worth noting that the description of fingerprint sampling and partitioned storage uses the fingerprint index provided in the embodiments of this application as an example. In fact, the fingerprint sampling and partitioned storage method is also applicable to fingerprint indexes or fingerprint tables with other structures. The implementation principle is similar to the fingerprint sampling and partitioned storage method performed on the fingerprint index provided in the embodiments of this application, and will not be repeated here.

[0438] Based on the same inventive concept as the method embodiments, this application also provides a construction apparatus for executing the method executed by the construction apparatus in the above method embodiments. For example... Figure 14 As shown, the construction device 1400 includes a first construction module 1401 and a second construction module 1402. Optionally, it also includes a third construction module 1403, a sampling module 1404, and a storage module 1405. Specifically, in the construction device 1400, the modules are connected through a communication path. The fingerprint of the first fingerprint set includes a first field and a second field. The construction device 1400 includes:

[0439] The first construction module 1401 is used to construct the first index layer of the fingerprint index. Constructing the first index layer of the fingerprint index includes setting multiple index items in the first index layer, where one index item in the first index layer represents a value of the first field.

[0440] The second construction module 1402 is used to construct the second index layer of the fingerprint index. Constructing the second index layer of the fingerprint index includes setting multiple sets of index items in the second index layer. One index item in the first index layer corresponds to one set of index items in the second index layer. One index item in the set of index items in the second index layer represents a value of the second field.

[0441] As one possible implementation, the second construction module 1402 stores information related to fingerprints in the first fingerprint set in the index entries of the second index layer. The information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0442] In one possible implementation, the fingerprints in the first fingerprint index set further include a third field, and the construction device further includes a third construction module 1403, which constructs a third index layer of the fingerprint index. Constructing the third index of the fingerprint index includes: setting multiple sets of index entries in the third index layer; one index entry in the second index layer corresponds to one set of index entries in the third index layer; one index entry in the set of index entries in the third index layer represents a value of the third field; and storing relevant information about the fingerprints in the first fingerprint set in the index entries of the third index layer. The relevant information about the fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0443] As one possible implementation, the third field includes a first subfield and a second subfield. The third construction module 1403 sets at least one set of buckets in the third index layer. An index item in the second index layer corresponds to a set of buckets in the third index layer. Each bucket in the set of buckets includes one or more index items. Each bucket represents a value of the first subfield, and an index item in each bucket represents a value of the second subfield.

[0444] In one possible implementation, multiple index entries in the first index layer include a first index entry, and a set of index entries in the second index layer corresponding to the first index entry includes a second index entry; the second index entry stores relevant information about the first fingerprint in the first fingerprint set, the value of the first field in the first fingerprint is consistent with the value represented by the first index entry, and the value of the second field in the first fingerprint is consistent with the value represented by the second index entry.

[0445] In one possible implementation, the first index layer includes a third index item among multiple index items, the third index item is followed by a fourth index item among a set of index items in the second index layer, the fourth index item is followed by a fifth index item among a set of index items in the third index layer, and the fifth index item stores relevant information about the second fingerprint in the first fingerprint set. The value of the first field in the second fingerprint is consistent with the value represented by the third index item, the value of the second field in the second fingerprint is consistent with the value represented by the fourth index item, and the value of the third field in the second fingerprint is consistent with the value represented by the fifth index item.

[0446] In one possible implementation, the construction apparatus further includes a sampling module 1404, which samples fingerprints from a second fingerprint set based on a sampling ratio. The first fingerprint set includes the sampled fingerprints. The second fingerprint set includes fingerprints of data in a storage system or storage node, and the number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

[0447] In one possible implementation, the fingerprint values ​​in the first fingerprint set are greater than a first value and less than a second value, where the second value is determined based on the sampling ratio.

[0448] As one possible implementation, the fingerprint values ​​in the first fingerprint set are greater than a first value and less than a second value, where the first value is determined based on the sampling ratio.

[0449] As one possible implementation, the construction device also includes a storage module 1405, which distributes the fingerprint index across N partitions of the storage device, each partition being a segment of storage space, where N is a positive integer.

[0450] As one possible implementation, the storage device includes N nodes, each node including a partition.

[0451] Based on the same inventive concept as the method embodiments, this application also provides a query device for querying a target fingerprint in a first fingerprint set based on a fingerprint index, wherein the fingerprint index is used to store the first fingerprint set. This query device is used to execute the method executed by the query device in the above method embodiments. Figure 15 As shown, the query device 1500 includes a first query module 1501, a second query module 1502, and optionally, a third query module 1503. Specifically, in the query device 1500, the modules are connected through a communication path.

[0452] The query device 1500 is used to query a target fingerprint based on a fingerprint index in a first fingerprint set. The fingerprint index stores the first fingerprint set and includes a first index layer and a second index layer. The first index layer stores the first field of the fingerprints in the first fingerprint set, and the second index layer stores the second field of the fingerprints in the first fingerprint set. One index entry in the first index layer corresponds to a set of index entries in the second index layer. The query device 1500 includes:

[0453] The first query module 1501 is used to determine whether a first index item exists among multiple index items in the first index layer, and the value of the first index item is consistent with the value of the first field of the target fingerprint.

[0454] The second query module 1502 is used to determine whether a second index item exists in a set of index items in the second index layer corresponding to the first index item if a first index item exists among multiple index items in the first index layer. The value represented by the second index item is consistent with the value of the second field in the target fingerprint.

[0455] As one possible implementation, the second query module 1502 obtains relevant information about the target fingerprint from the second index item. The relevant information about the target fingerprint includes some or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the region where the target data is located.

[0456] As one possible implementation, the fingerprint index also includes a third index layer, which is used to store the third field of the fingerprint in the first fingerprint set. One of the index entries in the third index layer represents a value of the third field. The query device 1500 also includes a third query module 1503, which determines whether a third index entry exists in a set of index entries in the third-level index corresponding to the second index entry. The value represented by the third index entry is consistent with the value of the third field in the target fingerprint.

[0457] If a third index entry exists in a set of index entries in the third-level index corresponding to the second index entry, the relevant information of the target fingerprint is obtained from the third index entry. The relevant information of the target fingerprint includes some or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the region where the target data is located.

[0458] As one possible implementation, the third index layer is also used to store information related to fingerprints in the first fingerprint set. The information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0459] In one possible implementation, the third field includes a first subfield and a second subfield. The third index layer includes at least one set of buckets. An index term of the second index layer corresponds to a set of buckets in the third index layer. Each set of buckets includes one or more buckets, and each bucket includes one or more index terms. Each bucket represents a value of the first subfield, and an index term of each bucket represents a value of the second subfield. The third query module 1503 determines the target bucket in a set of buckets in the third-level index corresponding to the second index term. The value represented by the target bucket is consistent with the value of the first subfield in the target fingerprint. The third index term is determined in the target bucket. The value represented by the third index term is consistent with the value of the second subfield in the target fingerprint.

[0460] As one possible implementation, if the first index item is not present among the multiple index items in the first index layer, the first query module 1501 determines that the target fingerprint does not exist in the first fingerprint set.

[0461] If the second index entry is not present in a set of index entries in the second index layer corresponding to the first index entry, the second query module 1502 determines that the target fingerprint does not exist in the first fingerprint set.

[0462] If the third index term does not exist in a set of index terms of the third index layer corresponding to the second index term, the third query module 1503 determines that the target fingerprint does not exist in the first fingerprint set.

[0463] In one possible implementation, the first fingerprint set includes fingerprints obtained by sampling from the second fingerprint set based on a sampling ratio. The second fingerprint set includes fingerprints of data in a storage system or storage node. The number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

[0464] As one possible implementation, the fingerprint index is distributed across N partitions of the storage device, where each partition is a segment of storage space and N is a positive integer.

[0465] In one possible implementation, the storage device includes N nodes, each node including a partition.

[0466] The module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0467] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0468] This application also provides, for example Figure 16 The computing device 1600 shown includes a bus 1601, a processor 1602, a communication interface 1603, and a memory 1604. The processor 1602, the memory 1604, and the communication interface 1603 communicate with each other via the bus 1601.

[0469] The processor 1602 can be a CPU, or other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0470] Memory 1604 can be DRAM. Besides DRAM, memory 1604 can also be other random access memory (such as SRAM). Additionally, memory 1602 can also be ROM. For read-only memory, for example, it could be PROM, EPROM, etc. Memory 1604 can also be flash memory, HDD, or SSD.

[0471] The memory 1604 stores computer program instructions, and the processor 1602 executes these computer program instructions to perform the aforementioned tasks. Figure 3 , Figure 4 , Figure 5 , Figure 9 , Figure 10A , Figure 10B , Figure 10C The described method comprises the steps performed by the construction apparatus. The memory 1604 may also include other software modules required for running processes, such as an operating system (e.g., multiple modules in the construction apparatus 1400). The operating system may be LINUX. TM UNIX TM WINDOWS TM wait.

[0472] Alternatively, the memory 1604 stores computer program instructions, and the processor 1602 executes these computer program instructions to perform the aforementioned tasks. Figure 6 , Figure 7 , Figure 8 , Figure 11 The steps performed by the query device in the described method. The memory 1604 may also include other software modules required for running processes, such as an operating system (e.g., multiple modules in the query device 1500). The operating system may be LINUX. TM UNIX TM WINDOWS TM wait.

[0473] This application also provides a computing device system, the computing device system including at least one such as Figure 17 The computing device 1700 shown includes a bus 1701, a processor 1702, a communication interface 1703, and a memory 1704. The processor 1702, memory 1704, and communication interface 1703 communicate with each other via the bus 1701. At least one computing device 1700 in the computing device system communicates with each other via a communication path.

[0474] The specific types of processor 1702 and memory 1704 can be found in the relevant descriptions of processor 1602 and memory 1604, and will not be repeated here. Processor 1702 executes the computer program instructions stored in memory 1704 to perform the aforementioned tasks. Figure 3 , Figure 4 , Figure 5 , Figure 9 , Figure 10A , Figure 10B , Figure 10C The described method includes some or all of the steps performed by the construction device. The memory may also include other software modules required for running processes, such as an operating system. The operating system may be Linux. TM UNIX TM WINDOWS TM wait.

[0475] At least one computing device 1700 in the computing device system establishes communication with each other through a communication network, and each computing device 1700 runs any one or any multiple modules of the configuration device 700.

[0476] Alternatively, processor 1702 executes the computer program instructions stored in memory 1704 to perform the aforementioned actions. Figure 6 , Figure 7 , Figure 8 , Figure 11 The described method includes some or all of the steps performed by the query device. The memory may also include other software modules required for running processes, such as an operating system. The operating system may be Linux. TM UNIX TM WINDOWS TM wait.

[0477] At least one computing device 1700 in the computing device system establishes communication with each other through a communication network, and each computing device 1700 runs any one or any multiple modules of the management device 800.

[0478] The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis. For parts of a process that are not described in detail, please refer to the relevant descriptions of other processes.

[0479] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, in the form of a computer program product. A computer program product includes computer program instructions, which, when loaded and executed on a computer, generate, in whole or in part, the product according to the embodiments of the present invention. Figure 3 , Figure 4 , Figure 5 , Figure 9 , Figure 10A , Figure 10B , Figure 10C , Figure 6 , Figure 7 , Figure 8 ,or Figure 11 The process or function described.

[0480] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD).

[0481] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A storage device, characterized in that, The storage device is used to store a fingerprint index, which includes a first index layer and a second index layer. The first index layer is used to store a first field of a fingerprint in a first fingerprint set, and the second index layer is used to store a second field of a fingerprint in the first fingerprint set. The first index layer includes multiple index entries, and one index entry in the first index layer represents a value of the first field; one index entry in the first index layer corresponds to a set of index entries in the second index layer; one index entry in the second index layer represents a value of the second field.

2. The storage device as claimed in claim 1, characterized in that, The second index layer is also used to store information related to fingerprints in the first fingerprint set. The information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

3. The storage device as claimed in claim 1, characterized in that, The fingerprint index further includes a third index layer, which is used to store the third field of the fingerprints in the first fingerprint set and related information of the fingerprints. One index item of the second index layer corresponds to a set of index items of the third index layer. One index item of the third index layer represents a value of the third field. The related information of the fingerprint includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

4. The storage device as claimed in claim 3, characterized in that, The third field includes a first subfield and a second subfield. The third index layer includes at least one set of buckets. An index item of the second index layer corresponds to a set of buckets in the third index layer. Each bucket in the set of buckets includes one or more index items. Each bucket represents a value of the first subfield, and an index item of each bucket represents a value of the second subfield.

5. The storage device as claimed in claim 2, characterized in that, The first index layer includes multiple index entries, including a first index entry. The second index layer, corresponding to the first index entry, includes a second index entry. The second index entry stores relevant information about the first fingerprint in the first fingerprint set. The value of the first field in the first fingerprint is consistent with the value represented by the first index entry, and the value of the second field in the first fingerprint is consistent with the value represented by the second index entry.

6. The storage device as claimed in claim 3, characterized in that, The first index layer includes a third index item, and the third index item is included in a set of index items in the second index layer. The fourth index item is included in a set of index items in the third index layer. The fifth index item stores relevant information about the second fingerprint in the first fingerprint set. The value of the first field in the second fingerprint is consistent with the value represented by the third index item, the value of the second field in the second fingerprint is consistent with the value represented by the fourth index item, and the value of the third field in the second fingerprint is consistent with the value represented by the fifth index item.

7. The storage device according to any one of claims 1 to 6, characterized in that, The first fingerprint set includes fingerprints obtained by sampling from the second fingerprint set based on a sampling ratio. The second fingerprint set includes fingerprints of data in a storage system or storage node. The number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

8. The storage device as claimed in claim 7, characterized in that, In the first fingerprint set, the value of the fingerprint is greater than a first value and less than a second value, which is determined according to the sampling ratio.

9. The storage device according to any one of claims 1 to 8, characterized in that, The fingerprint index is distributed across N partitions of the storage device, where each partition is a storage space and N is a positive integer.

10. The storage device as claimed in claim 8, characterized in that, The storage device includes N nodes, and each node includes one partition.

11. A fingerprint index construction method, characterized in that, The method is used to construct a fingerprint index for a first fingerprint set, wherein the fingerprints of the first fingerprint set include a first field and a second field, and the method includes: Construct a first index layer for the fingerprint index, and construct a second index layer for the fingerprint index, wherein: The construction of the first index layer of the fingerprint index includes: setting multiple index items in the first index layer, wherein one index item in the first index layer represents a value of the first field; The construction of the second index layer of the fingerprint index includes: setting multiple sets of index items in the second index layer, wherein an index item of the first index layer corresponds to a set of index items of the second index layer, and an index item of the set of index items of the second index layer represents a value of the second field.

12. The method as described in claim 11, characterized in that, The method further includes: The index entries of the second index layer store information related to fingerprints in the first fingerprint set. The information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

13. The method as described in claim 11, characterized in that, The fingerprint in the first fingerprint index set also includes a third field, and the method further includes: Constructing a third index layer for the fingerprint index; wherein, constructing the third index for the fingerprint index includes: setting multiple sets of index entries in the third index layer, wherein an index entry in the second index layer corresponds to a set of index entries in the third index layer, and one index entry in the set of index entries in the third index layer represents a value of the third field; storing relevant information of fingerprints in the first fingerprint set in the index entries of the third index layer, wherein the relevant information of fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

14. The method as described in claim 13, characterized in that, The third field includes a first subfield and a second subfield. Multiple sets of index items are set in the third index layer, including: At least one set of buckets is set in the third index layer. An index item of the second index layer corresponds to a set of buckets in the third index layer. Each bucket in the set of buckets includes one or more index items. Each bucket represents a value of the first sub-field, and an index item of each bucket represents a value of the second sub-field.

15. A fingerprint query method, characterized in that, The method is used to query a target fingerprint based on a fingerprint index in a first fingerprint set. The fingerprint index stores the first fingerprint set and includes a first index layer and a second index layer. The first index layer stores a first field of the fingerprints in the first fingerprint set, and the second index layer stores a second field of the fingerprints in the first fingerprint set. One index entry in the first index layer corresponds to a set of index entries in the second index layer. The method includes: Determine whether a first index entry exists among multiple index entries in the first index layer, wherein the value of the first index entry is consistent with the value of the first field of the target fingerprint. If the first index item exists among multiple index items in the first index layer, determine whether a second index item exists among a set of index items in the second index layer corresponding to the first index item. The value represented by the second index item is consistent with the value of the second field in the target fingerprint.

16. The method as described in claim 15, characterized in that, The method further includes: The relevant information of the target fingerprint is obtained from the second index item. The relevant information of the target fingerprint includes some or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the region where the target data is located.

17. The method as described in claim 15, characterized in that, The fingerprint index further includes a third index layer, which stores the third field of the fingerprints in the first fingerprint set. One index entry in a set of index entries in the third index layer represents a value of the third field. The method further includes: In the third-level index corresponding to the second index item, it is determined whether a third index item exists in a set of index items, and the value represented by the third index item is consistent with the value of the third field in the target fingerprint; If a third index entry exists in a set of index entries in the third-level index corresponding to the second index entry, the relevant information of the target fingerprint is obtained from the third index entry. The relevant information of the target fingerprint includes some or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the region where the target data is located.

18. The method as described in claim 17, characterized in that, The third field includes a first subfield and a second subfield. The third index layer includes at least one set of buckets. An index entry in the second index layer corresponds to a set of buckets in the third index layer. Each set of buckets includes one or more buckets, and each bucket includes one or more index entries. Each bucket represents a value of the first subfield, and an index entry in each bucket represents a value of the second subfield. Determining the third index entry from a set of index entries in the third-level index corresponding to the second index entry includes: In the third-level index corresponding to the second index item, a target bucket is determined from a set of buckets, and the value represented by the target bucket is consistent with the value of the first sub-field in the target fingerprint; A third index item is determined in the target bucket, and the value represented by the third index item is consistent with the value of the second sub-field in the target fingerprint.

19. A construction apparatus, characterized in that, The construction device is used to construct a fingerprint index for a first fingerprint set, wherein the fingerprints of the first fingerprint set include a first field and a second field, and the construction device includes: The first construction module is used to construct the first index layer of the fingerprint index, wherein constructing the first index layer of the fingerprint index includes: setting multiple index items in the first index layer, wherein one index item in the first index layer represents a value of the first field; The second construction module is used to construct the second index layer of the fingerprint index, wherein constructing the second index layer of the fingerprint index includes: setting multiple sets of index items in the second index layer, wherein an index item of the first index layer corresponds to a set of index items of the second index layer, and an index item of the set of index items of the second index layer represents a value of the second field.

20. The apparatus as claimed in claim 19, characterized in that, The second construction module is further configured to: store information related to fingerprints in the first fingerprint set in the index entries of the second index layer, wherein the information related to fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

21. The apparatus as claimed in claim 19, characterized in that, The fingerprints in the first fingerprint index set also include a third field, and the construction device further includes a third construction module, which is used to: Constructing a third index layer for the fingerprint index; wherein, constructing the third index for the fingerprint index includes: setting multiple sets of index entries in the third index layer, wherein an index entry in the second index layer corresponds to a set of index entries in the third index layer, and one index entry in the set of index entries in the third index layer represents a value of the third field; storing relevant information of fingerprints in the first fingerprint set in the index entries of the third index layer, wherein the relevant information of fingerprints includes some or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

22. The apparatus as claimed in claim 21, characterized in that, The third field includes a first subfield and a second subfield. The third construction module is used for: At least one set of buckets is set in the third index layer. An index item of the second index layer corresponds to a set of buckets in the third index layer. Each bucket in the set of buckets includes one or more index items. Each bucket represents a value of the first sub-field, and an index item of each bucket represents a value of the second sub-field.

23. A query device, characterized in that, The query device is used to query a target fingerprint based on a fingerprint index in a first fingerprint set. The fingerprint index is used to store the first fingerprint set and includes a first index layer and a second index layer. The first index layer is used to store a first field of the fingerprints in the first fingerprint set, and the second index layer is used to store a second field of the fingerprints in the first fingerprint set. One index entry in the first index layer corresponds to a set of index entries in the second index layer. The query device includes: The first query module is used to determine whether a first index item exists among multiple index items in the first index layer, wherein the value of the first index item is consistent with the value of the first field of the target fingerprint. The second query module is used to determine whether a second index item exists in a set of index items in the second index layer corresponding to the first index item if the first index item exists among multiple index items in the first index layer. The value represented by the second index item is consistent with the value of the second field in the target fingerprint.

24. The apparatus as claimed in claim 23, characterized in that, The second query module is also used for: The relevant information of the target fingerprint is obtained from the second index item. The relevant information of the target fingerprint includes some or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the region where the target data is located.

25. The apparatus as claimed in claim 23, characterized in that, The fingerprint index further includes a third index layer, which stores the third field of the fingerprints in the first fingerprint set. One index entry in a set of index entries in the third index layer represents a value of the third field. The device further includes a third query module, which is used to: In the third-level index corresponding to the second index item, it is determined whether a third index item exists in a set of index items, and the value represented by the third index item is consistent with the value of the third field in the target fingerprint; If a third index entry exists in a set of index entries in the third-level index corresponding to the second index entry, the relevant information of the target fingerprint is obtained from the third index entry. The relevant information of the target fingerprint includes some or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the region where the target data is located.

26. The apparatus as claimed in claim 25, characterized in that, The third field includes a first subfield and a second subfield. The third index layer includes at least one set of buckets. An index entry in the second index layer corresponds to a set of buckets in the third index layer. Each set of buckets includes one or more buckets, and each bucket includes one or more index entries. Each bucket represents a value of the first subfield, and an index entry in each bucket represents a value of the second subfield. The third query module is used for: In the third-level index corresponding to the second index item, a target bucket is determined from a set of buckets, and the value represented by the target bucket is consistent with the value of the first sub-field in the target fingerprint; A third index item is determined in the target bucket, and the value represented by the third index item is consistent with the value of the second sub-field in the target fingerprint.

27. A computing device, characterized in that, The computing device includes a processor and memory; The memory is used to store computer program instructions; The processor executes computer program instructions that invoke the memory to perform the method as described in any one of claims 11 to 18.

28. A computer-readable storage medium, characterized in that, When the computer-readable storage medium is executed by a computing device, the computing device performs the method of any one of claims 11 to 18.

Images