Storage apparatus, fingerprint index construction method, query method, and device

Abstract

A storage apparatus, a fingerprint index construction method, a query method, and a device. In the present application, the storage apparatus stores a fingerprint index, the fingerprint index comprises a first index layer and a second index layer, the first index layer is used for storing a first field of a fingerprint in a first fingerprint set, and the second index layer is used for storing a second field of the fingerprint in the first fingerprint set. The first index layer comprises a plurality of index items, and one index item in the first index layer represents a value of the first field; one index item in the first index layer corresponds to a group of index items in the second index layer; and one index item in the second index layer represents a value of the second field. The structure of the fingerprint index is simpler, so that the storage space occupied by the fingerprint index can be effectively reduced, and the difficulty of constructing the fingerprint index is reduced. Switching between the index layers is simpler due to the correspondence between the index layers in the fingerprint index, so that the query efficiency of the fingerprint index is improved.

Description

A storage device, a fingerprint index construction method, a query method and equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application No. 202411140755.1, filed on August 16, 2024, entitled "A storage device, a fingerprint index construction method, a query method and equipment", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of communication technology, and in particular to a storage device, a fingerprint index construction method, a query method and equipment. BACKGROUND

[0004] For a storage system used for storing backup data or a storage system with frequent data updates, when new data is written into the storage system, the data often needs to be re-deleted. The re-deletion is performed based on a fingerprint table: first, for any data shard in the data, the value of the query key of the key-value pair included in the fingerprint table is the key-value pair of the data shard fingerprint, if the key-value pair is found, it means that the data shard has been stored in the storage system, and the data re-deletion can delete the data shard. If the key-value pair is not found, it means that the data shard has not been stored in the storage system, and the data re-deletion does not need to delete the data shard.

[0005] Generally, the fingerprint table records a plurality of key-value pairs, and the number of key-value pairs is the same as the number of data shards stored in the storage system. The larger the number of data shards in the storage system, the larger the fingerprint table, which will occupy more storage space. SUMMARY

[0006] The storage device, the fingerprint index construction method, the query method and the equipment provided by the embodiments of the present application are used to reduce the storage space occupied by the fingerprint index.

[0007] In a first aspect, the embodiments of the present application further provide a storage device, which is used for storing a fingerprint index. The fingerprint index includes a plurality of index layers. The structure of the fingerprint index is described below by taking two adjacent index layers in the plurality of index layers as an example. In order to distinguish them, the two adjacent index layers are referred to as a first index layer and a second index layer. The first index layer is used for storing a first field (such as field A in the embodiments of the present application) of a fingerprint in a first fingerprint set, and the second index layer is used for storing a second field (such as field B in the embodiments of the present application) of the fingerprint in the first fingerprint set.

[0008] The first index layer includes a plurality of index items, one index item in the first index layer represents a value of the first field; one index item in the first index layer corresponds to a group of index items in the second index layer; one index item in the second index layer represents a value of the second field. The lengths of the plurality of index items in the same index layer are consistent.

[0009] In the storage device, the fingerprint index has a multi-layer structure, the index items in the fingerprint index can represent the values of the fields in the fingerprints, and there is a corresponding relationship between adjacent two layers, so that a large number of fingerprints do not need to be stored in the fingerprint index, and the storage space occupied by the fingerprint index can be effectively reduced. The structure of the fingerprint index is simpler, and the construction difficulty of the fingerprint index can be reduced. Due to the corresponding relationship between the index layers in the fingerprint index, after a certain index item in the previous index layer is determined, a group of index items in the next index layer can be located, the jump between the index layers is simpler, and the query efficiency of the fingerprint index can be improved.

[0010] In a possible implementation, the second index layer is further configured to store the related information of the fingerprints in the first fingerprint set, and the related information of the fingerprints includes part or all of the following: the data corresponding to the fingerprints, the storage addresses of the data, the identifiers of the nodes where the data is located, and the identifiers of the regions where the data is located.

[0011] In the storage device, the second index layer stores the related information of the fingerprints in the first fingerprint set, so that the fingerprint index is applicable to different query scenarios.

[0012] In a possible implementation, the fingerprint index further includes a third index layer, the third index layer is configured to store the third field (for example, the field C in the embodiments of the present application) of the fingerprints in the first fingerprint set and the related information of the fingerprints, one index item in the second index layer corresponds to a group of index items in the third index layer; one index item in the third index layer represents a value of the third field, and the related information of the fingerprints includes part or all of the following: the data corresponding to the fingerprints, the storage addresses of the data, the identifiers of the nodes where the data is located, and the identifiers of the regions where the data is located.

[0013] In the storage device, the third index layer stores the related information of the fingerprints in the first fingerprint set, so that the fingerprint index is applicable to different query scenarios.

[0014] In a possible implementation, the third field includes a first subfield and a second subfield, the third index layer includes at least one group of buckets, one index item in the second index layer corresponds to a group of buckets in the third index layer, each bucket in the group of buckets includes one or more index items, each bucket represents a value of the first subfield, and one index item in each bucket represents a value of the second subfield.

[0015] In the storage device, in the third index layer, index items with the same value of the first sub-field are clustered into a bucket, i.e., the value of the first sub-field is represented by a bucket. The setting of the bucket enables locating a bucket in a group of buckets in the third index layer by using the specific value of the first sub-field when jumping to the group of buckets, and further reduces the number of index items in the group of index items when jumping from the second index layer to the third index layer.

[0016] In a possible implementation, the plurality of index items of the first index layer includes a first index item, the group of index items of the second index layer corresponding to the first index item includes a second index item, the second index item stores the related information of a first fingerprint in the first fingerprint set, the value of the first field in the first fingerprint is consistent with the value represented by the first index item, and the value of the second field in the first fingerprint is consistent with the value represented by the second index item.

[0017] In the storage device, the first index item and the second index item corresponding to the first index item can represent a first fingerprint in the first fingerprint set. It can be seen that, in the fingerprint index item, the index items having the corresponding relationship in the index layers can represent a fingerprint in the first fingerprint set.

[0018] In a possible implementation, the plurality of index items of the first index layer includes a third index item, the group of index items of the second index layer corresponding to the third index item includes a fourth index item, the group of index items of the third index layer corresponding to the fourth index item includes a fifth index item, the fifth index item stores the related information of a second fingerprint in the first fingerprint set, the value of the first field in the second fingerprint is consistent with the value represented by the third index item, the value of the second field in the second fingerprint is consistent with the value represented by the fourth index item, and the value of the third field in the second fingerprint is consistent with the value represented by the fifth index item.

[0019] In the storage device, the third index item and the fourth index item and the fifth index item corresponding to the third index item can represent the second fingerprint in the first fingerprint set. In the fingerprint index item, the index items having the corresponding relationship in the index layers can represent the fingerprint in the first fingerprint set.

[0020] In a possible implementation, the first fingerprint set includes the fingerprints obtained by sampling the second fingerprint set based on a sampling ratio, the second fingerprint set includes the fingerprints of the data in the storage system or the storage node, and the number of the fingerprints included in the first fingerprint set is less than the number of the fingerprints included in the second fingerprint set.

[0021] In the storage device, only part of the fingerprints in the second fingerprint set are included in the first fingerprint set, so that the number of the fingerprints recorded in the fingerprint index is reduced, and the storage space occupied by the fingerprint index is further reduced.

[0022] In a possible implementation, the upper limit value and the lower limit value of the fingerprints in the first fingerprint set can be determined based on the sampling ratio. For example, the value of the fingerprints in the first fingerprint set is greater than a first value (the first value is the lower limit value) and less than a second value (the second value is the upper limit value). The upper limit value and / or the lower limit value of the fingerprints in the first fingerprint set is related to the sampling ratio. For example, the second value is determined according to the sampling ratio. For another example, the first value is determined according to the sampling ratio.

[0023] In the storage device, the sampling ratio can determine the upper limit value and the lower limit value of the fingerprints in the first fingerprint set. When the sampling ratio is changed, only the upper limit value or the lower limit value of the fingerprints in the first fingerprint set needs to be changed. When only the upper limit value or the lower limit value of the fingerprints in the first fingerprint set is changed, the change of the fingerprints included in the first fingerprint set is relatively small.

[0024] In a possible implementation, the fingerprint index is distributed in N partitions of the storage device, each partition is a piece of storage space, and N is a positive integer.

[0025] In the storage device, the fingerprint index can be stored in partitions, and the storage manner of the fingerprint index is relatively flexible.

[0026] In a possible implementation, the storage device includes N nodes, and each node includes a partition. The node can be a memory or a computing device.

[0027] In the storage device, the fingerprint index can be stored across nodes, which effectively expands the applicable storage manner of the fingerprint index, and the deployment manner of the fingerprint index is more flexible.

[0028] In a second aspect, an embodiment of the present application further provides a fingerprint index construction method, and related beneficial effects can be referred to the related description of the first aspect, which will not be repeated here. The method is used for constructing a fingerprint index for a first fingerprint set, the fingerprints of the first fingerprint set include a first field and a second field, and the method can be executed by a construction device. Here, two index layers in the fingerprint index are taken as an example for description, and in the method, the construction device constructs a first index layer of the fingerprint index, and constructs a second index layer of the fingerprint index.

[0029] When constructing the first index layer of the fingerprint index, the construction device sets a plurality of index items in the first index layer, and each index item in the first index layer represents a value of the first field. That is, the construction device can store the first field by using the first index item layer. Each index item represents a value of the first field.

[0030] The constructing device sets multiple groups of index items in the second index layer when constructing the second index layer of the fingerprint index, one index item of the first index layer corresponds to one group of index items of the second index layer, and one index item in one group of index items of the second index layer represents one value of the second field. That is, the constructing device can store the second field by using the second index item layer. Each index item represents one value of the second field.

[0031] By the above method, the constructing device can quickly construct the fingerprint index, and the structure of the fingerprint index is relatively simple.

[0032] In a possible implementation, the related information of the fingerprints in the first fingerprint set is stored in the index items of the second index layer, and the related information of the fingerprints includes part or all of the following: the data corresponding to the fingerprints, the storage address of the data, the identifier of the node where the data is located, and the identifier of the area where the data is located.

[0033] In a possible implementation, the fingerprints in the first fingerprint set further include a third field, and the constructing device can construct a third index layer of the fingerprint index.

[0034] The constructing device sets multiple groups of index items in the third index layer when constructing the third index layer of the fingerprint index, one index item of the second index layer corresponds to one group of index items of the third index layer, and one index item in one group of index items of the third index layer represents one value of the third field; and the related information of the fingerprints in the first fingerprint set is stored in the index items of the third index layer, and the related information of the fingerprints includes part or all of the following: the data corresponding to the fingerprints, the storage address of the data, the identifier of the node where the data is located, and the identifier of the area where the data is located.

[0035] In a possible implementation, the third field includes a first subfield and a second subfield, and when the constructing device sets multiple groups of index items in the third index layer, the constructing device sets at least one group of buckets in the third index layer, one index item of the second index layer corresponds to one group of buckets of the third index layer, each bucket in one group of buckets includes one or more index items, each bucket represents one value of the first subfield, and one index item in each bucket represents one value of the second subfield.

[0036] In a possible implementation, the multiple index items of the first index layer include a first index item, the group of index items of the second index layer corresponding to the first index item includes a second index item; the second index item stores the related information of a first fingerprint in the first fingerprint set, the value of the first field in the first fingerprint is consistent with the value represented by the first index item, and the value of the second field in the first fingerprint is consistent with the value represented by the second index item.

[0037] In a possible implementation, the plurality of index items of the first index layer includes a third index item, the group of index items of the second index layer corresponding to the third index item includes a fourth index item, the group of index items of the third index layer corresponding to the fourth index item includes a fifth index item, the fifth index item stores information of a second fingerprint in the first fingerprint set, a value of a first field in the second fingerprint is consistent with a value represented by the third index item, a value of a second field in the second fingerprint is consistent with a value represented by the fourth index item, and a value of a third field in the second fingerprint is consistent with a value represented by the fifth index item.

[0038] In a possible implementation, the constructing module samples the fingerprints in the second fingerprint set based on the sampling ratio, the first fingerprint set includes the sampled fingerprints, and the second fingerprint set includes the fingerprints of the data in the storage system or the storage node, and the number of the fingerprints included in the first fingerprint set is less than the number of the fingerprints included in the second fingerprint set.

[0039] In a possible implementation, based on the sampling ratio, an upper limit value and a lower limit value of the fingerprints in the first fingerprint set can be determined, for example, the value of the fingerprint in the first fingerprint set is greater than a first value (the first value is the lower limit value) and less than a second value (the second value is the upper limit value). The upper limit value and / or the lower limit value of the fingerprints in the first fingerprint set are related to the sampling ratio. For example, the second value is determined according to the sampling ratio. For another example, the first value is determined according to the sampling ratio.

[0040] In a possible implementation, the fingerprint index is distributed in N partitions of the storage device, each partition is a piece of storage space, and N is a positive integer.

[0041] In a possible implementation, the storage device includes N nodes, and each node includes one partition.

[0042] In a third aspect, an embodiment of the present application further provides a fingerprint query method, and the related beneficial effects are described above, which will not be repeated here. The method is used for querying a target fingerprint in a first fingerprint set based on a fingerprint index, and the fingerprint index is used for storing the first fingerprint set, and the fingerprints in the first fingerprint set are fingerprints of data in a data set.

[0043] The fingerprint index includes a first index layer and a second index layer, the first index layer is used for storing a first field of the fingerprints in the first fingerprint set, the second index layer is used for storing a second field of the fingerprints in the first fingerprint set, and one index item of the first index layer corresponds to a group of index items of the second index layer.

[0044] There are many scenarios for querying a target fingerprint in a first fingerprint set. For example, determining whether the target fingerprint corresponds to target data in the data set. For another example, determining whether a certain fingerprint has been stored in the first fingerprint set. Embodiments of the present application do not limit the scenarios to which the fingerprint query method is applicable. Any scenario that needs to determine whether a target fingerprint exists in a first fingerprint set is applicable to the fingerprint query method provided by the present application. The method is executed by a querying device, and in the method:

[0045] The querying device determines whether a first index item exists in the plurality of index items in the first index layer, the value of the first index item being consistent with the value of the first field of the target fingerprint.

[0046] If the first index item exists in the plurality of index items in the first index layer, the querying device determines whether a second index item exists in a group of index items in the second index layer corresponding to the first index item, the value represented by the second index item being consistent with the value of the second field of the target fingerprint.

[0047] If the second index item exists in the group of index items in the second index layer corresponding to the first index item, it can be determined that the target fingerprint exists in the first fingerprint set.

[0048] Through the above method, when the querying device performs a query based on the fingerprint index, the querying device can jump from the first index layer to the second index layer based on the correspondence between the first index layer and the second index layer, locate the group of index items in the second index layer, and perform fingerprint query in a simpler and more efficient manner.

[0049] In a possible implementation, the querying device obtains, from the second index item, related information of the target fingerprint, the related information of the target fingerprint including some or all of the following: target data corresponding to the target fingerprint, a storage address of the target data, an identifier of a node where the target data is located, and an identifier of a region where the target data is located.

[0050] Through the above method, after locating the second index item, the querying device can quickly obtain the related information of the target fingerprint from the second index item.

[0051] In a possible implementation, the fingerprint index further includes a third index layer, the third index layer being configured to store a third field of the fingerprints in the first fingerprint set, and an index item in a group of index items in the third index layer representing a value of the third field.

[0052] The query device determines whether a third index item exists in a group of index items in the third layer index corresponding to the second index item, the third index item representing a value consistent with the value of the third field in the target fingerprint. If the third index item exists in the group of index items in the third layer index corresponding to the second index item, the query device obtains the relevant information of the target fingerprint from the third index item, the relevant information of the target fingerprint including part or all of the following: the target data corresponding to the target fingerprint, the storage address of the target data, the identifier of the node where the target data is located, and the identifier of the area where the target data is located.

[0053] Through the above method, after the query device jumps from the second index layer to the third index layer, the third index item in the third index layer can be located, and then the relevant information of the target fingerprint is obtained from the second index item.

[0054] In a possible implementation, the third index layer is further configured to store the relevant information of the fingerprints in the first fingerprint set, the relevant information of the fingerprints including part or all of the following: the data corresponding to the fingerprints, the storage address of the data, the identifier of the node where the data is located, and the identifier of the area where the data is located.

[0055] In a possible implementation, the third field includes a first subfield and a second subfield, the third index layer includes at least one group of buckets, one index item in the second index layer corresponds to one group of buckets in the third index layer, each group of buckets includes one or more buckets, and each bucket includes one or more index items, each bucket representing a value of the first subfield, and each index item in each bucket representing a value of the second subfield.

[0056] When the query device determines whether the third index item exists in the group of index items in the third layer index corresponding to the second index item, the query device determines a target bucket in the group of buckets in the third layer index corresponding to the second index item, the target bucket representing a value consistent with the value of the first subfield in the target fingerprint.

[0057] If the third index item exists in the group of index items in the third layer index corresponding to the second index item, the third index item representing a value consistent with the value of the second subfield in the target fingerprint.

[0058] Through the above method, the third index layer aggregates multiple index layers through buckets, and when the query device jumps to a group of buckets in the third index layer, the search range is quickly reduced to a target bucket in the group of buckets with the value of the first subfield of the target fingerprint.

[0059] In a possible implementation, if the first index item, the second index item, or the third index item does not exist, the query device can determine that the target fingerprint does not exist in the first fingerprint set.

[0060] By the method, if no corresponding index item is found in any index layer, it is determined that the target fingerprint does not exist in the first fingerprint set, and the query process is stopped,

[0061] In a possible implementation, the first fingerprint set includes fingerprints obtained by sampling the second fingerprint set based on a sampling ratio, the second fingerprint set includes fingerprints of data in the storage system or the storage node, and the number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

[0062] In a possible implementation, based on the sampling ratio, an upper limit value and a lower limit value of the fingerprints in the first fingerprint set can be determined, for example, the value of the fingerprint in the first fingerprint set is greater than a first value (the first value is the lower limit value) and less than a second value (the second value is the upper limit value). The upper limit value and / or the lower limit value of the fingerprints in the first fingerprint set are related to the sampling ratio. For example, the second value is determined according to the sampling ratio. For another example, the first value is determined according to the sampling ratio.

[0063] In a possible implementation, the fingerprint index is distributed in N partitions of the storage device, each partition is a piece of storage space, and N is a positive integer.

[0064] In a possible implementation, the storage device includes N nodes, and each node includes one partition.

[0065] In a fourth aspect, the embodiments of the present application further provide a fingerprint index. The related advantages can be referred to the foregoing description and will not be repeated here. The fingerprint index includes a first index layer and a second index layer. The first index layer is configured to store a first field of a fingerprint in a first fingerprint set. The second index layer is configured to store a second field of the fingerprint in the first fingerprint set.

[0066] The first index layer includes a plurality of index items. One index item in the first index layer represents a value of the first field. One index item in the first index layer corresponds to a group of index items in the second index layer. One index item in the second index layer represents a value of the second field.

[0067] In a possible implementation, the second index layer is further configured to store related information of the fingerprint in the first fingerprint set. The related information of the fingerprint includes part or all of the following: data corresponding to the fingerprint, a storage address of the data, an identifier of a node where the data is located, and an identifier of a region where the data is located.

[0068] In a possible implementation, the fingerprint index further includes a third index layer, the third index layer is configured to store a third field of the fingerprints in the first fingerprint set and related information of the fingerprints, one index item of the second index layer corresponds to a group of index items of the third index layer; one index item of the third index layer represents a value of the third field, and the related information of the fingerprints includes part or all of the following: data corresponding to the fingerprint, a storage address of the data, an identifier of a node where the data is located, and an identifier of a region where the data is located.

[0069] In a possible implementation, the third field includes a first subfield and a second subfield, the third index layer includes at least one group of buckets, one index item of the second index layer corresponds to a group of buckets of the third index layer, and each bucket in the group of buckets includes one or more index items, each bucket represents a value of the first subfield, and one index item in each bucket represents a value of the second subfield.

[0070] In a possible implementation, the multiple index items of the first index layer include a first index item, a group of index items of the second index layer corresponding to the first index item includes a second index item, the second index item stores related information of a first fingerprint in the first fingerprint set, a value of the first field in the first fingerprint is consistent with a value represented by the first index item, and a value of the second field in the first fingerprint is consistent with a value represented by the second index item.

[0071] In a possible implementation, the multiple index items of the first index layer include a third index item, a group of index items of the second index layer corresponding to the third index item includes a fourth index item, a group of index items of the third index layer corresponding to the fourth index item includes a fifth index item, the fifth index item stores related information of a second fingerprint in the first fingerprint set, a value of the first field in the second fingerprint is consistent with a value represented by the third index item, a value of the second field in the second fingerprint is consistent with a value represented by the fourth index item, and a value of the third field in the second fingerprint is consistent with a value represented by the fifth index item.

[0072] In a possible implementation, the first fingerprint set includes fingerprints obtained by sampling, based on a sampling ratio, from a second fingerprint set, the second fingerprint set includes fingerprints of data in the storage system or the storage node, and the number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

[0073] In a possible implementation, a value of a fingerprint in the first fingerprint set is greater than a first value and less than a second value, and the second value is determined according to the sampling ratio.

[0074] In a possible implementation, a value of a fingerprint in the first fingerprint set is greater than a first value and less than a second value, and the first value is determined according to the sampling ratio.

[0075] In a possible implementation, the fingerprint index is distributed in N partitions of the storage device, each partition is a piece of storage space, and N is a positive integer.

[0076] In a possible implementation, the storage device includes N nodes, and each node includes one partition.

[0077] In a fifth aspect, the embodiments of the present application further provide a construction device, which has the functions of implementing the behaviors in the method examples of the second aspect, and the beneficial effects can be referred to the description of the second aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the structure of the construction device includes a first construction module and a second construction module, and optionally, a third construction module, a sampling module and a storage module. These modules can perform the corresponding functions in the method examples of the second aspect, and the specific description can be referred to the detailed description in the method examples.

[0078] In a sixth aspect, the embodiments of the present application further provide a query device, which has the functions of implementing the behaviors in the method examples of the third aspect, and the beneficial effects can be referred to the description of the third aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the structure of the device includes a first query module and a second query module, and optionally, a third query module. These modules can perform the corresponding functions in the method examples of the third aspect, and the specific description can be referred to the detailed description in the method examples.

[0079] In a seventh aspect, the present application further provides a computing device, which includes a processor and a memory, and can further include a communication interface. The processor executes computer program instructions in the memory to perform the method provided by the second aspect or any possible implementation of the second aspect. Or the processor executes computer program instructions in the memory to perform the method provided by the third aspect or any possible implementation of the third aspect. The memory is coupled with the processor, which stores necessary computer program instructions and data in the fingerprint construction process or the fingerprint query process. The communication interface is used for communication with other devices, such as obtaining a target fingerprint.

[0080] In an eighth aspect, the present application provides a computing device system, which comprises at least one computing device. Each computing device comprises a memory and a processor. The processor of the at least one computing device is configured to access computer program instructions in the memory to perform the method provided in the second aspect or any possible implementation of the second aspect, or the processor of the at least one computing device is configured to access computer program instructions in the memory to perform the method provided in the third aspect or any possible implementation of the third aspect.

[0081] In a ninth aspect, the present application provides a computer readable storage medium, which, when executed by a computing device, causes the computing device to perform the method provided in the second aspect or any possible implementation of the second aspect, or the method provided in the third aspect or any possible implementation of the third aspect. The storage medium stores computer program instructions. The storage medium can include, but is not limited to, a volatile memory, such as a random access memory (RAM), a non-volatile memory, such as a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0082] In a tenth aspect, the present application provides a computing device program product, which comprises computer program instructions, and when executed by a computing device, causes the computing device to perform the method provided in the second aspect or any possible implementation of the second aspect, or the method provided in the third aspect or any possible implementation of the third aspect. The computing device program product can be a software package, which can be downloaded and executed on the computing device when the method provided in the second aspect or any possible implementation of the second aspect, or the method provided in the third aspect or any possible implementation of the third aspect is needed.

[0083] In an eleventh aspect, the present application further provides a computer chip, which is connected to a memory and is configured to read and execute computer program instructions stored in the memory to perform the method in the second aspect and each possible implementation of the second aspect, or to perform the method in the third aspect and each possible implementation of the third aspect.

[0084] The technical effects achieved by the second aspect to the eleventh aspect can refer to the description of the technical effects achieved by the corresponding design scheme in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0085] FIG. 1 is a structural diagram of a fingerprint index provided by the present application;

[0086] Figures 2A and 2B are schematic diagrams of the structure of a fingerprint index provided in an embodiment of this application;

[0087] Figure 3 is a schematic diagram of a fingerprint index construction method provided in this application;

[0088] Figures 4 and 5 are schematic diagrams of a fingerprint index construction process provided in this application;

[0089] Figure 6 is a schematic diagram of a fingerprint query method provided in this application;

[0090] Figures 7 and 8 are schematic diagrams of a fingerprint query process provided in this application;

[0091] Figure 9 is a schematic diagram of a fingerprint index construction method provided in this application;

[0092] Figures 10A to 10C are schematic diagrams of a fingerprint index structure provided in this application;

[0093] Figure 11 is a schematic diagram of a fingerprint query method provided in this application;

[0094] Figures 12A and 12B are schematic diagrams of a fingerprint index deployment provided in this application;

[0095] Figures 13A and 13B are schematic diagrams illustrating the splitting of a fingerprint index provided in this application;

[0096] Figure 14 is a structural schematic diagram of a construction device provided in this application;

[0097] Figure 15 is a schematic diagram of the structure of a query device provided in this application;

[0098] Figures 16 and 17 are schematic diagrams of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0099] Before describing the storage device, fingerprint index construction method, query method, and equipment provided in the embodiments of this application, the concepts involved in the embodiments of this application will be explained first:

[0100] (1) Data fingerprint.

[0101] A data fingerprint is information formed based on the data itself and can be used to represent the data. A data fingerprint can be understood as a "summary" extracted from the data. Data fingerprints can be used to compare data; when two sets of data have the same data fingerprint, it means that the two sets of data are very likely the same.

[0102] The embodiments of the present application do not limit the generation manner of the data fingerprint. Generally, the fingerprint of the data can be generated by calling a data fingerprint algorithm. The data fingerprint algorithm includes but is not limited to MD5, xxhash, sha1, sha256, sha3, blake3. The present application does not limit the specific type of the data fingerprint algorithm.

[0103] When generating the fingerprint of the data, the data can be first split to form multiple data portions; for the convenience of description, one of the multiple data portions is referred to as a data slice. After obtaining multiple data slices, the fingerprint of each data slice is generated. The fingerprint of the data includes the fingerprints of the multiple data slices.

[0104] The granularity of the splitting of the data can be fixed, that is, the size of each data slice formed is the same. The granularity of the splitting of the data can be variable, that is, the size of each data slice formed is not completely the same; in this splitting manner, the value range of the granularity can be set, that is, the value range of the size of the data slice formed is limited, and the size of each data slice formed in the splitting of the data needs to belong to the value range.

[0105] (2) De-duplication domain.

[0106] For any data slice, the de-duplication domain of the data slice refers to the storage area (the storage area can also be referred to as a region) allocated for the data slice and needed to write the data slice. Generally, when performing data de-duplication, a de-duplication domain is allocated for each data slice included in the data. For any data slice, the data slice needs to be written into the de-duplication domain allocated for it. If the same data slice as the data slice has been stored in the de-duplication domain of the data slice, when the data slice is written into the de-duplication domain, the write operation of the data slice does not need to be performed, that is, the data slice does not need to occupy the storage space in the de-duplication domain, and only the storage address of the data slice already stored in the de-duplication domain needs to be recorded, which is the storage address of the data slice needed to be written into the de-duplication domain. If the same data slice as the data slice has not been stored in the de-duplication domain of the data slice, when the data slice is written into the de-duplication domain, the write operation of the data slice needs to be performed, and the data slice needs to occupy the storage space in the de-duplication domain, and the storage address of the data slice in the de-duplication domain needs to be recorded. It can be seen that the de-duplication domain of the data slice limits the de-duplication range of the data slice, in other words, when de-duplicating the data slice, the data already stored in the de-duplication domain of the data slice is taken as the data for comparison with the data slice.

[0107] The specific form of the deduplication domain of the data shard is related to the data writing scenario and the granularity of the division of the storage space in the data writing scenario. For example, when data is written in a node, a deduplication domain can be allocated for each data shard included in the data in the node when deduplicating the data, and the deduplication domain can be the memory of the node or a storage pool of the node, where the storage pool can be understood as a piece of storage space obtained by logically dividing the storage space in the node. For another example, when data is written in a storage system, a deduplication domain can be allocated for each data shard included in the data in the storage system when deduplicating the data, and the deduplication domain can be a node in the storage system or the memory of the node, and the like. The specific form of the deduplication domain of the data shard is flexible, and the embodiments of the present application do not limit the specific form of the deduplication domain.

[0108] First, a fingerprint index provided by an embodiment of the present application is introduced.

[0109] 1), Structure of the fingerprint index.

[0110] The fingerprint index is an index constructed based on a first fingerprint set, and information related to the fingerprint is obtained by querying the fingerprint index with the fingerprint. The "information related to the fingerprint" obtained by the query is the "data" to which the fingerprint index is directed. The "data" to which the fingerprint index is directed is different in different scenarios in which the fingerprint index is applied. For example, in a scenario in which the data corresponding to the fingerprint is needed, the "data" to which the fingerprint index is directed is the data corresponding to the fingerprint, that is, the data corresponding to the fingerprint can be obtained by querying the fingerprint index with the fingerprint. In a scenario in which the storage location of the data is needed, the "data" to which the fingerprint index is directed is the storage address of the data, that is, the storage address of the data can be obtained by querying the fingerprint index with the fingerprint. In a scenario in which the node in which the data is located is needed, the "data" to which the fingerprint index is directed is the node in which the data is located, that is, the information of the node in which the data is located (such as the identifier of the node, the physical location of the node, and the like) can be obtained by querying the fingerprint index with the fingerprint. In a scenario in which the area in which the data is located is needed, the "data" to which the fingerprint index is directed is the area in which the data is located, that is, the information of the area in which the data is located (such as the identifier of the area, and the like) can be obtained by querying the fingerprint index with the fingerprint. This scenario is commonly used to query the deduplication domain of the data, that is, the area mentioned above is the deduplication domain.

[0111] In addition, there is a special scenario in which it is necessary to determine whether to store certain data or whether certain data exists. In this scenario, the fingerprint is queried in the fingerprint index. If the fingerprint is queried, it indicates that the data corresponding to the fingerprint exists or that the data corresponding to the fingerprint has been stored. Otherwise, the data corresponding to the fingerprint does not exist or the data is not stored.

[0112] For convenience of description, the information related to the fingerprint is referred to as fingerprint-related information. There is a corresponding relationship between the fingerprint and the fingerprint-related information. The corresponding relationship between the fingerprint and the fingerprint-related information can exist in the form of a key-value pair, that is, the fingerprint is used as a key, and the corresponding relationship between the fingerprint and the fingerprint-related information is used as a value corresponding to the key. In the embodiments of the present application, only the corresponding relationship between the fingerprint and the fingerprint-related information in the form of a key-value pair is described as an example. The corresponding relationship between the fingerprint and the fingerprint-related information can also exist in other forms, which is not limited in the embodiments of the present application.

[0113] The fingerprint in the first data set can be split into multiple segments as a string, that is, the fingerprint in the first data set includes multiple fields, and possible values of the fields are the basis for constructing the fingerprint index. It should be noted that the values of the fields are related to the type of the fingerprint. For example, the specific values of the fields can be character segments containing 0 and / or 1 when the fingerprint is a binary character type. For another example, the specific values of the fields can be character segments containing part or all of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F when the fingerprint is a hexadecimal character type.

[0114] The structure of the fingerprint index is described below.

[0115] As shown in FIG. 1, the structure of a fingerprint index provided by the embodiments of the present application is shown. The fingerprint index is a multi-layer structure, and the fingerprint index includes multiple index layers. The multiple index layers in the fingerprint index are arranged layer by layer, and one layer in the multi-layer structure is an index layer. The multiple index layers of the fingerprint index have an up-down relationship, that is, there is an adjacent index layer (which can be referred to as a previous layer index layer) above any index layer in the fingerprint index except the topmost index layer, and there is an adjacent index layer (which can be referred to as a next layer index layer) below any index layer in the fingerprint index except the bottommost index layer.

[0116] Each index layer corresponds to a field in the fingerprint, and an index layer is used to store a field of the fingerprint. The index layer covers all possible values of the field.

[0117] Each index layer includes a plurality of index items, and any index item of the index layer represents a value of a field corresponding to the index layer.

[0118] Any index layer of the fingerprint index except the bottom index layer, an index item in the index layer corresponds to a group of index items in the next index layer. A group of index items includes one or more index items, and an index item in the group of index items represents a value of a field corresponding to the index layer where the index item is located.

[0119] In other words, any index item of an index layer points to a group of index items in the next index layer. From the perspective of the correspondence between the index items of the upper and lower layers, any index item in the pointed group of index items represents a value of a field corresponding to the next index layer, under the condition that the field corresponding to the index layer is in a value represented by the index item of the index layer.

[0120] According to the correspondence between the upper and lower index layers, starting from any index item of the top index layer, an index item is selected in each index layer, and the selected index item corresponds to the selected index item in the previous index layer, until an index item is selected from the bottom index layer. That is, starting from any index item of the top index layer, a group of index items in the second index layer pointed by the index item is determined; an index item is selected in the determined group of index items in the second index layer, and a group of index items in the third index layer pointed by the index item is determined; an index item is selected in the determined group of index items in the third index layer, and a group of index items in the fourth index layer pointed by the index item is determined; an index item is selected in the determined group of index items in the fourth index layer, and the cycle is executed until the bottom index layer is reached, and an index item is selected in the group of index items in the bottom index layer.

[0121] Based on this selection method of index items, the selected index item in the bottom index layer represents a fingerprint. The values of the fields of the fingerprint are consistent with the values represented by the selected index items in the index layers.

[0122] To explain the statement "the index entry selected in the lowest index layer represents a fingerprint" from another perspective: Due to the correspondence between the upper and lower index layers, starting from the lowest index layer and working upwards, each index layer always identifies an index entry that corresponds to the index entry identified in the next lower index layer. In this way, there is a correspondence between the index entries identified in each index layer; that is, an index entry in any index layer always has a corresponding index entry in any index layer above it. Therefore, the index entry in the lowest index layer can represent a fingerprint, and this fingerprint is consistent with the value of the field represented by the index entry identified in each index layer.

[0123] Optionally, the index entries in the lowest index layer include fingerprint-related information. The fingerprint is the fingerprint represented by the index entry. In this embodiment, in any index layer other than the lowest index layer, each index entry has the same length, meaning each index entry occupies the same storage space. For the lowest index layer, it may contain fingerprint-related information, and the length of each index entry is related to the length of the fingerprint-related information. Therefore, the length of each index entry in the lowest index layer can be the same or different. The case where the lengths of index entries in the lowest index layer are different will be explained below.

[0124] As described in the structural description of the fingerprint index, for any index layer, the index entries in that layer represent a value of the corresponding field. Here, we will explain one way in which an index entry represents a value of the corresponding field, as mentioned in the embodiments of this application.

[0125] In any index layer other than the lowest level, each index entry records the storage address of a set of index entries in the next lower index layer. For example, each index entry can record the starting address of each index entry in a set of index entries in the next lower index layer. Furthermore, since each index entry in this index layer has the same length, each index entry can record the starting address of the first index entry in a set of index entries in the next lower index layer.

[0126] For any index entry in this index layer, in the next index layer corresponding to that index entry, the index entries in that group are arranged in order of the value represented by each index entry. For example, the index entries in that group are arranged from largest to smallest according to the value represented by each index entry. Or, the index entries in that group are arranged from smallest to largest according to the value represented by each index entry.

[0127] Since the index items in the set of index items are arranged in a certain order, the position of each index item in the set of index items can represent the value represented by the index item.

[0128] For example, the set of index items includes eight index items, which represent eight values of the field in the corresponding index layer, such as 0, 1, 2, 3, 4, 5, 6, and 7. The index items in the set of index items are arranged in ascending order according to the values represented by the index items. Then, the value represented by the index item arranged at the first position in the set of index items is 0, the value represented by the index item arranged at the second position in the set of index items is 1, the value represented by the index item arranged at the third position in the set of index items is 2, the value represented by the index item arranged at the fourth position in the set of index items is 3, the value represented by the index item arranged at the fifth position in the set of index items is 4, the value represented by the index item arranged at the sixth position in the set of index items is 5, the value represented by the index item arranged at the seventh position in the set of index items is 6, and the value represented by the index item arranged at the eighth position in the set of index items is 7.

[0129] For the bottommost index layer, a set of index items corresponding to any index item of the upper index layer in the bottommost index layer can be arranged in the following two ways.

[0130] In the first way, the index items in the set of index items are arranged in order according to the values represented by the index items. For example, the index items in the set of index items are arranged in descending order according to the values represented by the index items. For another example, the index items in the set of index items are arranged in descending order according to the values represented by the index items.

[0131] In this way, each index item in the set of index items can not contain the value of the corresponding field, that is, the value represented by the index item is not written in the storage space where the index item is located. The position of any index item in the set of index items is related to the value represented by the index item. The position of any index item in the set of index items represents the value represented by the index item.

[0132] In the second way, each index item in the set of index items contains the value represented by the index item, that is, the value represented by the index item is written in the storage space where the index item is located. In this way, the arrangement order of the index items in the set of index items is not limited, for example, the index items in the set of index items can be arranged in order according to the values represented by the index items, or can be arranged randomly.

[0133] For the topmost index layer, the plurality of index entries in the topmost index layer are arranged in order according to the values represented by the respective index entries. Each index entry can not contain the value of the corresponding field, that is, the value represented by the index entry is not written in the storage space where the index entry is located. The position of the index entry in the plurality of index entries can indicate the value of the corresponding field. In this way, the position of any index entry in the plurality of index entries is related to the value represented by the index entry. The position of any index entry in the plurality of index entries represents the value represented by the index entry.

[0134] These two ways have their own advantages. Way one is suitable for the scenario where the storage space of the fingerprint index is sufficient. In the bottommost index layer, there can be index entries representing all possible values of the field corresponding to any index entry in the upper index layer. In the scenario of querying fingerprints by means of the fingerprint index, this way not only ensures the accuracy of the query result, but also ensures the query efficiency (i.e., fast query) of the fingerprint index. Way two is suitable for the scenario where the storage space of the bottommost index layer of the fingerprint index is limited, that is, the number of index entries required to be contained in the bottommost index layer is fixed. In this scenario, the number of fingerprints that can be recorded by the fingerprint index is small, but the fingerprints recorded by the fingerprint index are always updated. Therefore, in way two, the index entries can be replaced, that is, some index entries written for a long time can be deleted in time to release storage space for writing new index entries. The index entries in the bottommost index layer of the fingerprint index can change.

[0135] From the above description of the "way of representing a value of a field by an index entry", when an index entry in a certain index layer is located, the storage address of a group of index entries corresponding to the index entry in the next index layer can be obtained from the located index entry, and then the group of index entries corresponding to the next index layer is located based on the storage address. In the group of index entries, an index entry can be located according to the value of the field corresponding to the next index layer in the known fingerprint. The above operation is repeatedly performed until an index entry in the bottommost index layer is located. In this way of representing a value of a field by an index entry, the specific way of searching by means of the fingerprint index will be described below, and will not be described in detail here.

[0136] In the following, the structure of the fingerprint index will be further introduced by taking the fingerprint index including a two-layer structure and a three-layer structure as an example, so as to more clearly understand the structure of the fingerprint index.

[0137] The fingerprints in the first fingerprint set include two fields, field A and field B.

[0138] As shown in FIG. 2A, the fingerprint index includes two index layers, which are referred to as a first index layer and a second index layer. The first index layer corresponds to field A and is used to store field A. The first index layer includes a plurality of index items, each of which represents a value of field A. The second index layer corresponds to field B and is used to store field B. An index item in the first index layer corresponds to a group of index items in the second index layer, and the index items in the group represent values of field B.

[0139] Suppose that field A can have integer values from 0 to 7, and field B can have integer values from 0 to 7. The first index layer includes eight index items, each of which represents a value of field A, and the eight index items represent the eight integers from 0 to 7. Any index item in the first index layer points to a group of index items in the second index layer, and each group of index items in the second index layer includes a plurality of index items. Each group of index items covers all possible values of field B. Since field B can have integer values from 0 to 7, each group of index items includes eight index items. The second index layer includes a total of eight groups of index items, and a total of 64 index items.

[0140] In the case of a known fingerprint, the values of field A and field B in the known fingerprint are determined. It is assumed that field A is 0 and field B is 7. Starting from the first index layer in the two-layer structure, a search operation is performed layer by layer: search for a first index item representing a value of field A as 0 in all index items in the first index layer, and jump to a group of index items in the second index layer corresponding to the first index item. In the second index layer, search for a second index item representing a value of field B as 7 in the group of index items jumped to.

[0141] If a second index item is searched in the second index layer, it means that the known fingerprint is queried in the fingerprint index, indicating that the data corresponding to the known fingerprint exists. If the second index item searched in the second index layer further includes fingerprint-related information, the fingerprint-related information in the second index item is the relevant information of the known fingerprint.

[0142] If no index item is searched in any layer in the two-layer structure, it means that the known fingerprint is not recorded in the fingerprint index.

[0143] The fingerprints in the first fingerprint set include three fields, namely field A, field B, and field C.

[0144] As shown in FIG. 2B, the fingerprint index includes three index layers, which are referred to as a first index layer, a second index layer, and a third index layer. The first index layer corresponds to field A and is used to store field A. The first index layer includes a plurality of index items, each of which represents a value of field A. The second index layer corresponds to field B and is used to store field B. One index item in the first index layer corresponds to a group of index items in the second index layer, and one index item in the group of index items represents a value of field B. The third index layer corresponds to field C and is used to store field C. One index item in the second index layer corresponds to a group of index items in the third index layer, and one index item in the group of index items represents a value of field C.

[0145] Suppose that field A can have a value of an integer from 0 to 7, field B can have a value of an integer from 0 to 7, and field C can have a value of an integer from 0 to 7. The correspondence between the first index layer and the second index layer and the number of index items included are similar to those in FIG. 2A, and details are described above and will not be repeated here. For the third index layer, any index item in the second index layer points to a group of index items in the third index layer, and each group of index items in the third index layer includes a plurality of index items. Each group of index items covers all possible values of field C. Since field C can have a value of an integer from 0 to 7, each group of index items includes 8 index items. The third index layer includes 64 groups of index items, and a total of 512 index items.

[0146] In the case of a known fingerprint, the values of field A, field B, and field C in the known fingerprint are determined. It is assumed that field A is 0, field B is 7, and field C is 7. Starting from the first index layer in the three-layer structure, a search operation is performed layer by layer: the determination of index items in the first index layer and the second index layer is similar to that in the first index layer and the second index layer in FIG. 2A. After the second index item is determined in the second index layer, jump to a group of index items in the third index layer corresponding to the second index item. In the third index layer, search for a third index item representing that field C has a value of 7 in the group of index items.

[0147] If a third index item is searched in the third index layer, it indicates that the known fingerprint is queried in the fingerprint index, and that the data corresponding to the known fingerprint exists. If the third index item searched in the third index layer further includes fingerprint-related information of the fingerprint, the fingerprint-related information in the third index item is the relevant information required for the known fingerprint.

[0148] If no index item is searched in any layer in the three-layer structure, it indicates that the known fingerprint is not recorded in the fingerprint index.

[0149] The above-mentioned way of querying the known fingerprints in the first fingerprint set based on the fingerprint index is only a simple description, and the way of querying based on the fingerprint index will be described in detail below by taking a query target fingerprint as an example.

[0150] It should be noted that in the description of the structure of the fingerprint index, it is assumed that any field in the fingerprint can be equal to all possible values. For example, it is assumed that a field in the fingerprint is a binary field containing two bits, and all possible values of the field are 00, 01, 11, and 10, a total of four possible values. In actual applications, the value of a field in a fingerprint is related to the specific fingerprints included in the first fingerprint set. Taking a field in any fingerprint as a binary field containing two bits as an example, due to the different specific fingerprints included in the first fingerprint set, there can be partial values of the four possible values of the field, or when the values of other fields in the fingerprint are fixed, there can be partial values of the four possible values of the field. Then, the index layer corresponding to the field in the fingerprint index only needs to cover the partial values.

[0151] In another aspect, the fingerprints recorded in the fingerprint index constructed for the first fingerprint set are the fingerprints in the first fingerprint set. Each index layer in the fingerprint index corresponds to a field in the fingerprint, and each index layer covers all possible values of the field in each fingerprint in the first fingerprint set.

[0152] In combination with the index item representing a value of the corresponding field, the above-mentioned "each index layer covers all possible values of the field in each fingerprint in the first fingerprint set" is further described as follows:

[0153] In the two coverage manners provided in the embodiments of the present application, the following are introduced in turn:

[0154] Two numerical values are introduced here. One data value is the number of possible values of the field, and for the convenience of description, this kind of data value is referred to as the first numerical value. One numerical value is the number of possible values of the field in each fingerprint in the first fingerprint set, and for the convenience of description, this kind of numerical value is referred to as the second numerical value. The first numerical value is the number of possible values of the field without limiting the first fingerprint set. The second numerical value is the number of possible values of the field under the condition of limiting the first fingerprint set. The first numerical value is greater than or equal to the second numerical value.

[0155] Coverage manner one: invalid index items are allowed to be set in each layer index layer, and the invalid index item in any index layer does not have a corresponding set of index items in the next layer index layer.

[0156] Since the position of an index item in a set of index items in the fingerprint index represents the value represented by the index item, on the premise that the set of index items can contain multiple index items, the number of the multiple index items is equal to the first value. That is, the set of index items reserves multiple index items representing all possible values. For all possible values of the field in each fingerprint in the first fingerprint set, the storage address of a set of index items in the next index layer corresponding to the value is recorded in the index item representing the value. For values not in the field in each fingerprint in the first fingerprint set, no storage address is recorded in the index item representing the value, or an invalid identifier is recorded in the index item representing the value, to indicate that the value does not exist or that the index item is invalid. In this way, the number of index items with recorded storage addresses in the index layer (that is, valid index items) is equal to the second value.

[0157] Based on this setting mode, there are two kinds of index items in the fingerprint index, one is a valid index item, the valid index item records a storage address, that is, corresponds to a set of index items in the next index layer, in the embodiment of the present application, the valid index item can represent a value and correspond to a set of index items in the next index layer. One is an invalid index item, the invalid index item does not record a storage address, and there is no set of index items in the next index layer corresponding to the invalid index item.

[0158] For the index items in the bottommost index layer, if the index items in the bottommost index layer are arranged in mode one, the number of index items in the bottommost index layer is consistent with the first value. There are valid index items or invalid index items in the bottommost index layer. The invalid index item includes an invalid identifier to indicate that the value does not exist. The valid index item includes a valid identifier and / or related information of the fingerprint. The valid identifier indicates that the value exists or that the index item is valid. If the index items are arranged in mode two, the index items included in the bottommost index layer are all valid index items. That is, each index item in the bottommost index layer includes a valid identifier and / or related information of the fingerprint.

[0159] Mode two of coverage: invalid index items are allowed to be set in the bottommost index layer, the invalid index item of the bottommost index layer indicates that the fingerprint represented by the invalid index item does not exist. The invalid index item of the bottommost index layer can include an invalid identifier, and further, the invalid index item does not include related information of the fingerprint.

[0160] Since the position of an index item in a group of index items in the fingerprint index represents the value represented by the index item, on the premise that the group of index items can contain a plurality of index items, the number of the plurality of index items is equal to the first value. That is, the group of index items reserves a plurality of index items representing all possible values. Any index item in the plurality of index items corresponds to a group of index items in the next index layer; that is, one index item corresponds to a group of index items in the next index layer. Any index item in the plurality of index items records the storage address of the corresponding group of index items in the next index layer. The index items in each index layer are arranged in this way.

[0161] For the index items in the bottommost index layer, if the index items in the bottommost index layer are arranged in way one, there can be valid index items or invalid index items in the bottommost index layer. If the fingerprint index is applicable to the scenario of querying whether data is stored or whether data exists in a data set, the invalid index item includes an invalid identifier to indicate that the fingerprint represented by the invalid index item does not exist. If the fingerprint index is applicable to the scenario of querying fingerprint-related information, the invalid index item does not include fingerprint-related information, that is, it can be determined that the index item is an invalid index item without obtaining fingerprint-related information from the index item. Further, the invalid index item can also include an invalid identifier to indicate that the fingerprint represented by the invalid index item does not exist. The valid index item includes a valid identifier and / or fingerprint-related information. The valid identifier indicates that the fingerprint value represented by the valid index item exists or that the index item is valid. For the description of the index item in the bottommost index layer representing a certain fingerprint, please refer to the foregoing content, which will not be repeated here.

[0162] If the index items are arranged in way two, the index items included in the bottommost index layer are all valid index items. That is, each index item in the bottommost index layer includes a valid identifier and / or fingerprint-related information.

[0163] As shown in the fingerprint index in FIG. 1, the fingerprint index has a multi-layer structure, and the structure of the fingerprint index is simpler, which can simplify the construction process of the fingerprint index and improve the query efficiency based on the fingerprint index. In the fingerprint index, an index item in the fingerprint index can represent the value of a certain field in the fingerprint, and only fingerprint-related information needs to be stored in the bottommost index layer of the fingerprint index. Compared with the fingerprint table that directly records the key-value pair (that is, the key in the key-value pair is the fingerprint, and the value is the data corresponding to the fingerprint), the fingerprint index does not need to record a large number of fingerprints, and the storage space occupied by the fingerprint index is smaller.

[0164] 2) Construction process of the fingerprint index.

[0165] The following describes a process of constructing a fingerprint index for fingerprints in the first fingerprint set. As shown in FIG. 3, a fingerprint index construction method provided by an embodiment of the present application includes the following steps:

[0166] Step 300: The construction device samples the fingerprints in the second fingerprint set to obtain the first fingerprint set.

[0167] When constructing the fingerprint index, the construction device can construct the fingerprint index for only part of the fingerprints, which can effectively reduce the size of the fingerprint index and reduce the storage space occupied by the fingerprint index.

[0168] For example, the second fingerprint set includes fingerprints of all data in a data set, and each fingerprint is a fingerprint of a data slice in the data set. The data set can be a database, data in a node, data in a node cluster, or data in a hard disk, or data in a hard disk group, etc. When constructing the fingerprint index, the construction device can take the second fingerprint set as the first fingerprint set, or can perform step 300 to extract part of the fingerprints from the second fingerprint set, and the part of the fingerprints constitutes the first fingerprint set.

[0169] Embodiments of the present application do not limit the execution manner of step 300, and the fingerprints in the second fingerprint set can be sampled in a modulo manner, or other manners. The sampling manner of the fingerprints will be further described below, and thus will not be described in detail here.

[0170] Step 301: The construction device splits the fingerprints in the first fingerprint set into N fields. The number of fields determines the number of layers of the fingerprint index, and embodiments of the present application do not limit the number of fields. N is a positive integer.

[0171] The fingerprint is usually a string. In the case that the fingerprint generation algorithm is fixed, the length of the fingerprint generated by calling the fingerprint generation algorithm for data is fixed. It can be seen that the fingerprint is a string with a fixed length. The type of the string can be different according to different fingerprint generation algorithms. For example, the fingerprint can be a hexadecimal string, or a decimal string.

[0172] When performing step 301, the construction device can directly split the fingerprint into multiple segments, and each segment is a field. Alternatively, the construction device can first convert the fingerprint, such as converting the fingerprint into a binary string; and then split the converted fingerprint into multiple segments, and each segment is a field.

[0173] Step 302: The construction device constructs each index layer starting from the topmost index layer. For any index layer, the index layer corresponds to a field in the fingerprint, and constructing the index layer includes: setting one or more groups of index items in the index layer, each group of index items including one or more index items, and each index item in the index layer representing a value of the field. An index item in the upper index layer of the index layer corresponds to a group of index items in the index layer.

[0174] The index layer includes a plurality of index items covering all possible values of the field corresponding to the index layer, and each index item represents a value. The number of groups of index items in the index layer is equal to the number of index items in the upper index layer. The topmost index layer of the fingerprint index includes a group of index items.

[0175] In combination with the description of each index layer covering all possible values of the field in each fingerprint in the first fingerprint set, valid index items and invalid index items, the way of constructing the index layer is described as follows:

[0176] If the covering mode one is adopted, the construction process is as follows:

[0177] When constructing the index layer, the construction device sets one or more groups of index items in the index layer, each group of index items including one or more index items, and each index item recording the storage address of a group of index items in the next index layer corresponding to the index item. The position of the index item in the group of index items represents the value represented by the index item.

[0178] Specifically, when the construction device sets any group of index items in the index layer, a plurality of index items are set in the group of index items, and the number of set index items is equal to a first value, which is equal to the number of possible values of the field corresponding to the index layer. The position of each index item in the plurality of index items represents a possible value of the field. In other words, the possible values of the field corresponding to the index layer are sequentially sorted according to the size of the value, and the sorting position of a specific value is the position of the index item representing the specific value in the plurality of index items.

[0179] For all possible values of the field in each fingerprint in the first fingerprint set, the storage address of a group of index items in the next index layer corresponding to the value is recorded in the index item representing the value. The index item is a valid index item.

[0180] For the values of the field that are not in each fingerprint in the first fingerprint set, no storage address is recorded in the index item representing the value, or an invalid identifier is recorded in the index item representing the value, to indicate that the value does not exist or that the index item is invalid. The index item is an invalid index item.

[0181] In this way, the number of index items (i.e. valid index items) in the index layer recording the storage addresses is equal to the second value. The second value is the number of possible values of the field under the condition that the first fingerprint set is defined.

[0182] For the bottommost index layer, if the set of index items in the bottommost index layer is arranged in the first way, the way of constructing the set of index items in the bottommost index layer is similar to the way mentioned above, with the difference that there are also valid index items or invalid index items in the bottommost index layer. The invalid index items can include invalid identifiers to indicate that the value does not exist, or the invalid index items can include fingerprint-related information to indicate that the fingerprint represented by the invalid index item does not exist. The valid index items include valid identifiers and / or fingerprint-related information, and the valid identifiers indicate that the value exists or that the index item is valid. If the set of index items in the bottommost index layer is arranged in the second way, when constructing the set of index items in the bottommost index layer corresponding to the index items of the upper index layer, a plurality of valid index items are arranged in the set of index items, i.e. a plurality of valid index items are arranged, and each index item includes the value represented by the index item, and optionally, includes a valid identifier and / or fingerprint-related information.

[0183] If the second covering way is adopted, the construction process is as follows:

[0184] When constructing the index layer, the construction device arranges one or more sets of index items in the index layer, each set of index items including one or more index items, and in each index item, the storage address of a set of index items in the next index layer corresponding to the index item is recorded. The position of the index item in the set of index items represents the value represented by the index item.

[0185] Specifically, when the construction device arranges any set of index items in the index layer, a plurality of index items are arranged in the set of index items, and the number of arranged index items is equal to the first value, which is equal to the number of possible values of the field corresponding to the index layer. The position of each index item in the plurality of index items represents a possible value of the field. In other words, the possible values of the field corresponding to the index layer are sequentially sorted according to the size of the values, and the sorting position of a specific value is the position of the index item representing the specific value in the plurality of index items. In any index item in the plurality of index items, the storage address of a set of index items in the next index layer corresponding to the index item is recorded. The index items in each index layer are arranged in this way.

[0186] The construction process of the bottommost index layer is similar to the construction process of the bottommost index layer using the first covering way, and specific details can be found in the foregoing description, which will not be repeated here.

[0187] According to the construction manner, the index items in each index layer are in a corresponding relationship. For example, from any index item in the lowest index layer, a corresponding index item can be found in each index layer above.

[0188] Therefore, one index item in the lowest index layer can represent one fingerprint. The values of the fields in the fingerprint are consistent with the values represented by the index item in the lowest index layer and the values represented by the corresponding index items in the index layers above.

[0189] Here, the "the index items included in the index layer cover all possible values of the field corresponding to the index layer" is explained as follows: all possible values here refer to all possible values of the field in each fingerprint in the first fingerprint set. The corresponding relationship between the index items in each index layer is introduced. For a group of index items in any index layer, all possible values of the field represented by the group of index items are all possible values of the field in the fingerprints in the first fingerprint set that satisfy the following condition: the fingerprints include one or more upper layer fields, and the values of the upper layer fields in the fingerprints are consistent with the values represented by the corresponding index items in the upper index layer. The upper layer field is the field corresponding to the index layer above the index layer where the group of index items is located. The index layer corresponding to one upper layer field is referred to as the upper index layer.

[0190] After steps 301 to 302, the index items that can be introduced in the fingerprint index are arranged based on the current first fingerprint set. For the fingerprint index used in a scenario where fingerprint-related information needs to be obtained, the construction device needs to add the fingerprint-related information of the fingerprint in each index item in the lowest index layer, and the fingerprint is the fingerprint represented by the index item. For the fingerprint index used in a scenario where it is necessary to determine whether data is stored or whether data already exists, it is not necessary to add the fingerprint-related information of the fingerprint in each index item in the lowest index layer. Of course, the fingerprint index including the fingerprint-related information in each index item in the lowest index layer is also applicable to the scenario where it is necessary to determine whether data is stored or whether data already exists.

[0191] In the embodiments of the present application, the fingerprints in the first fingerprint set are allowed to change, such as adding new fingerprints in the first fingerprint set, deleting existing fingerprints in the first fingerprint set, and modifying the fingerprints in the first fingerprint set. When the fingerprints in the first fingerprint set change, the fingerprint index needs to be updated.

[0192] When a new fingerprint is added in the first fingerprint set, the building device adds an index item representing the fingerprint in the fingerprint index. The value of each field in the new fingerprint is determined. The building device starts from the topmost index layer of the fingerprint index, and performs the following judgment operation in each index layer: it is determined whether there is an index item representing the value of the specified field in the new fingerprint in the search range of each index layer (e.g., whether the storage address is recorded in the index item representing the value in the search range), the specified field being the field corresponding to the index layer, and if there is (e.g., the storage address is recorded in the index item representing the value in the search range), it jumps to a group of index items in the next index layer corresponding to the index item. If there is not (e.g., the storage address is not recorded in the index item representing the value in the search range), an index item representing the value of the specified field in the new fingerprint is added in the index layer, e.g., a storage address is recorded in the index item representing the value, the storage address being the storage address of a group of index items in the next index layer, and a group of index items is added in the next index layer, the group of index items temporarily not writing index items, e.g., the plurality of index items not recording any information, and jumping to the group of index items. The search range of each index layer is the group of index items jumped to when the judgment operation is performed in the previous index layer. For the topmost index layer, the search range is all index items of the topmost index layer. The judgment operation is performed layer by layer until the bottommost index layer is reached.

[0193] In the bottommost index layer, after the index item is added, if the fingerprint index is used to query fingerprint-related information, the building device adds the fingerprint-related information of the new fingerprint in the added index item.

[0194] It should be noted that the above method of adding an index item in the fingerprint abbreviation is based on the aforementioned coverage mode one. If the coverage mode two is adopted, when a new fingerprint is added in the first fingerprint set, the building device only needs to add the index item of the fingerprint in the bottommost index layer. For example, when the mode one is adopted in the bottommost index item, the building device can mark the index item representing the value of the specified field in the new fingerprint in the bottommost index layer as a valid index item, e.g., storing the related information of the new fingerprint in the index item (the fingerprint index item is applicable to the scenario of querying fingerprint-related information), or adding a valid identifier in the index item (the fingerprint index item is applicable to the scenario of querying whether data is stored or whether data exists). For another example, when the mode two is adopted in the bottommost index item, the building device can add an index item in the bottommost index layer, the index item including the value of the specified field in the new fingerprint, and further, the index item can add the related information of the new fingerprint (the fingerprint index item is applicable to the scenario of querying fingerprint-related information).

[0195] When an existing fingerprint is deleted from the first fingerprint set, the constructing device deletes the index item representing the deleted fingerprint in the lowest index layer of the fingerprint index. When the index items in the lowest index layer are arranged in the first manner, the constructing device can mark the index item representing the deleted fingerprint in the lowest index layer as an invalid index item, for example, by deleting the relevant information of the fingerprint in the index item (the fingerprint index item is applicable to the scenario of querying the relevant information of the fingerprint), or by adding an invalid identifier in the index item (the fingerprint index item is applicable to the scenario of querying whether data is stored or whether the data exists). When the index items in the lowest index layer are arranged in the second manner, the constructing device can delete the index item representing the deleted fingerprint in the lowest index layer.

[0196] When a fingerprint in the first fingerprint set is modified, since the modification of the fingerprint in the first fingerprint set can cause the fingerprint of the fingerprint to change, the fingerprint of the fingerprint before the modification is referred to as an old fingerprint, and the fingerprint of the fingerprint after the modification is referred to as a new fingerprint. When the fingerprint index is updated, the constructing device deletes the index item representing the old fingerprint in the lowest index layer. The constructing device adds an index item representing the new fingerprint in the fingerprint index, and the manner of adding the index item representing the new fingerprint in the fingerprint index and the manner of deleting the index item representing the old fingerprint can be referred to the foregoing description, and will not be described herein again.

[0197] In addition, the length of the fingerprint-related information in the fingerprint index is allowed to be dynamically adjusted. That is, the lengths of the relevant information recorded in the fingerprint index can be inconsistent.

[0198] For example, the fingerprint-related information is a node identifier where the data is located. Since there are many ways to set the node identifier, there can be a case where the lengths of the node identifiers in the same node cluster are consistent, and the lengths of the node identifiers in different node clusters are inconsistent. In the embodiment of the present application, the fingerprint-related information is recorded in the index item, and the lengths of the fingerprint-related information are different, which does not affect the structure of the fingerprint index. Accordingly, even if a new index item needs to be added in the fingerprint index, and the length of the fingerprint-related information in the new index item is different from the length of the fingerprint-related information in the existing index item, according to the values of the fields in the fingerprint, the position of the new index item in the fingerprint index can be uniquely determined, and then the new index item is added.

[0199] After the fingerprint index is built, if a new index item needs to be added in the fingerprint index, the new index item contains fingerprint-related information of a larger length, the existing index items in the fingerprint index do not need to be changed, and only the corresponding index item needs to be added in the fingerprint index. The way of adding the index item is described above, and will not be repeated here. If a new index item needs to be added in the fingerprint index, the new index item contains fingerprint-related information of a smaller length, the existing index items in the fingerprint index do not need to be changed, and only the corresponding index item needs to be added in the fingerprint index. The way of adding the index item is described above, and will not be repeated here.

[0200] Because the length of the fingerprint-related information changes, the length of the index item in the bottommost index layer of the fingerprint index may be different.

[0201] Of course, the length of the fingerprint in the fingerprint index is also allowed to change. For example, new fingerprints can be added in the first fingerprint set, and the length of the new fingerprint can be inconsistent with the length of the fingerprint recorded in the fingerprint index. For example, some fingerprints need to be added in the first fingerprint set, and each fingerprint identifies a data shard. In order to identify these data shards, a character can be added to the fingerprint of the data shard after the fingerprint of the data shard is generated. In this way, the length of the fingerprint of the data shard is greater than the length of the fingerprint recorded in the fingerprint index. For example, some fingerprints need to be added in the first fingerprint set, and each fingerprint identifies a data shard. In order to identify these data shards, a character can be deleted from the fingerprint of the data shard after the fingerprint of the data shard is generated. In this way, the length of the fingerprint of the data shard is less than the length of the fingerprint recorded in the fingerprint index.

[0202] ①After the fingerprint index is built, the length of the fingerprint increases.

[0203] That is, the length of the fingerprint represented by the index item to be recorded in the fingerprint index later becomes larger. For the convenience of description, the field corresponding to the bottommost index layer of the fingerprint index after the fingerprint index is built is referred to as field P, and the field added after the length of the fingerprint increases is referred to as field Q. Then, after the fingerprint index is built, the length of the fingerprint increases, and the construction device can add a new index item in the fingerprint index, and the fingerprint represented by the new index item is the fingerprint after the length increases. The way of adding the index item can be referred to the description above, and the difference is that the new index item added in the bottommost index layer includes the field Q and the field P. That is, the new index item added in the bottommost index layer can represent a value of the field Q and a value of the field P.

[0204] ②After the fingerprint index is built, the length of the fingerprint decreases.

[0205] That is, the length of the fingerprint represented by the index item to be recorded in the fingerprint index subsequently changes, for the convenience of description, the field corresponding to the bottom layer of the index layer after the fingerprint index is constructed is called field P, and the field P is reduced to field K after the fingerprint length is reduced. Then, after the fingerprint index is constructed, the fingerprint length is reduced, and the construction device can add a new index item in the fingerprint index, the fingerprint represented by the new index item is the fingerprint after the length is reduced, and the way of adding the index item can be referred to the foregoing description, the difference is that when the new index item is added in the bottom layer of the index layer, the field corresponding to the new index item is the field K. That is, the new index item added in the bottom layer of the index layer can represent a value of the field K.

[0206] Considering the possible scenario of fingerprint length reduction, the arrangement mode of the index item in the bottom layer of the index layer in the fingerprint index suitable for this scenario can adopt mode two. That is, the index item includes the value of the corresponding field, and optionally, the index item can also include the related information of the fingerprint. Since the values included in the index item can be different, this also leads to the possibility that the lengths of the index items in the bottom layer of the index layer of the fingerprint index are different.

[0207] Next, taking the construction of the fingerprint index shown in FIGS. 2A and 2B as an example, the construction process of the fingerprint index is further introduced, so that the construction method of the fingerprint index can be more clearly understood.

[0208] I. The fingerprint index includes a double-layer structure:

[0209] As shown in FIG. 4, a fingerprint index construction method provided by an embodiment of the present application includes the following steps:

[0210] Step 400: The construction device samples the fingerprints in the second fingerprint set to obtain a first fingerprint set. The specific execution mode can be referred to the related description of step 300, which will not be described here.

[0211] Step 401: The construction device splits the fingerprints in the first fingerprint set into two fields. The two fields include field A and field B.

[0212] Step 402: The construction device constructs the first index layer of the fingerprint index: multiple index items are set in the first index layer, and one index item in the first index layer represents a value of field A.

[0213] Suppose that the possible specific values of field A of the fingerprints in the first fingerprint set are integers from 0 to 9, and the construction device sets 10 index items in the first index layer, which represent 10 integers from 0 to 9.

[0214] The first index layer is constructed as follows:

[0215] If the first index layer is constructed in the first way, the construction process is as follows:

[0216] In constructing the first index layer, the construction device sets a plurality of index items in the first index layer, and the number of the index items set is equal to the first value, which is equal to the number of possible values of the field A. The position of each index item in the plurality of index items represents a possible value of the field A. In other words, the possible values of the field A corresponding to the index layer are sequentially sorted according to the size of the values, and the sorting position of a specific value is the position of the index item representing the specific value in the plurality of index items.

[0217] For all possible values of the field A in each fingerprint in the first fingerprint set, the storage address of a group of index items in the next index layer corresponding to the value is recorded in the index item representing the value. The index item is an effective index item.

[0218] For the values of the field A that are not in each fingerprint in the first fingerprint set, no storage address is recorded in the index item representing the value, or an invalid identifier is recorded in the index item representing the value, to indicate that the value does not exist or that the index item is invalid. The index item is an invalid index item.

[0219] Still taking the example of the possible specific values of the field A in the fingerprints in the first fingerprint set being integers from 0 to 9, and assuming that the possible values of the field A are integers from 0 to 15 without limitation of the first fingerprint set. That is, the first value is equal to 16, and the second value is equal to 10. In constructing the first index layer, the construction device sets 16 index items in the first index layer, and the arrangement position of any index item in the 16 index items indicates a specific value of the field A. For example, the value represented by the index item at the first position is 0, and the value represented by the index item at the end is 15.

[0220] For all possible values of the field A (such as integers from 0 to 9) in each fingerprint in the first fingerprint set, the storage address of a group of index items in the second index layer corresponding to the value is recorded in the index item representing the value. The index item is an effective index item.

[0221] For the values of the field A that are not in each fingerprint in the first fingerprint set (such as integers from 10 to 15), no storage address is recorded in the index item representing the value, or an invalid identifier is recorded in the index item representing the value, to indicate that the value does not exist or that the index item is invalid. The index item is an invalid index item.

[0222] The construction process of the second covering mode is similar to the construction process of the first covering mode, and the difference is that there is no concept of invalid index item in the first index layer, the first index layer sets multiple index items, and the number of the set index items is equal to the first value, which is equal to the number of possible values of the field A. Each index item in the first index layer corresponds to a group of index items in the second index layer, and each index item records the storage address of a group of index items in the next index layer corresponding thereto.

[0223] Step 403: constructing the second index layer of the fingerprint index: multiple groups of index items are set in the second index layer, one index item of the first index layer corresponds to one group of index items of the second index layer, and one index item in a group of index items of the second index layer represents one value of the field B.

[0224] Suppose the possible specific values of the field B are integers from 0 to 9, the construction device sets 10 groups of index items in the second index layer, and one index item of the first index layer corresponds to one group of index items. Each group of index items includes 10 index items, which represent 10 integers from 0 to 9.

[0225] If the fingerprint index is used to query fingerprint-related information, the construction device stores the related information of the fingerprint in each index item of the second index layer. In view of the corresponding relationship between the first index layer and the second index layer, one index item in the second index layer represents one fingerprint, and the related information stored in the index item is the related information of the represented fingerprint.

[0226] The second index layer is the bottommost index layer, and the construction method of the second index layer is similar to the construction method of the bottommost index layer in step 302. For details, please refer to the foregoing content, which will not be repeated here.

[0227] II. The fingerprint index includes three layers of structure:

[0228] As shown in FIG. 5, a fingerprint index construction method provided by an embodiment of the present application includes the following steps:

[0229] Step 500: the construction device samples the fingerprints in the second fingerprint set to obtain a first fingerprint set. For details, please refer to the related description of step 300, which will not be repeated here.

[0230] Step 501: the construction device splits the fingerprints in the first fingerprint set into three fields. The two fields include field A, field B, and field C.

[0231] Step 502: the construction device constructs the first index layer of the fingerprint index. For details, please refer to the related description of step 402, which will not be repeated here.

[0232] Step 503: The constructing device constructs a second index layer of the fingerprint index. The construction of any group of index items in the second index layer is similar to the construction of the plurality of index items in the first index layer in step 502. For details, refer to the foregoing description.

[0233] Step 504: The constructing device constructs a third index layer of the fingerprint index. A plurality of groups of index items are arranged in the third index layer. One index item of the second index layer corresponds to one group of index items of the third index layer. One index item in one group of index items of the third index layer represents one value of the field C.

[0234] Suppose the possible specific values of the field C are integers from 0 to 9. 100 groups of index items are arranged in the third index layer. One index item of the second index layer corresponds to one group of index items. Each group of index items includes 10 index items, which represent 10 integers from 0 to 9.

[0235] If the fingerprint index is used to query fingerprint-related information, the constructing device stores the fingerprint-related information in each index item of the third index layer. Since there is a corresponding relationship among the first index layer, the second index layer, and the third index layer, one index item in the third index layer represents one fingerprint, and the related information stored in the index item is the related information of the represented fingerprint.

[0236] The third index layer is the bottommost index layer. The construction of the third index layer is similar to the construction of the second index layer in step 402, except that the corresponding field is different. For details, refer to the foregoing description.

[0237] It should be noted that in the foregoing description of the fingerprint index construction method, the constructing device is taken as an example to illustrate the fingerprint index construction method. The present embodiment does not limit the relationship between the constructing device and the device where the first fingerprint set is located. The device where the first fingerprint set is located can also be understood as a device that stores data corresponding to the fingerprints in the first fingerprint set. The constructing device can be deployed on the device, or it can be a device before the device for constructing the fingerprint index. After the constructing device constructs the fingerprint index, it can send the fingerprint index to the device, and the device receives and stores the fingerprint index.

[0238] Due to the simple structure of the fingerprint index, the construction of the fingerprint index is relatively easy, and the update of the fingerprint index is more convenient. In addition, the fingerprint index supports changes in the length of fingerprint-related information, and has stronger scalability.

[0239] 3) Query method based on fingerprint index.

[0240] Taking the structure of the fingerprint index as shown in FIG. 1 as an example, the query method based on the fingerprint index is described. As shown in FIG. 6, it is a schematic diagram of a query method based on a fingerprint index provided by an embodiment of the present application. In the method, the query device queries the target fingerprint based on the fingerprint index in the first fingerprint set.

[0241] Step 600: The query device acquires the target fingerprint to be queried and determines the value of each field in the target fingerprint.

[0242] There are many scenarios in which the query device queries the target fingerprint in the first fingerprint set. For example, the query device receives a fingerprint query instruction, the fingerprint query instruction is used to indicate that the target fingerprint is queried in the first fingerprint set, and the fingerprint query instruction carries the target fingerprint. For another example, the query device receives a data query instruction, the data query instruction is used to indicate that the target data is queried in the data set. That is, it is determined whether the target data exists in the data set. After receiving the data query instruction, the query device generates the fingerprint of the target data (that is, the target fingerprint). For another example, the query device receives a data write request, the data write request is used to request to write the target data. After receiving the data write request, the query device acquires the target data from the data write request and generates the fingerprint of the target data.

[0243] Step 601: The query device starts from the topmost index layer of the fingerprint index and performs the following first search operation layer by layer until the lowest index layer is reached. The first search operation performed in any index layer is as follows: search for an index item representing the value of the first target field in the target fingerprint in the search range of the index layer, the first target field being the field corresponding to the index layer, and if the index item is searched, jump to a group of index items in the next index layer corresponding to the index item. If no index item is searched, the query method based on the fingerprint index can be terminated.

[0244] The search range of the index layer is a group of index items jumped to by the first search operation performed in the previous index layer. For the topmost index layer, the search range is all index items of the topmost index layer.

[0245] In combination with the aforementioned manner in which the index item represents a value of the corresponding field and the valid index item and the invalid index item, the manner in which the query device performs step 601 is described.

[0246] If the coverage mode one is adopted, the query process is as follows:

[0247] The first search operation performed by the query device in any index layer is as follows: the query device searches the search range of the index layer according to the value of the first target field in the target fingerprint to find an index item representing the value. If an effective index item is found, the query device jumps to the storage address of a group of index items in the next index layer recorded in the searched effective index item. If no effective index item is found, the query method based on the fingerprint index is terminated. It can be seen that the query device in step 602 searches an effective index item representing the value of each field in the target fingerprint layer by layer. The index item searched in the bottommost index layer is an effective index item, which indicates that the target fingerprint is found. Otherwise, the target fingerprint is not found.

[0248] If the second coverage mode is adopted, the query process is as follows:

[0249] The first search operation performed by the query device in any index layer is as follows: the query device searches the search range of the index layer according to the value of the first target field in the target fingerprint to find an index item representing the value. According to the storage address of a group of index items in the next index layer recorded in the searched index item, the query device jumps to the storage address. Until the bottommost index layer is reached. It can be seen that the query device in step 602 searches an index item representing the value of each field in the target fingerprint layer by layer. The index item searched in the bottommost index layer is an effective index item, which indicates that the target fingerprint is found. Otherwise, the target fingerprint is not found.

[0250] Step 602: The fingerprint represented by the index item searched by the query device in the search range of the bottommost index layer is the target fingerprint.

[0251] The fingerprint index is applied to a scenario in which fingerprint-related information needs to be obtained. Then, the query device obtains the fingerprint-related information of the target fingerprint from the index item searched in the search range of the bottommost index layer. If step 601 is performed, no corresponding index item is searched in a certain layer, and then the query method based on the fingerprint index is terminated, indicating that the target fingerprint is not recorded in the fingerprint index, or the fingerprint-related information of the target fingerprint does not exist.

[0252] The fingerprint index is applied to a scenario in which fingerprint-related information needs to be obtained. Then, the query device obtains the fingerprint-related information of the target fingerprint from the index item searched in the search range of the bottommost index layer.

[0253] The fingerprint index is applied to a scenario in which it is necessary to determine whether data exists or whether data has been stored. Then, the query device searches an index item in the search range of the bottommost index layer, which indicates that the data corresponding to the target fingerprint exists or the data corresponding to the target fingerprint has been stored.

[0254] Since the fingerprint index has a hierarchical structure and there is a corresponding relationship between the index layers, when the query is performed based on the fingerprint index, a smaller index item range can be located as a search range layer by layer. The corresponding relationship between the index layers can not only simplify the query process, but also quickly locate the search range during the layer-by-layer search. Compared with the query based on the fingerprint table, the search range can be gradually reduced in the query process with the help of the fingerprint index by means of layer-by-layer search. That is, the query based on the fingerprint table takes the fingerprints recorded in the entire fingerprint table as the search range, and the layer-by-layer search in the query process with the help of the fingerprint index is essentially to exclude the fingerprints whose sub-segment values are different from the sub-segment values of the target fingerprint.

[0255] The query process based on the fingerprint index will be further introduced below by taking the query based on the fingerprint index shown in FIGS. 2A and 2B as an example, so as to more clearly understand the query method based on the fingerprint index.

[0256] I. The fingerprint index includes a double-layer structure.

[0257] As shown in FIG. 7, it is a query method based on a fingerprint index provided by an embodiment of the present application.

[0258] Step 700: The query device acquires a target fingerprint to be queried, and determines the values of field A and field B in the target fingerprint.

[0259] Step 701: The query device determines a first index item from the plurality of index items in the first index layer of the fingerprint index, and the value represented by the first index item is consistent with the value of field A in the target fingerprint.

[0260] In combination with the aforementioned manner in which the index item represents a value of the corresponding field and the valid index item and the invalid index item, the manner in which the query device performs step 701 will be described.

[0261] If the coverage mode one is adopted, the query process is as follows:

[0262] The query device performs the following in the first index layer: the query device finds the valid index item representing the value of field A in the first index layer according to the value of field A in the target fingerprint.

[0263] The query device can learn the starting address P of the first index layer in the fingerprint index. The query device calculates the starting address of the index item representing the value of field A in the target fingerprint according to the value of field A in the target fingerprint. The value of field A in the target fingerprint is in the S-th position in the order of all possible values of field A. That is, there are S-1 values before the value of field A in the target fingerprint in the order of all possible values of field A, and there are S-1 index items before the index item representing the value. Then, the starting address p of the index item representing the value satisfies: p = P + (S-1) * D, where D is the length of the index item in the first index layer. In other words, the starting address of the index item representing the value is the starting address P of the first index layer in the fingerprint index item plus an offset, and the offset is equal to the total length of the index items before the index item representing the value, that is, the offset is (S-1) * D.

[0264] After the query device calculates the starting address of the index item representing the value, the query device acquires the index item at the starting address of the index item representing the value, and determines whether the storage addresses of a group of index items in the corresponding second index layer are recorded in the index item. If the storage addresses of a group of index items in the corresponding second index layer are recorded in the index item, the index item is the first index item. If the storage addresses of a group of index items in the corresponding second index layer are not recorded in the index item, the index item is an invalid index item, and is not the first index item. That is, there is no first index item in the first index layer.

[0265] After the query device finds the first index item, the query device acquires the storage addresses of a group of index items in the second index layer corresponding to the first index item from the first index item, and performs step 702.

[0266] If the coverage mode two is adopted, the query process is as follows:

[0267] In the first index layer, the query device finds the index item representing the value of field A in the target fingerprint according to the value of field A in the target fingerprint. The finding method of the index item is similar to the finding method of the valid index item in the query process when the coverage mode one is adopted, and the difference is that the index item representing the value can be found in the query process, and the storage addresses of a group of index items in the second index layer corresponding to the index item are recorded in the index item.

[0268] Step 702: Jump to a group of index items in the second index layer corresponding to the first index item, and the query device determines a second index item in the group of index items in the second index layer corresponding to the first index item, and the value represented by the second index item is consistent with the value of field B in the target fingerprint.

[0269] The fingerprint index is applied to a scenario in which fingerprint-related information needs to be acquired. Then, the query device acquires the fingerprint-related information of the target fingerprint from the second index item.

[0270] The fingerprint index is applied to a scenario in which it is needed to determine whether data exists or whether data has been stored. Then, the query device searches the second index layer to the second index item, which indicates that data corresponding to the target fingerprint exists or data corresponding to the target fingerprint has been stored.

[0271] In combination with the aforementioned index item representing a value of a corresponding field and the valid index item and the invalid index item, the manner in which the query device performs step 702 is described.

[0272] The second index layer is the bottommost index layer. If the index items in the second index layer are arranged in the first manner, the query device determines the second index item in a group of index items in the second index layer corresponding to the first index item in a manner similar to that in which the query device determines the first index item from the plurality of index items in the first index layer in step 701. Details can be found in the foregoing description, which will not be repeated here. When the query device searches the second index layer to the valid index item representing the value of the field B of the target fingerprint, the query device can acquire the fingerprint-related information of the target fingerprint. If the index items in the second index layer are arranged in the second manner, the query device can traverse each index item in a group of index items in the second index layer corresponding to the first index item until the valid index item containing the value of the field B of the target fingerprint is found. If the valid index item containing the value of the field B of the target fingerprint is not found, it indicates that the target fingerprint is not queried.

[0273] II. The fingerprint index includes a three-layer structure.

[0274] As shown in FIG. 8, it is a schematic diagram of a query method based on a fingerprint index provided by an embodiment of the present application.

[0275] Step 800: The query device acquires a target fingerprint to be queried and determines values of a field A, a field B, and a field C in the target fingerprint.

[0276] Step 801: The query device determines a first index item from a plurality of index items in a first index layer of the fingerprint index. Details can be found in the related description of step 701, which will not be repeated here.

[0277] Step 802: The query device determines a second index item in a group of index items in a second index layer corresponding to the first index item, the value represented by the first index item being consistent with the value of the field B in the target fingerprint. The query device determines the second index item in a group of index items in the second index layer corresponding to the first index item in a manner similar to that in which the query device determines the first index item from the plurality of index items in step 701. Details can be found in the foregoing description, which will not be repeated here.

[0278] Step 803: Jump to the set of index items of the third index layer corresponding to the second index item, and the query device determines the third index item in the set of index items of the third index layer corresponding to the second index item, and the value represented by the third index item is consistent with the value of field C in the target fingerprint.

[0279] The fingerprint index is applied to a scenario in which fingerprint-related information needs to be obtained, and then the query device obtains the fingerprint-related information of the target fingerprint from the third index item.

[0280] The fingerprint index is applied to a scenario in which it is needed to determine whether data exists or whether data has been stored, and then the query device searches for the third index item in the third index layer, which indicates that the data corresponding to the target fingerprint exists or the data corresponding to the target fingerprint has been stored.

[0281] The third index layer is the bottommost index layer, and the way in which the query device determines the third index item in the set of index items of the third index layer corresponding to the second index item is similar to the way in which the second index item is determined in the set of index items of the second index layer corresponding to the first index item in step 703. For details, refer to the foregoing description, which will not be repeated here.

[0282] Based on the fingerprint index shown in FIG. 1, another fingerprint index is provided in an embodiment of the present application. The fingerprint index is introduced as follows.

[0283] 1. Construction process of the fingerprint index.

[0284] As shown in FIG. 9, the method for constructing a fingerprint index provided in an embodiment of the present application includes the following steps:

[0285] Step 901: The construction device splits the fingerprints in the first fingerprint set into N fields. The execution manner of step 901 is the same as that of step 301, and for details, refer to the foregoing description, which will not be repeated here.

[0286] Step 902: Each field corresponds to an index layer. The construction device constructs each index layer except the bottommost index layer. For any index layer except the bottommost index layer, the index layer corresponds to one field in the fingerprint, and the construction of the index layer includes: setting one or more sets of index items in the index layer, each set of index items includes one or more index items, and one index item in the index layer represents a value of the field. One index item in the upper index layer of the index layer corresponds to a set of index items in the index layer. The execution manner of step 901 is similar to that of step 302, except that the bottommost index layer is not constructed in step 902. For details, refer to the foregoing description, which will not be repeated here.

[0287] It should be noted that, since the bottommost index layer is provided with one or more groups of buckets, each group of buckets includes one or more buckets, and each bucket includes one or more index items. An index item in the upper layer of the bottommost index layer corresponds to a group of buckets in the bottommost index layer. The storage address of the corresponding group of buckets in the bottommost index layer is recorded in the index item in the upper layer of the bottommost index layer.

[0288] Step 903: The construction device constructs the bottommost index layer. For the bottommost index layer, the field corresponding to the bottommost index layer is further split into two subfields. In order to distinguish the two subfields, the two subfields are referred to as a first subfield and a second subfield, respectively.

[0289] The bottommost index layer is constructed, including: providing the bottommost index layer with one or more groups of buckets, each group of buckets including a plurality of buckets, and each group of buckets being used to store the first subfield and covering all possible values of the first subfield, and each bucket in each group of buckets representing a value of the first subfield. Each bucket includes a plurality of index items, and each bucket is used to store the second subfield, and the plurality of index items included in the bucket cover all possible values of the second subfield, and each index item represents a value of the second subfield.

[0290] For an index item in any bucket, the index item represents a value of the second subfield when the first subfield is a certain value, wherein the value of the first subfield is the value represented by the bucket. A bucket can be understood as a storage space of a specified size, and an index item can be written in the storage space. The size of the bucket can be pre-set, and the size of each bucket can be the same or different.

[0291] Similarly, according to this construction, there is a corresponding relationship between the index items of each layer of index layers. From any index item in the bottommost index layer, a corresponding index item can always be found in each layer of index layers. An index item in the lowest layer of index layers can represent a fingerprint. The values of the fields of the fingerprint are consistent with the values represented by the index item in the lowest layer and the values represented by the corresponding index items in the upper layers of the index layers.

[0292] In the bottommost index layer, the index items of the bottommost index layer are "clustered" in the granularity of buckets, and some index items can be gathered in a bucket. This design can simplify the structure of the fingerprint index on the one hand. On the other hand, when searching based on the fingerprint index, the search range can be effectively reduced when reaching the bottommost index layer; for example, if the bottommost index layer is not clustered when constructing the bottommost index layer, when searching based on the fingerprint index, the search range will be a group of index items corresponding to the upper layer index item when reaching the bottommost index layer. If clustering is performed when constructing the bottommost index layer, when searching based on the fingerprint index, the search range will be a group of buckets corresponding to the upper layer index item when reaching the bottommost index layer, and the value of the first subfield in the known fingerprint can quickly reduce the search range to one bucket in the group of buckets.

[0293] For the fingerprint index used in the scene where the fingerprint related information needs to be obtained, the construction device also needs to add the fingerprint related information of the fingerprint in each index item of the bottommost index layer, and the fingerprint is the fingerprint represented by the index item. For the fingerprint index used in the scene where it is necessary to determine whether the data is stored or whether the data already exists, it is not necessary to add the fingerprint related information of the fingerprint in each index item of the bottommost index layer.

[0294] In combination with each index layer covering all possible values of the field in each fingerprint in the first set of fingerprints, valid index items and invalid index items, the way of constructing the bottommost index layer is described:

[0295] If the first covering mode is adopted, the construction process is:

[0296] When the construction device sets any group of buckets in the bottommost index layer, a plurality of buckets are set in the group of buckets, and the number of the set buckets is equal to a third value, the third value is equal to the number of possible values of the first subfield, and the third value is the number of possible values of the first subfield when the first set of fingerprints is not limited. The position of each bucket in the plurality of buckets represents a possible value of the first subfield. In other words, the possible values of the first subfield are sequentially sorted according to the size of the value, and the sorting position of a specific value is the position of the bucket representing the specific value in the plurality of buckets.

[0297] For all possible values of the first subfield in each fingerprint in the first set of fingerprints, a plurality of index items are set in the bucket representing the value. The bucket is a valid bucket.

[0298] If the multiple index items in the valid bucket are arranged in way one: the construction device sets multiple index items in the bucket, the number of the set index items is equal to the fourth value, the fourth value is equal to the number of possible values of the second subfield. The fourth value is the number of possible values of the second subfield without limiting the first fingerprint set. The position of each index item in the multiple index items represents a possible value of the second subfield. In other words, the possible values of the second subfield are sequentially sorted according to the size of the value, and the sorting position of a specific value is the position of the index item representing the specific value in the multiple index items.

[0299] For all possible values of the second subfield in each fingerprint in the first fingerprint set, the index item representing the value is a valid index item. The valid identifier and / or fingerprint related information are stored in the index item.

[0300] For the value of the second subfield that does not exist in each fingerprint in the first fingerprint set, the index item representing the value does not record fingerprint related information (applicable to the scene of querying fingerprint related information) or records an invalid identifier in the index item representing the value, indicating that the value does not exist or that the index item is invalid. The index item is an invalid index item.

[0301] If the multiple index items in the valid bucket are arranged in way two: the construction device writes multiple valid index items in the valid bucket, each valid index item includes the represented value, and optionally includes the valid identifier and / or fingerprint related information

[0302] For the value of the first subfield that does not exist in each fingerprint in the first fingerprint set, the bucket representing the value is an invalid bucket, and no information or an invalid identifier can be recorded in the bucket.

[0303] If way two of coverage is used, the construction process is:

[0304] When the construction device sets any group of buckets in the bottommost index layer, the construction device sets multiple buckets in the group of buckets, the number of the set buckets is equal to the third value, the third value is equal to the number of possible values of the first subfield, and the third value is the number of possible values of the first subfield without limiting the first fingerprint set. The position of each bucket in the multiple buckets represents a possible value of the first subfield. In other words, the possible values of the first subfield are sequentially sorted according to the size of the value, and the sorting position of a specific value is the position of the bucket representing the specific value in the multiple buckets.

[0305] If the index items in the bucket are arranged in the first way, the construction device sets the index items in each bucket, and the number of index items in each bucket is equal to the fourth value. The fourth value is equal to the number of possible values of the second subfield. The fourth value is the number of possible values of the second subfield when the first fingerprint set is not limited. The position of each index item in the plurality of index items represents a possible value of the second subfield. In other words, the possible values of the second subfield are sequentially sorted according to the size of the value, and the sorting position of a specific value is the position of the index item representing the specific value in the plurality of index items.

[0306] For all possible values of the second subfield in each fingerprint in the first fingerprint set, the index item representing the value is a valid index item. The valid index item means that there is a fingerprint represented by the valid index item, and the valid identifier and / or fingerprint related information is stored in the index item.

[0307] For the value of the second subfield that does not exist in each fingerprint in the first fingerprint set, no storage address is recorded in the index item representing the value, or an invalid identifier is recorded in the index item representing the value, to indicate that the value does not exist or that the index item is invalid. The index item is an invalid index item. The valid index item means that there is no fingerprint represented by the valid index item

[0308] If the index items in the bucket are arranged in the second way, the construction device writes a plurality of valid index items in the bucket, and each valid index item includes the represented value, and optionally includes the valid identifier and / or fingerprint related information

[0309] In the embodiments of the present application, the size of each bucket in the bottommost index layer can be the same. The length of the index item in each bucket can be the same or different, and the length of the index item is related to the included fingerprint related information.

[0310] According to the construction method shown in FIG. 9, a fingerprint index including multiple index layers can be constructed. Taking the construction of a two-layer structure fingerprint index and a three-layer structure fingerprint index as examples. The construction method is similar to the construction method shown in FIG. 4 and FIG. 5, and the difference lies in the construction method of the bottommost index layer.

[0311] For the two-layer structure fingerprint index, the bottommost layer is the second index layer. When the second index layer is constructed, the field B corresponding to the second index layer is further split into two subfields. In order to distinguish the two subfields, the two subfields are called subfield B1 and subfield B2, respectively.

[0312] The second index layer is constructed, including: setting one or more groups of buckets in the second index layer. Corresponding to one group of buckets, one group of buckets is used to store the subfield B1, and the group of buckets includes a plurality of buckets. Each bucket in each group of buckets represents a value of the subfield B1. Each bucket includes a plurality of index items, and each index item represents a value of the subfield B2.

[0313] For the three-layer structure of the fingerprint index, the bottom layer is the second index layer. When the second index layer is constructed, the field B corresponding to the second index layer is further split into two subfields. In order to distinguish the two subfields, the two subfields are referred to as subfield B1 and subfield B2.

[0314] The third index layer is constructed, including: setting one or more groups of buckets in the third index layer. Corresponding to one group of buckets, one group of buckets is used to store the subfield B1, and the group of buckets includes a plurality of buckets. Each bucket in each group of buckets represents a value of the subfield B1. Each bucket includes a plurality of index items, and each index item represents a value of the subfield B2.

[0315] In combination with the valid index items and the invalid index items, when the second index layer in the two-layer structure and the third index layer in the three-layer structure are constructed, the construction manner is similar to the construction manner of the bottom layer index layer. For details, refer to the foregoing description, which will not be repeated here.

[0316] 2. Specific structure of the fingerprint index.

[0317] As shown in FIG. 10A, it is a structure schematic diagram of a fingerprint index provided by an embodiment of the present application.

[0318] The fingerprint index is a multi-layer structure, and the fingerprint index includes a plurality of index layers. Each index layer corresponds to a field in the fingerprint, and each index layer is used to store the corresponding field.

[0319] Except for the bottom layer index layer, each index layer includes a plurality of index items, and each index item of the index layer represents a value of the field corresponding to the index layer. Each index item of each index layer corresponds to a group of index items in the next layer index layer.

[0320] Each group of buckets in the bottommost index layer corresponds to a first subfield in a target field, the target field being a field corresponding to the bottommost index layer. Each group of buckets corresponds to an index item in an index layer above it. Each group of buckets is used to store the first subfield, each group of buckets covering all possible values of the first subfield, one bucket in each group of buckets representing one value of the first subfield. Each bucket is used to store a second subfield, the index items in each bucket covering all possible values of the second subfield in the target field, each index item representing one possible value of the second subfield.

[0321] Each index item in the bottommost index layer includes fingerprint-related information of a fingerprint represented by the index item.

[0322] In combination with the description of the first subfield covering all possible values of the field in each fingerprint in the first set of fingerprints, the index item representing one value of the corresponding field, the valid index item, and the invalid index item, since the bottommost index layer is provided with one or more groups of buckets, each group of buckets including one or more buckets, and each bucket including one or more index items. One index item in an index layer above the bottommost index layer corresponds to a group of buckets in the bottommost index layer. The storage address of the corresponding group of buckets in the bottommost index layer is recorded in the index item in the index layer above the bottommost index layer.

[0323] If the covering mode one is used, the structure of the bottommost index layer in the fingerprint index is as follows:

[0324] Each group of buckets in the bottommost index layer includes a plurality of buckets, the number of buckets in each group of buckets being equal to a third value, the third value being equal to the number of possible values of the first subfield, the third value not being limited to the number of possible values of the first subfield when the first set of fingerprints is not limited. The position of each bucket in the plurality of buckets represents a possible value of the first subfield. In other words, the possible values of the first subfield are sequentially sorted according to the size of the value, and the sorting position of a specific value is the position of the bucket representing the specific value in the plurality of buckets.

[0325] For all possible values of the first subfield in each fingerprint in the first set of fingerprints, a plurality of index items are provided in the bucket representing the value. The bucket is a valid bucket.

[0326] The multiple index items in the valid bucket are arranged in a first manner: the bucket includes multiple index items, and the number of index items is equal to a fourth value, which is equal to the number of possible values of the second subfield. The fourth value is the number of possible values of the second subfield when the first fingerprint set is not limited. The position of each index item in the multiple index items represents a possible value of the second subfield. In other words, the possible values of the second subfield are sequentially sorted according to the size of the value, and the sorting position of a specific value is the position of the index item representing the specific value in the multiple index items.

[0327] For all possible values of the second subfield in each fingerprint in the first fingerprint set, the index item representing the value is a valid index item. The index item includes a valid identifier and / or related information of the fingerprint.

[0328] For a value of the second subfield that does not exist in each fingerprint in the first fingerprint set, the index item representing the value does not record any storage address, or the index item representing the value records an invalid identifier, to indicate that the value does not exist or that the index item is invalid. The index item is an invalid index item.

[0329] If the multiple index items in the valid bucket are arranged in a second manner: the valid bucket includes multiple valid index items, and each valid index item includes a represented value, and optionally, a valid identifier and / or related information of the fingerprint

[0330] If the second coverage manner is used, the structure of the bottommost index layer in the fingerprint index is as follows:

[0331] The structure of the bottommost index layer in the fingerprint index using the second coverage manner is similar to the structure of the bottommost index layer in the fingerprint index using the first coverage manner, and the difference is that in the bottommost index layer using the second coverage manner, there is no concept of invalid bucket, and each bucket includes multiple index items.

[0332] Each index item can represent a fingerprint. If the fingerprint represented by the index item does not exist in the first fingerprint set, the index item is an invalid index item. If the fingerprint index is applicable to a scenario of querying fingerprint-related information, that is, it is required that the fingerprint index item records the related information of the fingerprint, the invalid index item does not include the related information of the fingerprint, and further, can include an invalid identifier to indicate that the represented fingerprint does not exist. If the fingerprint index is applicable to a scenario of determining whether to store data or whether a data set exists data, the invalid index item includes an invalid identifier to indicate that the represented fingerprint does not exist.

[0333] For the multiple index items in each bucket in the bottommost index layer, the arrangement manner of the index items is similar to the arrangement manner of the index items in the bottommost index layer of the fingerprint index using the first coverage manner.

[0334] From the fingerprint index construction process and the specific structure of the fingerprint index, it can be seen that the parameters affecting the specification of the fingerprint index also include the bucket size, the number of buckets. The number of buckets and the number of index items in the bucket are related to the division method of the target field. For example, the length of the first subfield formed after the division of the target field is relatively long, so the value category of the first subfield is also relatively large, and correspondingly, the length of the second subfield is relatively short, so the value category of the second subfield is also relatively small. Under this division method, the number of buckets is relatively large, the number of index items in the bucket is relatively small, and the bucket size can be pre-set.

[0335] Next, taking the fingerprint index including a double-layer structure and a three-layer structure as an example, the structure of the fingerprint index is further introduced, so as to more clearly understand the structure of the fingerprint index.

[0336] The fingerprints in the first fingerprint set include two fields, field A and field B.

[0337] As shown in FIG. 10B, the fingerprint index includes two index layers, which are referred to as the first index layer and the second index layer. The fields corresponding to the first index layer and the second index layer and the meanings represented by the index items therein can be referred to the related description of FIG. 2A, which will not be repeated here.

[0338] Different from the fingerprint index shown in FIG. 2A, in the fingerprint index shown in FIG. 10B, the field B corresponding to the second index layer is further split into two subfields, subfield B1 and subfield B2. The second index layer includes multiple groups of buckets. One index item in the first index layer corresponds to one group of buckets in the second index layer. Since the first index layer includes 10 index items, the second index layer includes 10 groups of buckets.

[0339] Corresponding to one group of buckets, one group of buckets is used to store the subfield B1, and one group of buckets includes multiple buckets. Each bucket in each group of buckets represents one value of the subfield B1. Therefore, the number of buckets in each group of buckets is the same as the number of possible values of the subfield B1. Each bucket includes multiple index items, and each index item represents one value of the subfield B2. Therefore, the number of index items in each bucket is the same as the number of possible values of the subfield B2.

[0340] The fingerprints in the first fingerprint set include three fields, field A, field B, and field C.

[0341] As shown in FIG. 10C, the fingerprint index includes three index layers, which are referred to as the first index layer, the second index layer, and the third index layer. The fields corresponding to the first index layer, the second index layer, and the third index layer and the meanings represented by the index items therein can be referred to the related description of FIG. 2B, which will not be repeated here.

[0342] The fingerprint index shown in FIG. 2A is different from the fingerprint index shown in FIG. 10C. In the fingerprint index shown in FIG. 10C, the field C corresponding to the third index layer is further split into two subfields, subfield C1 and subfield C2. The third index layer includes multiple groups of buckets. One index item in the second index layer corresponds to one group of buckets in the third index layer. Since the second index layer includes 64 index items, the second index layer includes 64 groups of buckets.

[0343] Corresponding to one group of buckets, one group of buckets is used to store the subfield C1, and the group of buckets includes multiple buckets. Each bucket in each group of buckets represents one value of the subfield C1. Therefore, the number of buckets in each group of buckets is the same as the number of possible values of the subfield C1. Each bucket includes multiple index items, and each index item represents one value of the subfield C2. Therefore, the number of index items in each bucket is the same as the number of possible values of the subfield C2.

[0344] In combination with the valid index item and the invalid index item, the structure of the second index layer in the double-layer structure and the third index layer in the triple-layer structure is similar to that of the bottommost index layer, which can be referred to the foregoing description and will not be described here.

[0345] 3. A query method based on the fingerprint index.

[0346] The query method based on the fingerprint index is described by taking the structure of the fingerprint index shown in FIG. 10A as an example. As shown in FIG. 11, the structure of the fingerprint index provided in the embodiment of the present application is shown.

[0347] In step 1100, the query device acquires a target fingerprint to be queried and determines the values of the fields in the target fingerprint.

[0348] In step 1101, the query device starts from the topmost index layer of the fingerprint index and performs a first search operation in each index layer until the last but one index layer of the bottommost index layer. The first search operation is described in the related description of step 601, which will not be described here.

[0349] In step 1102, the query device performs a second search operation in the last but one index layer of the bottommost index layer: searching for an index item representing the value of a second target field in the target fingerprint in the search range of the index layer, the second target field being the field corresponding to the index layer, and if the index item is found, jumping to a group of buckets in the bottommost index layer corresponding to the index item. If the index item is not found, the query method based on the fingerprint index is terminated.

[0350] The index item representing the value of the second target field in the target fingerprint records the storage address of a group of buckets in the corresponding bottommost index layer, and the query device can jump to the group of buckets in the corresponding bottommost index layer based on the storage address.

[0351] Step 1103: The query device performs a third search operation in the bottommost index layer: searching for a target bucket representing the value of the first subfield in the target fingerprint in the search range of the index layer, and searching for a target index item representing the value of the second subfield in the target fingerprint in the target bucket. The search range of the index layer is the group of buckets jumped to in step 1102.

[0352] The query device can obtain the starting address BP of the group of buckets, and calculate the starting address of the bucket representing the value of the first subfield in the target fingerprint according to the value of the first subfield. The ranking position of the value of the first subfield in the target fingerprint among the various possible values of the first subfield is BS. That is, there are BS-1 values before the value of the first subfield in the target fingerprint among the various possible values of the first subfield, and there are S-1 buckets before the bucket representing the value. Therefore, the starting address bp of the target bucket representing the value satisfies: bp=BP+(BS-1)*d, where d is the size of the bucket. In other words, the starting address of the target bucket representing the value is the starting address BP of the group of buckets in the fingerprint bucket plus an offset, and the offset is equal to the total size of the buckets before the bucket representing the value, that is, the offset is (BS-1)*d.

[0353] After determining the target bucket, the query device searches for a target index item representing the value of the second subfield in the target fingerprint from the target bucket. If the multiple index items in the target bucket are arranged in mode one, the query device searches for a target index item representing the value of the second subfield in the target fingerprint in the multiple index items in a manner similar to the execution manner of step 701, and specific details can be referred to the related description of step 701, which will not be repeated here.

[0354] If the multiple index items in the target bucket are arranged in mode two: the query device can traverse each index item in the target bucket until an effective index item containing the value of the second subfield in the target fingerprint is found. If an effective index item containing the value of the second subfield B in the target fingerprint is not found, it means that the target fingerprint is not queried.

[0355] Step 1104: If a target index item is searched in the target bucket, it means that the target fingerprint is searched in the fingerprint index, and the query device can obtain the fingerprint-related information of the target fingerprint from the target index item. If a target index item is not searched in the target bucket, it means that the target fingerprint is not searched in the fingerprint index.

[0356] The following further introduces the query process based on the fingerprint index by taking the query based on the fingerprint index as shown in FIG. 10C and FIG. 10B as an example, so as to more clearly understand the query method based on the fingerprint index.

[0357] I. The fingerprint index includes a double-layer structure.

[0358] The query method based on the fingerprint index as shown in FIG. 10B is similar to the query method based on the fingerprint index as shown in FIG. 7, and the difference lies in that when the query based on the fingerprint index as shown in FIG. 10B is performed, the query device jumps to a group of buckets of the second index layer corresponding to the first index item after the first index item is determined, and determines a target bucket in the group of buckets, the value represented by the target bucket being consistent with the value of the subfield B1 in the target fingerprint. After the target bucket is determined, a second index item is determined from a plurality of index items in the target bucket, the value represented by the second index item being consistent with the value of the subfield B2 in the target fingerprint.

[0359] II. The fingerprint index includes a three-layer structure.

[0360] The query method based on the fingerprint index as shown in FIG. 10B is similar to the query method based on the fingerprint index as shown in FIG. 8, and the difference lies in that when the query based on the fingerprint index as shown in FIG. 10B is performed, the query device jumps to a group of buckets of the third index layer corresponding to the second index item after the second index item is determined, and determines a target bucket in the group of buckets, the value represented by the target bucket being consistent with the value of the subfield C1 in the target fingerprint. After the target bucket is determined, a third index item is determined from a plurality of index items in the target bucket, the value represented by the third index item being consistent with the value of the subfield C2 in the target fingerprint.

[0361] The third index layer is the bottommost index layer, and the query device determines the third index item in a group of index items of the third index layer corresponding to the second index item in a manner similar to the execution manner of step 1103. For details, refer to the foregoing description, which will not be repeated here.

[0362] The following lists two deployment manners of the fingerprint index:

[0363] Deployment manner I: The fingerprint index is deployed in a distributed storage system. The fingerprint index is stored in a data management node in the distributed storage system.

[0364] As shown in FIG. 12A, a system architecture diagram of a distributed storage system provided by an embodiment of the present application is shown, which includes a server cluster. The server cluster includes one or more servers 130 (three servers 130 are shown in FIG. 12A, but the number of servers 130 is not limited to two). The servers 130 can communicate with each other. The server 130 is a device with both computing and storage capabilities, such as a server, a desktop computer, etc. One server is one node in the distributed storage system.

[0365] In hardware, as shown in FIG. 12A, the server 130 includes at least a processor 132, a memory 133, a network card 134, and a hard disk 135. The processor 132, the memory 133, the network card 134, and the hard disk 135 are connected through a bus.

[0366] The network card 134 can be used to communicate with other servers 130. The network card 134 can also be used to communicate with the hard disk 135 to expand the capacity of the server 130.

[0367] The processor 132 can be a central processing unit (CPU), or other specific integrated circuits. The processor 132 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc.

[0368] The memory 133 refers to an internal memory directly exchanging data with the processor 132, and can be a dynamic random access memory (DRAM). In addition to the DRAM, the memory 133 can also be other random access memories, such as a static random access memory (SRAM), etc. In addition, the memory 133 can also be a read only memory (ROM). For the read only memory, for example, it can be a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), etc. The memory 133 can also be other memories such as a FLASH memory, etc. The memory 133 stores computer program instructions required by the processor 132 to perform data operations.

[0369] The hard disk 135 is used to provide storage resources, such as storing data. It can be a magnetic disk or other types of storage media, such as a solid state disk or a shingled magnetic recording hard disk, etc.

[0370] The fingerprint index deployed in the distributed storage system shown in FIG. 12A can be a "global" fingerprint index, which records the fingerprints of data in all servers (such as the hard disks of the servers) in the storage system. Although not shown, the storage system can also include a server for managing data, which has a structure similar to that of the server 130, and can be the foregoing, which will not be described here. The server can serve as a data management node in the distributed storage system, and the global fingerprint index is stored in the server.

[0371] For example, the fingerprint-related information included in the index item of the global fingerprint index is the node where the data is located. In the distributed storage system, the node where a certain data is located needs to be queried, and the server storing the global fingerprint index can query the global fingerprint index using the fingerprint of the data. If the target index item is searched in the lowest index layer in the global fingerprint index, the node where the data is located is obtained from the target index item. If the target index item is not searched in the lowest index layer in the global fingerprint index, it means that the data is not stored in the distributed storage system.

[0372] In the foregoing description, the server for managing data is an independent server, in actual application, one or more servers 130 in the distributed storage system can undertake the data management function, that is, the global fingerprint index can be shared by the one or more servers 130, and the one or more servers 130 can access the global fingerprint index. For example, a shared storage space (which can be located in the memory of the server 130) is respectively deployed in the one or more servers 130, the global fingerprint index is stored in the shared storage space, and the one or more servers 130 can access the shared storage space and read the global fingerprint index. For another example, the global fingerprint index is distributed in the one or more servers 130, the global fingerprint index includes a plurality of fingerprint indexes, one server 130 in the one or more servers 130 stores one fingerprint index, and the structure of each fingerprint index is shown in FIG. 1 or FIG. 10A. Any server 130 can query the fingerprint index and determine the node where the data corresponding to the fingerprint to be queried is located.

[0373] It should be noted that in the foregoing description, the global fingerprint index records the fingerprints of all data in the servers (such as the hard disks of the servers) in the storage system. In actual application, the global fingerprint index can only record the fingerprints of part of the data in the servers in the storage system. That is, when the global fingerprint index is constructed, the fingerprints of the data in the storage system are sampled, only part of the fingerprints of the data are extracted, and the global fingerprint index is constructed only for the part of the fingerprints of the data. By means of sampling, the number of fingerprints recorded in the global fingerprint index can be reduced, and the storage space occupied by the global fingerprint index can be reduced.

[0374] The second deployment manner is that the fingerprint index is deployed in the distributed storage system. The fingerprint index is stored in each node in the distributed storage system.

[0375] As shown in FIG. 12B, it is a system architecture schematic diagram of a distributed storage system provided by an embodiment of the present application, in which the computing function and the storage function required by the distributed storage system are deployed on different nodes, wherein the node with the computing function is referred to as a computing node, and the node with the storage function is referred to as a storage node.

[0376] In FIG. 12B, the distributed storage system includes a computing node cluster and a storage node cluster. The computing node cluster includes one or more computing nodes 140 (two computing nodes 140 are shown in FIG. 12B, but the number of computing nodes 140 is not limited to two), and the computing nodes 140 can communicate with each other. The computing node 140 is a computing device, such as a server, a desktop computer, or a controller of a storage array.

[0377] The computing node 140 is provided with a computing function, which is capable of receiving a data request sent from outside the storage system, and processing the data request. If the data request is used to request access to data or to request processing of data, the computing node 140 can send a data access request to the storage node 150 to request access to the data from the storage node or to obtain the data from the storage node.

[0378] In hardware, as shown in FIG. 12B, the computing node 140 at least includes a processor 142, a memory 143 and a network card 144. The specific form of the processor 142 is similar to that of the processor 132, and specific reference can be made to the foregoing description, which will not be repeated here. The processor 142 is used to process data requests and trigger the network card 144 to send data access requests to the storage node.

[0379] The memory 143 is used to store computer program instructions required by the processor 142 to process data requests, and can also temporarily store data. When the data stored in the memory 143 reaches a certain threshold, the computing node 140 can send a data access request to the storage node 150 through the network card 144 to perform persistent storage of the data. The network card 144 is used to communicate with the storage node 150.

[0380] Any one computing node 140 can access any one storage node 150 in the storage node cluster through a network. The storage node cluster includes a plurality of storage nodes 150 (three storage nodes 150 are shown in FIG. 12B, but are not limited to three storage nodes 150). One storage node 150 includes one or more control units 151, a network card 154 and a plurality of hard disks 155. The network card 154 is used to communicate with the computing node 140. The hard disk 155 is used to store data, which can be a magnetic disk or other types of storage media, such as a solid state disk or a shingled magnetic recording hard disk, etc.

[0381] The functions of the control unit 151 (equivalent to the processor in the storage node 150), the network card 154 and the plurality of hard disks 155 are similar to those of the processor 142, the network card 144 and the hard disk 135 mentioned above, and specific reference can be made to the foregoing description, which will not be repeated here.

[0382] In the distributed storage system shown in FIG. 12B, a fingerprint index can be deployed in each storage node, which records the fingerprints of the data in the storage node (such as the hard disk of the storage node).

[0383] For example, the fingerprint-related information included in the index item of the fingerprint index in each storage node is the storage address of the data. When a certain storage node needs to query a certain data, the fingerprint index can be queried by using the fingerprint of the data. If the target index item is searched in the bottommost index layer of the fingerprint index, the storage address of the data is obtained from the target index item, and then the data is obtained. If the target index item is not searched in the bottommost index layer of the fingerprint index, it indicates that the data is not stored in the storage node.

[0384] For example, the fingerprint-related information included in the index item of the fingerprint index in each storage node is the storage address of the data. When a certain storage node needs to query a certain data, the fingerprint index can be queried by using the fingerprint of the data. If the target index item is searched in the bottommost index layer of the fingerprint index, the storage address of the data is obtained from the target index item, and then the data is obtained. If the target index item is not searched in the bottommost index layer of the fingerprint index, it indicates that the data is not stored in the storage node.

[0385] For a data set, the larger the data set (that is, the larger the number of data shards contained therein), the more fingerprints of the data shards will be. If all the data fingerprints in the data set are recorded in the fingerprint index, the fingerprint index will be relatively large, and storing the fingerprint index will occupy a large storage space.

[0386] In order to reduce the fingerprint storage space occupied by the fingerprint index, the fingerprint index can be saved in a fingerprint sampling manner. That is, the index item (that is, the index item of the bottommost index layer) included in the fingerprint index only needs to represent the fingerprints of part of the data shards in the data set. In addition, the fingerprint index can be saved in a partitioned manner. That is, the fingerprint index is distributed and stored in multiple partitions. Each partition can be understood as a piece of storage space, and the multiple partitions can be located in the same storage device or different storage devices. The storage device can be a memory or a computing device with storage function.

[0387] At present, the sampling saving manner is commonly used to save a fingerprint table. The fingerprint table includes a plurality of key-value pairs. The key of each key-value pair is a fingerprint, and the value corresponding to the key can be the data corresponding to the fingerprint or the storage address of the data. In the sampling saving manner, the modulo 0 method can be used to determine which key-value pair is retained in the fingerprint table. For example, the modulus is 5, and the key-value pair with the fingerprint value being a multiple of 5 is retained in the fingerprint table. That is, the key-value pairs with the fingerprint values of 0, 5, 10, 15, 20, 25, 30, 35, … can be saved. These key-value pairs can be distributed in multiple partitions.

[0388] This sampling preservation method has some disadvantages. If the sampling ratio of the sampled and preserved fingerprint table needs to be changed (i.e. the modulus is changed), the key-value pairs in the fingerprint table will fluctuate greatly, and new key-value pairs may need to be added to the fingerprint table, and some existing key-value pairs may need to be deleted.

[0389] For example, if the modulus of the sampled and preserved fingerprint table is 5, the key-value pairs with the fingerprint values of 0, 5, 10, 15, 20, 25, 30, 35, … are preserved in the current sampled and preserved fingerprint table.

[0390] If the sampling ratio of the fingerprint table is changed at this time, and the modulus is changed to 7, the key-value pairs with the fingerprint values of 0, 7, 14, 21, 28, 35, 42, 49, … need to be preserved in the fingerprint table.

[0391] As can be seen, only a small part of the key-value pairs preserved in the fingerprint table before and after the change of the sampling ratio are the same. Most of the key-value pairs are different, and most of the key-value pairs stored in the fingerprint table need to be deleted when the sampling ratio is changed.

[0392] If the fingerprint table is stored in partitions, most of the key-value pairs included in the partitions distributed in the fingerprint table need to be adjusted when the sampling ratio is changed.

[0393] Taking the change of the sampling ratio from the modulus 5 to the modulus 7 as an example, and assuming that the fingerprint table is distributed in 8 partitions, the serial numbers of the key-value pairs stored in each partition before and after the change of the sampling ratio are shown in Table 1.

[0394] Table 1

[0395] As can be seen from Table 1, the change of the sampling ratio will cause the key-value pairs recorded in the 7 partitions to change, and the deletion rate will be lost.

[0396] In the embodiments of the present application, a sampling preservation method of a fingerprint index is provided. In the sampling preservation method, the sampling ratio can be reflected by a fingerprint value range, i.e. the fingerprints of the data shards with the fingerprint values in the fingerprint value range are extracted from the fingerprints of all the current data shards, and the proportion of the extracted fingerprints of the data shards with the fingerprint values in the fingerprint value range in the fingerprints of all the current data shards is equal to the sampling ratio. The method comprises the following steps:

[0397] Step 1, obtaining a sampling ratio, and determining a value range of the fingerprints. For example, the sampling ratio is 16%, and the fingerprints of the current all data slices are 100, then the value range of the fingerprints can be 0-16. For another example, the sampling ratio is 12%, and the fingerprints of the current all data slices are 100, then the value range of the fingerprints can be 0-12. Assuming that the value range of the fingerprints is: the value of the fingerprints is greater than the first value, and less than the second value. The first value and the second value are the lower limit value and the upper limit value of the fingerprints. The upper limit value and / or the lower limit value of the fingerprints changes with the sampling ratio.

[0398] Step 2, extracting the fingerprints of the data slices whose values are in the value range from the fingerprints of the current all data slices (such as the fingerprints included in the second fingerprint set mentioned above).

[0399] Step 3, constructing a fingerprint index for the extracted fingerprints of the data slices (the extracted fingerprints of the data slices can be the fingerprints included in the first fingerprint set mentioned above).

[0400] Step 4, storing the fingerprint index in multiple partitions.

[0401] Regarding Step 4, the embodiments of the present application do not limit the storage manner of the fingerprint index partitions, and the following two storage manners of the partitions are listed by taking the fingerprint index shown in FIG. 1 as an example:

[0402] Manner one, when storing the fingerprint index in the partitions, the fingerprint index can be split into multiple first fingerprint sub-indices. The structure of each first fingerprint sub-index is similar to that of the fingerprint index, that is, each first fingerprint sub-index is a multi-layer structure, and each first fingerprint sub-index includes multiple index layers. Each index layer is used to store a field in the fingerprints, and each index item in each index layer represents a value of the stored field. An index item in one index layer in the multiple index layers corresponds to a group of index items in the next index layer. One first fingerprint sub-index is stored in each partition.

[0403] As shown in FIG. 13A, it is a schematic diagram of the fingerprint index and the first fingerprint sub-index. The structure of the first fingerprint sub-index is the same as that of the fingerprint index, and the difference is that the fingerprints recorded by the first fingerprint sub-index are only part of the fingerprints recorded in the fingerprint index.

[0404] The embodiments of the present application do not limit the manner of splitting the fingerprint index into multiple first fingerprint sub-indexes. For example, for any first fingerprint sub-index, the first fingerprint sub-index includes the partial index items of the topmost index layer in the fingerprint index. Further, the first fingerprint sub-index also includes the corresponding index items of the partial index items in other index layers (i.e., all index layers except the topmost index layer) in the fingerprint index.

[0405] For another example, for any first fingerprint sub-index, the first fingerprint sub-index includes the partial index items of the bottommost index layer in the fingerprint index. Further, the first fingerprint sub-index also includes the corresponding index items of the partial index items in other index layers (i.e., all index layers except the bottommost index layer) in the fingerprint index.

[0406] In this manner, the first fingerprint sub-index of each partition records part of the first fingerprint set. That is, the index items in the first fingerprint sub-index represent part of the first fingerprint set.

[0407] When initially constructing the first fingerprint sub-indexes, the construction device can first determine the fingerprints that need to be recorded by the first fingerprint sub-index items of each partition. That is, the construction device can split the first fingerprint set into multiple fingerprint sub-sets. The embodiments of the present application do not limit the manner of splitting the first fingerprint set into multiple fingerprint sub-sets by the construction device.

[0408] For example, the construction device splits the first fingerprint set into multiple fingerprint sub-sets based on the fourth field in the first fingerprint set. Each fingerprint sub-set includes fingerprints with the same value of the fourth field. The fingerprints in different fingerprint sub-sets have different values of the fourth field. The first fingerprint index items in each partition are used to record the fingerprints in each fingerprint sub-set.

[0409] The fourth field can be different from, the same as, or partially the same as (e.g., the fourth field has an overlapping character segment with the first field) the first field, the second field, or the third field.

[0410] For another example, the construction device splits the first fingerprint set into multiple fingerprint sub-sets based on the hash values of the fingerprints in the first fingerprint set. Each fingerprint sub-set includes fingerprints with the same hash value. The fingerprints in different fingerprint sub-sets have different hash values.

[0411] After the plurality of fingerprint sub-sets are determined, the constructing device constructs a first fingerprint sub-index for each fingerprint sub-set based on the fingerprint sub-set. The way the constructing device constructs the first fingerprint sub-index based on the fingerprint sub-set is similar to the way the fingerprint index is constructed, except that the fingerprint index is constructed based on the first fingerprint set in the foregoing description, while the fingerprint sub-index is constructed based on the fingerprint sub-set in this case.

[0412] After the first fingerprint sub-indexes are constructed, each first fingerprint sub-index is stored in a partition.

[0413] When the querying device needs to query a target fingerprint, the searching device can first determine the first fingerprint sub-index that is likely to record the target fingerprint, i.e., determine the partition where the first fingerprint sub-index is located. After the partition is determined,

[0414] For example, when the constructing device splits the first fingerprint set into a plurality of fingerprint sub-sets based on the fourth field in the first fingerprint set, the querying device can determine the fingerprint sub-set to which the target fingerprint belongs based on the fourth field of the target fingerprint, and then determine the partition where the first fingerprint sub-index is located.

[0415] When the constructing device splits the first fingerprint set into a plurality of fingerprint sub-sets based on the hash value of each fingerprint in the first fingerprint set, the querying device can determine the fingerprint sub-set to which the target fingerprint belongs based on the hash value of the target fingerprint, and then determine the partition where the first fingerprint sub-index is located.

[0416] Of course, in actual applications, the querying device can also not determine the first fingerprint sub-index that is likely to record the target fingerprint, i.e., not determine the partition where the first fingerprint sub-index is located. The querying device can synchronously query the first fingerprint sub-indexes in each partition.

[0417] In the second way, when the fingerprint index is stored in the partition, the fingerprint index can be split into a plurality of second fingerprint sub-indices according to the multi-layer structure of the fingerprint index, and each second fingerprint sub-index includes part of the index layers in the fingerprint index. Each partition stores a second fingerprint sub-index. That is, each second fingerprint sub-index only contains part of the layers in the fingerprint index.

[0418] As shown in FIG. 13B, it is a schematic diagram of the fingerprint index and the second fingerprint sub-index. Each second fingerprint sub-index only contains part of the layers in the fingerprint index. When two adjacent index layers in the fingerprint index are located in different second fingerprint sub-indices. The second fingerprint sub-index where the upper index layer in the two adjacent index layers is located can point to the second fingerprint sub-index where the lower index layer in the two adjacent index layers is located. Further, any index item in the upper index layer can point to a group of index items in the lower index layer in the second fingerprint sub-index where the lower index layer is located.

[0419] In this way, there is a correspondence between the second fingerprint sub-indexes of each partition (i.e., the correspondence between the index layers).

[0420] When the query device needs to query a target fingerprint, the query device can start querying from the second fingerprint sub-index containing the topmost index layer and jump between the second fingerprint sub-indexes. The query process is similar to the aforementioned way of querying by using the fingerprint index, with the difference being that each index layer can be located in a different partition. For details, refer to the foregoing description, which will not be repeated here.

[0421] For example, if the value range of the fingerprint is 0-16, then in the fingerprint index, the fingerprints with the values 0, 1, 2, 3,..., 16 need to be recorded.

[0422] If the sampling ratio (i.e., the value range of the fingerprint) is to be changed, assuming that the value range of the fingerprint becomes 0-12, i.e., only the upper limit value of the fingerprint is changed, then in the fingerprint index, the fingerprints with the values 0, 1, 2, 3,..., 12 need to be recorded.

[0423] As can be seen, before and after the change of the sampling ratio, the proportion of the same index items in the index items that need to be retained in the fingerprint index is large, i.e., most of the index items are the same, and only a small part of the index items are different. The change of the sampling ratio will not cause a large range of changes to the fingerprints recorded in the fingerprint index.

[0424] If the fingerprint index is stored in a partitioned storage manner, then when the sampling ratio is changed, the index items included in the small part of the partitions distributed by the fingerprint index need to be adjusted.

[0425] The fingerprint index is stored in the partitioned storage manner. Still taking the aforementioned example in which the sampling ratio is changed from the value range (0-16) to the value range (0-12) as an example, assuming that the fingerprint table is distributed in 8 partitions, and the serial numbers of the key-value pairs stored in each partition before and after the change of the sampling ratio are shown in Table 2.

[0426] Table 2

[0427] As can be seen from Table 2, the change of the sampling ratio will cause the index items recorded in the small part of the partitions to change, and the re-deletion rate is almost lossless.

[0428] In the embodiment set of the present application, the fingerprint index can be stored in a partitioned storage manner, i.e., the fingerprint index can be distributed and stored in multiple partitions. In the embodiment of the present application, the partitioned storage fingerprint index supports the expansion and contraction of the number of partitions.

[0429] ①, Expansion of the number of partitions.

[0430] The expansion of the number of partitions refers to increasing the number of partitions of the fingerprint index distribution. The increase in the number of partitions indicates that the storage space that can be occupied by the fingerprint index is increased. When the number of partitions is increased, the new index items that need to be written into the fingerprint index can be stored in the newly added partitions, or part of the existing index items in the fingerprint index are migrated to the newly added partitions.

[0431] When the fingerprint index is stored in the first way, when the number of partitions is increased, the new index items that need to be written into the fingerprint index can be stored in the newly added partitions. The new index items refer to the index items that need to be added in the bottommost index layer and the corresponding index items of the index items added in the bottommost index layer in other index layers. Alternatively, part of the existing index items in the fingerprint index are migrated to the newly added partitions. The part of the existing index items refers to part of the index items in the bottommost index layer of the fingerprint index and the corresponding index items of the part of the index items in other index layers. That is, a first fingerprint sub-index is stored in the added partition.

[0432] When the fingerprint index is stored in the second way, when the number of partitions is increased, the second fingerprint sub-index can be further split into a plurality of third fingerprint sub-indices, and the third fingerprint sub-indices are stored in the added partitions. The embodiment of the present application does not limit the splitting manner of the second fingerprint sub-index. For example, the second fingerprint sub-index can be split in layers as a granularity.

[0433] ②, Reduction of the number of partitions.

[0434] The reduction of the number of partitions refers to reducing the number of partitions of the fingerprint index distribution. The reduction of the number of partitions indicates that the storage space that can be occupied by the fingerprint index is reduced. When the number of partitions needs to be reduced, part of the partitions need to be deleted and part of the partitions need to be retained. The index items stored in the to-be-deleted partitions in the fingerprint index can be migrated to the retained partitions, or the index items in the to-be-deleted partitions are directly deleted. In the embodiment of the present application, the manner of migrating the index items stored in the to-be-deleted partitions in the fingerprint index to the retained partitions is not limited. For example, according to the order in which the index items are written into the fingerprint index, the index items written in the recent period of time can be stored in the retained partitions. That is, the index items written in the to-be-deleted partitions in the period of time are migrated to the retained partitions, and the index items written in the retained partitions before the recent period of time are deleted.

[0435] If the fingerprint index is stored in the first way, when the number of partitions is reduced, the first fingerprint sub-index of one or more partitions to be deleted can be directly deleted. The order of writing the index items in the lowest index layer of each first fingerprint sub-index can be preserved, and the index items written into the lowest index layer of each first fingerprint sub-index in a recent period of time can be preserved. Then, the first fingerprint sub-index in the partition to be deleted is merged into the first fingerprint sub-index of the preserved partition. That is, the number of first fingerprint sub-indexes is reduced.

[0436] If the fingerprint index is stored in the second way, when the number of partitions is reduced, the order of writing the index items in the lowest index layer of the fingerprint index can be preserved, and the index items written into the lowest index layer in a recent period of time can be preserved. Then, the second fingerprint sub-index in the partition to be deleted is merged into the second fingerprint sub-index of the preserved partition. That is, the number of second fingerprint sub-indexes is reduced.

[0437] It is worth noting that in the description of the fingerprint sampling and the partition storage, the fingerprint index provided by the embodiments of the present application is taken as an example for description. In fact, the fingerprint sampling and the partition storage are also applicable to fingerprint indexes or fingerprint tables of other structures, and the implementation principles are similar to the fingerprint sampling and the partition storage when the fingerprint index provided by the embodiments of the present application is taken as an example, which will not be described herein.

[0438] Based on the same inventive concept as the method embodiments, the embodiments of the present application also provide a construction device for performing the method performed by the construction device in the above method embodiments. As shown in FIG. 14, the construction device 1400 includes a first construction module 1401, a second construction module 1402. Optionally, it also includes a third construction module 1403, a sampling module 1404, and a storage module 1405. Specifically, in the construction device 1400, the modules are connected through a communication channel. The fingerprints of the first fingerprint set include a first field and a second field, and the construction device 1400 includes:

[0439] The first construction module 1401 is configured to construct a first index layer of a fingerprint index, and the construction of the first index layer of the fingerprint index includes: setting a plurality of index items in the first index layer, and one index item in the first index layer represents one value of the first field.

[0440] The second construction module 1402 is configured to construct a second index layer of the fingerprint index, and the construction of the second index layer of the fingerprint index includes: setting a plurality of groups of index items in the second index layer, one index item in the first index layer corresponds to one group of index items in the second index layer, and one index item in one group of index items in the second index layer represents one value of the second field.

[0441] As a possible implementation, the second construction module 1402 stores the related information of the fingerprints in the first fingerprint set in the index items of the second index layer, and the related information of the fingerprints includes part or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0442] As a possible implementation, the fingerprint in the first fingerprint index set further includes a third field, and the construction device further includes a third construction module 1403, which constructs a third index layer of the fingerprint index; wherein the third index layer of the fingerprint index is constructed by: setting a plurality of groups of index items in the third index layer, one index item of the second index layer corresponding to one group of index items of the third index layer, and one index item in one group of index items of the third index layer representing one value of the third field; and storing the related information of the fingerprints in the first fingerprint set in the index items of the third index layer, and the related information of the fingerprints includes part or all of the following: the data corresponding to the fingerprint, the storage address of the data, the identifier of the node where the data is located, and the identifier of the region where the data is located.

[0443] As a possible implementation, the third field includes a first subfield and a second subfield, and the third construction module 1403 sets at least one group of buckets in the third index layer, one index item of the second index layer corresponding to one group of buckets of the third index layer, each bucket in one group of buckets including one or more index items, each bucket representing one value of the first subfield, and one index item in each bucket representing one value of the second subfield.

[0444] As a possible implementation, the plurality of index items of the first index layer includes a first index item, and a second index item is included in the group of index items of the second index layer corresponding to the first index item; the second index item stores the related information of a first fingerprint in the first fingerprint set, the value of the first field in the first fingerprint being consistent with the value represented by the first index item, and the value of the second field in the first fingerprint being consistent with the value represented by the second index item.

[0445] As a possible implementation, the plurality of index items of the first index layer includes a third index item, a fourth index item is included in the group of index items of the second index layer corresponding to the third index item, a fifth index item is included in the group of index items of the third index layer corresponding to the fourth index item, and the fifth index item stores the related information of a second fingerprint in the first fingerprint set, the value of the first field in the second fingerprint being consistent with the value represented by the third index item, the value of the second field in the second fingerprint being consistent with the value represented by the fourth index item, and the value of the third field in the second fingerprint being consistent with the value represented by the fifth index item.

[0446] As a possible implementation, the constructing apparatus further comprises a sampling module 1404 configured to sample the fingerprints in the second fingerprint set based on a sampling ratio, the first fingerprint set comprising the sampled fingerprints. The second fingerprint set comprises fingerprints of data in the storage system or the storage node, and the number of fingerprints in the first fingerprint set is less than the number of fingerprints in the second fingerprint set.

[0447] As a possible implementation, the value of the fingerprints in the first fingerprint set is greater than a first value and less than a second value, and the second value is determined according to the sampling ratio.

[0448] As a possible implementation, the value of the fingerprints in the first fingerprint set is greater than a first value and less than a second value, and the first value is determined according to the sampling ratio.

[0449] As a possible implementation, the constructing apparatus further comprises a storage module 1405 configured to store the fingerprint index distribution in N partitions of the storage apparatus, each partition being a piece of storage space, and N being a positive integer.

[0450] As a possible implementation, the storage apparatus comprises N nodes, and each node comprises one partition.

[0451] Based on the same inventive concept as the method embodiments, the embodiments of the present application further provide a querying apparatus configured to query a target fingerprint in a first fingerprint set based on a fingerprint index, the fingerprint index being configured to store the first fingerprint set. The querying apparatus is configured to perform the method performed by the querying apparatus in the above method embodiments. As shown in FIG. 15, the querying apparatus 1500 comprises a first querying module 1501 and a second querying module 1502, and optionally, a third querying module 1503. Specifically, in the querying apparatus 1500, the modules are connected through a communication channel.

[0452] The querying apparatus 1500 is configured to query a target fingerprint in a first fingerprint set based on a fingerprint index, the fingerprint index being configured to store the first fingerprint set, the fingerprint index comprising a first index layer and a second index layer, the first index layer being configured to store a first field of the fingerprints in the first fingerprint set, the second index layer being configured to store a second field of the fingerprints in the first fingerprint set, one index item in the first index layer corresponding to a group of index items in the second index layer, and the querying apparatus 1500 comprising:

[0453] The first querying module 1501 is configured to determine whether there is a first index item in the plurality of index items in the first index layer, the value of the first index item being consistent with the value of the first field of the target fingerprint.

[0454] The second query module 1502 is configured to: if the first index layer includes the first index item, determine whether the second index layer corresponding to the first index item includes a second index item, the second index item representing a value identical to a value of a second field in the target fingerprint.

[0455] As a possible implementation, the second query module 1502 obtains, from the second index item, information related to the target fingerprint, the information related to the target fingerprint including some or all of the following: target data corresponding to the target fingerprint, a storage address of the target data, an identifier of a node where the target data is located, and an identifier of a region where the target data is located.

[0456] As a possible implementation, the fingerprint index further includes a third index layer, the third index layer being configured to store a third field of the fingerprints in the first fingerprint set, one index item in a group of index items in the third index layer representing a value of the third field, and the query device 1500 further includes a third query module 1503, the third query module 1503 being configured to: determine whether the third index layer corresponding to the second index item includes a third index item, the third index item representing a value identical to a value of a third field in the target fingerprint.

[0457] If the third index layer corresponding to the second index item includes the third index item, the second query module 1502 obtains, from the third index item, information related to the target fingerprint, the information related to the target fingerprint including some or all of the following: target data corresponding to the target fingerprint, a storage address of the target data, an identifier of a node where the target data is located, and an identifier of a region where the target data is located.

[0458] As a possible implementation, the third index layer is further configured to store information related to the fingerprints in the first fingerprint set, the information related to the fingerprints including some or all of the following: data corresponding to the fingerprints, a storage address of the data, an identifier of a node where the data is located, and an identifier of a region where the data is located.

[0459] As a possible implementation, the third field includes a first subfield and a second subfield, the third index layer includes at least one group of buckets, one index item in the second index layer corresponding to one group of buckets in the third index layer, each group of buckets including one or more buckets, and each bucket including one or more index items, each bucket representing a value of the first subfield, and each index item in each bucket representing a value of the second subfield, the third query module 1503 being configured to: determine a target bucket in the third index layer corresponding to the second index item, the target bucket representing a value identical to a value of the first subfield in the target fingerprint; and determine the third index item in the target bucket, the third index item representing a value identical to a value of the second subfield in the target fingerprint.

[0460] As a possible implementation, if the first index item does not exist in the plurality of index items in the first index layer, the first query module 1501 determines that the target fingerprint does not exist in the first fingerprint set.

[0461] If the second index item does not exist in the group of index items corresponding to the second index layer of the first index item, the second query module 1502 determines that the target fingerprint does not exist in the first fingerprint set.

[0462] If the third index item does not exist in the group of index items corresponding to the second index layer of the second index item, the third query module 1503 determines that the target fingerprint does not exist in the first fingerprint set.

[0463] As a possible implementation, the first fingerprint set includes fingerprints obtained by sampling from the second fingerprint set based on a sampling ratio, the second fingerprint set includes fingerprints of data in the storage system or the storage node, and the number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

[0464] As a possible implementation, the fingerprint index is distributed in N partitions of the storage device, each partition is a piece of storage space, and N is a positive integer.

[0465] As a possible implementation, the storage device includes N nodes, and each node includes a partition.

[0466] The division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, another division manner can be used. In addition, the functional modules in each embodiment of the present application can be integrated in one processor, or can be physically separated, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0467] When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing an end device (which can be a personal computer, a mobile phone, or a network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage program codes.

[0468] The present application also provides a computing device 1600 as shown in FIG. 16. The computing device 1600 includes a bus 1601, a processor 1602, a communication interface 1603, and a memory 1604. The processor 1602, the memory 1604, and the communication interface 1603 communicate with each other through the bus 1601.

[0469] The processor 1602 can be a CPU, and can also be other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0470] The memory 1604 can be DRAM. In addition to DRAM, the memory 1604 can also be other random access memories such as SRAM, etc. In addition, the memory 1604 can also be ROM. For read-only memories, for example, can be PROM, EPROM, etc. The memory 1604 can also be flash memory media, HDD or SSD, etc.

[0471] The memory 1604 stores computer program instructions, and the processor 1602 executes the computer program instructions to perform the steps performed by the construction device in the methods described in the foregoing FIG. 3, FIG. 4, FIG. 5, FIG. 9, FIG. 10A, FIG. 10B, FIG. 10C. The memory 1604 can also include software modules required by other running processes such as the plurality of modules in the construction device 1400. The operating system can be LINUX TM , UNIX TM , WINDOWS TM , etc.

[0472] Alternatively, the memory 1604 stores computer program instructions, and the processor 1602 executes the computer program instructions to perform the steps performed by the query device in the methods described in the foregoing FIG. 6, FIG. 7, FIG. 8, FIG. 11. The memory 1604 can also include software modules required by other running processes such as the plurality of modules in the query device 1500. The operating system can be LINUX TM , UNIX TM , WINDOWS TM , etc.

[0473] The present application also provides a computing device system, which comprises at least one computing device 1700 as shown in FIG. 17. The computing device 1700 comprises a bus 1701, a processor 1702, a communication interface 1703 and a memory 1704. The processor 1702, the memory 1704 and the communication interface 1703 communicate through the bus 1701. The at least one computing device 1700 in the computing device system communicates through a communication channel.

[0474] The specific types of the processor 1702 and the memory 1704 can refer to the related descriptions of the processor 1602 and the memory 1604, which will not be repeated here. The processor 1702 executes the computer program instructions stored in the memory 1704 to perform part or all of the steps performed by the construction device in the methods described in FIG. 3, FIG. 4, FIG. 5, FIG. 9, FIG. 10A, FIG. 10B, FIG. 10C. The memory can also include software modules required by other running processes such as operating system. The operating system can be LINUX TM , UNIX TM , WINDOWS TM , etc.

[0475] The at least one computing device 1700 in the computing device system communicates with each other through a communication network, and any one or more modules in the construction device 1400 run on each computing device 1700.

[0476] Alternatively, the processor 1702 executes the computer program instructions stored in the memory 1704 to perform part or all of the steps performed by the construction device in the methods described in FIG. 6, FIG. 7, FIG. 8, FIG. 11. The memory can also include software modules required by other running processes such as operating system. The operating system can be LINUX TM , UNIX TM , WINDOWS TM , etc.

[0477] The at least one computing device 1700 in the computing device system communicates with each other through a communication network, and any one or more modules in the query device 1500 run on each computing device 1700.

[0478] Alternatively, the processor 1702 executes the computer program instructions stored in the memory 1704 to perform part or all of the steps performed by the query device in the methods described in FIG. 6, FIG. 7, FIG. 8, FIG. 11. The memory can also include software modules required by other running processes such as operating system. The operating system can be LINUX TM , UNIX TM , WINDOWS TM , etc.

[0479] The descriptions of the processes corresponding to the respective figures above each have their own emphasis, and the parts not described in detail in a certain process can be referred to the relevant descriptions of other processes.

[0480] In the embodiments described above, the entire or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, the entire or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes computer program instructions. When the computer program instructions are loaded and executed on a computer, the entire or part of the processes or functions according to the embodiments of the present application shown in FIG. 3, FIG. 4, FIG. 5, FIG. 9, FIG. 10A, FIG. 10B, FIG. 10C, FIG. 6, FIG. 7, FIG. 8, or FIG. 11 are generated.

[0481] The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatuses. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as an SSD).

[0482] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A memory device, comprising: The storage device is configured to store a fingerprint index, the fingerprint index comprising a first index layer and a second index layer, the first index layer being configured to store a first field of fingerprints in a first fingerprint set, and the second index layer being configured to store a second field of fingerprints in the first fingerprint set; The first index layer comprises a plurality of index entries, one index entry in the first index layer representing a value of the first field; one index entry in the first index layer corresponds to a group of index entries in the second index layer; one index entry in the second index layer represents a value of the second field.

2. The memory device of claim 1, wherein, The second index layer is further configured to store related information of the fingerprints in the first fingerprint set, the related information of the fingerprints comprising some or all of the following: data corresponding to the fingerprints, storage addresses of the data, identities of nodes where the data is stored, and identities of areas where the data is stored.

3. The memory device of claim 1, wherein, The fingerprint index further comprises a third index layer, the third index layer being configured to store a third field of fingerprints in the first fingerprint set and the related information of the fingerprints, one index entry in the second index layer corresponding to a group of index entries in the third index layer; one index entry in the third index layer represents a value of the third field, and the related information of the fingerprints comprises some or all of the following: data corresponding to the fingerprints, storage addresses of the data, identities of nodes where the data is stored, and identities of areas where the data is stored.

4. The memory device of claim 3, wherein, The third field comprises a first subfield and a second subfield, and the third index layer comprises at least one group of buckets, one index entry in the second index layer corresponding to a group of buckets in the third index layer, each bucket in the group of buckets comprising one or more index entries, each bucket representing a value of the first subfield, and one index entry in each bucket representing a value of the second subfield.

5. The memory device of claim 2, wherein, The plurality of index entries in the first index layer comprises a first index entry, a group of index entries in the second index layer corresponding to the first index entry comprises a second index entry, the second index entry storing related information of a first fingerprint in the first fingerprint set, a value of the first field in the first fingerprint being consistent with a value represented by the first index entry, and a value of the second field in the first fingerprint being consistent with a value represented by the second index entry.

6. The memory device of claim 3, wherein, The plurality of index entries in the first index layer comprises a third index entry, a group of index entries in the second index layer corresponding to the third index entry comprises a fourth index entry, a group of index entries in the third index layer corresponding to the fourth index entry comprises a fifth index entry, the fifth index entry storing related information of a second fingerprint in the first fingerprint set, a value of the first field in the second fingerprint being consistent with a value represented by the third index entry, a value of the second field in the second fingerprint being consistent with a value represented by the fourth index entry, and a value of the third field in the second fingerprint being consistent with a value represented by the fifth index entry.

7. The memory device of any one of claims 1-6, wherein, The first fingerprint set includes fingerprints sampled from a second fingerprint set based on a sampling ratio, the second fingerprint set including fingerprints of data in the storage system or storage nodes, and the number of fingerprints included in the first fingerprint set is less than the number of fingerprints included in the second fingerprint set.

8. The memory device of claim 7, wherein, The value of the fingerprints in the first fingerprint set is greater than a first value and less than a second value, and the second value is determined according to the sampling ratio.

9. The memory device of any one of claims 1-8, wherein, The fingerprint index is distributed in N partitions of the storage device, each partition being a storage space, and N being a positive integer.

10. The memory device of claim 8, wherein, The storage device includes N nodes, and each node includes one of the partitions.

11. A fingerprint index construction method, characterized in that, The method is used for constructing a fingerprint index for a first fingerprint set, the fingerprints of the first fingerprint set including a first field and a second field, and the method includes: constructing a first index layer of the fingerprint index, and constructing a second index layer of the fingerprint index, wherein: the first index layer includes a plurality of index items, and one index item in the first index layer represents a value of the first field; the second index layer includes a plurality of groups of index items, one index item in the first index layer corresponds to one group of index items in the second index layer, and one index item in one group of index items in the second index layer represents a value of the second field.

12. The method of claim 11, wherein, The method further includes: storing related information of the fingerprints in the first fingerprint set in the index items of the second index layer, the related information of the fingerprints including some or all of the following: data corresponding to the fingerprints, storage addresses of the data, identifiers of nodes where the data is located, and identifiers of regions where the data is located.

13. The method of claim 11, wherein, The fingerprints in the first fingerprint set further include a third field, and the method further includes: constructing a third index layer of the fingerprint index; wherein the third index layer includes a plurality of groups of index items, one index item in the second index layer corresponds to one group of index items in the third index layer, and one index item in one group of index items in the third index layer represents a value of the third field; and storing related information of the fingerprints in the first fingerprint set in the index items of the third index layer, the related information of the fingerprints including some or all of the following: data corresponding to the fingerprints, storage addresses of the data, identifiers of nodes where the data is located, and identifiers of regions where the data is located.

14. The method of claim 13, wherein, The third field includes a first subfield and a second subfield, and the plurality of groups of index items in the third index layer includes: at least one group of buckets in the third index layer, one index item in the second index layer corresponds to one group of buckets in the third index layer, each bucket in the group of buckets includes one or more index items, each bucket represents a value of the first subfield, and one index item in each bucket represents a value of the second subfield.

15. A method of fingerprint searching, the method comprising: The method is used for querying a target fingerprint in a first fingerprint set based on a fingerprint index, the fingerprint index is used for storing the first fingerprint set, the fingerprint index comprises a first index layer and a second index layer, the first index layer is used for storing a first field of a fingerprint in the first fingerprint set, the second index layer is used for storing a second field of the fingerprint in the first fingerprint set, one index item of the first index layer corresponds to a group of index items of the second index layer, and the method comprises the following steps: determining whether there is a first index item in a plurality of index items in the first index layer, the value of the first index item is consistent with the value of the first field of the target fingerprint if the first index item exists in the plurality of index items in the first index layer, determining whether there is a second index item in a group of index items of the second index layer corresponding to the first index item, the value represented by the second index item is consistent with the value of the second field in the target fingerprint.

16. The method of claim 15, wherein, The method further comprises: obtaining the related information of the target fingerprint from the second index item, the related information of the target fingerprint comprises part or all of the following: target data corresponding to the target fingerprint, a storage address of the target data, an identifier of a node where the target data is located, and an identifier of a region where the target data is located.

17. The method of claim 15, wherein, The fingerprint index further comprises a third index layer, the third index layer is used for storing a third field of a fingerprint in the first fingerprint set, one index item in a group of index items of the third index layer represents a value of the third field, and the method further comprises the following steps: determining whether there is a third index item in a group of index items in the third layer index corresponding to the second index item, the value represented by the third index item is consistent with the value of the third field in the target fingerprint; if the third index item exists in the group of index items in the third layer index corresponding to the second index item, obtaining the related information of the target fingerprint from the third index item, the related information of the target fingerprint comprises part or all of the following: target data corresponding to the target fingerprint, a storage address of the target data, an identifier of a node where the target data is located, and an identifier of a region where the target data is located.

18. The method of claim 17, wherein, The third field comprises a first subfield and a second subfield, the third index layer comprises at least one group of buckets, one index item of the second index layer corresponds to a group of buckets of the third index layer, each group of buckets comprises one or more buckets, each bucket comprises one or more index items, each bucket represents a value of the first subfield, and one index item in each bucket represents a value of the second subfield, and the determination of the third index item in the group of index items in the third layer index corresponding to the second index item comprises the following steps: determining a target bucket in a group of buckets in the third layer index corresponding to the second index item, the value represented by the target bucket is consistent with the value of the first subfield in the target fingerprint; determining a third index item in the target bucket, the value represented by the third index item is consistent with the value of the second subfield in the target fingerprint.

19. A construction device, characterized in that The constructing device is used for constructing a fingerprint index of a first fingerprint set, a fingerprint of the first fingerprint set comprising a first field and a second field, and the constructing device comprises: a first constructing module, configured to construct a first index layer of the fingerprint index, wherein the first index layer of the fingerprint index is constructed by: setting a plurality of index items in the first index layer, and one index item in the first index layer representing one value of the first field; a second constructing module, configured to construct a second index layer of the fingerprint index, wherein the second index layer of the fingerprint index is constructed by: setting a plurality of groups of index items in the second index layer, one index item in the first index layer corresponding to one group of index items in the second index layer, and one index item in one group of index items in the second index layer representing one value of the second field.

20. The apparatus of claim 19, wherein, The second constructing module is further configured to: store, in the index items of the second index layer, related information of the fingerprints in the first fingerprint set, the related information of the fingerprints comprising some or all of the following: corresponding data of the fingerprints, storage addresses of the data, identities of nodes where the data is located, and identities of areas where the data is located.

21. The apparatus of claim 19, wherein, The fingerprints in the first fingerprint index set further comprise a third field, and the constructing device further comprises a third constructing module, the third constructing module being configured to: construct a third index layer of the fingerprint index, wherein the third index layer of the fingerprint index is constructed by: setting a plurality of groups of index items in the third index layer, one index item in the second index layer corresponding to one group of index items in the third index layer, and one index item in one group of index items in the third index layer representing one value of the third field; and storing, in the index items of the third index layer, related information of the fingerprints in the first fingerprint set, the related information of the fingerprints comprising some or all of the following: corresponding data of the fingerprints, storage addresses of the data, identities of nodes where the data is located, and identities of areas where the data is located.

22. The apparatus of claim 21, wherein, The third field comprises a first subfield and a second subfield, and the third constructing module is configured to: set at least one group of buckets in the third index layer, one index item in the second index layer corresponding to one group of buckets in the third index layer, each bucket in the group of buckets comprising one or more index items, each bucket representing one value of the first subfield, and one index item in each bucket representing one value of the second subfield.

23. A query device, characterized in that The querying device is used for querying a target fingerprint in a first fingerprint set based on a fingerprint index, the fingerprint index being used for storing the first fingerprint set, the fingerprint index comprising a first index layer and a second index layer, the first index layer being used for storing a first field of a fingerprint in the first fingerprint set, the second index layer being used for storing a second field of the fingerprint in the first fingerprint set, and one index item in the first index layer corresponding to one group of index items in the second index layer, and the querying device comprises: The first query module is configured to determine whether a first index item exists in the plurality of index items in the first index layer, the first index item having a value consistent with a value of the first field of the target fingerprint The second query module is configured to, if the first index item exists in the plurality of index items in the first index layer, determine whether a second index item exists in a group of index items of the second index layer corresponding to the first index item, the second index item representing a value consistent with a value of the second field of the target fingerprint.

24. The apparatus of claim 23, wherein, The second query module is further configured to: obtain, from the second index item, relevant information of the target fingerprint, the relevant information of the target fingerprint including some or all of the following: target data corresponding to the target fingerprint, a storage address of the target data, an identifier of a node where the target data is located, and an identifier of an area where the target data is located.

25. The apparatus of claim 23, wherein, The fingerprint index further includes a third index layer configured to store a third field of a fingerprint in the first set of fingerprints, an index item in a group of index items of the third index layer representing a value of the third field, and the device further includes a third query module configured to: determine whether a third index item exists in a group of index items of the third index layer corresponding to the second index item, the third index item representing a value consistent with a value of the third field of the target fingerprint; if the third index item exists in the group of index items of the third index layer corresponding to the second index item, obtain, from the third index item, relevant information of the target fingerprint, the relevant information of the target fingerprint including some or all of the following: target data corresponding to the target fingerprint, a storage address of the target data, an identifier of a node where the target data is located, and an identifier of an area where the target data is located.

26. The apparatus of claim 25, wherein, The third field includes a first subfield and a second subfield, the third index layer includes at least one group of buckets, an index item of the second index layer corresponding to a group of buckets of the third index layer, each group of buckets including one or more buckets, each bucket including one or more index items, each bucket representing a value of the first subfield, and each index item in each bucket representing a value of the second subfield, and the third query module is configured to: determine a target bucket in a group of buckets of the third index layer corresponding to the second index item, the target bucket representing a value consistent with a value of the first subfield of the target fingerprint; determine a third index item in the target bucket, the third index item representing a value consistent with a value of the second subfield of the target fingerprint.

27. A computing device, comprising: The computing device includes a processor and a memory; The memory is configured to store computer program instructions; The processor executes the computer program instructions in the memory to perform the method of any one of claims 11 to 18.

28. A computer-readable storage medium, characterized in that, The computer readable storage medium is executed by a computing device, and the computing device performs the method of any one of claims 11 to 18.

Images