Fingerprint information storage method, device, equipment, medium, product and filter

Abstract

The invention provides a fingerprint information storage method and device, equipment, a medium, a product and a filter, and relates to the technical field of big data. The method comprises the following steps: determining fingerprint information of a target element, determining a plurality of candidate buckets based on the target element, and respectively determining heat states of the plurality of candidate buckets; and based on the popularity states of the plurality of candidate buckets, determining a target bucket of the fingerprint information, and storing the fingerprint information to the target bucket. According to the method, by sensing the heat state of the candidate bucket in real time and dynamically determining the storage position of the fingerprint information, delay jitter caused by chained kicking-out is remarkably reduced, performance jitter caused by burst traffic is avoided, stability and real-time performance of storage operation are ensured, and stability of a system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a fingerprint information storage method, device, equipment, medium, product and filter. BACKGROUND

[0002] In the fields of finance, e-commerce, Internet of Things and the like which need to process massive data in real time, a cuckoo filter is widely used for quickly judging whether data exists (member query) by virtue of its high space utilization and excellent query performance as a kind of probabilistic data structure.

[0003] However, in the cuckoo filter, when a high-frequency insertion operation triggers recursive kick-out, the length of a conflict chain exceeds a threshold (such as > 10 memory reallocations), resulting in a sudden increase of single insertion delay from microsecond level to millisecond level, and forming response time fluctuation (i.e. delay jitter) with uncertainty. SUMMARY

[0004] The present application provides a fingerprint information storage method, device, equipment, medium, product and filter to solve the technical problem of insertion delay jitter caused by chain kick-out operation in the traditional cuckoo filter under high load.

[0005] In a first aspect, the present application provides a fingerprint information storage method, comprising:

[0006] determining fingerprint information of a target element, the fingerprint information being used for identifying the target element;

[0007] determining a plurality of candidate buckets based on the target element, and respectively determining a hotness state of each of the plurality of candidate buckets, the hotness state being used for quantifying storage pressure of the candidate bucket;

[0008] determining a target bucket of the fingerprint information based on the hotness state of the plurality of candidate buckets, and storing the fingerprint information into the target bucket.

[0009] In a second aspect, the present application provides a fingerprint information storage device, comprising:

[0010] a determining module configured to determine fingerprint information of a target element, the fingerprint information being used for identifying the target element;

[0011] The determining module is further configured to determine a plurality of candidate buckets based on the target element, and respectively determine a hotness state of each of the plurality of candidate buckets, the hotness state being used for quantifying storage pressure of the candidate bucket;

[0012] The determining module is further configured to determine a target bucket of the fingerprint information based on the hotness state of the plurality of candidate buckets.

[0013] The processing module is configured to store the fingerprint information into the target bucket.

[0014] In a third aspect, the present application provides a cuckoo filter, comprising a bucket array, a bucket state module, a flexible bucket controller and a control unit.

[0015] The bucket array is configured to store fingerprint information of elements.

[0016] The bucket state module is configured to collect and store a hotness state of each bucket in real time.

[0017] The flexible bucket controller is configured to dynamically allocate shadow buckets according to the hotness state.

[0018] The control unit is configured to use the fingerprint information storage method of the first aspect and various possible implementation manners of the first aspect to dynamically determine a storage location of the fingerprint information based on the hotness state of each bucket.

[0019] In a fourth aspect, the present application provides an electronic device, comprising:

[0020] A processor and a memory connected with the processor in communication;

[0021] The memory stores computer execution instructions.

[0022] The processor executes the computer execution instructions stored in the memory to implement the fingerprint information storage method of the first aspect and various possible implementation manners of the first aspect.

[0023] In a fifth aspect, the present application provides a computer storage medium having computer execution instructions stored thereon, and the computer execution instructions are executed by a processor to implement the fingerprint information storage method of the first aspect and various possible implementation manners of the first aspect.

[0024] In a sixth aspect, the present application provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the fingerprint information storage method of the first aspect and various possible implementation manners of the first aspect.

[0025] The fingerprint information storage method provided in the application determines the fingerprint information of a target element, determines a plurality of candidate buckets based on the target element, and determines the hotspot values of the plurality of candidate buckets respectively; in the case where the hotspot value of a candidate bucket exceeds a preset threshold value, the hotness state of the candidate bucket is determined as a first hotness state, and in the case where the hotspot value of the candidate bucket does not exceed the preset threshold value, the hotness state of the candidate bucket is determined as a second hotness state; in the case where the hotness states of the plurality of candidate buckets are all the first hotness state, a first candidate bucket in the plurality of candidate buckets is determined, and a first path length corresponding to the execution of a dynamic kicking-out strategy by the first candidate bucket is determined based on a path prediction model; in the case where the first path length exceeds a preset path length, a shadow bucket corresponding to the first candidate bucket is determined, the shadow bucket corresponding to the first candidate bucket is determined as a target bucket, and the fingerprint information is stored in the target bucket. The method can realize real-time sensing of the hotness state of the candidate bucket, dynamically determine the storage position of the fingerprint information, significantly reduce the delay jitter caused by the chain kicking-out, effectively reduce the storage pressure of the candidate bucket by using the shadow bucket mechanism, avoid the concentrated outbreak of conflicts caused by high load, and thus realize the stabilization and efficiency of the insertion performance, and improve the stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0027] Figure 1 is a flowchart of the fingerprint information storage method provided in the application Figure 1 .

[0028] Figure 2 is a flowchart of the fingerprint information storage method provided in the application Figure 2 .

[0029] Figure 3 is a structural schematic diagram of the fingerprint information storage device provided in the application.

[0030] Figure 4 is a structural schematic diagram of the electronic device provided in the application.

[0031] The above drawings have shown the specific embodiments of the present application, and will be described in more detail hereinafter. These drawings and the written description are not intended to limit the scope of the present application concept in any way, but to illustrate the present application concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0032] The exemplary embodiments will be described in detail below with reference to the drawings. In the following description, the same numbers are used to denote the same elements throughout the several views. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they only represent examples of apparatuses and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0033] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, public order and good customs are not violated, and appropriate operation portals are provided for users to choose authorization or refusal.

[0034] And the present application involves big data analysis of user information (including but not limited to personal biological characteristics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and uses artificial intelligence technology for automatic decision-making, and makes technical solutions based on automatic decision-making results that have a significant impact on personal rights and interests, provides appropriate operation portals for users to choose to agree or refuse automatic decision-making results; if the user chooses to refuse, enter the expert decision-making process.

[0035] It should be noted that the fingerprint information storage method, device, equipment, medium, product and filter provided by the present application can be used in the field of big data technology, and can also be used in any field other than big data. The application field of the fingerprint information storage method, device, equipment, medium, product and filter in the present application is not limited.

[0036] First, the terms involved in the present application are explained.

[0037] Cuckoo filter: a probabilistic data structure based on cuckoo hashing algorithm, used to quickly determine whether an element belongs to a certain set, while supporting dynamic insertion and deletion operations, and has better space efficiency and lower false positive rate than Bloom filter in certain scenarios.

[0038] In the fields of finance, e-commerce, Internet of Things, etc. that need to process massive data in real time, the performance of data structure is crucial to system efficiency. As a probabilistic data structure, cuckoo filter is widely used in scenarios that need to quickly determine whether data exists (member query) due to its high space utilization and excellent query performance, and has become a key technical means to handle massive data query requirements.

[0039] In the prior art, the commonly used operation mechanism of the cuckoo filter is to process data according to the established algorithm logic (such as hash function mapping, storage rules of buckets and fingerprints) in the data insertion and query process, so as to provide relatively stable member query service.

[0040] However, in the cuckoo filter, when the high-frequency insertion operation triggers the recursive kick-out, the conflict chain length exceeds the threshold (such as > 10 memory reallocations), causing the single insertion delay to suddenly increase from the microsecond level to the millisecond level, forming a response time fluctuation (i.e. delay jitter) with uncertainty, and further affecting the real-time performance of transaction processing.

[0041] In addition, high-concurrency insertion operations may trigger a hash conflict amplification effect, causing the 99th percentile delay to be much higher than the average value, ultimately leading to user request timeout or system load abnormality.

[0042] To solve the above problems, the present application provides a fingerprint information storage method.

[0043] The fingerprint information storage method provided by the present application quickly locates the candidate bucket through the hash function, and combines the bucket state module to realize the real-time sensing of the bucket heat state. In the dynamic insertion process of the cuckoo filter, the dynamic path selection strategy is combined with the shadow bucket chain elastic expansion mechanism to reduce the pressure of the main bucket, avoid the chain kick-out and the hash conflict amplification effect, so as to realize the stabilization of the insertion performance. The method realizes the real-time sensing of the bucket state and actively selects the low-load path, so that the insertion operation is no longer dependent on the random kick-out strategy, significantly reduces the delay jitter caused by the chain kick-out, reduces the conflict probability, makes the insertion time distribution tend to be stable, avoids the abnormal increase of the 99th percentile delay, and improves the stability of the system.

[0044] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail in the specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0045] Figure 1 is the flowchart of the fingerprint information storage method provided by the present application Figure 1 The execution subject of the embodiment may, for example, be a control unit of the cuckoo filter. As Figure 1 shown, the fingerprint information storage method provided by the present application comprises:

[0046] S101: Determine the fingerprint information of the target element.

[0047] The fingerprint information is used to identify the target element.

[0048] In the embodiments of the present application, the key-value database is the core component of data storage, and the basic requirement is to successfully write data; only by ensuring that the data is accurately stored in the database, can the subsequent reading, updating and deleting operations have a reliable data basis; the key-value database stores data in the form of key-value pairs, wherein the key-value pair is composed of a key and a value, the key is used to quickly locate and retrieve the corresponding value in the database, and the value refers to the actual data content associated with the key, which can be of any type (such as string, number, object, array, etc.); for example, in an e-commerce system, user order information, commodity inventory data, etc. need to be stably stored in the key-value database to support order processing, inventory management and other business logic; if data writing fails, it may cause order loss, inaccurate inventory data, etc., which seriously affects the normal operation of the business.

[0049] In a large-scale data storage scenario (high-concurrency scenario), the control unit in the cuckoo filter performs existence detection on the data to be stored, to avoid writing duplicate data into the key-value database, thereby effectively saving storage space; wherein the key-value database stores the key-value pair of the data to be stored at this time, that is, the specific data content and the key (that is, the element) corresponding to the data content, and the bucket of the cuckoo filter stores the fingerprint information obtained by hashing the element.

[0050] As can be understood, the element refers to the key in the key-value pair of the stored data, and the element can be used to quickly locate and query the corresponding data in the database; the target element refers to the key in the key-value pair of the data to be stored at this time; the fingerprint information refers to the identification value obtained by hashing the target element, which can distinguish different elements.

[0051] For example, if the data to be stored at this time is data A, the corresponding key is 5, that is, the target element is 5, at this time, the fingerprint information obtained by hashing the target element is f1, specifically, the fingerprint information can be the lower 4 bits of the hash value of the target element.

[0052] In some embodiments, the fingerprint information can also be the target element itself.

[0053] S102: Determine a plurality of candidate buckets based on the target element, and determine the hotness state of each candidate bucket.

[0054] Wherein, the candidate bucket is used to indicate the bucket position set where the fingerprint information can be stored, and the hotness state is used to quantify the storage pressure of the candidate bucket.

[0055] In the embodiments of the present application, the cuckoo filter includes a bucket state module, which can collect and store the hotness state of each bucket in the cuckoo filter in real time, and provide data support for real-time decision of the control unit.

[0056] For the target element of the determined data to be stored, multiple candidate buckets are determined by performing multiple hash functions on the target element; and the heat state of each candidate bucket is determined by using the bucket state module.

[0057] It can be understood that in the cuckoo filter, the determination of multiple candidate buckets is obtained by performing multiple independent hash function calculations on the target element (i.e., the key to be inserted or queried); each hash function maps the target element to a different bucket index position, thereby generating a set of candidate buckets.

[0058] For example, using a three-hash cuckoo filter to store fingerprint information, if the number of current buckets is 8, namely "Bucket0, Bucket1, Bucket2, Bucket3, Bucket4, Bucket5, Bucket6, Bucket7", and the capacity of each bucket is 2, the hash functions used for the hash calculation of the target element include "H1(x) = x%8, H2(x) = (x+1)%8, H3(x) = (x+3)%8", and in the case of a target element x = 5, the set of multiple candidate buckets of the target element is calculated by using the corresponding hash functions H1(x), H2(x), and H3(x), and the corresponding calculation results are H1(5)%8 = 5, H2(5)%8 = 6, and H3(5)%8 = 0. At this time, the multiple candidate buckets can be determined as "Bucket0, Bucket5, Bucket6". After determining the candidate buckets that can currently store fingerprint information, the heat state of the corresponding bucket is obtained by using the bucket state module. Specifically, the heat state of Bucket0 is hot, the heat state of Bucket5 is cold, and the heat state of Bucket6 is hot.

[0059] S103: Determine the target bucket of the fingerprint information based on the heat states of the multiple candidate buckets, and store the fingerprint information in the target bucket.

[0060] The target bucket is used to store the fingerprint information of the target element.

[0061] It can be understood that the heat state represents the storage density of the fingerprint information in the bucket, and the load degree and operation pressure of the corresponding bucket; different heat states represent that the buckets have different storage pressures, such as the heat state of the hot bucket, that is, the hot bucket, the storage density of the hot bucket is high (the load factor is equal to or exceeds the threshold value, such as 0.9), the operation frequency is high, the number of fingerprint information corresponding to the hot bucket is close to the capacity upper limit, the storage space is few or no storage space, and the relocation is frequently triggered. At this time, the hot bucket is easy to become the performance bottleneck of the cuckoo filter. The heat state of the cold bucket, that is, the cold bucket, the storage density of the cold bucket is low (the load factor is less than 0.5), the operation frequency is low, the number of fingerprints in the bucket is far lower than the capacity, and the storage space is sufficient. At this time, the cold bucket is easy to exist idle resources. Therefore, when determining the target bucket for storing the fingerprint information, according to the heat state of the current candidate bucket, the bucket with a relatively low heat state is preferentially selected.

[0062] In the case that the heat state indicates that the plurality of candidate buckets are all hot buckets, a candidate bucket is determined from any of the plurality of candidate buckets, an additional storage space is allocated for the candidate bucket, and the storage space is used as an overflow area of the candidate bucket. The current to-be-stored fingerprint information is stored in the overflow area.

[0063] The heat states of the plurality of candidate buckets are compared, the candidate bucket with the lowest heat state is selected from the plurality of candidate buckets, and the candidate bucket is used as the target bucket. The fingerprint information is stored in the target bucket. In the process of selecting the candidate bucket with the lowest heat state from the plurality of candidate buckets, if the candidate bucket with the lowest heat state cannot be selected, that is, the heat states of the plurality of candidate buckets are all hot, at this time, any one of the plurality of candidate buckets can be determined. An additional storage space is allocated for the candidate bucket as an overflow area of the candidate bucket, and the current to-be-stored fingerprint information is stored in the overflow area.

[0064] For example, if the heat states of the plurality of candidate buckets are currently determined to be: the heat state of Bucket0 is hot, the heat state of Bucket5 is cold, and the heat state of Bucket6 is hot, the bucket with the lowest heat state is selected as the target bucket, that is, Bucket5 is determined as the target bucket, and the fingerprint information f1 is stored in Bucket5. If the heat states of the plurality of candidate buckets are currently determined to be: the heat state of Bucket0 is hot, the heat state of Bucket5 is hot, and the heat state of Bucket6 is hot, the heat states of the candidate buckets are all hot, at this time, an overflow area Bucket5-1 can be allocated for Bucket5, and the Bucket5-1 is determined as the target bucket. The fingerprint information f1 is stored in the Bucket5-1.

[0065] In the embodiment of the present application, the control unit may, for example, be a path decision engine. The path decision engine is configured to predict an insertion path cost based on the heat state of the bucket, and dynamically select a target bucket or a shadow bucket, so as to store the current fingerprint information.

[0066] The fingerprint information storage method provided in the embodiment determines the fingerprint information of a target element, determines a plurality of candidate buckets based on the target element, and determines the heat states of the plurality of candidate buckets respectively. The target bucket of the fingerprint information is determined based on the heat states of the plurality of candidate buckets, and the fingerprint information is stored in the target bucket. The method dynamically determines the storage location of the fingerprint information by real-time sensing the heat states of the candidate buckets, significantly reduces the delay jitter caused by the chain kickout, avoids the performance jitter caused by burst traffic, ensures the stability and real-time performance of the storage operation, and improves the stability of the system.

[0067] Figure 2 FIG. 1 is a flowchart of the fingerprint information storage method provided in the present application. Figure 2 As shown in FIG. 2, the embodiment provides a detailed description of the fingerprint information storage method based on the embodiment. Figure 1 Figure 3 The fingerprint information storage method provided in the embodiment includes the following steps.

[0068] S201: Determine the fingerprint information of a target element.

[0069] Step S201 is similar to step S101 described above, and will not be described again here.

[0070] S202: Determine a plurality of candidate buckets based on the target element, and determine the hot value of each candidate bucket.

[0071] S203: If the hot value of the candidate bucket exceeds a preset threshold, determine the heat state of the candidate bucket as a first heat state.

[0072] S204: If the hot value of the candidate bucket does not exceed the preset threshold, determine the heat state of the candidate bucket as a second heat state.

[0073] The hot value is used to quantify the heat state of the candidate bucket, and the preset threshold may, for example, be 8.

[0074] ​It can be understood that the hotspot value is a score of the candidate bucket in the cuckoo filter, which quantifies the frequency of access, load intensity (such as the number of fingerprints, the frequency of insertion conflict), and the like. According to the comparison result of the hotspot value and the preset threshold, the heat state can be divided into a first heat state and a second heat state. Specifically, in the case where the hotspot value of the candidate bucket exceeds the preset threshold, it indicates that the current candidate bucket is frequently accessed or operated, and the candidate bucket has a high hotspot value. At this time, it is determined that the heat state of the candidate bucket is the first heat state. In the case where the hotspot value of the candidate bucket does not exceed the preset threshold, it indicates that the access amount of the current candidate bucket is low or the number of operations is low, and the candidate bucket has a low hotspot value. At this time, it is determined that the heat state of the candidate bucket is the second heat state.

[0075] Based on the target element, a plurality of candidate buckets are determined. The hotspot values of the plurality of candidate buckets are determined by the bucket state module, and the hotspot values of different candidate buckets are compared with the preset threshold. In the case where the hotspot value of the candidate bucket exceeds the preset threshold, it is determined that the heat state of the candidate bucket is the first heat state. In the case where the hotspot value of the candidate bucket does not exceed the preset threshold, it is determined that the heat state of the candidate bucket is the second heat state.

[0076] For example, the first heat state can be a high heat state, and the second heat state can be a low heat state. In the case where the target element x = 5, the plurality of candidate buckets can be determined as: “Bucket0, Bucket5, Bucket6”. After determining the candidate bucket in which the fingerprint information can be stored, the hotspot value of the corresponding bucket is obtained by using the bucket state module. Specifically, the hotspot value of Bucket0 is 9, the hotspot value of Bucket5 is 5, and the hotspot value of Bucket6 is 10. At this time, it can be determined that the heat state of Bucket0 is the high heat state, the heat state of Bucket5 is the low heat state, and the heat state of Bucket6 is the high heat state.

[0077] In some embodiments, the determination of the hotspot value can be: respectively determining the number of fingerprint information, the conflict frequency and the relocation frequency of the plurality of candidate buckets; determining the weighted values of the number of fingerprint information, the conflict frequency and the relocation frequency of the plurality of candidate buckets according to the preset weight information, and determining the plurality of weighted values as the hotspot values of the corresponding candidate buckets.

[0078] It can be understood that the number of fingerprint information refers to the number of currently stored fingerprint information in the candidate bucket, which is used to reflect the occupation degree of the corresponding candidate bucket. For example, the capacity of Bucket0 is 10, and 8 fingerprint information has been stored currently, so the number of fingerprint information is 8; the conflict frequency refers to the number of times of conflict of the candidate bucket due to storage operation within a predetermined time (i.e., the corresponding candidate bucket is full and needs to kick out the original fingerprint information in the corresponding candidate bucket), which is used to reflect the access frequency and load pressure of the corresponding candidate bucket. For example, Bucket5 has a conflict of 20 times within 1h due to storage operation, so the conflict frequency of Bucket5 is 20 times / h; the relocation frequency refers to the frequency of fingerprint information in the candidate bucket being kicked out, relocated and stored to other candidate buckets, which is used to reflect the stability of the corresponding candidate bucket. For example, the fingerprint information in Bucket6 is relocated 5 times within 1h, so the relocation frequency of Bucket6 is 5 times / h.

[0079] The number of fingerprint information, the conflict frequency and the relocation frequency all belong to the load indicators of the candidate bucket, and the hotspot value is a multi-dimensional comprehensive load indicator; therefore, when calculating the hotspot value, the multiple load indicators (i.e., the number of fingerprint information, the conflict frequency and the relocation frequency) of the candidate bucket are combined with the preset weight information to perform weighted calculation on the multiple load indicators, so as to obtain the hotspot value of the corresponding candidate bucket.

[0080] For example, the preset weight information of the number of fingerprint information, the conflict frequency and the relocation frequency is: “number of fingerprint information-0.5, conflict frequency-0.3, relocation frequency-0.2”, and the hotspot values of the multiple candidate buckets are calculated based on the preset weight information. The hotspot value is not specially limited in the present application.

[0081] S205: In the case that the heat states of the multiple candidate buckets are all the first heat state, for a first candidate bucket in the multiple candidate buckets, a first path length corresponding to the execution of the dynamic kicking-out strategy by the first candidate bucket is determined based on a path prediction model.

[0082] S206: In the case that the first path length exceeds a preset path length, a shadow bucket corresponding to the first candidate bucket is determined, and the shadow bucket corresponding to the first candidate bucket is determined as the target bucket.

[0083] S207: The fingerprint information is stored to the target bucket.

[0084] The path prediction model is used to predict the path length of the fingerprint information kicked out of the candidate bucket, the dynamic kicking-out strategy is used to kick out any fingerprint information in the candidate bucket, and the first path length refers to the path length of any fingerprint information in the first candidate bucket kicked out.

[0085] In the embodiment of the present application, the cuckoo filter further comprises an elastic bucket controller, which is configured to dynamically allocate shadow buckets according to the hotness state of the plurality of candidate buckets.

[0086] In the cuckoo filter, when new fingerprint information is stored, if all candidate buckets are full, a dynamic kicking-out strategy is triggered, part of the original fingerprint information in the current candidate bucket is selected and kicked out (the kicked-out original fingerprint information is stored in the corresponding alternative position), and a storage space is provided for the new fingerprint information to be stored.

[0087] However, during the execution of the dynamic kicking-out strategy, if the number of times of continuously kicking out the original fingerprint information in the candidate bucket exceeds a preset threshold, it indicates that the kicked-out fingerprint information cannot find an effective storage space in its alternative position, thereby causing a circular blockage of the chain kicking-out, and the new fingerprint information cannot be provided with a usable storage space, i.e., the dynamic kicking-out strategy fails, and the storage operation of the new fingerprint information is terminated.

[0088] The path prediction model can predict the execution path of the dynamic kicking-out strategy before the execution of the dynamic kicking-out strategy, and obtain a predicted value of the kicking-out path length of the dynamic kicking-out strategy. For example, when new fingerprint information is stored, the chain kicking-out is triggered, fingerprint 1 is kicked out from candidate bucket A for the first time, fingerprint 1 is migrated to alternative bucket B, and it is found that the bucket is full, and then fingerprint 2 is kicked out; fingerprint 2 is migrated to alternative bucket C, and an empty slot is found to complete the storage, and the whole process is continuously kicked out twice, and the path length corresponding to the chain kicking-out is 2.

[0089] In some embodiments, the path length recorded when the dynamic kicking-out strategy is executed can be the number of times of kicking out the kicked-out fingerprint information, or can be a quantized value of the fingerprint migration span value or the kicking-out link complexity. The embodiment of the present application does not specially limit the path length.

[0090] For example, the double-hash cuckoo filter allocates two candidate positions for each fingerprint information through two hash functions. and If both positions are occupied, an existing fingerprint information is randomly kicked out, and an attempt is made to reinsert it into the other candidate position. If the kicked-out fingerprint information encounters the same situation again (i.e., its other candidate position is also occupied) when reinserting, other fingerprint information is continuously kicked out, and a loop is formed. If the number of loops exceeds a preset threshold (such as 500 times), it is determined that the kicking-out fails, and the corresponding preset path length can be 500.

[0091] It can be understood that the first candidate bucket refers to any candidate bucket selected from a plurality of candidate buckets in the case that the plurality of candidate buckets are in the first heat state; the first candidate bucket is not used for storing the fingerprint information at this time, and the fingerprint information at this time is stored in the shadow bucket of the first candidate bucket.

[0092] In the case that the plurality of candidate buckets are in the first heat state, that is, the plurality of candidate degrees are high heat values, a candidate bucket is randomly selected from the plurality of candidate buckets, and the randomly selected candidate bucket is taken as the first candidate bucket; for the first candidate bucket, a path prediction model is used to predict the first path length (that is, the length of the kick-out path of the predicted execution of the dynamic kick-out strategy) when any fingerprint information in the first candidate bucket is kicked out; in the case that the first path length exceeds the preset path length, an elastic bucket controller is used to allocate a corresponding overflow area, that is, a shadow bucket, for the first candidate bucket, and the shadow bucket corresponding to the first candidate bucket is determined as the target bucket, and the fingerprint information is stored in the target bucket determined at this time.

[0093] In the case that the first path length does not exceed the preset path length, the first candidate bucket is taken as the target bucket, the dynamic kick-out strategy is executed, and the storage of the corresponding fingerprint information is performed.

[0094] For example, if the preset path length is 50, the first heat state is the high heat state, and the second heat state is the low heat state; the fingerprint information is f1, the plurality of candidate buckets are "Bucket0, Bucket5, Bucket6", and the heat state of Bucket0 is the high heat state, the heat state of Bucket5 is the high heat state, and the heat state of Bucket6 is the high heat state. At this time, one candidate bucket is randomly selected from "Bucket0, Bucket5, Bucket6" as the first candidate bucket, and a corresponding shadow bucket is allocated for the first candidate bucket; if Bucket0 is selected as the first candidate bucket at this time, a path prediction model is used to predict the first path length when the dynamic kick-out strategy is executed for Bucket0, if the first path length is 55, and the first path length exceeds the preset path length 50, the shadow bucket allocated for Bucket0 is Bucket0-1, Bucket0-1 is determined as the target bucket, and the fingerprint information f1 is stored in Bucket0-1.

[0095] In some embodiments, when there is at least one second hotness state in the plurality of hotness states, a candidate bucket corresponding to the second hotness state is determined as a second candidate bucket; when there is a storage vacancy in the second candidate bucket, the second candidate bucket is determined as the target bucket; when there is no storage vacancy in the second candidate bucket, a second path length corresponding to the execution of the dynamic eviction strategy by the second candidate bucket is determined based on the path prediction model; when the second path length exceeds the preset path length, a shadow bucket corresponding to the second candidate bucket is determined, and the shadow bucket corresponding to the second candidate bucket is determined as the target bucket.

[0096] The second path length refers to the path length of the eviction of any fingerprint information in the second candidate bucket.

[0097] It can be understood that the second candidate bucket refers to the candidate bucket corresponding to the second hotness state when there is at least one second hotness state in the plurality of hotness states.

[0098] For example, the first hotness state is a high hotness state, and the second hotness state is a low hotness state; the fingerprint information is f1, the plurality of candidate buckets are “Bucket0, Bucket5, Bucket6” respectively, the hotness state of Bucket0 is a high hotness state, the hotness state of Bucket5 is a low hotness state, and the hotness state of Bucket6 is a high hotness state. At this time, Bucket5 is determined as the second candidate bucket, and it is determined whether there is a storage vacancy in Bucket5; if there is a storage vacancy in Bucket5, it indicates that the fingerprint information can be directly stored in Bucket5, at this time, Bucket5 is determined as the target bucket, and the fingerprint information f1 is stored in Bucket5; if there is no storage vacancy in Bucket5, it indicates that the fingerprint information cannot be directly stored in Bucket5, at this time, the second path length corresponding to the execution of the dynamic eviction strategy by Bucket5 is predicted by using the path prediction model, if the second path length is 55, and the second path length exceeds the preset path length 50, the shadow bucket for Bucket5 is allocated by the elastic bucket controller, and the shadow bucket is Bucket5-1, Bucket5-1 is determined as the target bucket, and the fingerprint information f1 is stored in Bucket5-1.

[0099] In some embodiments, a lightweight machine learning model (such as a decision tree or a random forest) is integrated in the control unit (such as a path decision engine), and the path prediction model is obtained by training the historical insertion data.

[0100] The model input includes the current load state of the candidate bucket (fingerprint number, collision frequency), timestamp (such as periodic characteristics of insertion time), element hash distribution characteristics, and the output is the path length of triggering chain kicking after inserting the bucket; the control unit determines whether to use the shadow bucket to store the fingerprint information according to the prediction result. The application does not make special restrictions on the path prediction model.

[0101] In some other embodiments, after determining the shadow bucket, a plurality of storage levels of the shadow bucket are determined according to the collision frequency of the fingerprint information stored in the third candidate bucket; a target collision frequency of the fingerprint information is determined; a target storage level of the fingerprint information is determined based on the target collision frequency, and the fingerprint information is stored to the target storage level of the target bucket.

[0102] Among them, the plurality of storage levels are used to store fingerprint information of different collision frequencies, and the third candidate bucket is the first candidate bucket or the second candidate bucket.

[0103] It can be understood that the third candidate bucket refers to the candidate bucket allocated with the shadow bucket; when using the shadow bucket to store the fingerprint information, different storage levels can be set in the shadow bucket according to the collision severity of different fingerprint information, for example, the storage levels of the shadow bucket include a first storage level (corresponding to storing fingerprint information of low collision frequency) and a second storage level (corresponding to storing fingerprint information of high collision frequency), the first storage level is used for short-term overflow and only stores a small amount of collision fingerprints, and when migrating the fingerprint information in the shadow bucket to the corresponding candidate bucket, the fingerprint information in the first storage level can be preferentially migrated; the second storage level is used for long-term overflow and stores high-frequency collision fingerprints, and the corresponding fingerprint information is migrated only when the corresponding candidate bucket meets the data migration condition.

[0104] In some other embodiments, the fingerprint information stored in the shadow bucket is migrated to the third candidate bucket when the third candidate bucket meets the data migration condition.

[0105] Among them, the data migration condition includes at least one of the following:

[0106] The number of fingerprint information stored in the third candidate bucket is lower than a preset capacity threshold;

[0107] The collision frequency of the third candidate bucket shows a downward trend within a preset time window;

[0108] The storage capacity of the shadow bucket reaches a preset capacity upper limit.

[0109] Among them, the preset capacity threshold can be 80%, the preset time window can be 20min, and the preset capacity upper limit can be 4.

[0110] It can be understood that in the case that the third candidate bucket meets the data rollback condition, the actual storage pressure of the third candidate bucket allocated with the shadow bucket is reduced, at this time the fingerprint information stored in the shadow bucket can be migrated to the third candidate bucket, realizing the data migration balance between the shadow bucket and the corresponding candidate bucket, reducing resource waste and maintaining the stability of the corresponding candidate bucket.

[0111] In the embodiments of the present application, the first candidate bucket, the second candidate bucket and the third candidate bucket are substantially the same, and there is an association relationship among the three; the first candidate bucket and the second candidate bucket are determined from a plurality of candidate buckets for different heat states, and can store fingerprint information; the third candidate bucket refers to the first candidate bucket or the second candidate bucket allocated with the shadow bucket.

[0112] The fingerprint information storage method provided by the embodiment determines the fingerprint information of the target element, determines a plurality of candidate buckets based on the target element, and determines the heat value of each candidate bucket; in the case that the heat value of the candidate bucket exceeds the preset threshold, the heat state of the candidate bucket is determined as the first heat state, and in the case that the heat value of the candidate bucket does not exceed the preset threshold, the heat state of the candidate bucket is determined as the second heat state; in the case that the heat states of the plurality of candidate buckets are all the first heat state, the first candidate bucket in the plurality of candidate buckets is determined, and the first path length corresponding to the execution of the dynamic kick-out strategy of the first candidate bucket is determined based on the path prediction model; in the case that the first path length exceeds the preset path length, the shadow bucket corresponding to the first candidate bucket is determined, and the shadow bucket corresponding to the first candidate bucket is determined as the target bucket, and the fingerprint information is stored in the target bucket. The method can dynamically determine the storage position of the fingerprint information by real-time sensing the heat state of the candidate bucket, significantly reducing the delay jitter caused by the chain kick-out, effectively reducing the storage pressure of the candidate bucket by using the shadow bucket mechanism, avoiding the concentrated outbreak of conflicts caused by high load, thereby realizing the stabilization and efficiency of the insertion performance, and improving the stability of the system.

[0113] Figure 3 is a structural schematic diagram of the fingerprint information storage device provided by the present application. As Figure 4 shown, the present application provides a fingerprint information storage device, which comprises:

[0114] The determination module 301 is configured to determine the fingerprint information of the target element, and the fingerprint information is used to identify the target element.

[0115] The determination module 301 is further configured to determine a plurality of candidate buckets based on the target element, and determine the heat state of each candidate bucket.

[0116] The determination module 301 is further configured to determine the target bucket of the fingerprint information based on the heat states of the plurality of candidate buckets.

[0117] The processing module 302 is configured to store the fingerprint information into the target bucket.

[0118] Optionally, the determining module 301 is further configured to determine the hotspot values of the plurality of candidate buckets respectively.

[0119] The determining module 301 is further configured to determine, in a case where the hotspot value of the candidate bucket exceeds a preset threshold value, that the heat state of the candidate bucket is a first heat state,

[0120] The determining module 301 is further configured to determine, in a case where the hotspot value of the candidate bucket does not exceed the preset threshold value, that the heat state of the candidate bucket is a second heat state.

[0121] Optionally, the processing module 302 is further configured to, in a case where the heat states of the plurality of candidate buckets are all the first heat state, determine, for a first candidate bucket in the plurality of candidate buckets, a first path length of the first candidate bucket executing a dynamic eviction strategy based on a path prediction model, the path prediction model being used to predict a path length of evicting fingerprint information in the candidate bucket, the dynamic eviction strategy being used to evict any fingerprint information in the candidate bucket, and the first path length being used to represent the path length of evicting any fingerprint information in the first candidate bucket.

[0122] The determining module 301 is further configured to, in a case where the first path length exceeds a preset path length, determine a shadow bucket corresponding to the first candidate bucket, and determine the shadow bucket corresponding to the first candidate bucket as the target bucket.

[0123] Optionally, the determining module 301 is further configured to, in a case where there is at least one second heat state in the plurality of heat states, determine a candidate bucket corresponding to the second heat state as a second candidate bucket.

[0124] The determining module 301 is further configured to, in a case where there is a storage space in the second candidate bucket, determine the second candidate bucket as the target bucket.

[0125] The processing module 302 is further configured to, in a case where there is no storage space in the second candidate bucket, determine a second path length of the second candidate bucket executing the dynamic eviction strategy based on the path prediction model, the dynamic eviction strategy being used to evict any fingerprint information in the candidate bucket, and the second path length being used to represent the path length of evicting any fingerprint information in the second candidate bucket.

[0126] The determining module 301 is further configured to, in a case where the second path length exceeds the preset path length, determine a shadow bucket corresponding to the second candidate bucket, and determine the shadow bucket corresponding to the second candidate bucket as the target bucket.

[0127] Optionally, the determining module 301 is further configured to determine, after determining the shadow bucket, a plurality of storage levels of the shadow bucket according to a conflict frequency of the fingerprint information stored in the third candidate bucket, the plurality of storage levels being used to store the fingerprint information of different conflict frequencies, and the third candidate bucket being the first candidate bucket or the second candidate bucket.

[0128] The determining module 301 is further configured to determine a target conflict frequency of the fingerprint information.

[0129] The determining module 301 is further configured to determine, based on the target conflict frequency, a target storage level of the fingerprint information.

[0130] The processing module 302 is further configured to store the fingerprint information to the target storage level of the target bucket.

[0131] Optionally, the processing module 302 is further configured to, in a case where the third candidate bucket satisfies a data back migration condition, migrate the fingerprint information stored in the shadow bucket to the third candidate bucket.

[0132] The data back migration condition includes at least one of the following:

[0133] The number of the fingerprint information stored in the third candidate bucket is lower than a preset capacity threshold;

[0134] The conflict frequency of the third candidate bucket presents a downward trend within a preset time window;

[0135] The storage capacity of the shadow bucket reaches a preset capacity upper limit.

[0136] Optionally, the determining module 301 is further configured to determine the number of the fingerprint information, the conflict frequency and the relocation frequency of each of the plurality of candidate buckets.

[0137] The determining module 301 is further configured to determine, according to preset weight information, a weighted value of the number of the fingerprint information, the conflict frequency and the relocation frequency of each of the plurality of candidate buckets, and determine the plurality of weighted values as hotspot values of the corresponding candidate buckets respectively.

[0138] Figure 4 The structure of the electronic device provided in the present application is shown in the figure. ​ As shown in the figure, the electronic device provided in the present application includes a receiver 401, a transmitter 402, a processor 403 and a memory 404.

[0139] The receiver 401 is configured to receive instructions and data.

[0140] The transmitter 402 is configured to transmit instructions and data.

[0141] The memory 404 is configured to store computer execution instructions.

[0142] The processor 403 is configured to execute the computer-executable instructions stored in the memory 404 to implement each step of the fingerprint information storage method described in the above embodiments. For details, refer to the related description in the fingerprint information storage method embodiments.

[0143] Optionally, the memory 404 can be independent or integrated with the processor 403.

[0144] When the memory 404 is independent, the electronic device further includes a bus for connecting the memory 404 and the processor 403.

[0145] The present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the processor executes the computer-executable instructions, the fingerprint information storage method performed by the electronic device described above is implemented.

[0146] The present application also provides a computer program product, which includes a computer program. When the processor executes the computer program, the fingerprint information storage method of any one of the above embodiments is implemented.

[0147] The present application also provides a cuckoo filter, which includes a bucket array, a bucket state module, a flexible bucket controller, and a control unit.

[0148] The bucket array is configured to store the fingerprint information of elements.

[0149] The bucket state module is configured to collect and store the hotness state of each bucket in real time.

[0150] The flexible bucket controller is configured to dynamically allocate shadow buckets according to the hotness state.

[0151] The control unit is configured to use the fingerprint information storage method of any one of the above embodiments to dynamically determine the storage location of the fingerprint information based on the hotness state of each bucket.

[0152] It should be noted that, for the above method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0153] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0154] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0155] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0156] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0157] If the integrated units / modules are implemented in the form of software program modules and sold or used as independent products, they can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0158] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0159] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains or can relate. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the application are indicated by the following claims.

[0160] It should be understood that the application is not limited to the precise construction that has been described and illustrated herein and that various modifications and changes can be made therein without departing from the scope thereof. The scope of the application is indicated by the appended claims rather than by the description.

Claims

1. A method for storing fingerprint information, characterized in that, The method includes: Determine the fingerprint information of the target element, wherein the fingerprint information is used to identify the target element; Based on the target element, multiple candidate buckets are determined, and the heat status of each candidate bucket is determined. The heat status is used to quantify the storage pressure of the candidate bucket. Based on the heat status of the multiple candidate buckets, the target bucket for the fingerprint information is determined, and the fingerprint information is stored in the target bucket.

2. The method according to claim 1, characterized in that, Determining the heat status of the multiple candidate buckets includes: Determine the hotspot values ​​of the multiple candidate buckets respectively; If the hotspot value of the candidate bucket exceeds a preset threshold, the hotspot state of the candidate bucket is determined to be the first hotspot state. If the hotspot value of the candidate bucket does not exceed a preset threshold, the hotspot state of the candidate bucket is determined to be the second hotspot state.

3. The method according to claim 2, characterized in that, The step of determining the target bucket for the fingerprint information based on the heat status of the multiple candidate buckets includes: When the heat state of the multiple candidate buckets is the first heat state, for the first candidate bucket among the multiple candidate buckets, based on the path prediction model, the first path length for the first candidate bucket to execute the dynamic kick-out strategy is determined. The path prediction model is used to predict the path length for kicking out fingerprint information in the candidate bucket. The dynamic kick-out strategy is used to kick out any fingerprint information in the candidate bucket. The first path length is used to represent the path length for kicking out any fingerprint information in the first candidate bucket. If the length of the first path exceeds the preset path length, determine the shadow bucket corresponding to the first candidate bucket, and determine the shadow bucket corresponding to the first candidate bucket as the target bucket.

4. The method according to claim 2, characterized in that, The step of determining the target bucket for the fingerprint information based on the heat status of the multiple candidate buckets further includes: If at least one of the second heat states exists among multiple heat states, the candidate bucket corresponding to the second heat state is determined as the second candidate bucket; If there is a storage space in the second candidate bucket, the second candidate bucket will be determined as the target bucket; If there is no empty storage space in the second candidate bucket, the second path length corresponding to the dynamic kick-out strategy of the second candidate bucket is determined based on the path prediction model. The dynamic kick-out strategy is used to kick out any fingerprint information in the candidate bucket, and the second path length is used to represent the path length of any fingerprint information kicked out of the second candidate bucket. If the length of the second path exceeds the preset path length, the shadow bucket corresponding to the second candidate bucket is determined, and the shadow bucket corresponding to the second candidate bucket is determined as the target bucket.

5. The method according to claim 3 or 4, characterized in that, The method further includes: After determining the shadow bucket, multiple storage levels of the shadow bucket are determined based on the collision frequency of the fingerprint information stored in the third candidate bucket. The multiple storage levels are used to store fingerprint information with different collision frequencies. The third candidate bucket is either the first candidate bucket or the second candidate bucket. The step of storing the fingerprint information into the target bucket includes: Determine the target collision frequency of the fingerprint information; Based on the target collision frequency, the target storage level of the fingerprint information is determined, and the fingerprint information is stored in the target storage level of the target bucket.

6. The method according to claim 5, characterized in that, The method further includes: If the third candidate bucket meets the data migration conditions, the fingerprint information stored in the shadow bucket will be migrated to the third candidate bucket. The data migration conditions include at least one of the following: The amount of fingerprint information stored in the third candidate bucket is lower than a preset capacity threshold; The collision frequency of the third candidate bucket shows a decreasing trend within a preset time window; The storage capacity of the shadow bucket has reached the preset capacity limit.

7. The method according to claim 2, characterized in that, The step of determining the hotspot values ​​of the plurality of candidate buckets includes: The number of fingerprint information, collision frequency, and relocation frequency of the multiple candidate buckets are determined respectively; According to preset weight information, the weighted values ​​of the number of fingerprint information, the collision frequency and the relocation frequency of the multiple candidate buckets are determined respectively, and the multiple weighted values ​​are respectively determined as the hot spot values ​​of the corresponding candidate buckets.

8. A cuckoo-shaped filter, characterized in that, Includes: bucket array, bucket status module, flexible bucket controller, and control unit; The bucket array is used to store the fingerprint information of the elements; The bucket status module is used to collect and store the heat status of each bucket in real time; The elastic bucket controller is used to dynamically allocate shadow buckets according to the heat status; The control unit is used to dynamically determine the storage location of the fingerprint information based on the heat state of each bucket, using the method described in any one of claims 1-7.

9. A fingerprint information storage device, comprising: A determining module is used to determine the fingerprint information of a target element, wherein the fingerprint information is used to identify the target element; The determining module is further configured to determine multiple candidate buckets based on the target element, and to determine the heat status of each of the multiple candidate buckets, wherein the heat status is used to quantify the storage pressure of the candidate buckets; The determining module is further configured to determine the target bucket of the fingerprint information based on the heat status of the multiple candidate buckets; The processing module is used to store the fingerprint information into the target bucket.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images