Data positioning method, network equipment and storage medium
By gradually determining the filter parameters in the hash table group, the hash table insertion and search operations are simplified, and the hash table operation problems of high computing resources and high power consumption in the existing technology are solved, and a more efficient data positioning method is realized .
Patent Information
- Application Number
- CN202310802607.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-30
- Publication Date
- 2025-01-07
AI Technical Summary
In the prior art, the storage mode of multiple hash tables consumes a lot of computing resources during insertion and search operations, high power consumption and cumbersome steps, making it difficult to optimize while ensuring memory utilization.
By obtaining the key values of the target data, determine the filtering parameters in the hash table group, and gradually determine whether the target data belongs to each subtable until all subtables are traversed, simplifying the insertion and search operations, reducing the number of memory fetches and computing resources.
On the premise of ensuring memory utilization, the hash table insertion and search operations are simplified, computing resources and power consumption are reduced, and query efficiency is improved.
Smart Images
Figure CN119271663A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communication technology, and in particular to a data positioning method, a network device and a storage medium. Background Art
[0002] Hash table is also called hash table in Chinese. It can provide fast insertion and search operations. In the prior art, the performance parameters of data storage methods are generally evaluated by query performance and memory utilization. The storage method using multiple hash subtables can not only ensure high memory utilization, but also query results within a fixed number of times. The maximum number of table lookups is the number of subtables.
[0003] However, when using the storage method of multiple hash sub-tables, it is necessary to traverse the rows corresponding to all sub-tables during storage query, and the steps of inserting and searching operations are cumbersome, requiring more computing resources and relatively high power consumption. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to provide a data positioning method, a network device and a storage medium, which aims to reduce the energy consumption overhead when querying the hash table while ensuring its memory utilization when performing insertion operations and search operations on the hash table.
[0005] In a first aspect, an embodiment of the present application provides a data positioning method, the method comprising:
[0006] Obtaining a key value and data location requirement corresponding to the target data, and determining a filtering parameter corresponding to a first subtable of a plurality of subtables of a hash table group according to the key value;
[0007] Based on the data location requirement, determining whether the target data belongs to the first sub-table according to the filtering parameter and the key value;
[0008] If it is determined that the target data does not belong to the first sub-table, the filtering parameters corresponding to the second sub-table in the hash table group are determined according to the key value, and whether the target data belongs to the second sub-table is determined according to the filtering parameters and the key value, until the sub-table corresponding to the target data is determined or all sub-tables in the hash table group are traversed.
[0009] In a second aspect, an embodiment of the present application provides a network device, the network device comprising:
[0010] A processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of the data locating method as described above are realized.
[0011] In a third aspect, an embodiment of the present application provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the data positioning method as described above.
[0012] The embodiment of the present application provides a data positioning method, a network device and a storage medium, which obtains the key value and data positioning requirements corresponding to the target data, and determines the filtering parameters corresponding to the first subtable in the multiple subtables of the hash table group according to the key value; based on the data positioning requirements, determines whether the target data belongs to the first subtable according to the filtering parameters and the key value; if it is determined that the target data does not belong to the first subtable, determines the filtering parameters corresponding to the second subtable in the hash table group according to the key value, and determines whether the target data belongs to the second subtable according to the filtering parameters and the key value, until the subtable corresponding to the target data is determined or all subtables in the hash table group are traversed. In this way, when inserting and searching the hash table, the steps of the insertion and search operations can be simplified under the premise of ensuring its memory utilization, thereby greatly reducing the number of memory accesses of the hash table, and further reducing the computing resources and power consumption when querying the hash table. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0014] Figure 1 A flowchart of a data location method provided in an embodiment of the present application;
[0015] Figure 2 A structural example diagram of a filter for a hash table group provided in an embodiment of the present application;
[0016] Figure 3 A flowchart of another data location method provided in an embodiment of the present application;
[0017] Figure 4 A flowchart of another data location method provided in an embodiment of the present application;
[0018] Figure 5 A flowchart of another data location method provided in an embodiment of the present application;
[0019] Figure 6 Another structural example diagram of a filter for a hash table group provided in an embodiment of the present application;
[0020] Figure 7 A structural example diagram of a Bloom filter for a hash table group provided in an embodiment of the present application;
[0021] Figure 8 is a schematic block diagram of a data positioning device provided in an embodiment of the present application;
[0022] Fig. 9 A schematic block diagram of the structure of a network device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0025] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0026] In the prior art, a bloom filter can be used to query a hash table. However, the bloom filter has a false positive problem, that is, the bloom filter indicates that the entry exists in the hash subtable, but in fact the entry may not exist in the hash subtable. The bloom filter is essentially multiple hash subtables plus a bit map. If an entry is inserted into the table, the bit map corresponding to multiple hash values of the entry is set to 1, but the hash values of different entries in the table may be the same, that is, the 1 in the bit map cannot determine which entries are mapped. Therefore, when querying, the bloom filter hits are all 1, but it cannot be said that the entry is definitely in the table. If the false positive is high, the bloom filter basically loses its effect. In addition, if the bloom filter is to support deletion, each bit of the bit map needs to be replaced by a counter, which will result in a large memory overhead for the bloom filter.
[0027] The embodiment of the present application provides a data positioning method, a network device and a storage medium. The method can be applied to a network device, wherein the network device can be a server, a base station and an operation and maintenance device, but is not limited thereto. Thus, when inserting and searching a hash table, the steps of the inserting and searching operations can be simplified while ensuring the memory utilization, thereby greatly reducing the number of memory accesses of the hash table, and further reducing the computing resources and power consumption when searching the hash table.
[0028] Please refer to Figure 1 , Figure 1 A flow chart of a data location method provided in an embodiment of the present application. The data location method can simplify the steps of inserting and searching operations on a hash table while ensuring its memory utilization, thereby greatly reducing the number of memory accesses of the hash table, and further reducing the computing resources and power consumption when querying the hash table.
[0029] like Figure 1 As shown, the data positioning method may include S101 to S103.
[0030] S101. Acquire a key value and data location requirement corresponding to target data, and determine a filtering parameter corresponding to a first sub-table among multiple sub-tables of a hash table group according to the key value.
[0031] Among them, the target data can be an entry to be queried or an entry to be inserted, and the entry can be stored in a hash table. The key value (key) can be used to determine the memory storage location corresponding to the entry in the hash table. Data location requirements may include data query requirements and data insertion requirements. The hash table group may include multiple hash subtables. The first subtable can be any hash subtable in the hash table group, and is not specifically limited here. The filtering parameter can be a filter value (filter_parameter, fp), which is a parameter for the filter. The filter value is a pre-stored value and can be determined according to actual conditions. The key value corresponding to the target data needs to be hashed independently in each subtable, so the keywords corresponding to each subtable are independent, so the values or filtering parameters obtained for the same key value in different subtables are independent.
[0032] like Figure 2 As shown, the hash table group is composed of N hash sub-tables, and the depth of each hash sub-table is D. In the embodiment of the present application, corresponding filter parameters are additionally stored in each row of each hash sub-table. The filter parameters are associated with the key values stored in the same row, that is, the corresponding filter parameters in each hash sub-table can be determined by the key values.
[0033] For example, if the key value corresponding to the target data is 14, the filtering parameter corresponding to the first sub-table in the multiple sub-tables of the hash table group can be determined as 20 according to the key value; then, the filtering parameter corresponding to the second sub-table in the multiple sub-tables of the hash table group can be determined as 30 according to the key value, and so on. The corresponding filtering parameters in each hash sub-table can be determined by the key value.
[0034] In some embodiments, the key value is hashed to determine the row where the filter parameter is located in the first sub-table, and the filter parameter is extracted from the row where the filter parameter is located in the first sub-table to obtain the filter parameter, thereby accurately determining the filter parameter corresponding to the sub-table.
[0035] The row where the filter parameter is located in the first sub-table is a key value mapped to the corresponding row in the first sub-table.
[0036] For example, if the key value corresponding to the target data is 14, the key value can be hashed first to determine the row in which the filter parameter is located in the first subtable, such as the 5th row, and then the filter parameter stored in the 5th row of the first subtable is extracted to obtain the filter parameter corresponding to the first subtable as 20.
[0037] S102: Based on data location requirements, determine whether the target data belongs to the first sub-table according to filtering parameters and key values.
[0038] In some embodiments, if the data location requirement is a data query requirement, it is determined whether the comparison result between the key value and the filter parameter meets the preset location condition; if the comparison result between the key value and the filter parameter meets the preset location condition, the preset key value corresponding to the row where the filter parameter is located is obtained, and it is determined whether the key value is the preset key value; if the key value is the preset key value, it is determined that the target data belongs to the first sub-table; if the comparison result between the key value and the filter parameter does not meet the preset location condition, or the key value is not the preset key value, it is determined that the target data does not belong to the first sub-table. Therefore, it is possible to accurately determine whether the target data belongs to the first sub-table through the comparison result between the key value and the filter parameter.
[0039] Among them, the data query demand is the demand to find the target data in multiple sub-tables of the hash table group. The preset positioning condition can be used to determine whether the target data belongs to the preliminary condition of the first sub-table, and its specific judgment condition can be determined according to the actual situation, for example, it can be determined according to the value of the filter parameter. The preset key value can be the key value corresponding to each entry stored in the row where the filter parameter is located. At this time, the filter parameter can be the maximum key value or the minimum key value in the row where the filter parameter is located, etc., which is not specifically limited here. The value of the filter parameter can correspond to the preset positioning condition.
[0040] Specifically, when the comparison result between the key value and the filtering parameter meets the preset positioning condition and the key value is the preset key value, it is determined that the target data belongs to the first sub-table; when the comparison result between the key value and the filtering parameter does not meet the preset positioning condition or the key value is not the preset key value, it is determined that the target data does not belong to the first sub-table.
[0041] Specifically, if the data positioning requirement is a data query requirement, determining whether the comparison result of the key value and the filtering parameter meets the preset positioning condition can be considered as determining whether the key value is greater than the filtering parameter; if the key value is not greater than the filtering parameter, it is considered that the comparison result of the key value and the filtering parameter meets the preset positioning condition; if the key value is greater than the filtering parameter, it is considered that the comparison result of the key value and the filtering parameter does not meet the preset positioning condition.
[0042] For example, if the filter parameter is the maximum key value in its corresponding row, for example 40, if the key value corresponding to the target data is 30, and the data positioning requirement is a data query requirement, it can be determined that the key value is not greater than the filter parameter. Then, the preset key value corresponding to the row where the filter parameter is located is obtained, and it is determined whether the key value is the preset key value.
[0043] Specifically, if the data positioning requirement is a data query requirement, determining whether the comparison result of the key value and the filter parameter meets the preset positioning condition can be considered as determining whether the key value is less than the filter parameter; if the key value is not less than the filter parameter, it is considered that the comparison result of the key value and the filter parameter meets the preset positioning condition; if the key value is less than the filter parameter, it is considered that the comparison result of the key value and the filter parameter does not meet the preset positioning condition.
[0044] Exemplarily, if the filter parameter is the minimum key value in its corresponding row, for example 40, if the key value corresponding to the target data is 50, and the data positioning requirement is a data query requirement, it can be determined that the key value is not less than the filter parameter, then the corresponding preset key value in the row where the filter parameter is located is obtained, and it is determined whether the key value is the preset key value.
[0045] like Figure 3 As shown, exemplarily, the data processing method provided by the embodiment of the present application is introduced with the data location requirement as the data query requirement and the entry is only inserted into the hash table, specifically including S201-S209.
[0046] S201. Obtain the key value corresponding to the target data, and i=0.
[0047] S202: Set the filtering parameters to the corresponding key values in sub-table i.
[0048] S203, determine whether the comparison result between the key value and the filter parameter meets the preset positioning condition; if so, go to S204, if not, go to S207.
[0049] S204: Read the preset key value corresponding to the row where the filtering parameter is located.
[0050] S205: Determine whether the key value is a preset key value. If so, go to S206; if not, go to S207.
[0051] S206: Determine whether the target data exists.
[0052] S207, i++.
[0053] S208: Is i less than N? If so, go to S202; if not, go to S209.
[0054] S209: Determine that the target data does not exist.
[0055] In some embodiments, if the key value is not a preset key value, it is determined whether the identification information corresponding to the filter parameter is the preset identification information; if it is determined that the identification information corresponding to the filter parameter is the preset identification information, it is determined that the target data does not belong to the first sub-table; if it is determined that the identification information corresponding to the filter parameter is not the preset identification information, it is determined that the target data does not belong to the sub-table in the hash table group. In this way, the memory access during hash table query can be reduced in consideration of repeated insertion and deletion.
[0056] The preset identification information is preset identification information, which is used for comparison with the identification information to avoid false positive situations.
[0057] Taking the comparison result between the key value and the filter parameter as an example to determine whether the key value is greater than the filter parameter to determine whether it meets the preset positioning condition, this is explained. Due to repeated insertions and deletions, in order to ensure the fill rate, it is necessary to slightly increase the number of memory accesses. When inserting an entry, if a row is empty, but the filter parameter corresponding to the row may be smaller than the key value corresponding to the target data, not inserting the row will affect the fill rate, while directly inserting the key value and updating the filter parameter will affect the correctness of the query.
[0058] In this case, you can add identification information to the filter parameters corresponding to the row. When you query the row with the identification information, unless you hit the key value stored in the row, you must continue to query the following subtable. Hitting a row with identification information means one more memory access. With repeated insertion and deletion of entries, the filter parameter may drop to the previous value. At this time, you can remove the identification information in the filter parameter to reduce memory access during query.
[0059] Specifically, when it is determined that the identification information corresponding to the filter parameter is the preset identification information, unless the key value stored in the row is hit, if the key value stored in the row is not hit, it is determined that the target data does not belong to the first sub-table, and the subsequent sub-table must be queried; when it is determined that the identification information corresponding to the filter parameter is not the preset identification information, it can be determined that the target data does not belong to the sub-table in the hash table group, and there is no need to continue to query the subsequent sub-table, thereby reducing memory access during hash table queries.
[0060] like Figure 4 As shown, exemplarily, the data processing method provided by the embodiment of the present application is introduced with the data location requirement as the data query requirement and the situation where entries are repeatedly inserted and deleted, specifically including S301-S310.
[0061] S301. Obtain the key value corresponding to the target data, and i=0.
[0062] S302: Set the filtering parameters to the corresponding key values in sub-table i.
[0063] S303, determine whether the comparison result between the key value and the filter parameter meets the preset positioning condition; if so, go to S304, if not, go to S308.
[0064] S304: Read the preset key value corresponding to the row where the filtering parameter is located.
[0065] S305: Determine whether the key value is a preset key value. If so, go to S306; if not, go to S307.
[0066] S306: Determine whether the target data exists.
[0067] S307 , determine whether the identification information corresponding to the filtering parameter is the preset identification information, if so, go to S308 , if not, go to S310 .
[0068] S308, i++.
[0069] S309: Is i less than N? If so, go to S302; if not, go to S310.
[0070] S310: Determine that the target data does not exist.
[0071] In some embodiments, if the data location requirement is a data insertion requirement, the comparison result of the key value and the filter parameter is determined; if the comparison result of the key value and the filter parameter meets the preset location condition, it is determined that the target data belongs to the first sub-table; if the comparison result of the key value and the filter parameter does not meet the preset location condition, it is determined that the target data does not belong to the first sub-table. In this way, the hash sub-table corresponding to the target data can be accurately determined, so that the target data can be inserted into the corresponding position.
[0072] The data insertion requirement is the requirement to insert the target data into multiple subtables of the hash table group. The preset positioning condition can be used as a preliminary condition to determine whether the target data belongs to the first subtable. The specific judgment condition can be determined according to the actual situation, for example, according to the value of the filtering parameter.
[0073] Specifically, if the data positioning requirement is a data insertion requirement, determining whether the comparison result of the key value and the filter parameter meets the preset positioning condition can be considered as determining whether the key value is greater than the filter parameter; if the key value is not greater than the filter parameter, it is considered that the comparison result of the key value and the filter parameter meets the preset positioning condition; if the key value is greater than the filter parameter, it is considered that the comparison result of the key value and the filter parameter does not meet the preset positioning condition.
[0074] Exemplarily, if the filter parameter is the maximum key value in its corresponding row, for example 40, if the key value corresponding to the target data is 30, and the data positioning requirement is a data query requirement, it can be determined that the key value is not greater than the filter parameter, then it is determined that the target data belongs to the first subtable.
[0075] Specifically, if the data positioning requirement is a data insertion requirement, determining whether the comparison result of the key value and the filter parameter meets the preset positioning condition can be considered as determining whether the key value is less than the filter parameter; if the key value is not less than the filter parameter, it is considered that the comparison result of the key value and the filter parameter meets the preset positioning condition; if the key value is less than the filter parameter, it is considered that the comparison result of the key value and the filter parameter does not meet the preset positioning condition.
[0076] Exemplarily, if the filter parameter is the minimum key value in its corresponding row, for example 40, if the key value corresponding to the target data is 50, and the data positioning requirement is a data query requirement, it can be determined that the key value is not less than the filter parameter, then it is determined that the target data belongs to the first subtable.
[0077] In some embodiments, after determining that the target data belongs to the first subtable, determine whether there is an empty space in the row where the filter parameter is located; if it is determined that there is no empty space in the row where the filter parameter is located, remove the target parameter in the row where the filter parameter is located to form an empty space, and insert the key value in the empty space in the row where the filter parameter is located; update the key value and the filter parameter according to the target parameter, and insert the updated key value into the empty space in the row where the corresponding subtable is located until the key value is no longer updated. In this way, the target data can be accurately inserted into the corresponding hash subtable.
[0078] The target parameter is a key value that needs to be removed in the row where the filter parameter is located, such as the maximum key value in the row where the filter parameter is located, or the minimum key value in the row where the filter parameter is located.
[0079] Specifically, it is determined whether there is a vacancy in the row where the filtering parameter is located; if it is determined that there is a vacancy in the row where the filtering parameter is located, the key value is directly inserted into the corresponding vacancy in the row where the filtering parameter is located, and the filtering parameter is updated.
[0080] Taking determining whether the comparison result between the key value and the filtering parameter satisfies the preset positioning condition as an example to determine whether the key value is greater than the filtering parameter, it is explained. If it is determined that there is no vacancy in the row where the filtering parameter is located, the maximum key value can be removed from the row where the filtering parameter is located to form a vacancy, and the key value corresponding to the target data is inserted into the vacancy in the row where the filtering parameter is located; after the insertion is completed, the removed maximum key value needs to be inserted into the corresponding hash subtable, so the key value to be inserted is updated to the maximum key value, and at the same time, because the maximum key value is removed from the row where the filtering parameter is located, the filtering parameter needs to be re-determined, and then the data insertion process is performed through the updated key value, so that the updated key value is inserted into the vacancy in the row where the corresponding subtable is located, until the key value is no longer updated.
[0081] Exemplarily, the filter parameter is the maximum value in the row where the filter parameter is located, and determining whether the comparison result between the key value and the filter parameter meets the preset positioning condition is used as an example to explain whether the key value is greater than the filter parameter. If the filter parameter is 40, the key value to be inserted is 30. If it is determined that there is no vacancy in the row where the filter parameter is located, the maximum key value, that is, 40, can be removed from the row where the filter parameter is located, thereby forming a vacancy, and the key value corresponding to the target data is inserted into the vacancy in the row where the filter parameter is located; after the insertion is completed, the key value to be inserted is updated to the maximum key value, that is, 40. At the same time, since the maximum key value is removed from the row where the filter parameter is located, it is necessary to re-determine the filter parameter, such as updating it to 38, and then perform the data insertion process through the updated key value, so that the updated key value is inserted into the vacancy in the row where the corresponding subtable is located, until the key value is no longer updated.
[0082] like Figure 5 As shown, exemplarily, the data processing method provided in the embodiment of the present application is introduced with the data location requirement as the data insertion requirement, specifically including S401-S409.
[0083] S401. Obtain the key value corresponding to the target data, with i=0.
[0084] S402: Set the filtering parameters to the corresponding key values in sub-table i.
[0085] S403, determine whether the comparison result between the key value and the filter parameter meets the preset positioning condition; if so, go to S404, if not, go to S407.
[0086] S404, determine whether there is an empty space in the row where the filtering parameter is located, if yes, go to S410, if not, go to S405.
[0087] S405: Remove the target parameter in the row where the filtering parameter is located to form a vacancy, and insert a key value into the vacancy in the row where the filtering parameter is located.
[0088] S406: Update key values and filter parameters.
[0089] S407, i++.
[0090] S408: Is i less than N? If so, go to S402; if not, go to S409.
[0091] S409: Target data insertion failed.
[0092] S410: Insert key values and update filtering parameters.
[0093] S411. Target data is inserted successfully.
[0094] In some embodiments, if the comparison result between the key value and the filter parameter does not meet the preset positioning condition, it is determined whether there is a vacancy in the row where the filter parameter is located; if it is determined that there is a vacancy in the row where the filter parameter is located, the preset identification information corresponding to the filter parameter is updated. In this way, the preset identification information can be accurately updated, thereby effectively avoiding false positives in the case of repeated insertions and deletions.
[0095] Specifically, determine whether there is an empty space in the row where the filtering parameter is located; if it is determined that there is an empty space in the row where the filtering parameter is located, update the preset identification information corresponding to the filtering parameter, and then traverse the next hash subtable; if it is determined that there is no empty space in the row where the filtering parameter is located, directly traverse the next hash subtable.
[0096] In some embodiments, if the data location requirement is a data query requirement or a data insertion requirement, it is determined whether the number of bits of the filter parameter and the key value are the same; if it is determined that the number of bits of the filter parameter and the key value are the same, it is determined that the target data belongs to the first sub-table; if it is determined that the number of bits of the filter parameter and the key value are different, it is determined that the target data does not belong to the first sub-table. In this way, it can be accurately determined whether the target data belongs to the first sub-table.
[0097] Here, the filtering parameter at this time may be the common bit number or average bit number of all key values in the row where the filtering parameter is located, etc., which is not specifically limited here.
[0098] Specifically, if the data positioning requirement is a data query requirement, first calculate the number of bits of the key value corresponding to the target data to determine whether the number of bits of the filter parameter and the key value are the same; if it is determined that the number of bits of the filter parameter and the key value are the same, then determine that the target data belongs to the first sub-table; if it is determined that the number of bits of the filter parameter and the key value are not the same, then determine that the target data does not belong to the first sub-table.
[0099] Specifically, if the data location requirement is a data insertion requirement, it can be determined first whether there is a vacant position in the row where the filter parameter is located. If there is a vacant position in the row where the filter parameter is located, the key value of the target data is directly inserted, and the maximum common bit number of all key values in the row where the filter parameter is located after insertion is extracted as the filter parameter; if there is no vacant position in the row where the filter parameter is located, determine whether the number of bits of the filter parameter and the key value are the same; if it is determined that the number of bits of the filter parameter and the key value are the same, it is necessary to move the target key value in the row where the filter parameter is located and update the filter parameter; if it is determined that the number of bits of the filter parameter and the key value are not the same, traverse the next hash subtable.
[0100] The target key value is the key value to be moved, which can be determined by calculating the number of common bits. Since the more common bits, the better the filtering effect, the target key value to be moved needs to make the number of common bits of the remaining entries after the move the largest among all combinations.
[0101] S103: If it is determined that the target data does not belong to the first sub-table, determine the filtering parameters corresponding to the second sub-table in the hash table group according to the key value, and determine whether the target data belongs to the second sub-table according to the filtering parameters and the key value, until the sub-table corresponding to the target data is determined or all sub-tables in the hash table group are traversed.
[0102] The second subtable can be any hash subtable in the hash table group, for example, the hash subtable corresponding to the second smallest key value in the hash table group, which is not specifically limited here. Generally, all subtables in the hash table group can be sorted according to the key value, for example, the first subtable is the hash subtable corresponding to the smallest key value in the hash table group, the second subtable is the hash subtable corresponding to the second smallest key value in the hash table group, and so on.
[0103] Specifically, if it is determined that the target data belongs to the first subtable, a query or insert operation is performed on the target data in the corresponding row in the first subtable. If it is determined that the target data does not belong to the first subtable, the filter parameter corresponding to the second subtable in the hash table group is determined according to the key value, and whether the target data belongs to the second subtable is determined according to the filter parameter and key value corresponding to the second subtable; and so on, until the subtable corresponding to the target data is determined or all subtables in the hash table group are traversed.
[0104] like Figure 6As shown, for example, if the hash table group contains 32 sub-tables, the sub-table depth is 4*1024 and the width is 256 bits. Now insert an entry with a key value of 32 bits and a width of 64 bits into the table. A sub-table row can store 4 entries, and each sub-table can store 16*1024 entries. The maximum capacity of the entries in the hash table group is 512*1024.
[0105] The required RAM resources are 32Mb, and an additional 32 CRC32 functions are required, one corresponding to each sub-table.
[0106] Since the row entry capacity of the single subform is 4, according to the following formula:
[0107]
[0108]
[0109] Among them, m is the width of the filter, n is the capacity of the subtable entries, p is the false positive probability, and k is the number of hash functions. It can be determined that when the value of m is 32 and the number of hash functions k is 5, the false positive probability is about 2%.
[0110] like Figure 7 As shown, for example, the RAM resources need to be increased by 4 Mb on the basis of the original Bloom filter resources, and 5 additional hash functions need to be added for use by the Bloom filter.
[0111] As shown in Table 1, Table 1 is a performance comparison table of the filter provided in this application and the Bloom filter.
[0112]
[0113] Table 1
[0114] As shown in Table 1, the fill rate of the filter provided by the present application is similar to that of the Bloom filter, but the number of memory accesses and the number of bits accessed are less than those of the Bloom filter, the RAM and CRC costs are low, and the effect is better than that of the Bloom filter. In addition, if the Bloom filter needs to add a counter to support deletion, the RAM resources need to be doubled.
[0115] The embodiment of the present application provides a data locating method, which obtains the key value and data locating requirement corresponding to the target data, and determines the filtering parameters corresponding to the first subtable in the multiple subtables of the hash table group according to the key value; based on the data locating requirement, determines whether the target data belongs to the first subtable according to the filtering parameters and the key value; if it is determined that the target data does not belong to the first subtable, determines the filtering parameters corresponding to the second subtable in the hash table group according to the key value, and determines whether the target data belongs to the second subtable according to the filtering parameters and the key value, until the subtable corresponding to the target data is determined or all subtables in the hash table group are traversed. In this way, when inserting and searching the hash table, the steps of the inserting and searching operations can be simplified under the premise of ensuring its memory utilization, thereby greatly reducing the number of memory accesses of the hash table, and further reducing the computing resources and power consumption when querying the hash table.
[0116] See also Figure 8 , Figure 8 It is a schematic block diagram of a data locating device provided in an embodiment of the present application. The data locating device can be configured in a server to execute the aforementioned data locating method.
[0117] like Figure 8 As shown, the data locating device 500 includes: a filtering parameter determining module 501 , a sub-table determining module 502 and a sub-table traversing module 503 .
[0118] A filtering parameter determination module 501 is used to obtain a key value and a data location requirement corresponding to the target data, and determine a filtering parameter corresponding to a first sub-table in a plurality of sub-tables of a hash table group according to the key value;
[0119] A sub-table determination module 502, configured to determine whether the target data belongs to the first sub-table based on the data location requirement, the filtering parameter and the key value;
[0120] The subtable traversal module 503 is used to determine the filtering parameters corresponding to the second subtable in the hash table group according to the key value if it is determined that the target data does not belong to the first subtable, and determine whether the target data belongs to the second subtable according to the filtering parameters and the key value, until the subtable corresponding to the target data is determined or all subtables in the hash table group are traversed.
[0121] The sub-table determination module 502 is also used to determine whether the comparison result between the key value and the filter parameter meets the preset positioning condition if the data positioning requirement is a data query requirement; if the comparison result between the key value and the filter parameter meets the preset positioning condition, obtain the preset key value corresponding to the row where the filter parameter is located, and determine whether the key value is the preset key value; if the key value is the preset key value, determine that the target data belongs to the first sub-table; if the comparison result between the key value and the filter parameter does not meet the preset positioning condition, or the key value is not the preset key value, determine that the target data does not belong to the first sub-table.
[0122] The sub-table determination module 502 is also used to determine whether the identification information corresponding to the filtering parameter is the preset identification information if the key value is not the preset key value; if it is determined that the identification information corresponding to the filtering parameter is the preset identification information, then determine that the target data does not belong to the first sub-table; if it is determined that the identification information corresponding to the filtering parameter is not the preset identification information, then determine that the target data does not belong to the sub-table in the hash table group.
[0123] The sub-table determination module 502 is also used to determine the comparison result between the key value and the filtering parameter if the data location requirement is a data insertion requirement; if the comparison result between the key value and the filtering parameter meets the preset location condition, determine that the target data belongs to the first sub-table; if the comparison result between the key value and the filtering parameter does not meet the preset location condition, determine that the target data does not belong to the first sub-table.
[0124] The sub-table determination module 502 is also used to determine whether there is a vacant position in the row where the filtering parameter is located; if it is determined that there is no vacant position in the row where the filtering parameter is located, remove the target parameter in the row where the filtering parameter is located to form a vacant position, and insert the key value into the vacant position in the row where the filtering parameter is located; update the key value and the filtering parameter according to the target parameter, and insert the updated key value into the vacant position in the row where the corresponding sub-table is located until the key value is no longer updated.
[0125] The sub-table determination module 502 is further used to determine whether there is a vacancy in the row where the filtering parameter is located; if it is determined that there is a vacancy in the row where the filtering parameter is located, the preset identification information corresponding to the filtering parameter is updated.
[0126] The sub-table determination module 502 is also used to determine whether the number of bits of the filter parameter and the key value are the same if the data location requirement is a data query requirement or a data insertion requirement; if it is determined that the number of bits of the filter parameter and the key value are the same, then it is determined that the target data belongs to the first sub-table; if it is determined that the number of bits of the filter parameter and the key value are not the same, then it is determined that the target data does not belong to the first sub-table.
[0127] The filter parameter determination module 501 is further used to perform hash processing on the key value to determine the row where the filter parameter is located in the first sub-table; perform parameter extraction on the row where the filter parameter is located in the first sub-table to obtain the filter parameter.
[0128] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0129] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer network devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.
[0130] See also Fig. 9 , Fig. 9 A schematic block diagram of the structure of a network device provided in an embodiment of the present application.
[0131] like Fig. 9 As shown, the network device 600 may include a processor 601 and a memory 602 , and the processor 601 and the memory 602 are connected via a bus 603 , such as an I2C (Inter-integrated Circuit) bus.
[0132] In an exemplary embodiment, the processor 601 can be used to provide computing and control capabilities to support the operation of the entire network device. The processor 601 can be a central processing unit (CPU), and the processor 601 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0133] Specifically, the memory 602 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.
[0134] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present application, and does not constitute a limitation on the network device to which the embodiment of the present application is applied. The specific server may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0135] The processor 601 is used to run a computer program stored in the memory, and implement any one of the data positioning methods provided in the embodiments of the present application when executing the computer program.
[0136] In one embodiment, the processor 601 is used to run a computer program stored in a memory, and implement the following steps when executing the computer program: obtain a key value and a data location requirement corresponding to the target data, and determine a filtering parameter corresponding to a first subtable in multiple subtables of a hash table group according to the key value; based on the data location requirement, determine whether the target data belongs to the first subtable according to the filtering parameter and the key value; if it is determined that the target data does not belong to the first subtable, determine the filtering parameter corresponding to the second subtable in the hash table group according to the key value, and determine whether the target data belongs to the second subtable according to the filtering parameter and the key value, until the subtable corresponding to the target data is determined or all subtables in the hash table group are traversed.
[0137] In one embodiment, when the processor 601 determines whether the target data belongs to the first sub-table based on the data positioning requirement and the filtering parameters and the key value, it is used to implement: if the data positioning requirement is a data query requirement, then determine whether the comparison result of the key value and the filtering parameter meets the preset positioning condition; if the comparison result of the key value and the filtering parameter meets the preset positioning condition, obtain the preset key value corresponding to the row where the filtering parameter is located, and determine whether the key value is the preset key value; if the key value is the preset key value, determine that the target data belongs to the first sub-table; if the comparison result of the key value and the filtering parameter does not meet the preset positioning condition, or the key value is not the preset key value, determine that the target data does not belong to the first sub-table.
[0138] In one embodiment, after determining whether the key value is the preset key value, the processor 601 is used to implement: if the key value is not the preset key value, determining whether the identification information corresponding to the filtering parameter is the preset identification information; if it is determined that the identification information corresponding to the filtering parameter is the preset identification information, determining that the target data does not belong to the first sub-table; if it is determined that the identification information corresponding to the filtering parameter is not the preset identification information, determining that the target data does not belong to the sub-table in the hash table group.
[0139] In one embodiment, when the processor 601 determines whether the target data belongs to the first sub-table based on the data positioning requirement and according to the filtering parameters and the key value, it is used to implement: if the data positioning requirement is a data insertion requirement, then determine the comparison result between the key value and the filtering parameter; if the comparison result between the key value and the filtering parameter meets the preset positioning condition, then determine that the target data belongs to the first sub-table; if the comparison result between the key value and the filtering parameter does not meet the preset positioning condition, then determine that the target data does not belong to the first sub-table.
[0140] In one embodiment, after determining that the target data belongs to the first sub-table, the processor 601 is used to implement: determining whether there is a vacant position in the row where the filtering parameter is located; if it is determined that there is no vacant position in the row where the filtering parameter is located, removing the target parameter in the row where the filtering parameter is located to form a vacant position, and inserting the key value into the vacant position in the row where the filtering parameter is located; updating the key value and the filtering parameter according to the target parameter, and inserting the updated key value into the vacant position in the row of the corresponding sub-table until the key value is no longer updated.
[0141] In one embodiment, after the comparison result between the key value and the filtering parameter does not meet the preset positioning condition, the processor 601 is used to: determine whether there is a vacancy in the row where the filtering parameter is located; if it is determined that there is a vacancy in the row where the filtering parameter is located, update the preset identification information corresponding to the filtering parameter.
[0142] In one embodiment, when the processor 601 determines whether the target data belongs to the first sub-table based on the data location requirement and according to the filter parameter and the key value, it is used to implement: if the data location requirement is a data query requirement or a data insertion requirement, then determine whether the number of bits of the filter parameter and the key value are the same; if it is determined that the number of bits of the filter parameter and the key value are the same, then determine that the target data belongs to the first sub-table; if it is determined that the number of bits of the filter parameter and the key value are not the same, then determine that the target data does not belong to the first sub-table.
[0143] In one embodiment, when the processor 601 determines the filtering parameters corresponding to the first sub-table in the hash table group according to the key value, it is used to implement: hashing the key value to determine the row in which the filtering parameters are located in the first sub-table; and extracting parameters from the row in which the filtering parameters are located in the first sub-table to obtain the filtering parameters.
[0144] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the network device described above can refer to the corresponding process in the aforementioned data positioning method embodiment, and will not be repeated here.
[0145] An embodiment of the present application also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any data detection method provided in the description of the embodiment of the present application.
[0146] The storage medium may be an internal storage unit of the network device described in the foregoing embodiment, such as a hard disk or memory of the network device. The storage medium may also be an external storage device of the network device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., provided on the network device.
[0147] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0148] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.
[0149] The serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A data positioning method, characterized in that: The method comprises: Obtaining a key value and data location requirement corresponding to the target data, and determining a filtering parameter corresponding to a first subtable of a plurality of subtables of a hash table group according to the key value; Based on the data location requirement, determining whether the target data belongs to the first sub-table according to the filtering parameter and the key value; If it is determined that the target data does not belong to the first sub-table, the filtering parameters corresponding to the second sub-table in the hash table group are determined according to the key value, and whether the target data belongs to the second sub-table is determined according to the filtering parameters and the key value, until the sub-table corresponding to the target data is determined or all sub-tables in the hash table group are traversed.
2. The method according to claim 1, characterized in that The determining, based on the data location requirement and according to the filtering parameter and the key value, whether the target data belongs to the first sub-table includes: If the data positioning requirement is a data query requirement, determining whether a comparison result between the key value and the filtering parameter satisfies a preset positioning condition; If the comparison result between the key value and the filter parameter satisfies the preset positioning condition, obtaining the preset key value corresponding to the row where the filter parameter is located, and determining whether the key value is the preset key value; If the key value is the preset key value, determining that the target data belongs to the first sub-table; If the comparison result between the key value and the filter parameter does not satisfy the preset positioning condition, or the key value is not the preset key value, it is determined that the target data does not belong to the first sub-table.
3. The method according to claim 2, characterized in that After determining whether the key value is the preset key value, the method further includes: If the key value is not the preset key value, determining whether the identification information corresponding to the filtering parameter is the preset identification information; If it is determined that the identification information corresponding to the filtering parameter is the preset identification information, it is determined that the target data does not belong to the first sub-table; If it is determined that the identification information corresponding to the filter parameter is not the preset identification information, it is determined that the target data does not belong to a sub-table in the hash table group.
4. The method according to claim 1, characterized in that: The determining, based on the data location requirement and according to the filtering parameter and the key value, whether the target data belongs to the first sub-table includes: If the data location requirement is a data insertion requirement, determining a comparison result between the key value and the filtering parameter; If the comparison result between the key value and the filtering parameter meets the preset positioning condition, determining that the target data belongs to the first sub-table; If the comparison result between the key value and the filtering parameter does not satisfy the preset positioning condition, it is determined that the target data does not belong to the first sub-table.
5. The method according to claim 4, characterized in that After determining that the target data belongs to the first sub-table, the method further includes: Determine whether there is a vacancy in the row where the filtering parameter is located; If it is determined that there is no empty space in the row where the filtering parameter is located, then removing the target parameter in the row where the filtering parameter is located to form an empty space, and inserting the key value into the empty space in the row where the filtering parameter is located; The key value and the filter parameter are updated according to the target parameter, and the updated key value is inserted into the empty position in the row of the corresponding subtable until the key value is no longer updated.
6. The method according to claim 4, characterized in that If the comparison result between the key value and the filtering parameter does not meet the preset positioning condition, the method further includes: Determine whether there is a vacancy in the row where the filtering parameter is located; If it is determined that there is a vacancy in the row where the filtering parameter is located, the preset identification information corresponding to the filtering parameter is updated.
7. The method according to claim 1, characterized in that The determining, based on the data location requirement and according to the filtering parameter and the key value, whether the target data belongs to the first sub-table includes: If the data location requirement is a data query requirement or a data insertion requirement, determining whether the number of bits of the filtering parameter and the key value are the same; If it is determined that the number of bits of the filtering parameter and the key value is the same, determining that the target data belongs to the first sub-table; If it is determined that the number of bits of the filtering parameter and the key value are different, it is determined that the target data does not belong to the first sub-table.
8. The method according to claim 1, characterized in that The determining, according to the key value, a filtering parameter corresponding to the first sub-table in the hash table group includes: Performing hash processing on the key value to determine the row where the filtering parameter is located in the first sub-table; Parameter extraction is performed on the row where the filtering parameter is located in the first sub-table to obtain the filtering parameter.
9. A network device, characterized in that: The network equipment includes: A processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of the data locating method according to any one of claims 1 to 8 are realized.
10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the data locating method according to any one of claims 1 to 8.