Frequency estimation method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a frequency estimation method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of big data. The redundancy counting frequency group is a two-dimensional array, each row is used for carrying out independent frequency recording on all data items in streaming data, each element value in each row represents the estimated frequency of a plurality of data items in the streaming data, and the plurality of data items corresponding to each element value in each column are different. According to the method, multiple data items can share one position to achieve frequency estimation, it is guaranteed that the multiple data items sharing each position in each column are different, and therefore it is guaranteed that frequency estimation can be compatibly achieved on mass data items in a limited storage space through multi-row redundancy counting processing.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and more specifically, to a frequency estimation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, the common approach is to use a "counting" method, which involves using a hash table to record the number of times each element appears in the data stream. The key of this structure is the element itself, and the value is the current cumulative number of times that element has appeared. As the data stream continues to arrive, elements are read one by one, and the corresponding count is updated. Specifically, for each element read, it is first checked whether the element already exists in the hash table; if it exists, its corresponding count is incremented by one; if it does not exist, it is added to the hash table, and its count is initialized to one.

[0003] This method can accurately track the frequency of each element's occurrence without any estimation errors. When you need to query the frequency of a specific element, simply check its corresponding value in the hash table. If the element is in the table, its count is returned; if not, it means it has never appeared, and zero is returned.

[0004] In scenarios with fixed memory space, existing technologies can only achieve counting of more new elements by expanding memory. This is because the number of elements that can be counted is limited in a fixed memory space. Each new element requires allocated storage space in memory. When the number of possible element types (i.e., the number of keys) in the data stream is very large or unpredictable, the hash table will rapidly expand, consuming a large amount of memory. When the surge in new elements causes the hash table to use up all its memory space, counting errors will occur.

[0005] In summary, while existing counting methods are simple to implement and accurate, they are only suitable for scenarios with small data volumes or sufficient memory resources, and are difficult to adapt to high-throughput data stream processing scenarios with limited memory. Summary of the Invention

[0006] The purpose of this invention is to provide a frequency estimation method, apparatus, electronic device, and computer-readable storage medium to improve the problems existing in the prior art.

[0007] The embodiments of the present invention can be implemented as follows: In a first aspect, the present invention provides a frequency estimation method applied to a server, wherein a preset storage space of the server is used to store a redundant frequency count array; the redundant frequency count array is a two-dimensional array, wherein each row is used to independently record the frequency of all data items in the streaming data, each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different; the method includes: Receive a count request, and determine the current data item based on the count request; the current data item is any one of the data items in the streaming data. Using a preset family of hash functions and a direction function, multiple current mapping positions and multiple current direction values ​​corresponding to the current data item are determined; the current direction value is 1 or -1. The element value corresponding to each current mapping position in the redundant count array is added to the current direction value to update the element value corresponding to each current mapping position; one of the updated element values ​​corresponding to each current mapping position reflects the estimated frequency of the current data item.

[0008] In an optional implementation, the family of hash functions includes A hash function, the redundant count array includes OK, The step of determining multiple mapping positions and multiple direction values ​​corresponding to the current data item using a preset family of hash functions and direction functions includes: Input the current data items respectively A hash function is obtained. The current hash value; Will Each of the current hash values ​​is moduloed by w to obtain... The current column position; The redundancy count group The lines are respectively with Combine each current column position one by one to get One current mapping position; Will Each of the current hash values ​​is input into the direction function to obtain... The current direction value.

[0009] In an optional implementation, the method further includes: Receive a frequency lookup request and determine the target data item based on the frequency lookup request; Using a preset family of hash functions and direction functions, multiple target mapping positions and multiple target direction values ​​corresponding to the target data item are determined; Based on the multiple target mapping positions and multiple target direction values ​​corresponding to the target data item, the estimated frequency of the target data item is determined.

[0010] In an optional implementation, the family of hash functions includes A hash function, the redundant count array includes OK, List; The step of determining multiple target mapping positions and multiple target direction values ​​corresponding to the target data item using a preset family of hash functions and direction functions includes: Input the target data items respectively A hash function is obtained. One target hash value; Will Each target hash value is moduloed by w to obtain... Target column positions; The redundancy count group The lines are respectively with By combining the positions of each target column one by one, we obtain One target mapping location; Will Each target hash value is input into the direction function to obtain... One target direction value; The step of determining the estimated frequency of the target data item based on multiple target mapping positions and multiple target direction values ​​corresponding to the target data item includes: Will The target direction values ​​are respectively related to Multiplying the corresponding element values ​​of each target mapping position in the redundancy count array yields... Frequency of target estimation; The median of the estimated frequencies of the target is used to obtain the estimated frequency of the target data item.

[0011] In an optional implementation, the hash function family includes d hash functions, and the redundant count array is... OK, A two-dimensional array of columns; When the maximum capacity of the preset storage space is M satisfy: or In an optional implementation, the d hash functions in the hash function family are represented as follows: The first of them The hash functions are:

[0012] In the formula, This represents the input data item. Preset step size; The direction function is:

[0013] In the formula, This represents the modulo operation. Indicates the calculated first... One direction value.

[0014] Secondly, the present invention provides a frequency estimation device applied to a server, wherein a preset storage space of the server is used to store a redundant frequency count array; the redundant frequency count array is a two-dimensional array, wherein each row is used to independently record the frequency of all data items in the streaming data, each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different; the device includes: An update trigger module is used to receive a count request and determine the current data item based on the count request; the current data item is any type of data item in the streaming data. The position determination module is used to determine multiple current mapping positions and multiple current direction values ​​corresponding to the current data item using a preset family of hash functions and a direction function; the current direction value is 1 or -1. The frequency update module is used to add the element value corresponding to each current mapping position in the redundant count array to each current direction value to update the element value corresponding to each current mapping position; one of the updated element values ​​corresponding to each current mapping position reflects the estimated frequency of the current data item.

[0015] In an optional implementation, it further includes: The frequency lookup triggering module is used to receive frequency lookup requests and determine target data items based on the frequency lookup requests; The location determination module is also used to determine multiple target mapping positions and multiple target direction values ​​corresponding to the target data item by using a preset family of hash functions and direction functions; The frequency query module is used to determine the estimated frequency of the target data item based on multiple target mapping positions and multiple target direction values ​​corresponding to the target data item.

[0016] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a software program, and when the electronic device is running, the processor executes the software program to implement the frequency estimation method as described in the first aspect above.

[0017] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the frequency estimation method described in the first aspect.

[0018] Compared with existing technologies, embodiments of the present invention provide a frequency estimation method, apparatus, electronic device, and computer-readable storage medium. A preset storage space in the server is used to store a redundant frequency counting array. The redundant frequency counting array is a two-dimensional array, where each row is used to independently record the frequency of all data items in the streaming data. Each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different. The method is as follows: receiving a frequency counting request; determining the current data item based on the frequency counting request; the current data item is any type of data item in the streaming data; using a preset hash function family and direction function, determining multiple current mapping positions and multiple current direction values ​​corresponding to the current data item; the current direction value is 1 or -1; adding the element value corresponding to each current mapping position in the redundant frequency counting array to each current direction value to update the element value corresponding to each current mapping position; one of the updated element values ​​corresponding to each current mapping position reflects the estimated frequency of the current data item. In this invention, in each row of the redundant counting array, multiple data items share a position to achieve frequency estimation, and it is ensured that the multiple data items sharing each position in each column are different. In this way, through the redundant counting processing of multiple rows, it is ensured that frequency estimation of massive data items can be achieved compatiblely in a limited storage space. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is one of the schematic diagrams of a redundant counting group provided in an embodiment of the present invention.

[0021] Figure 2 This is a flowchart illustrating a frequency estimation method provided in an embodiment of the present invention.

[0022] Figure 3 This is a second schematic diagram of a redundant counting array provided in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of a frequency estimation device provided in an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0027] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0028] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0029] Here, we will first introduce the keywords or key terms involved in this invention: 1. Streaming data: refers to continuously generated and transmitted data streams in real time. Common sources include sensor data from Internet of Things (IoT) devices, real-time updates from social media, financial transaction data streams, network traffic logs, online game player activity, and product sales data streams.

[0030] 2. Hash Collision: This refers to a situation where, when using hash tables (such as HashMap and Hashtable) for data storage, different inputs (keys) are mapped to the same hash value or the same hash table index position through a hash function. In other words, even if two keys are different, their hash values ​​may be the same, causing them to be assigned to the same storage location (or bucket) when stored in the hash table. This is a hash collision.

[0031] Taking streaming data as the product sales data stream in an e-commerce platform as an example, if the existing technology of using hash tables to record the sales of each product in real time is adopted, each product link has an independent key-value pair in the hash table, so the sales records of massive products require a large amount of storage resources.

[0032] Furthermore, during major sales events (discount sales), many products are not discounted in the original product links. Merchants will create a new product link, which requires a new key-value pair to be formed in the hash table. Thus, during major sales events, the emergence of a large number of new product links will cause the memory usage of the hash table to expand rapidly in the short term. The only way to meet the sales records of the entire platform is to expand the storage capacity. After the major sales event, the memory usage will slowly decrease, and the increased storage will be idle, resulting in a waste of storage resources.

[0033] Similarly, if the streaming data is financial transaction data stream or other streaming data, the above problems will also occur if the existing counting method is used.

[0034] This invention provides a frequency estimation method that can be applied to servers. To accommodate the frequency requirements of massive data items, this invention pre-constructs a family of hash functions and a directional function, and reserves a portion of the server's storage space to store a redundant frequency array. Based on this redundant frequency array, frequency recording is achieved for all data items in the streaming data.

[0035] The streaming data addressed in this invention can be, but is not limited to, the following types: (1) Sensor data of Internet of Things (IoT) devices. The data items that need to be counted can be: the number of reports for each device ID, the frequency of occurrence of abnormal events (such as excessive device temperature or abnormal voltage), the triggering frequency of sensor events (such as smoke sensor), etc. (2) Real-time social media updates, the data items that need to be counted can be: the number of times some hashtags appear, the frequency of some keywords (such as economy, typhoon, high temperature), the frequency of some users' posts / forwards / likes, etc.; (3) Financial transaction data flow, the data items that need to be counted can be: the number of transactions of the trading account, the frequency of certain specific trading behaviors, the number of transactions of financial products, etc.; (4) Network traffic logs, the data items that need to be counted can be: the access frequency of IP addresses, the number of times URLs are accessed, the number of times network attack types occur, etc. (5) For online game player activities, the data items that need to be counted can be: the frequency of player operations (such as moving, using skills), the number of times game items are used, the frequency of occurrence of a certain game event (such as losing a game, winning a game, upgrading equipment), etc. For product sales data streams, the data items that need to be counted can be: total sales of all products, frequency of product clicks or orders during promotional activities, sales frequency of a certain product category, etc.

[0036] The pre-built hash function family includes d hash functions, which can be represented as follows: The first of them The hash functions are:

[0037] In the formula, This represents the input data item, and step is the preset step size (step can be 1, 2, 3, etc.). For a preset quantity value (e.g.) It can be a value of 4, 5, 6, 8, etc.

[0038] The pre-constructed direction function is:

[0039] In the formula, This represents the modulo operation. Indicates the calculated first... One direction value.

[0040] Among them, such as Figure 1 The pre-constructed redundant counting array is OK, A two-dimensional array of columns, the redundancy count array of the first column. line, number The element value of the column is represented as the first... Understandable. The relationship between the value of and the frequency estimation error e can be expressed as: , The term "tight bound" indicates the exact bound, used to describe the equivalence of upper and lower bounds on the asymptotic growth rate of a function. That is: and The relationship between them can be equivalent to a direct proportional relationship.

[0041] Given a maximum preset storage capacity of M, if the goal is to minimize error... Then the value of w satisfies If the goal is to achieve optimal space utilization, then the value of w must satisfy... .

[0042] The redundant count values ​​provided by this invention are used to independently record the frequency of all data items in the streaming data in each row. Each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different.

[0043] The frequency estimation method provided in this embodiment of the invention will be introduced below, taking the frequency update and frequency query of a data item as an example.

[0044] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a frequency estimation method provided in an embodiment of the present invention. The method includes the following steps S101~S103: S101. Receive the count request and determine the current data item based on the count request.

[0045] In this embodiment, when a data item appears in the streaming data, a count request for that data item will be generated. The current data item is any type of data item in the streaming data.

[0046] For example, taking the application scenario of recording product sales as an example, the current data item can be the link identifier of any product link. This example is only for illustration and is not intended to be limiting.

[0047] S102. Using a preset family of hash functions and direction functions, determine multiple current mapping positions and multiple current direction values ​​corresponding to the current data item.

[0048] In this embodiment, each current mapping position represents the row and column positions in the redundant count array, and each current direction value is either 1 or -1.

[0049] S103. Add the element value corresponding to each current mapping position in the redundant count array to each current direction value to complete the update of the element value corresponding to each current mapping position.

[0050] In this embodiment, one of the updated element values ​​corresponding to each current mapping position reflects the estimated frequency of the current data item.

[0051] The frequency estimation method provided in this embodiment of the invention uses a two-dimensional redundant counting array, where each row is used to independently record the frequency of all data items in the streaming data. Each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different. This allows multiple data items to share a position for frequency estimation, and ensures that the multiple data items sharing each position in each column are different. In this way, through redundant counting processing of multiple rows, frequency estimation of massive data items can be compatibly achieved in a limited storage space.

[0052] Among the optional implementation methods, step S102 can be implemented in the following ways: S1021. Input the current data items respectively. A hash function is obtained. The current hash value; S1022, will Each of the current hash values ​​is moduloed by w to obtain... The current column position; S1023, Set the redundant counter groups The lines are respectively with Combine each current column position one by one to get One current mapping position; S1024, will Each of the current hash values ​​is input into the direction function to obtain... The current direction value.

[0053] Among them, the The element value at the current mapped position is: , No. The updated element value at the current mapping position is represented as: .

[0054] This example is merely illustrative and is not intended to be limiting.

[0055] The frequency update of a data item is achieved through the steps S101~S103 described above.

[0056] When there is a frequency lookup requirement for a specific data item, the method may also include the following steps: S106. Receive a frequency lookup request and determine the target data item based on the frequency lookup request; S107. Using a family of hash functions and a direction function, determine the estimated frequency of the target data item from the redundant count array.

[0057] The implementation of step S107 includes: S1071, will Each target hash value is moduloed by w to obtain... Target column positions; S1072, Set up redundant counter groups The lines are respectively with By combining the positions of each target column one by one, we obtain One target mapping location; S1073, will Each target hash value is input into the direction function to obtain... One target direction value; S1074, will The target direction values ​​are respectively related to Multiplying the element values ​​corresponding to each target mapping position in the redundancy count array, we get... Frequency of target estimation; S1075. Calculate the median of the estimated frequency of the target data item to obtain the predicted frequency.

[0058] In this embodiment, the first The element values ​​at each target mapping location are: , No. The frequency of each target estimate is expressed as: ,So The frequency of target estimation is: The estimated frequency of the target data item is... of the median.

[0059] In the case of limited storage resources, in order to achieve frequency estimation of massive data items, this invention uses a family of hash functions to map each data item to multiple positions in a redundant count array (one position per row), and the update value (i.e., direction value) of each position is no longer simply "+1", but "+1 or -1", which can reduce the accumulation of errors caused by hash collisions.

[0060] From a mathematical perspective, even if multiple data items are mapped to the same location due to hash collisions, their update values ​​may cancel each other out at that location because the update value is 1 or -1. This can reduce the accumulation of errors caused by collisions. In frequency lookup, the element value at each location mapped to the data item is restored to sign using the direction value, and then the median of multiple locations is selected as the estimated frequency, making the estimate closer to the true value.

[0061] From a statistical perspective, the directional function used in this invention is independent and symmetrical (i.e., +1 and -1 have the same probability of occurrence). Therefore, when multiple data items have hash collisions, their expected contribution to that position is 0, which makes the error caused by the collision have the characteristic of "zero mean noise". Finally, by selecting the median during the query, this noise can be effectively filtered out.

[0062] For example, combined Figure 3 Assuming the redundancy counter is a 4×7 two-dimensional array, if a data item is the link identifier of product A, and its corresponding four current column positions are determined by a family of hash functions to be 0, 2, 6, and 3, then the four current mapping positions are: row 0, column 0; row 1, column 2; row 2, column 6; and row 3, column 3. Figure 3 The four gray areas are shown. If product A needs a sales update (sales +1), then: like , So after the update ; like , So after the update ; like , So after the update ; like , So after the update .

[0063] At the same time, if product B is also mapped to If the corresponding direction value is 1, then the sales updates of the two will cancel each other out, avoiding cumulative errors.

[0064] Next, if you want to find the sales volume of product A, you can find: , , The median of the four Therefore, the estimated sales volume of product A is 826.

[0065] The above examples are merely illustrations, and the embodiments of the present invention do not limit the scale of the redundant count values ​​or the size of each element value in the array.

[0066] In order to perform the corresponding steps in the above method embodiments and various possible implementations, an implementation of a frequency estimation device is given below.

[0067] Please see Figure 4 , Figure 4A schematic diagram of the frequency estimation device provided in an embodiment of the present invention is shown. The frequency estimation device 200 is applied to a server, and the server's preset storage space is used to store a redundant count array; the redundant count array is a two-dimensional array, in which each row is used to independently record the frequency of all data items in the streaming data, each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different.

[0068] The frequency estimation device 200 includes: an update trigger module 210, a location determination module 220, and a frequency update module 230.

[0069] The update trigger module 210 is used to receive count requests and determine the current data item based on the count requests; the current data item is any type of data item in the streaming data. The position determination module 220 is used to determine multiple current mapping positions and multiple current direction values ​​corresponding to the current data item using a preset family of hash functions and direction functions; the current direction value is 1 or -1. The frequency update module 230 is used to add the element value corresponding to each current mapping position in the redundant count array to each current direction value to complete the update of the element value corresponding to each current mapping position; one of the updated element values ​​corresponding to each current mapping position reflects the estimated frequency of the current data item.

[0070] Optionally, the position determination module 220 can be used to: input the current data item respectively A hash function is obtained. The current hash value; will Each of the current hash values ​​is moduloed by w to obtain... The current column position; the redundant count group The lines are respectively with Combine each current column position one by one to get The current mapping position; will Each of the current hash values ​​is input into the direction function to obtain... The current direction value.

[0071] Optionally, the frequency estimation device 200 further includes a frequency lookup trigger module 240 and a frequency query module 250: the frequency lookup trigger module 240 can be used to receive a frequency lookup request and determine the target data item based on the frequency lookup request; the position determination module 220 can be used to determine multiple target mapping positions and multiple target direction values ​​corresponding to the target data item using a preset hash function family and direction function; the target direction value is 1 or -1; the frequency query module 250 can be used to determine the estimated frequency of the target data item based on the multiple target mapping positions and multiple target direction values ​​corresponding to the target data item.

[0072] Optionally, the location determination module 220 can be used to: input the target data items respectively A hash function is obtained. One target hash value; will Each target hash value is moduloed by w to obtain... Target column positions; Redundancy count groups The lines are respectively with By combining the positions of each target column one by one, we obtain One target mapping location; will Each target hash value is input into the direction function to obtain... Each target direction value.

[0073] Optional, the frequency query module 250 can be used to: The target direction values ​​are respectively related to Multiplying the element values ​​corresponding to each target mapping position in the redundancy count array, we get... The estimated frequency of each target is determined by statistically analyzing the median of the estimated frequencies, thus obtaining the predicted frequency of the target data item.

[0074] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the frequency estimation device 200 described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0075] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 300 includes a processor 310, a memory 320, and a bus 330, with the processor 310 connected to the memory 320 via the bus 330.

[0076] The memory 320 can be used to store software programs or firmware, for example, the software program or firmware corresponding to the frequency estimation device 200 described above. The processor 310 executes various functional applications and data processing by running the software program stored in the memory 320 to implement the frequency estimation method provided in the embodiments of the present invention.

[0077] The memory 320 may be, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), FLASH (Flash Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0078] The processor 310 can be an integrated circuit chip with signal processing capabilities, capable of executing software programs, such as the software program corresponding to the frequency estimation device 200 described above. The processor 310 can be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), SoC (System on Chip), etc.; it can also be: DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0079] Understandable. Figure 5 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown. Figure 5 The components shown can be implemented using hardware, software, or a combination thereof.

[0080] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the frequency estimation method disclosed in the above embodiments. The computer-readable storage medium can be, but is not limited to, various media capable of storing program code, such as a USB flash drive, portable hard drive, ROM, RAM, PROM, EPROM, EEPROM, FLASH disk, or optical disk.

[0081] In summary, this invention provides a frequency estimation method, apparatus, electronic device, and computer-readable storage medium. The server's preset storage space stores a redundant frequency counting array. The redundant frequency counting array is a two-dimensional array, where each row is used to independently record the frequency of all data items in the streaming data. Each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different. The method is as follows: receiving a frequency counting request; determining the current data item based on the frequency counting request; the current data item is any type of data item in the streaming data; using a preset hash function family and direction function, determining multiple current mapping positions and multiple current direction values ​​corresponding to the current data item; the current direction value is 1 or -1; adding the element value corresponding to each current mapping position in the redundant frequency counting array to each current direction value to update the element value corresponding to each current mapping position; one of the updated element values ​​corresponding to each current mapping position reflects the estimated frequency of the current data item. In this invention, in each row of the redundant counting array, multiple data items share a position to achieve frequency estimation, and it is ensured that the multiple data items sharing each position in each column are different. In this way, through the redundant counting processing of multiple rows, it is ensured that frequency estimation of massive data items can be achieved compatiblely in a limited storage space.

[0082] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A frequency estimation method, characterized in that, The method is applied to a server, wherein the server's preset storage space is used to store a redundant count array; the redundant count array is a two-dimensional array, wherein each row is used to independently record the frequency of all data items in the streaming data, each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different; the method includes: Receive a count request, and determine the current data item based on the count request; the current data item is any one of the data items in the streaming data. Using a preset family of hash functions and a direction function, multiple current mapping positions and multiple current direction values ​​corresponding to the current data item are determined; the current direction value is 1 or -1. The element value corresponding to each current mapping position in the redundant count array is added to the current direction value to update the element value corresponding to each current mapping position; one of the updated element values ​​corresponding to each current mapping position reflects the estimated frequency of the current data item.

2. The frequency estimation method according to claim 1, characterized in that, The hash function family includes A hash function, the redundant count array includes OK, The step of determining multiple mapping positions and multiple direction values ​​corresponding to the current data item using a preset family of hash functions and direction functions includes: Input the current data items respectively A hash function is obtained. The current hash value; Will Each of the current hash values ​​is moduloed by w to obtain... The current column position; The redundancy count group The lines are respectively with Combine each current column position one by one to get One current mapping position; Will Each of the current hash values ​​is input into the direction function to obtain... The current direction value.

3. The frequency estimation method according to claim 1, characterized in that, The method further includes: Receive a frequency lookup request and determine the target data item based on the frequency lookup request; Using a preset family of hash functions and direction functions, multiple target mapping positions and multiple target direction values ​​corresponding to the target data item are determined; Based on the multiple target mapping positions and multiple target direction values ​​corresponding to the target data item, the estimated frequency of the target data item is determined.

4. The frequency estimation method according to claim 3, characterized in that, The hash function family includes A hash function, the redundant count array includes OK, List; The step of determining multiple target mapping positions and multiple target direction values ​​corresponding to the target data item using a preset family of hash functions and direction functions includes: Input the target data items respectively A hash function is obtained. One target hash value; Will Each target hash value is moduloed by w to obtain... Target column positions; The redundancy count group The lines are respectively with By combining the positions of each target column one by one, we obtain One target mapping location; Will Each target hash value is input into the direction function to obtain... One target direction value; The step of determining the estimated frequency of the target data item based on multiple target mapping positions and multiple target direction values ​​corresponding to the target data item includes: Will The target direction values ​​are respectively related to Multiplying the corresponding element values ​​of each target mapping position in the redundancy count array yields... Frequency of target estimation; The median of the estimated frequencies of the target is used to obtain the estimated frequency of the target data item.

5. The frequency estimation method according to any one of claims 1-4, characterized in that, The hash function family includes d hash functions, and the redundant counting array is... OK, A two-dimensional array of columns; When the maximum capacity of the preset storage space is M satisfy: or .

6. The frequency estimation method according to any one of claims 1-4, characterized in that, The d hash functions in the hash function family are represented as follows: The first of them The hash functions are: In the formula, This represents the input data item. Preset step size; The direction function is: In the formula, This represents the modulo operation. Indicates the calculated first... One direction value.

7. A frequency estimation device, characterized in that, The device is applied to a server, wherein the server's preset storage space is used to store a redundant count array; the redundant count array is a two-dimensional array, wherein each row is used to independently record the frequency of all data items in the streaming data, each element value in each row represents the estimated frequency of multiple data items in the streaming data, and the multiple data items corresponding to each element value in each column are different; the device includes: An update trigger module is used to receive a count request and determine the current data item based on the count request; the current data item is any one of the data items in the streaming data. The position determination module is used to determine multiple current mapping positions and multiple current direction values ​​corresponding to the current data item using a preset family of hash functions and a direction function; the current direction value is 1 or -1. The frequency update module is used to add the element value corresponding to each current mapping position in the redundant count array to each current direction value to update the element value corresponding to each current mapping position; one of the updated element values ​​corresponding to each current mapping position reflects the estimated frequency of the current data item.

8. The frequency estimation device according to claim 7, characterized in that, Also includes: The frequency lookup triggering module is used to receive frequency lookup requests and determine target data items based on the frequency lookup requests; The location determination module is also used to determine multiple target mapping positions and multiple target direction values ​​corresponding to the target data item by using a preset family of hash functions and direction functions; The frequency query module is used to determine the estimated frequency of the target data item based on multiple target mapping positions and multiple target direction values ​​corresponding to the target data item.

9. An electronic device, characterized in that, include: The electronic device includes a memory and a processor, wherein the memory stores a software program, and the processor executes the software program to implement the frequency estimation method as described in any one of claims 1-6 when the electronic device is running.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the frequency estimation method according to any one of claims 1-6.

Images