Data stream processing method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention provides a data stream processing method and device, computer equipment and a readable storage medium, and relates to the field of network measurement. The method comprises the following steps: acquiring a to-be-processed data stream, and determining a plurality of to-be-processed elements from the to-be-processed data stream; performing hash calculation on each to-be-processed element through a quotient and remainder hash function to obtain quotients and remainders of the to-be-processed elements; determining a quotient of the to-be-processed element as a first bucket index, determining a remainder as a first entry index, determining a first hash bucket of the to-be-processed element from the data storage space according to the first bucket index, and determining a first target entry of the to-be-processed element from a plurality of storage entries of the first hash bucket according to the first entry index; and in the first target entry, updating the count of the corresponding to-be-processed element. Therefore, the processing efficiency of the data stream and the utilization rate of the memory can be improved.

Description

Technical Field

[0001] The present application relates to the field of network measurement, and in particular to a data stream processing method, device, computer equipment and readable storage medium. Background Art

[0002] A data stream is a sequence of data that is continuously generated and can be composed of a series of elements that arrive in time order. In the context of data stream processing, an element usually refers to a single data item or record in a data stream. For example, in network traffic monitoring, an element may be an IP address or a data packet; in network log analysis, an element may be a user behavior record.

[0003] In a data stream, some elements may appear repeatedly. By counting these elements, the frequency of occurrence of each element can be estimated to discover abnormal behavior or optimize resource allocation, such as identifying high-frequency IP addresses and popular keywords.

[0004] In the related art, it is generally necessary to perform multiple hash operations to locate and count elements. For example, when using an XOR operation, it is necessary to hash the elements multiple times to obtain multiple hash values, and then use these hash values ​​to determine the location of the elements. However, multiple hash operations increase the computational burden, especially for high-speed data streams that need to be processed in real time, which significantly reduces processing efficiency; on the other hand, since each hash operation requires at least one memory access, multiple hash operations will affect the processing efficiency of the data stream and reduce memory utilization. In summary, the efficiency of data stream processing and memory utilization in the related art are low. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a data stream processing method, apparatus, computer equipment and readable storage medium, which can improve the processing efficiency of data streams and the utilization rate of memory.

[0006] To achieve the above object, a first aspect of an embodiment of the present application provides a data stream processing method, the method comprising:

[0007] Acquire a data stream to be processed, and determine a plurality of elements to be processed from the data stream to be processed;

[0008] Performing hash calculation on each of the elements to be processed by using a quotient-remainder hash function to obtain a quotient and a remainder of the element to be processed;

[0009] Determine the quotient of the element to be processed as a first bucket index, determine the remainder of the element to be processed as a first entry index, determine a first hash bucket of the element to be processed from a data storage space according to the first bucket index, and determine a first target entry of the element to be processed from a plurality of storage entries of the first hash bucket according to the first entry index;

[0010] In the first target entry, the count value of the corresponding element to be processed is updated.

[0011] Accordingly, a second aspect of an embodiment of the present application provides a data stream processing device, the device comprising:

[0012] An acquisition module, used for acquiring a data stream to be processed and determining a plurality of elements to be processed from the data stream to be processed;

[0013] A calculation module, used for performing hash calculation on each of the elements to be processed by using a quotient and remainder hash function to obtain the quotient and remainder of the element to be processed;

[0014] a determination module, configured to determine the quotient of the element to be processed as a first bucket index, determine the remainder of the element to be processed as a first entry index, determine a first hash bucket of the element to be processed from a data storage space according to the first bucket index, and determine a first target entry of the element to be processed from a plurality of storage entries of the first hash bucket according to the first entry index;

[0015] An updating module is used to update the count value of the corresponding element to be processed in the first target entry.

[0016] In some implementations, the data stream processing device further includes a removal module, configured to:

[0017] Obtaining a total storage resource occupancy of a plurality of the storage entries in the first hash bucket;

[0018] When the total storage resource occupancy exceeds the storage space of the first hash bucket, at least one storage entry is removed from the first hash bucket according to the count value of each storage entry to release the storage space of the first hash bucket.

[0019] In some embodiments, the removal module is further used for:

[0020] When the total storage resource occupancy exceeds the storage space of the first hash bucket, determining a first storage entry with a minimum count value;

[0021] removing the first storage entry from the first hash bucket;

[0022] Repeating the step of removing the smallest first storage entry from the first hash bucket until the total storage resource occupancy is less than the storage space of the first hash bucket, thereby obtaining a first hash bucket with released storage space.

[0023] In some implementations, the data stream processing device further includes a storage module, configured to:

[0024] When the first target entry corresponding to the first entry index cannot be matched from the first hash bucket, obtaining a filling state attribute of the first hash bucket corresponding to the first bucket index;

[0025] When the filling state attribute indicates that there is available storage space in the first hash bucket, a second storage entry corresponding to the first entry index and the count value of the element to be processed is stored in the first hash bucket.

[0026] In some embodiments, the storage module is further used for:

[0027] When the filling state attribute indicates that there is no available storage space in the first hash bucket, obtaining a count value of each storage entry in the first hash bucket;

[0028] When there is a third storage entry whose count value is less than or equal to the count value of the element to be processed, removing the corresponding third storage entry from the first hash bucket to release storage space of the first hash bucket;

[0029] A second storage entry corresponding to the first entry index and the count value of the element to be processed is stored in the first hash bucket.

[0030] In some implementations, the data stream processing device further includes a sorting module, configured to:

[0031] Obtaining a count value of each storage entry in the first hash bucket;

[0032] Bubble sort each of the storage entries based on the count value to obtain a sorting result;

[0033] Based on the sorting result, the position of each storage entry in the first hash bucket is adjusted to obtain an adjusted first hash bucket; wherein the count value of each storage entry in the first hash bucket is positively correlated with the first hash bucket level.

[0034] In some implementations, the count value of each storage entry in each hash bucket is positively correlated with the level of the hash bucket; the data stream processing device further includes a push module, which is used to:

[0035] The method further comprises:

[0036] For each current hash bucket, obtain the updated first count value of the fourth storage entry corresponding to the lowest level of the hash bucket;

[0037] Determine, from the data storage space, a candidate hash bucket for the element corresponding to the fourth storage entry, and a second count value corresponding to the highest level of the candidate hash bucket;

[0038] When the second count value is smaller than the first count value, remove the storage entry with the smallest count value in the candidate hash bucket;

[0039] Pushing down each storage entry in the candidate hash bucket one level in turn to obtain the highest level in the candidate hash bucket that is in an empty state;

[0040] The fourth storage entry is stored in the highest level of the candidate hash bucket.

[0041] In some implementations, the data stream processing device further includes a query module, configured to:

[0042] Get the element to be queried;

[0043] Performing hash calculation on the element to be queried by using a quotient-remainder hash function to obtain a quotient and a remainder of the element to be queried;

[0044] Determine the quotient of the element to be queried as a second bucket index, determine the remainder as a second entry index, determine a query hash bucket of the element to be queried according to the second bucket index, and determine a query entry of the element to be queried from the query hash bucket according to the second entry index;

[0045] The count value of the element to be queried is read from the query entry.

[0046] Correspondingly, the third aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the data stream processing method described in any one of the embodiments of the first aspect of the present application when executing the computer program.

[0047] Correspondingly, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the data stream processing method described in any one of the embodiments of the first aspect of the present application.

[0048] The embodiment of the present application obtains a data stream to be processed, and determines multiple elements to be processed from the data stream to be processed; performs hash calculation on each element to be processed through a quotient-remainder hash function to obtain the quotient and remainder of the element to be processed; determines the quotient of the element to be processed as the first bucket index, determines the remainder as the first entry index, and determines the first hash bucket of the element to be processed from the data storage space according to the first bucket index, and determines the first target entry of the element to be processed from multiple storage entries of the first hash bucket according to the first entry index; in the first target entry, updates the count value of the corresponding element to be processed. In this way, the bucket index and entry index of the element to be processed can be obtained by using the quotient-remainder hash, and the corresponding hash bucket and entry can be directly determined through the bucket index and entry index, thereby reducing the number of hash operations, optimizing the memory access mode, and thus improving the processing efficiency of the data stream, while also improving the memory utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the architecture of the data stream processing system provided in an embodiment of the present application;

[0050] Figure 2 is a flow chart of a data stream processing method provided by an embodiment of the present application;

[0051] Figure 3 It is a schematic diagram of element insertion of the data stream processing method provided in an embodiment of the present application;

[0052] Figure 4 It is another element insertion schematic diagram of the data stream processing method provided in the embodiment of the present application;

[0053] Figure 5 It is another element insertion schematic diagram of the data stream processing method provided in the embodiment of the present application;

[0054] Figure 6 It is a functional module diagram of a data stream processing device provided in an embodiment of the present application;

[0055] Figure 7 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] A data stream is a sequence of data that is continuously generated and can be composed of a series of elements that arrive in time order. In the context of data stream processing, an element usually refers to a single data item or record in a data stream. For example, in network traffic monitoring, an element may be an IP address or a data packet; in network log analysis, an element may be a user behavior record.

[0060] In a data stream, some elements may appear repeatedly. By counting these elements, the frequency of occurrence of each element can be estimated to discover abnormal behavior or optimize resource allocation, such as identifying high-frequency IP addresses and popular keywords.

[0061] In the related art, it is generally necessary to locate and count elements through multiple hash operations. For example, when using an XOR operation, it is necessary to hash the elements multiple times to obtain multiple hash values. Specifically, the first hash value is obtained using the first hash function, the second hash value is obtained using the second hash function, the hash table is searched according to the first hash bucket, the corresponding bucket list is obtained, and the bucket list is searched according to the second hash value to obtain the address information stored in the corresponding hash bucket. However, multiple hash operations increase the computational burden, especially for high-speed data streams that need to be processed in real time, which will significantly reduce processing efficiency; on the other hand, since each hash operation requires at least one memory access, multiple hash operations will affect the processing efficiency of the data stream and reduce memory utilization. In summary, the efficiency of data stream processing and memory utilization in the related art are low.

[0062] Based on this, the embodiments of the present application provide a data stream processing method, apparatus, computer equipment and readable storage medium, which can improve the processing efficiency of data streams and the utilization rate of memory.

[0063] The data stream processing method, apparatus, computer device and readable storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data stream processing system in the embodiments of the present application is described.

[0064] Please refer to Figure 1 In some implementations, an embodiment of the present application provides a data stream processing system, including a terminal 11 and a server 12.

[0065] Specifically, the terminal 11 may be a device that generates or receives a data stream and sends the data stream to the server for processing. Exemplarily, the terminal 11 may be a user's smart phone, personal computer, IoT device, etc. The terminal 11 collects or generates a raw data stream, formats the raw data stream into a form suitable for transmission, and then sends the processed data stream to the server 12 for further processing.

[0066] Exemplarily, the server end 12 may be a server cluster in a data center, or a computing resource provided by a cloud service provider. The server end 12 may be responsible for receiving the data stream from the terminal 11, and counting and frequency counting the elements to be processed in the data stream. Furthermore, the server end 12 may also store the processed data or intermediate results, and feed back the processing results (such as a list of the most frequent elements) to the terminal 11 or other system components as needed. At the same time, the server end 12 also supports users to query the processed data.

[0067] The data stream processing method in the embodiment of the present application can be illustrated by the following embodiment.

[0068] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0069] In the embodiments of the present application, the data stream processing device will be described from the perspective of the data stream processing device, which can be integrated into a computer device. Figure 2 , Figure 2 This is a flowchart of the steps of the data stream processing method provided in an embodiment of the present application. In the embodiment of the present application, the data stream processing device is specifically integrated in a terminal or a server as an example. When the processor on the terminal or the server executes the program instructions corresponding to the data stream processing method, the specific process is as follows:

[0070] Step 101: Obtain a data stream to be processed, and determine a plurality of elements to be processed from the data stream to be processed.

[0071] In some embodiments, in order to count the frequency of occurrence of each element in the data stream, the data stream to be processed can be obtained, and multiple elements to be processed can be determined from the data stream to be processed, so as to analyze the frequency of occurrence of the elements to be processed and discover important patterns, anomalies or trends.

[0072] The data stream to be processed may be a continuously generated data sequence, and the data sequence may be a network data packet, a log record, a sensor data, etc.

[0073] The element to be processed may be a single data item or record extracted from the data stream to be processed, and the element to be processed is an object that needs to be counted.

[0074] Exemplarily, a network monitoring tool can be used to capture a network data packet flow from a data source, analyze the data flow, and extract multiple elements to be processed from the data flow. The elements to be processed may be IP addresses, URLs, keywords, etc. For example, when the element to be processed is an IP address and it is necessary to count the frequency of occurrence of the IP address, a network monitoring tool can be used to capture the network data packet flow, and extract the source IP address and the destination IP address from each data packet, and use these IP addresses as elements to be processed.

[0075] By obtaining a data stream to be processed and determining a plurality of elements to be processed from the data stream to be processed, it is possible to facilitate subsequent calculation of the elements to be processed by using a quotient remainder hash function.

[0076] Step 102: Perform hash calculation on each element to be processed by using a quotient-remainder hash function to obtain the quotient and remainder of the element to be processed.

[0077] In some embodiments, in order to enable the system to process data streams more efficiently and quickly and accurately count the frequency of occurrence of each element to be processed, each element to be processed can be hashed using a quotient and remainder hash function to obtain the quotient and remainder of the element to be processed, thereby improving the processing efficiency of the elements to be processed.

[0078] The quotient-remainder hash function may be a special hash function that can simultaneously output the quotient and remainder of the element to be processed after performing a hash operation on the element to be processed.

[0079] Exemplarily, the hash value of the element to be processed may be calculated by a quotient-remainder hash function, and the hash value may be divided into a quotient and a remainder, thereby obtaining the quotient and the remainder of the element to be processed.

[0080] Exemplarily, for an element to be processed e, the quotient and remainder hash value of the element to be processed may be calculated as h(e), and then the hash value may be divided into a quotient and a remainder. Specifically, the quotient Q(e) may be calculated by the following formula:

[0081]

[0082] The remainder Q(e) can be calculated by the following formula:

[0083] Q(e)=h(e)mod 2 r

[0084] Among them, r represents the number of bits of the quotient and remainder hash value. In order to avoid hash conflicts or problems where the quotient and remainder are too large to match, the value of r can be selected according to the actual calculation scenario.

[0085] By performing hash calculation on each element to be processed through the quotient and remainder hash function, the quotient and remainder of the element to be processed are obtained, which can facilitate the rapid determination of the counting position of the element to be processed in the data storage space.

[0086] Step 103, determine the quotient of the element to be processed as the first bucket index, determine the remainder as the first entry index, and determine the first hash bucket of the element to be processed from the data storage space according to the first bucket index, and determine the first target entry of the element to be processed from multiple storage entries of the first hash bucket according to the first entry index.

[0087] In some embodiments, in order to simplify the positioning and processing of elements to be processed and improve the efficiency and accuracy of data stream processing, the first hash bucket in which the elements to be processed are stored can be determined by taking the quotient of the elements to be processed as the first bucket index, and the position of the entry corresponding to the elements to be processed in the first hash bucket can be determined by taking the remainder of the elements to be processed as the first entry index. In this way, only one hash calculation is needed to quickly locate the entry of the elements to be processed in the data storage space.

[0088] The first bucket index may be a quotient obtained by performing a hash calculation on the element to be processed through a quotient-remainder hash function, and is used to determine the first hash bucket corresponding to the element to be processed from the data storage space.

[0089] The first entry index may be a remainder obtained by performing a hash calculation on the element to be processed through a quotient remainder hash function, and is used to determine the first target entry corresponding to the element to be processed from the first hash bucket.

[0090] The data storage space may be a storage space for buckets for storing multiple elements and their count values, and the data storage space includes at least a first array and a second array, wherein the first array is used to store a primary hash bucket, and the second array is used to store an alternative hash bucket, so that when the primary hash bucket cannot store an entry corresponding to an element to be processed, the entry of the element to be processed is stored in the alternative hash bucket of the element to be processed. Each array has multiple hash buckets.

[0091] The first hash bucket may be a hash bucket determined by a quotient obtained by calculating the quotient residue hash function on the element to be processed. Specifically, for the element to be processed, the quotient calculated by the hash function is used as the first bucket index, thereby determining the first hash bucket corresponding to the element to be processed from the first array in the data storage space. Furthermore, the candidate hash bucket of the element to be processed can also be determined from the second array of the data storage space at the same time through the first bucket index.

[0092] The storage entry may refer to a basic storage unit in each hash bucket, and each storage entry stores an entry index of an element (ie, the remainder of the element to be processed) and a corresponding count value.

[0093] The first target entry may be a storage entry for counting the elements to be processed, which is determined from a plurality of storage entries in the first hash bucket through the first entry index.

[0094] For example, when the quotient and remainder hash value of the element to be processed is divided to obtain a quotient of 7 and a remainder of 9, the first hash bucket can be determined from the first array of the data storage space according to the quotient, and the candidate hash bucket can be determined from the second array. Since the main hash bucket, i.e., the first hash bucket, is generally used to count the elements to be processed, the first target entry for counting the elements to be processed can be further determined in the first hash bucket according to the remainder.

[0095] For example, when the quotient is 7 and the remainder is 9, the 7th hash bucket in the first array can be determined as the first hash bucket, and the 9th entry in the first hash bucket can be determined as the first target entry.

[0096] By determining the quotient of the element to be processed as the first bucket index, determining the remainder as the first entry index, and determining the first hash bucket of the element to be processed from the data storage space according to the first bucket index, and determining the first target entry of the element to be processed from multiple storage entries of the first hash bucket according to the first entry index, the quotient and remainder obtained by a hash calculation can quickly locate the position of the element to be processed in the data storage space, reducing unnecessary memory access and hash operations, reducing the number of memory accesses, and improving data processing speed, which can greatly improve the processing efficiency of real-time, large-scale data streams.

[0097] Step 104: in the first target entry, update the count value of the corresponding element to be processed.

[0098] In some embodiments, in order to accurately track the frequency of occurrence of each element in the data stream and provide accurate frequency statistics when needed, the count value can be updated in the corresponding first target entry for each element to be processed to ensure the accuracy of the count value statistics of the element to be processed.

[0099] The count value can be a value used to record the number of times an element appears in a data stream in data stream processing. The count value is a basic indicator in data stream processing and can be used to measure the frequency of an element, that is, the number of times the element appears in a data stream.

[0100] Exemplarily, when the original count value of the first target entry is 85, when a to-be-processed element appears in the data stream, the count value of the to-be-processed element in the first target entry is increased by 1, and the updated count value of the to-be-processed element is 86.

[0101] Furthermore, during the processing, it is possible to check whether the current element to be processed is the same as the subsequent element to be processed. If they are the same, there is no need to repeat the quotient and remainder hash calculation, and continuous counting can be performed directly in the first target entry determined by the first identical element to be processed, so as to improve the processing efficiency of the data stream.

[0102] In some implementations, the size of the counter in each storage entry can be set for different traffic scenarios. For example, in a high-traffic scenario, a larger counter can be set to accurately count the number of occurrences of high-frequency elements, maintaining high accuracy even in high concurrency. In a low-traffic scenario, a smaller counter can be set to reduce storage space usage while still meeting statistical requirements.

[0103] It is understandable that in traditional data stream processing methods, such as using counters to count the frequency of occurrence of elements, because it is more efficient to use shift operations to access memory addresses in hardware implementation, and in order to simplify it to bitwise AND operations, which correspond to powers of 2, the size of the counter is often set to a power of 2. However, setting the size of the counter to a power of 2 will limit the flexibility of the system, making it difficult for the counter to adapt to data streams of different sizes, resulting in a waste of storage space. As a result, it is not possible to set the size of the counter in each storage entry for different traffic scenarios.

[0104] The present application locates elements by using the quotient and remainder calculated by the quotient-remainder hash function, thereby realizing fast element insertion and query operations. At this time, the location of the element to be processed is no longer limited by the requirement that the size of the counter must be a power of 2, and the size of the counter can be adjusted more flexibly, for example, the size of the counter is set to 10, 12, etc. In this way, the size of the counter in each storage entry can be set according to different traffic scenarios, thereby improving the flexibility and efficiency of the system.

[0105] The embodiment of the present application obtains a data stream to be processed, and determines multiple elements to be processed from the data stream to be processed; performs hash calculation on each element to be processed through a quotient-remainder hash function to obtain the quotient and remainder of the element to be processed; determines the quotient of the element to be processed as the first bucket index, determines the remainder as the first entry index, and determines the first hash bucket of the element to be processed from the data storage space according to the first bucket index, and determines the first target entry of the element to be processed from multiple storage entries of the first hash bucket according to the first entry index; in the first target entry, updates the count value of the corresponding element to be processed. In this way, the bucket index and entry index of the element to be processed can be obtained by using the quotient-remainder hash, and the corresponding hash bucket and entry can be directly determined through the bucket index and entry index, thereby reducing the number of hash operations, optimizing the memory access mode, and thus improving the processing efficiency of the data stream, while also improving the memory utilization.

[0106] In some implementations, as data continues to flow in, the storage entries in the first hash bucket will be continuously filled and updated, and the total storage resource usage of the first hash bucket will exceed the storage space of the first hash bucket. Therefore, the storage space of the first hash bucket can be released to ensure the stability and efficiency of data flow processing. For example, after step 104, it also includes:

[0107] (A1) obtaining a total storage resource occupancy of a plurality of storage entries in a first hash bucket;

[0108] (A2) When the total storage resource occupancy exceeds the storage space of the first hash bucket, at least one storage entry is removed from the first hash bucket according to the count value of each storage entry to release the storage space of the first hash bucket.

[0109] The total storage resource occupancy may be the total actual storage space occupied by all storage entries in the first hash bucket. The total storage resource occupancy includes the space occupied by the entry index (also referred to as element fingerprint) and count value of the elements stored in all storage entries, as well as the space occupied by other related metadata (such as element identifiers, etc.).

[0110] The storage space may be the size of an available storage area allocated to the first hash bucket.

[0111] Exemplarily, all storage entries in the first hash bucket may be traversed, the space occupied by the count value of each storage entry and the space occupied by the entry index corresponding to each element may be calculated, and the total storage resource occupancy of the first hash bucket may be accumulated.

[0112] Furthermore, when the total storage resource occupancy exceeds the storage space of the first hash bucket, the storage entry with the smallest count value can be moved out of the first hash bucket. Specifically, the storage entry with the smallest count value can be discarded or moved to the alternative hash bucket of the corresponding storage entry. In this way, storage space can be released while retaining more valuable high-frequency elements as much as possible.

[0113] Exemplarily, there are three storage entries in the first hash bucket, among which the count value of storage entry 1 is 200, the count value of storage entry 2 is 50, and the count value of storage entry 3 is 5. Then, when the total storage resource occupancy exceeds the storage space of the first hash bucket, storage entry 3 can be removed from the first hash bucket to release the storage space of the first hash bucket and ensure that high-frequency elements are retained.

[0114] By optimizing the storage space of the first hash bucket according to the count value of each storage entry, the processing performance of the system for processing large-scale data streams and high-frequency elements can be improved, and the stability of the system and the efficiency of processing data streams are improved.

[0115] In some implementations, in order to ensure the stability and efficiency of data stream processing, when the total storage resource occupancy exceeds the storage space of the first hash bucket, the storage entry with the smallest count value can be removed from the first hash bucket to avoid the first hash bucket being unable to continue processing new data stream elements due to insufficient storage space, thereby ensuring that the data stream processing system can maintain stable operation when processing large amounts of data. For example, A2 may include:

[0116] (A2.1) when the total storage resource occupancy exceeds the storage space of the first hash bucket, determining a first storage entry with a minimum count value;

[0117] (A2.2) removing the first storage entry from the first hash bucket;

[0118] (A2.3) Repeatedly remove the smallest first storage entry from the first hash bucket until the total storage resource occupancy is less than the storage space of the first hash bucket, thereby obtaining the first hash bucket with released storage space.

[0119] The first storage entry may be a storage entry with the smallest count value in the first hash bucket. After the first storage entry is removed from the first hash bucket, the first storage entry may be re-determined in the first hash bucket.

[0120] Exemplarily, when the total storage resource occupancy exceeds the storage space of the first hash bucket, all storage entries in the first hash bucket can be traversed to find the first storage entry with the smallest count value, and the first storage entry can be removed from the first hash bucket. At this time, the total storage resource occupancy of the first hash bucket continues to be compared with the storage space of the first hash bucket. When the total storage resource occupancy still exceeds the storage space of the first hash bucket, the smallest first storage entry is repeatedly removed from the first hash bucket until the total storage resource occupancy is less than the storage space of the first hash bucket, or the ratio of the total storage resource occupancy to the storage space of the first hash bucket is lower than a preset ratio, then the removal of the first storage entry can be stopped, and the first hash bucket with released storage space is obtained.

[0121] Exemplarily, if the first hash bucket has 5 storage entries, where the count value of storage entry 1 is 86, the count value of storage entry 2 is 70, the count value of storage entry 3 is 52, the count value of storage entry 4 is 15, and the count value of storage entry 5 is 2, when the total storage resource occupancy exceeds the storage space of the first hash bucket, storage entry 5 can be determined as the first storage entry based on the count value, and at this time, storage entry 5 is removed from the first hash bucket. When the total storage resource occupancy at this time still exceeds the storage space of the first hash bucket, storage entry 4 can be determined as the first storage entry based on the count value, and at this time, storage entry 4 is removed from the first hash bucket. If the total storage resource occupancy at this time is less than the storage space of the first hash bucket, the determination of the first storage entry can be stopped to obtain the first hash bucket with released storage space.

[0122] Exemplarily, the preset ratio can be set according to actual conditions. For example, the preset ratio can be 3 / 4. That is to say, when the ratio of the total storage resource occupancy to the storage space of the first hash bucket is lower than 3 / 4, the removal of the first storage entry can be stopped to obtain the first hash bucket with released storage space.

[0123] In some embodiments, the first storage entry that is removed can be discarded directly, or its corresponding alternative hash bucket can be determined through a quotient hash function, and when there is a vacant space in the alternative hash bucket, the alternative hash bucket can be inserted. When there is no vacant space in the alternative hash bucket, the storage entry with the smallest count value can also be removed from the alternative hash bucket.

[0124] Furthermore, if the same first storage entry still cannot be stored in other hash buckets after trying the preset number of attempts, the first storage entry can be discarded. Exemplarily, the preset number of attempts k can be determined by the following formula:

[0125]

[0126] Here, α is a load factor, which affects the frequency and number of times of removing a first storage entry. The value of α can be adjusted according to actual conditions.

[0127] It can be understood that the above embodiment is applicable to all hash buckets that need to release storage space.

[0128] By releasing the storage space of the first hash bucket, problems such as data processing interruption due to insufficient storage space can be avoided, while optimizing the use of storage resources and improving the overall efficiency of data stream processing.

[0129] In some embodiments, when the first target entry corresponding to the first entry index cannot be matched from the first hash bucket, a match can be first performed in the alternative hash bucket of the element to be processed. If the first target entry corresponding to the first entry index exists in the alternative hash bucket, the count value of the element to be processed is updated in the alternative hash bucket. If the first target entry corresponding to the first entry index still cannot be matched from the alternative hash bucket, the first target entry is inserted into the main hash bucket, that is, the first hash bucket, so that the element to be processed is counted in the main hash bucket first. Alternatively, it can also be counted in the alternative hash bucket, and the embodiment of the present application does not impose specific restrictions on this.

[0130] In some embodiments, when the first target entry corresponding to the first entry index cannot be matched from the first hash bucket, it indicates that there is no storage entry related to the element to be processed in the first hash bucket. At this time, the fill state attribute of the first hash bucket can be judged, and it can be determined whether the second storage entry corresponding to the element to be processed can be stored in the hash bucket according to the fill state attribute, so as to ensure the stability and efficiency of the data stream processing system and avoid data processing interruption caused by insufficient storage space. Exemplarily, the data stream processing method also includes the following situations:

[0131] (B.1) when the first target entry corresponding to the first entry index cannot be matched from the first hash bucket, obtaining a fill state attribute of the first hash bucket corresponding to the first bucket index;

[0132] (B.2) When the fill state attribute indicates that there is available storage space in the first hash bucket, a second storage entry corresponding to the first entry index and the count value of the element to be processed is stored in the first hash bucket.

[0133] The fill state attribute may be an attribute for describing the current fill level of the first hash bucket, and is used to indicate the state of available storage space in the first hash bucket. The fill state attribute is usually used to determine whether the first hash bucket still has available entry space to store the entry index and count value of the new element to be processed.

[0134] The second storage entry may be a second storage entry composed of the first entry index of the element to be processed and the count value of the element to be processed when the first target entry corresponding to the first entry index cannot be matched from the first hash bucket.

[0135] Specifically, the fill status attribute can be a simple Boolean value (true or false) or a specific numerical value indicating the size of the remaining space, for example, it can be the percentage of the remaining available space. When the percentage of the remaining available space is greater than a percentage threshold (for example, 20%), it indicates that there is available storage space in the first hash bucket.

[0136] Exemplarily, if the element to be processed is the IP address 192.168.1.2, this address appears for the first time. By calculating the quotient and remainder of the element to be processed, and determining the first hash bucket in the data storage space by using the quotient as the first bucket index, at this time, since the element to be processed appears for the first time, it is impossible to match the first target entry corresponding to the first entry index from the first hash bucket. At this time, the filling state attribute of the first hash bucket can be obtained. For example, if it is assumed that there are 100 storage entries in the first hash bucket, the storage space occupied by each storage entry is 4 bytes (count value) + 16 bytes (entry index) = 20 bytes. If 90 storage entries in the first hash bucket have been used, then the number of remaining available storage entries is 10, that is, there is available storage space in the first hash bucket. At this time, the second storage entry corresponding to the first entry index and the count value of the element to be processed can be stored in the first hash bucket. Due to the initial count, at this time, in the second storage entry of the hash bucket, the count value is generally 1.

[0137] Through the above method, the stability and efficiency of the data stream processing system can be effectively ensured. By checking the filling status attribute of the first hash bucket, it is determined whether the new second storage entry can be stored in the first hash bucket, thereby avoiding the problem of data processing interruption caused by insufficient storage space. At the same time, the use of storage resources is optimized and the overall efficiency of data stream processing is improved.

[0138] In some embodiments, as data continues to flow in, the storage entries in the first hash bucket will be continuously filled and updated. When the first target entry corresponding to the first entry index cannot be matched from the first hash bucket, and the filling state attribute indicates that there is no available storage space in the first hash bucket, the storage entry with a count value less than the element to be processed can be removed from the first hash bucket to release the storage space of the first hash bucket so that the new second storage entry can be stored in the first hash bucket. For example, after B.1, that is, the above B.2, can be replaced by the following implementation:

[0139] (C.1) when the fill state attribute indicates that there is no available storage space in the first hash bucket, obtaining a count value of each storage entry in the first hash bucket;

[0140] (C.2) when there is a third storage entry whose count value is less than or equal to the count value of the element to be processed, the corresponding third storage entry is removed from the first hash bucket to release storage space of the first hash bucket;

[0141] (C.3) The second storage entry corresponding to the first entry index and the count value of the element to be processed is stored in the first hash bucket.

[0142] The third storage entry may be a storage entry in the first hash bucket whose count value is smaller than the count value of the element to be processed.

[0143] Specifically, when the first target entry corresponding to the first entry index cannot be matched from the first hash bucket, and the fill state attribute indicates that there is no available storage space in the first hash bucket, the storage space of the first hash bucket can be released by removing the third storage entry with the smallest count value from the first hash bucket, and new elements can be retained as much as possible to better allocate limited computing and storage resources.

[0144] Exemplarily, the element to be processed is a new IP address 192.168.1.2, which appears for the first time. At this time, the quotient and remainder of the element to be processed are calculated by the quotient-remainder hash function to determine the first hash bucket corresponding to the element to be processed. However, since the element to be processed appears for the first time, there is no corresponding entry in the first hash bucket.

[0145] Further, when the fill state attribute indicates that there is no available storage space in the first hash bucket, a count value of each storage entry in the first hash bucket is obtained. Assuming that there are 100 storage entries in the first hash bucket and all 100 storage entries are used, there are no remaining available storage entries.

[0146] At this time, when there is a third storage entry whose count value is less than or equal to the count value of the element to be processed, for example, the count value of the element to be processed is 1 and the count value of the third storage entry is 1, the third storage entry is removed from the first hash bucket to release the storage space of the first hash bucket so that the first hash bucket can continue to process new data stream elements.

[0147] Furthermore, after the storage space of the first hash bucket is released, the second storage entry corresponding to the first entry index and the count value of the element to be processed may be stored in the first hash bucket.

[0148] By releasing the storage space in the first hash bucket to store new entries, data processing interruption due to insufficient storage space can be effectively avoided, while optimizing the use of storage resources and improving the overall efficiency of data stream processing.

[0149] In some implementations, in order to ensure that high-frequency elements are preferentially stored in the first hash bucket, each storage entry may be bubble sorted based on the count value at regular intervals, or each time the count value of the storage entry in the first hash bucket is updated, each storage entry may be bubble sorted based on the count value, thereby effectively optimizing the layout of the storage entries in the first hash bucket and improving the overall performance of the data stream processing system. Exemplarily, the data stream processing method may also include:

[0150] (D.1) Obtaining the count value of each storage entry in the first hash bucket;

[0151] (D.2) Bubble sort each storage entry based on the count value to obtain a sorting result;

[0152] (D.3) Based on the sorting result, the position of each storage entry in the first hash bucket is adjusted to obtain an adjusted first hash bucket; wherein the count value of each storage entry in the first hash bucket is positively correlated with the level of the first hash bucket.

[0153] The sorting result may be a result obtained by sorting the storage entries in the first hash bucket based on the count value, and the sorting result is used to guide the position of the storage entries in the first hash bucket to optimize storage space usage and improve data processing efficiency.

[0154] Exemplarily, if there are 3 storage entries in the first hash bucket, the count value of storage entry a at the highest level of the first hash bucket is 100, the count value of storage entry b below storage entry a is 33, and the count value of storage entry c below storage entry b (that is, the storage entry at the lowest level of the first hash bucket) is 33. Further, if the count value corresponding to storage entry c increases by 2 to 35 after acquiring the real-time data stream, then bubble sort is performed on storage entry a, storage entry b, and storage entry c, and the sorting result indicates that the count value of storage entry b is less than the count value of storage entry c.

[0155] Furthermore, based on the sorting results, the storage entries that need to be adjusted can be determined, and the positions of the corresponding storage entries in the hash bucket can be adjusted, thereby obtaining storage entry a, storage entry c, and storage entry b, wherein storage entry a is located at the highest level of the first hash bucket, and storage entry c is located at the lowest level of the first hash bucket.

[0156] It is understandable that the above solution of adjusting the position of each storage entry in the hash bucket by bubble sort is not only applicable to the first hash bucket, but can also be used in other hash buckets to optimize the internal storage of each hash bucket and retain high-frequency data as much as possible.

[0157] By bubbling, the most frequent elements are adjusted to the highest level of the first hash bucket, and the least frequent elements are adjusted to the lowest level of the first hash bucket, so that the layout of the storage entries in the first hash bucket can be effectively optimized, ensuring that the high-frequency elements are stored preferentially and improving the data processing efficiency.

[0158] In some embodiments, in order to facilitate the extraction of high-frequency elements, the count value of each storage entry in each hash bucket is positively correlated with the level of the hash bucket, so as to directly extract the count value of the high-frequency element at the highest level of the hash bucket. In order to effectively optimize the layout of the storage entries between multiple hash buckets and ensure that high-frequency elements are stored preferentially, the storage position of the fourth storage entry corresponding to the lowest level of the hash bucket in the corresponding alternative hash bucket can be re-determined to avoid the high-frequency element being mistakenly discarded when a new element with a large count value is inserted into the current hash bucket. Exemplarily, the data stream processing method also includes:

[0159] (E.1) for each current hash bucket, obtaining the updated first count value of the fourth storage entry corresponding to the lowest level of the hash bucket;

[0160] (E.2) determining, from the data storage space, a candidate hash bucket for the element corresponding to the fourth storage entry, and a second count value corresponding to the highest level of the candidate hash bucket;

[0161] (E.3) When there is a second count value that is smaller than the first count value, remove the storage entry with the smallest count value in the candidate hash bucket;

[0162] (E.4) Push down each storage entry in the candidate hash bucket one level in turn to obtain the highest level in the candidate hash bucket that is vacant;

[0163] (E.5) The fourth storage entry is stored in the highest level of the candidate hash bucket.

[0164] Among them, the fourth storage entry can be a storage entry at the lowest level in the current hash bucket. Since the count value of each storage entry in each hash bucket is positively correlated with the level of the hash bucket, the count value of the storage entry at the lowest level is also the smallest.

[0165] The first count value may be a count value of a fourth storage entry at the lowest level in the current hash bucket.

[0166] The second count value may be a count value corresponding to the highest level in the candidate hash bucket that can be used to store the fourth storage entry and is determined in the data space according to the bucket index of the fourth storage entry.

[0167] The candidate hash bucket may be a candidate hash bucket of a hash bucket in another array. Exemplarily, the quotient of element a may be used as a bucket index to determine a hash bucket in the first array and a candidate hash bucket in the second array. The hash bucket and the corresponding candidate hash bucket may both be used to store the entry index of element a and count element a by 3.

[0168] Exemplarily, for the current hash bucket 4, if hash bucket 4 has 3 storage entries, corresponding to 3 levels, from the highest level to the lowest level, the count values ​​corresponding to the storage entries are: the count value corresponding to storage entry a1 is 198, the count value corresponding to storage entry a2 is 86, and the count value corresponding to storage entry a3 is 52. The lowest level of hash bucket 4 corresponds to storage entry a3, and storage entry a3 is the fourth storage entry, and the corresponding first count value is 52.

[0169] If storage entry a3 corresponds to element a, by performing quotient remainder hash calculation on element a, hash bucket 4 of element a in the first array of the data storage space and candidate hash bucket 2 in the second array can be determined according to the calculated bucket index.

[0170] Furthermore, if candidate hash bucket 2 has 4 storage entries, corresponding to 4 levels, from the highest level to the lowest level, the count values ​​corresponding to the storage entries are: storage entry b1 corresponds to a count value of 33, storage entry b2 corresponds to a count value of 21, storage entry b3 corresponds to a count value of 8, and storage entry b4 corresponds to a count value of 2. The storage entry corresponding to the highest level of candidate hash bucket 2 is storage entry b1, and the second count value is 33.

[0171] By comparison, it can be determined that the second count value (33) is less than the first count value (52), then the smallest storage entry in the candidate hash bucket 2 can be removed, that is, the storage entry b4 can be removed. And each storage entry in the candidate hash bucket is pushed down one level in turn. Specifically, the storage entry b3 can be pushed down to the original position of the storage entry b4, the storage entry b2 can be pushed down to the original position of the storage entry b3, and the storage entry b1 can be pushed down to the original position of the storage entry b2. Thus, the highest level in the candidate hash bucket 2 that is in a vacant state is obtained, that is, the original position of the storage entry b1 that is in a vacant state.

[0172] Furthermore, the fourth storage entry, that is, the storage entry a3, may be stored in the original position of the storage entry b1 which is in a vacant state.

[0173] It is understandable that the above solution is only used as an example. In fact, for each hash bucket, the above method can be used to optimize the storage inside the hash bucket, which will not be described in detail here.

[0174] Through the above methods, the storage space of the hash bucket can be effectively managed and optimized, and the relevant data of more frequently appearing elements can be stored in the higher level of the hash bucket as much as possible to improve the retrieval speed of high-frequency elements, so that the storage mode of the hash bucket can better adapt to the changes in data scale and access mode.

[0175] In some embodiments, since the present application uses a quotient-residual hash function to calculate the hash value of an element, and stores the relevant data of the element according to the bucket index and entry index divided by the hash value, it is possible not only to identify hot elements, but also to query the count value of each element according to the bucket index and entry index calculated by the quotient-residual hash function. Exemplarily, in the first target entry, after updating the count value of the corresponding element to be processed, a query process for the count value of the element may also be included, which may specifically include:

[0176] (F.1) Obtain the element to be queried;

[0177] (F.2) performing hash calculation on the query element using the quotient-remainder hash function to obtain the quotient and remainder of the query element;

[0178] (F.3) determining the quotient of the element to be queried as the second bucket index, determining the remainder as the second entry index, and determining the query hash bucket of the element to be queried according to the second bucket index, and determining the query entry of the element to be queried from the query hash bucket according to the second entry index;

[0179] (F.4) Read the count value of the element to be queried from the query entry.

[0180] The element to be queried may be a specific element whose occurrence frequency (ie, count value) needs to be queried during the data stream processing.

[0181] The second bucket index may be a quotient part obtained by calculating the hash value of the element to be queried through a quotient-remainder hash function, and is used to determine the hash bucket position of the element to be queried in the data storage space.

[0182] The entry index may be a remainder obtained by calculating the hash value of the element to be queried through a quotient remainder hash function, and is used to determine the position of the storage entry of the element to be queried in the second hash bucket.

[0183] The query hash bucket may be a hash bucket determined according to the second bucket index, and is used to store entries of elements to be queried.

[0184] The query entry may be a specific storage entry determined in the query hash bucket according to the entry index, and is used to count the elements to be queried.

[0185] Exemplarily, if the count value of the element e to be queried is needed, if the quotient of the queried element is hashed using the quotient-remainder hash function and the quotient is 1 and the remainder is 3, the quotient can be used as the second bucket index to determine the query hash bucket located in the first array and the alternative query hash bucket located in the second array from the data storage space.

[0186] Furthermore, the remainder can be used as the second entry index to search for the query entry of the element to be queried e in the query hash bucket. Specifically, the storage entry with the entry index 3 can be determined as the query entry in the query hash bucket according to the second entry index 3. If the count value of the element to be queried e in the query entry is 55, then the count value of the element to be queried is 55.

[0187] Exemplarily, if the element e to be queried cannot be found in the query hash bucket according to the second entry index, the element e to be queried can be found in the alternative query hash bucket according to the second entry index.

[0188] Most queries can be completed in constant time

[0189] The above scheme is only used as an example. All elements to be queried can find the corresponding count value from the hash bucket in the above manner, which will not be described in detail here.

[0190] By querying the element to be queried through the quotient hash function, the corresponding query entry can be quickly located, which greatly improves the efficiency and accuracy of obtaining the count value of the element to be queried.

[0191] Please refer to Figures 3 to 5 , the embodiments of the present application are combined with Figures 3 to 5 , the insertion process of the data stream processing method is introduced.

[0192] Exemplarily, if each hash bucket contains three storage entries (ie, B=3), the storage entries are represented as (e, freq), where e represents the entry index of the element, also called the element fingerprint, and freq represents the frequency of the entry index e, ie, the count value.

[0193] Please refer to Figure 3 , when inserting element e1, calculate the quotient and remainder hash value of e1, and divide the quotient and remainder hash value into quotient and remainder, where quotient Q(e) = 1 and remainder R(e) = 3, so that the quotient can be used as the bucket index to determine hash bucket 1 from the first array. Use the remainder as the entry index, and determine the storage entry corresponding to e1 from hash bucket 1 as entry 3. The original count value of entry 3 is 90. At this time, increase the count of element e1 in entry 3, and the new count value of element e1 is 91.

[0194] Please refer to Figure 3 , illustratively, the count value of each storage entry in each hash bucket is positively correlated with the level of the hash bucket. After inserting element e2, the quotient Q(e)=1 and remainder R(e)=1 are calculated by the quotient-remainder hash function, so hash bucket 1 can be determined by the quotient, and the count value of the storage entry corresponding to element e2 is determined by the remainder to be 33. At this time, the count of element e2 in entry 2 is increased, and the new count value of element e2 is 34. Through bubble sorting, it is obtained that the count value of e2 in hash bucket 1 is greater than the count value of e4, so the storage entries corresponding to e2 and e4 can be exchanged in hash bucket 1.

[0195] Please refer to Figure 4 For example, after inserting element e3, the quotient Q(e)=2 of e3 is calculated by the quotient-remainder hash function, and the remainder R(e)=2. If the quotient is used as the bucket index at this time, hash bucket 2 is determined in the first array. However, based on the remainder as the entry index, the corresponding storage entry cannot be matched in hash bucket 2. At this time, if there is vacant space in hash bucket 2, the entry index and count value corresponding to element e3 can be stored in the vacant space.

[0196] Please refer to Figure 4 For example, after inserting element e4, the quotient Q(e)=2 of e4 is calculated by the quotient-remainder hash function, and the remainder R(e)=1. If the quotient is used as the bucket index at this time, hash bucket 2 is determined in the first array, and the alternative hash bucket is determined in the second array. However, according to the remainder as the entry index, the corresponding storage entry cannot be matched in hash bucket 2 and the alternative hash bucket, and both hash bucket 2 and the alternative hash bucket have no remaining storage space. At this time, the storage entry with the smallest count value can be removed from hash bucket 2. If the storage entry with the smallest count value in hash bucket 2 is the storage entry corresponding to element e7, at this time, element e7 is removed from hash bucket 2, and element e4 is stored in the original storage position corresponding to the storage entry of element e7.

[0197] Please refer to Figure 5For example, after inserting element e5, the quotient Q(e)=1 and remainder R(e)=2 of e5 are calculated by the quotient-remainder hash function. At this time, the quotient is used as the bucket index to determine hash bucket 3 in the first array, and the alternative hash bucket 21 of element e5 is determined in the second array. Furthermore, the storage entry corresponding to e5 can be determined in hash bucket 3 according to the remainder as the entry index, and the count value of e5 is updated to obtain the count value of e5 as 72. Furthermore, if at this time, in the alternative hash bucket 21, the count value of element e5 is greater than the count values ​​of all storage entries, and the highest level of the alternative hash bucket 21 is vacant, in order to facilitate the distinction of all hot elements, element e5 can be transferred to the alternative hash bucket 21 as a large data stream.

[0198] Please refer to Figure 5 , illustratively, after inserting element e6, the quotient Q(e)=1 of e6 is calculated by the quotient-remainder hash function, and the remainder R(e)=1. At this time, the quotient is used as the bucket index to determine hash bucket 3 in the first array, and the alternative hash bucket 31 of element e6 is determined in the second array. Further, the storage entry corresponding to e6 can be determined in hash bucket 3 according to the remainder as the entry index, and the count value of e6 is updated to obtain the count value of e6 as 61. Further, if at this time, in the alternative hash bucket 31, the count value of element e6 is greater than the count values ​​of all storage entries, and there is no remaining storage space in the alternative hash bucket 31, at this time, the storage entry with the smallest count value in the alternative hash bucket 31 can be removed, and all levels can be pushed down, so that the highest level of the alternative hash bucket 31 is idle, and the storage entry corresponding to element e6 is moved from hash bucket 3 to the highest level of the alternative hash bucket 31.

[0199] In some implementations, a query process for an element is introduced.

[0200] For example, please refer to Figure 3 If you need to query the element e1, the quotient and remainder hash function is used to calculate the quotient Q(e) = 1 and the remainder R(e) = 3 of e1. At this time, the quotient is used as the bucket index and the remainder is used as the entry index to determine the count value corresponding to the element e1, which is 91. If the corresponding hash bucket is determined by the quotient, but the corresponding storage entry cannot be matched in the data storage space by the remainder, the corresponding element does not exist.

[0201] The embodiment of the present application obtains a data stream to be processed, and determines multiple elements to be processed from the data stream to be processed; performs hash calculation on each element to be processed through a quotient-remainder hash function to obtain the quotient and remainder of the element to be processed; determines the quotient of the element to be processed as the first bucket index, determines the remainder as the first entry index, and determines the first hash bucket of the element to be processed from the data storage space according to the first bucket index, and determines the first target entry of the element to be processed from multiple storage entries of the first hash bucket according to the first entry index; in the first target entry, updates the count value of the corresponding element to be processed. In this way, the bucket index and entry index of the element to be processed can be obtained by using the quotient-remainder hash, and the corresponding hash bucket and entry can be directly determined through the bucket index and entry index, thereby reducing the number of hash operations, optimizing the memory access mode, and thus improving the processing efficiency of the data stream, while also improving the memory utilization.

[0202] See also Figure 6 The embodiment of the present application further provides a data stream processing device, which can implement the above data stream processing method, and the data stream processing device includes:

[0203] An acquisition module 61 is used to acquire a data stream to be processed and determine a plurality of elements to be processed from the data stream to be processed;

[0204] A calculation module 62 is used to perform hash calculation on each element to be processed by using a quotient and remainder hash function to obtain a quotient and remainder of the element to be processed;

[0205] A determination module 63, configured to determine the quotient of the element to be processed as a first bucket index, determine the remainder as a first entry index, determine a first hash bucket of the element to be processed from the data storage space according to the first bucket index, and determine a first target entry of the element to be processed from a plurality of storage entries of the first hash bucket according to the first entry index;

[0206] The updating module 64 is used to update the count value of the corresponding element to be processed in the first target entry.

[0207] The specific implementation of the data stream processing device is basically the same as the specific implementation of the data stream processing method described above, and will not be repeated here. On the premise of meeting the requirements of the embodiment of the present application, the data stream processing device can also be provided with other functional modules to implement the data stream processing method in the above embodiment.

[0208] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data stream processing method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0209] See also Figure 7 , Figure 7 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0210] The processor 71 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0211] The memory 72 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 72 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 72, and the processor 71 calls and executes the data stream processing method of the embodiment of this application;

[0212] Input / output interface 73, used to implement information input and output;

[0213] Communication interface 74, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0214] A bus 75 that transmits information between the various components of the device (e.g., the processor 71, the memory 72, the input / output interface 73, and the communication interface 74);

[0215] The processor 71 , the memory 72 , the input / output interface 73 and the communication interface 74 are connected to each other in communication within the device via a bus 75 .

[0216] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data stream processing method is implemented.

[0217] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0218] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0219] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0220] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0222] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0223] It should be understood that in the present application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0224] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0225] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0227] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0228] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data stream processing method, characterized in that: The method comprises: Acquire a data stream to be processed, and determine a plurality of elements to be processed from the data stream to be processed; Performing hash calculation on each of the elements to be processed by using a quotient-remainder hash function to obtain a quotient and a remainder of the element to be processed; Determine the quotient of the element to be processed as a first bucket index, determine the remainder of the element to be processed as a first entry index, determine a first hash bucket of the element to be processed from a data storage space according to the first bucket index, and determine a first target entry of the element to be processed from a plurality of storage entries of the first hash bucket according to the first entry index; In the first target entry, the count value of the corresponding element to be processed is updated.

2. The data stream processing method according to claim 1, characterized in that: After updating the count value of the corresponding element to be processed in the first target entry, the method further includes: Obtaining a total storage resource occupancy of a plurality of the storage entries in the first hash bucket; When the total storage resource occupancy exceeds the storage space of the first hash bucket, at least one storage entry is removed from the first hash bucket according to the count value of each storage entry to release the storage space of the first hash bucket.

3. The data stream processing method according to claim 2, characterized in that: When the total storage resource occupancy exceeds the storage space of the first hash bucket, at least one storage entry is removed from the first hash bucket according to the count value of each storage entry to release the storage space of the first hash bucket, including: When the total storage resource occupancy exceeds the storage space of the first hash bucket, determining a first storage entry with a minimum count value; removing the first storage entry from the first hash bucket; Repeating the step of removing the smallest first storage entry from the first hash bucket until the total storage resource occupancy is less than the storage space of the first hash bucket, thereby obtaining a first hash bucket with released storage space.

4. The data stream processing method according to claim 1, characterized in that: The method further comprises: When the first target entry corresponding to the first entry index cannot be matched from the first hash bucket, obtaining a filling state attribute of the first hash bucket corresponding to the first bucket index; When the filling state attribute indicates that there is available storage space in the first hash bucket, a second storage entry corresponding to the first entry index and the count value of the element to be processed is stored in the first hash bucket.

5. The data stream processing method according to claim 4, characterized in that: The method further comprises: When the filling state attribute indicates that there is no available storage space in the first hash bucket, obtaining a count value of each storage entry in the first hash bucket; When there is a third storage entry whose count value is less than or equal to the count value of the element to be processed, removing the corresponding third storage entry from the first hash bucket to release storage space of the first hash bucket; A second storage entry corresponding to the first entry index and the count value of the element to be processed is stored in the first hash bucket.

6. The data stream processing method according to claim 1, characterized in that: The method further comprises: Obtaining a count value of each storage entry in the first hash bucket; Bubble sort each of the storage entries based on the count value to obtain a sorting result; Based on the sorting result, the position of each storage entry in the first hash bucket is adjusted to obtain an adjusted first hash bucket; wherein the count value of each storage entry in the first hash bucket is positively correlated with the first hash bucket level.

7. The data stream processing method according to claim 1, characterized in that: The count value of each storage entry in each hash bucket is positively correlated with the level of the hash bucket; the method further includes: For each current hash bucket, obtain the updated first count value of the fourth storage entry corresponding to the lowest level of the hash bucket; Determine, from the data storage space, a candidate hash bucket for the element corresponding to the fourth storage entry, and a second count value corresponding to the highest level of the candidate hash bucket; When the second count value is smaller than the first count value, remove the storage entry with the smallest count value in the candidate hash bucket; Pushing down each storage entry in the candidate hash bucket one level in turn to obtain the highest level in the candidate hash bucket that is in an empty state; The fourth storage entry is stored in the highest level of the candidate hash bucket.

8. The data stream processing method according to claim 1, characterized in that: After updating the count value of the corresponding element to be processed in the first target entry, the method further includes: Get the element to be queried; Performing hash calculation on the element to be queried by using a quotient-remainder hash function to obtain a quotient and a remainder of the element to be queried; Determine the quotient of the element to be queried as a second bucket index, determine the remainder as a second entry index, determine a query hash bucket of the element to be queried according to the second bucket index, and determine a query entry of the element to be queried from the query hash bucket according to the second entry index; The count value of the element to be queried is read from the query entry.

9. A data stream processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a data stream to be processed and determining a plurality of elements to be processed from the data stream to be processed; A calculation module, used for performing hash calculation on each of the elements to be processed by using a quotient and remainder hash function to obtain the quotient and remainder of the element to be processed; a determination module, configured to determine the quotient of the element to be processed as a first bucket index, determine the remainder of the element to be processed as a first entry index, determine a first hash bucket of the element to be processed from a data storage space according to the first bucket index, and determine a first target entry of the element to be processed from a plurality of storage entries of the first hash bucket according to the first entry index; An updating module is used to update the count value of the corresponding element to be processed in the first target entry.

10. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the data stream processing method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data stream processing method according to any one of claims 1 to 8 is implemented.

Images