Data flow measurement method, device, electronic device and readable storage medium

Abstract

The present application discloses a data flow measurement method, device, electronic device, and readable storage medium. When a data flow is detected, the fingerprint information of the current data flow is determined; if a first target element exists in a hash table, the count value corresponding to the first target element is increased; if the count value corresponding to the first target element overflows, the storage content of the second target element is swapped with the storage content of the first target element. Since the storage space used by the second target element to store the count value is larger than the space occupied by the count value of the first target element, and the storage space used by the first target element to store the count value is larger than the space occupied by the count value of the second target element, the storage content of the second target element is swapped with the storage content of the first target element. After the swap, neither the first target element nor the second target element will overflow. Therefore, the present application can measure the data flow and reduce storage overhead.

Description

Technical Field

[0001] The present application relates to the field of data processing and computing, and in particular to a data flow measurement method, device, electronic device, and readable storage medium. Background Art

[0002] A data stream is an ordered sequence of bytes with a starting and ending point. Its size and value range are often unpredictable. Data stream measurement uses certain data structures to record statistical information about the arrival of multiple data streams. Statistical measurement of data streams has widespread application in database optimization, network monitoring, data analysis, and other fields.

[0003] In related technologies, data flow measurement methods usually measure data flows through simple data structures such as hash tables and linked lists. For scenarios such as big data and high-bandwidth networks, when measuring data flows, it is necessary to process the arrival of millions of data flows in a very short time. Conventional measurement methods require a large amount of storage overhead. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to provide a data flow measurement method, device, electronic device and readable storage medium, which can measure data flow and reduce storage overhead.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a data flow measurement method, including:

[0006] When a data flow is detected, fingerprint information of the current data flow is determined;

[0007] Detecting whether a first target element exists in multiple hash tables, where fingerprint information of the first target element is the same as the fingerprint information of the current data stream; wherein each of the hash tables is provided with multiple hash buckets, each of the hash buckets stores multiple elements, and each of the elements stores the fingerprint information and count value of the data stream;

[0008] If the first target element exists in the hash table, increasing the count value corresponding to the first target element;

[0009] Detecting whether the count value corresponding to the first target element overflows;

[0010] If the count value corresponding to the first target element overflows, detecting whether a second target element exists in the target hash bucket corresponding to the first target element, where the storage space used by the second target element to store the count value is larger than the space occupied by the count value of the first target element, and the storage space used by the first target element to store the count value is larger than the space occupied by the count value of the second target element;

[0011] If the second target element exists in the target hash bucket, the storage content of the second target element is exchanged with the storage content of the first target element.

[0012] In one embodiment, after detecting whether a second target element exists in a target hash bucket corresponding to the first target element if the count value corresponding to the first target element overflows, the method further includes:

[0013] If the second target element does not exist in the target hash bucket, changing the type of the target hash bucket; wherein the storage space of each hash bucket is equal, and the type is used to determine the number of the elements in the hash bucket and the storage space of each element;

[0014] The storage content of the element in the target hash bucket before the type is changed is correspondingly stored in the target hash bucket after the type is changed.

[0015] In one embodiment, after changing the type of the target hash bucket, the method further includes:

[0016] If the number of elements in the target hash bucket before the type is changed is greater than the number of elements in the target hash bucket after the type is changed, determining to remove elements from the elements in the target hash bucket before the type is changed;

[0017] Repeating the elimination and insertion processing operation according to the elimination element until no new elimination element is generated or the number of repetitions reaches a preset number;

[0018] The insertion and removal processing operation includes:

[0019] Determine the target hash table corresponding to the removed element according to the cuckoo hash algorithm;

[0020] Detecting whether a third target element exists in the target hash table, the third target element does not store fingerprint information, and a storage space used by the third target element for storing a count value is larger than a space occupied by the count value of the eliminated element;

[0021] If the third target element exists in the target hash table, storing the content of the removed element in the third target element;

[0022] If the third target element does not exist in the target hash table, determining a fourth target element from the hash bucket of the target hash table; wherein the storage space of the fourth target element for storing the count value is larger than the space occupied by the count value of the eliminated element;

[0023] The content of the removed element is stored in the fourth target element, and the original stored content of the fourth target element is used as the new removed element.

[0024] In one embodiment, after detecting whether the first target element exists in multiple hash tables, the method further includes:

[0025] If the first target element does not exist in the plurality of hash tables, detecting whether a fifth target element exists in the plurality of hash tables, the fifth target element not storing fingerprint information;

[0026] If the fifth target element exists in the hash table, the fingerprint information of the current data stream is stored in the fifth target element, and the count value of the fifth target element is set to 1.

[0027] In one embodiment, after detecting whether the fifth target element exists in the plurality of hash tables, the method further includes:

[0028] If the fifth target element does not exist in the hash table, determining a sixth target element with the smallest count value from the plurality of hash tables;

[0029] The content originally stored in the sixth target element is used as the removed element, and the removal and insertion processing operation is repeatedly performed on the removed element.

[0030] The fingerprint information of the current data stream is stored in the sixth target element, and the count value of the sixth target element is set to 1.

[0031] In one embodiment, it further includes:

[0032] Obtain fingerprint information of the target data stream;

[0033] detecting whether a seventh target element exists in the plurality of hash tables, where fingerprint information of the seventh target element is the same as fingerprint information of the target data stream;

[0034] If the count value in the seventh target element exists in the hash table, the count value in the seventh target element is a statistical value of the target data flow.

[0035] In one embodiment, after detecting whether the seventh target element exists in the plurality of hash tables, the method further includes:

[0036] If the seventh target element does not exist in the plurality of hash tables, and at least one element that does not store fingerprint information exists in the plurality of hash tables, the statistical value of the target data stream is 0;

[0037] If the seventh target element does not exist in the multiple hash tables and the elements that do not store fingerprint information do not exist in the multiple hash tables, the minimum value of all count values ​​in the multiple hash tables is used as the statistical value of the target data flow.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data flow measurement device, comprising:

[0039] A determination module, configured to determine fingerprint information of a current data flow when a data flow is detected;

[0040] a first detection module, configured to detect whether a first target element exists in a plurality of hash tables, wherein fingerprint information of the first target element is the same as fingerprint information of the current data stream; wherein each of the hash tables is provided with a plurality of hash buckets, each of the hash buckets stores a plurality of elements, and each of the elements stores fingerprint information and a count value of the data stream;

[0041] a counting module, configured to increase a count value corresponding to the first target element if the first target element exists in the hash table;

[0042] A second detection module is used to detect whether the count value corresponding to the first target element overflows;

[0043] a third detection module, configured to, if the count value corresponding to the first target element overflows, detect whether there is a second target element in the target hash bucket corresponding to the first target element, wherein the storage space used by the second target element to store the count value is larger than the space occupied by the count value of the first target element, and the storage space used by the first target element to store the count value is larger than the space occupied by the count value of the second target element;

[0044] The exchange module is configured to exchange the storage content of the second target element with the storage content of the first target element if the second target element exists in the target hash bucket.

[0045] To achieve the above-mentioned purpose, the third aspect of the embodiments of the present application proposes an electronic device, which includes a memory and a processor, the memory storing a computer program, and the processor implementing the data flow measurement method described in any one of the embodiments of the first aspect above when executing the computer program.

[0046] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements the data flow measurement method described in any one of the embodiments of the first aspect above.

[0047] The data flow measurement method, device, electronic device and readable storage medium proposed in the embodiments of the present application determine the fingerprint information of the current data flow when a data flow is detected; detect whether a first target element exists in multiple hash tables, and the fingerprint information of the first target element is the same as the fingerprint information of the current data flow; wherein each hash table is provided with multiple hash buckets, each hash bucket stores multiple elements, and each element stores the fingerprint information and count value of the data flow; if the first target element exists in the hash table, the count value corresponding to the first target element is increased; detect whether the count value corresponding to the first target element overflows; if the count value corresponding to the first target element overflows, detect whether a second target element exists in the target hash bucket corresponding to the first target element, the storage space of the second target element for storing the count value is larger than the space occupied by the count value of the first target element, and the storage space of the first target element for storing the count value is larger than the space occupied by the count value of the second target element; if the second target element exists in the target hash bucket, the storage content of the second target element is exchanged with the storage content of the first target element. If the count value corresponding to the first target element overflows, it means that the storage space used to store the count value in the first target element is smaller than the space occupied by the count value of the first target element. Since the storage space used to store the count value in the second target element is larger than the space occupied by the count value of the first target element, and the storage space used to store the count value in the first target element is larger than the space occupied by the count value of the second target element, the storage content of the second target element is exchanged with the storage content of the first target element. In this way, after the exchange, neither the first target element nor the second target element will overflow, and the storage cost will not increase. Therefore, the embodiment of the present application can measure the data stream and reduce storage overhead.

[0048] Additional aspects and advantages of the present application will be given in part in the following description and in part will become obvious from the following description or will be learned through practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present application is further described below with reference to the accompanying drawings and embodiments, wherein:

[0050] Figure 1 This is a flowchart of the steps of the data flow measurement method according to an embodiment of the present application;

[0051] Figure 2 This is a schematic diagram of a sub-process for changing the hash bucket type in the data flow measurement method according to an embodiment of the present application;

[0052] Figure 3 This is another sub-process diagram of changing the hash bucket type in the data flow measurement method according to an embodiment of the present application;

[0053] Figure 4Schematic diagram of the process of the elimination and insertion processing operation of the data flow measurement method according to an embodiment of the present application;

[0054] Figure 5 This is a schematic diagram of a sub-process of the data flow measurement method according to an embodiment of the present application when the first target element does not exist in multiple hash tables;

[0055] Figure 6 This is another sub-flow diagram of the data flow measurement method according to an embodiment of the present application when the first target element does not exist in multiple hash tables;

[0056] Figure 7 A schematic diagram of a sub-process of statistical query of the data flow measurement method according to an embodiment of the present application;

[0057] Figure 8 This is another sub-flow diagram of the statistical query of the data flow measurement method according to an embodiment of the present application;

[0058] Figure 9 A schematic diagram of the structure of a data flow measurement device provided in an embodiment of the present application;

[0059] Figure 10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0061] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0063] First, let’s analyze some of the terms used in this application:

[0064] Data stream: It is an ordered sequence of bytes with an end and a starting point. Its data size and data value range are often unpredictable.

[0065] Fingerprint information of a data stream: used to uniquely identify a data stream. In one embodiment, a hash calculation is performed on the data stream to obtain the fingerprint information of the data stream.

[0066] Hash buckets: Hash buckets are the basic unit of data storage in hash tables, also known as hash slots or buckets. In a hash table, a hash function maps a key to a corresponding hash bucket. Each hash bucket can hold one or more key-value pairs. Hash tables utilize hash functions and hash bucket structures to achieve efficient data storage and retrieval. Different keys may be mapped to the same hash value by the hash function, resulting in hash collisions. Hash buckets are designed to handle these collisions, allowing multiple key-value pairs to be stored in a single bucket.

[0067] To reduce storage overhead when measuring data streams, referring to Table 1, an embodiment of the present application provides a data structure comprising M hash tables, each using a different hash function. Each hash table contains N hash buckets, resulting in a total of M*N hash buckets. It should be noted that this application does not specifically limit M and N, and those skilled in the art may set the values ​​of M and N based on actual needs.

[0068] Table 1

[0069]

[0070] Referring to Table 2, there are k types of hash buckets. The number of elements stored in each hash bucket may vary. The total storage space of each hash bucket is equal, but the storage space of each element may be unequal. The Flag field stored in the hash bucket identifies the hash bucket type. The hash bucket type is used to determine the number of elements in the hash bucket and the storage space of each element. Each element includes the fingerprint information of a data stream and the count value of that data stream. For example, the total storage space of a hash bucket is 64 bits, of which the hash bucket type occupies 4 bits and the fingerprint information of the data stream occupies 8 bits. For hash bucket type 1, there are 5 elements, and the storage space used to store the count value in the 5 elements is 2 bits, 3 bits, 4 bits, 5 bits, and 6 bits, respectively. Therefore, the total storage space of the hash bucket is 4 + 8 * 5 + 2 + 3 + 4 + 5 + 6 = 64 bits, which fully utilizes all the storage space of the hash bucket. For example, a hash bucket of type 2 has four elements, and the storage space used by each element to store the count value is 3 bits, 4 bits, 5 bits, and 16 bits, respectively. The total storage space of the hash bucket is also exactly 64 bits. The above hash bucket type and number of elements, and the allocation of storage space used by each element to store the count value are merely examples and are not to be construed as limitations of this application. Those skilled in the art can set the storage space allocation for different elements of different types of hash buckets according to actual needs.

[0071] Table 2

[0072]

[0073] In one embodiment, the elements in the hash bucket are key-value pairs of the hash bucket. Each element consists of fingerprint information of the data stream and a count value of the data stream. The fingerprint information of the data stream serves as the key and the count value of the data stream serves as the value. Within an element, the fingerprint information of the data stream and the calculated value of the data stream form a key-value pair.

[0074] Based on the above data structure, an embodiment of the present application proposes a data flow measurement method.

[0075] The data flow measurement method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running on a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data flow measurement method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0077] Reference Figure 1 , Figure 1 The first aspect of the present application provides a data flow measurement method. Figure 1 The data flow measurement method shown includes but is not limited to steps S110 to S160;

[0078] Step S110, when a data stream is detected, determining fingerprint information of the current data stream;

[0079] Exemplarily, when a data stream is detected to have arrived, the fingerprint information of the current data stream is calculated. In one embodiment, a hash calculation is performed on the current data stream to obtain the fingerprint information of the current data stream, and the fingerprint information is used to uniquely indicate the current data stream.

[0080] Step S120: Detect whether a first target element exists in multiple hash tables, and the fingerprint information of the first target element is the same as the fingerprint information of the current data stream; wherein each hash table is provided with multiple hash buckets, each hash bucket stores multiple elements, and each element stores the fingerprint information and count value of the data stream;

[0081] Step S130: if the first target element exists in the hash table, the count value corresponding to the first target element is increased;

[0082] Exemplarily, when it is detected that a first target element exists in a hash table, it indicates that the fingerprint information in the first target element is the same as the fingerprint information of the current data stream, and the count value of the first target element is increased by 1.

[0083] Step S140, detecting whether the count value corresponding to the first target element overflows;

[0084] In one embodiment, if the count value corresponding to the first target element does not overflow, it means that the storage space of the first target element is sufficient to store the count value.

[0085] Step S150: If the count value corresponding to the first target element overflows, detect whether there is a second target element in the target hash bucket corresponding to the first target element, and the storage space used by the second target element to store the count value is larger than the space occupied by the count value of the first target element, and the storage space used by the first target element to store the count value is larger than the space occupied by the count value of the second target element;

[0086] Step S160: If the second target element exists in the target hash bucket, the storage content of the second target element is exchanged with the storage content of the first target element.

[0087] In steps S110 to S160 illustrated in the embodiment of the present application, when a data stream is detected, fingerprint information of the current data stream is determined; whether a first target element exists in multiple hash tables, and the fingerprint information of the first target element is the same as the fingerprint information of the current data stream; wherein each hash table is provided with multiple hash buckets, each hash bucket stores multiple elements, and each element stores the fingerprint information and count value of the data stream; if the first target element exists in the hash table, the count value corresponding to the first target element is increased; whether the count value corresponding to the first target element overflows is detected; if the count value corresponding to the first target element overflows, whether a second target element exists in the target hash bucket corresponding to the first target element, the storage space of the second target element for storing the count value is larger than the space occupied by the count value of the first target element, and the storage space of the first target element for storing the count value is larger than the space occupied by the count value of the second target element; if the second target element exists in the target hash bucket, the storage content of the second target element is exchanged with the storage content of the first target element. If the count value corresponding to the first target element overflows, it means that the storage space used to store the count value in the first target element is smaller than the space occupied by the count value of the first target element. Since the storage space used to store the count value in the second target element is larger than the space occupied by the count value of the first target element, and the storage space used to store the count value in the first target element is larger than the space occupied by the count value of the second target element, the storage content of the second target element is exchanged with the storage content of the first target element. In this way, after the exchange, neither the first target element nor the second target element will overflow, and the storage cost will not increase. Therefore, the embodiment of the present application can measure the data stream and reduce storage overhead.

[0088] Reference Figure 2 , Figure 2 This is a schematic diagram of a sub-process for changing the hash bucket type of the data flow measurement method according to an embodiment of the present application. In some embodiments, after step S150, steps S210 to S220 are also included but are not limited to;

[0089] Step S210: If the second target element does not exist in the target hash bucket, then change the type of the target hash bucket; wherein the storage space of each hash bucket is equal, and the type is used to determine the number of elements in the hash bucket and the storage space of each element;

[0090] In step S220 , the stored content of the element in the target hash bucket before the type change is correspondingly stored in the target hash bucket after the type change.

[0091] By changing the type of the hash bucket, the storage space of each element in the hash bucket is reallocated, thereby increasing the storage space of the elements in the hash bucket. This prevents overflow of the count value stored in the element without increasing storage costs. For example, by reducing the number of elements in the hash bucket, the storage space of each element can be increased, so that the count value stored in the element does not overflow. In this way, the embodiments of the present application achieve flexible and variable storage space for hash bucket elements, avoiding waste of storage space.

[0092] For example, the Flag of the target hash bucket is 1, the first target element is Entry[1], the storage space for storing the count value in Entry[1] is 6 bits, and the count value of the first target element is 2 after the increase. 6 +1, therefore, the count value of the first target element will overflow, causing the data stream corresponding to the first target element to be unable to count normally. At this time, the Flag of the target hash bucket is changed to 2, and the storage space used by Entry[1] to store the count value is 16 bits, 2 16 >(2 6 +1), so the Entry[1] of the target hash bucket after the type change will not overflow. And the storage content of the elements of the target hash bucket before the type change is stored in the target hash bucket after the type change. For example, before the type change, the Flag of the target hash bucket is 1, and the target hash bucket includes Entry[1], Entry[2], Entry[3], and Entry[4]. After the type change, the Flag of the target hash bucket is 2, and the target hash bucket still includes Entry[1], Entry[2], Entry[3], and Entry[4]. Then the content of Entry[1] before the type change is stored in Entry[1] after the type change; the content of Entry[2] before the type change is stored in Entry[2] after the type change; the content of Entry[3] before the type change is stored in Entry[3] after the type change; and the content of Entry[4] before the type change is stored in Entry[4] after the type change.

[0093] It should be noted that in the process of changing the type of the hash bucket, the number of elements in the hash bucket may change.

[0094] In one embodiment, referring to Figure 3 , Figure 3 This is another sub-process diagram of changing the hash bucket type of the data flow measurement method according to an embodiment of the present application; after step S210, it may also include but not be limited to step S310 and step S320.

[0095] Step S310: If the number of elements in the target hash bucket before the type change is greater than the number of elements in the target hash bucket after the type change, determine to remove elements from the elements in the target hash bucket before the type change.

[0096] For example, before the type is changed, the Flag of the target hash bucket is 1, and the target hash bucket includes Entry[1], Entry[2], Entry[3], and Entry[4]. After the type is changed, the Flag of the target hash bucket is 2, and the target hash bucket only includes Entry[1], Entry[2], and Entry[3]. At this time, the contents of each element of the target hash bucket before the type is changed cannot be stored correspondingly in the target hash bucket after the type is changed. It is necessary to determine one element from the elements of the target hash bucket before the type is changed as the eliminated element, and the remaining elements are stored correspondingly in the target hash bucket after the type is changed. In one embodiment, before the type is changed, the element with the smallest count value in the target hash bucket is used as the eliminated element, which can minimize the impact on the statistics of the data stream corresponding to the element with a larger count value.

[0097] Step S320 , repeatedly performing the culling and inserting processing operation according to the culling elements until no new culling elements are generated or the number of repetitions reaches a preset number.

[0098] It should be noted that when the number of repetitions reaches the preset number of times and there is still a discarded element, the discarded element is discarded. Those skilled in the art can set the preset number of times according to actual needs.

[0099] Reference Figure 4 , Figure 4 It is a flowchart of the elimination and insertion processing operation of the data flow measurement method according to an embodiment of the present application.

[0100] The removal and insertion processing operations include but are not limited to steps S410 to S450;

[0101] Step S410, determining the target hash table corresponding to the removed element according to the cuckoo hash algorithm;

[0102] It should be noted that the purpose of the culling operation is to attempt to store the culled element in another hash table. Therefore, the target hash table corresponding to the culled element is different from the hash table to which the target hash bucket belongs. For example, if the hash table to which the target hash bucket belongs is the xth hash table, then according to the cuckoo hash algorithm, a hash table is identified from the other M-1 hash tables as the target hash table for the culled element. This target hash table can be any of the M-1 hash tables.

[0103] Step S420: Detect whether the target hash table has a third target element, the third target element does not store fingerprint information, and the storage space used by the third target element to store the count value is larger than the space occupied by the count value of the eliminated element;

[0104] Step S430: If the target hash table contains a third target element, the content of the removed element is stored in the third target element.

[0105] Step S440: If the target hash table does not contain the third target element, determine a fourth target element from the hash bucket of the target hash table, wherein the storage space of the fourth target element for storing the count value is larger than the space occupied by the count value of the eliminated element.

[0106] Step S450: storing the content of the removed element in the fourth target element, and using the original stored content of the fourth target element as a new removed element.

[0107] It should be noted that, in one embodiment, the count value of the fourth target element is smaller than the count value of the eliminated element. Thus, the original eliminated element is stored in the fourth target element, and the content of the original fourth target element is used as the new eliminated element. This can retain elements with larger count values ​​as much as possible, thereby avoiding affecting the statistics of the data stream corresponding to the elements with larger count values ​​as much as possible.

[0108] Reference Figure 5 , Figure 5 This is a schematic diagram of a sub-process of the data flow measurement method according to an embodiment of the present application when the first target element does not exist in multiple hash tables; after step S120, it may also include but is not limited to step S510 and step S520.

[0109] Step S510: If the first target element does not exist in the multiple hash tables, detect whether a fifth target element exists in the multiple hash tables, and the fifth target element does not store fingerprint information;

[0110] Step S520: If the fifth target element exists in the hash table, the fingerprint information of the current data stream is stored in the fifth target element, and the count value of the fifth target element is set to 1.

[0111] If the first target element does not exist in multiple hash tables, it means that the current data stream has not been counted and the current data stream is a new data stream. Therefore, it is necessary to find the fifth target element in multiple hash tables that does not store fingerprint information, store the fingerprint information of the current data stream in the fifth target element, and set the count value of the fifth target element to 1.

[0112] Reference Figure 6 , Figure 6This is another sub-flow diagram of the data flow measurement method according to an embodiment of the present application when the first target element does not exist in multiple hash tables. After step S510, the method further includes steps S610 to S630.

[0113] Step S610: if the fifth target element does not exist in the hash table, determine the sixth target element with the smallest count value from multiple hash tables;

[0114] Step S620, taking the content originally stored in the sixth target element as a removed element, and repeatedly performing the removal and insertion processing operation on the removed element;

[0115] Step S630: Store the fingerprint information of the current data stream in the sixth target element, and set the count value of the sixth target element to 1.

[0116] If the fifth target element does not exist in the hash table, it means that there are no empty elements in multiple hash tables, and each element stores the fingerprint information and count value of the data stream. Then, each element of all hash tables is traversed to determine the element with the smallest count value, that is, the sixth target element. The content originally stored in the sixth target element is used as the removed element, and the removal and insertion processing operation is repeated on the removed element; the fingerprint information of the current data stream is stored in the sixth target element, and the count value of the sixth target element is set to 1.

[0117] Reference Figure 7 , Figure 7 This is a schematic diagram of a sub-flow of statistical query of the data flow measurement method according to an embodiment of the present application; the data flow measurement method according to the present application may further include steps S710 to S730.

[0118] Step S710, obtaining fingerprint information of the target data stream;

[0119] Step S720: Detect whether there is a seventh target element in the multiple hash tables, and the fingerprint information of the seventh target element is the same as the fingerprint information of the target data stream;

[0120] Step S730: If the count value in the seventh target element exists in the hash table, the count value in the seventh target element is a statistical value of the target data flow.

[0121] Specifically, the target data stream refers to the data stream whose statistical value needs to be queried. The fingerprint information of the target data stream is determined based on the target data stream. When it is detected that there is a seventh target element in the hash table and the fingerprint information of the seventh target element is the same as the fingerprint information of the target data stream, the count value in the seventh target element is used as the statistical value of the target data stream.

[0122] Reference Figure 8 , Figure 8This is another sub-flow diagram of the statistical query of the data flow measurement method according to an embodiment of the present application; after step S720, it may further include steps S810 to S820;

[0123] Step S810: if the seventh target element does not exist in the multiple hash tables, and there is at least one element in the multiple hash tables that does not store fingerprint information, the statistical value of the target data stream is 0;

[0124] Step S820 : If the seventh target element does not exist in the multiple hash tables and the multiple hash tables do not contain elements that do not store fingerprint information, the minimum value of all count values ​​in the multiple hash tables is used as the statistical value of the target data stream.

[0125] Specifically, if the seventh target element does not exist in multiple hash tables, and at least one element that does not store fingerprint information exists in multiple hash tables, it means that the target data stream has never been counted, that is, the target data stream has never been reached, and therefore the statistical value of the target data stream is 0. If the seventh target element does not exist in multiple hash tables, and no element that does not store fingerprint information exists in multiple hash tables, it means that the count value of the target data stream may have been discarded as a discarded element, and therefore the count value of the target data stream is small. Therefore, the minimum value of all count values ​​in the multiple hash tables is used as the statistical value of the target data stream.

[0126] Reference Figure 9 , Figure 9 : is a schematic diagram of the structure of a data flow measurement device provided in an embodiment of the present application. The measurement device can implement the above-mentioned data flow measurement method. The measurement device includes:

[0127] The determination module 910 is configured to determine fingerprint information of the current data flow when a data flow is detected;

[0128] A first detection module 920 is configured to detect whether a first target element exists in multiple hash tables, where the fingerprint information of the first target element is the same as the fingerprint information of the current data stream; wherein each hash table has multiple hash buckets, each hash bucket stores multiple elements, and each element stores the fingerprint information and count value of the data stream;

[0129] a counting module 930 configured to increase a count value corresponding to the first target element if the first target element exists in the hash table;

[0130] A second detection module 940 is used to detect whether the count value corresponding to the first target element overflows;

[0131] a third detection module 950 for detecting, if the count value corresponding to the first target element overflows, whether a second target element exists in the target hash bucket corresponding to the first target element, where the storage space used by the second target element for storing the count value is larger than the space occupied by the count value of the first target element, and the storage space used by the first target element for storing the count value is larger than the space occupied by the count value of the second target element;

[0132] The exchange module 960 is configured to exchange the storage content of the second target element with the storage content of the first target element if the second target element exists in the target hash bucket.

[0133] The measurement device of an embodiment of the present application, when detecting a data stream, determines fingerprint information of the current data stream through a determination module 910; detects whether a first target element exists in multiple hash tables through a first detection module 920, and the fingerprint information of the first target element is the same as the fingerprint information of the current data stream. Each hash table is provided with multiple hash buckets, each hash bucket stores multiple elements, and each element stores fingerprint information of the data stream and a count value. If the first target element exists in the hash table, the count value corresponding to the first target element is increased through a counting module 930; the count value corresponding to the first target element is detected through a second detection module 940 whether it overflows; if the count value corresponding to the first target element overflows, the third detection module 950 detects whether a second target element exists in a target hash bucket corresponding to the first target element, and the storage space used by the second target element for storing the count value is larger than the space occupied by the count value of the first target element, and the storage space used by the first target element for storing the count value is larger than the space occupied by the count value of the second target element; if the second target element exists in the target hash bucket, the storage content of the second target element is exchanged with the storage content of the first target element through a swap module 960. If the count value corresponding to the first target element overflows, it means that the storage space used to store the count value in the first target element is smaller than the space occupied by the count value of the first target element. Since the storage space used to store the count value in the second target element is larger than the space occupied by the count value of the first target element, and the storage space used to store the count value in the first target element is larger than the space occupied by the count value of the second target element, the storage content of the second target element is exchanged with the storage content of the first target element. In this way, after the exchange, neither the first target element nor the second target element will overflow, and the storage cost will not increase. Therefore, the embodiment of the present application can measure the data stream and reduce storage overhead.

[0134] The specific implementation of the data flow measurement device is basically the same as the specific embodiment of the data flow measurement method described above, and will not be repeated here. Under the premise of meeting the requirements of the embodiment of the present application, the data flow measurement device can also be provided with other functional modules to implement the data flow measurement method in the above embodiment.

[0135] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned data flow measurement method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0136] See also Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0137] The processor 1001 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0138] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called by the processor 1001 to execute the data flow measurement method of the embodiments of this application.

[0139] Input / output interface 1003, used to implement information input and output;

[0140] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0141] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0142] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via a bus 1005 .

[0143] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data flow measurement method is implemented.

[0144] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0146] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0148] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0149] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0152] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0155] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A data flow measurement method, characterized in that: include: When a data flow is detected, fingerprint information of the current data flow is determined; Detecting whether a first target element exists in multiple hash tables, where fingerprint information of the first target element is the same as the fingerprint information of the current data stream; wherein each of the hash tables is provided with multiple hash buckets, each of the hash buckets stores multiple elements, and each of the elements stores the fingerprint information and count value of the data stream; If the first target element exists in the hash table, increasing the count value corresponding to the first target element; Detecting whether the count value corresponding to the first target element overflows; If the count value corresponding to the first target element overflows, detecting whether a second target element exists in the target hash bucket corresponding to the first target element, where the storage space used by the second target element to store the count value is larger than the space occupied by the count value of the first target element, and the storage space used by the first target element to store the count value is larger than the space occupied by the count value of the second target element; If the second target element exists in the target hash bucket, swapping the storage content of the second target element with the storage content of the first target element; After detecting whether a second target element exists in the target hash bucket corresponding to the first target element if the count value corresponding to the first target element overflows, the method further includes: If the second target element does not exist in the target hash bucket, changing the type of the target hash bucket; wherein the storage space of each hash bucket is equal, and the type is used to determine the number of the elements in the hash bucket and the storage space of each element; The storage content of the element in the target hash bucket before the type is changed is correspondingly stored in the target hash bucket after the type is changed.

2. The data flow measurement method according to claim 1, characterized in that: After changing the type of the target hash bucket, the method further includes: If the number of elements in the target hash bucket before the type is changed is greater than the number of elements in the target hash bucket after the type is changed, determining to remove elements from the elements in the target hash bucket before the type is changed; Repeating the elimination and insertion processing operation according to the elimination element until no new elimination element is generated or the number of repetitions reaches a preset number; The removal and insertion processing operation includes: Determine the target hash table corresponding to the removed element according to the cuckoo hash algorithm; Detecting whether a third target element exists in the target hash table, the third target element does not store fingerprint information, and a storage space used by the third target element for storing a count value is larger than a space occupied by the count value of the eliminated element; If the third target element exists in the target hash table, storing the content of the removed element in the third target element; If the third target element does not exist in the target hash table, determining a fourth target element from the hash bucket of the target hash table; wherein the storage space of the fourth target element for storing the count value is larger than the space occupied by the count value of the eliminated element; The content of the removed element is stored in the fourth target element, and the original stored content of the fourth target element is used as the new removed element.

3. The data flow measurement method according to claim 2, characterized in that: After detecting whether the first target element exists in the plurality of hash tables, the method further includes: If the first target element does not exist in the plurality of hash tables, detecting whether a fifth target element exists in the plurality of hash tables, the fifth target element not storing fingerprint information; If the fifth target element exists in the hash table, the fingerprint information of the current data stream is stored in the fifth target element, and the count value of the fifth target element is set to 1.

4. The data flow measurement method according to claim 3, characterized in that: After detecting whether the fifth target element exists in the plurality of hash tables, the method further includes: If the fifth target element does not exist in the hash table, determining a sixth target element with the smallest count value from the plurality of hash tables; The content originally stored in the sixth target element is used as the removed element, and the removal and insertion processing operation is repeatedly performed on the removed element. The fingerprint information of the current data stream is stored in the sixth target element, and the count value of the sixth target element is set to 1.

5. The data flow measurement method according to claim 1, characterized in that: Also includes: Obtain fingerprint information of the target data stream; detecting whether a seventh target element exists in the plurality of hash tables, where fingerprint information of the seventh target element is the same as fingerprint information of the target data stream; If the count value in the seventh target element exists in the hash table, the count value in the seventh target element is a statistical value of the target data flow.

6. The data flow measurement method according to claim 5, characterized in that: After detecting whether the seventh target element exists in the plurality of hash tables, the method further includes: If the seventh target element does not exist in the plurality of hash tables, and at least one element that does not store fingerprint information exists in the plurality of hash tables, the statistical value of the target data stream is 0; If the seventh target element does not exist in the multiple hash tables and the elements that do not store fingerprint information do not exist in the multiple hash tables, the minimum value of all count values ​​in the multiple hash tables is used as the statistical value of the target data flow.

7. A data flow measurement device, characterized in that: include: A determination module, configured to determine fingerprint information of a current data flow when a data flow is detected; a first detection module, configured to detect whether a first target element exists in a plurality of hash tables, wherein fingerprint information of the first target element is the same as fingerprint information of the current data stream; wherein each of the hash tables is provided with a plurality of hash buckets, each of the hash buckets stores a plurality of elements, and each of the elements stores fingerprint information and a count value of the data stream; a counting module, configured to increase a count value corresponding to the first target element if the first target element exists in the hash table; A second detection module is used to detect whether the count value corresponding to the first target element overflows; a third detection module, configured to, if the count value corresponding to the first target element overflows, detect whether there is a second target element in the target hash bucket corresponding to the first target element, wherein the storage space used by the second target element to store the count value is larger than the space occupied by the count value of the first target element, and the storage space used by the first target element to store the count value is larger than the space occupied by the count value of the second target element; an exchange module, configured to exchange the storage content of the second target element with the storage content of the first target element if the second target element exists in the target hash bucket; After detecting whether a second target element exists in the target hash bucket corresponding to the first target element if the count value corresponding to the first target element overflows, the method further includes: If the second target element does not exist in the target hash bucket, changing the type of the target hash bucket; wherein the storage space of each hash bucket is equal, and the type is used to determine the number of the elements in the hash bucket and the storage space of each element; The storage content of the element in the target hash bucket before the type is changed is correspondingly stored in the target hash bucket after the type is changed.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the data flow measurement method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data flow measurement method according to any one of claims 1 to 6 is implemented.

Images