A data stream counting method, device and storage medium based on a flag flag bit

Abstract

The application provides a data flow counting method based on a Flag flag bit, a device and a storage medium, which comprises the following steps: setting a parameter initial value; when a data packet arrives, presenting key value pair information and obtaining a calculator position to which the data packet is mapped; when the first data packet arrives, calling an insertion function of a CUS structure to update the counter value to which the data packet is mapped to 1; from the second data packet, comparing the counter position of each data packet with that of the previous data packet, judging whether the data packet and the previous data packet belong to the same data flow or not; performing data flow query and selecting the smallest value as a query result. The Flag flag bit is used to detect whether the data packets mapped to the same counter belong to the same data flow or not, different counting schemes are adopted, the accuracy of counting the number of data packets in the data flow is improved, and the data flow measurement precision is improved; and the Flag flag bit is of a Bool type, and the memory occupied in the actual environment can be ignored.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network management, and in particular to a data flow counting method based on a Flag flag bit, a device and a storage medium. BACKGROUND

[0002] In high-speed networks, massive data traffic brings great pressure to network measurement work. As an important means of next-generation network measurement, data flow-based network traffic measurement schemes have been widely used in network billing, traffic engineering, flow database, network security and other fields. The data flow-based network traffic measurement method only produces summary information that occupies a small amount of storage space, which can meet the conventional operation requirements of data processing. In particular, the sketch data flow technology is powerful and highly flexible, with fast update speed and low complexity. Sketch is a compact data structure and a Hash-based data structure, which can detect large flows and abnormal flows in high-speed networks, and has the characteristics of balancing memory and estimation accuracy.

[0003] The sketch-based data flow method mainly includes Count-Min Sketch (CMS), Count Sketch (CS), Conservative Update Sketch (CUS), Augmented Sketch (AS), etc. Such algorithms do not store elements themselves, but store the counts of elements. CMS is a typical and most widely used sketch structure. It maps the key-value pairs of data packets to counters through Hash function calculation, and each counter bucket mapped needs to be updated. CUS only needs to update the smallest counter bucket. CS is similar to CMS, but each array in CS is related to two Hash functions. AS adds an additional filter to sketch to dynamically capture frequent items. The above sketches provide functions in the frequency counting of network traffic measurement, but they have problems such as high memory access frequency, high Hash calculation resource consumption, low accuracy, large error, etc.

[0004] Due to the existence of hash collisions in the hash table, the query result of CMS can only be greater than or equal to the true value, and only overestimates the frequency of elements. In particular, for elements with fewer occurrences, the accuracy will be relatively low. CUS improves the accuracy of data flow counting by using a conservative update method based on CMS, but it cannot perform a delete operation. Because after inserting an element for a period of time, the minimum in the minimum bucket previously mapped by the element may not be itself, other elements may share the counting bucket, so performing a delete operation may affect other elements.

[0005] Based on this, in view of the overestimation problem of the CMS, on the basis of the CUS proposed for improving the accuracy of the CMS, the application proposes a data flow counting method based on a Flag flag bit, aiming to detect whether the data packets mapped to the same counter belong to the same data flow by setting a Flag flag bit, and then taking different counting schemes to improve the counting operation of the counter to improve the measurement accuracy. SUMMARY

[0006] This section is intended to summarize some aspects of the embodiments of the application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions cannot be used to limit the scope of the application.

[0007] In view of the problems existing in the prior art data flow counting method based on the Flag flag bit, the application is proposed.

[0008] Therefore, the purpose of the application is to provide a data flow counting method based on a Flag flag bit.

[0009] To solve the above technical problems, the application provides the following technical solutions:

[0010] Set the initial value of the parameter;

[0011] When a data packet arrives, first propose the key-value pair information in the data packet, then perform hash calculation according to the Hash function of the CUS algorithm to obtain the counter position to which the data packet is mapped;

[0012] When the first data packet arrives, call the insertion function of the CUS structure to update the counter value to which the data packet is mapped to 1;

[0013] Starting from the second data packet that arrives, compare the counter position of each data packet with that of the previous data packet to determine whether the data packet and the previous data packet mapped to the same counter belong to the same data flow;

[0014] Perform data flow query to determine all the mapped counter positions, compare the sizes of the counter values, and select the smallest value as the query result.

[0015] As a preferred scheme of the data flow counting method based on the Flag flag bit, wherein:

[0016] The width value and the depth value of the CUS structure are set to fixed values, and all the counter values in the CUS structure are initialized to 0, and the Flag flag bit is initialized to flag = 1.

[0017] As a preferred scheme of the data flow counting method based on the Flag flag bit, wherein: the CUS structure is a two-dimensional matrix array with a width of w and a depth of d, used to store the counting information of elements, each of the array contains w counters, and also contains d hash functions h j (j = 1, 2,..., d).

[0018] As a preferred scheme of the data flow counting method based on the Flag flag bit, wherein: the hash calculation is represented as h j (key), (j = 1, 2,..., d), and the counter position to which the data packet is mapped is obtained after the hash calculation, denoted as f(x i , p i );

[0019] Where s i represents the flow to which the data packet belongs, and p i represents the counter position to which the data packet is mapped.

[0020] As a preferred scheme of the data flow counting method based on the Flag flag bit, wherein: the insertion function of the CUS structure includes,

[0021] When the data packet arrives, the mapping counting is performed in the form of key-value pair, the hash function in the CUS structure is called first, the key field of the data packet is hashed d times to find the corresponding counter, and then only the smallest value in the mapped counter is increased, that is, the smallest counter value is updated to the current value plus the value field value of the data packet.

[0022] As a preferred scheme of the data flow counting method based on the Flag flag bit, wherein: from the second data packet, the counter position of each data packet is compared with the previous data packet to determine whether the data packet and the previous data packet belong to the same data flow which are mapped to the same counter, including,

[0023] If the counter positions of the two data packets are different, the counting is normally performed according to the counting rule of the CUS;

[0024] If the counter positions of the two data packets are the same, that is, a hash collision occurs, and belong to the same data flow, the flag is recorded as 1;

[0025] If the counter positions of the two data packets are the same, and belong to different data flows, the flag is recorded as 0;

[0026] Finally, the flag value is updated in the counter;

[0027] If flag = 1, then increment the packet counter value by 1;

[0028] If flag = 0, the packet counter value will be increased by 0.5.

[0029] As a preferred embodiment of the data stream counting method based on the Flag bit described in this invention, the data stream query includes:

[0030] The CUS query function is called first to perform a hash calculation based on the key field of the data packet to find the corresponding hash function mapping counter. Then, the minimum value among these counters is found as the frequency estimate of the data packet. Finally, the minimum value is selected as the estimate.

[0031] A data stream counting system based on a flag bit, characterized in that the system comprises:

[0032] The data collection module initializes relevant parameters to facilitate subsequent calculations.

[0033] The data determination module uses the flag bit to detect whether data packets mapped to the same counter belong to the same data stream, and then adopts different counting schemes to improve the counting operation of the counter.

[0034] The data query module determines the final estimated value, improving the measurement accuracy of the data stream.

[0035] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

[0036] A computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

[0037] The beneficial effects of this invention are as follows: By using the Flag bit to detect whether data packets mapped to the same counter belong to the same data stream, and then adopting different counting schemes to improve the counting operation of the counter, the accuracy of calculating the number of data packets in the data stream can be improved, the overestimation of CMS can be reduced, and the measurement accuracy of the data stream can be improved; moreover, the Flag bit is set to Bool type, which only occupies one byte of memory space, and the additional memory required by the Flag can be ignored in the actual environment. Attached Figure Description

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings. Among them:

[0039] Figure 1 The overall structure diagram of the data flow counting method based on the Flag flag bit of the present application.

[0040] Figure 2 The flow chart of inserting a data packet in the data flow counting method based on the Flag flag bit of the present application. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0042] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0043] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0044] Thirdly, the present application is described in detail in conjunction with the schematic diagram, and in the detailed description of the embodiments of the present application, the cross-sectional view of the device structure is partially enlarged without general proportion for the convenience of description, and the schematic diagram is only an example, which should not limit the scope of protection of the present application here. In addition, three-dimensional spatial dimensions including length, width and depth should be included in actual manufacture.

[0045] Embodiment 1

[0046] Referring to Figure 1 and Figure 2 A data flow counting method based on the Flag flag bit includes:

[0047] As Figure 1As shown, the method comprises two parts of element item and CUS; wherein the element item is composed of two modules of item and flag; the item records elements; the flag is a flag bit, which can be used to judge whether the data packet belongs to the same data stream; the CUS is a two-dimensional array with a width of w and a depth of d, which is used to store the counting information of elements; the principle is to detect whether the data packets mapped to the same counter belong to the same data stream according to the flag bit, and then classify and count.

[0048] wherein, Figure 1 The p is a data packet.

[0049] The data stream counting method based on the flag bit, the CUS structure can count the frequency of the data packet corresponding stream, and the structure is as shown in Figure 1 The right half part.

[0050] wherein, the data stream counting method based on the flag bit is recorded as CUSF.

[0051] wherein, the CUS structure is a two-dimensional array with a width of w and a depth of d, each array contains w counters, and also contains d hash functions h j (j=1, 2,..., d).

[0052] When a data packet arrives, it is stored in the form of key-value pair (key, value), wherein the key field can be the attribute information in the data packet header, such as five-tuple (source IP address, source port, destination IP address, destination port and transmission layer protocol), and the value field can be the number of stream occurrences or the size of the stream.

[0053] When updating the count, first, the key field of the arrived data packet is calculated by d times of hash to find the mapped counter, and then only the smallest one (or those) in the mapped counter is increased, that is, the smallest counter value is updated to the current value plus the value field value of the data packet.

[0054] When querying, the key-value pairs generated by different data streams are different, the hash function is calculated according to the key field of the data packet to find the counters mapped by the hash function, and then the smallest value in these counters is found as the frequency estimation value of the data packet, because there is hash conflict, the query result can only be greater than or equal to the real value, so selecting the smallest value as the estimation value is the most accurate.

[0055] Call the above method, as shown in Figure 2 The execution steps are as follows:

[0056] S1: set the initial value of the parameter. It should be noted that:

[0057] The width value and the depth value of the CUS structure are set as fixed values, all the counter values in the CUS structure are initialized as 0, and the Flag flag bit is initialized as flag = 1.

[0058] S2: When a data packet arrives, the key-value pair information in the data packet is first extracted, and then Hash calculation is performed according to the Hash function of the CUS algorithm to obtain the counter position to which the data packet is mapped. It should be noted that:

[0059] When a data packet arrives, the key-value pair (key, value) information in the data packet is first extracted, and then Hash calculation is performed according to the Hash function of the CUS algorithm to obtain the counter position to which the data packet is mapped, denoted as f(s i , p i ).

[0060] Wherein, the Hash calculation is represented as h j (key), (j = 1, 2, …, d), s i represents the flow to which the data packet belongs, and p i represents the counter position to which the data packet is mapped.

[0061] S3: When the first data packet arrives, the insertion function of the CUS structure is called to update the counter value to which the data packet is mapped to 1. It should be noted that:

[0062] The insertion function of the CUS structure is that when a data packet arrives, the key-value pair is mapped for counting, the Hash function in the CUS structure is first called to perform d times of Hash calculation on the key field of the data packet to find the corresponding counter, and then only the smallest value in the mapped counter is increased, that is, the smallest counter value is updated to the current value plus the value field value of the data packet.

[0063] S4: Starting from the second data packet, the counter position of each data packet is compared with that of the previous data packet to determine whether the data packet and the previous data packet mapped to the same counter belong to the same data flow. It should be noted that:

[0064] If the counter positions of the two data packets are different, the CUS counting rule is used for normal counting;

[0065] If the counter positions of the two data packets are the same, that is, a Hash collision occurs, and they belong to the same data flow, the flag is recorded as 1;

[0066] If the counter positions of the two data packets are the same, and they belong to different data flows, the flag is recorded as 0;

[0067] Finally, the counter is updated according to the value of the flag.

[0068] If flag = 1, the data packet counter value is increased by 1;

[0069] If flag = 0, the data packet counter value is increased by 0.5.

[0070] S5: Perform data stream query, determine all counter positions of mapping, compare the size of the counter value, and select the smallest value as the query result. It should be noted that:

[0071] The CUS query function is called, first, the corresponding hash function mapping counter is found according to the hash calculation of the key field of the data packet, then the smallest value in these counters is found as the frequency estimation value of the data packet, and finally the minimum value is selected as the estimation value.

[0072] Embodiment 2

[0073] The embodiment is the fourth embodiment of the application, which is different from the first embodiment in that a verification test of the data stream counting method based on the Flag flag is provided, and the technical effects adopted in the method are verified and explained.

[0074] When Count-Min Sketch handles the element counting problem of data stream, due to the hash collision of the hash table, CMS only overestimates the frequency of the element, especially for the elements with less occurrence, the accuracy is relatively low. In order to improve the accuracy of CMS, the Conservative Update Sketch is proposed, which uses the conservative update method to improve the accuracy of data stream counting, but it cannot perform the deletion operation, because after inserting the element for a period of time, the minimum in the minimum bucket previously mapped by the element may not be the element itself, and other elements may share the counting bucket, so if the deletion operation is performed, it may affect other elements. On the basis of CMS and CUS, the application proposes a data stream counting method CUSF based on the Flag flag, adds a Flag flag of Bool type, and adopts different counting schemes to improve the counting operation of the counter to improve the measurement precision.

[0075] CUSF and CUS experimental comparison: assuming that there are a and b two data streams that need to be updated to the counter in a period of time, the arrival sequence of the data packet is a, b, a, b, b, a, a, b, a, a, the counter is initialized to 0, flag = 1, the width w of CUS is 10, the depth d is 2, and the positions of a and b mapped to the counter are at least the same.

[0076] There are two cases: in the first case, the two flows a and b are mapped to the same location in the counter, i.e., h1(a)=h1(b), h2(a)=h2(b).

[0077] In the second case, the two flows a and b are mapped to the same location in the counter, i.e., h1(a)=h1(b), h2(a)=h2(b).

[0078] Taking the first case as an example, it is assumed that h1(a)=h1(b)=2, h2(a)=h2(b)=5, and the counting results of CUS and CUSF are shown in Table 1 and Table 2.

[0079]

[0080] Table 1: Counting results of data flows on CUS

[0081]

[0082] Table 2: Counting results of data flows on CUSF

[0083] In the CUS algorithm, the estimated value of the data flow a and b is queried, the query result is 10 packets, while the actual number of data packets of the data flow a is only 6 packets, and the number of data packets of the data flow b is only 4 packets, the query accuracy of the data flow a and b is 60% and 40% respectively, and in the CUSF algorithm, the query result is 7 packets, the query accuracy of the data flow a reaches 86%, and the accuracy of the data flow b is 57%, compared with the CUS algorithm, the query precision of the CUSF algorithm is improved by about 22%. It can be seen that the accuracy of the CUSF algorithm is higher, and the CUSF algorithm is more suitable for traffic measurement in high-speed network environment.

[0084] In summary, the application provides a data flow counting method based on a Flag flag bit, a Flag flag bit of a Bool type is set (the set Flag flag bit is of a Bool type, only occupies a byte of memory space, and the additional memory required by the Flag can be ignored in the actual environment), which is used for detecting whether the data packets mapped to the same counter belong to the same data flow as the previous data packets, if the two data packets mapped to the same counter are the same (i.e., a hash collision occurs), and belong to the same data flow, the flag is recorded as 1, if they belong to different data flows, the flag is recorded as 0, then the flag is updated in the counter according to the value, the number of data packets in the data flow can be more accurately calculated, the overestimation of the CMS is reduced, and the measurement accuracy of the data flow is improved.

[0085] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A method for counting data flow based on a flag bit, comprising: Comprising, ​ setting initial values of parameters; When a data packet arrives, the key-value pair information in the data packet is first proposed, and then the Hash function of the CUS algorithm is used for Hash calculation to obtain the calculator position to which the data packet is mapped; When the first data packet arrives, the insertion function of the CUS structure is called, and the counter value to which the data packet is mapped is updated to 1; From the second arriving data packet, the counter position of each data packet is compared with that of the previous data packet to determine whether the data packet and the previous data packet mapped to the same counter belong to the same data stream; Performing data stream query, determining all the counter positions mapped, comparing the counter values, and selecting the smallest value as the query result; From the second arriving data packet, the counter position of each data packet is compared with that of the previous data packet to determine whether the data packet and the previous data packet mapped to the same counter belong to the same data stream, comprising, If the counter positions mapped by the two data packets are different, the CUS counting rule is used for normal counting; If the counter positions mapped by the two data packets are the same, that is, a hash collision occurs, and belong to the same data stream, the flag is recorded as 1; If the counter positions mapped by the two data packets are the same, and belong to different data streams, the flag is recorded as 0; Finally, the flag value is updated in the counter; If the flag = 1, the data packet counter value is increased by 1; If the flag = 0, the data packet counter value is increased by 0.

5.

2. The method of claim 1, wherein the method is based on a flag bit. Setting initial values of parameters comprises, Setting the width value and the depth value of the CUS structure to fixed values, and initializing all the counter values in the CUS structure to 0 and the flag flag to 1.

3. The method of claim 2, wherein: the flag is a flag bit of a packet header of the data stream. The CUS structure is a two-dimensional matrix array of width w and depth d for storing count information of elements, each of the array containing w counters and also d hash functions . ​ 4. The method of claim 1, wherein the method is based on a flag bit. The hash calculation is represented as After the hash calculation, the counter position to which the data packet is mapped is obtained, denoted as ; wherein represents the flow to which the data packet belongs, represents the counter position to which the data packet is mapped.

5. The method of claim 1, wherein: the flag is a Flag flag. The insertion function of the CUS structure comprises, ​ When a data packet arrives, the key-value pair is mapped for counting, the Hash function in the CUS structure is first called to perform d times of Hash calculation on the key field of the data packet to find the corresponding counter, and then only the smallest value in the mapped counter is increased, that is, the smallest counter value is updated to the current value plus the value field value of the data packet.

6. The method of claim 1, wherein: Performing the data stream query comprises, The CUS query function is called, the Hash calculation is first performed on the key field of the data packet to find the corresponding counter mapped by the Hash function, then the smallest value in the counters is found as the frequency estimation value of the data packet, and finally the smallest value is selected as the estimation value.

7. A system for counting data flows based on a Flag marker, applying a method for counting data flows based on a Flag marker according to any one of claims 1 to 6, characterized in that, Comprising: A data collection module initializes relevant parameters for subsequent calculation; A data determination module uses a flag to detect whether the data packets mapped to the same counter belong to the same data stream, and then uses different counting schemes to improve the counting operation of the counter; A data query module determines the final estimation value to improve the measurement accuracy of the data stream.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images