Network traffic monitoring method and device, electronic equipment and storage medium

Abstract

The invention provides a network traffic monitoring method and device, electronic equipment and a storage medium. The method is applied to the network traffic monitoring device (comprising a statistical module), the statistical module comprises a key value table used for counting a first type of traffic and a count-minimum rough drawing used for counting a second type of traffic, and the first type of traffic is greater than the second type of traffic. The method comprises the following steps: determining traffic characteristics corresponding to each data packet according to header information of each to-be-monitored data packet; performing Hash calculation on the traffic features to obtain Hash features corresponding to the data packets; according to the hash features and the traffic features, storing traffic information corresponding to each data packet to a key value table or a count-minimum approximate graph; and determining the network traffic according to the statistical information in the key value table or the count-minimum approximate graph. According to the invention, the waste of space resources caused by the highly inclined characteristic of the data flow can be avoided, the missing of effective information related to the network flow can be prevented, and the accuracy of the flow monitoring result is improved.

Description

Technical Field

[0001] The present application relates to the field of network monitoring technology, and in particular to a network traffic monitoring method, device, electronic device and storage medium. Background Art

[0002] Network measurement plays a vital role in modern computer networks. It is the basis for tasks such as traffic engineering, anomaly detection, traffic engineering, and network security monitoring. Accurately measuring the size and characteristics of network traffic is essential for efficient resource allocation and stable network performance. As the size and complexity of networks grow, it becomes increasingly challenging to achieve accurate and real-time measurements while minimizing the overhead. Simple graph algorithms have attracted great attention from academia and industry due to their ability to provide compact representations of large-scale network data.

[0003] Existing graph techniques can be divided into two categories: traditional graph algorithms and learning-based solutions. Traditional graph algorithms, such as CM Graph, CU Graph, and Count Graph, use simple counter arrays to record flow information. However, due to the highly skewed nature of data flows, directly using regular counters can lead to significant space waste. In large networks, a few flows (i.e., elephant flows) account for the majority of the traffic, while many other flows are small and sparse (i.e., rat flows) only account for a small portion of the traffic. In this case, traditional graph algorithms have difficulty in effectively handling these heterogeneous traffic patterns. In addition, most counters are underutilized for small flows, while large flows accumulate significant errors due to counter conflicts, and finding the right balance between accuracy and memory usage is a non-trivial task, as increasing the counter size to improve accuracy results in significant memory overhead.

[0004] In recent years, researchers have tried to use machine learning methods to improve the performance of graphs. For example, the learned CM graph uses RNN to learn and infer whether the flow is large flow and uses an additional hash table to record large flows. Other solutions use machine learning to enhance the hashing, updating, and querying processes inside the graph to improve performance. However, the main difference between these methods is that they only learn features from flow IDs and distributions without taking advantage of other valid information carried by packets, which limits their potential in terms of accuracy and adaptability. Summary of the invention

[0005] The present application provides a network traffic monitoring method, device, electronic device and storage medium to solve the defects of the prior art such as waste of space resources due to the inability to cope with the highly skewed characteristics of data flows, and poor accuracy and adaptability due to failure to comprehensively consider the effective information carried by data packets for traffic monitoring.

[0006] The present application provides a network traffic monitoring method, which is applied to a network traffic monitoring device, wherein the network traffic monitoring device includes a statistical module, wherein the statistical module includes a key value table for counting first-class traffic and a count-minimum profile for counting second-class traffic, wherein the first-class traffic is greater than the second-class traffic, and the network traffic monitoring method includes: Obtaining each data packet to be monitored and header information of each data packet; Determining the flow characteristics corresponding to each of the data packets according to the header information of each of the data packets; Performing hash calculation on the traffic characteristics corresponding to each of the data packets to obtain the hash characteristics corresponding to each of the data packets; According to the hash features and traffic features corresponding to each of the data packets, the traffic information corresponding to each of the data packets is stored in the key-value table or the count-minimum profile graph; The network traffic value is determined according to the statistical information in the key-value table or the count-minimum profile map.

[0007] According to the network traffic monitoring method provided by the present application, the data packet includes a target data packet, the target data packet is any one of the data packets, the key-value table includes at least one bucket, each of the buckets includes at least one cell, one of the cells is used to store a key-value pair, and one of the key-value pairs is used to uniquely store the traffic information corresponding to the data packet that meets a traffic characteristic; according to the hash characteristics and traffic characteristics corresponding to each of the data packets, the traffic information corresponding to each of the data packets is stored in the key-value table or the count-minimum profile graph, including: Determine the target bucket corresponding to the target data packet in the key-value table according to the hash feature corresponding to the target data packet; Determining whether there is a first cell in the target bucket that matches the traffic feature corresponding to the target data packet; If a preset condition is not met, storing the flow information corresponding to the target data packet in the target bucket, wherein the preset condition is that the first cell does not exist, there is no empty cell in the target bucket, and the flow corresponding to the target data packet belongs to the second type of flow; When the preset condition is met, the flow information corresponding to the target data packet is stored in the count-minimum profile.

[0008] According to the network traffic monitoring method provided by the present application, the value in the key-value pair in the first cell is used to store the cumulative number of data packets that meet the traffic characteristics corresponding to the first cell, and when the preset condition is not met, the traffic information corresponding to the target data packet is stored in the target bucket, including: If the first cell exists, updating the value of the key-value pair in the first cell according to the traffic information corresponding to the target data packet; If the first cell does not exist, and there is an empty cell in the target bucket, set the value in the key-value pair in a second cell in the empty cell according to the traffic information corresponding to the target data packet; and if there is no empty cell in the target bucket, if the traffic corresponding to the target data packet belongs to the first type of traffic, determine the third cell in the target bucket, and set the value in the key-value pair in the third cell according to the traffic information corresponding to the target data packet, the third cell being the cell with the smallest value in the key-value pair among all cells in the target bucket.

[0009] According to the network traffic monitoring method provided by the present application, setting the value of the key-value pair in the third cell according to the traffic information corresponding to the target data packet includes: After storing the information of the key-value pair in the third cell in the count-minimum profile, the third cell is cleared, and the value of the key-value pair in the cleared third cell is set according to the traffic information corresponding to the target data packet.

[0010] According to the network traffic monitoring method provided by the present application, the count-minimum profile includes at least one counter, and the counter is used to store the cumulative number of data packets that meet a traffic characteristic; the traffic characteristic includes data volume information of the data packet; and the determining of the network traffic value according to the key-value table or the statistical information in the count-minimum profile includes: Obtaining a traffic query request, wherein the traffic query request is used to query a network traffic value that meets target traffic characteristics; Determining a network traffic value that matches the target traffic feature according to a value in a key-value pair in a cell that matches the target traffic feature and data volume information in the target traffic feature; or; According to the value of the counter corresponding to the target traffic feature in the count-minimum profile and the data volume information in the target traffic feature, the network traffic value that meets the target traffic feature is determined.

[0011] According to the network traffic monitoring method provided by the present application, setting the value of the key-value pair in a second cell in the empty cell according to the traffic information corresponding to the target data packet includes: Adjusting the sizes of storage spaces occupied by the key and the value in the second cell respectively by Fibonacci coding to obtain an adjusted second cell; According to the traffic information corresponding to the target data packet, the value in the key-value pair in the adjusted second cell is set.

[0012] According to the network traffic monitoring method provided by the present application, when there are no empty cells in the target bucket, the method further includes: The header information of the target data packet is input into a preset traffic classifier to obtain a result that the traffic corresponding to the target data packet belongs to the first type of traffic, or a result that the traffic corresponding to the target data packet belongs to the second type of traffic.

[0013] The present application also provides a network traffic monitoring device, comprising: A statistics module, comprising a key-value table for counting first-class traffic and a count-minimum profile for counting second-class traffic, wherein the first-class traffic is greater than the second-class traffic; A network measurement module obtains each data packet to be monitored and the header information of each data packet; determines the flow characteristics corresponding to each data packet according to the header information of each data packet; performs hash calculation on the flow characteristics corresponding to each data packet to obtain the hash characteristics corresponding to each data packet; stores the flow information corresponding to each data packet in the key-value table or the count-minimum profile according to the hash characteristics and flow characteristics corresponding to each data packet; and determines the network flow value according to the statistical information in the key-value table or the count-minimum profile.

[0014] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a network traffic monitoring method as described in any one of the above is implemented.

[0015] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a network traffic monitoring method as described in any one of the above is implemented.

[0016] The present application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the network traffic monitoring method as described in any one of the above is implemented.

[0017] The network traffic monitoring method of the present application is implemented, firstly, each data packet to be monitored and the header information of each data packet are obtained; then, the traffic characteristics corresponding to each data packet are determined according to the header information of each data packet, and the traffic characteristics corresponding to each data packet are hashed to obtain the hash characteristics corresponding to each data packet; then, according to the hash characteristics and traffic characteristics corresponding to each data packet, the traffic information corresponding to each data packet is stored in a key value table or a count-minimum profile, and finally the network traffic value is determined according to the statistical information in the key value table or the count-minimum profile. In the present application, a key value table for recording a larger traffic flow and a count-minimum profile for recording a smaller traffic flow are set, and the traffic information corresponding to each data packet is automatically recorded in the key value table or the count-minimum profile according to the hash characteristics and traffic characteristics corresponding to each data packet, and no longer heavily relies on a simple counter array to record the traffic information. On the one hand, it can avoid the defects of waste of space resources in the related technology that cannot cope with the highly tilted characteristics of the data flow, and the defects of excessive memory overhead caused by increasing the counter size to improve the accuracy. On the other hand, by utilizing all the header information of the data packet, it can prevent the leakage of effective information related to the network traffic, and improve the accuracy and adaptability of the traffic monitoring results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a flow chart of a network traffic monitoring method shown in an embodiment of the present application; Figure 2 This is a schematic diagram showing a method of recording flow information corresponding to a target data packet according to an embodiment of the present application; Figure 3 It is a schematic diagram of a process of recording flow information corresponding to a target data packet according to an embodiment of the present application; Figure 4 is a flow chart of another network traffic measurement method shown in an embodiment of the present application; Figure 5 It is a schematic diagram of the physical structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The network traffic monitoring method provided in the present application is applied to a network traffic monitoring device, which includes a statistical module, which includes a key-value table for counting the first type of traffic and a count-minimum profile for counting the second type of traffic, and the first type of traffic is greater than the second type of traffic.

[0022] Figure 1 1 is a flow chart of a network traffic monitoring method shown in an embodiment of the present application. Figure 1 The network traffic monitoring method of the present application may include the following steps: Step 101: Acquire each data packet to be monitored and header information of each data packet.

[0023] In this application, network traffic is monitored by monitoring data packets during network transmission.

[0024] In step 101, it is necessary to obtain all the information of the data packet header (not limited to the five-tuple). Among them, the five-tuple includes the source IP address, the destination IP address, the source port number, the destination port number and the protocol type. The header information of the data packet varies depending on the network protocol. The specific content of the header information of the data packet is not limited in this application. For example, in addition to the five-tuple, the header information of the data packet may also include version information, data volume information, service type, verification method, etc.

[0025] Step 102: Determine the traffic characteristics corresponding to each data packet according to the header information of each data packet.

[0026] In one implementation, for each data packet, the header information of the data packet can be directly used as the flow characteristics corresponding to the data packet, thereby simplifying the data processing steps. For example, the header information of data packet 1 can be directly used as the flow characteristics corresponding to data packet 1.

[0027] In another embodiment, for each data packet, a series of processing can be performed on the header information of the data packet, such as data cleaning (removing noise, processing missing values), feature extraction (selecting key fields, combining features), data conversion (encoding processing, normalization), etc., to obtain the traffic characteristics corresponding to the data packet, thereby enhancing the adaptability of the traffic monitoring method of the present application.

[0028] In specific implementation, any of the above methods may be used to obtain the traffic characteristics corresponding to each data packet.

[0029] Step 103: Perform hash calculation on the traffic characteristics corresponding to each data packet to obtain the hash characteristics corresponding to each data packet.

[0030] Hash calculation is the process of converting input data of arbitrary length into fixed-length output through a specific hash function.

[0031] In this application, for each data packet, by performing a hash calculation on its corresponding traffic characteristics, its corresponding hash characteristics can be obtained. For example, performing a hash calculation on the traffic characteristics corresponding to data packet 1 can obtain the hash characteristics corresponding to data packet 1; for another example, performing a hash calculation on the traffic characteristics corresponding to data packet 2 can obtain the hash characteristics corresponding to data packet 2.

[0032] When implementing step 103, any type of hash function can be used according to actual needs, and this application does not specifically limit this.

[0033] Step 104: According to the hash features and traffic features corresponding to each data packet, the traffic information corresponding to each data packet is stored in a key-value table or a count-minimum sketch (CMS).

[0034] Among them, the count-minimum probability graph is a probabilistic data structure with high space efficiency, which is often used for approximate statistics of element counts in data streams. Based on the characteristics of hash functions, the count-minimum probability graph uses multiple independent hash functions to map the elements in the data stream to a two-dimensional counter array. When an element arrives, each hash function is used to calculate the multiple positions of the element in the counter array, and the counter values ​​at these positions are increased by 1.

[0035] In this application, the first type of traffic can be understood as a larger traffic, such as elephant traffic, and the second type of traffic can be understood as a smaller traffic, such as mouse traffic. The types of traffic in this application are divided into the first type of traffic and the second type of traffic.

[0036] The statistics module is essentially a storage module, which records large flows through key-value tables and small flows through count-minimum profiles. Therefore, the key-value table is equivalent to the heavy part of the statistics module, and the count-minimum profile is equivalent to the light part of the statistics module.

[0037] When implementing step 104, the network traffic monitoring device can automatically record the traffic information corresponding to the data packets into a key-value table or a count-minimum profile according to the hash features and traffic features corresponding to each data packet, so as to facilitate subsequent traffic statistics.

[0038] Step 105: Determine the network traffic value according to the statistical information in the key-value table or the count-minimum profile.

[0039] In the present application, after the flow information of all the data packets to be monitored is recorded in a key value table or a count-minimum profile, the network flow can be obtained based on the statistical information obtained by the record. For example, in a scenario where device 1 sends data X to device 2, device 1 splits data X into multiple data packets for transmission, and the network flow monitoring device can monitor all data packets and record the flow information corresponding to each data packet in a key value table or a count-minimum profile. Therefore, after the data packet is sent, the network flow value of the data transmission between device 1 and device 2 can be determined based on the information obtained from the record in the key value table or the count-minimum profile. It can be understood that when the network flow monitoring method of the present application is actually applied, it is not limited to the above-mentioned scenarios, but can also be applied to scenarios where multiple devices transmit network data to each other.

[0040] The network traffic monitoring method of the present application is implemented, firstly, each data packet to be monitored and the header information of each data packet are obtained; then, the traffic characteristics corresponding to each data packet are determined according to the header information of each data packet, and the traffic characteristics corresponding to each data packet are hashed to obtain the hash characteristics corresponding to each data packet; then, according to the hash characteristics and traffic characteristics corresponding to each data packet, the traffic information corresponding to each data packet is stored in a key value table or a count-minimum profile, and finally the network traffic value is determined according to the statistical information in the key value table or the count-minimum profile. In the present application, a key value table for recording a larger traffic flow and a count-minimum profile for recording a smaller traffic flow are set, and the traffic information corresponding to each data packet is automatically recorded in the key value table or the count-minimum profile according to the hash characteristics and traffic characteristics corresponding to each data packet, and no longer heavily relies on a simple counter array to record the traffic information. On the one hand, it can avoid the defects of waste of space resources in the related technology that cannot cope with the highly tilted characteristics of the data flow, and the defects of excessive memory overhead caused by increasing the counter size to improve the accuracy. On the other hand, it can also prevent the leakage of effective information related to the network traffic by utilizing all the header information of the data packet, and improve the accuracy of the traffic monitoring results.

[0041] In combination with the above embodiments, in one implementation, a data packet includes a target data packet, the target data packet is any one of the data packets, the key-value table includes at least one bucket, each bucket includes at least one cell, a cell is used to store a key-value pair, and a key-value pair is used to uniquely store the traffic information corresponding to a data packet that meets a traffic characteristic.

[0042] In the present application, the size of the key-value table may be w×d, where w is the number of buckets in the key-value table, and d is the number of cells in each bucket. The values ​​of w and d may be set according to actual needs.

[0043] Accordingly, step 104 may include: Step 1041: Determine the target bucket corresponding to the target data packet in the key-value table according to the hash feature corresponding to the target data packet.

[0044] In the present application, according to the hash feature corresponding to a single data packet, it can be uniquely mapped to a bucket in the key-value table. For example, if the traffic feature corresponding to a data packet is f1, and the hash feature is h(f1), the bucket corresponding to h(f1) can be obtained according to the mapping relationship between h(f1) and the bucket position, for example, it can be B1; for another example, if the traffic feature corresponding to a data packet is f2, and the hash feature is h(f2), the bucket corresponding to h(f2) can be obtained according to the mapping relationship between h(f2) and the bucket position, for example, it can be B2.

[0045] Among them, the mapping method between the hash features corresponding to the data packet and the positions of each bucket in the key-value table can be set according to actual needs, and this application does not impose specific restrictions on this.

[0046] Step 1042: Determine whether there is a first cell in the target bucket that matches the traffic feature corresponding to the target data packet.

[0047] In the present application, since one cell stores one key-value pair, and one key-value pair is used to uniquely record the traffic information corresponding to a data packet that meets a traffic characteristic, therefore, by implementing step 1042, it is possible to search in the target bucket whether there is a first cell that matches the traffic characteristic corresponding to the target data packet.

[0048] For example, if there are 10 cells in bucket 1, the key in the key-value pair in cell 1 is uniquely associated with traffic feature 1, the key in the key-value pair in cell 2 is uniquely associated with traffic feature 2, and cells 3-10 are empty. If the traffic feature corresponding to data packet X is traffic feature 1, then it means that there is a cell in bucket 1 that matches traffic feature 1 corresponding to data packet X, that is, cell 1 is the first cell. If the traffic feature corresponding to data packet X is traffic feature 4, since there are only cells corresponding to traffic feature 1 and traffic feature 2 in bucket 1, it means that there is no cell in bucket 1 that matches traffic feature 1 corresponding to data packet X, that is, there is no first cell.

[0049] Step 1043: If the preset conditions are not met, the traffic information corresponding to the target data packet is stored in the target bucket. The preset conditions are that there is no first cell, there is no empty cell in the target bucket, and the traffic corresponding to the target data packet belongs to the second type of traffic.

[0050] Implement step 1043. Once it is determined that the preset condition is not met, it is necessary to record the flow information corresponding to the target data packet in the target bucket.

[0051] In the present application, the value in the key-value pair in the first cell is used to store the cumulative number of data packets that meet the traffic characteristics corresponding to the first cell.

[0052] For example, if cell 1 is used to store the cumulative number of data packets that meet traffic feature 1, then the key (i.e., key) in the key-value pair in cell 1 is uniquely associated with traffic feature 1, and the value (i.e., value) in the key-value pair in cell 1 represents the cumulative number of all monitored data packets that meet traffic feature 1.

[0053] Accordingly, step 1043 may specifically include: If the first cell exists, update the value of the key-value pair in the first cell according to the flow information corresponding to the target data packet; If the first cell does not exist, and there is an empty cell in the target bucket, the value in the key-value pair in the second cell in the empty cell is set according to the traffic information corresponding to the target data packet; and if there is no empty cell in the target bucket, if the traffic corresponding to the target data packet belongs to the first category of traffic, the third cell in the target bucket is determined, and the value in the key-value pair in the third cell is set according to the traffic information corresponding to the target data packet, the third cell being the cell with the smallest value in the key-value pair among all cells in the target bucket.

[0054] According to the flow information corresponding to the target data packet, setting the value of the key-value pair in the third cell may specifically include: After storing the information of the key-value pair in the third cell in the count-minimum profile, the third cell is cleared, and the value of the key-value pair in the cleared third cell is set according to the traffic information corresponding to the target data packet.

[0055] Step 1044: When the preset conditions are met, the flow information corresponding to the target data packet is stored in the count-minimum profile.

[0056] Step 1044 is implemented. If it is determined that the preset condition is met, the flow information corresponding to the target data packet needs to be recorded in the count-minimum profile graph, such as Figure 2 shown. Figure 2 It is a schematic diagram of a method of recording flow information corresponding to a target data packet shown in an embodiment of the present application.

[0057] In the present application, the count-minimum profile includes at least one counter, and one counter is used to store the cumulative number of data packets that meet a traffic feature. For example, if counter 1 is used to store the cumulative number of data packets that meet traffic feature 1, then the value of counter 1 represents the cumulative number of all data packets that meet traffic feature 1 monitored.

[0058] In the present application, the counting-minimum profile is composed of a w×d counter array. Of course, the size of the counter array can also be set according to actual needs.

[0059] In the present application, if the first cell exists in the target bucket, the value in the key-value pair in the first cell is directly increased by 1 to update the traffic information recorded in the first cell.

[0060] If the first cell does not exist, further determine whether there is an empty cell in the target bucket. If there is an empty cell, obtain a second cell in the empty cell, record the traffic information corresponding to the target data packet in the second cell, and specifically set the value in the key-value pair in the second cell to 1. If there is no empty cell, determine whether the traffic corresponding to the target data packet belongs to the first type of traffic. If it does not belong to the first type of traffic (that is, it belongs to the second type of traffic), find out whether there is at least one counter matching the traffic feature corresponding to the target data packet in the count-minimum profile graph. If there is a matching counter, directly add 1 to the counter value. If there is no matching counter, directly record the traffic information corresponding to the target data packet in at least one new counter, specifically setting the count value of the new counter to 1.

[0061] In the present application, if there are no empty cells and the traffic corresponding to the target data packet belongs to the first category of traffic, then first find the cell with the smallest value in the key-value pair among all cells in the target bucket, use that cell as the third cell, and then migrate all the key-value pair information in the third cell to a new counter in the count-minimum profile and record it. After that, clear the third cell, and record the traffic information corresponding to the target data packet in the cleared third cell. Specifically, associate the key in the key-value pair in the cleared third cell with the traffic feature corresponding to the target data packet, and set the value in the key-value pair to 1.

[0062] Figure 3 This is a schematic diagram of a process for recording flow information corresponding to a target data packet according to an embodiment of the present application. Figure 3 When it is necessary to record the flow information corresponding to a packet f1, the network flow monitoring device first locates the packet f1 to the corresponding bucket, that is, bucket B1, through the hash function. Since f1 has been recorded in bucket B1, it only needs to increase the corresponding value by one, for example,<f1,1000> Change to<f1,1001> , where f1 can be represented by the traffic characteristics corresponding to the packet. When you need to record the traffic information corresponding to a packet f2, locate the corresponding bucket, that is, B2. Since bucket B2 is empty, insert it directly into an empty cell.<f2,1> , where f2 can be represented by the traffic characteristics corresponding to the packet. When it is necessary to record the traffic information corresponding to a packet f5, locate the corresponding bucket B3. Since bucket B3 is full and the type of traffic corresponding to packet f5 is small traffic (second type of traffic), it is directly recorded.<f5,1> Record it in the corresponding counter (there can be multiple counters here) in the count-minimum profile. When you need to record the traffic information corresponding to a packet f8, locate the corresponding bucket, that is, bucket B4. Since bucket B4 is full and the type of traffic corresponding to packet f8 is large traffic (first type of traffic), first record the smallest value in bucket B4.<f7,5> Evict to the corresponding counter in the count-minimum summary graph, and<f8,1> Record to original<f7,5> location.

[0063] In the present application, the traffic information corresponding to each data packet is automatically recorded in a key-value table or a count-minimum profile according to the hash features and traffic features corresponding to each data packet. This can avoid the waste of space resources caused by the highly skewed characteristics of the data flow. It can also prevent the omission of valid information related to network traffic by utilizing all the header information of the data packet, thereby improving the accuracy of the traffic monitoring results.

[0064] In combination with the above embodiments, in one implementation, the traffic characteristics include data volume information of the data packets; determining the network traffic value according to the statistical information in the key-value table or the count-minimum profile may specifically include: Obtaining a traffic query request, where the traffic query request is used to query a network traffic value that meets target traffic characteristics; Determining a network traffic value that matches the target traffic feature based on a value in a key-value pair in a cell that matches the target traffic feature and data volume information in the target traffic feature; or; According to the value of the counter corresponding to the target traffic feature in the count-minimum profile and the data volume information in the target traffic feature, the network traffic value that meets the target traffic feature is determined.

[0065] Specifically, taking the network traffic value that meets traffic feature 1 as an example, first search for a cell that matches traffic feature 1 in the key-value table. If there is a matching cell, read the value in the cell (the cumulative number of monitored data packets that meet traffic feature 1). After that, the network traffic value that meets traffic feature 1 can be determined based on the data volume information and value of the data packets recorded in traffic feature 1. Of course, the data volume information, value of the data packets recorded in traffic feature 1, and other pre-defined rules for determining the size of network traffic can also be considered to determine the final network traffic value. This application does not specifically limit other rules for determining the size of network traffic.

[0066] If there is no matching cell, continue to search in the count-minimum profile to see if there is a counter matching traffic feature 1. If there is a matching counter, read the value of the counter (the cumulative number of monitored data packets that meet traffic feature 1). After that, the network traffic value that meets traffic feature 1 can be determined based on the data volume information of the data packets recorded in traffic feature 1 and the value of the counter. Of course, the final network traffic value can also be determined by comprehensively considering the data volume information of the data packets recorded in traffic feature 1, the value of the counter, and other pre-defined rules for determining the size of network traffic, and this application does not limit this.

[0067] The above gives a method for obtaining a network traffic value corresponding to a traffic feature. It can be understood that if you want to obtain the sum of the network traffic values ​​corresponding to multiple different traffic features, you can first obtain the network traffic value corresponding to each traffic feature, and then sum the network traffic values ​​corresponding to each traffic feature to obtain the sum of the network traffic values.

[0068] In combination with the above embodiment, according to the traffic information corresponding to the target data packet, the value of the key-value pair in a second cell in the empty cell is set, including: Adjust the size of the storage space occupied by the key and the value in the second cell respectively by Fibonacci coding to obtain an adjusted second cell; According to the traffic information corresponding to the target data packet, the value in the key-value pair in the adjusted second cell is set.

[0069] In this application, if the storage space of a single cell is fixed, the storage space proportion of the key and value in the cell can be adjusted by Fibonacci coding.

[0070] Fibonacci coding uses the characteristics of the Fibonacci sequence to encode data. In this coding method, data is represented by a combination of different Fibonacci numbers. For the symbol part (key) and the count part (value), they can be regarded as a coding space composed of different combinations of Fibonacci numbers. For example, in a fixed-length coding bit, according to the conventional coding method, the symbol part and the count part may each occupy half of the coding bits. However, after the introduction of Fibonacci coding, some coding bits originally allocated to the symbol part or the count part can be reallocated according to the combination rules of Fibonacci numbers.

[0071] The present application uses Fibonacci coding to adjust the size ratio of the symbol part and the count part, which can make the storage structure more efficiently adapt to different traffic conditions. For example, when a large amount of new and different characteristics of traffic appear in the network traffic, the symbol part needs more space to accurately identify these different traffic characteristics. At this time, using Fibonacci coding, some space of the count part can be adjusted to the symbol part, so that the system can more accurately classify and identify these small flows, thereby enhancing the adaptability of the network traffic monitoring method of the present application.

[0072] In combination with the above embodiments, in one implementation, when there are no empty cells in the target bucket, the method further includes: The header information of the target data packet is input into a preset traffic classifier to obtain a result that the traffic corresponding to the target data packet belongs to the first type of traffic, or a result that the traffic corresponding to the target data packet belongs to the second type of traffic.

[0073] In the present application, the step of obtaining a preset traffic classifier may include: Step 1: Collect a large number of data packet samples from the network environment, which cover various first-class traffic and various second-class traffic. The data packet samples contain complete header information.

[0074] Step 2: For each collected data packet sample, mark it according to its actual traffic size to determine whether it belongs to the first type of traffic (large traffic) or the second type of traffic (small traffic).

[0075] Step 3: Extract relevant features from all header information of the data packet sample, filter the extracted features, remove some features that are less relevant to the traffic size or are not helpful for model training, and reduce the complexity and amount of calculation of the model. Secondly, some features can be transformed, such as normalization and standardization, to make the features have better numerical characteristics.

[0076] Step 4: Select a suitable machine learning model as the traffic classifier, such as decision tree, random forest, support vector machine (SVM), neural network, etc. The specific model can be determined according to actual needs.

[0077] Step 5: Divide the prepared labeled dataset into training set, validation set and test set. The training set is used to train the model, the validation set is used to adjust the model's hyperparameters during the training process to prevent the model from overfitting, and the test set is used to evaluate the performance of the trained model on unknown data.

[0078] Step 6: Use the training set to train the selected model and adjust the model parameters by minimizing the loss function through the optimization algorithm. During the training process, the characteristics of the data packet are input into the model. The model predicts the type of traffic based on the current parameters and compares it with the labeled real traffic type to calculate the loss value. Then, the parameters of the model are updated through methods such as the back propagation algorithm to gradually reduce the loss value.

[0079] Step 7: Adjust the model’s hyperparameters using the validation set.

[0080] In the present application, a traffic classifier is trained using all the information in the packet header and is further used to classify traffic, rather than just the key of the traffic (such as a quintuple). This method is similar to the idea in a large language model, that is, using contextual information (similar to the information in the packet header) to better understand and predict traffic patterns, thereby being able to better predict the type of traffic corresponding to the packet, so that the traffic information corresponding to the packet can be more accurately stored in the appropriate location (key-value table or count-minimum profile), avoiding the problem of space waste caused by the highly skewed characteristics of the data flow. In summary, the present application provides a Sketch-based network traffic measurement method enhanced by a large language model (LLM). By introducing the design concept of a large language model, the traffic classifier can more intelligently separate large and small traffic, and more effectively utilize the entire header information of the packet, which can significantly improve the accuracy and efficiency of network traffic measurement. The implementation principle of the entire network traffic measurement method can be as follows: Figure 4 shown. Figure 4 FIG. 1 is a flow chart of another network traffic measurement method shown in an embodiment of the present application. Figure 4The process of the method in has been described in detail in the previous article, so it will not be repeated here.

[0081] The present application also provides a network traffic monitoring device, comprising: A statistics module, including a key-value table for counting first-class traffic and a count-minimum profile for counting second-class traffic, wherein the first-class traffic is greater than the second-class traffic; The network measurement module obtains each data packet to be monitored and the header information of each data packet; determines the traffic characteristics corresponding to each data packet according to the header information of each data packet; performs hash calculation on the traffic characteristics corresponding to each data packet to obtain the hash characteristics corresponding to each data packet; stores the traffic information corresponding to each data packet in a key-value table or a count-minimum profile according to the hash characteristics and traffic characteristics corresponding to each data packet; determines the network traffic value according to the statistical information in the key-value table or the count-minimum profile.

[0082] In this application, the statistical module is mainly used to realize the recording of flow information, and the network measurement module is mainly used to realize the control algorithm in the whole network flow measurement process.

[0083] According to a network traffic monitoring device provided by the present application, the data packet includes a target data packet, the target data packet is any one of the data packets, the key-value table includes at least one bucket, each of the buckets includes at least one cell, one of the cells is used to store a key-value pair, and one of the key-value pairs is used to uniquely store traffic information corresponding to a data packet that meets a traffic characteristic; the network measurement module includes a storage module, and the storage module includes: A first determination submodule, configured to determine a target bucket corresponding to the target data packet in the key-value table according to a hash feature corresponding to the target data packet; A second determination submodule is used to determine whether there is a first cell in the target bucket that matches the traffic feature corresponding to the target data packet; A first storage submodule, configured to store the flow information corresponding to the target data packet in the target bucket when a preset condition is not met, wherein the preset condition is that the first cell does not exist, there is no empty cell in the target bucket, and the flow corresponding to the target data packet belongs to the second type of flow; The second storage submodule is used to store the flow information corresponding to the target data packet into the count-minimum profile when the preset condition is met.

[0084] According to a network traffic monitoring device provided by the present application, the value in the key-value pair in the first cell is used to store the cumulative number of data packets that meet the traffic characteristics corresponding to the first cell; the first storage submodule includes: An updating submodule, configured to update the value of the key-value pair in the first cell according to the flow information corresponding to the target data packet if the first cell exists; The first setting submodule is used for setting the value of a key-value pair in a second cell in the empty cell according to the traffic information corresponding to the target data packet if the first cell does not exist and there is an empty cell in the target bucket, and for determining a third cell in the target bucket and setting the value of the key-value pair in the third cell according to the traffic information corresponding to the target data packet if there is no empty cell in the target bucket and the traffic corresponding to the target data packet belongs to the first category of traffic, wherein the third cell is the cell with the smallest value in the key-value pair among all cells in the target bucket.

[0085] According to a network traffic monitoring device provided by the present application, the first setting submodule includes: The second setting submodule is used to store the information of the key-value pair in the third cell in the count-minimum profile, clear the third cell, and set the value of the key-value pair in the cleared third cell according to the traffic information corresponding to the target data packet.

[0086] According to a network traffic monitoring device provided by the present application, the counting-minimum profile includes at least one counter, one counter is used to store the cumulative number of data packets that meet a traffic feature, and the traffic feature includes data volume information of the data packets; the network monitoring module includes a traffic calculation module, and the traffic calculation module includes: An acquisition submodule, used to acquire a traffic query request, wherein the traffic query request is used to query a network traffic value that meets the target traffic characteristics; a third determination submodule, configured to determine a network traffic value that meets the target traffic feature according to a value in a key-value pair in a cell that matches the target traffic feature and data volume information in the target traffic feature; or; The fourth determination submodule is used to determine the network traffic value that meets the target traffic feature according to the value of the counter corresponding to the target traffic feature in the count-minimum profile and the data volume information in the target traffic feature.

[0087] According to a network traffic monitoring device provided by the present application, the first setting submodule includes: An adjustment submodule, used for adjusting the size of the storage space occupied by the key and the value in the second cell respectively through Fibonacci coding to obtain an adjusted second cell; The third setting submodule is used to set the value of the key-value pair in the adjusted second cell according to the traffic information corresponding to the target data packet.

[0088] According to a network traffic monitoring device provided by the present application, the network monitoring module also includes a traffic classifier, and the traffic classifier is used to input the header information of the target data packet into a preset traffic classifier to obtain a result that the traffic corresponding to the target data packet belongs to the first category of traffic, or to obtain a result that the traffic corresponding to the target data packet belongs to the second category of traffic. Figure 5 is a schematic diagram of the physical structure of an electronic device shown in an embodiment of the present application, such as Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communications interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a network traffic monitoring method, which includes: Obtaining each data packet to be monitored and header information of each data packet; Determining the flow characteristics corresponding to each of the data packets according to the header information of each of the data packets; Performing hash calculation on the traffic characteristics corresponding to each of the data packets to obtain the hash characteristics corresponding to each of the data packets; According to the hash features and traffic features corresponding to each of the data packets, the traffic information corresponding to each of the data packets is stored in the key-value table or the count-minimum profile graph; The network traffic value is determined according to the statistical information in the key-value table or the count-minimum profile map.

[0089] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0090] On the other hand, the present application also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute a network traffic monitoring method provided by the above methods, the method comprising: obtaining each data packet to be monitored and header information of each of the data packets; Determining the flow characteristics corresponding to each of the data packets according to the header information of each of the data packets; Performing hash calculation on the traffic characteristics corresponding to each of the data packets to obtain the hash characteristics corresponding to each of the data packets; According to the hash features and traffic features corresponding to each of the data packets, the traffic information corresponding to each of the data packets is stored in the key-value table or the count-minimum profile graph; The network traffic value is determined according to the statistical information in the key-value table or the count-minimum profile map.

[0091] In another aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being implemented when executed by a processor to perform a network traffic monitoring method provided by the above methods, the method comprising: obtaining each data packet to be monitored and header information of each of the data packets; Determining the flow characteristics corresponding to each of the data packets according to the header information of each of the data packets; Performing hash calculation on the traffic characteristics corresponding to each of the data packets to obtain the hash characteristics corresponding to each of the data packets; According to the hash features and traffic features corresponding to each of the data packets, the traffic information corresponding to each of the data packets is stored in the key-value table or the count-minimum profile graph; The network traffic value is determined according to the statistical information in the key-value table or the count-minimum profile map.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A network traffic monitoring method, characterized in that: Applied to a network traffic monitoring device, the network traffic monitoring device includes a statistical module, the statistical module includes a key value table for counting first-class traffic and a count-minimum profile for counting second-class traffic, the first-class traffic is greater than the second-class traffic, and the network traffic monitoring method includes: Obtaining each data packet to be monitored and header information of each data packet; Determining the flow characteristics corresponding to each of the data packets according to the header information of each of the data packets; Performing hash calculation on the traffic characteristics corresponding to each of the data packets to obtain the hash characteristics corresponding to each of the data packets; According to the hash features and traffic features corresponding to each of the data packets, the traffic information corresponding to each of the data packets is stored in the key-value table or the count-minimum profile graph; The network traffic value is determined according to the statistical information in the key-value table or the count-minimum profile map.

2. The network traffic monitoring method according to claim 1, characterized in that: The data packet includes a target data packet, the target data packet is any one of the data packets, the key-value table includes at least one bucket, each of the buckets includes at least one cell, one of the cells is used to store a key-value pair, and one of the key-value pairs is used to uniquely store the flow information corresponding to a data packet that meets a flow characteristic; according to the hash characteristics and flow characteristics corresponding to each of the data packets, the flow information corresponding to each of the data packets is stored in the key-value table or the count-minimum profile graph, including: Determine the target bucket corresponding to the target data packet in the key-value table according to the hash feature corresponding to the target data packet; Determining whether there is a first cell in the target bucket that matches the traffic feature corresponding to the target data packet; If a preset condition is not met, storing the flow information corresponding to the target data packet in the target bucket, wherein the preset condition is that the first cell does not exist, there is no empty cell in the target bucket, and the flow corresponding to the target data packet belongs to the second type of flow; When the preset condition is met, the flow information corresponding to the target data packet is stored in the count-minimum profile.

3. The network traffic monitoring method according to claim 2, characterized in that: The value in the key-value pair in the first cell is used to store the cumulative number of data packets that meet the traffic characteristics corresponding to the first cell, and if the preset condition is not met, the traffic information corresponding to the target data packet is stored in the target bucket, including: If the first cell exists, updating the value of the key-value pair in the first cell according to the traffic information corresponding to the target data packet; If the first cell does not exist, and there is an empty cell in the target bucket, set the value in the key-value pair in a second cell in the empty cell according to the traffic information corresponding to the target data packet; and if there is no empty cell in the target bucket, if the traffic corresponding to the target data packet belongs to the first type of traffic, determine the third cell in the target bucket, and set the value in the key-value pair in the third cell according to the traffic information corresponding to the target data packet, the third cell being the cell with the smallest value in the key-value pair among all cells in the target bucket.

4. The network traffic monitoring method according to claim 3, characterized in that: The step of setting the value of the key-value pair in the third cell according to the flow information corresponding to the target data packet includes: After storing the information of the key-value pair in the third cell in the count-minimum profile, the third cell is cleared, and the value of the key-value pair in the cleared third cell is set according to the traffic information corresponding to the target data packet.

5. The network traffic monitoring method according to claim 3, characterized in that: The counting-minimum profile includes at least one counter, one counter being used to store the accumulated number of data packets conforming to a traffic characteristic, wherein the traffic characteristic includes data volume information of the data packets; The determining of the network traffic value according to the statistical information in the key-value table or the count-minimum profile graph includes: Obtaining a traffic query request, wherein the traffic query request is used to query a network traffic value that meets target traffic characteristics; Determining a network traffic value that matches the target traffic feature according to a value in a key-value pair in a cell that matches the target traffic feature and data volume information in the target traffic feature; or; According to the value of the counter corresponding to the target traffic feature in the count-minimum profile and the data volume information in the target traffic feature, the network traffic value that meets the target traffic feature is determined.

6. The network traffic monitoring method according to claim 3, characterized in that: The step of setting the value of the key-value pair in a second cell in the empty cell according to the flow information corresponding to the target data packet includes: Adjusting the sizes of storage spaces occupied by the key and the value in the second cell respectively by Fibonacci coding to obtain an adjusted second cell; According to the traffic information corresponding to the target data packet, the value in the key-value pair in the adjusted second cell is set.

7. The network traffic monitoring method according to claim 3, characterized in that: In the case where there is no empty cell in the target bucket, the method further comprises: The header information of the target data packet is input into a preset traffic classifier to obtain a result that the traffic corresponding to the target data packet belongs to the first type of traffic, or a result that the traffic corresponding to the target data packet belongs to the second type of traffic.

8. A network traffic monitoring device, characterized in that: include: A statistics module, comprising a key-value table for counting first-class traffic and a count-minimum profile for counting second-class traffic, wherein the first-class traffic is greater than the second-class traffic; The network measurement module obtains each data packet to be monitored and the header information of each data packet; determines the flow characteristics corresponding to each data packet according to the header information of each data packet; performs hash calculation on the flow characteristics corresponding to each data packet to obtain the hash characteristics corresponding to each data packet; stores the flow information corresponding to each data packet in the key-value table or the count-minimum profile according to the hash characteristics and flow characteristics corresponding to each data packet; The network traffic value is determined according to the statistical information in the key-value table or the count-minimum profile map.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the network traffic monitoring method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a network traffic monitoring method as described in any one of claims 1 to 7 is implemented.

Images