Route-independent whole network large flow detection method based on adaptive sampling

Abstract

The invention firstly provides a sampling method applied to network measurement, and provides a routing-independent whole-network large flow detection method based on adaptive sampling on the basis of the sampling method of network measurement, and the detection method comprises the following steps: a measurement point sends a sampled data packet to a central controller according to the sampling method; the central controller determines the minimum value in the number of the received data packets sampled at all the measurement points at the end of the current fixed time interval, and selects kmin data packets corresponding to different Hash values from the data packets according to the sequence of the Hash values from small to large; and the central controller counts the number of data packets of different streams in the selected kmin data packets according to the stream identifiers. According to the routing-independent whole-network large flow detection method based on adaptive sampling, topology between measurement points and the routing condition of flow are not concerned, the occupied data surface memory is small, and the whole-network large flow is accurately detected.

Description

Technical Field

[0001] The present invention relates to the field of network measurement, in particular to the field of large flow detection, and more specifically to a method for detecting large flows in the entire network that is independent of routing. Background Art

[0002] With the rapid development of the Internet, the network has become an indispensable part of people's lives. However, the reliability and stability of the network are always under attack and threat. According to the report "Cisco Visual Networking Index: Forecast and Methodology (2016-2021)" (Reference 1), global IP traffic will reach 3.3ZB per year in 2021, or 278EB per month. The global IP traffic in 2021 is equivalent to 127 times the IP traffic in 2005. Internet traffic in mid-March 2020 increased by 25% to 35% compared with February. A technical report (Reference 3) shows that with the sharp increase in Internet traffic, web application attacks such as DDoS (Distributed Denial of Service) are also increasing. In order to better maintain the security and reliability of the network, network measurement has become an indispensable part of modern network management.

[0003] Network measurement can help operators understand the performance, status and anomalies of the network by using specific algorithms and data structures to count and analyze useful information in the original traffic, so as to take timely measures to optimize the network service quality and stability. For example, network algorithms such as routing (reference 4), load balancing (reference 5), service quality assurance (reference 6), and anomaly detection (reference 7) all need to understand certain key indicators of the traffic through network measurement.

[0004] Heavy hitters, also known as elephant flows, refer to a small number of flows that make up most of the network traffic in a short period of time. Researchers usually refer to flows whose proportion of the total traffic exceeds a certain threshold as heavy hitters. Heavy hitters carry a large portion of the total traffic, increase the completion time of delay-sensitive small flows, and are the main source of congestion. Therefore, heavy hitter detection is of great significance. For example, in order to detect and mitigate distributed denial of service attacks or TCP Incast in real time (Reference 8), network operators often monitor the destinations of a large number of traffic from different sources. In traditional networks, such detection relies on analyzing packet samples or flow logs (References 9 and 10). With the emergence of software-defined networks (SDN) and the recent progress in programmable data plane design (Reference 11), it is possible to program directly on the data plane using domain-specific languages ​​(such as P4).

[0005] One problem faced by network heavy hitter detection is that heavy hitters usually pass through multiple measurement points, that is, there may be only a small number of packets at one measurement point, but after aggregation at multiple measurement points, there are a large number of packets, which exceeds the threshold of network-wide heavy hitters. Some work designed for heavy hitter detection at a single measurement point (reference 12) cannot meet the needs of accurate heavy hitter detection, while network operators often need to track network-wide heavy hitters, so further research on network-wide heavy hitter detection is very necessary.

[0006] The whole network large flow detection usually uses multiple measurement points as the data plane, and the central controller as the control plane aggregates the measurement results of the data plane to form a whole network view, reducing the probability of missing large network flows at the cost of communication overhead. The development of programmable switches provides a new technical means for the whole network large flow detection. Programmable switches can be used as measurement points to deploy special data structures and algorithms to achieve multiple measurement functions.

[0007] Due to the large measurement network, it is difficult to obtain the full topology, and the measurement node failure leads to topology update. In many cases, the accurate topology information of the measurement network cannot be obtained. Moreover, with the development of multi-channel TCP and other protocols, the assumption of a single fixed path for traffic routing can no longer model the current network. Therefore, it is necessary to implement full-network large flow detection in an independent way under arbitrary topology and routing. Routing independence means not caring about the topology and flow routing between measurement points. In this way, on the one hand, the system does not need to maintain complex topology information, and on the other hand, when there is a node failure during the measurement process, it does not affect the continuation of the full-network measurement task. The implementation of full-network large flow detection in an independent way includes the following two challenges: First, the same data packet may pass through one or more measurement points and has a certain probability of being sampled to the central controller at multiple measurement points, resulting in the central controller repeatedly counting the same data packet, and the measurement results of the full-network large flow are inaccurate; second, due to the limited storage space of the data plane in the existing programmable switches, such as Tofino, which has only 20MB of on-chip storage space, it is necessary to reduce the storage space occupied in the data plane while ensuring accurate detection.

[0008] Some work has implemented a method for detecting large flows in the entire network regardless of routing. IP Mark (Reference 13) uses an unused bit in the IP header. An attacker can avoid being sampled and discovered to escape detection by marking the bit in the data packet as 1 in advance, so it lacks security. INVEST (Reference 14) uses the merging property of Hyperloglog (Reference 15) to eliminate duplicate packet calculations, but cannot record flow ID-related information, so it is not reversible. AROMA (Reference 16) uses the merging property of the KMV (K-Minimum Value) algorithm (Reference 17) to solve the problem of packet duplication, and can perform flow ID-related measurement tasks, but it requires a large amount of memory on the data plane. When the number of sampled packets of the KMV algorithm is set to 10 6 In the IPv4 scenario, each data plane device requires 13.6MB of storage space, while in the IPv6 scenario, it requires 32.8MB of storage space, which is unacceptable for existing data plane devices.

[0009] The reference list is as follows:

[0010] [1]Cisco VN I.Cisco visual networking index:forecast andmethodology2016–2021[J].CISCO White paper, 2017.

[0011] [2]Bienkowski T. Covid-19 network traffic patterns: A worldwide perspective from our customers [EB / OL] 2020. https: / / www.netscout.com / blog / Network-Traffic-inthe-Age-of-C.

[0012] [3]lexander Gutnikov Oleg Kupreev E B.Ddos attacks in q1 2020,[EB / OL]2020.https: / / securelist.com / ddos-attacks-in-q1-2020 / 96.

[0013] [4]Benson T,Anand A,Akella A,et al.MicroTE:Fine grained trafficengineering for data centers[C] / / Proceedings of the seventh conference onemerging networking experiments and technologies.2011:1-12.

[0014] [5]Dittmann G,Herkersdorf A.Network processor load balancing forhigh-speed links[C] / / Proceedings of the 2002International Symposium onPerformance Evaluation of Computer and Telecommunication Systems.Citeseer,2002,735.

[0015] [6]Kabbani A,Alizadeh M,Yasuda M,et al.AF-QCN:Approximate fairnesswith quantized congestion notification for multi-tenanted data centers[C] / / 2010 18th ieee symposium on high performance interconnects.IEEE,2010:58-65.

[0016] [7]Garcia-Teodoro P,Diaz-Verdejo J,Maciá-Fernández G,et al.Anomaly-based network intrusion detection:Techniques,systems and challenges[J].computers&security,2009,28(1-2):18-28.

[0017] [8]Chen Y,Griffith R,Liu J,et al.Understanding TCP incast throughputcollapse in datacenter networks[C] / / Proceedings of the 1st ACM workshop onResearch on enterprise networking.2009:73-82.

[0018] [9]Schmoll C,Aitken P,Claise B.Guidelines for IP flow informationexport(IPFIX)testing[R].2009.

[0019]

[10] RFC 3954:Cisco systems NetFlow services export version 9[J].2004.

[0020]

[11] Pit-Claudel B,Desmouceaux Y,Pfister P,et al.Stateless load-awareload balancing in p4[C] / / 2018IEEE 26th International Conference on NetworkProtocols(ICNP).IEEE,2018:418-423.

[0021]

[12] Sivaraman V,Narayana S,Rottenstreich O,et al.Heavy-hitterdetection entirely in the data plane[C] / / Proceedings of the Symposium on SDNResearch.2017:164-176.

[0022]

[13] Afek Y,Bremler-Barr A,Feibish S L,et al.Detecting heavy flows inthe SDN match and action model[J].Computer Networks,2018,136:1-12.

[0023]

[14] Ding D, Savi M, Pederzolli F, et al. INVEST: Flow-based traffic volume estimation in data-plane programmable networks [C] / / 2021IFIP NetworkingConference (IFIP Networking). IEEE, 2021:1-9.

[0024]

[15] Flajolet P, Fusy Gandouet O, et al. Hyperloglog: the analysis of anear-optimal cardinality estimation algorithm [C] / / Discrete Mathematics and Theoretical Computer Science. Discrete Mathematics and Theoretical Computer Science, 2007: 137-156.

[0025]

[16] Basat RB, Chen

[0026]

[17] Beyer K, Haas PJ, Reinwald B, et al. On synopses for distinct-value estimation under multiset operations[C] / / Proceedings of the 2007ACM SIGMODinternational conference on Management of data.2007:199-210. Summary of the invention

[0027] Therefore, the purpose of the present invention is to overcome the defects of the prior art and provide a sampling method for network measurement and a method for detecting large flows in the entire network.

[0028] According to a first aspect of the present invention, there is provided a sampling method for network measurement, wherein the network architecture includes a central controller and a plurality of measurement points, and the method comprises the following steps:

[0029] S10: the measuring point uses a predetermined hash function to perform a hash operation on the flow identifier and the packet identifier of the data packet that first passes through the measuring point to obtain a hash value of the data packet;

[0030] S20: the measuring point sends the flow identifier and hash value of the data packet whose hash value is less than or equal to the local dynamic threshold to the central controller;

[0031] S30: The central controller estimates the kth minimum hash value in the current fixed time interval of the measurement point, where the k value is a preset value and an integer;

[0032] S40: The central controller predicts the kth minimum hash value of the measuring point in the next fixed time interval based on the collected kth hash value sequences of the measuring point in previous and current fixed time intervals, and sends it to the measuring point as a local dynamic threshold.

[0033] According to an embodiment of the present invention, the step S10 further includes selecting the data packet that passes through the measurement point for the first time by using a Bloom filter.

[0034] According to an embodiment of the present invention, the hash function in step S10 is the same hash function for all measurement points.

[0035] According to an embodiment of the present invention, the initial value of the local dynamic threshold in step S20 is set to a default value, such as 0.3.

[0036] According to an embodiment of the present invention, the step S30 estimates the kth minimum hash value in the tth fixed time interval of the measurement point using the following formula:

[0037]

[0038] where h j NMP is the measuring point j The maximum hash value sampled, k j NMP is the measuring point j The actual number of sampled packets.

[0039] According to an embodiment of the present invention, in step S40, the algorithm used by the central controller to predict the kth minimum hash value of the measurement point in the next fixed time interval is EWMA, LSTM or Transformer.

[0040] According to a second aspect of the present invention, a method for detecting a large flow in the entire network independently of routing based on adaptive sampling is provided, wherein the detection architecture includes a central controller and multiple measurement points, and the detection method includes the following steps:

[0041] Step S100: the measuring point sends a sampled data packet to the central controller according to the sampling method for network measurement of the present invention, wherein the predetermined hash function is the same hash function for all measuring points;

[0042] S200: The central controller determines the minimum value k of the number of data packets sampled from all measurement points received at the end of the current fixed time interval. min ;

[0043] S300: The central controller selects k from all sampled data packets received at the current fixed time interval in ascending order of hash values. min Data packets corresponding to different hash values;

[0044] S400: The central controller counts the selected k according to the flow identifier min The number of packets of different flows in a packet;

[0045] S500: According to the number of data packets, min The proportion of the network flow in the data packets determines whether the network flow is a large network flow.

[0046] According to an embodiment of the present invention, the central controller estimates the size of the total network traffic according to the following formula:

[0047]

[0048] Among them, h t For the k min The largest hash value among the different hash values.

[0049] According to a third aspect of the present invention, there is provided a network measurement system, comprising a central controller and a plurality of measurement points, characterized in that the sampling method for network measurement according to the present invention samples data packets passing through the measurement points.

[0050] According to a fourth aspect of the present invention, a whole-network large flow detection system is provided, comprising a central controller and multiple measurement points, characterized in that the whole-network large flow is detected according to the routing-independent whole-network large flow detection method based on adaptive sampling of the present invention.

[0051] Compared with the prior art, the advantages of the present invention are: the routing-independent full-network large flow detection method based on adaptive sampling is proposed in the scenario of arbitrary topology and routing, does not care about the topology and flow routing between measurement points, occupies data plane storage space, and solves the problem of repeated calculation of data packets while accurately detecting the full-network large flow and is not vulnerable to attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0053] Figure 1 The whole network measurement architecture of the present invention includes a central controller and multiple measurement points;

[0054] Figure 2 The framework of the routing-independent full-network large flow detection method based on adaptive sampling proposed by the present invention;

[0055] Figure 3 The process of adaptive sampling of measurement points proposed by the present invention;

[0056] Figure 4 The central controller proposed in the present invention predicts the dynamic sampling threshold, estimates the size of the entire network traffic, and detects the process of large flows in the entire network. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following is a detailed description of the self-adaptive routing oblivious sampling method for detecting large flows in the entire network (SAROS) based on adaptive sampling from three aspects: working principle, specific implementation steps and experimental verification, in conjunction with the accompanying drawings.

[0058] The Bloom filter and the exponentially weighted moving average (EWMA) algorithm used in the present invention are common knowledge in the art and will not be described in detail.

[0059] The KMV algorithm used in the present invention is briefly introduced as follows.

[0060] The KMV algorithm (K-Minimum Value) is based on the idea of ​​hash distance estimation and uses the properties of random hash operations to approximately estimate the number of different elements in the data stream. The intuition of KMV is: suppose there is a very uniform hash function, and the element set S = {x1, x2, ... x n} is uniformly mapped to the interval [0,1]. Since the hash operation is uniform, the number of different elements in the set S can be estimated by the average spacing between the hash values ​​of the elements.

[0061] For example, we can estimate how many different elements there are in set S by using only the difference between the minimum hash value and the value 0. Assume that set S has 8 different elements, and the minimum hash value after mapping is 0.1. We use 0.1 to replace the average distance of the element hash values, and finally estimate that the number of different elements in set S is 10.

[0062] However, estimating the number of different elements using only one element hash value has a large error. Therefore, the average distance between multiple adjacent element hash values ​​can be calculated to more accurately estimate the number of different elements in the set S. For the data set S, the set obtained by taking the first K hash values ​​(referring to the first K after the hash values ​​are arranged in ascending order) after the hash operation is KMV = {h1,h2,...h K}, then the average distance between the first K hash values ​​can be calculated as Since the range of hash values ​​is [0,1], the estimated number of different elements in the set is Where K-1 is the algorithm correction.

[0063] The KMV algorithm supports three operations: update (ADD), query (QUERY), and merge (UNION). The update operation is to hash the incoming elements into the interval [0,1]. The query operation is to take the first K hash values ​​after ascending order as the KMV set, and use the maximum value in the KMV set to estimate the number of elements as the query output. The merge is to merge the operation results of multiple instances of KMV into the overall operation result. Suppose there are two different instance hash operation results A and B, and the number of element hash values ​​is K respectively. A and K B , let K = min(K A ,K B ), when A and B are merged, the first K different hash values ​​after ascending order are taken as the return value of the UNION operation. Similarly, the KMV algorithm can also merge multiple instances through multiple merge operations. In the SAROS proposed by the present invention, the merging property of the KMV algorithm is used to achieve unbiased uniform sampling of the entire network, thereby solving the problem that repeated sampling of data packets is difficult to remove duplicates.

[0064] 1. Working Principle

[0065] The measurement architecture of the present invention is as follows Figure 1As shown, a central controller 101 and multiple measurement points 102 (such as NMP1, NMP2, NMP3, NMP4), the network topology and the flow routing (such as p1, p2, p3) are arbitrary. The measurement point 102 performs data plane sampling, and the central controller 101 samples each measurement point 102 at a fixed time interval T p The sampled data sent within the time interval is used to predict the dynamic sampling threshold of the next time interval of the measurement point 102, estimate the size of the entire network traffic, and detect the large network flow.

[0066] The framework of the routing-independent full-network large flow detection method based on adaptive sampling proposed in the present invention is as follows: Figure 2 As shown. In order to reduce the communication volume between the data plane and the control plane and avoid the same measurement point 102 from repeatedly sampling the same data packet 201 to the central controller 101, the measurement point 102 uses a Bloom filter 202 (Bloom Filter) to filter out the data packet 201 that passes through the measurement point 102 for the first time, and then uses the same hash function of all measurement points 102 in the network to hash the flow identifier and packet identifier of the data packet 201 to the range of 0 to 1, and obtain the hash value 203 of the packet to ensure that the hash values ​​203 of the same data packet 201 at different measurement points are the same, so as to facilitate the central controller 101 to aggregate the sampling results. The measurement point 102 samples the data packet 201 according to the hash value 203 of the packet and the local threshold 204. If the hash value 203 of the packet is less than or equal to the local threshold 204, the relevant information of the data packet 201 is sent to the central controller 101. The local threshold 204 is set by the central controller 101 at a fixed time interval T p Periodic update, first fixed time interval T p The local threshold 204 is the default value, usually set to 0.3. p In the example, the measurement point 102 actually sends about k packets 205 with the smallest hash values ​​to the central controller 101. k is an integer value set by the administrator. The larger the value, the higher the measurement accuracy. However, since the local threshold is periodically updated according to the sampling situation, the packets actually sent usually vary around k within a certain range. The k packets with the smallest hash values ​​refer to the packets 201 corresponding to the first k values ​​after the hash values ​​203 of the packets are arranged in ascending order. The same applies to the following. The central controller 101 merges the sampled packets 201 sent by all the measurement points 102 to obtain k packets. min The k packets with the smallest hash values ​​206 are used to estimate and predict the network-wide high flow. min The data packet with the smallest hash value is a packet sent by the central controller 100 to a fixed time interval T p After the hash values ​​of the packets received in 203 are sorted in ascending order, the first k min The hash values ​​of different packets correspond to the data packets, the same as below.

[0067] The adaptive sampling process of the data plane measurement point 102 is as follows: Figure 3 As shown. Take any measurement point NMP j For example, when a data packet 201 passes through the measurement point NMP j When , the data packet is represented as (x, i), where x is the flow identifier and i is the packet identifier. j The data packet (x,i) is not processed, and the first time it passes the measurement point NMP j The data packet (x,i) uses the same hash function H at all measurement points 102 KMV Perform a hash operation on the flow identifier x and the packet identifier i of the data packet (x,i) to obtain the packet's hash value h (x,i) Measuring point NMP j The resulting hash value h (x,i) With the local dynamic threshold T j Compare, if h (x,i) Less than or equal to T j Then sample the data packet information, and the flow identifier and hash value (x,h (x,i) ) is sent to the central controller 101. In this way, the measuring point NMP j In a fixed time interval period Tp, send about k (set as k j ) packet 205 with the smallest hash value. In the central controller 101, the flow identifier x is used to detect the measurement tasks such as the large flow of the entire network, ensuring that SAROS is reversible (the flow identifier can be checked), and the hash value h (x,i) It is used to remove the data packets 201 that are repeatedly sampled in the whole network and estimate the flow size of the whole network.

[0068] According to one embodiment of the present invention, the central controller 101 predicts the dynamic sampling threshold (a fixed time interval T of the measurement point 102) p The local threshold 204), estimating the size of the entire network traffic, and detecting the large flow of the entire network, the process is as follows Figure 4 As shown. At each fixed time interval T p At the end of the data plane, the central controller 101 first updates the local dynamic sampling threshold 204 of each measurement point 102 in the data plane to reduce the delay of the adaptive sampling cycle and avoid undefined time between time intervals due to long processing time. j , at the current tth fixed time interval T p At the end, the central controller 101 actually receives its sampled k j data packets, let k j The largest hash value among the packets is h jSince all measurement points use the hash function H KMV is uniform, so according to the measuring point NMP j The actual number of sampled packets k j , the largest hash value h j , the measurement point NMP can be estimated j The average packet spacing in Then the measuring point NMP j At the current tth fixed time interval T p The kth smallest hash value in 403T (j,t) can be estimated as k average spacings It is expressed as follows:

[0069]

[0070] The kth minimum hash value refers to the hash value at the kth position after the hash values ​​are arranged in ascending order, and the same applies hereinafter. The central controller 101 collects historical data at each fixed time interval T p The kth hash value sequence 404[T (j,0) ,T (j,1) ,…,T (j,t-1) ,T (j,t) ,], using the exponentially weighted moving average EWMA algorithm to predict the measurement point NMP j At the t+1th fixed time interval T p The kth smallest hash value 405 in is approximately T (j,t+1) , and issue T (j,t+1) , updated to the measurement point NMP j The local dynamic sampling threshold 204 is used for the next fixed time interval T p According to other embodiments of the present invention, the prediction algorithm EWMA can also be replaced by a more complex prediction model, such as LSTM, Transforme, etc.

[0071] The central controller 101 adopts the core idea of ​​estimating the number of elements based on the average spacing of the hash values ​​of the first K elements in the KMV algorithm, and uses the merged sampled data to estimate the size of the entire network traffic. However, since the number of sampled data packets at each measurement point is slightly different due to traffic changes or the influence of the predicted threshold, in order to ensure that the merged sampling information is uniform across the entire network, K here is taken as the minimum number of sampled data packets of all measurement points. Suppose the central controller 101 is at the tth fixed time interval T p The number of data packets sampled at the n measurement points received at the end of j ,...,k n , remember kmin The central controller 101 sorts the received hash values ​​of the data packets sampled by all measurement points in the whole network in ascending order and takes the first k min There are data packets corresponding to different hash values. The largest hash value among these data packets is recorded as h t , then according to the KMV algorithm, the total network traffic size is estimated to be:

[0072]

[0073] Since the k min The hash values ​​of the k packets are different, so there are no duplicate packets. min At the same time, the measurement data sent from multiple measurement points to the central controller 101 are merged into one measurement data, and repeated sampling of the measurement data of the whole network is avoided, and the merged data is uniformly sampled for the whole network.

[0074] k min The distribution of k packets can represent the traffic distribution of the entire network. min The number of packets in a flow among the packets k x k min If the ratio is greater than the network large flow threshold θ set by the network administrator, the flow is determined to be a large flow in the entire network, and its flow identifier x and traffic size information are output. In addition, the sampling rate can be calculated based on the number of merged sampled data packets and the estimated size of the total network traffic, and the size of the total network flow frequency can be estimated.

[0075] In addition, if the network administrator needs to reset the fixed sampling threshold θ or the number of sampled data packets k according to measurement requirements, there is no need to restart the data plane measurement point, but to directly modify it in the control plane code and re-run the control plane. Therefore, the present invention has good operability.

[0076] The present invention does not elaborate on some of the well-known technologies that belong to those skilled in the art.

[0077] 2. Specific implementation steps

[0078] According to the above description, a specific embodiment of the present invention is as follows.

[0079] Step 1: Measuring point NMP j Use Bloom filter to select the data packets that pass through the measurement point for the first time

[0080] Step 2: Measuring point NMP j Sample packets based on their hash values ​​and local thresholds:

[0081] 1) The measurement points use the same hash function H KMVPerform a hash operation on the flow identifier x and packet identifier i of the data packet (x,i) to obtain the hash value h of the data packet (x,i) ;

[0082] 2) The hash value h (x,i) With the local dynamic threshold T j Compare, if the hash value h (x,i) Less than or equal to the local dynamic threshold T j , then the flow identifier and hash value (x,h (x,i) ) is sent to the central controller.

[0083] Step 3: The central controller estimates and outputs the next fixed time interval T p Local sampling threshold at each measurement point

[0084] 1) The central controller estimates NMP according to the following formula j The tth fixed time interval T of the measurement point p The kth smallest hash value T in (j,t)

[0085]

[0086] where h j NMP is the measuring point j The maximum hash value sampled, k j NMP is the measuring point j The number of packets actually sampled;

[0087] 2) The central controller collects historical measurement points NMP j Every fixed time interval T p The kth hash value sequence in [T (j,0) ,T (j,1) ,…,T (j,t-1) ,T (j,t) ,], using the EWMA algorithm to predict the measurement point NMP j The kth minimum hash value T in the t+1th fixed time interval (j,t+1) ;

[0088] 3) The central controller sends T (j,t+1) , updated to the measurement point NMP j The local dynamic sampling threshold.

[0089] Step 4: The central controller estimates and outputs the total network flow size for the previous time interval

[0090] 1) The central controller at the tth fixed time interval T p At the end of j,...,k n Find the minimum value k in min ;

[0091] 2) The central controller sends the tth fixed time interval T p After the hash values ​​of the data packets sampled by all measurement points in the whole network are sorted in ascending order, the first k min The data packets corresponding to different hash values ​​are found, and the largest value h among the hash values ​​of these data packets is found. t ;

[0092] 3) The central controller estimates the size of the entire network traffic according to the following formula:

[0093]

[0094] Step 5: The central controller outputs the network-wide traffic information for the previous time interval

[0095] 1) According to the flow identifier x, the selected k min The number of packets of different flows in a packet k x ;

[0096] 2) Calculation

[0097] 3) If Determine the network as a large flow, and output its flow identifier x and traffic size information

[0098] 3. Experimental Verification

[0099] In order to verify and illustrate the effect of the technical solution of the present invention, the inventor built an experimental environment, collected three segments of data under different data sets, and compared and analyzed the experimental results of the SAROS algorithm of the present invention and the AROMA algorithm, as described below.

[0100] 1. Data Source

[0101] The present invention uses two 900-second segments of real traffic in the MAWI data set collected by the ISP measurement point and the UNIV1 data set as the traffic used for distribution in the experiment. The specific experimental environment built is as follows:

[0102] a) Server: one server with 376 GB memory, 16-core CPU, and Ubuntu 20.04 system.

[0103] b) Experimental environment: Install mininet network simulation tool, install bmv2 to simulate software switch supporting P4 programming, install PI as P4 Runtime Server, install p4c as P4 front-end compiler, install protobuf as grpc default option, install grpc as the basis of PI, install p4-utils (mininet extension)

[0104] c) Test topology: The experimental topology uses Fat-Tree, which is a switch-centric topology. It is divided into three layers: core layer, aggregation layer, and access layer. The experimental test network is set as a Fat-Tree with 4 pods, including 20 programmable switches and 8 hosts. In addition, a packet sending host is used to inject traffic into the test network, and a switch is used to randomly distribute traffic to the access layer switch of the test network.

[0105] 2. Evaluation Metrics

[0106] a) Average relative error

[0107] The average relative error (ARE) is used to evaluate the estimation result of the number of packets in the whole network, which is defined as the average relative error of all consecutive time intervals t all The average relative error of the estimated data packets in the entire network is calculated by the following formula:

[0108]

[0109] in, and |s i | are the i-th time interval t p The system estimate and actual value of the number of packets in the entire network.

[0110] The smaller the mean relative error value, the better the estimation effect.

[0111] b) F1 value

[0112] The F1 value (F1 Score) is an indicator used in statistics to measure the accuracy of a binary classification (or multi-task binary classification) model. It takes into account both the precision and recall of the classification model (it should be further pointed out that the calculation method corresponding to the present invention) and is defined as:

[0113]

[0114] The F1 score can be regarded as a weighted average of the model's accuracy and recall. Its maximum value is 1 and its minimum value is 0. The larger the value, the better the model.

[0115] c) Occupied space

[0116] The footprint measures the size of the data plane storage space used by the system on the programmable data plane. Since the memory resources on the programmable data plane are limited, the smaller the footprint, the better.

[0117] 3. Comparative experimental results and analysis

[0118] The SAROS algorithm of the present invention is compared with the AROMA algorithm of the literature

[16] from four aspects under three data sets, and the experimental results are analyzed. The four aspects are: the estimation error of the number of data packets (flow size) in the whole network when the number of sampled data packets is different (i.e., k value), the detection accuracy of large flows in the whole network (F1 value) when the number of sampled data packets is different, the size of the data plane storage space occupied, and the detection accuracy of large flows in the whole network when the threshold value θ of large flows in the whole network is different.

[0119] a) Estimation error of the number of packets in the entire network (flow size)

[0120] Under three data sets, the estimation error comparison of the number of packets in the whole network by the two algorithms at different numbers of sampled packets (k values) is shown in Table 1.

[0121] Table 1. Network flow size estimation error (Error%)

[0122]

[0123] It can be observed from Table 1 that, on the one hand, the error percentage (Error%) decreases with the increase of the number of samples k at each measurement point, which means that the error in estimating the size of the entire network flow decreases; on the other hand, the error percentages of the two algorithms are almost the same at different k values.

[0124] The above conclusion is because both SAROS and AROMA estimate the size of the entire network flow based on the KMV algorithm. The difference between the two algorithms is that SAROS calculates the average distance of the hashes of all data packets based on the hashes of k consecutive data packets, while AROMA first divides the data into k buckets, selects the smallest hash in each bucket, and calculates the average hash distance of all data packets based on these k hash values. In essence, both use the hash values ​​of k data packets to calculate the average hash distance of all data packets, so they have similar estimation errors. In addition, under the conditions of the UNIV1 dataset, the reason why AROMA's estimation error is about 6 percentage points smaller than SAROS when the number of sampled data packets is 5000 is that SAROS adaptive sampling produces a certain error and the KMV algorithm has a certain randomness.

[0125] b) Detection accuracy of large flows in the entire network (F1 value)

[0126] Under three data sets, the comparison of the detection accuracy of the two algorithms for large flows in the entire network at different numbers of sampled data packets (k values) is shown in Table 2.

[0127] Table 2: Detection accuracy of large flows in the whole network (F1 value)

[0128]

[0129] As can be seen from Table 2, with the increase of the number of sampled data packets k, the accuracy of large flow detection in the whole network of the SAROS algorithm and the AROMA algorithm both tends to increase. For example, under MAWI1 traffic, the F1 value of the SAROS algorithm is 0.38 when k is 100, and it rises to 0.84 when k is 5000; under the MAWI2 data set, the F1 value is only 0.25 when k is 100, but it can reach 0.98 when k is 5000; in the UNIV1 data set, the F1 value rises from 0.75 to 0.9 as k increases. Similarly, the AROMA algorithm is more accurate in detecting large flows in the whole network as the number of sampled data packets k increases.

[0130] At different k values, the F1 value of the SAROS algorithm is higher than that of AROMA, and has better accuracy in detecting large flows in the entire network. When k is 5000, the F1 value of SAROS for large flow detection in the entire network is about 0.45 higher than that of AROMA in the MAWI1 dataset, about 0.28 higher than that of AROMA in the MAWI2 dataset, and about 0.33 higher than that of AROMA in the UNIV1 dataset. At other k values, the F1 value of SAROS is also about 0.3 higher than that of AROMA.

[0131] The above phenomenon is caused by the different mechanisms of AROMA and SAROS to implement the KMV algorithm on the data plane. AROMA needs to hash once before performing the KMV algorithm hash, divide the data packet set passing through a certain measurement point into different buckets, and sample a data packet in each bucket. Because the distribution of data packets in each bucket is different from the distribution of data packets passing through the measurement point due to this bucketing, the sampling rate of some flows in a bucket is greater or less than the sampling rate of the measurement point (the sampling rate of the measurement point can be understood as the ratio of the set k value to the number of data packets passing through the measurement point). Therefore, the distribution of data packets at each measurement point is less similar to the distribution of the data packet set passing through each measurement point. AROMA estimates that based on these sampled data packets, a certain amount of false positives and missed detections will occur. SAROS does not adopt a bucketing strategy, so the effect of large-scale flow detection in the entire network is better than AROMA.

[0132] c) Data plane storage space occupied

[0133] Under the three data sets, the data plane storage space occupied by the AROMA algorithm with different numbers of sampled data packets (k values) is shown in Table 3. Since the SAROS algorithm only needs to store a 32-bit dynamic sampling threshold on the data plane to achieve adaptive sampling, no matter how the data packet sampling number k is set, the SAROS algorithm only occupies 32 bits in the data plane.

[0134] Table 3 Data plane storage space occupied (KB)

[0135] Packet Protocol k＝100 k＝500 k＝1000 k＝5000 IPv4 1.36 6.8 13.6 68 Ipv6 3.28 16.4 32.8 164

[0136] As can be seen from Table 3, as k increases, the data plane storage space occupied by the AROMA algorithm becomes larger and larger. This is because the AROMA algorithm needs to store k groups of flow identifiers and hash values ​​on the data plane, so it occupies a large amount of memory on the data plane, especially in the IPv6 scenario. Taking the flow identifier as a five-tuple as an example, the IP address of an IPv4 data packet occupies 32 bits, and the five-tuple size is 104 bits; the IP address in an IPv6 data packet occupies 128 bits, and the five-tuple size is 296 bits. If the hash value is stored using 32 bits, the memory size required by the AROMA algorithm on the data plane in the IPv4 scenario is (136 bits*k); in the IPv6 scenario or in the scenario where IPv4 and IPv6 data packets are mixed, the memory size required by the AROMA algorithm on the data plane is (328 bits*k). When k reaches 10 6 When the AROMA algorithm is used, the data plane memory size is 13.6MB in the IPv4 scenario, and the memory size is 32.8MB in the IPv6 or mixed scenario, while the Tofino data plane memory size is only 20MB.

[0137] The accuracy of the two measurement tasks of the AROMA algorithm, network-wide flow size estimation and network-wide large flow detection, is positively correlated with the number of sampled data packets k set, and k is positively correlated with the fixed memory size set on the data plane. Since the memory in the data plane is limited, for example, the on-chip memory of the Tofino programmable switch is only 20M, the AROMA algorithm has a visible upper limit on accuracy. Since only a 32-bit dynamic sampling threshold is stored in the data plane, the accuracy of the SAROS algorithm for network-wide flow size estimation and network-wide large flow detection is not constrained by the data plane memory space.

[0138] d) Detection accuracy of large network flows (F1 value) at different large network flow thresholds θ

[0139] The comparison of F1 values ​​of the two algorithms in the whole network large flow detection task is shown in Table 4.

[0140] Table 4 Effect of different network-wide large flow thresholds θ on the detection accuracy of network-wide large flows

[0141] algorithm θ=0.001 θ=0.005 θ=0.01 θ=0.05 SAROS 0.36062 0.78926 0.85185 0.87654 AROMA 0.09273 0.30269 0.43694 0.56962

[0142] As can be seen from Table 4, under different conditions of the whole network large flow threshold θ, the F1 value of the SAROS algorithm is higher than that of the AROMA algorithm, showing better whole network detection accuracy. At the same time, as the value of θ increases, the F1 values ​​of the two algorithms also increase. This is because according to the same number of samples (for example, 1000 in this group of experiments), the smaller the flow, the less likely it is to be estimated accurately. When the whole network threshold is set to a small value, flows with a small frequency but exceeding the whole network threshold may be missed due to inaccurate estimation, or some flows with a small frequency but a frequency in the sample exceeding the threshold may be misreported due to inaccurate estimation.

[0143] The present invention adopts a routing-independent full-network large flow detection scheme based on adaptive sampling, which does not need to know the topological structure of the network and can adjust the sampling threshold according to the data packet sampling result of the measurement point. Under the premise of ensuring the error accuracy of the full-network flow size estimation, not only the full-network large flow detection accuracy is improved, but also the memory space occupied by the data plane is fixed to 32 bits, which does not increase with the increase of the number of data packet sampling, thereby reducing the storage space requirement for the data plane measurement point.

[0144] Although the present invention has been described through preferred embodiments, the present invention is not limited to the embodiments described herein but includes various changes and modifications that may be made without departing from the scope of the present invention.

Claims

1. A sampling method for network measurement, wherein the network architecture includes a central controller and multiple measurement points, characterized in that The following steps are involved: S10: the measuring point uses a predetermined hash function to perform a hash operation on the flow identifier and the packet identifier of the data packet that first passes through the measuring point to obtain a hash value of the data packet; S20: the measuring point sends the flow identifier and hash value of the data packet whose hash value is less than or equal to the local dynamic threshold to the central controller; S30: The central controller estimates the kth minimum hash value in the current fixed time interval of the measurement point, where the k value is a preset value and an integer; S40: The central controller predicts the kth minimum hash value of the measuring point in the next fixed time interval based on the collected kth hash value sequences of the measuring point in previous and current fixed time intervals, and sends it to the measuring point as a local dynamic threshold.

2. The sampling method for network measurement according to claim 1, characterized in that: The step S10 further includes selecting the data packet that passes through the measurement point for the first time by using a Bloom filter.

3. The sampling method for network measurement according to claim 1 or 2, characterized in that: The hash function in step S10 is the same hash function for all measurement points.

4. The sampling method for network measurement according to claim 1, characterized in that: The initial value of the local dynamic threshold in step S20 is set to a default value, such as 0.

3.

5. The sampling method for network measurement according to claim 1, characterized in that: The step S30 estimates the kth minimum hash value in the tth fixed time interval of the measurement point using the following formula: where h j NMP is the measuring point j The maximum hash value sampled, k j NMP is the measuring point j The actual number of sampled packets.

6. The sampling method for network measurement according to claim 1, characterized in that: In the step S40, the algorithm used by the central controller to predict the kth minimum hash value of the measurement point in the next fixed time interval is EWMA, LSTM or Transformer.

7. A method for detecting large flows in the entire network regardless of routing based on adaptive sampling, wherein the detection architecture includes a central controller and multiple measurement points, and the detection method includes the following steps: Step S100: the measuring point sends a sampled data packet to the central controller according to the sampling method of claim 1, wherein the predetermined hash function is the same hash function for all measuring points; S200: The central controller determines the minimum value k of the number of data packets sampled from all measurement points received at the end of the current fixed time interval. min ; S300: The central controller selects k from all sampled data packets received at the current fixed time interval in ascending order of hash values. min Data packets corresponding to different hash values; S400: The central controller counts the selected k according to the flow identifier min The number of packets of different flows in a packet; S500: According to the number of data packets, min The proportion of the network flow in the data packets determines whether the network flow is a large network flow.

8. The method for detecting large flows in the entire network based on adaptive sampling and routing independence according to claim 7, characterized in that: The central controller estimates the size of the entire network traffic according to the following formula Among them, h t For the k min The largest hash value among the different hash values.

9. A network measurement system, comprising a central controller and a plurality of measurement points, characterized in that: According to the method described in any one of claims 1 to 6, data packets passing through the measurement point are sampled.

10. A network-wide high-flow detection system, comprising a central controller and multiple measurement points, characterized in that: The method according to claim 7 or 8 is used to detect the large flow of the entire network.

Images