Route-independent network measurement system and method

Abstract

The invention provides a route-independent network measurement system which comprises a plurality of network measurement points and a centralized controller, each network measurement point maintains a route-independent data sketch, each network measurement point is of a multi-layer structure, each layer comprises a plurality of units provided with bitmaps, and initial values of all bits in the bitmaps are first numerical values; the network measurement point is configured to obtain a packet keyword of a received data packet and a flow keyword of a data flow to which the data packet belongs according to a measurement task; updating a data sketch of the network measurement point according to the keyword, determining an associated layer according to the packet keyword, and determining an associated unit in the associated layer according to the stream keyword; mapping the packet keyword to a position of the bitmap of the associated unit, and updating the value of the position to a second value; and the centralized controller is configured to estimate the number of data packets transmitted by each data stream in the network according to a global data sketch obtained by collecting and aggregating the updated data sketches of the plurality of network measurement points so as to complete a measurement task.

Description

Technical Field

[0001] The present invention relates to the field of network operation and management, in particular to the field of network measurement technology, and more particularly to a routing-independent network measurement system and method. Background Art

[0002] Realizing efficient operation and secure management of the network is inseparable from the real-time collection and analysis of basic network data. Therefore, network administrators usually regularly check, monitor and track network operation to detect abnormal conditions such as network performance loss and paralysis caused by sudden traffic, network attacks, and facility failures. This indispensable work is usually called network measurement. A single measurement point usually cannot completely cover the traffic in the entire network. Therefore, in order to be able to grasp the global view of the network and perform network measurement tasks, such as identifying heavy hitters distributed at multiple network measurement points in the entire network, or using multiple network measurement points to detect Distributed Denial of Service (DDoS) attacks in the entire network, the network measurement system usually needs to combine the data of multiple measurement points to form a measurement of each network flow in the entire network, that is, use multiple network measurement points to perform full network measurement.

[0003] Network-wide measurement usually uses multiple measurement points as the data plane, and the central controller as the control plane aggregates the measurement results of the data plane to form a network-wide view, and performs various network measurement tasks based on the measurement results in the network-wide view, such as measuring the traffic (number of packets) of each flow in the network. However, the challenge of achieving network-wide network measurement is the duplicate counting problem, that is, the data packets that pass through multiple measurement points will be recorded repeatedly, resulting in inconsistencies between the network measurement results and the actual state of the network. Due to the large size of the network to be measured, it is difficult to obtain the full topology, and the topology update caused by the failure of the measurement node, the assumption of a single fixed path for traffic routing can no longer model the current network.

[0004] Existing network measurement technologies are usually based on sketch algorithms. That is, sketch algorithms (such as count-minimum sketch (reference [1] (called Count-Min Sketch), reference [2] (called Bloom Filter)) are deployed on network measurement points, usually using programmable technology to deploy them on the data plane of programmable switches in the network. At this time, the switch deployed with the corresponding data structure is used as a network measurement point. The central controller periodically communicates with each network measurement point in the network to obtain the local data of each network measurement point, and finally merges the local data to obtain the final measurement results. However, this type of network measurement system has the duplicate counting problem mentioned above, such as reference [3] (called Marple) and reference [4] (called This problem is faced in measurement systems such as MAFIA, Flymon, Poseidon, and Jaqen. These measurement systems collect traffic information in the network based on the sketch algorithm on the data plane. However, in order to avoid the problem of double counting, strict restrictions need to be imposed on the distribution of measurement points and traffic patterns in such work. For example, Jaqen uses a data plane based on Univmon in [8], which requires limiting the deployment location of measurement points. It is difficult to deal with problems such as measurement node failure or network topology update in actual networks.

[0005] Some works have focused on route-independent network detection methods. The basic idea of ​​implementing route-independent network measurement is to consider the uniqueness of each data packet when updating the measurement results at the measurement point, and use a globally consistent (i.e., the same at all measurement points) hash algorithm to update the state, so that the updates generated by the same data packet at multiple measurement points are the same. For example, reference [9] (called INVEST) uses the merging property of reference

[10] (called Hyperloglog) to eliminate packet duplication, but this method can only estimate the total flow size in the network and cannot support the recording and query of the state of each flow in the network. Reference

[11] (called AROMA) uses the merging property of reference

[12] (K-Minimum Value, abbreviated as KMV algorithm) to achieve network-wide total flow size estimation and uniform sampling of data packets. This method is mainly used in the field of large flow detection in the entire network. The types of network measurement tasks that can be implemented are limited, and it is impossible to query the information of small flows in the network.

[0006] Therefore, although the existing route-independent full-network measurement method can avoid duplicate counting of data packets to a certain extent, it only counts the total traffic and cannot separately count the counting information of each data flow. As a result, the existing measurement method cannot support the recording and query of the status of each traffic in the network, and cannot query the information of small traffic in the network, resulting in the problem that the types of network measurement tasks that can be implemented are limited and the application scope is small.

[0007] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art.

[0008] References are as follows:

[0009] [1] G.Cormode and S.Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms, 55(1):58–75, 2005.

[0010] [2] Geravand, Shahabeddin, and Mahmood Ahmadi. "Bloom filter applications in network security: A state-of-the-art survey." Computer Networks 57.18 (2013): 4047-4064.

[0011] [3]Narayana S, Sivaraman A, Nathan V, et al. Language-Directed HardwareDesign for Network Performance Monitoring[C] / / Conference of the Acm SpecialInterest Group.ACM, 2017.

[0012] [4]Laffranchini,Paolo,et al.Measurements as first-class artifacts[C] / / IEEE INFOCOM 2019-IEEE Conference on Computer Communications.IEEE,2019.

[0013] [5]Zheng,Hao,et al.Flymon:enabling on-the-fly task reconfigurationfor network measurement[C] / / Proceedings of the ACM SIGCOMM 2022Conference.2022.

[0014] [6]Zhang M,Li G,Wang S,et al.Poseidon:Mitigating Volumetric DDoSAttacks with Programmable Switches.[C] / / Network and Distributed SystemSecurity Symposium.Internet Society,2020.

[0015] [7]Liu Z,Namkung H,Nikolaidis G,et al.Jaqen:A High-PerformanceSwitch-Native Approach for Detecting and Mitigating Volumetric DDoS Attackswith Programmable Switches.[C] / / USENIX Security Symposium.2021.

[0016] [8]Liu,Zaoxing,et al.One sketch to rule them all:Rethinking networkflow monitoring with univmon.[C] / / Proceedings of the 2016 ACM SIGCOMMConference.2016.

[0017] [9]Ding D, Savi M, Pederzolli F, et al. INVEST: Flow-based Traffic Volume Estimation in Data-plane Programmable Networks[C] / / IFIP Networking.2021.

[0018]

[10] Flajolet P, Fusy,Gandouet O,et al.HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm[J].2007.DOI:doi:http: / / dx.doi.org / .

[0019]

[11] Basat,Ran Ben,et al.Routing oblivious measurement analytics[C] / / 2020 IFIP Networking Conference(Networking).IEEE,2020.

[0020]

[12] Beyer K, Haas PJ, Reinwald B, et al. On synopses for distinct-value estimation under multiset operations[C] / / Proceedings of the 2007 ACM SIGMOD international conference on Management of data.2007. Summary of the invention

[0021] Therefore, the object of the present invention is to overcome the above-mentioned defects of the prior art and provide a network measurement system and method that is independent of routing.

[0022] The objective of the present invention is achieved through the following technical solutions:

[0023] According to a first aspect of the present invention, there is provided a network measurement system that is independent of routing, the system comprising a plurality of distributed network measurement points and a centralized controller, wherein each network measurement point maintains a data sketch that is independent of routing, wherein the data sketch is a multi-layer structure, each layer comprising a plurality of units arranged in sequence according to position, each unit being provided with a bitmap, and the initial values ​​of all bits in the bitmap being a first value;

[0024] Each network measurement point is configured to: receive and forward a data packet, obtain a keyword of the data packet according to a network measurement task, including a packet keyword and a flow keyword of a data flow to which the data packet belongs, and update a data sketch of the network measurement point according to the keyword of the data packet, wherein the updating method includes: determining an associated layer according to a hash value of the packet keyword, and determining the associated unit in the associated layer according to the hash value of the flow keyword; mapping the packet keyword to a position in a bitmap of the associated unit, and updating the value of the position to a second value;

[0025] The centralized controller is configured to collect updated data sketches from multiple network measurement points and aggregate all collected data sketches to obtain a global data sketch, and estimate the number of data packets transmitted by each data stream in the network based on the global data sketch to complete the network measurement task.

[0026] In some embodiments of the present invention, at each network measurement point, a threshold range is preset for each layer in its data sketch, and the threshold ranges of all layers of the data sketch do not intersect;

[0027] The method of determining the associated layer includes:

[0028] Performing hierarchical hashing on the packet keyword using a hierarchical hash function to obtain a layer hash value, wherein a hash range of the hierarchical hash function is the sum of threshold ranges of all layers of the data sketch;

[0029] According to the threshold range of the layer where the layer hash value is located, the layer corresponding to the threshold range is used as the associated layer.

[0030] In some embodiments of the present invention, in each network measurement point, the associated unit is determined by:

[0031] In the associated layer, the stream keyword is hashed using a unit hash function to obtain a unit hash value, and the associated unit in the layer is obtained according to the unit hash value mapping, wherein the hash range of the unit hash function is greater than or equal to the total number of units in the layer.

[0032] In some embodiments of the present invention, the number of bits of the bitmaps of the plurality of units of each layer of the data sketch is the same, and in each network measurement point, the manner of mapping to a position in the bitmap of the associated unit includes:

[0033] A bitmap hash function is used to perform bitmap hashing on the packet keyword to obtain a bit hash value, and the bit hash value is mapped to a position in the bitmap of the associated unit according to the bit hash value, wherein the hash range of the bitmap hash function is greater than or equal to the bit size of the bitmap of the corresponding associated unit.

[0034] In some embodiments of the present invention, in the centralized controller, the aggregation method includes:

[0035] The bitmaps of the cells at the same position in all the collected data sketches are aggregated to obtain a global data sketch with the same structure as the data sketch of the network measurement point. The bitmap of each cell in each layer in the global data sketch is obtained by ORing the bitmaps of the cells at the corresponding positions in all the data sketches.

[0036] In some embodiments of the present invention, each network measurement point includes:

[0037] A flow keyword mask register group, used for storing a flow keyword mask, which includes a plurality of flow mask values ​​respectively associated with a plurality of fields of a data packet, each flow mask value being used for indicating whether a flow keyword needs a field associated with the flow mask value;

[0038] The packet keyword mask register group is used to store the packet keyword mask, which includes multiple packet mask values ​​respectively associated with multiple fields of the data packet, and each packet mask value is used to indicate whether a field in the packet keyword needs to be associated with the packet mask value.

[0039] In some embodiments of the present invention, at each network measurement point, a method of obtaining the keyword of the data packet includes:

[0040] Processing the data packet according to the flow keyword mask to obtain a flow keyword including one or more fields, wherein the processing comprises performing a bitwise AND operation on each of the multiple fields of the data packet and a flow mask value associated with the field to obtain a field required for the flow keyword;

[0041] The data packet is processed according to the packet keyword mask to obtain a packet keyword, which includes one or more fields, wherein the processing method includes bitwise ANDing each field of multiple fields of the data packet with a packet mask value associated with the field to obtain the field required for the packet keyword.

[0042] In some embodiments of the present invention, the centralized controller is further configured to:

[0043] According to the fields required by the flow keyword and the fields required by the packet keyword in the network measurement task, the flow keyword mask and the packet keyword mask are determined, and the flow keyword mask and the packet keyword mask are distributed to multiple network measurement points.

[0044] In some embodiments of the present invention, in the centralized controller, the method of estimating the number of data packets transmitted by each data flow includes:

[0045] Obtain a given stream keyword for each data stream, and for each data stream, estimate the number of data packets transmitted by each data stream in the network based on the global data sketch in the following manner: query the global data sketch based on the stream keyword of each data stream to obtain information corresponding to each data stream, wherein the query method includes hashing the stream keyword of the data stream using a unit hash function to obtain a unit hash value, and obtaining a bitmap of a unit in each layer in the global data sketch according to the unit hash value and the total number of units in each layer to obtain a plurality of bitmaps; traverse the plurality of bitmaps in sequence according to the number of layers, and when the ratio of the number of bits in the bitmap being zero to the total number of bits in the bitmap is greater than a preset ratio, estimate the number of data packets transmitted by the data stream using a linear counting algorithm based on the ratio of the bitmap and the threshold range corresponding to the layer where the bitmap is located to the sum of the threshold ranges of all layers; and obtain the data packet counting information of the corresponding task based on the obtained estimated number of data packets transmitted by each data stream and the network measurement task.

[0046] According to a second aspect of the present invention, a whole network measurement method based on the system according to the first aspect of the present invention is provided, the method comprising:

[0047] Based on the network measurement task issued by the centralized controller, each of the multiple network measurement points obtains the keyword of the data packet when receiving and forwarding the data packet, including the packet keyword and the flow keyword of the data flow to which the data packet belongs; each of the multiple network measurement points updates the data sketch of the network measurement point according to the keyword of the data packet, and the updating method includes: determining the associated layer according to the hash value of the packet keyword, and determining the associated unit in the associated layer according to the hash value of the flow keyword; mapping the packet keyword to a position in the bitmap of the associated unit, and updating the value of the position to a second value; collecting the updated data sketches of the multiple network measurement points respectively and aggregating all the collected data sketches through the centralized controller to obtain a global data sketch, and estimating the number of data packets transmitted by each data flow in the whole network according to the global data sketch to complete the network measurement task.

[0048] Compared with the prior art, the advantages of the present invention are:

[0049] On the one hand, in the process of updating the sketch by combining the structure of the data sketch with the measurement point of the present invention, under any measurement point topology and routing path, by obtaining the bitmap at a specific position, it is possible to more accurately measure or query the data packet information transmitted by different data streams, support the subsequent recording and query of each flow status in the network, and record the relevant information on each bitmap of the sketch regardless of the flow size, so as to support the query of the information of small flow in the network. On the other hand, the data sketch structure of multiple network measurement points is the same and the update method of the data sketch is the same. Each measurement point updates the value of the same place of the data sketch to the second data for the same data packet of the same data stream transmitted, that is, the value of the bitmap is set at the same position in the sketch, avoiding repeated counting of the data packet, so that the global data sketch aggregated by the centralized controller also has no repeated counting, and the estimated number of data packets transmitted by each data stream is more accurate. After the estimation of the number of data packets corresponding to each data stream is realized, a variety of network measurement task types can be realized, and the application range is wide. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0051] Figure 1 A schematic diagram of the overall architecture of a measurement system and its measurement process principle according to an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the structural design of a data sketch and its updating principle according to an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the principle of a method for aggregating a global data sketch according to an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a query process of a data sketch according to an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of a network measurement task configuration method and a workflow of a system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] As mentioned in the background technology section, the existing route-independent full-network measurement method can avoid duplicate counting of data packets to a certain extent, but it only counts the total traffic and cannot separately count the counting information of each data flow. As a result, the existing measurement method cannot support the recording and query of the status of each traffic in the network, and cannot query the information of small traffic in the network. As a result, there are problems such as limited types of network measurement tasks that can be implemented and a small scope of application.

[0058] Based on the above problems, the present invention proposes a routing-independent network measurement system, including multiple network measurement points and a centralized controller. First, each network measurement point maintains a data sketch, which is a multi-layer structure, each layer includes multiple units with bitmaps, and the initial values ​​of all bits in the bitmap are the first numerical value. The present invention uses the first numerical value to indicate that the measurement point does not receive any data packets of the data stream at the initial time, so as to facilitate the correct statistics of subsequent data packets. Secondly, the network measurement point: obtains the packet keyword of the data packet received and the flow keyword of the data stream to which the data packet belongs according to the measurement task; updates the data sketch of the network measurement point according to the keyword, including: determining the associated layer according to the packet keyword, determining the associated unit according to the flow keyword in the associated layer; mapping the packet keyword to a position in the bitmap of the associated unit, and updating the value of the position to the second numerical value. In the process of updating the sketch by combining the structure of the data sketch with the measurement point of the present invention, under any measurement point topology and routing path, by obtaining the bitmap of a specific position, it is possible to more accurately measure or query the data packet information transmitted by different data streams, support the subsequent recording and query of each flow state in the network, and record the relevant information on each bitmap of the sketch regardless of the flow size, so as to support the query of the information of small flow in the network. Finally, the centralized controller estimates the number of data packets transmitted by each data stream in the network according to the global data sketch obtained by the updated data sketch of the aggregated multiple network measurement points. The data sketch structure and update method of multiple network measurement points of the present invention are the same, that is, they have global consistency. Each measurement point updates the value of the same place of the data sketch to the second data for the same data packet of the same data stream transmitted, avoids repeated counting of the data packet, and makes the global data sketch also have no repeated counting. The estimated number of data packets transmitted by each data stream is more accurate, and after the estimation of the number of data packets corresponding to each data stream is realized, a variety of network measurement task types can be realized, and the application range is wide.

[0059] In general, according to an embodiment of the present invention, in the network measurement system of the present invention, each network measurement point maintains a data sketch that is independent of routing, wherein the data sketch is a multi-layer structure, each layer includes a plurality of units arranged in sequence according to position, each unit is provided with a bitmap, and the initial value of all bits in the bitmap is a first value; each network measurement point is configured to: receive and forward a data packet, obtain a keyword of the data packet according to a network measurement task, including a packet keyword and a flow keyword of a data flow to which the data packet belongs, and update the data sketch of the network measurement point according to the keyword of the data packet, and the updating method includes: determining an associated layer according to a hash value of the packet keyword, determining the associated unit in the associated layer according to the hash value of the flow keyword; mapping the packet keyword to a position in the bitmap of the associated unit, and updating the value of the position to a second value; the centralized controller is configured to: collect the updated data sketches of each of the plurality of network measurement points and aggregate all the collected data sketches to obtain a global data sketch, and estimate the number of data packets transmitted by each data flow in the network according to the global data sketch to complete the network measurement task. In order to better understand the present invention, the network measurement system is described in detail below in conjunction with specific embodiments.

[0060] (I) The overall architecture of the network measurement system of the present invention and its measurement process are described

[0061] According to one embodiment of the present invention, see Figure 1, which is a schematic diagram of the overall architecture of the measurement system of the present invention and the principle of its measurement process. The overall architecture of the system includes a centralized controller and a plurality of distributed network measurement nodes. Six network measurement points are shown in the figure, each of which is regarded as a data plane, and the centralized controller is regarded as a control plane. During the measurement process, first, the measurement task is sent to each network measurement point through the centralized controller. In the case of unknown routing (i.e., the routing of the data packet and the topology of the measurement point are unknown), each network measurement point updates its data sketch according to the received data packet, obtains the updated data sketch, and uses the updated data sketch as the local measurement result of the measurement point. In the measurement process, each data packet is recorded after passing through the network measurement point, and a data packet is allowed to pass through multiple network measurement points. For example, in the figure, a data packet passes through three network measurement points, and the three network measurement points are all recorded through their data planes (i.e., network measurement points, and the network measurement points are used to represent the data plane in the following), that is, the data sketch in the network measurement point is updated. The control plane (i.e., the centralized controller, which will be referred to as the control plane in the following) collects local measurement results of all network measurement points (i.e., the updated data sketch), aggregates all the collected local measurement results to obtain a global data sketch, and queries and obtains network measurement information based on the global data sketch, including the bitmap information corresponding to each flow keyword, thereby realizing various network measurement tasks, such as measurement task 1, measurement task 2, and measurement task 3.

[0062] According to one embodiment of the present invention, the network measurement point in the system is a device that directly undertakes the task of traffic forwarding in the network, and such a device is required to have a certain data processing capability, such as a programmable switch device that can directly update the data sketch, or a device that can mirror the traffic and process it on an external server. There are no special requirements for the location deployment of the network measurement node, such as deployment at multiple core nodes of the network. The centralized controller in the system needs to collect and process data from each network measurement point. Therefore, the centralized controller can use a conventional general-purpose server device or a cloud server device, and it is necessary to ensure that the centralized controller can interconnect with each network measurement point and receive data from the network measurement point.

[0063] According to one embodiment of the present invention, the general process of the system performing a network measurement task includes:

[0064] 1) The centralized controller issues a measurement task, and after issuing a measurement task, initializes the data sketch of each network measurement point, that is, sets the values ​​of all bits in all bitmaps to the first value.

[0065] 2) Each network measurement point starts measuring and ends measuring after a preset measurement time, and obtains a data sketch updated at each network measurement point within the preset measurement time. During the measurement process, when a data packet passes through a network measurement point and passes through the switch forwarding chip through the network measurement point, the flow keyword and packet keyword related to the network measurement task are obtained from the data packet, that is, <flow keyword, packet keyword>, and the data packets with the same flow keyword are regarded as data packets in the same data flow, and the data packets with different packet keywords are regarded as different data packets. The network measurement point updates its data sketch according to the obtained keyword, thereby recording the information of the data flow to which the data packet belongs.

[0066] 3) The centralized controller collects the updated data sketches of each network measurement point within a preset measurement time, completes aggregation, and obtains a global data sketch. The data sketches of each network measurement point can also be collected in a periodic manner. For example, the data sketches updated by each network measurement point within the 5 minutes are collected every 5 minutes to obtain the measurement results of the task in each cycle. The centralized controller further obtains network measurement information from the global data sketch, and uses queries to obtain the bitmap information corresponding to each flow keyword, thereby realizing a variety of network measurement tasks. Schematically, the basic network measurement tasks in the network are all to measure the statistical information of the data flow in the network, such as the number of data packets, etc., taking the common measurement tasks as an example: measuring the number of data packets of each data flow and measuring how many data packets each source IP has (i.e., finding the IP that sends more data packets in the network). The network measurement tasks in the present invention include measuring the amount of data of each data flow, that is, by querying how many different packet keywords there are under each flow keyword, the number of data packets of each data flow can be counted.

[0067] (ii) Explanation of the data sketch maintained at each network measurement point

[0068] According to one embodiment of the present invention, see Figure 2 , which is a schematic diagram of the structural design of the data sketch maintained by the network measurement point and its updating principle. The structure and updating principle of the data sketch are explained below:

[0069] 1) Structure of the data sketch

[0070] The data sketch is a multi-layer structure. Figure 2 As shown, the data sketch is an n-layer structure, namely layer 1, layer 2 to layer n. A threshold range is preset for each layer in the data sketch, and the threshold ranges of all layers of the data sketch do not intersect. For each layer of the data sketch, a threshold T is maintained, that is, the i-th layer has a threshold T i , and at the same time, for any layer i, T i >T i+1 ,T0=H max,like Figure 2 , the threshold range of layer 1 is [T1, T0), the threshold range of layer 2 is [T2, T1), and the threshold range of layer n is [T n ,T n-1 ), the sum of the threshold ranges of all layers of the data sketch is 0 to H max .

[0071] In a multi-layer structure, each layer includes multiple data units. Figure 2 As shown, the number of data units in each layer from layer 1 to layer n may be the same or different.

[0072] Each of the multiple data units in each layer includes a bitmap. The bit size of the bitmaps of the multiple units in each layer is the same, such as Figure 2 The bit size of the bitmap of all units in layer 2 is 6.

[0073] The data sketch also includes a large flow record part, which is an array of multiple registers. Each register is used as a large flow record unit to record the flow key of the current data packet. The number of registers (i.e., large flow record units) in the large flow record part is denoted as H. The flow key of the data packet that meets the specified conditions will be recorded in the large flow record unit, and the layer corresponding to the large flow record is denoted as layer 0, T h Indicates the threshold corresponding to the large flow record part.

[0074] 2) Data Sketch Update Principle

[0075] like Figure 2 As shown in the figure, when a data packet is transmitted to the network measurement point, the update process principle is as follows:

[0076] ①. The network measurement point obtains the flow keyword (Key flow) and the packet keyword (Key pkg) of the data packet and obtains <flow keyword, packet keyword>.

[0077] According to one embodiment of the present invention, two keywords can be freely defined according to the measurement task. Here, taking the measurement of how many data packets are contained in each data flow as an example, the specified flow keyword is a five-tuple, and the packet keyword is the checksum of the IP header and the TCP / UDP header in the data packet. Schematically, the form of the five-tuple is: "source IP field, destination IP field, source port number field, destination port number field, protocol type field", checksum: the checksum field of the data packet header, that is, a number of a specified number of digits and different for each data packet.

[0078] ②. Determine the associated layer based on the hash value of the package keyword.

[0079] According to an embodiment of the present invention, the method for determining the associated layer includes: performing hierarchical hashing on the packet keyword using a hierarchical hash function to obtain a layer hash value, wherein the hash range of the hierarchical hash function is the sum of the threshold ranges of all layers of the data sketch; and according to the threshold range of the layer where the layer hash value is located, the layer corresponding to the threshold range is used as the associated layer. Figure 2 The sum of the threshold ranges of all layers in is 0 to H max For example, using the hierarchical hash function H d Hash the packet keyword and record the resulting layer hash value as h layer , and the hash range of the hash function of the hierarchical hash is 0 to H max If h layer In the interval [T i ,T i+1 ), then the associated layer is layer i.

[0080] ③. In the associated layer, the associated unit is determined according to the hash value of the stream keyword.

[0081] According to an embodiment of the present invention, in each network measurement point, the associated unit is determined by: in the associated layer, using the unit hash function H s The stream keyword is hashed to obtain a unit hash value, and the associated unit in the layer is obtained according to the unit hash value mapping, wherein the hash range of the unit hash function is greater than or equal to the total number of units in the layer.

[0082] Indicatively, when the hash range of the unit hash function is set to be larger than the total number of units in the layer, the unit hash function is used to hash the stream keyword, and the resulting unit hash value is recorded as h unit At this time, the total number of units in the associated layer i determined in step ② is recorded as N i , the hash range is set to 0 to S i , and S i >N i , at this time, it is necessary to determine in the associated layer that the associated unit is the hth unit in the i-th layer unit %N i Unit, h unit %N i Indicates the remainder operation. When h unit %N i When the remainder of is 3, it means that the associated unit is unit 3 in the i-th layer.

[0083] Indicatively, when the hash range of the unit hash function is set equal to the total number of units in the layer, if the total number of units in the layer is N i , the hash range is set to 0 to N i. Use the unit hash function to hash the stream keyword, and the resulting unit hash value is recorded as h unit ′ . Unit hash value h unit ′ is 5. According to the associated layer i determined in step ②, the associated unit is the 5th unit in the i-th layer.

[0084] ④. Map the package keyword to a position in the bitmap of the associated unit, and update the value of the position to the second value.

[0085] According to an embodiment of the present invention, in each network measurement point, the mapping to a position in the bitmap of the associated unit includes: using a bitmap hash function H b The bitmap hash function is used to obtain the bit hash value of the packet keyword, and is mapped to a position in the bitmap of the associated unit according to the bit hash value, wherein the hash range of the bitmap hash function is greater than or equal to the bit size of the bitmap of the corresponding associated unit. Schematically, after determining the associated unit in the associated layer, the packet keyword is hashed using the bitmap hash function, and the obtained hash value is recorded as h bitmap , at this time, the hth bitmap The value of the bit is set to 1, the range of the bitmap hash is consistent with the bitmap size in the data unit, and the bitmap size in all data units in the same layer is the same. Figure 2 For example, currently, according to the data packet, the associated layer is determined to be layer 2, and the associated unit in layer 2 is determined to be the first unit 1, and the position mapped to the bitmap of the associated unit is the fifth position (the red number part in the figure). Therefore, the value of the fifth bit in the bitmap of unit 1 of layer 2 is updated to the second value, where 0 represents the first value and 1 represents the second value, and the current bitmap of unit 1 is 011010.

[0086] According to one embodiment of the present invention, a threshold T is maintained for the large flow record portion. h , for the hierarchical hash function H through step ② d Hash the package keyword to get the layer hash value h layer , if H max <h layer <T h Then, through step ③, use the unit hash function H s Hash the stream keyword to get the unit hash value h unit , and in the hth unit %H large flow record units record the flow keyword of the current data packet. H represents the total number of large flow record units in the large flow record part, h unit %H represents the remainder operation. unitWhen the %H remainder is 1, it means that the associated unit is unit 1 in the i-th layer.

[0087] The technical solution of the embodiment corresponding to the above-mentioned aspect (ii) can at least achieve the following beneficial technical effects: no matter how the data packet is routed, that is, the same data packet passes through any network measurement point, and the same update will be performed at different measurement points, and there will be no repeated counting. This allows the centralized controller to accurately obtain network traffic after data aggregation, and achieve accurate network measurement that is independent of routing. Therefore, the benefits of route-independent network measurement include: 1. Accurate network measurement, that is, avoiding the problem of repeated counting. 2. More robust, that is, the number and topology of measurement points can be changed at will.

[0088] (III) Explanation of the method of sketching the data of the aggregated measurement points of the centralized controller

[0089] According to one embodiment of the present invention, in a centralized controller, the aggregation method includes: aggregating the bitmaps of the units at the same position in all collected data sketches to obtain a global data sketch with the same structure as the data sketch of the network measurement point, and the bitmap of each unit in each layer in the global data sketch is obtained by OR operation based on the bitmaps of the units at the corresponding positions in all data sketches.

[0090] According to one embodiment of the present invention, see Figure 3 , which shows a schematic diagram of the principle of the aggregation method of the global data sketch. It demonstrates the centralized controller collecting the updated data sketches from two measurement points and the aggregation process of the two data sketches. Specifically, after the centralized controller collects the data sketch of measurement point 1 and the data sketch of measurement point 2, it performs an OR operation on the bitmaps of the cells at the same position in the two data sketches to obtain the global data sketch. Figure 3 Taking the position of unit 1 in the middle layer 2 as an example, the bitmap of unit 1 in the middle layer 2 in the global data sketch is constructed by performing an OR operation on the bitmaps of the units corresponding to the position of unit 1 in the middle layer 2 in all data sketches, that is, the bitmap 010010 at the corresponding position in measurement point 1 and the bitmap 001010 at the corresponding position in measurement point 2 are bitwise ORed, and the bitmap of unit 1 in the middle layer 2 in the global data sketch is 011010. For the large flow record unit in the large flow record part, the flow key recorded in the measurement point is updated to the corresponding position in the global data sketch, that is, the first large flow record unit is used to store the flow key f1.

[0091] According to an embodiment of the present invention, since the probability of flow conflict in the large flow record part is small, usually there will not be multiple different flow keyword identifiers at the same position of multiple measurement points. If there is no conflict, the flow keyword in the measurement point is directly selected for update. If a conflict occurs, the measurement point is selected to estimate the size of the corresponding data flow for the flow keyword corresponding to each position, and the flow keyword corresponding to the maximum estimated value is selected and updated to the corresponding position of the global data sketch. Schematically, if at the same position, such as Figure 3 In the first large flow record unit, the flow keyword recorded at measurement point 1 is f2, and the flow keyword recorded at measurement point 2 is f1. If the data flow corresponding to the flow keyword f1 recorded at measurement point 2 is larger, then the keyword f1 is updated to the position of the first large flow record unit in the global data sketch.

[0092] According to an embodiment of the present invention, when the centralized controller collects updated data sketches from multiple measurement points and there are more than two data sketches to be aggregated, the same method as the above embodiment is adopted, that is, the bitmaps in the units at the same position (units at the same layer and with the same label) in all network measurement points are operated with an OR operation, and the bitmap is stored in the corresponding position of the global data sketch (the global data sketch is also irrelevant to routing). The OR operation means that when one of the values ​​at the same position in each bitmap is 1, the result is 1. Schematically, if the values ​​of the bitmaps at the unit 1 position of layer 2 of the three data sketches to be aggregated are 000010, 001010 and 011000 respectively, the three bitmaps are operated with an OR operation to obtain 011010, and 011010 is used as the value of the bitmap of unit 1 of layer 2 of the global data sketch. The second bit value of 000010 and 001010 is 1, indicating that the same data packet passes through two different measurement points. For the second bit of the two bitmaps, the existing method may directly regard the same data packet passing through two different measurement points as 2 different data packets, that is, directly add them to 2, resulting in the problem of repeated counting of the same data packet. However, the OR operation adopted in the present invention during aggregation still obtains a result of 1, avoiding repeated counting of the same data packet.

[0093] The technical solution of the above embodiment can at least achieve the following beneficial technical effects: the centralized controller aggregation method can effectively avoid the problem of repeated counting. The centralized controller collects data sketches from all measurement points and aggregates them into a global data sketch with global information, thereby providing an overall network measurement result.

[0094] (IV) Explain how to query and obtain network measurement information

[0095] 1) Obtain network measurement information of the entire network based on global data sketch query

[0096] According to one embodiment of the present invention, in a centralized controller, a method for estimating the number of data packets transmitted by each data stream includes steps S1 and S2:

[0097] Step S1: Obtain a given flow keyword of each data flow, and estimate the number of data packets transmitted by each data flow in the network based on the global data sketch for the flow keyword of each data flow.

[0098] According to one embodiment of the present invention, a method for estimating the number of data packets transmitted by each data flow in a network based on a global data sketch includes the following steps S11 and S12:

[0099] Step S11, based on the flow keyword of each data flow, query from the global data sketch to obtain the information corresponding to each data flow, the query method includes using a unit hash function to hash the flow keyword of the data flow to obtain a unit hash value, and according to the unit hash value and the total number of units in each layer, obtain a bitmap of a unit in each layer in the global data sketch to obtain multiple bitmaps.

[0100] According to one embodiment of the present invention, when a stream keyword is given, the unit hash function H in step ③ is used. s Hash the stream keyword and record the resulting unit hash value as h unit , and for each layer, the hash range of the unit hash function is the same as the hash range used in each layer in step ③. According to the unit hash value and the total number of units in each layer, query the bitmap of a unit in each layer in the global data sketch.

[0101] Schematically, for each layer, when the hash range is set to be larger than the total number of units in each layer: the total number of units in layer i of the global data sketch is denoted as N i , the unit to be obtained at this time is the hth unit in layer i unit %N i units, the total number of units in the corresponding layer i-1 is recorded as N i-1 , the unit to be obtained is the hth unit in layer i unit %N i-1 Units, that is, repeat the above operation for all layers, and then obtain the bitmaps of n units in n layers respectively. i and N i-1 The sizes may be the same or different.

[0102] Schematically, for each layer, when the hash range is set to be equal to the total number of units in each layer: the stream keyword is hashed using the unit hash function and the hash range of the corresponding layer, and the resulting unit hash value is recorded as h unit ′ If the unit hash value h unit ′If 5 is 5, the unit obtained is the bitmap of the 5th unit of the corresponding layer. Repeat the above operation for all layers to obtain the bitmaps of n units.

[0103] Step S12, traverse the multiple bitmaps in sequence according to the number of layers. When the ratio of the number of bits of zero in the bitmap to the total number of bits of the bitmap is greater than a preset ratio, a linear counting algorithm is used to estimate the number of data packets transmitted by the data stream based on the ratio of the threshold range corresponding to the bitmap and the layer where the bitmap is located to the sum of the threshold ranges of all layers.

[0104] According to an embodiment of the present invention, for the bitmaps of the n units obtained in step S11, the bitmaps of the units from each layer (i.e., the units from layer 1, the units from layer 2, and so on) are traversed from low to high according to the layer number, and the preset ratio is set to one-fourth, and when the bitmap of a certain unit meets the condition: the ratio of the number of zero bits in the bitmap of the unit to the total number of bits of the bitmap is greater than one-fourth of the total number of bits of the current bitmap, the traversal stops. At this time, the bitmap of the unit that meets this condition is used to estimate the number of data packets transmitted by the data stream. If the unit belongs to layer i, the corresponding sampling rate of the unit is (T i-1 -T i ) / H max . It should be understood that the setting of the preset ratio is only for illustration. One quarter of the present invention is an empirical value and can be customized according to the specific usage scenario. For example, the preset ratio can also be set to one fifth, etc. After finding a bitmap that meets the conditions, a linear counting algorithm is used for estimation. Combined with the corresponding sampling rate of the unit where the bitmap is located at this time, the number of data packets transmitted by the data stream (the number of data packets is also called the data stream size) is estimated. The estimation method is as follows:

[0105]

[0106] Where F is the estimated value, m is the length of the bitmap, and u is the number of bits that are 0 in the current bitmap.

[0107] For example, the preset ratio is one quarter. Figure 4 , which is a schematic diagram of the query process of the data sketch. According to the stream keyword, in the manner of the above embodiment, the bitmap of the corresponding unit in each layer of the data sketch is obtained, that is, the bitmap of the unit of layer 1 is 11101, and the corresponding sampling rate is (H max -T1) / H max , the bitmap of the unit of layer 2 is 10101010, and the corresponding sampling rate is (T1-T2) / H max , the bitmap of the unit of layer n is 001000…00, and the corresponding sampling rate is (T n-1 -T n ) / H max. Then, traverse from layer 1 to layer n in sequence. When traversing to layer 2, the bitmap of its unit is 10101010. It can be seen that the ratio of the number of zero bits to the total number of bits of the bitmap is greater than one quarter of the total number of bits of the current bitmap. Therefore, the estimation is performed according to the above estimation method. Among them, the linear counting algorithm in the above embodiment can adopt the method described in reference

[13] :

[0108]

[13] : Whang, Kyu-Young, Brad T. Vander-Zanden, and Howard M. Taylor. "Alinear-time probabilistic counting algorithm for database applications." ACM Transactions on Database Systems (TODS) 15.2 (1990): 208-229.

[0109] Step S2: based on the estimated number of data packets transmitted by each data flow and the network measurement task, obtain data packet counting information of the corresponding task.

[0110] According to one embodiment of the present invention, after estimating the number of data packets transmitted by each data stream, the network measurement task is to count the number of data packets transmitted by the first, second or third data stream. The measurement task can also be the sum of the number of data packets transmitted by any several data streams, so as to estimate the data packet counting information of the corresponding task.

[0111] 2) Based on the data sketch query of the network measurement point, obtain the network measurement information of the measurement point

[0112] According to one embodiment of the present invention, when a flow keyword is given, the network measurement information of the measurement point can be queried and obtained in the manner of steps S1 and S2 in the above embodiment, wherein, when querying, the query is performed based on the data sketch of the measurement point, and the obtained network measurement information is local information (i.e., the information of the measurement point), while the above embodiment is based on the global data sketch for query, and the obtained network measurement information is global information.

[0113] (V) Explain how the centralized controller in the network measurement system issues measurement tasks

[0114] According to one embodiment of the present invention, see Figure 5 , which is a schematic diagram of the system's network measurement task configuration method and workflow. The centralized controller implements the measurement task distribution by configuring flow keywords and packet keywords.

[0115] Before introducing the centralized controller to implement the measurement task delivery, the following description is given of the structure related to keyword processing in the network measurement point:

[0116] According to one embodiment of the present invention, each network measurement point includes: a flow keyword mask register group, which is used to store a flow keyword mask, which includes a plurality of flow mask values ​​respectively associated with a plurality of fields of a data packet, each flow mask value is used to indicate whether a flow keyword needs to be associated with a field of the flow mask value; a packet keyword mask register group, which is used to store a packet keyword mask, which includes a plurality of packet mask values ​​respectively associated with a plurality of fields of a data packet, each packet mask value is used to indicate whether a field in a packet keyword needs to be associated with the packet mask value. Figure 5 As shown, the stream keyword mask includes 6 stream mask values, and is associated with the source IP field (ScrIP), the destination IP field (DstIP), the source port number field (Sport), the destination port number field (Dport), the protocol type field (Protocol) and the ID field respectively. Similarly, the packet keyword mask also includes 6 stream mask values, and is associated with the source IP field (ScrIP), the destination IP field (DstIP), the source port number field (Sport), the destination port number field (Dport), the protocol type field (Protocol) and the ID field respectively. Wherein, the checksum of the IP header and the TCP / UDP header is used as the ID field in the present invention. When the stream mask value and the packet mask value are represented by 1 or 0, and 1 represents a field associated with the stream mask value or the packet mask value, and 0 represents that it is not required. Schematically, if the stream keyword mask is 110000, it means that the information of the source IP field (ScrIP) and the destination IP field (DstIP) is needed in the stream keyword, and the information of the remaining 4 fields is not needed.

[0117] The following is an explanation of the method and workflow of the centralized controller issuing measurement tasks:

[0118] 1) The centralized controller configures the keywords required for the measurement task.

[0119] According to one embodiment of the present invention, the configuration of the keyword is realized by configuring the flow keyword mask and the packet keyword mask. That is, the centralized controller is also configured to: determine the flow keyword mask and the packet keyword mask according to the fields required by the flow keyword and the fields required by the packet keyword in the network measurement task, and distribute the flow keyword mask and the packet keyword mask to multiple network measurement points to implement the sending of the measurement task. Among them, the flow keyword masks stored in the flow keyword mask register groups of multiple network measurement points are the same, and the packet keyword masks stored in the packet keyword mask register groups of multiple network measurement points are the same. Schematically, if the measurement task is: measuring how many data packets are contained in each data stream, the flow keyword is set to the quintuple form in the above embodiment, and the flow keyword needs to contain the five fields in the quintuple, then the flow keyword mask needs to be configured as 111110, specifically, the flow mask values ​​corresponding to the five fields in the quintuple need to be set to 1.

[0120] According to one embodiment of the present invention, the centralized controller modifies the measurement task by modifying the flow keyword mask and the packet keyword mask in the network measurement point. The remaining measurement tasks can be configured in the same way, including but not limited to counting the number of data packets received by a certain IP address (i.e., setting the flow keyword to the destination IP address and the packet keyword to the ID mentioned above), and counting how many different source addresses a certain destination IP address communicates with (setting the flow keyword to the destination IP address and the packet keyword to the source IP address).

[0121] 2) After the flow keyword mask and the packet keyword mask are distributed to multiple network measurement points, when a data packet is transmitted to the multiple network measurement points, the data packet is processed according to the flow keyword mask and the packet keyword mask stored in the network measurement point to obtain the keyword of the data packet.

[0122] According to one embodiment of the present invention, at each network measurement point, a method for obtaining a keyword of the data packet includes: processing the data packet according to a flow keyword mask to obtain a flow keyword, which includes one or more fields, wherein the processing method includes bitwise ANDing each field of the multiple fields of the data packet with a flow mask value associated with the field to obtain the field required for the flow keyword; processing the data packet according to a packet keyword mask to obtain a packet keyword, which includes one or more fields, wherein the processing method includes bitwise ANDing each field of the multiple fields of the data packet with a packet mask value associated with the field to obtain the field required for the packet keyword. Indicatively, if the flow mask value is 010000, and the 6th bit value 0 is associated with the source IP field (ScrIP), the 5th bit value 1 is associated with the destination IP field (DstIP), the 4th bit value 0 is associated with the source port number field (Sport), the 3rd bit value 0 is associated with the destination port number field (Dport), the 2nd bit value 0 is associated with the protocol type field (Protocol), and the 1st bit value 0 is associated with the ID field, and a bitwise AND operation is performed between the mutually associated flow mask values ​​and fields, for example, the 6th bit value 0 is ANDed with each bit of the source IP field (ScrIP) associated with it (bitwise AND operation), so that the source IP field (ScrIP) is all set to 0, and 1 and each bit of the destination IP field (DstIP) associated with 1 are ANDed (bitwise AND operation), and the destination IP field (DstIP) is obtained, that is, the field information required for the flow keyword. The remaining fields are deduced by analogy to obtain the flow keyword of the data packet, and the packet keyword of the data packet is obtained in the same way.

[0123] 3) The flow keyword of the data packet and the packet keyword are input into the data sketch that is not related to routing to perform an update operation. The packet keyword is hierarchically hashed by a hierarchical hash function to determine the layer associated with the data packet. In the associated layer, the unit hash function H is used to s The stream key is hashed to determine the associated unit and the bitmap hash function H is used to b The packet key is bitmap hashed to determine a position in the bitmap of the associated unit that needs to be updated, thereby obtaining an updated data sketch that is independent of routing.

[0124] 4) After the update is completed, the centralized controller collects the updated data sketch from the network measurement points.

[0125] According to an embodiment of the present invention, a whole network measurement method based on the system described in the above embodiment is provided, the method comprising: obtaining a keyword of a data packet, including a packet keyword and a flow keyword of a data flow to which the data packet belongs, when receiving and forwarding a data packet based on a network measurement task issued by a centralized controller by each of a plurality of network measurement points; updating a data sketch of the network measurement point according to the keyword of the data packet by each of the plurality of network measurement points, the updating method comprising: determining an associated layer according to a hash value of the packet keyword, and determining an associated unit in the associated layer according to the hash value of the flow keyword; mapping the packet keyword to a position in a bitmap of the associated unit, and updating the value of the position to a second value; collecting the updated data sketches of the plurality of network measurement points respectively and aggregating all the collected data sketches by the centralized controller to obtain a global data sketch, estimating the number of data packets transmitted by each data flow in the whole network according to the global data sketch, so as to complete the network measurement task.

[0126] In order to verify the beneficial effects of the present invention, the following comparative experiments were conducted:

[0127] The network measurement task is set to measure the number of data packets received by each destination address. During the measurement process, the total amount of data packets input to the network measurement point is 8 million to 10 million, and the total amount of data packets is represented by 8M, 9M and 10M respectively. Under the total amount of three data packets, the system of the present invention and the two measurement systems of the prior art (i.e., the above-mentioned reference [7], referred to as Jaqen) are compared in an experiment. During the experiment, the weighted average percentage error is calculated by the estimated value of the number of data packets obtained by the two systems when performing the measurement task. The error comparison results are shown in Table 1 below:

[0128] Table 1: Weighted average percentage error comparison results

[0129]

[0130] According to the data in Table 1 above, compared with the existing method Jaqen, the measurement result of the present invention has a smaller error. In addition, the present invention can not only estimate the total flow, but also estimate the number of data packets transmitted by each data flow separately, and can realize various types of network measurement tasks, with a wide range of applications.

[0131] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0132] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0133] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0134] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A routing-independent network measurement system, characterized in that: The system includes multiple distributed network measurement points and a centralized controller, among which: Each network measurement point maintains a data sketch that is independent of routing, wherein the data sketch is a multi-layer structure, each layer includes a plurality of units arranged in sequence according to position, each unit is provided with a bitmap, and the initial value of all bits in the bitmap is a first value; Each network measurement point is configured as: Receive and forward a data packet, obtain a keyword of the data packet according to the network measurement task, including a packet keyword and a flow keyword of a data flow to which the data packet belongs, and update a data sketch of the network measurement point according to the keyword of the data packet, wherein the updating method includes: Determine an associated layer according to a hash value of the packet keyword, and determine the associated unit in the associated layer according to a hash value of the flow keyword; Mapping the packet key to a position in a bitmap of the associated unit and updating the value of the position to a second value; The centralized controller is configured as: Collect updated data sketches from multiple network measurement points and aggregate all collected data sketches to obtain a global data sketch. Estimate the number of data packets transmitted by each data flow in the network based on the global data sketch to complete the network measurement task.

2. The system according to claim 1, characterized in that In each network measurement point, a threshold range is preset for each layer in its data sketch, and the threshold ranges of all layers of the data sketch do not intersect; The method of determining the associated layer includes: Performing hierarchical hashing on the packet keyword using a hierarchical hash function to obtain a layer hash value, wherein a hash range of the hierarchical hash function is the sum of threshold ranges of all layers of the data sketch; According to the threshold range of the layer where the layer hash value is located, the layer corresponding to the threshold range is used as the associated layer.

3. The system according to claim 1 or 2, characterized in that: In each network measurement point, the associated unit is determined in the following manner: In the associated layer, the stream keyword is hashed using a unit hash function to obtain a unit hash value, and the associated unit in the layer is obtained according to the unit hash value mapping, wherein the hash range of the unit hash function is greater than or equal to the total number of units in the layer.

4. The system according to claim 3, characterized in that The number of bits of the bitmaps of the multiple units of each layer of the data sketch is the same, and in each network measurement point, the mapping to a position in the bitmap of the associated unit includes: A bitmap hash function is used to perform bitmap hashing on the packet keyword to obtain a bit hash value, and the bit hash value is mapped to a position in the bitmap of the associated unit according to the bit hash value, wherein the hash range of the bitmap hash function is greater than or equal to the bit size of the bitmap of the corresponding associated unit.

5. The system according to claim 1, characterized in that In the centralized controller, the aggregation method includes: The bitmaps of the cells at the same position in all the collected data sketches are aggregated to obtain a global data sketch with the same structure as the data sketch of the network measurement point. The bitmap of each cell in each layer in the global data sketch is obtained by ORing the bitmaps of the cells at the corresponding positions in all the data sketches.

6. The system according to claim 1, characterized in that Each network measurement point includes: A flow keyword mask register group, used for storing a flow keyword mask, which includes a plurality of flow mask values ​​respectively associated with a plurality of fields of a data packet, each flow mask value being used for indicating whether a flow keyword needs a field associated with the flow mask value; The packet keyword mask register group is used to store the packet keyword mask, which includes multiple packet mask values ​​respectively associated with multiple fields of the data packet, and each packet mask value is used to indicate whether a field in the packet keyword needs to be associated with the packet mask value.

7. The system according to claim 6, characterized in that At each network measurement point, the methods for obtaining the keyword of the data packet include: Processing the data packet according to the flow keyword mask to obtain a flow keyword including one or more fields, wherein the processing comprises performing a bitwise AND operation on each of the multiple fields of the data packet and a flow mask value associated with the field to obtain a field required for the flow keyword; The data packet is processed according to the packet keyword mask to obtain a packet keyword, which includes one or more fields, wherein the processing method includes bitwise ANDing each field of multiple fields of the data packet with a packet mask value associated with the field to obtain the field required for the packet keyword.

8. The system according to claim 6, characterized in that The centralized controller is also configured to: According to the fields required by the flow keyword and the fields required by the packet keyword in the network measurement task, the flow keyword mask and the packet keyword mask are determined, and the flow keyword mask and the packet keyword mask are distributed to multiple network measurement points.

9. The system according to any one of claims 2 to 8, characterized in that: In the centralized controller, the methods for estimating the number of packets transmitted by each data flow include: Get the flow key of each given data flow, and for each data flow flow key, estimate the number of data packets transmitted by each data flow in the network based on the global data sketch in the following way: Based on the flow keyword of each data flow, query the global data sketch to obtain information corresponding to each data flow, wherein the query method includes hashing the flow keyword of the data flow using a unit hash function to obtain a unit hash value, and obtaining a bitmap of a unit of each layer in the global data sketch according to the unit hash value and the total number of units of each layer to obtain multiple bitmaps; Traversing the multiple bitmaps in order according to the number of layers, when the ratio of the number of bits that are zero in the bitmap to the total number of bits in the bitmap is greater than a preset ratio, using a linear counting algorithm to estimate the number of data packets transmitted by the data stream based on the ratio of the threshold range corresponding to the bitmap and the layer where the bitmap is located to the sum of the threshold ranges of all layers; Based on the estimated number of data packets transmitted by each data flow and the network measurement task, the data packet counting information of the corresponding task is obtained.

10. A whole network measurement method based on the system according to any one of claims 1 to 9, characterized in that: Methods include: By each of the plurality of network measurement points, based on the network measurement task issued by the centralized controller, when receiving and forwarding a data packet, a keyword of the data packet is obtained, including a packet keyword and a flow keyword of a data flow to which the data packet belongs; Each network measurement point in the plurality of network measurement points updates the data sketch of the network measurement point according to the keyword of the data packet, and the updating method includes: Determine an associated layer according to a hash value of the packet keyword, and determine the associated unit in the associated layer according to a hash value of the flow keyword; Mapping the packet key to a position in a bitmap of the associated unit and updating the value of the position to a second value; The updated data sketches of multiple network measurement points are collected through a centralized controller and all collected data sketches are aggregated to obtain a global data sketch. The number of data packets transmitted by each data stream in the entire network is estimated based on the global data sketch to complete the network measurement task.

Images