A routing-independent network measurement system and method

Abstract

The application provides a routing-independent network measurement system, comprising a plurality of network measurement points and a centralized controller, each network measurement point maintaining a routing-independent data sketch which is a multi-layer structure, each layer comprising a plurality of units provided with a bitmap, and the initial value of all bits in the bitmap being a first numerical value; the network measurement point is configured to: acquire a packet key of a data packet received by the network measurement point and a flow key of a data flow to which the data packet belongs according to a measurement task; update the data sketch of the network measurement point according to the keys, comprising: determining an associated layer according to the packet key, determining an associated unit in the associated layer according to the flow key; and mapping the packet key to a position of the bitmap of the associated unit, and updating the numerical value of the position to a second numerical value; and the centralized controller is configured to: estimate the number of data packets transmitted by each data flow in the network according to a global data sketch obtained by collecting and aggregating the updated data sketches of the plurality of network measurement points, so as to complete the measurement task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network operation and management, in particular to the field of network measurement technology, and more particularly to a routing-independent network measurement system and method. BACKGROUND

[0002] Real-time collection and analysis of network basic data is indispensable for efficient operation and safe management of a network. Therefore, network administrators usually check, monitor and track the network operation regularly to find abnormal conditions such as network performance loss and paralysis caused by sudden traffic, network attacks, facility failures, etc. This indispensable work is usually referred to as network measurement. A single measurement point usually cannot completely cover the traffic in the entire network. Therefore, in order to obtain a global view of the network and perform network measurement tasks, such as identifying heavy-hitters distributed in multiple network measurement points in the entire network, or detecting DDoS attacks using multiple network measurement points, the network measurement system usually needs to combine data from multiple measurement points to form measurements of each network flow in the entire network, i.e. to perform network-wide measurement using multiple network measurement points.

[0003] Network-wide measurement usually uses multiple measurement points as the data plane and a central controller as the control plane to aggregate the measurement results of the data plane to form a network-wide view, and to perform various network measurement tasks based on the measurement results in the network-wide view, such as measuring the traffic (number of data packets) of each flow in the network. However, the challenge in implementing network-wide measurement is the problem of repeated counting, i.e. packets traversing multiple measurement points are recorded multiple times, resulting in inconsistency between the network measurement results and the actual state of the network. Due to the large size of the network to be measured, it is difficult to know the overall topology, the topology is updated due to failure of the measurement node, etc. Therefore, a single fixed path assumption for traffic routing cannot model the current network.

[0004] Existing network measurement techniques are usually based on sketch algorithms. That is, sketch algorithms (e.g., Count-Min Sketch (referred to as [1]), Bloom Filter (referred to as [2])) are deployed on network measurement points, usually programmable switches in the data plane of the network by using programmable technology, and the switch with the corresponding data structure deployed is taken as a network measurement point. A central controller periodically communicates with each network measurement point in the network to obtain the local data of each network measurement point, and finally combines the local data to obtain the final measurement result. However, the network measurement system of such design has the repeated counting problem described above, such as the measurement systems in [3] (referred to as Marple), [4] (referred to as MAFIA), [5] (referred to as Flymon), [6] (referred to as Poseidon), and [7] (referred to as Jaqen). These measurement systems collect traffic information in the network based on sketch algorithms in the data plane. However, in order to avoid the problem of repeated counting, strict restrictions need to be imposed on the distribution of measurement points and traffic patterns in such works. For example, Jaqen uses a data plane based on [8] (referred to as Univmon), which needs to limit the deployment location of the measurement point, and it is difficult to deal with the problems of measurement node failure or network topology update in the actual network.

[0005] Some works have focused on routing-independent network detection methods. The basic idea of implementing routing-independent network measurement is to consider the uniqueness of each data packet when updating the measurement result in the measurement point, and use a globally consistent (i.e., the same for each measurement point) hash algorithm for state update, so that the same data packet produces the same update at multiple measurement points. For example, [9] (referred to as INVEST) uses the merging property of

[10] (referred to as Hyperloglog) to eliminate data packet repeated counting, but this method can only estimate the total traffic size in the network, and cannot support the recording and querying of each traffic state in the network.

[11] (referred to as AROMA) uses the merging property of

[12] (K-Minimum Value, referred to as KMV algorithm) to implement total traffic size estimation and uniform sampling of data packets in the whole network. This method is mainly applied to the field of large flow detection in the whole network, and the type of network measurement tasks it can implement is limited, and it cannot query the information of small flows in the network.

[0006] Therefore, in the existing routing-independent network-wide measurement method, although the repeated counting of data packets is avoided to a certain extent, only the total traffic count is counted, and the count information of each data flow cannot be counted separately, resulting in the problems that the existing measurement method cannot support the recording and querying of the state of each traffic in the network, and the information of small traffic in the network cannot be queried, thereby existing the problems of limited types of network measurement tasks and small application range.

[0007] It should be noted that the background art is only used to introduce the related information of the present application, so as to help understand the technical solutions of the present application, but does not mean that the related information must be the prior art. In the absence of evidence that the related information has been disclosed before the filing date of the present application, the related information should not be regarded as prior art.

[0008] The references are as follows:

[0009] [1] G. Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms, 55(1):58-75, 2005.

[0010] [2] Geravand, Shahabeddin, and Mahmood Ahmadi. "Bloom filter applications in network security: A state-of-the-art survey." Computer Networks 57.18 (2013): 4047-4064.

[0011] [3] Narayana S, Sivaraman A, Nathan V, et al. Language-Directed Hardware Design for Network Performance Monitoring [C] / / Conference of the Acm Special Interest Group. ACM, 2017.

[0012] [4] Laffranchini, Paolo, et al. Measurements as first-class artifacts[C] / / IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019.

[0013] [5] Zheng, Hao, et al. Flymon: enabling on-the-fly task reconfiguration for network measurement[C] / / Proceedings of the ACM SIGCOMM 2022 Conference. 2022.

[0014] [6] Zhang M, Li G, Wang S, et al. Poseidon: Mitigating Volumetric DDoS Attacks with Programmable Switches. [C] / / Network and Distributed System Security Symposium. Internet Society, 2020.

[0015] [7] Liu Z, Namkung H, Nikolaidis G, et al. Jaqen: A High-Performance Switch-Native Approach for Detecting and Mitigating Volumetric DDoS Attacks with Programmable Switches. [C] / / USENIX Security Symposium. 2021.

[0016] [8] Liu, Zaoxing, et al. One sketch to rule them all: Rethinking network flow monitoring with univmon. [C] / / Proceedings of the 2016 ACM SIGCOMM Conference. 2016.

[0017] [9] Ding D, Savi M, Pederzolli F, et al. INVEST: Flow-based Traffic Volume Estimation in Data-plane Programmable Networks[C] / / IFIP Networking. 2021.

[0018]

[10] Flajolet P, Fusy, Gandouet O, et al. HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm[J]. 2007. DOI: doi:http: / / dx.doi.org / .

[0019]

[11] Basat, Ran Ben, et al. Routing oblivious measurement analytics[C] / / 2020 IFIP Networking Conference (Networking). IEEE, 2020.

[0020]

[12] Beyer K, Haas P J, Reinwald B, et al. On synopses for distinct-value estimation under multiset operations[C] / / Proceedings of the 2007 ACM SIGMOD international conference on Management of data. 2007. SUMMARY

[0021] Therefore, the purpose of the present application is to overcome the defects of the prior art, and to provide a routing-independent network measurement system and method.

[0022] The purpose of the present application is achieved by the following technical solutions:

[0023] According to a first aspect of the present application, a routing-independent network measurement system is provided, the system comprising a plurality of distributed network measurement points and a centralized controller, wherein each network measurement point maintains a routing-independent data sketch, wherein the data sketch is a multi-layer structure, each layer comprising a plurality of units arranged in order of position, and each unit is provided with a bitmap, and the initial value of all bits in the bitmap is a first value;

[0024] Each network measurement point is configured to: receive and forward a data packet, obtain a key of the data packet according to a network measurement task, the key including a packet key and a flow key of a data flow to which the data packet belongs, update a data sketch of the network measurement point according to the key of the data packet, and the updating manner includes: determining an associated layer according to a hash value of the packet key, determining an associated unit in the associated layer according to a hash value of the flow key, mapping the packet key to a position in a bitmap of the associated unit, and updating a value of the position to a second value.

[0025] The centralized controller is configured to: collect the updated data sketch of each network measurement point, aggregate all the collected data sketches to obtain a global data sketch, and estimate a number of data packets transmitted by each data flow in the network according to the global data sketch, so as to complete the network measurement task.

[0026] In some embodiments of the present application, a threshold range is preset for each layer of the data sketch in each network measurement point, and the threshold ranges of all the layers of the data sketch are disjointed.

[0027] The determination manner of the associated layer includes:

[0028] The packet key is subjected to layered hashing by using a layered hashing function to obtain a layer hash value, wherein a hashing range of the layered hashing function is a sum of the threshold ranges of all the layers of the data sketch.

[0029] The layer corresponding to the threshold range of the layer hash value is taken as the associated layer.

[0030] In some embodiments of the present application, in each network measurement point, the determination manner of the associated unit includes:

[0031] In the associated layer, the flow key is subjected to hashing by using a unit hashing function to obtain a unit hash value, and the associated unit in the layer is obtained according to the unit hash value, wherein a hashing range of the unit hashing function is greater than or equal to a total number of units of the layer.

[0032] In some embodiments of the present application, the bit sizes of the bitmaps of the units of each layer of the data sketch are the same, and in each network measurement point, the manner of mapping the packet key to the position in the bitmap of the associated unit includes:

[0033] The packet key is subjected to bitmap hashing by using a bitmap hashing function to obtain a bit hash value, and the bit hash value is mapped to a position in the bitmap of the associated unit, wherein a hashing range of the bitmap hashing function is greater than or equal to a bit size of the bitmap of the corresponding associated unit.

[0034] In some embodiments of the present application, in the centralized controller, the aggregating manner comprises:

[0035] The bitmaps of the units in the same position in all the collected data sketches are aggregated to obtain a global data sketch with the same structure as the data sketch of the network measurement point, and the bitmap of each unit in each layer in the global data sketch is obtained by an OR operation of the bitmaps of the units in the corresponding position in all the data sketches.

[0036] In some embodiments of the present application, each network measurement point comprises:

[0037] a flow key mask register group for storing a flow key mask comprising a plurality of flow mask values respectively associated with a plurality of fields of the data packet, each flow mask value being used to indicate whether a flow key needs one field associated with the flow mask value;

[0038] a packet key mask register group for storing a packet key mask comprising a plurality of packet mask values respectively associated with a plurality of fields of the data packet, each packet mask value being used to indicate whether a packet key needs one field associated with the packet mask value.

[0039] In some embodiments of the present application, in each network measurement point, the manner of obtaining the key of the data packet comprises:

[0040] processing the data packet according to the flow key mask to obtain a flow key comprising one or more fields, wherein the processing manner comprises performing a bitwise AND operation between each of the plurality of fields of the data packet and the flow mask value associated with the field to obtain the required field of the flow key;

[0041] processing the data packet according to the packet key mask to obtain a packet key comprising one or more fields, wherein the processing manner comprises performing a bitwise AND operation between each of the plurality of fields of the data packet and the packet mask value associated with the field to obtain the required field of the packet key.

[0042] In some embodiments of the present application, the centralized controller is further configured to:

[0043] determining the flow key mask and the packet key mask according to the required fields of the flow key and the required fields of the packet key in the network measurement task, and distributing the flow key mask and the packet key mask to the plurality of network measurement points.

[0044] In some embodiments of the present application, in the centralized controller, the manner of estimating the number of data packets transmitted by each data flow comprises:

[0045] Obtaining the flow key of each data stream given, for the flow key of each data stream, estimating the number of data packets transmitted by each data stream in the network based on the global data sketch in the following manner: querying the information corresponding to each data stream from the global data sketch based on the flow key of each data stream, the querying manner including hashing the flow key of the data stream using a unit hash function to obtain a unit hash value, and obtaining a bitmap of each unit in each layer in the global data sketch according to the unit hash value and the total number of units in each layer to obtain a plurality of bitmaps; sequentially traversing the plurality of bitmaps by layer, when the ratio of the number of zeros in the bitmap to the total number of bits in the bitmap is greater than a preset ratio, estimating the number of data packets transmitted by the data stream based on the bitmap and the ratio of the threshold range corresponding to the layer of the bitmap to the sum of the threshold ranges of all layers using a linear counting algorithm; and obtaining data packet counting information corresponding to the task based on the estimated number of data packets transmitted by each data stream and the network measurement task.

[0046] According to a second aspect of the present application, a network-wide measurement method based on the system of the first aspect of the present application is provided, the method comprising:

[0047] Through each network measurement point in the plurality of network measurement points, based on the network measurement task issued by the centralized controller, obtaining the key of the data packet including the packet key and the flow key of the data stream to which the data packet belongs when receiving and forwarding the data packet; through each network measurement point in the plurality of network measurement points, updating the data sketch of the network measurement point according to the key of the data packet, the updating manner including: determining the associated layer according to the hash value of the packet key, and determining the associated unit in the associated layer according to the hash value of the flow key; mapping the packet key to a position in the bitmap of the associated unit, and updating the value of the position to a second value; collecting the updated data sketch of each network measurement point by the centralized controller and aggregating all collected data sketches to obtain a global data sketch, estimating the number of data packets transmitted by each data stream in the network-wide network based on the global data sketch, and completing the network measurement task.

[0048] Compared with the prior art, the present application has the following advantages:

[0049] In one aspect, in the process of sketch updating of the structure of the sketch of the measurement point combined data, under any measurement point topology and routing path, by obtaining the bitmap of a specific position, more accurate measurement or query of the data packet information of different data stream transmission can be realized, the subsequent recording and query of the state of each flow in the network are supported, and the relevant information is recorded on each bitmap of the sketch regardless of the size of the flow, so as to support the query of the information of small flow in the network. In another aspect, the data sketch structures of the plurality of network measurement points are the same and the updating methods of the data sketch are the same, the same data packet of the same data stream transmitted to each measurement point is updated to the second data by the value of the same position of the data sketch, that is, the value of the bitmap at the same position in the sketch is set, the repeated counting of the data packet is avoided, the global data sketch aggregated by the centralized controller is also free of repeated counting, the estimated number of data packets of each data stream transmission is more accurate, and after the estimation of the number of data packets corresponding to each data stream is realized, a plurality of network measurement task types can be realized, and the application range is wide. BRIEF DESCRIPTION OF DRAWINGS

[0050] The embodiments of the present application are further described below with reference to the accompanying drawings, in which:

[0051] Figure 1 It is a schematic diagram of the overall architecture of the measurement system according to the embodiment of the present application and the principle of the measurement process thereof;

[0052] Figure 2 It is a schematic diagram of the structure design of the data sketch according to the embodiment of the present application and the principle of the updating thereof;

[0053] Figure 3 It is a schematic diagram of the principle of the aggregation method of the global data sketch according to the embodiment of the present application;

[0054] Figure 4 It is a schematic diagram of the query process of the data sketch according to the embodiment of the present application;

[0055] Figure 5 It is a schematic diagram of the network measurement task configuration method and the working process of the system according to the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0057] As mentioned in the part of background, in the existing routing-independent network measurement method, although the repeated counting of data packets is avoided to some extent, the counting is only for the total traffic, and the counting information of each data flow cannot be counted separately, which leads to the problems that the existing measurement method cannot support the recording and querying of each traffic state in the network and the querying of the information of small traffic in the network, thereby the problems of limited types of network measurement tasks and small application range in the implementation exist.

[0058] Based on the above problems, the present application provides a routing-independent network measurement system, which comprises a plurality of network measurement points and a centralized controller. Firstly, each network measurement point maintains a data sketch, which is a multi-layer structure, and each layer comprises a plurality of units provided with a bitmap, and the initial value of all bits in the bitmap is a first value. The first value is used to represent that the measurement point does not receive any data packet of data flow at the initial time, so as to facilitate the correct counting of subsequent data packets. Secondly, the network measurement point: obtains the packet key of the received data packet and the flow key of the data flow to which the data packet belongs according to the measurement task; updates the data sketch of the network measurement point according to the keys, including: determining the associated layer according to the packet key, and determining the associated unit in the associated layer according to the flow key; and maps the packet key to a position of the bitmap of the associated unit, and the value of the position is updated to a second value. In the process of sketch updating of the measurement point combined with the structure of the data sketch, under any measurement point topology and routing path, the data packet information of different data flow transmission can be measured or queried more accurately by obtaining the bitmap of a specific position, the recording and querying of each traffic state in the network can be supported, and the relevant information of any traffic size is recorded on each bitmap of the sketch to support the querying of the information of small traffic in the network. Finally, the centralized controller: obtains the global data sketch from the updated data sketches of a plurality of network measurement points aggregated by the centralized controller, and estimates the number of data packets of each data flow transmission in the network. The data sketch structure and updating mode of the plurality of network measurement points are the same, i.e., the global consistency is achieved, the same value of the data sketch of each measurement point is updated to the second data for the same data packet of the same data flow transmitted, the repeated counting of the data packet is avoided, the global data sketch is also free of repeated counting, the estimated number of data packets of each data flow transmission is more accurate, and after the estimation of the number of data packets corresponding to each data flow is achieved, a plurality of types of network measurement tasks can be implemented, and the application range is wide.

[0059] In summary, according to one embodiment of the present application, in the network measurement system of the present application, each network measurement point maintains a routing-independent data sketch, wherein the data sketch is a multi-layer structure, each layer includes a plurality of units arranged in sequence according to positions, and each unit is provided with a bitmap, and the initial value of all bits in the bitmap is a first value; each network measurement point is configured to receive and forward a data packet, obtain a key of the data packet according to a network measurement task, the key including a packet key and a flow key of a data flow to which the data packet belongs, and update the data sketch of the network measurement point according to the key of the data packet, the updating manner including: determining an associated layer according to a hash value of the packet key, determining the associated unit in the associated layer according to a hash value of the flow key, mapping the packet key to a position in the bitmap of the associated unit, and updating the value of the position to a second value; and a centralized controller is configured to collect the updated data sketch of each network measurement point and aggregate all the collected data sketches to obtain a global data sketch, estimate the number of data packets transmitted by each data flow in the network according to the global data sketch, and complete the network measurement task. In order to better understand the present application, the network measurement system will be described in detail below in combination with specific embodiments.

[0060] (I) The overall architecture of the network measurement system of the present application and its measurement process are described

[0061] According to one embodiment of the present application, referring to Figure 1, which is the overall architecture of the measurement system of the present application and the schematic diagram of the principle of the measurement process. The overall architecture of the system includes a centralized controller and a plurality of distributed network measurement nodes. Six network measurement points are shown in the figure, each of which is regarded as a data plane, and the centralized controller is regarded as a control plane. In the measurement process, first, the centralized controller issues a measurement task to each network measurement point. Each network measurement point updates its data sketch according to the data packet it receives in the case of unknown routing (i.e. the routing of the data packet and the topology of the measurement point are unknown), obtains the updated data sketch, and takes the updated data sketch as the local measurement result of the measurement point. And in the measurement process, each data packet is recorded after passing through the network measurement point, and one data packet is allowed to pass through multiple network measurement points, for example, a data packet passes through three network measurement points in the figure, and the three network measurement points all record (i.e. update the data sketch in) the network measurement point. The control plane (i.e. the centralized controller, hereinafter referred to as the control plane) collects the local measurement results (i.e. the updated data sketch) of all network measurement points, aggregates all collected local measurement results to obtain a global data sketch, and queries the network measurement information according to the global data sketch, including the bitmap information corresponding to each flow keyword, thereby realizing various network measurement tasks, such as measurement task 1, measurement task 2 and measurement task 3.

[0062] According to an embodiment of the present application, the network measurement point in the system is a device directly undertaking the traffic forwarding task in the network, and such a device needs to have a certain data processing capability, for example, a programmable switch device that can directly update the data sketch, or a device that can mirror the traffic and process it on an external server. There is no special requirement for the location deployment of the network measurement node, such as deployment at multiple core nodes of the network. The centralized controller in the system needs to collect data from each network measurement point and process it, therefore, the centralized controller can adopt a conventional general server device or a cloud server device, and needs to ensure that the centralized controller can interconnect with each network measurement point and receive data from the network measurement point.

[0063] According to an embodiment of the present application, the approximate process of the system for network measurement task includes:

[0064] 1) The centralized controller issues a measurement task, and after issuing a measurement task, initializes the data sketch of each network measurement point, i.e. sets the value of all bits in all bitmaps to the first value.

[0065] 2) Each network measurement point starts to measure and ends the measurement after a preset measurement time, obtaining the updated data sketch of each network measurement point in the preset measurement time. During the measurement, when a data packet passes through the network measurement point and is forwarded through the switch forwarding chip, the flow key and the packet key related to the network measurement task are obtained from the data packet, i.e. <flow key, packet key>, the data packets with the same flow key are data packets in the same data flow, and the data packets with different packet keys are different data packets. The network measurement point updates its data sketch according to the obtained key, thereby recording the information of the data flow to which the data packet belongs.

[0066] 3) The centralized controller collects the updated data sketch of each network measurement point in the preset measurement time, completes aggregation, and obtains a global data sketch. The data sketch of each network measurement point can also be collected in a periodic manner, for example, the updated data sketch of each network measurement point in 5 minutes is collected every 5 minutes to obtain the measurement result of the task in each period. The centralized controller further obtains the network measurement information from the global data sketch, uses query to obtain the bitmap information corresponding to each flow key, thereby realizing various network measurement tasks. Illustratively, the basic network measurement task in the network is to measure the statistical information of the data flow in the network, such as the number of data packets. Taking a common measurement task as an example, the number of data packets of each data flow is measured, and the number of data packets of each source IP is measured (i.e. finding the IP with more data packets in the network). In the present application, the network measurement task includes measuring the data amount of each data flow, i.e. the number of data packets of each data flow can be counted by querying how many different packet keys are under each flow key.

[0067] (II) Explanation of the data sketch maintained by each network measurement point

[0068] According to an embodiment of the present application, referring to Figure 2 , which is a structure design of the data sketch maintained by the network measurement point and a schematic diagram of the updating principle. The structure and updating principle of the data sketch are described as follows:

[0069] 1) Structure of the data sketch

[0070] The data sketch is a multi-layer structure. As shown in Figure 2 , the data sketch is an n-layer structure, which is layer 1, layer 2 to layer n, a threshold range is preset for each layer of the data sketch, and the threshold ranges of all layers of the data sketch are disjoint. For each layer of the data sketch, a threshold T is maintained, i.e. the i-th layer has a threshold T i , and for any layer i, T i > T i+1 , T0= H max,like Figure 2 , the threshold range of layer 1 is [T1, T0), the threshold range of layer 2 is [T2, T1), and the threshold range of layer n is [T n ,T n-1 ), the sum of the threshold ranges of all layers of the data sketch is 0 to H max .

[0071] In a multi-layer structure, each layer includes multiple data units. Figure 2 As shown, the number of data units in each layer from layer 1 to layer n may be the same or different.

[0072] Each of the multiple data units in each layer includes a bitmap. The number of bits in the bitmaps of the multiple units in each layer is the same, such as Figure 2 The bit size of the bitmap of all units in layer 2 is 6.

[0073] The data sketch also includes a large flow record part, which is an array of multiple registers. Each register is used as a large flow record unit to record the flow key of the current data packet. The number of registers (i.e., large flow record units) in the large flow record part is denoted as H. The flow key of the data packet that meets the specified conditions will be recorded in the large flow record unit, and the layer corresponding to the large flow record is denoted as layer 0, T h Indicates the threshold corresponding to the large flow record part.

[0074] 2) Data Sketch Update Principle

[0075] like Figure 2 As shown in the figure, when a data packet is transmitted to the network measurement point, the update process principle is as follows:

[0076] ①. The network measurement point obtains the flow keyword (Key flow) and the packet keyword (Key pkg) of the data packet and obtains <flow keyword, packet keyword>.

[0077] According to one embodiment of the present invention, the two keywords can be freely defined based on the measurement task. Here, taking the measurement of the number of data packets contained in each data flow as an example, the flow keyword is specified as a five-tuple, and the packet keyword is the checksum of the IP header and TCP / UDP header in the data packet. In an illustrative embodiment, the five-tuple is in the form of: "Source IP field, destination IP field, source port number field, destination port number field, protocol type field." The checksum is the checksum field in the data packet header, i.e., a number of a specified number of digits that is different for each data packet.

[0078] ②. Determine the associated layer based on the hash value of the package keyword.

[0079] According to one embodiment of the present application, the determination of the associated layer comprises: layering the packet key by using a layering hash function to obtain a layer hash value, wherein the hash range of the layering hash function is the sum of the threshold ranges of all layers of the data sketch; and according to the threshold range in which the layer hash value is located, the layer corresponding to the threshold range is taken as the associated layer. Illustratively, the sum of the threshold ranges of all layers in the data sketch is 0 to H Figure 2 max For example, the layering hash function H d is used to hash the packet key to obtain a layer hash value h layer , and the hash range of the layering hash function is 0 to H max . If h layer is in the interval [T i , T i+1 ), then the associated layer is layer i.

[0080] 3) In the associated layer, the associated unit is determined according to the hash value of the flow key.

[0081] According to one embodiment of the present application, in each network measurement point, the determination of the associated unit comprises: in the associated layer, hashing the flow key by using a unit hash function H s to obtain a unit hash value, and according to the unit hash value, the associated unit in the layer is obtained, wherein the hash range of the unit hash function is greater than or equal to the total number of units in the layer.

[0082] Illustratively, when the hash range of the unit hash function is set to be greater than the total number of units in the layer, the flow key is hashed by using the unit hash function to obtain a unit hash value h unit . At this time, according to the total number of units in the associated layer i determined in step 2), N i , the hash range is set to be 0 to S i , and S i > N i . At this time, the associated unit in the associated layer needs to be determined as the h unit th unit in the i i th layer, h unit % N i represents the modulo operation, and when the remainder of h unit % N i is 3, it means that the associated unit is the third unit in the i

[0083] Illustratively, when the hash range of the unit hash function is set to be equal to the total number of units in the layer, if the total number of units in the layer is N i , the hash range is set to be 0 to N i .The flow key is hashed using a unit hash function to obtain a unit hash value, denoted as h unit ′ The unit hash value h unit ′ If the associated layer i is determined in step 2, the associated unit is the fifth unit in the i-th layer.

[0084] 4. Map the packet key to a position in the bitmap of the associated unit, and update the value of the position to a second value.

[0085] According to an embodiment of the present application, in each network measurement point, the way of mapping the packet key to a position in the bitmap of the associated unit comprises: using a bitmap hash function H b The bitmap hash value obtained by bitmap hashing the packet key is mapped to a position in the bitmap of the associated unit according to the bitmap hash value, wherein the hash range of the bitmap hash function is greater than or equal to the size of the bitmap of the corresponding associated unit. Illustratively, after determining the associated unit in the associated layer, the packet key is hashed using the bitmap hash function to obtain a hash value, denoted as h bitmap At this time, the value of the h bitmap bit in the bitmap is set to 1. The range of bitmap hashing is consistent with the size of the bitmap in the data unit, and the bitmap size in all data units in the same layer is the same. Taking Figure 2 For example, the associated layer determined according to the data packet is layer 2, the associated unit determined in the layer 2 is the first unit 1, and the position in the bitmap of the associated unit is the fifth position (the red number part in the figure). Therefore, the value of the fifth bit in the bitmap of the unit 1 in the layer 2 is updated to a second value, wherein 0 represents a first value and 1 represents a second value, and the current bitmap of the unit 1 is obtained as 011010.

[0086] According to an embodiment of the present application, a threshold T h For the layer hash value h d obtained by hashing the packet key using the layer hash function H layer If H max < h layer < T h , then the flow key is hashed using the unit hash function H s to obtain a unit hash value h unit , and the h unit % H-th large flow record unit in the large flow record part records the flow key of the current data packet. H represents the total number of large flow record units in the large flow record part, and h unit % H represents a modulo operation, and h unitWhen the remainder of %H is 1, it indicates that the associated unit is unit 1 in the i-th layer.

[0087] The technical scheme of the embodiment corresponding to the second aspect can achieve the following beneficial technical effects: no matter how the data packet is routed, i.e. the same data packet passes through any network measurement point, the same update will be performed in different measurement points, and there is no repeated counting. This makes the centralized controller accurately obtain the network traffic after data aggregation, and realizes accurate network measurement regardless of routing. Therefore, the benefits of the routing-independent network measurement include: 1. accurate network measurement, i.e. avoiding the problem of repeated counting; 2. more robust, i.e. the number and topology of measurement points can be changed at will.

[0088] (Three) Description of the way of aggregating the data sketch of the measurement point of the centralized controller

[0089] According to an embodiment of the present application, in the centralized controller, the aggregation manner includes: aggregating the bitmaps of the units at the same position in all collected data sketches to obtain a global data sketch having the same structure as the data sketch of the network measurement point, and the bitmap of each unit in each layer in the global data sketch is obtained by performing an OR operation on the bitmaps of the units at the corresponding position in all data sketches.

[0090] According to an embodiment of the present application, referring to Figure 3 which shows a schematic diagram of the aggregation method of the global data sketch. It demonstrates the process of the centralized controller collecting the updated data sketches from two measurement points and aggregating the two data sketches. Specifically, it includes: after the centralized controller collects the data sketch of measurement point 1 and the data sketch of measurement point 2, it performs an OR operation on the bitmaps of the units at the same position in the two data sketches to obtain a global data sketch. Taking Figure 3 the position of unit 1 in layer 2 as an example, the bitmap of unit 1 in layer 2 in the global data sketch is obtained by performing an OR operation on the bitmaps of the units at the corresponding position in all data sketches, i.e. performing a bitwise OR operation on the bitmap 010010 at the corresponding position in measurement point 1 and the bitmap 001010 at the corresponding position in measurement point 2 to obtain the bitmap 011010 of unit 1 in layer 2 in the global data sketch. For the flow record units in the large flow record part, the flow key recorded in the measurement point is updated to the corresponding position in the global data sketch, i.e. the first flow record unit is used to store the flow key f1.

[0091] According to one embodiment of the present application, since the probability of flow conflict in the large flow record part is small, the same position of multiple measurement points usually does not have the identification of multiple different flow keys. If there is no conflict, the flow key in the measurement point is directly selected for updating. If a conflict occurs, the measurement point estimates the size of the corresponding data flow for the flow key corresponding to the position of each measurement point, and selects the flow key corresponding to the maximum estimated value to update the corresponding position of the global data sketch. Illustratively, if in the same position, such as the first large flow record unit in the first embodiment, the flow key recorded by measurement point 1 is f2, and the flow key recorded by measurement point 2 is f1, if the data flow corresponding to the flow key f1 recorded by measurement point 2 is larger, the flow key f1 is updated to the position of the first large flow record unit of the global data sketch. Figure 3

[0092] According to one embodiment of the present application, when the centralized controller collects the updated data sketches from multiple measurement points and the number of data sketches to be aggregated is more than 2, the same method as in the above embodiment is used, that is, the bitmaps in the units of the same position (units of the same layer and having the same label) in all network measurement points are operated, and the bitmap is stored in the corresponding position of the global data sketch (the global data sketch is also independent of the route). The operation means that when the value of the same position in each bitmap is 1, the result is 1. Illustratively, if in the three data sketches to be aggregated, the values of the bitmaps of unit 1 in layer 2 of the three sketches are 000010, 001010 and 011000 respectively, the result of the operation of the three bitmaps is 011010, which is the value of the bitmap of unit 1 in layer 2 of the global data sketch. The second bit value in 000010 and 001010 is 1, indicating that the same data packet passes through two different measurement points. For the second bit of the two bitmaps, the prior art method may directly regard the same data packet passing through two different measurement points as two different data packets, that is, directly add them to 2, resulting in the problem of repeated counting of the same data packet. However, the operation used in the aggregation of the present application still obtains a result of 1, avoiding the repeated counting of the same data packet.

[0093] The technical solutions of the above embodiments can at least achieve the following beneficial technical effects: the aggregation method of the centralized controller can effectively avoid the problem of repeated counting. The centralized controller collects the data sketches from all measurement points and aggregates them into a global data sketch with global information, thereby providing the overall network measurement result.

[0094] (IV) Description of the method of querying and obtaining network measurement information

[0095] 1) Querying and obtaining the network measurement information of the entire network based on the global data sketch

[0096] ​According to one embodiment of the present application, in the centralized controller, the way of estimating the number of data packets transmitted by each data flow comprises steps S1 and S2:

[0097] Step S1, obtain the flow key of each given data flow, and for the flow key of each data flow, estimate the number of data packets transmitted by each data flow in the network based on the global data sketch.

[0098] According to one embodiment of the present application, the way of estimating the number of data packets transmitted by each data flow in the network based on the global data sketch comprises steps S11 and S12:

[0099] Step S11, query the information corresponding to each data flow from the global data sketch based on the flow key of each data flow, and the querying way comprises hashing the flow key of the data flow by using a unit hash function to obtain a unit hash value, and obtaining the bitmap of one unit in each layer in the global data sketch according to the unit hash value and the total number of units in each layer, so as to obtain a plurality of bitmaps.

[0100] According to one embodiment of the present application, when a flow key is given, the unit hash function H s in step ③ is used. unit The unit hash value obtained by hashing the flow key is denoted as h i , and for each layer, the hash range of the unit hash function is the same as the hash range used in each layer in step ③. According to the unit hash value and the total number of units in each layer, the bitmap of one unit in each layer in the global data sketch is queried.

[0101] Illustratively, for each layer, when the hash range is set to be greater than the total number of units in each layer: the total number of units in layer i of the global data sketch is denoted as N unit , the total number of units in layer i-1 is denoted as N i , and the unit to be obtained is the h i-1 th unit in layer i, i.e. the h unit th unit in layer i-1, and so on. That is, the above operation is repeated for all layers, and at this time, n bitmaps of n units are obtained in n layers. Wherein, N i-1 and N i have the same size or different sizes.

[0102] Illustratively, for each layer, when the hash range is set to be equal to the total number of units in each layer: the flow key is hashed by using the unit hash function and the hash range of the corresponding layer, and the unit hash value obtained is denoted as h i-1 unit . If the unit hash value h ′ unit ′ ​For 5, the acquired unit is the bitmap of the 5th unit of the corresponding layer. Repeat the above operation for all layers to acquire the bitmap of n units.

[0103] In step S12, the plurality of bitmaps are traversed in order of the number of layers. When the ratio of the number of bits being zero in the bitmap to the total number of bits of the bitmap is greater than a preset ratio, the number of data packets transmitted by the data stream is estimated based on the bitmap and the ratio of the threshold range corresponding to the layer where the bitmap is located to the sum of the threshold ranges of all layers using a linear counting algorithm.

[0104] According to one embodiment of the present application, for the n unit bitmaps acquired in step S11, the bitmaps of the units from each layer are traversed from low to high according to the layer number (i.e. the units from layer 1, the units from layer 2, and so on), the preset ratio is set to one quarter, and the traversal stops when the bitmap of a unit satisfies the condition that the ratio of the number of bits being zero in the bitmap to the total number of bits of the bitmap is greater than one quarter of the total number of bits of the current bitmap. At this time, the bitmap of the unit that satisfies this condition is used to estimate the number of data packets transmitted by the data stream, and if the unit belongs to layer i, the corresponding sampling rate of the unit is (T i-1 -T i ) / H max It should be understood that the setting of the preset ratio is only illustrative, and one quarter of the present application is an empirical value that can be customized according to specific use scenarios, for example, the preset ratio can also be set to one fifth, etc. After finding the bitmap that satisfies the condition, the linear counting algorithm is used for estimation, and in combination with the corresponding sampling rate of the unit where the bitmap is located, the number of data packets transmitted by the data stream (also referred to as the data stream size) is estimated. The estimation method is as follows:

[0105]

[0106] Wherein, F is the estimated value, m is the length of the bitmap, and u is the number of bits being zero in the current bitmap.

[0107] Taking one quarter as an example, see Figure 4 which is a schematic diagram of the query process of the data sketch. According to the stream key, the bitmap of the corresponding unit in each layer in the data sketch is acquired in the manner of the above embodiment, i.e. the acquired bitmap of the unit of layer 1 is 11101, the corresponding sampling rate is (H max -T1) / H max , the acquired bitmap of the unit of layer 2 is 10101010, the corresponding sampling rate is (T1-T2) / H max , and the acquired bitmap of the unit of layer n is 001000…00, the corresponding sampling rate is (T n-1 -T n ) / H max. Then, the layer 1 to layer n is traversed in turn, when the layer 2 is traversed, the bitmap of the unit is 10101010, and it can be known that the ratio of the number of zeros to the total number of the bitmap is greater than one fourth of the total number of bits of the current bitmap. Therefore, the estimation is performed according to the estimation manner described above. The linear counting algorithm in the above embodiment can be performed according to the manner described in reference

[13] :

[0108]

[13] : Whang, Kyu-Young, Brad T. Vander-Zanden, and Howard M. Taylor. "A linear-time probabilistic counting algorithm for database applications." ACM Transactions on Database Systems (TODS) 15.2 (1990): 208-229.

[0109] In step S2, the data packet counting information corresponding to the network measurement task is obtained based on the estimated data packet number of each data stream transmission and the network measurement task.

[0110] According to an embodiment of the present application, after the data packet number of each data stream transmission is estimated, the network measurement task is to count the data packet number of the first, second or third data stream transmission, and the measurement task can also be the sum of the data packet number of any several data stream transmissions, so as to estimate the data packet counting information corresponding to the task.

[0111] 2) Obtain the network measurement information of the measurement point based on the data sketch query of the measurement point

[0112] According to an embodiment of the present application, when a flow key is given, the network measurement information of the measurement point can be obtained by querying according to the manner of steps S1 and S2 in the above embodiment, wherein, when querying, the network measurement information is obtained based on the data sketch of the measurement point, and the obtained network measurement information is local information (i.e. the information of the measurement point), while the above embodiment is based on the global data sketch to query, and the obtained network measurement information is a global information.

[0113] (V) The measurement task issuing manner of the centralized controller in the network measurement system is described

[0114] According to an embodiment of the present application, referring to Figure 5 which is the network measurement task configuration method and working flow diagram of the system. The centralized controller can realize the measurement task issuing by configuring the flow key and the packet key.

[0115] Before introducing the centralized controller to implement the measurement task issuing, the following description is given to the structure related to the keyword processing in the network measurement point:

[0116] According to one embodiment of the present application, each network measurement point comprises: a flow keyword mask register group for storing a flow keyword mask comprising a plurality of flow mask values respectively associated with a plurality of fields of a data packet, each flow mask value being used to indicate whether a flow keyword needs one field associated with the flow mask value; and a packet keyword mask register group for storing a packet keyword mask comprising a plurality of packet mask values respectively associated with a plurality of fields of a data packet, each packet mask value being used to indicate whether a packet keyword needs one field associated with the packet mask value. As shown in Figure 5 The flow keyword mask comprises 6 flow mask values and is respectively associated with a source IP field (ScrIP), a destination IP field (DstIP), a source port number field (Sport), a destination port number field (Dport), a protocol type field (Protocol) and an ID field, and the packet keyword mask also comprises 6 flow mask values and is respectively associated with a source IP field (ScrIP), a destination IP field (DstIP), a source port number field (Sport), a destination port number field (Dport), a protocol type field (Protocol) and an ID field. In the present application, the checksum of the IP header and the TCP / UDP header is used as the ID field. When the flow mask value and the packet mask value are represented by 1 or 0, 1 represents that one field associated with the flow mask value or the packet mask value is needed, and 0 represents that one field associated with the flow mask value or the packet mask value is not needed. Illustratively, if the flow keyword mask is 110000, it means that the information of the source IP field (ScrIP) and the destination IP field (DstIP) in the flow keyword is needed, and the information of the other 4 fields is not needed.

[0117] The following describes the measurement task issuing mode and the working flow of the centralized controller:

[0118] 1) The centralized controller configures the needed keywords for the measurement task.

[0119] According to one embodiment of the present application, the configuration of the key is realized by configuring the flow key mask and the packet key mask. That is, the centralized controller is further configured to determine the flow key mask and the packet key mask according to the fields required by the flow key and the fields required by the packet key in the network measurement task, and distribute the flow key mask and the packet key mask to the plurality of network measurement points to realize the issuing of the measurement task. Wherein, the flow key mask stored in the flow key mask register group of the plurality of network measurement points is the same, and the packet key mask stored in the packet key mask register group of the plurality of network measurement points is the same. Illustratively, if the measurement task is to measure how many data packets are contained in each data flow, the flow key is set to the five-tuple form in the above embodiment, and the flow key needs to contain the five fields in the five-tuple, so the flow key mask needs to be configured as 111110, specifically, the five fields in the five-tuple are respectively set to 1.

[0120] According to one embodiment of the present application, the centralized controller modifies the measurement task by modifying the flow key mask and the packet key mask in the network measurement point. The rest of the measurement tasks can be configured in the same way, including but not limited to counting the number of data packets received by a certain IP address (i.e. setting the flow key to the destination IP address and the packet key to the ID described above), counting the number of different source addresses that communicate with a certain destination IP address (setting the flow key to the destination IP address and the packet key to the source IP address).

[0121] 2) After the flow key mask and the packet key mask are distributed to the plurality of network measurement points, when the data packet is transmitted to the plurality of network measurement points, the data packet is processed according to the flow key mask and the packet key mask stored in the network measurement point to realize the acquisition of the key of the data packet.

[0122] According to one embodiment of the present application, in each network measurement point, the way of obtaining the key of the data packet comprises: processing the data packet according to the flow key mask to obtain the flow key, which contains one or more fields, wherein the processing way comprises performing a bitwise AND operation between each field of the data packet and the flow mask value associated with the field to obtain the required field of the flow key; processing the data packet according to the packet key mask to obtain the packet key, which contains one or more fields, wherein the processing way comprises performing a bitwise AND operation between each field of the data packet and the packet mask value associated with the field to obtain the required field of the packet key. Illustratively, if the flow mask value is 010000, and the value 0 of the 6th bit is associated with the source IP field (ScrIP), the value 1 of the 5th bit is associated with the destination IP field (DstIP), the value 0 of the 4th bit is associated with the source port number field (Sport), the value 0 of the 3rd bit is associated with the destination port number field (Dport), the value 0 of the 2nd bit is associated with the protocol type field (Protocol), and the value 0 of the 1st bit is associated with the ID field, performing a bitwise AND operation between the flow mask value and the associated fields, for example, performing a bitwise AND operation between the value 0 of the 6th bit and each bit of the source IP field (ScrIP) associated with the value 0 of the 6th bit, so that the source IP field (ScrIP) is set to 0, and performing a bitwise AND operation between 1 and each bit of the destination IP field (DstIP) associated with 1, so as to obtain the destination IP field (DstIP), i.e. the field is the required field information of the flow key. The other fields are processed in the same way to obtain the flow key of the data packet, and the packet key of the data packet is obtained in the same way.

[0123] 3) The flow key and the packet key of the data packet are input to the updated routing-independent data sketch. The packet key is hashed by a hierarchical hash function to determine the associated layer, and in the associated layer, a unit hash function H s The flow key is hashed to determine the associated unit, and a bitmap hash function H b The packet key is bitmap hashed to determine a position to be updated in the bitmap of the associated unit, so as to obtain the updated routing-independent data sketch.

[0124] 4) After the update is completed, the centralized controller collects the updated data sketch from the network measurement points.

[0125] According to one embodiment of the present application, a network-wide measurement method based on the system described in the above embodiment is provided, the method comprising: through each network measurement point in the plurality of network measurement points, based on the network measurement task issued by the centralized controller, when receiving and forwarding a data packet, obtaining the key of the data packet, including the packet key and the flow key of the data flow to which the data packet belongs; through each network measurement point in the plurality of network measurement points, updating the data sketch of the network measurement point according to the key of the data packet, the updating manner comprising: determining the associated layer according to the hash value of the packet key, and determining the associated unit in the associated layer according to the hash value of the flow key; mapping the packet key to a position in the bitmap of the associated unit, and updating the value of the position to a second value; through the centralized controller, collecting the updated data sketch of each network measurement point and aggregating all collected data sketches to obtain a global data sketch, and estimating the number of data packets transmitted by each data flow in the network-wide measurement according to the global data sketch, so as to complete the network measurement task.

[0126] In order to verify the beneficial effects of the present application, the following comparative experiments were carried out:

[0127] The network measurement task is set to measure the number of data packets received by each destination address. During the measurement process, the total amount of data packets input into the network measurement point is eight million to ten million, represented by 8M, 9M and 10M respectively. Under the three total amounts of data packets, the present application system and the prior art (i.e. the above-mentioned reference [7], referred to as Jaqen) are compared in the experiment, and the weighted average percentage error is calculated by the estimated value of the number of data packets obtained by the two systems during the execution of the measurement task. The error comparison results are shown in Table 1 as follows:

[0128] Table 1: Comparison of weighted average percentage error

[0129]

[0130] According to the data in Table 1 above, compared with the prior art Jaqen, the measurement result of the present application has smaller error. Moreover, the present application can not only estimate the total flow, but also estimate the number of data packets transmitted by each data flow, and can realize various network measurement task types, with wide application range.

[0131] It should be noted that although the above describes the steps in a specific order, it does not mean that the steps must be performed in the above specific order, in fact, some of the steps can be executed concurrently, or even in a changed order, as long as the required function can be realized.

[0132] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0133] Computer-readable storage media can be a tangible device that holds and stores the instructions used by an instruction execution device. Computer-readable storage media can, for example, include, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, for example, and any suitable combination thereof.

[0134] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A route-independent network measurement system, characterized in that: The system includes multiple distributed network measurement points and a centralized controller, among which, Each network measurement point maintains a data sketch that is not related to routing, wherein the data sketch is a multi-layer structure, each layer includes a plurality of units arranged in sequence according to position, each unit is provided with a bitmap, and the initial value of all bits in the bitmap is a first value; Each network measurement point is configured as: Receive and forward a data packet, obtain a keyword of the data packet according to the network measurement task, including a packet keyword and a flow keyword of a data flow to which the data packet belongs, and update a data sketch of the network measurement point according to the keyword of the data packet, wherein the updating method includes: Determine an associated layer according to a hash value of the packet keyword, and determine the associated unit in the associated layer according to a hash value of the flow keyword; Mapping the packet key to a position in the bitmap of the associated unit and updating the value of the position to a second value; The centralized controller is configured as: Collect updated data sketches from multiple network measurement points and aggregate all collected data sketches to obtain a global data sketch. Based on the global data sketch, estimate the number of data packets transmitted by each data flow in the network to complete the network measurement task.

2. The system according to claim 1, wherein: At each network measurement point, a threshold range is preset for each layer in its data sketch, and the threshold ranges of all layers of the data sketch do not intersect; The associated layer is determined in the following manner: Performing hierarchical hashing on the packet keyword using a hierarchical hash function to obtain a layer hash value, wherein a hash range of the hierarchical hash function is the sum of threshold ranges of all layers of the data sketch; According to the threshold range of the layer where the layer hash value is located, the layer corresponding to the threshold range is used as the associated layer.

3. The system according to claim 1 or 2, characterized in that At each network measurement point, the associated unit is determined in the following manner: In the associated layer, the flow keyword is hashed using the unit hash function to obtain a unit hash value, and the associated unit in the layer is obtained according to the unit hash value mapping, wherein the hash range of the unit hash function is greater than or equal to the total number of units in the layer.

4. The system according to claim 3, characterized in that The bitmaps of the plurality of units in each layer of the data sketch have the same bit size. At each network measurement point, the mapping to a position in the bitmap of the associated unit includes: A bitmap hash function is used to perform a bitmap hash on the packet keyword to obtain a bit hash value, and the bit hash value is mapped to a position in the bitmap of the associated unit according to the bit hash value, wherein a hash range of the bitmap hash function is greater than or equal to the number of bits of the bitmap of the corresponding associated unit.

5. The system according to claim 1, wherein: In the centralized controller, the aggregation methods include: The bitmaps of the cells at the same position in all the collected data sketches are aggregated to obtain a global data sketch with the same structure as the data sketch of the network measurement point. The bitmap of each cell in each layer in the global data sketch is obtained by performing an OR operation based on the bitmaps of the cells at the corresponding position in all the data sketches.

6. The system according to claim 1, wherein: Each network measurement point includes: a flow keyword mask register group, used for storing a flow keyword mask, which includes a plurality of flow mask values ​​respectively associated with a plurality of fields of a data packet, each flow mask value being used to indicate whether a flow keyword needs a field associated with the flow mask value; The packet keyword mask register group is used to store the packet keyword mask, which includes multiple packet mask values ​​respectively associated with multiple fields of the data packet, and each packet mask value is used to indicate whether a field in the packet keyword needs to be associated with the packet mask value.

7. The system according to claim 6, characterized in that At each network measurement point, methods for obtaining the keyword of the data packet include: Processing the data packet according to the flow keyword mask to obtain a flow keyword comprising one or more fields, wherein the processing comprises performing a bitwise AND operation on each of the plurality of fields of the data packet and a flow mask value associated with the field to obtain the fields required for the flow keyword; The data packet is processed according to the packet keyword mask to obtain a packet keyword, which includes one or more fields, wherein the processing includes performing a bitwise AND operation on each of the multiple fields of the data packet and a packet mask value associated with the field to obtain the fields required for the packet keyword.

8. The system according to claim 6, wherein: The centralized controller is further configured to: According to the fields required by the flow keyword and the fields required by the packet keyword in the network measurement task, the flow keyword mask and the packet keyword mask are determined, and the flow keyword mask and the packet keyword mask are distributed to multiple network measurement points.

9. The system according to any one of claims 2 to 8, characterized in that: In the centralized controller, the methods for estimating the number of packets transmitted by each data flow include: Get the flow key of each data flow, and for each data flow flow key, estimate the number of packets transmitted by each data flow in the network based on the global data sketch in the following way: Based on the flow keyword of each data flow, query the global data sketch to obtain information corresponding to each data flow, wherein the query method includes hashing the flow keyword of the data flow using a unit hash function to obtain a unit hash value, and obtaining a bitmap of a unit in each layer in the global data sketch according to the unit hash value and the total number of units in each layer to obtain multiple bitmaps; Traversing the plurality of bitmaps in order by layer, and when a ratio of the number of zero bits in a bitmap to the total number of bits in the bitmap is greater than a preset ratio, estimating the number of data packets transmitted by the data stream using a linear counting algorithm based on a ratio of a threshold range corresponding to the bitmap and the layer where the bitmap is located to the sum of the threshold ranges of all layers; Based on the estimated number of data packets transmitted by each data flow and the network measurement task, data packet counting information of the corresponding task is obtained.

10. A network-wide measurement method based on the system according to any one of claims 1 to 9, characterized in that: Methods include: obtaining, by each of the plurality of network measurement points, a keyword of the data packet when receiving and forwarding the data packet based on the network measurement task issued by the centralized controller, including the packet keyword and the flow keyword of the data flow to which the data packet belongs; Each of the plurality of network measurement points updates a data sketch of the network measurement point according to the keyword of the data packet, and the updating method includes: Determine an associated layer according to a hash value of the packet keyword, and determine the associated unit in the associated layer according to a hash value of the flow keyword; Mapping the packet key to a position in the bitmap of the associated unit and updating the value of the position to a second value; The centralized controller collects the updated data sketches of multiple network measurement points and aggregates all the collected data sketches to obtain a global data sketch. Based on the global data sketch, the number of data packets transmitted by each data flow in the entire network is estimated to complete the network measurement task.

Images