A network monitoring system based on multi-pipeline switch

Abstract

The application discloses a network monitoring system based on a multi-pipeline switch, wherein a data plane of the switch is used for processing and forwarding various types of data messages on different ports of the switch, and reporting flows exceeding a pipeline threshold to a control plane of the switch; the control plane of the switch is used for controlling and managing the operation of all modules in the switch, and monitoring the reported flows. When a data packet enters a pipeline of the switch, a program on the pipeline judges whether the data packet is a potential target, and if yes, the pipeline reports the data packet to the control plane of the switch. A controller located at the control plane of the switch pulls and merges data of all pipelines of the data plane at the end of each epoch, and checks whether the flows reported by the data plane are real targets. After integrating information collected in the current epoch, the controller sends new control information to each pipeline of the data plane, so that the pipelines work more accurately and efficiently.

Description

Technical Field

[0001] This invention belongs to the field of network monitoring technology, specifically, it relates to a network monitoring system based on a multi-pipeline switch. Background Technology

[0002] With the explosive growth of network traffic, increasingly complex networks inevitably encounter problems such as configuration errors, malicious attacks, and hardware failures, which affect network security and stability. Network measurement can efficiently collect information from the data plane, helping to detect network bottlenecks and abnormal behavior, and effectively supporting network management. Switches, due to their superior network location, are well-suited for network measurement tasks. Furthermore, the emergence of programmable switches has greatly expanded their functionality, making it possible for switches to perform complex functions.

[0003] The core of a programmable switch is a programmable pipeline that processes data packets at line speed. Existing commercial programmable switches employ an architecture with multiple parallel data packet processing pipelines, as shown in the diagram. Figure 1 As shown, each switch pipeline is divided into two parts: Ingress and Egress. Ingress connects to the ingress port of the switch, while Egress connects to the egress port of the switch. The two are interconnected through a Traffic Manager (TM). Multiple pipelines are independent of each other and cannot share state and data.

[0004] There are two main types of metrics in network measurement: HeavyHitter and Distinct counting, such as DDoS and SuperSpreader. HeavyHitter is mainly used to monitor hotspot traffic; DDoS and SuperSpreader are used to monitor network attacks. For ease of explanation, we will use (source IP, destination IP) pairs as the flow key and the number of packets in the flow as the measure of flow size.

[0005] Current network monitoring solutions include systems such as Marple, Sonata, and PacketScope, as well as network measurement algorithms such as various Sketch and BeauCoup algorithms. Taking Sonata as an example, this monitoring system supports a variety of query requests. If Sonata is used to detect which flows are Heavy Hitters: the flow key of each packet is extracted from the switch's data plane, hashed, and the hash value is used as an index in a register. The count in the corresponding register is accumulated to count the flow size. Once the count corresponding to a packet reaches a threshold, the data plane reports the flow. Taking the BeauCoup algorithm as an example, it is inspired by the "coupon collection problem." BeauCoup divides packet information into two parts: keywords and attributes, and maintains a coupon array for each received keyword. Taking the SuperSpreader problem as an example, the keyword is the source IP, and the attribute is the destination IP. Whenever a packet enters the switch, BeauCoup maps it to a coupon based on the value of its destination IP, indicating that the packet has collected that coupon. Packets from different destination IPs may be mapped to a single coupon, but will only be counted once. Ultimately, the number of destination IPs can be approximated based on the number of coupons collected, thus detecting SuperSpreader. Additionally, some distributed monitoring systems using multiple switches reduce the communication volume between multiple switches and a central coordinator while maintaining accuracy.

[0006] However, most of these methods assume that the programmable switch has only one pipeline and do not take into account the actual architecture of the switch. Therefore, when applied to existing programmable switches, they cannot accurately measure the corresponding metrics to reflect the true state of the network. Furthermore, existing distributed multi-machine measurement methods are not applicable to single-machine multi-pipeline switch architectures and mostly only provide heavy hitter measurement methods.

[0007] In summary, current network measurement methods can lead to false alarms and missed alarms in a single-switch, multi-pipeline architecture. For example, there is the HeavyHitter problem, where traffic is sent to multiple pipelines at different rates. Even if the total traffic has reached the HeavyHitter threshold, a single pipeline may not have reached it and therefore will not report it. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a network monitoring system based on a multi-pipeline switch, which combines the data plane line-speed processing capability of the switch with the global perspective of the control plane to solve the network monitoring problem of multi-pipeline switches.

[0009] To achieve the above technical objectives, the present invention adopts the following technical solution: a network monitoring system based on a multi-pipeline switch, comprising: a switch control plane and a switch data plane, wherein the switch data plane is used to process and forward various types of data packets on different ports of the switch, and to report flows exceeding the pipeline threshold to the switch control plane, wherein the switch control plane is used to control and manage the operation of all modules in the switch, and to monitor all pipelines of the switch data plane.

[0010] Furthermore, the switch data plane includes: a HeavyHitter measurement module, a Distinct measurement module, and a forwarding module. The HeavyHitter measurement module measures the HeavyHitter index of data packets entering the switch through a single pipeline, reports the flow containing data packets with a HeavyHitter index pipeline threshold to the switch control plane, and forwards the data packets out of the switch through the forwarding module. The Distinct measurement module measures the Distinct index of data packets entering the switch through a single pipeline, reports the flow containing the Distinct index pipeline threshold to the switch control plane, and forwards the data packets out of the switch through the forwarding module.

[0011] Furthermore, the specific working process of the HeavyHitter measurement module is as follows:

[0012] Step 1: Calculate the memory address of the incoming data packet based on the flow key;

[0013] Step 2: Obtain the corresponding counter based on the memory address and increment the counter by 1;

[0014] Step 3: Determine whether the count in the counter is greater than the HeavyHitter pipeline threshold. If it is, report the flow in which the data packet is located and forward the data packet traffic normally.

[0015] Furthermore, the specific working process of the Distinct measurement module is as follows:

[0016] Step 1: Calculate the coupon array index based on the flow key for the data packets entering the switch;

[0017] Step 2: Set the corresponding coupon array position to 1 according to the coupon array index;

[0018] Step 3: Check if the number of 1s in the coupon array is greater than the Distinct indicator pipeline threshold. If it is, report the flow in which the data packet is located and forward the data packet traffic normally.

[0019] Furthermore, the switch control plane includes a receiving module, a statistics module, and a sending module. The receiving module is used to receive the flow key reported by the HeavyHitter measurement module and the Distinct measurement module. The statistics module determines which flows meet the global threshold based on the flow key and the measurement value of the switch control plane. The sending module sends the updated global threshold to the switch data plane.

[0020] Furthermore, the working process of the statistics module is as follows: obtain the cumulative measurement value of the switch data plane, traverse the flow key reported by the HeavyHitter measurement module or the Distinct measurement module, and determine whether the cumulative measurement value of all pipelines corresponding to the flow key meets the global threshold; if it does, record the flow corresponding to the flow key; otherwise, recalculate the global threshold.

[0021] Furthermore, the process of acquiring the cumulative measurement value of the switch data plane is as follows: check whether the time of the switch control plane has reached one epoch. If the time has passed one epoch, collect the measurement values ​​of all pipelines in the switch data plane.

[0022] Furthermore, when the measurement metric is HeavyHitter, the process of recalculating the global threshold is as follows: the global threshold is allocated to each pipeline of the switch according to the actual traffic ratio, and the allocation of the global threshold is updated.

[0023] Furthermore, when the measurement metric is Distinct, the process of recalculating the global threshold is as follows: set two parameters: coupon threshold and maximum number of draws. Calculate the maximum number of draws for each pipeline according to the actual traffic ratio in the switch. Input the maximum number of draws for each pipeline into the mapping array couponToNumber between the number of draws and the number of coupons to obtain the corresponding coupon threshold.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) This invention utilizes the multi-pipeline of a programmable switch, thereby making more effective use of the physical resources of the multi-pipeline of the switch, reducing network measurement costs, and improving the scalability of the network monitoring system.

[0026] (2) The present invention is based on a multi-pipeline switch, which has a higher degree of parallelism than a single pipeline and can process multiple data streams at the same time, thereby improving measurement efficiency and throughput.

[0027] (3) Compared with existing distributed measurement methods, this invention dynamically updates the reporting conditions of the switch data plane by setting the HeavyHitter measurement module and the Distinct measurement module, which reduces a lot of data communication overhead and can maintain a fairly high measurement accuracy at the same time.

[0028] (4) This invention addresses the problem of Distinct network monitoring and measurement by combining global threshold updates with the coupon threshold and maximum number of extractions in the BeauCoup algorithm, thereby reducing a large amount of physical memory overhead of the switch.

[0029] (5) This invention can be used very conveniently in different network architectures and has good openness and scalability. Attached Figure Description

[0030] Figure 1 The existing commercial programmable switches employ an architecture with multiple parallel packet processing pipelines.

[0031] Figure 2 This is a framework diagram of the network monitoring system based on a multi-pipeline switch according to the present invention;

[0032] Figure 3 This is a flowchart illustrating the workflow of the HeavyHitter measurement module in this invention.

[0033] Figure 4 This is a flowchart illustrating the workflow of the Distinct measurement module in this invention.

[0034] Figure 5 This is a flowchart of the statistical module in this invention. Detailed Implementation

[0035] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.

[0036] like Figure 2This is a framework diagram of the network monitoring system based on a multi-pipeline switch according to the present invention. The network monitoring system includes a switch control plane 201 and a switch data plane 208. The switch data plane 208 is used to process and forward various types of data packets on different ports of the switch, and reports flows exceeding pipeline thresholds to the switch control plane 201. The switch control plane 201 is used to control and manage the operation of all modules in the switch and monitor all pipelines in the switch data plane 208. This invention, based on multi-pipeline switching, has superior parallel processing capabilities compared to traditional single-pipeline methods, enabling the simultaneous processing of multiple data flows, thereby significantly improving measurement efficiency and system throughput. Compared to current distributed measurement methods, this invention successfully reduces a large amount of data communication overhead by dynamically updating the reporting conditions of the switch data plane, while maintaining a high level of measurement accuracy. Furthermore, this invention exhibits great flexibility in different network architectures, possessing excellent openness and scalability, making its application in various network environments more convenient and widespread.

[0037] In this invention, the switch data plane 208 includes a HeavyHitter measurement module 205, a Distinct measurement module 206, and a forwarding module 207. The HeavyHitter measurement module 205 measures the HeavyHitter index of data packets entering the switch via a single pipeline, reports flows containing data packets with a HeavyHitter index pipeline threshold to the switch control plane 201, and forwards the data packets out of the switch via the forwarding module 207. The Distinct measurement module 206 measures the Distinct index of data packets entering the switch via a single pipeline, reports flows with a Distinct index pipeline threshold to the switch control plane 201, and forwards the data packets out of the switch via the forwarding module 207. The HeavyHitter measurement module 205 is mainly used to measure flows exceeding a preset threshold within an epoch, also known as "elephant flows," which are often the source of network congestion. The Distinct measurement module 206 is mainly used to measure the initiator and victim of network attacks. The flow keys that meet the corresponding pipeline thresholds in the measurement results of the two measurement modules are reported. These reported flow keys serve as the source for the statistics module's global statistics. The measurement modules initially filter out traffic that meets the conditions for the statistics module, reducing network communication overhead, saving control plane computation and communication overhead, and greatly improving measurement efficiency. Finally, the traffic is forwarded normally through the forwarding module, achieving transparency to other devices in the network and causing no damage to network data.

[0038] like Figure 3 The specific working process of the HeavyHitter measurement module 205 is as follows:

[0039] Step 1: Calculate the memory address of the incoming data packet based on the flow key;

[0040] Step 2: Obtain the corresponding counter based on the memory address and increment the counter by 1;

[0041] Step 3: Determine if the count in the counter is greater than the HeavyHitter pipeline threshold. If it is, report the flow where the data packet is located, thereby filtering out flows that may meet the global threshold. The control plane only needs to perform global statistics on this part of the flowkey, reducing the size of the flow set traversed by the control plane, thereby reducing the communication and computing overhead when the control plane performs global statistics, and finally forwarding the data packet traffic normally.

[0042] like Figure 4 The specific working process of the Distinct measurement module 206 is as follows:

[0043] Step 1: Calculate the coupon array index based on the flow key for the data packets entering the switch;

[0044] Step 2: Set the corresponding coupon array position to 1 according to the coupon array index;

[0045] Step 3: Check if the number of 1s in the coupon array is greater than the Distinct indicator pipeline threshold. If it is, report the flow where the data packet is located, thereby filtering out flows that may meet the global threshold. The control plane only needs to perform global statistics on this part of the flowkey, reducing the size of the flow set traversed by the control plane, thereby reducing the communication and computing overhead when the control plane performs global statistics, and finally forwarding the data packet traffic normally.

[0046] In this invention, the switch control plane 201 includes a receiving module 202, a statistics module 203, and a distribution module 204. The receiving module 202 receives the flow keys reported by the HeavyHitter measurement module 205 and the Distinct measurement module 206. The statistics module 203 determines which flows meet the global threshold based on the flow key and the measurement values ​​of the switch control plane 201. The distribution module 204 distributes the updated global threshold to the switch data plane 208. The statistics module 203, located in the switch control plane 201, can perform measurements from the global perspective of the switch, obtaining measurement values ​​from multiple pipelines on the data plane. Measurement results from the perspective of a single pipeline may have errors; global statistics improve the accuracy of network measurements.

[0047] like Figure 5The specific working process of the statistics module 203 is as follows: It acquires the cumulative measurement values ​​of the switch data plane 208, iterates through the flow keys reported by the HeavyHitter measurement module 205 or the Distinct measurement module 206, and determines whether the cumulative measurement values ​​of all pipelines corresponding to the flow key meet the global threshold. If they do, it records the flow corresponding to the flow key; otherwise, it recalculates the global threshold. The statistics module 203 accumulates and statistically analyzes all pipelines. Compared to the measurement results from the perspective of a single pipeline, global statistics have higher accuracy. This is because it can comprehensively consider the information of different pipelines within the switch, thereby gaining a more comprehensive understanding of the characteristics of network traffic and reducing the errors that may exist in a single pipeline. By adopting a global perspective measurement method, the statistics module effectively improves the accuracy of network measurement and its global awareness capability.

[0048] The process of acquiring the cumulative measurement value of the switch data plane 208 is as follows: check whether the time of the switch control plane 201 has reached one epoch. If the time has passed one epoch, collect the measurement values ​​of all pipelines in the switch data plane 208.

[0049] When the measured metric is HeavyHitter, the process of recalculating the global threshold involves allocating the global threshold to each pipeline of the switch according to the actual traffic ratio, and updating the global threshold allocation. By allocating the global threshold according to the actual traffic ratio, the probability of false alarms in the next epoch is successfully reduced, thereby significantly improving the measurement efficiency of the control plane statistics module. This strategy fully considers the changes under different traffic scenarios, making the global threshold more adaptable to the actual network conditions. Therefore, in network measurement, by more accurately identifying and reporting meaningful events, unnecessary false alarms are reduced, and the overall efficiency of the network monitoring system is improved.

[0050] When the measurement metric is Distinct, the process of recalculating the global threshold is as follows: Two parameters are set: a coupon threshold and a maximum number of draws. The maximum number of draws is calculated for each pipeline based on the actual traffic ratio in the switch. The maximum number of draws for each pipeline is then input into the mapping array `couponToNumber` between the number of draws and the number of coupons to obtain the corresponding coupon threshold. The global threshold set based on the Beaucoup algorithm reduces memory overhead, thus enabling network measurement of the Distinct problem within the limited memory resources of the switch. By allocating the global threshold according to the actual traffic ratio, the probability of false alarms in the next epoch is successfully reduced, thereby significantly improving the measurement efficiency of the control plane statistics module.

[0051] Example

[0052] The monitoring effectiveness of the network monitoring system based on multi-pipeline switches in this invention was verified through NS3 simulation experiments: network traces collected in 2020 using the WIDE MAWI database were used to evaluate the performance of different schemes. The MAWI database collects traces from real-world networks to help researchers evaluate their traffic anomaly detection methods. In this invention, a 15-second trace was selected, containing approximately 8.59 million data packets. The epoch duration was set to 1 second, and the replay rate of the trace was set to 1 Mpkt / s.

[0053] This method simulates the distribution of network traffic across n different pipelines on a switch, referencing distributed traffic distribution. For each source IP, the flow is assigned to a specific pipeline on the switch with probability p, called the "primary pipeline," and then to three other pipelines with probability (1-p) / (n-1). This is to simulate the locality of traffic, meaning that most flows corresponding to a given source IP will likely pass through the "primary pipeline," while a smaller portion will pass through other pipelines. In this verification method, the probability p is set to 0.8, and the number of pipelines n is set to 4.

[0054] To address the Heavy Hitter problem, this experiment primarily compares the results with distributed switch measurement methods. The comparison metric is the communication overhead between the data plane and control plane, specifically the number of messages reported from the data plane to the control plane per epoch. In this invention, setting the global threshold to 1000, 5000, and 10000 for the Heavy Hitter problem shows no significant difference in measurement performance, which is generally applicable. The following experiment uses a threshold of 5000. Compared to distributed switch measurement methods, this invention reduces communication overhead by 99.73%, a difference of several orders of magnitude, significantly reducing communication costs. Simultaneously, the dynamic threshold method reduces the size of the reported flow set per epoch by 35%, greatly reducing the computational overhead of the control plane and improving the computational efficiency of the switch's CPU.

[0055] For the Distinct counting problem, the comparison metric remains the communication overhead between the data plane and the control plane. In the Heavy Hitter problem, this invention sets the global threshold to 100 and 200, with no significant difference in measurement results. Without loss of generality, the following experiments use a threshold of 100. In DDoS detection, communication overhead is reduced by 96%; in SuperSpreader detection, communication overhead is reduced by 94%, with the communication volume differing by several orders of magnitude, significantly reducing communication overhead.

[0056] BeauCoup is a probabilistic algorithm, and these transformation operations inevitably result in some loss of precision, thus causing a decrease in the effectiveness of dynamic adjustment in some cases. The experimental evaluation metrics are recall and precision. Recall refers to the percentage of actual SuperSpreaders or DDoS Victims identified, while precision refers to the percentage of streams identified by this invention that are genuine SuperSpreaders and DDoS Victims. Experiments show that BeauCoup has quite high precision; in most settings and epochs, BeauCoup's recall and precision are both 1, and it has not fallen below 0.5 in any setting.

[0057] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A network monitoring system based on a multi-pipeline switch, characterized in that, include: The switch control plane (201) and the switch data plane (208) are used to process and forward various types of data packets on different ports of the switch, and report flows exceeding the pipeline threshold to the switch control plane (201). The switch control plane (201) is used to control and manage the operation of all modules in the switch and monitor all pipelines of the switch data plane (208). The switch data plane (208) includes: a HeavyHitter measurement module (205), a Distinct measurement module (206), and a forwarding module (207). The HeavyHitter measurement module (205) measures the HeavyHitter index of data packets entering the switch through a single pipeline, reports the flow containing data packets with a HeavyHitter index pipeline threshold to the switch control plane (201), and forwards the data packets out of the switch through the forwarding module (207). The Distinct measurement module (206) measures the Distinct index of data packets entering the switch through a single pipeline, reports the flow containing data packets with a Distinct index pipeline threshold to the switch control plane (201), and forwards the data packets out of the switch through the forwarding module (207). The switch control plane (201) includes a receiving module (202), a statistics module (203), and a sending module (204). The receiving module (202) is used to receive the flow key reported by the HeavyHitter measurement module (205) and the Distinct measurement module (206). The statistics module (203) determines which flows meet the global threshold based on the flow key and the measurement value of the switch control plane (201). The sending module (204) sends the updated global threshold to the switch data plane (208). The working process of the statistics module (203) is as follows: obtain the cumulative measurement value of the switch data plane (208), traverse the flow key reported by the HeavyHitter measurement module (205) or the Distinct measurement module (206), and determine whether the cumulative measurement value of all pipelines corresponding to the flow key meets the global threshold; if it does, record the flow corresponding to the flow key; otherwise, recalculate the global threshold.

2. The network monitoring system based on a multi-pipeline switch according to claim 1, characterized in that, The specific working process of the HeavyHitter measurement module (205) is as follows: Step 1: Calculate the memory address of the incoming data packet based on the flow key; Step 2: Obtain the corresponding counter based on the memory address and increment the counter by 1; Step 3: Determine whether the count in the counter is greater than the HeavyHitter pipeline threshold. If it is, report the flow in which the data packet is located and forward the data packet traffic normally.

3. The network monitoring system based on a multi-pipeline switch according to claim 1, characterized in that, The specific working process of the Distinct measurement module (206) is as follows: Step 1: Calculate the coupon array index based on the flow key for the data packets entering the switch; Step 2: Set the corresponding position in the coupon array to 1 according to the coupon array index; Step 3: Check if the number of 1s in the coupon array is greater than the Distinct indicator pipeline threshold. If it is, report the flow in which the data packet is located and forward the data packet traffic normally.

4. A network monitoring system based on a multi-pipeline switch according to claim 1, characterized in that, The process of acquiring the cumulative measurement value of the switch data plane (208) is as follows: check whether the time of the switch control plane (201) has reached one epoch. If the time has passed one epoch, collect the measurement values ​​of all pipelines in the switch data plane (208).

5. A network monitoring system based on a multi-pipeline switch according to claim 1, characterized in that, When the measurement metric is HeavyHitter, the process of recalculating the global threshold is as follows: the global threshold is allocated to each pipeline of the switch according to the actual traffic ratio, and the allocation of the global threshold is updated.

6. A network monitoring system based on a multi-pipeline switch according to claim 1, characterized in that, When the measurement metric is Distinct, the process of recalculating the global threshold is as follows: set two parameters: coupon threshold and maximum number of draws. Calculate the maximum number of draws for each pipeline according to the actual traffic ratio in the switch. Input the maximum number of draws for each pipeline into the mapping array couponToNumber between the number of draws and the number of coupons to obtain the corresponding coupon threshold.

Images