Network flow measurement method based on Sketch-INT cooperation, electronic equipment and medium

Abstract

The invention discloses a network flow measurement method based on Sketch-INT cooperation, electronic equipment and a medium, and the method comprises the steps: firstly obtaining network topology and source-destination pair information, and constructing a candidate forwarding path set; thirdly, a multi-target optimization model is constructed with the maximization of the flow coverage rate and the minimization of the distance to the control plane as targets, a measurement point set and an assignment relation are solved through Lagrangian relaxation and subgradient optimization, and Sketch and INT measurement programs are cooperatively deployed at selected points; then, flow is divided into a first class flow and a second class flow according to real-time distribution, and Sketch statistics and INT packet-by-packet collection are adopted respectively; and finally, based on a constrained Markov decision process, in combination with states such as a link utilization rate, a queue length and a worst sending rate, intelligently planning a report path and a distribution proportion, and ensuring non-congestion transmission of measurement data. According to the method, high-precision and low-resource-overhead whole-network flow monitoring is realized.

Description

Technical Field

[0001] This invention belongs to the field of network monitoring technology, and particularly relates to a network traffic measurement method, electronic device, and medium based on Sketch-INT collaboration. Background Technology

[0002] Network measurement is the foundation of network management and operation. By monitoring network traffic statistics in real time (such as packet count, flow size, etc.), network administrators can accurately grasp the real-time status of the network and effectively detect abnormal events such as distributed denial-of-service (DDoS) attacks, network congestion, and link failures.

[0003] Currently, mainstream programmable network measurement technologies are mainly divided into data summarization (Sketch) technology and in-band network telemetry (INT) technology: 1. Data Summarizing (Sketch) Technology: This technology (such as Count-Min Sketch) uses hash functions to map massive network flows to a fixed-size counter matrix, offering significant advantages such as high memory resource efficiency and good hardware compatibility. However, in high-speed network environments, the number of network flows far exceeds the limited memory capacity of switches, making hash collisions inevitable. This leads to a significant decrease in measurement accuracy under high-traffic scenarios, and mapping multiple flows to the same counter generates significant statistical errors, making it difficult to achieve high-precision flow identification and statistics.

[0004] 2. In-band Telemetry (INT) Technology: This technology collects detailed information such as path, latency, and queue depth packet by packet and hop by hop by packet by embedding telemetry instructions in the packet header, providing extremely high measurement accuracy and network-wide visibility. However, the application of INT technology is limited by huge bandwidth overhead. In high-speed networks, adding an INT header to each packet consumes a large amount of payload bandwidth, and the reporting of massive amounts of packet-by-packet telemetry data places a heavy burden on network links and control plane processing nodes.

[0005] To balance resource consumption and measurement accuracy, several hybrid measurement schemes have emerged in existing technologies. However, these schemes typically simply combine two technologies or employ static task partitioning, lacking the ability to perceive real-time traffic distribution and network conditions. For example, some schemes utilize Sketch structures to compress INT data, but are still limited by accuracy bottlenecks caused by hash collisions; others use INT to collect Sketch fragments, but face high reconstruction latency issues in large-scale networks.

[0006] In summary, in complex environments with incomplete routing information and limited network resources, how to achieve deep collaboration between Sketch and INT, and adjust measurement strategies in real time based on dynamic traffic distribution to minimize network resource overhead while ensuring high measurement accuracy, is a technical challenge that urgently needs to be solved in the field of network measurement. Summary of the Invention

[0007] To address the shortcomings of existing technologies, embodiments of the present invention provide a network traffic measurement method, electronic device, and medium based on Sketch-INT collaboration.

[0008] In a first aspect, embodiments of the present invention provide a network traffic measurement method based on Sketch-INT collaboration, the method comprising: Obtain the topology information, programmable switch set, and control plane node set of the network under test, and obtain the source and destination pairs corresponding to the flow in the network under test; Based on the source and destination pairs, construct a set of candidate forwarding paths for each network flow; Set constraints; construct a multi-objective optimization model with the objectives of maximizing flow coverage and minimizing the distance from the measurement point to the control plane; solve the multi-objective optimization model based on the constraints using Lagrange relaxation and subgradient optimization to obtain the set of measurement points and the assignment relationship between the measurement points and the control plane nodes; and collaboratively deploy the data digest Sketch measurement program and the in-band network telemetry INT measurement program on the programmable switch corresponding to the measurement point. Based on real-time traffic distribution, network traffic is divided into Class I network traffic and Class II network traffic. The Class I network traffic is measured using the Data Digest Sketch measurement program, and the Class II network traffic is measured using the In-Band Network Telemetry INT measurement program. Based on a constrained Markov decision process, actions are selected according to the state, thereby sending Sketch summary data and INT telemetry data of each measurement point to the control plane. The state includes link utilization, link queue length, and the total worst transmission rate of measurement data for each measurement point. The actions include the reporting path selected from the candidate forwarding path set and its corresponding offloading ratio.

[0009] In a second aspect, embodiments of the present invention provide an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-described network traffic measurement method based on Sketch-INT collaboration.

[0010] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the above-described network traffic measurement method based on Sketch-INT collaboration.

[0011] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, characterized in that the computer program / instruction, when executed by a processor, implements the above-described network traffic measurement method based on Sketch-INT collaboration.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention divides network traffic into Class I and Class II network flows based on real-time traffic distribution. It fully utilizes the complementarity of Sketch and INT technologies, employing the data digest Sketch measurement program to measure Class I network flows (i.e., large flows) and the in-band network telemetry INT measurement program to measure Class II network flows (i.e., small flows). Differentiated measurement strategies are applied to large and small flows based on real-time traffic distribution. While effectively reducing the measurement error of small flows, it significantly reduces the resource overhead of switches and the processing load of the control plane, achieving a balance between high accuracy and low overhead.

[0013] (2) By establishing the worst-case transmission rate estimate of measurement data and performing path planning and traffic splitting decisions under the constraint of link security threshold, this invention significantly reduces the risk of network congestion and data loss caused by measurement data and enhances the stability and reliability of measurement data reporting.

[0014] (3) This invention supports multi-control plane node collaboration and distributed traffic reporting mechanism. This architecture design can not only effectively distribute the pressure of a single control node, but also significantly improve the scalability and robustness in dealing with node failures in large-scale network environments.

[0015] (4) The present invention combines online dynamic rerouting decision-making with reinforcement learning algorithms, enabling the measurement system to perceive fine-grained changes in network status. When faced with sudden traffic or link topology changes, the system can adaptively adjust the reporting strategy, ensuring extremely high congestion-free measurement data protection capability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a network traffic measurement method based on Sketch-INT collaboration provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of solving the measurement points provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the congestion-free measurement data reporting path selection and diversion provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0020] like Figure 1 As shown, this embodiment of the invention provides a network traffic measurement method based on Sketch-INT collaboration. Its core idea is "divide and conquer," that is, using Sketch to efficiently statistically analyze the first type of network flow (i.e., large flows) to reduce collision errors, and using INT to collect the second type of network flow (i.e., small flows) packet-by-packet and hop-by-hop to obtain high-precision, fine-grained information. Furthermore, it achieves measurement point selection and congestion-free reporting even when routing information is incomplete, thus balancing measurement accuracy, resource efficiency, and reporting reliability. The method includes the following steps: Step S1: Obtain the topology information, programmable switch set, and control plane node set of the network under test, and obtain the source-destination pairs corresponding to the network flow under test; construct a candidate forwarding path set for each network flow based on the source-destination pairs.

[0021] Specifically, step S1 includes the following steps: Step S101: Obtain the topology information G=(V,E), the set of programmable switches P, and the set of control plane nodes C of the network under test, and obtain the source-destination pairs corresponding to each flow in the network under test. .

[0022] Specifically, obtain the topology information G=(V,E) of the network under test, where V is the set of network nodes and E is the set of links; obtain the set of programmable switches. The programmable switch supports the deployment of Sketch and INT; and the acquisition of the control plane node set. .

[0023] The control plane obtains the set F of network flows to be tested, and obtains the source-destination pair corresponding to each network flow f∈F. , Indicates the source node corresponding to network flow f. This represents the destination point corresponding to network flow f.

[0024] At the same time, initialize the distance metric. p represents a programmable switch node, and c represents a control plane node; initialize distance tradeoff parameters. ∈[0,1], and the iteration stopping threshold or maximum number of iterations, etc.

[0025] Step S102: When routing information is incomplete (e.g., the forwarding path of the network flow is uncertain), for each source-destination pair of the network flow to be tested... The connection source point is calculated. and destination At least one shortest path between them; determine the set of programmable switches located on any shortest path as candidate forwarding paths P for the network flow. f .

[0026] Therefore, even if the exact actual forwarding table or complete route cannot be obtained in advance, this embodiment can still characterize the range of candidate measurement points that the flow may pass through by the method of "source-destination pair + shortest path candidate".

[0027] Step S2: Set constraints; construct a multi-objective optimization model with the goal of maximizing flow coverage and minimizing the distance from the measurement point to the control surface; solve the multi-objective optimization model based on the constraints using Lagrange relaxation and subgradient optimization to obtain the set of measurement points and the assignment relationship between the measurement points and the control surface nodes; and collaboratively deploy the data digest Sketch measurement program and the in-band network telemetry INT measurement program on the switches corresponding to the measurement points.

[0028] Furthermore, the constraints include: First constraint: When network flow f is covered, there must be at least one programmable switch node serving as a measurement point on the forwarding path of network flow f; that is, when W f When =1, there exists at least p∈P f Make u p =1; and, Second constraint: When a programmable switch node is selected as a measurement point, there must exist at least one control plane node such that the measurement point is assigned to a control plane node; that is, u p When z = 1, there exists at least one c ∈ C such that zp,c =1, to ensure that the data from each measurement point can be assigned to the control plane for reception.

[0029] Furthermore, a multi-objective optimization model is constructed with the objectives of maximizing flow coverage and minimizing the distance from the measurement point to the control surface, as shown in the following expression: In the formula, Indicates the distance tradeoff parameter. ∈{0,1} indicates whether network flow f is covered, u p ∈{0,1} indicates whether the programmable switch is selected as a measurement point, z p,c ∈{0,1} indicates whether the measurement point is assigned to a control plane node, F represents the set of network flows to be measured, p represents a programmable switch node, P represents the set of programmable switches, c represents a control plane node, and C represents the set of control plane nodes. Represents a distance metric.

[0030] Furthermore, the process of obtaining the set of measurement points and the assignment relationship between measurement points and control surface nodes by solving the multi-objective optimization model based on Lagrange relaxation and subgradient optimization includes: This embodiment proposes a joint solution scheme based on Lagrangian relaxation and subgradient optimization to address the network measurement point selection and assignment problem with NP-hard characteristics. The scheme employs a master-slave architecture, iteratively updating the "flow coverage penalty coefficient" (i.e., the Lagrange multiplier λ) to minimize deployment costs while satisfying flow coverage constraints.

[0031] The solution process is mainly completed by two core algorithms working together: Algorithm 2 (the main algorithm) is responsible for the subgradient optimization of the dual problem and controls the global penalty coefficient; Algorithm 1 (the slave algorithm) is responsible for solving the Lagrangian function under the given penalty coefficient and outputting the specific measurement point selection scheme.

[0032] Figure 2 This is a schematic diagram of the solution process, which can be viewed as a closed-loop feedback control system. Feedback loop: Algorithm 2 (controller) sends a price signal λ, Algorithm 1 (actuator) calculates a local optimum based on the price, Algorithm 1 provides feedback on the coverage result, and Algorithm 2 adjusts the price based on the deviation. Convergence mechanism: Through continuous signal (λ) and response (u, w), the system eventually finds an approximately optimal balance between "maximizing flow coverage" and "minimizing distance cost".

[0033] Specifically, it involves the following stages: 1. Initialization phase: The main algorithm initializes the Lagrange multiplier vector λ (representing the "implicit price" of the uncovered flow).

[0034] 2. Parameter distribution: The main algorithm distributes the current multiplier λ. t Send it to the algorithm.

[0035] 3. Solving subproblems: The algorithm solves the subproblems based on the received λ. t Calculate the potential benefits and costs of each switch and make an activation decision (u p ) and assignment decision ( ), and provide feedback on the current flow coverage status ( ).

[0036] 4. Gradient Update: The main algorithm calculates the subgradient based on the feedback coverage result, evaluates the deviation between the current solution and the constraints, and updates the multiplier λ for the next round accordingly. t+1 .

[0037] 5. Iterative loop: Repeat the above process until the maximum number of iterations is reached or the solution converges.

[0038] Specifically, the solution process is as follows: Step S3: Based on the real-time traffic distribution, the network traffic is divided into Class I network traffic and Class II network traffic; the Class I network traffic is measured using the Data Digest Sketch measurement program, and the Class II network traffic is measured using the In-Band Network Telemetry INT measurement program.

[0039] Specifically, set the flow threshold θ; At each measurement point, the number of data packets or bytes of each network flow is statistically analyzed based on the flow key. When the statistical value is greater than the flow threshold θ, the current network flow is classified as a first-class network flow, i.e., a large flow; when the statistical value is less than the flow threshold θ, the current network flow is classified as a second-class network flow, i.e., a small flow.

[0040] For the first type of network flow, the data digest Sketch measurement program is used for measurement. The flow key is extracted and updated by hash mapping to the Sketch counter matrix to obtain the packet / byte count estimate of the network flow. For the second type of network flow, the in-band network telemetry INT measurement program is used for measurement. The INT instruction is embedded in the packet header and information such as switch identification, enqueue / dequeue time, and queue depth are recorded along the way, so that the control plane can obtain high-precision fine-grained path and delay information of the second type of network flow.

[0041] Furthermore, when a network flow is not marked as a first-class or second-class network flow, both the data digest Sketch measurement procedure and the in-band network telemetry INT measurement procedure are executed. When a network flow is marked as a first-class network flow, INT telemetry acquisition for that network flow is stopped, and only Sketch updates are executed; and a timeout recovery or periodic reset mechanism is maintained for the marking in the data plane to ensure that the state of large and small flows is released or reset when the flow ends or the cycle rotates.

[0042] Step S4: Model the dynamic path selection problem as a constrained Markov decision process and train the policy network based on reinforcement learning.

[0043] Specifically, step S4 includes the following steps: Step S401: Estimate the worst-case sending rate reported by Sketch.

[0044] Specifically, for Sketch measurement data, the worst-case transmission rate for Sketch reporting is determined by the ratio of Sketch's storage size to the reporting window length. Let the storage size of Sketch k at a given measurement node be... The reporting window is The expression for the worst-case sending rate reported by Sketch is as follows: In the formula, This indicates that Sketch k reports the worst-case sending rate.

[0045] Step S402: Estimate the worst-case transmission rate reported by INT.

[0046] For INT measurement data, the worst-case transmission rate for INT reporting is determined based on link capacity, the maximum bandwidth consumption by small flows, the average packet length of small flows, and the INT header size per packet. Let the relevant link capacity at the measurement point be... The maximum bandwidth ratio of small streams is The average message length of a small stream is The size of the INT header for each packet is [size missing]. The expression for INT reporting the worst transmission rate is as follows: In the formula, This indicates the worst-case transmission rate reported by INT.

[0047] Step S403: Sum the worst-case transmission rates reported by Sketch and INT deployed at the same measurement point to obtain the total worst-case transmission rate of measurement data for that measurement point, expressed as follows: In the formula, The Sketch collection deployed at measurement point p.

[0048] Step S404, given the rate estimate The dynamic path selection problem is modeled as a constrained Markov decision process (CMDP) to avoid network path saturation.

[0049] Specifically, the optimization objectives and constraints are as follows: in, The queue length (in bytes) on link e at time t is used to reflect the current congestion level of the link. This indicates the splitting ratio of the measurement data stream from programmable switch node p to control plane node c; This represents the data plane traffic on link e (i.e., the original user background traffic). This represents the safety threshold for link e (e.g., 80% of link capacity), used to reserve a margin to avoid congestion; This indicates the bandwidth capacity of link e; The measurement data flow on link e is represented by the following formula: in, For the indicator function, if link e belongs to the selected path from programmable switch node p to control plane node c. If the value is 1, then the value is 1; otherwise, it is 0.

[0050] This example trains a policy network. This problem will be solved. Specifically, this example uses a simulation environment that defines the following components: The state is defined as: In the formula, For link utilization, The queue length (in bits) on link e at time t. To measure the data transmission rate.

[0051] Action is defined as: (The set of paths from measurement point p to control node c) and (The proportion of traffic splitting in the path set).

[0052] The reward function is designed as follows: In the formula, the first term The second term is used to minimize network congestion (queue length). This is a penalty for violating link security threshold constraints (when traffic exceeds the threshold). hour ), This represents the weighting coefficient.

[0053] In the above environment, this method uses Constrained Policy Optimization (CPO) to train the policy network. Specifically, this method maximizes the reward while employing Lagrange relaxation to satisfy the problem's constraints. Once training converges, the policy model is deployed to the system for real-time dynamic inference.

[0054] Step S5, based on the trained policy network Based on the status, an action is selected to send the Sketch summary data and INT telemetry data of each measurement point to the control plane; wherein, the status includes at least the link utilization, the link queue length, and the total worst transmission rate of the measurement data of each measurement point; the action includes at least the reporting path selected from the candidate forwarding path set under the link capacity safety threshold and its corresponding traffic offloading ratio.

[0055] Step 501, Initialization: Calculate the k shortest paths between all programmable switch nodes p and control plane nodes c as a candidate path set. And initialize the split ratio to a uniform distribution (i.e. ).

[0056] Step 502, State Construction: At each time step t, collect state information from the entire network to construct a state vector. : Step 503: Action Selection: Change State Input the trained policy network Output action That is, a set of definite paths and the original diversion ratio .

[0057] Step 504: Security Constraint Execution: Calculate the expected measurement traffic for each link. And according to the safety threshold Adjust the diversion ratio : This step ensures that measuring traffic does not cause the total link traffic to exceed [a certain limit]. Subsequently, on Normalize.

[0058] Step 505: Deployment: Send the final selected path and traffic splitting ratio to the data plane switch for execution.

[0059] like Figure 3 As shown, Figure 3 (a) shows the network topology and capacity settings, with each link having a capacity of 10 and a congestion safety threshold of 80% (i.e., 8). Figure 3 (b) shows the network traffic distribution and node roles, where p2 and p3 are measurement nodes that deploy Sketch and need to report data to the control plane (connected to p2); Figure 3 (c) shows the initial path selection scheme. Node p3 directly sends measurement data at a rate of 2 through the p3→p2 link. Since there is already service data at a rate of 7 on this link, the total rate reaches 9, which exceeds the threshold of 8, resulting in congestion. Figure 3 (d) in the diagram shows the optimized path selection of this system. The data of node p3 is detoured through p1 (p3→p1→p2). At this time, the p1→p2 link aggregates the measurement data of p1 (rate 3) and p3 (rate 2). The total load is 5, which is still below the threshold after adding the background traffic, thus successfully avoiding congestion.

[0060] Step S6: The control plane receives and aggregates Sketch summary data and INT telemetry data for each measurement point; it monitors the traffic distribution in real time and performs iterative optimization.

[0061] Specifically, the control plane receives and aggregates Sketch summary data (for the first type of network flow statistics) and INT telemetry data (for the second type of network flow and queue status) from multiple measurement points, and uses flow identifiers (Flow IDs) to associate the multi-source data to reconstruct the entire network flow view.

[0062] Furthermore, this example also constructs a feedback-based dynamic update mechanism, which transforms control plane optimization decisions into specific rules for the data plane through a standardized southbound interface (such as P4Runtime). The specific implementation process is as follows: 1. Performance monitoring and trigger determination: Coverage monitoring: Control plane analysis of application layer feedback indicates the presence of traffic with unknown paths or newly emerging source-destination pairs not included in the existing set of measurement points. Coverage (i.e., covering indicator variables) If the proportion of traffic in this part exceeds the preset threshold, it is determined to be insufficient coverage, and the measurement point reselection is triggered.

[0063] Congestion awareness: Real-time monitoring of link utilization on reporting paths. If a link exceeds the security threshold for N consecutive time slices... Or receive the queue depth reported by INT. If an alarm is triggered, it is determined to be "reporting congestion" and the reporting path is replanned.

[0064] Accuracy calibration: Compare the Sketch estimate with the precise value of the sampled INT. If the false positive / false negative rate of the large-scale flow detection increases, it indicates that the threshold for determining the large-scale flow needs to be adjusted.

[0065] 2. Strategy recalculation: Measurement point update: The Lagrange relaxation algorithm solver is invoked, taking the current network topology G and the updated flow set F as input, and outputting a new measurement point deployment scheme (i.e., update) in polynomial time. , state).

[0066] Path and traffic splitting update: update the current network state Input the pre-trained reinforcement learning policy network (including link load and queue depth). Output a new set of reporting paths. and diversion ratio .

[0067] 3. Incremental deployment and distribution: Utilize P4Runtime's WriteRequest message to distribute updates to the data plane via the gRPC channel, ensuring that the update process does not interrupt normal business operations.

[0068] Example 1 This embodiment provides a large and small traffic orchestration and measurement system implemented on the control plane of a programmable switch and a server, to illustrate the deployment method and effects of the method of the present invention in a real network.

[0069] This embodiment implements the method of the present invention in a network environment consisting of multiple programmable switches. The data plane uses 32×400 Gbps programmable switches based on the Intel Barefoot Tofino2 chip, and the switches are connected by 100 Gbps links to form a two-level Fat-tree data center topology. The control plane consists of several high-performance servers, each equipped with a multi-core CPU and large-capacity memory, used to run optimization algorithms and network management applications. In a wide area network scenario, the applicability of the method of the present invention in networks of different sizes can be verified by simulating real WAN topologies and actual traffic such as CAIDA.

[0070] Regarding the switch data plane, this embodiment uses the P4 language to implement the data structure and algorithm logic of this invention. Specifically, a Sketch measurement program supporting various Sketch structures (e.g., Count-Min Sketch, Count Sketch, Elastic Sketch, UnivMon, etc.) and an INT measurement program supporting in-band network telemetry are deployed on each programmable switch. The P4 program includes logic for enabling or disabling Sketch / INT according to the configuration issued by the control plane, maintaining pipeline entries for flow-by-flow statistical counting, performing hash mapping and updating the counter matrix, inserting or stripping INT headers into packets, and encapsulating INT reports before sending them to the control plane. This embodiment can provide two versions of the P4 implementation for different generations of switch chips: one using... The syntax is designed to be compatible with existing devices, while another uses... The syntax is adapted to the latest switch hardware.

[0071] In terms of control plane and optimization algorithm implementation, this embodiment implements the measurement point selection and congestion-free reporting path planning functions of the present invention on a server. The measurement point selection part is implemented in C++, and uses the mathematical programming library to call the Lagrange relaxation and subgradient iteration algorithm to solve the multi-objective optimization model with coverage and distance to the control plane as the objectives, thereby obtaining an approximately optimal set of measurement points and their control plane assignment under the condition of incomplete routing information. The congestion-free reporting part periodically counts parameters such as the Sketch size, reporting window, link capacity, small flow bandwidth ratio, and average packet length of each measurement point in the control plane, calculates the worst-case transmission rate of Sketch and INT reporting, and summarizes the total worst-case transmission rate of each measurement point. Based on this, the control plane selects the reporting path according to the link safety threshold and calculates the traffic splitting ratio. If necessary, a reinforcement learning framework can be used to perform offline training and online inference on the path and traffic splitting strategy, so as to achieve dynamic rerouting and load balancing while ensuring that the measurement data does not exceed the available bandwidth of the link.

[0072] In terms of controller software, this embodiment runs a centralized controller on the control plane server. This controller is responsible for converting the measurement point selection results, surge thresholds, Sketch parameters, INT sampling ratios, reporting paths, and splitting ratios output by the optimization algorithm into configurations that can be understood by the switch. It then distributes the corresponding entries and rules to the data plane through the programmable switch's device development environment. The controller also receives Sketch summaries and INT reports from each measurement point, aggregating them into a unified data storage or stream processing framework. This provides input for various network management applications such as hit flow detection, super-spreader detection, DDoS flow identification, entropy estimation, and micro-burst detection.

[0073] During system operation, this embodiment executes the method of the present invention according to the aforementioned steps. First, the control plane acquires the network topology, the set of programmable switches, the set of control plane nodes, and the source-destination pair information of network flows. It constructs a candidate set of measurable switches based on the shortest path set and solves a multi-objective optimization model using Lagrange relaxation to obtain the set of measurement points and their assignment results to control plane nodes. Subsequently, Sketch and INT measurement programs are collaboratively deployed on the selected measurement points, and flow states for identifying large and small flows are maintained in the data plane. When traffic arrives, the measurement points dynamically divide the flow into large and small flows based on thresholds. Sketch is used for statistical measurement of large flows, while INT is used for packet-by-packet and hop-by-hop information collection of small flows. For flows switching from small to large flows, only Sketch measurement is retained to control INT overhead. Based on this, the control plane dynamically adjusts the large flow threshold, Sketch parameters, and INT sampling and reporting strategies in conjunction with resource constraints such as the memory, processing capacity, and link bandwidth of the measurement points. It also plans the measurement data reporting path and splitting ratio through worst-case transmission rate estimation and link security threshold constraints to ensure that the superposition of measurement data and service data does not exceed the available link bandwidth. Finally, the control plane analyzes the aggregated Sketch summary and INT report, identifies various abnormal events, and iteratively updates the measurement point selection, threshold, and reporting strategy to achieve closed-loop measurement orchestration.

[0074] In the above implementation environment, compared with traditional solutions that only use Sketch or only use INT, this embodiment can significantly improve detection accuracy in various applications such as hit flow detection, super spreader detection, DDoS flow identification, and aggregation index estimation. At the same time, it keeps the bandwidth and processing overhead brought by INT within an acceptable range, and maintains a low measurement data loss rate and reporting latency even under high load or traffic burst scenarios.

[0075] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the network traffic measurement method based on Sketch-INT collaboration as described above. Figure 4 The diagram shown illustrates a hardware structure of any data processing-capable device for implementing the network traffic measurement method based on Sketch-INT collaboration provided in this embodiment of the invention, except... Figure 4 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0076] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the network traffic measurement method based on Sketch-INT collaboration as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network traffic measurement method based on Sketch-INT collaboration, characterized in that, The method includes: Obtain the topology information, programmable switch set, and control plane node set of the network under test, and obtain the source-destination pairs corresponding to the network flows under test; construct a set of candidate forwarding paths for each network flow based on the source-destination pairs; Set constraints; construct a multi-objective optimization model with the objectives of maximizing flow coverage and minimizing the distance from the measurement point to the control plane; solve the multi-objective optimization model based on the constraints using Lagrange relaxation and subgradient optimization to obtain the set of measurement points and the assignment relationship between the measurement points and the control plane nodes; and collaboratively deploy the data digest Sketch measurement program and the in-band network telemetry INT measurement program on the programmable switch corresponding to the measurement point. Based on real-time traffic distribution, network traffic is divided into Class I network traffic and Class II network traffic. The Class I network traffic is measured using the Data Digest Sketch measurement program, and the Class II network traffic is measured using the In-Band Network Telemetry INT measurement program. Based on the trained policy network, actions are selected according to the state, thereby sending the Sketch summary data and INT telemetry data of each measurement point to the control plane. The state includes link utilization, link queue length, and the total worst transmission rate of measurement data for each measurement point. The actions include the reporting path selected from the candidate forwarding path set under the link capacity safety threshold and its corresponding traffic offloading ratio.

2. The network traffic measurement method based on Sketch-INT collaboration according to claim 1, characterized in that, The process of constructing a set of candidate forwarding paths for each network flow based on the source and destination includes: In the case of incomplete routing information, for each source-destination pair of the network flow under test, at least one shortest path connecting the source and destination is calculated. The set of programmable switches located on any shortest path is determined as the candidate forwarding path for the network flow.

3. The network traffic measurement method based on Sketch-INT collaboration according to claim 1, characterized in that, The constraints include: First constraint: When network flow f is covered, there must be at least one programmable switch node serving as a measurement point on the forwarding path of network flow f; and, Second constraint: When a programmable switch node is selected as a measurement point, there must be at least one control plane node such that the measurement point is assigned to a control plane node.

4. The network traffic measurement method based on Sketch-INT collaboration according to claim 1, characterized in that, The expression for the multi-objective optimization model is as follows: ； In the formula, Indicates the distance tradeoff parameter. ∈{0,1} indicates whether network flow f is covered, z p,c ∈{0,1} indicates whether the measurement point is assigned to a control plane node, F represents the set of network flows to be measured, p represents a programmable switch node, P represents the set of programmable switches, c represents a control plane node, and C represents the set of control plane nodes. Represents a distance metric.

5. The network traffic measurement method based on Sketch-INT collaboration according to claim 1, characterized in that, The process of dividing network traffic into Class I and Class II network flows based on real-time traffic distribution, measuring Class I network flows using the Data Digest Sketch measurement program, and measuring Class II network flows using the In-Band Network Telemetry INT measurement program includes: At each measurement point, the number of data packets or bytes of each network flow is statistically analyzed based on the flow key. When the statistical value is greater than the flow threshold θ, the current network flow is classified as a first-class network flow; when the statistical value is less than the flow threshold θ, the current network flow is classified as a second-class network flow. For the first type of network flow, the data digest Sketch measurement program is used for measurement. The flow key is extracted and updated by hash mapping to the Sketch counter matrix to obtain the packet / byte count estimate of the first type of network flow. For the second type of network flow, the in-band network telemetry INT measurement program is used for measurement. The INT instruction is embedded in the packet header and information including switch identification, enqueue / dequeue time, and queue depth is recorded along the way, so that the control plane can obtain the path and delay information of the second type of network flow.

6. The network traffic measurement method based on Sketch-INT collaboration according to claim 1, characterized in that, The process of calculating the worst-case transmission rate of measurement data at each measurement point includes: The worst-case sending rate reported by Sketch is calculated using the following expression: ； In the formula, This indicates that Sketch reported the worst-case sending rate. This indicates the size of the Sketch file. This indicates the reporting window; The worst-case transmission rate for INT reporting is calculated using the following expression: ； In the formula, This indicates that INT reports the worst-case transmission rate. Indicates the link capacity related to the measurement point. This indicates the maximum bandwidth percentage of the second type of network flow. This indicates the average message length of the second type of network flow. Indicates the size of the INT header for each packet; The worst-case transmission rate of the measurement data at the current measurement point is obtained by summing the worst-case transmission rate of the Sketch data reported at the current measurement point and the worst-case transmission rate of the INT data reported at the current measurement point, as shown in the following expression: ； In the formula, The Sketch collection deployed at measurement point p.

7. The network traffic measurement method based on Sketch-INT collaboration according to claim 1, characterized in that, The method further includes: If an unknown forwarding path or a newly emerging source-destination pair is discovered that is not covered by the existing set of measurement points, the set of measurement points is reset; and / or, Real-time monitoring of link utilization on the reporting path; if a link exceeds the link capacity safety threshold or receives an INT-triggered link queue length prepayment, the reporting path is replanned; and / or, Determine whether the first type of network flow detection is a false positive or a false negative, and adjust the flow threshold θ accordingly.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more of the computer programs being executed by the at least one processor to enable the at least one processor to perform the network traffic measurement method based on Sketch-INT collaboration as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the network traffic measurement method based on Sketch-INT collaboration as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the network traffic measurement method based on Sketch-INT collaboration as described in any one of claims 1-7.

Images