Large-scale network delay measurement method based on KNN matrix completion

Abstract

The invention discloses a large-scale network delay measurement method based on KNN matrix completion. The large-scale network delay measurement method comprises the following steps: 1) forming a delay monitoring system; 2) the control points collect topology and delay information of the whole network; 3) the control point selects an SDN switch which needs to be directly measured; and 4) the control point predicts missing link delay. The method can be deployed in a single-controller single-domain SDN (Software Defined Network) or a multi-controller multi-domain SDN, the link delay of the whole network is speculated by measuring part of the link delay, the delay of all links or routes of the network is continuously monitored under the condition of not influencing the measurement index and the measurement result deviation, the overhead of delay measurement is reduced to the greatest extent, and the measurement efficiency is improved. And real-time and continuous measurement is provided for the delay of links and routes in a large-scale network.

Description

Technical Field

[0001] The present invention relates to a delay monitoring technology for large-scale networks, and in particular to a large-scale network delay measurement method based on KNN matrix completion. Background Art

[0002] Software-defined networking is a new type of network architecture that achieves centralized management and flexible programming by separating the control plane from the data plane of the network. With a centralized controller as the core, software-defined networking uniformly manages and configures network devices, provides a global view and dynamic optimization capabilities, and makes the network more intelligent, efficient, and scalable.

[0003] By utilizing the characteristics of software-defined networks, the controller of software-defined networks can directly serve as a control point for actively monitoring large-scale network delays. By sending delay measurement packets, recycling them, and collecting statistics, it can provide real-time and continuous delay monitoring for large-scale networks. Many delay measurement methods based on software-defined networks have emerged in current research. Most of these methods inject OpenFlow messages or delay detection packets into the data plane through the controller, and use various means to make the detection packets return to the controller after collecting link information on the data plane. The controller can monitor network delays through further processing. Although these methods are simple and easy to implement, most of these measurement methods do not consider the problems of multi-domain collaboration, cross-domain clock synchronization, and load balancing in the delay measurement of large-scale networks.

[0004] KNN-based matrix completion technology is an algorithm that uses similarity measurement to predict missing values. It calculates the similarity between matrix rows or columns, selects K most similar neighbors, and uses their known values ​​to weightedly estimate missing values ​​and fill the matrix. This method is intuitive and simple, has good local prediction capabilities, and performs well in solving sparse data problems. Using these characteristics of KNN-based matrix completion, the KNN-based completion method can make full use of the delay data of a small number of measured links to estimate the delay of other links, reducing resource consumption and improving the efficiency of network delay estimation. Summary of the invention

[0005] The purpose of the present invention is to provide a large-scale network delay measurement method based on KNN matrix completion in view of the deficiencies of the prior art. This method can be deployed in a single-controller single-domain or multi-controller multi-domain SDN, and measures the link delay of part of the link to infer the link delay of the entire network, and continuously monitors the delay of all links or routes in the network without affecting the measurement indicators and the deviation of the measurement results, thereby minimizing the overhead of delay measurement and providing real-time and continuous measurement of the delay of links and routes in large-scale networks.

[0006] The technical solution for achieving the purpose of the present invention is:

[0007] A large-scale network delay measurement method based on KNN matrix completion includes the following steps:

[0008] 1) Composition of the delay monitoring system: The delay monitoring system consists of a control point, a delay measurement package, and a network under test. The control point is composed of a controller modified from a software-defined network (SDN) based on the OpenFlow protocol. The control point expands the LLDP data packet into a delay measurement package. The network under test includes SDN switches and links.

[0009] 2) The control point collects the topology and delay information of the entire network: In order to determine the set of SDN switches that need to be directly measured, the control point uses the delay measurement package to collect the topology and link delay of the entire network in the first round of measurement;

[0010] 3) The control point selects the SDN switches that need to be directly measured: After mastering the topology and delay information of the entire network, the control point takes minimizing the delay prediction error, control channel bandwidth occupancy, and controller CPU consumption as the optimization goal, establishes a three-objective simultaneous optimization equation, and solves the equation through a genetic algorithm to obtain the set of SDN switches that need to be directly measured. In subsequent measurements, the control point injects delay measurement packets into the set of SDN switches that need to be directly measured, and injects ordinary LLDP data packets into the remaining set of SDN switches;

[0011] 4) The control point predicts the missing link delay: The control point constructs a link delay matrix based on the collected network topology information. The control point records the link delays collected through direct measurement in the delay matrix, uses ∞ to represent the link delays between non-adjacent nodes and the diagonal of the delay matrix, and the delays of links that are not directly measured are used as missing values ​​of the delay matrix. Missing values ​​are represented by 0 in the delay matrix. Finally, the control point calculates the predicted missing delay values ​​based on the existing delay measurement values ​​through a KNN-based matrix completion method.

[0012] In step 3), the control point selects the SDN switch that needs to be directly measured, including the following steps:

[0013] 3.1) Establishing a three-objective simultaneous optimization equation: In order to determine the set of SDN switches that need to be measured directly, in the first round of measurement, the control point uses a delay measurement packet to collect the topology and link delay of the entire network. Then, the control point takes minimizing the delay prediction error, the bandwidth occupancy of the control channel, and the CPU consumption of the controller as the optimization goal, and establishes a three-objective simultaneous optimization equation, where the delay prediction error is measured by the average error between the predicted value and the measured value of the delay; the bandwidth occupancy of the control channel is measured by the ratio of the total size of all delay data packets in a single measurement to the maximum bandwidth of the control channel; the CPU consumption of the controller is measured by the ratio of the time to process missing values ​​in a single measurement to the measurement period;

[0014] 3.2) Solving the three-objective simultaneous optimization equation: The three-objective simultaneous optimization equation is solved using a genetic algorithm. The set of SDN switches that need to be directly measured is taken as a chromosome, and each chromosome is an individual in the population. The initial population is randomly generated, and a new generation of population is iteratively generated by crossover mutation. In each iteration, the genetic algorithm inputs the SDN switch set in the form of chromosomes, calculates the fitness of each chromosome, and then selects the chromosome with the smallest fitness value as the optimal solution of the three-objective simultaneous optimization equation, that is, the set of SDN switches that need to be directly measured;

[0015] 3.3) Perform incomplete measurement: The control point sends a delay measurement packet to the SDN switch set that needs to be directly measured, and sends a normal LLDP data packet to the remaining SDN switch set. The control point collects the network topology of the entire network and performs incomplete measurement of the network delay;

[0016] 3.4) Update the set of SDN switches that need to be directly measured: When the control point completes multiple network delay measurements or when a significant change in link delay is found, the control point re-executes steps 3.1), 3.2) and 3.3) to update the set of SDN switches that need to be directly measured.

[0017] In step 4), the control point predicts the missing link delay, including the following steps:

[0018] 4.1) Build a link delay matrix: Based on the network delay information collected by incomplete measurement, build an n×n square matrix, where N is the total number of SDN switches in the network and M is ij is the link delay between SDN switch i and SDN switch j. When the delay of the link that is not directly measured in the incomplete measurement is the missing value of the delay matrix, it is initialized with 0 in the delay matrix. The link delay between non-adjacent nodes and the diagonal of the delay matrix are represented by ∞.

[0019] 4.2) Preprocessing the delay matrix: Before completing the delay matrix, the delay matrix needs to be preprocessed, that is, excluding the links that have been measured at least once. Assume that SDN switches i and j are adjacent. In the incomplete measurement, SDN switch i is not injected with the delay measurement packet by the control plane. Then the link delay from SDN switch i to SDN switch j is a missing value, that is, M ij =0, but the control plane determines the link delay M from SDN switch j to SDN switch i by injecting a delay measurement packet into SDN switch j. ji , let M ij =M ji ,At this time, the missing values ​​in the delay matrix after preprocessing only include the delay of the unmeasured link;

[0020] 4.3) Predict missing values ​​by completing the KNN matrix: In units of rows, the columns with known values ​​in two rows of the delay matrix are called shared features between the two rows, that is, missing values ​​and ∞ are ignored. The mean square error of the shared features is recorded as the distance between the two rows. Then find the k rows with the shortest distance as samples, and take the corresponding column average in the samples as the prediction of the missing value.

[0021] The technical problems mainly solved by this technical solution are as follows:

[0022] 1. This technical solution transforms the controller of the software-defined network control plane into a control point for delay measurement. The control point provides real-time and continuous delay monitoring for large-scale networks by generating, sending, recycling and decoding delay measurement packets;

[0023] 2. After the control point collects the topology and link delay of the entire network, this technical solution takes minimizing the delay prediction error, control channel bandwidth occupancy rate and controller CPU consumption as the optimization goal, establishes a three-objective simultaneous optimization equation, and solves the equation through a genetic algorithm to obtain the set of SDN switches that need to be directly measured. And when the control point completes multiple network delay measurements or when it finds that the link delay has changed significantly, it recalculates the set of SDN switches that need to be directly measured;

[0024] 3. After collecting the delays of the set of SDN switches directly measured at the control point, this technical solution builds a link delay matrix according to the network topology and delay conditions, and then uses the KNN-based matrix completion method to complete the missing values ​​of the link delay. The KNN-based matrix completion method mines potential correlations from limited measurement data by calculating the similarities between links. KNN will find K measured links with similar characteristics for each unmeasured link, and use the known delay data of these neighbors for weighted predictions to complete the missing parts of the matrix. KNN utilizes the local characteristics of the data and can more accurately capture the correlation between links, thereby improving prediction accuracy and providing reliable support for subsequent network optimization;

[0025] 4. This technical solution uses the extended LLDP data packet as a delay measurement data packet, and uses the extended LLDP data packet to collect the delay information of the network link, which avoids the need to create a special delay measurement packet for traditional active measurement and the impact of additional overhead on network resources, and does not affect its original function of discovering network topology;

[0026] 5. This technical solution supports general open source or commercial OpenFlow controllers. The SDN switch can be an open source or commercial software or physical SDN switch. The recommended improvement to the controller is based on the existing standard network topology discovery module to achieve comprehensive implementation of delay measurement and topology discovery, so as to further reduce the bandwidth consumption of delay measurement.

[0027] This method makes full use of existing measurement data, effectively reduces measurement resource consumption, and provides flexible similarity modeling and high-precision prediction capabilities, which can find the best balance between prediction accuracy and resource consumption. In practical applications, this method can be deployed in SDN with a single controller and a single domain or multiple controllers and multiple domains, which can help administrators implement network-wide delay monitoring and optimize measurement resource overhead in large-scale networks, ensuring the overall performance of the network and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural diagram of a delay measurement package in an embodiment;

[0029] Figure 2 A flowchart for confirming a set of SDN switches that need to be directly measured in an embodiment;

[0030] Figure 3 A flowchart of predicting link delay missing values ​​in an embodiment;

[0031] Figure 4 is a linear graph comparing the embodiment with the traditional Ping method, wherein Figure 4a is a linear graph of the results of measuring intra-domain delay in the embodiment when the link delay is set to 0.05, 0.1, 0.3, and 0.5 ms and the traditional Ping method, Figure 4 b is a linear graph of the results of measuring intra-domain delay in the embodiment when the link delay is set to 1, 5, and 10 ms and the traditional Ping method, Figure 4 c is a linear graph of the results of measuring cross-domain delay in the embodiment with the cross-domain link delay set to 0.15, 0.3, 0.9, and 1.5 ms and the traditional Ping method, Figure 4 d is a linear graph of the results of measuring the cross-domain delay using the embodiment and the traditional Ping method when the cross-domain link delay is set to 3, 15, and 30 ms;

[0032] Figure 5 It is a line graph showing the comparison results of the computing bandwidth of the control channel required to be occupied on the full fat tree topology with 24, 48, 72 and 96 SDN switch ports compared with directly using the delay measurement packet to measure the whole network link delay;

[0033] Figure 6 The time required to complete the delay missing items in the basic tree network topology with 4, 8, 16, and 24 SDN switch ports in the embodiment (the total number of network SDN switches is 21, 73, 273, and 601, respectively). DETAILED DESCRIPTION

[0034] The content of the present invention is further described below in conjunction with the drawings and embodiments, but the present invention is not limited thereto.

[0035] Example:

[0036] A large-scale network delay measurement method based on KNN matrix completion includes the following steps:

[0037] 1) Composition of the delay monitoring system: The delay monitoring system consists of a control point, a delay measurement package, and a network under test, wherein the control point is composed of a controller modified from a software defined network (SDN) based on the OpenFlow protocol; Figure 1 As shown, the control point expands the LLDP data packet into a delay measurement packet; the network under test includes SDN switches and links;

[0038] 2) The control point collects the topology and delay information of the entire network: In order to determine the set of SDN switches that need to be directly measured, the control point uses the delay measurement package to collect the topology and link delay of the entire network in the first round of measurement;

[0039] 3) The control point selects the SDN switches that need to be directly measured: After mastering the topology and delay information of the entire network, the control point takes minimizing the delay prediction error, control channel bandwidth occupancy, and controller CPU consumption as the optimization goal, establishes a three-objective simultaneous optimization equation, and solves the equation through a genetic algorithm to obtain the set of SDN switches that need to be directly measured. In subsequent measurements, the control point injects delay measurement packets into the set of SDN switches that need to be directly measured, and injects ordinary LLDP data packets into the remaining set of SDN switches;

[0040] 4) The control point predicts the missing link delay: The control point constructs a link delay matrix based on the collected network topology information. The control point records the link delays collected through direct measurement in the delay matrix, uses ∞ to represent the link delays between non-adjacent nodes and the diagonal of the delay matrix, and the delays of links that are not directly measured are used as missing values ​​of the delay matrix. Missing values ​​are represented by 0 in the delay matrix. Finally, the control point calculates the predicted missing delay values ​​based on the existing delay measurement values ​​through a KNN-based matrix completion method.

[0041] like Figure 2 As shown, in step 3), the control point selects the SDN switch that needs to be directly measured, including the following steps:

[0042] 3.1) Establishing a three-objective simultaneous optimization equation: In order to determine the set of SDN switches that need to be measured directly, in the first round of measurement, the control point uses a delay measurement packet to collect the topology and link delay of the entire network. Then, the control point takes minimizing the delay prediction error, the bandwidth occupancy of the control channel, and the CPU consumption of the controller as the optimization goal, and establishes a three-objective simultaneous optimization equation, where the delay prediction error is measured by the average error between the predicted value and the measured value of the delay; the bandwidth occupancy of the control channel is measured by the ratio of the total size of all delay data packets in a single measurement to the maximum bandwidth of the control channel; the CPU consumption of the controller is measured by the ratio of the time to process missing values ​​in a single measurement to the measurement period;

[0043] 3.2) Solving the three-objective simultaneous optimization equation: The three-objective simultaneous optimization equation is solved using a genetic algorithm. The set of SDN switches that need to be directly measured is taken as a chromosome, and each chromosome is an individual in the population. The initial population is randomly generated, and a new generation of population is iteratively generated by crossover mutation. In each iteration, the genetic algorithm inputs the SDN switch set in the form of chromosomes, calculates the fitness of each chromosome, and then selects the chromosome with the smallest fitness value as the optimal solution of the three-objective simultaneous optimization equation, that is, the set of SDN switches that need to be directly measured;

[0044] 3.3) Perform incomplete measurement: The control point sends a delay measurement packet to the SDN switch set that needs to be directly measured, and sends a normal LLDP data packet to the remaining SDN switch set. The control point collects the network topology of the entire network and performs incomplete measurement of the network delay;

[0045] 3.4) Update the set of SDN switches that need to be directly measured: When the control point completes multiple network delay measurements or when a significant change in link delay is found, the control point re-executes steps 3.1), 3.2) and 3.3) to update the set of SDN switches that need to be directly measured.

[0046] like Figure 3 As shown, in step 4), the control point predicts the missing link delay, including the following steps:

[0047] 4.1) Build a link delay matrix: Based on the network delay information collected by incomplete measurement, build an n×n square matrix, where N is the total number of SDN switches in the network and M is ij is the link delay between SDN switch i and SDN switch j. When the delay of the link that is not directly measured in the incomplete measurement is the missing value of the delay matrix, it is initialized with 0 in the delay matrix. The link delay between non-adjacent nodes and the diagonal of the delay matrix are represented by ∞.

[0048] 4.2) Preprocessing the delay matrix: Before completing the delay matrix, the delay matrix needs to be preprocessed, that is, excluding the links that have been measured at least once. Assume that SDN switches i and j are adjacent. In the incomplete measurement, SDN switch i is not injected with the delay measurement packet by the control plane. Then the link delay from SDN switch i to SDN switch j is a missing value, that is, M ij =0, but the control plane determines the link delay M from SDN switch j to SDN switch i by injecting a delay measurement packet into SDN switch j. ji , let M ij =M ji ,At this time, the missing values ​​in the delay matrix after preprocessing only include the delay of the unmeasured link;

[0049] 4.3) Predict missing values ​​by completing the KNN matrix: In units of rows, the columns with known values ​​in two rows of the delay matrix are called shared features between the two rows, that is, missing values ​​and ∞ are ignored. The mean square error of the shared features is recorded as the distance between the two rows. Then find the k rows with the shortest distance as samples, and take the corresponding column average in the samples as the prediction of the missing value.

[0050] This example verifies the performance of this example by comparing it with the traditional Ping method commonly used by network administrators, such as Figure 4As shown, Figure 4 a and Figure 4 b are linear graphs of the results of the delay in the control domain measured by this example and the traditional Ping method. The link delay is set to 0.05, 0.1, 0.3, 0.5, 1, 5 and 10 ms respectively, and the link delay is increased every minute. Figure 4 a shows the measurement results when the link delay is set to 0.05, 0.1, 0.3, and 0.5 ms. Figure 4 b shows the measurement results when the link delay is set to 1, 5, and 10 ms; Figure 4 c and Figure 4 d are linear graphs of the results of measuring the cross-domain link delay using this example and the traditional Ping method. The cross-domain link delay is set to 0.15, 0.3, 0.9, 1.5, 3, 15, and 30 ms, respectively. The link delay is increased every minute. Figure 4 c shows the measurement results when the cross-domain link delay is set to 0.15, 0.3, 0.9, and 1.5 ms. Figure 4 d shows the measurement results when the cross-domain link delay is set to 3, 15, and 30ms; the results show that this example has better accuracy in measuring the delay within the control domain and across domains than the traditional Ping method, especially in low-latency application scenarios, this example can provide more accurate delay measurement results. In order to better reflect the advantages of this example in implementing delay monitoring in large-scale networks, Figure 5 It is a line graph showing the comparison results of the computing bandwidth of the control channel required to be occupied on the full fat tree topology with 24, 48, 72 and 96 SDN switch ports compared with directly using the delay measurement packet to measure the whole network link delay; Figure 6 The time required to complete the missing delay items in the basic tree network topology with 4, 8, 16, and 24 SDN switch ports (the total number of network SDN switches is 21, 73, 273, and 601, respectively); the results show that this example can implement network-wide delay monitoring in large-scale networks and optimize the measurement resource overhead, thereby ensuring the overall performance of the network and improving user experience.

Claims

1. A large-scale network delay measurement method based on KNN matrix completion, characterized in that: The steps include: 1) Composition of the delay monitoring system: The delay monitoring system consists of a control point, a delay measurement package, and a network under test. The control point is composed of a controller modified from a software-defined network based on the OpenFlow protocol. The control point expands the LLDP data packet into a delay measurement package. The network under test includes SDN switches and links. 2) The control point collects the topology and delay information of the entire network: In order to determine the set of SDN switches that need to be directly measured, the control point uses the delay measurement package to collect the topology and link delay of the entire network in the first round of measurement; 3) The control point selects the SDN switches that need to be directly measured: After mastering the topology and delay information of the entire network, the control point takes minimizing the delay prediction error, control channel bandwidth occupancy, and controller CPU consumption as the optimization goal, establishes a three-objective simultaneous optimization equation, and solves the equation through a genetic algorithm to obtain the set of SDN switches that need to be directly measured. In subsequent measurements, the control point injects delay measurement packets into the set of SDN switches that need to be directly measured, and injects ordinary LLDP data packets into the remaining set of SDN switches; 4) The control point predicts the missing link delay: The control point constructs a link delay matrix based on the collected network topology information. The control point records the link delays collected through direct measurement in the delay matrix, uses ∞ to represent the link delays between non-adjacent nodes and the diagonal of the delay matrix, and the delays of links that are not directly measured are used as missing values ​​of the delay matrix. Missing values ​​are represented by 0 in the delay matrix. Finally, the control point calculates the predicted missing delay values ​​based on the existing delay measurement values ​​through a KNN-based matrix completion method.

2. The large-scale network delay measurement method based on KNN matrix completion according to claim 1 is characterized in that: In step 3), the control point selects the SDN switch that needs to be directly measured, including the following steps: 3.1) Establishing a three-objective simultaneous optimization equation: In order to determine the set of SDN switches that need to be measured directly, in the first round of measurement, the control point uses a delay measurement packet to collect the topology and link delay of the entire network. Then, the control point takes minimizing the delay prediction error, the bandwidth occupancy of the control channel, and the CPU consumption of the controller as the optimization goal, and establishes a three-objective simultaneous optimization equation, where the delay prediction error is measured by the average error between the predicted value and the measured value of the delay; the bandwidth occupancy of the control channel is measured by the ratio of the total size of all delay data packets in a single measurement to the maximum bandwidth of the control channel; the CPU consumption of the controller is measured by the ratio of the time to process missing values ​​in a single measurement to the measurement period; 3.2) Solving the three-objective simultaneous optimization equation: The three-objective simultaneous optimization equation is solved using a genetic algorithm. The set of SDN switches that need to be directly measured is taken as a chromosome, and each chromosome is an individual in the population. The initial population is randomly generated, and a new generation of population is iteratively generated by crossover mutation. In each iteration, the genetic algorithm inputs the SDN switch set in the form of chromosomes, calculates the fitness of each chromosome, and then selects the chromosome with the smallest fitness value as the optimal solution of the three-objective simultaneous optimization equation, that is, the set of SDN switches that need to be directly measured; 3.3) Perform incomplete measurement: The control point sends a delay measurement packet to the SDN switch set that needs to be directly measured, and sends a normal LLDP data packet to the remaining SDN switch set. The control point collects the network topology of the entire network and performs incomplete measurement of the network delay; 3.4) Update the set of SDN switches that need to be directly measured: When the control point completes multiple network delay measurements or when a significant change in link delay is found, the control point re-executes steps 3.1), 3.2) and 3.3) to update the set of SDN switches that need to be directly measured.

3. The large-scale network delay measurement method based on KNN matrix completion according to claim 1 is characterized in that: In step 4), the control point predicts the missing link delay, including the following steps: 4.1) Build a link delay matrix: Based on the network delay information collected by incomplete measurement, build an n×n square matrix, where N is the total number of SDN switches in the network and M is ij is the link delay between SDN switch i and SDN switch j. When the delay of the link that is not directly measured in the incomplete measurement is the missing value of the delay matrix, it is initialized with 0 in the delay matrix. The link delay between non-adjacent nodes and the diagonal of the delay matrix are represented by ∞. 4.2) Preprocessing the delay matrix: Before completing the delay matrix, the delay matrix needs to be preprocessed, that is, excluding the links that have been measured at least once. Assume that SDN switches i and j are adjacent. In the incomplete measurement, SDN switch i is not injected with the delay measurement packet by the control plane. Then the link delay from SDN switch i to SDN switch j is a missing value, that is, M ij =0, but the control plane determines the link delay M from SDN switch j to SDN switch i by injecting a delay measurement packet into SDN switch j. ji , let M ij =M ji ,At this time, the missing values ​​in the delay matrix after preprocessing only include the delay of the unmeasured link; 4.3) Predict missing values ​​by completing the KNN matrix: In units of rows, the columns with known values ​​in two rows of the delay matrix are called shared features between the two rows, that is, missing values ​​and ∞ are ignored. The mean square error of the shared features is recorded as the distance between the two rows. Then find the k rows with the shortest distance as samples, and take the corresponding column average in the samples as the prediction of the missing value.

Images