Adaptive monitoring method of SDN network traffic based on reinforcement learning

Abstract

The present invention discloses a SDN network traffic adaptive monitoring method based on reinforcement learning. By adding a decision module in the controller, the method continuously interacts with the network environment according to the load of the network link and the controller, and adaptively monitors the network traffic without affecting the network performance, so as to balance the monitoring frequency and granularity, monitor the network status at a relatively low cost, and timely adjust the monitoring scheme when the network is abnormal, provide the required information for attack detection and other methods, so that the network administrator can better deal with the network abnormality.

Description

Technical Field

[0001] The present invention relates to the technical field of network traffic monitoring, and in particular to a SDN network traffic adaptive monitoring method based on reinforcement learning. Background Art

[0002] SDN is a new network architecture that has emerged in recent years. It separates the control layer and data layer of network devices. This not only facilitates the centralized management of the network, but also makes the SDN switch only responsible for forwarding data packets according to the flow table rules. It is a strong candidate for the next generation Internet architecture.

[0003] For any network operator, monitoring network traffic is essential. Network traffic monitoring is a technology that provides reference for network management and improves network performance through analysis, evaluation, detection and prediction. Fine-grained traffic measurement is critical for many applications, such as attack detection, QoS routing, anomaly detection and load balancing. With an accurate global view, SDN can provide flexible traffic measurement by using direct flow or switch port, making it easier to access fine-grained traffic monitoring data.

[0004] SDN network traffic measurement methods can be divided into query-based collection methods and push-based collection methods. The push-based collection method usually pushes the statistical information of each flow table to the server after each flow table of the switch expires, but this method has poor real-time performance and cannot meet the needs of other network services. The push-based collection method usually uses the controller to periodically poll the information of each flow table on the switch. This method will cause a large network overhead, and when the control plane publishes a large number of downstream messages to the SDN switch, it may damage the establishment of new flows or even interrupt the connection between the SDN controller and the switch. Therefore, it is necessary to balance the frequency and granularity of monitoring. Summary of the invention

[0005] The purpose of the present invention is to provide a SDN network traffic adaptive monitoring method based on reinforcement learning, aiming to solve the technical problem in the prior art of balancing the frequency and granularity of monitoring.

[0006] To achieve the above object, the present invention adopts a SDN network traffic adaptive monitoring method based on reinforcement learning, which includes the following steps:

[0007] S1: The controller sends an initial traffic detection request to monitor the traffic of each port of each switch to obtain network traffic information;

[0008] S2: Extract the characteristic values ​​of the network traffic information returned by the switch, combine them into the current state and pass them to the decision module, and store historical data in the database;

[0009] S3: Selecting an action to be executed from an action set according to the current network state and strategy;

[0010] S4: After executing the selected action, modify the current network state and subsequent network states according to the new strategy, and update the network state in the database;

[0011] S5: According to the current monitoring frequency, port bandwidth, network entropy value and total monitoring overhead of each port, reward and punishment values ​​are given and stored in the database;

[0012] S6: Update the Q-table in the database according to the reward and punishment value r;

[0013] S7: Repeat S3 to S6 until the network status suddenly changes, then reset the per-flow monitoring strategy;

[0014] S8: Repeat S3 to S7 until the given number of training times is reached.

[0015] Extract the characteristic values ​​of the network traffic information returned by the switch, combine them into the current state and pass them to the decision module, and store the historical data in the database:

[0016] After receiving the network traffic message returned by the switch, the required feature values ​​are extracted and calculated. The CPU overhead of the controller monitored this time, the communication overhead of the controller, the n ports with the largest bandwidth occupancy in the switch, and the bandwidth and monitoring frequency corresponding to the corresponding ports are used as the current state, and the current state is stored in the database.

[0017] The calculation formula of reward and punishment value r is:

[0018] The reward and punishment rules are as follows: if the monitoring strategy is adjusted to reduce the controller load, a reward will be given; otherwise, a penalty will be given; if the monitoring strategy is adjusted to enable the port with a higher flow rate to obtain a faster monitoring frequency, a reward will be given; otherwise, a penalty will be given; other strategy adjustments will depend on the situation.

[0019] The calculation formula of reward and punishment value r is:

[0020]

[0021] Among them, β c , β u , β v is used to adjust the amplitude of each item, Θ is the flow table item to be monitored in fine-grained manner in the network, C(Θ) and C(Θ)* represent the communication channel overhead between the controller and the switch generated by fine-grained monitoring of this part of the flow in states s and s*, Us and Us* represent the CPU overhead of the controller generated by fine-grained monitoring of this part of the flow in states s and s*, P trepresents the set of all ports in the current state s, H(x) represents the information entropy of the network, and v p Indicates the current bandwidth of port p, f p Indicates the monitoring frequency of the flow table entries of port p.

[0022] The calculation formula of controller cost Us is:

[0023]

[0024] The CPU consumption is obtained by the number of instructions executed by the computer. Each piece of data must be calculated and compared with the original data, so a constant x is given as a coefficient to estimate the CPU consumption. P is the set of ports to be monitored in fine-grained manner. ij is the j port of the i switch, and m represents the length of the message returned by the port, that is, the number of flow table entries of the port.

[0025] The calculation formula of network information entropy is:

[0026]

[0027] Where X represents the set of IP addresses of all hosts in the network, x is an IP address in the set X, and P(x) represents the proportion of network traffic with IP x as the destination in the total network traffic.

[0028] The beneficial effects of the present invention are as follows: by adding a decision-making module to the controller, monitoring the network link and the load of the controller, and constantly interacting with the network environment, the monitoring frequency and granularity of the network traffic are balanced without affecting the network performance, thereby realizing adaptive monitoring of the network traffic, monitoring the network status at a relatively low cost, and when the network is abnormal, the monitoring plan can be adjusted in time to provide the required information for attack detection and other methods, so that the network administrator can better deal with network abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a flowchart of the steps of the SDN network traffic adaptive monitoring method based on reinforcement learning of the present invention.

[0031] Figure 2It is a system architecture diagram of the SDN network traffic adaptive monitoring method based on reinforcement learning of the present invention. DETAILED DESCRIPTION

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0033] See also Figure 1 and Figure 2 The present invention provides a method for adaptively monitoring SDN network traffic based on reinforcement learning, comprising the following steps:

[0034] S1: The controller sends an initial traffic detection request to monitor the traffic of each port of each switch to obtain network traffic information;

[0035] S2: Extract the characteristic values ​​of the network traffic information returned by the switch, combine them into the current state and pass them to the decision module, and store historical data in the database;

[0036] S3: Selecting an action to be executed from an action set according to the current network state and strategy;

[0037] S4: After executing the selected action, modify the current network state and subsequent network states according to the new strategy, and update the network state in the database;

[0038] S5: According to the current monitoring frequency, port bandwidth, network entropy value and total monitoring overhead of each port, reward and punishment values ​​are given and stored in the database;

[0039] S6: Update the Q-table in the database according to the reward and punishment values;

[0040] S7: Repeat S3 to S6 until the network status suddenly changes, then reset the per-flow monitoring strategy;

[0041] S8: Repeat S3 to S7 until the given number of training times is reached.

[0042] Specifically, the characteristic values ​​of the network traffic information returned by the switch are extracted, combined into the current state and passed to the decision module, and the historical data is stored in the database:

[0043] After receiving the network traffic message returned by the switch, the required feature values ​​are extracted and calculated. The bandwidth overhead of the controller sending the monitoring request and receiving the switch return message, the CPU consumption generated by the controller sending the monitoring request, the n ports with the largest bandwidth occupancy in the switch, the used bandwidth and monitoring frequency corresponding to the n ports are used as the current state, and the current state is stored in the database.

[0044] Specifically, in the step of selecting the action to be executed from the action set according to the current network status and strategy:

[0045] The action set is an array consisting of n elements, which is composed of elements in the available action set {0.1, 0, -0.1} (the same element can be selected repeatedly), where "+0.1" means that the monitoring interval increases by 1 second, "0" means that the monitoring frequency remains unchanged, and "-0.1" means that the monitoring interval decreases by 1 second.

[0046] The strategy is to select the link with the maximum reward value argmax with a probability of 1-ε a Q π (s,a), randomly selects a link with a probability of ε, where s represents the current network state, a represents the action with the largest reward value in the current state, and Q π (s,a) represents the reward value of action a with the largest reward value selected according to the strategy in state s, and a* represents the final selected action.

[0047] The specific formula is as follows:

[0048]

[0049] Furthermore, the network traffic monitoring strategy is adjusted based on the following goals:

[0050] 1. The monitoring strategy adjustment needs to consider the controller overhead and minimize the overhead;

[0051] 2. The monitoring strategy adjustment needs to consider the port bandwidth usage to maximize the benefits within limited resources.

[0052] That is, when adjusting the monitoring strategy, the controller overhead should not be too large, otherwise it may affect the normal communication between the switch and the controller; secondly, the adjustment of the port monitoring frequency should be positively correlated with the port flow rate as much as possible.

[0053] Based on the above principles, the status of the network monitoring solution can be defined as "controller CPU overhead", "controller and switch communication channel overhead" and "the n ports with the fastest current flow rate in all switches and their flow rate v p ”.

[0054] Specifically, in the steps of giving reward and punishment values:

[0055] The reward and punishment rules are as follows: if the monitoring strategy is adjusted so that the controller load is reduced, a reward will be given; otherwise, a penalty will be given; if the monitoring strategy is adjusted so that the port with a larger flow rate has a faster monitoring frequency, a reward and punishment value will be given; otherwise, a penalty value will be given; other strategy adjustments depend on the situation.

[0056] Furthermore, the reward and punishment value r is calculated as follows:

[0057]

[0058] Among them, β c , β u , β v is used to adjust the amplitude of each item, Θ is the flow table item to be monitored in fine-grained manner in the network, C(Θ) and C(Θ)* represent the communication channel overhead between the controller and the switch generated by fine-grained monitoring of this part of the flow in states s and s*, Us and Us* represent the CPU overhead of the controller generated by fine-grained monitoring of this part of the flow in states s and s*, P t represents the set of all ports in the current state s, H(x) represents the information entropy of the network, and v p Indicates the current bandwidth of port p, f p Indicates the monitoring frequency of the flow table entries of port p.

[0059] Furthermore, the calculation formula of the communication channel overhead between the controller and the switch is:

[0060]

[0061] Among them, l rq is the monitoring request message sent to the switch, P represents the set of ports to be monitored in fine-grained manner, |θ i |Indicates the number of existing flow table entries in the corresponding port of the switch.

[0062] Furthermore, the calculation formula of the controller cost Us is:

[0063]

[0064] The CPU consumption is obtained by the number of instructions executed by the computer. Each piece of data must be calculated and compared with the original data, so a constant x is given as a coefficient to estimate the CPU consumption. P is the set of ports to be monitored in fine-grained manner. ij is the j port of the i switch, and m represents the length of the message returned by the port, that is, the number of flow table entries of the port.

[0065] Furthermore, the calculation formula of the information entropy of the network is:

[0066]

[0067] Where X represents the set of IP addresses of all hosts in the network, x is an IP address in the set X, and P(x) represents the proportion of network traffic with IP x as the destination in the total network traffic.

[0068] Specifically, until the network status suddenly changes:

[0069] Since network traffic is mutated, taking the action with the highest reward value to update the state under the current network state may result in inconsistency with the ideal subsequent state due to changes in the network state.

[0070] Since the elephant flow in the network occupies 80% of the network bandwidth, in the present invention, it can be approximately considered that each time the subsequent network state is inconsistent with the action with the maximum reward value executed under the current network state, that is, when the network status suddenly changes, the training ends.

[0071] Specifically, in the steps of updating the Q-table in the database according to the reward and punishment values:

[0072] The update formula is Where Q π (s,a) represents the reward value of the action selected according to the strategy under the current network state, s is the current network state, a is the selected action, 0≤α≤1 is the learning rate, and 0≤γ≤1 is the discount factor.

[0073] The above steps are continuously repeated to form the training process of the decision module. When the given number of training times is reached, the decision module will obtain a better strategy for completing the task. In the present invention, the decision module will learn the optimal or approximately optimal solution for adjusting the monitoring strategy when facing a network attack. Under the condition of limited processing capacity of the controller and the switch, the network status can be monitored at a relatively low cost. When an abnormality occurs in the network, the monitoring plan can be adjusted in time to provide the required information for attack detection and other methods, so that the network administrator can better deal with network abnormalities.

[0074] The SDN network traffic adaptive monitoring method based on reinforcement learning of the present invention adds a decision module to the controller, continuously interacts with the network environment according to the network link and the load of the controller, and adaptively monitors the network traffic without affecting the network performance, thereby balancing the monitoring frequency and granularity.

[0075] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A SDN network traffic adaptive monitoring method based on reinforcement learning, characterized in that: The steps include: S1: The controller sends an initial traffic detection request to monitor the traffic of each port of each switch to obtain network traffic information; S2: Extract the characteristic values ​​of the network traffic information returned by the switch, combine the characteristic values ​​into the current network status and pass it to the decision module, and store the historical data in the database; S3: Selecting an action to be executed from an action set according to the current network state and strategy; S4: After executing the selected action, the current network state is modified according to the new strategy to obtain the subsequent network state, and the network state is updated in the database at the same time; S5: According to the current monitoring frequency, port bandwidth, network entropy value and total monitoring overhead of each port, reward and punishment values ​​are given and stored in the database; S6: Update the Q-table in the database according to the reward and punishment values; S7: Repeat S3 to S6 until the current network status suddenly changes, then reset the network traffic monitoring strategy; S8: Repeat S3 to S7 until the given number of training times is reached; In the steps of giving reward and penalty values: The reward and punishment rules are as follows: if the monitoring strategy is adjusted to reduce the controller load, then the reward will be given; otherwise, the penalty will be given; if the monitoring strategy is adjusted to make the port with a larger flow rate have a faster monitoring frequency, then the reward will be given; otherwise, the penalty will be given; The formula for calculating the reward and punishment value is: Among them, β c , β u , β v is used to adjust the amplitude of each item, Θ is the flow table item to be monitored in fine-grained manner in the network, C(Θ) and C(Θ)* represent the communication channel overhead between the controller and the switch generated by fine-grained monitoring of this part of the flow in states s and s*, Us and Us* represent the CPU overhead of the controller generated by fine-grained monitoring of this part of the flow in states s and s*, P t represents the set of all ports in the current state s, H(x) represents the information entropy of the network, x is the IP address, v p Indicates the current bandwidth of port p, f p Indicates the monitoring frequency of the flow table entry of port p; The calculation formula of controller cost Us is: The CPU consumption is obtained by the number of instructions executed by the computer. Each piece of data is calculated and compared with the original data. The constant a is given as a coefficient to estimate the CPU consumption. P is the set of ports to be monitored in fine-grained manner. ij is the j port of the i switch, and m represents the length of the message returned by the port, that is, the number of flow table entries of the port.

2. The SDN network traffic adaptive monitoring method based on reinforcement learning according to claim 1 is characterized in that: Extract the characteristic values ​​of the network traffic information returned by the switch, combine the characteristic values ​​into the current network state and pass it to the decision module, and store the historical data in the database: After receiving the network traffic information returned by the switch, the required feature values ​​are extracted and calculated. The CPU overhead of the controller monitored this time, the communication overhead of the controller, the n ports with the largest bandwidth occupancy in the switch, and the bandwidth and monitoring frequency corresponding to the corresponding ports are used as the current state, and the current state is stored in the database.

3. The SDN network traffic adaptive monitoring method based on reinforcement learning as claimed in claim 2 is characterized in that: The calculation formula of network information entropy is: Where X represents the set of IP addresses of all hosts in the network, x is an IP address in the set X, and P(x) represents the proportion of network traffic with IP x as the destination in the total network traffic.

Images