Flow detection system and method based on software defined network

Abstract

The invention discloses a flow detection system and method based on a software defined network, belongs to the technical field of machine learning, and aims to solve the technical problem of how to realize flexible network flow monitoring under the condition of balancing the detection interval and the detection precision. Comprising a monitoring manager, a statistical module, a data storage module, a state conversion module and an intelligent agent module which are configured in an SDN controller. The intelligent agent module continuously interacts with a network environment according to controller load, switch resource consumption, channel overhead and a flow rate in a network which are generated in a monitoring process, flexibly monitors the flow in the network and balances the monitoring precision and the detection interval under the condition of ensuring normal operation of the network, so that the flow rate in the network can be accurately detected. Compared with the prior art, the network flow is monitored at very low cost, and even if the network is suddenly changed, the detection interval can be adjusted in real time, so that good support is provided for network management, and network maintenance personnel can conveniently and quickly process abnormity occurring in the network.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a traffic detection system and method based on software defined network. Background Art

[0002] Traffic monitoring technology uses detection and evaluation methods to support network operation and maintenance, thereby providing better services to users. Fine-grained monitoring of network traffic is necessary for many applications, such as network anomaly detection, QoS services, and network load balancing.

[0003] Software-defined networking is a very flexible architecture that separates the data plane from the management plane, enabling fast and convenient deployment of network devices. The architecture achieves centralized management capabilities through the SDN controller, and because it is programmable, the controller can achieve more precise network management. Based on the characteristics of software-defined networking, the SDN controller can easily obtain a global view of the network and more easily obtain fine-grained traffic data by accessing the switch port.

[0004] In software-defined networks, existing traffic monitoring technologies are mainly divided into active monitoring methods and passive monitoring methods. The passive monitoring method refers to sending a piece of information about a flow entry in the switch to the controller after the survival time of the flow entry reaches the end. This method has the following disadvantages: first, it is not flexible enough; second, when the traffic changes drastically, many unmatched flow entries will appear inside the switch, causing the switch to send a large number of Packet-in messages to the controller, which incurs a lot of overhead for the controller. The active monitoring method refers to the controller sending request messages to the switch periodically. If the detection interval is too small, it will cause a great burden on the controller. If the detection interval is too large, it will lead to insufficient monitoring accuracy. Both of the above methods cannot balance the detection interval and detection accuracy well, and cannot achieve intelligent optimization of this trade-off mechanism by learning the behavior of the network.

[0005] How to achieve flexible network traffic monitoring while balancing the detection interval and detection accuracy is a technical problem that needs to be solved. Summary of the invention

[0006] The technical task of the present invention is to address the above shortcomings and provide a traffic detection system and method based on software-defined networks to solve the technical problem of how to achieve flexible network traffic monitoring while balancing the detection interval and detection accuracy.

[0007] In a first aspect, the present invention provides a traffic detection system based on a software defined network, comprising a monitoring manager, a statistics module, a data storage module, a state conversion module and an intelligent agent module configured in an SDN controller;

[0008] The data storage module stores the initialized detection interval;

[0009] The monitoring manager is used to obtain the detection interval from the data storage module, send a detection request message to all ports of each switch in the data plane based on the detection interval, receive the detection response message returned by all ports of each switch in the data plane, and extract the network flow information from the data response message, and merge the network flow information corresponding to all ports of each switch and upload it to the statistics module;

[0010] The statistical module is used to perform data preprocessing on the network traffic information, and to perform data cleaning and format conversion through data preprocessing to obtain the preprocessed network traffic information;

[0011] The state conversion module is used to obtain the pre-processed network traffic information from the statistical module, extract the network traffic characteristics from the network traffic information as the state of the environment at the corresponding time, and send the state to the data storage module for storage, wherein the state includes the CPU utilization rate of the SDN controller, the channel overhead between the controller and the data plane, the resource utilization rate of the monitored switch, and the traffic rate of the monitored switch;

[0012] The agent module is provided with an action set and a traffic detection scheduling model based on a DQN network. The action set is an array consisting of multiple detection intervals. The traffic detection scheduling model includes a decision network and a target network with the same structure. The decision network is used to take the current state as input and select the next detection interval from the action set as an action. The target network is used to optimize the parameters of the decision network based on the current state, action, action reward and the state at the next moment. The agent module is used to perform the following:

[0013] Action evaluation: Taking the current state as input, the decision network predicts and outputs the Q value of each action under the current state, selects an action based on the Q value, returns the action to the environment and stores it in the data storage module, obtains the state, action and reward corresponding to the current moment and the state at the next moment from the environment as four-element sample data, and stores the sample data in the cache pool;

[0014] Parameter optimization: Get sample data from the cache pool, take the state at the next moment as input, predict the Q value of each action at the next moment through the target network, and select a Q value as output. Based on the Q value predicted by the decision network and the Q value predicted by the target network, optimize the parameters of the decision network. During the parameter optimization process, when the number of iterations reaches the predetermined step size, periodically copy the network parameters of the decision network to the target network until the network state is wrong, and reset the detection interval.

[0015] Preferably, the reward calculation formula is as follows:

[0016]

[0017] in, as well as represents the weight, ΔV represents the difference between the CPU usage of the SDN controller at the previous moment and the current moment, ΔC represents the difference between the channel overhead between the controller and the data plane at the previous moment and the current moment, ΔS represents the difference between the resource overhead generated by the monitored switch at the previous moment and the current moment, and ΔU represents the difference between the traffic rate of the monitored port of the switch at the previous moment and the current moment;

[0018] The channel overhead calculation formula between the SDN controller and the data plane is as follows:

[0019] C=(g rq +g rp )*|V|+(h rq +h rp )*|θ|,

[0020] g rq Indicates port detection request message g rp Indicates the port detection response message, h rq Refers to the flow table entry detection request message sent by the SDN controller to the data plane. rp is the flow entry detection response message, V refers to the number of switches to be detected, and θ is the number of surviving flow entries of the port to be detected;

[0021] The formula for the SDN controller CPU usage is:

[0022]

[0023] Among them, the constant x is used to calculate the CPU consumption of the SDN controller;

[0024] The switch cost is expressed as:

[0025]

[0026] Where y is a constant value used to calculate the switch resource overhead, which refers to the number of survivable flow entries in the current switch;

[0027] The calculation formula of the traffic rate V of the monitored port is:

[0028]

[0029] V=|sendspeed+receivingspeed|,

[0030] sendspeed indicates the sending byte rate, sendbytes c Refers to the amount of bytes sent in the current state, sendbytes p It refers to the amount of bytes sent in the previous state;

[0031] ReceivingSpeed ​​indicates the receiving byte rate, receivingbytes c Refers to the amount of received bytes in the current state, receivingbytes p It refers to the amount of bytes sent in the previous state;

[0032] The action set is represented by {0,1,2,…i,…,9,10}, where 0 indicates that no probe request message is sent to the port, 1 to 10 indicate the detection interval for sending probe request messages to the port, 1 indicates that the detection interval is 1 second, and i indicates that the detection interval is i seconds.

[0033] As a preference, when evaluating actions, an action is selected based on the Q value through the ε-greedy strategy, and the steps are as follows:

[0034] Set an ε between 0 and 1;

[0035] Randomly select a random number. When the random number is in the range of 0 to ε, randomly select an action in the action set to execute. When the random number is in the range of ε to 1, select the most valuable action in the action set. The Q value corresponding to the action is expressed as argmax a Q π (s,a), where Q π (s,a) represents the expected return of executing action a in state s and following strategy π, and finally executing action a * It is expressed as:

[0036]

[0037] As a preference, when optimizing parameters, the error between the Q value predicted by the decision network and the Q value predicted by the target network is calculated, and the policy network is updated using the reverse gradient descent method based on the square of the error value.

[0038] The calculation formula of the error value TD is as follows:

[0039] TD=R t+1 +γmax a Q(S t+1 ,a)-Q(S t ,A t ),

[0040] Among them, R t+1 R represents the immediate reward after executing the action, γ represents the discount factor, max a Q(S t+1 ,a) represents the Q value of the state-action pair at the next moment output by the target network, Q(S t ,A t ) represents the Q value of the current state-action pair output by the decision network;

[0041] The policy network update formula is:

[0042] Q(S t ,A t )←Q(S t ,A t )+α*TD,

[0043] Among them, α represents the learning rate.

[0044] In a second aspect, the present invention provides a method for detecting traffic based on a software defined network, which performs traffic monitoring through a traffic monitoring system based on a software defined network as described in any one of the first aspects, comprising the following steps:

[0045] S100, configuring an initialization detection interval in a data storage module;

[0046] S200, the monitoring manager reads the detection interval from the data storage module, sends a detection request message to all ports of each switch in the data plane based on the detection interval, receives the detection response message returned by all ports of each switch in the data plane, extracts the network traffic information from the detection response message, merges the network traffic information corresponding to all ports of each switch, and uploads it to the statistics module;

[0047] S300, extracting network traffic features from network traffic information as the state of the environment at a corresponding time through a statistical module, and sending the state to a data storage module for storage, wherein the state includes a CPU utilization rate of the SDN controller, a channel overhead between the controller and the data plane, a resource utilization rate of the monitored switch, and a traffic rate of the monitored switch;

[0048] S400, the intelligent agent module interacts with the environment to perform action evaluation and parameter optimization. During action evaluation, the current state is used as input, and the Q value of each action under the current state is predicted and output through the decision network, and an action is selected based on the Q value, and the action is returned to the environment and stored in the data storage module, and the state, action and reward corresponding to the current moment and the state of the next moment are obtained from the environment as four-element sample data, and the sample data is stored in the cache pool; during parameter optimization, sample data is obtained from the cache pool, and the state of the next moment is used as input, and the Q value of each action under the state of the next moment is predicted through the target network, and a Q value is selected as output, and the parameters of the decision network are optimized based on the Q value predicted by the decision network and the Q value predicted by the target network. During the parameter optimization process, when the number of iterations reaches a predetermined step length, the network parameters of the decision network are periodically copied to the target network until the network state is wrong, and the detection interval is reset;

[0049] Correspondingly, the monitoring manager reads the updated detection interval from the data storage module, sends a detection request message to all ports of each switch in the data plane based on the new detection interval, receives the detection response message returned by all ports of each switch in the data plane, extracts the network traffic information from the detection response message, merges the network traffic information corresponding to all ports of each switch, and uploads it to the statistics module;

[0050] Correspondingly, the statistical module extracts network traffic features from the network traffic information as the state of the environment at the corresponding moment, and sends the state to the data storage module for storage;

[0051] S500, executing step S400 repeatedly until a predetermined number of cycles is reached.

[0052] The traffic detection system and method based on software-defined network of the present invention have the following advantages: reinforcement learning is combined with software-defined network architecture, and an intelligent body module is added to the control plane. The intelligent body module continuously interacts with the network environment based on the controller load, switch resource consumption, channel overhead and traffic rate in the network generated during the monitoring process. While ensuring the normal operation of the network, the traffic in the network is flexibly monitored and the monitoring accuracy and detection interval size are balanced. The network traffic is monitored at a very low cost. Even when a sudden change occurs in the network, the detection interval can be adjusted in real time, providing good support for network management and facilitating network maintenance personnel to quickly handle anomalies in the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] The present invention is further described below in conjunction with the accompanying drawings.

[0055] Figure 1 This is a structural diagram of a traffic detection system based on software defined network according to Example 1;

[0056] Figure 2 This is a flowchart of the action evaluation and parameter optimization process of a traffic detection scheduling model in a traffic detection system based on software-defined networking in Example 1. DETAILED DESCRIPTION

[0057] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0058] The embodiments of the present invention provide a traffic detection system and method based on software defined network, which are used to solve the technical problem of how to achieve flexible network traffic monitoring while balancing the detection interval and detection accuracy.

[0059] Embodiment 1:

[0060] The present invention discloses a flow detection system based on software defined network, comprising a monitoring manager, a statistical module, a data storage module, a state conversion module and an intelligent body module configured in an SDN controller.

[0061] The data storage module stores the initialized detection interval.

[0062] The monitoring manager is used to obtain the detection interval from the data storage module, send detection request messages to all ports of each switch in the data plane based on the detection interval, receive detection response messages returned by all ports of each switch in the data plane, and extract network traffic information from the data response messages, so as to merge the network traffic information corresponding to all ports of each switch and upload it to the statistics module.

[0063] The statistical module is used to perform data preprocessing on network traffic information, and through data preprocessing, data cleaning and format conversion are performed to obtain preprocessed network traffic information.

[0064] As a specific implementation of the statistical module, outliers and null values ​​are removed through data cleaning, and the cleaned network traffic information is formatted and converted into a predetermined format.

[0065] The state conversion module is used to obtain the preprocessed network traffic information from the statistical module, extract the network traffic characteristics from the network traffic information as the state of the environment at the corresponding time, and send the state to the data storage module for storage, where the state includes the CPU utilization rate of the SDN controller, the channel overhead between the controller and the data plane, the resource utilization rate of the monitored switch, and the traffic rate of the monitored switch.

[0066] In this embodiment, the detection scheme for traffic monitoring in the SDN network is adjusted as follows:

[0067] (1) The monitoring scheme adjustment should include the factors affecting the SDN controller load to reduce the controller load;

[0068] (2) The monitoring scheme should be adjusted to include the channel overhead between the data plane Openflow switch and the controller to increase the benefits under the constraint of limited bandwidth;

[0069] (3) The monitoring plan adjustment should include the CPU usage of the monitored switch. When monitoring the switch, the usage of the monitored switch should be considered, and the normal function of the switch should be avoided as much as possible to avoid affecting the normal operation of the entire network.

[0070] (4) The monitoring plan adjustment should include the traffic rate of the monitored port to monitor the maximum traffic changes within a limited time.

[0071] That is, when adjusting the monitoring plan, first, the purpose is to reduce the SDN control load. Excessive controller load will affect the overall performance of the controller, which will in turn have a negative impact on the operation of the entire network; second, to reduce the load of the monitored switch; then, to reduce the channel overhead between the controller and the data plane during the monitoring process; then, the size of the detection interval should be proportional to the rate of change of the traffic on the Openflow switch port.

[0072] Under the above conditions, the state of the state conversion module can be clearly defined as the CPU usage of the SDN controller, the channel overhead between the controller and the data plane, the resource usage of the monitored switch, and the traffic rate of the monitored switch.

[0073] The agent module is provided with an action set and a traffic detection scheduling model based on a DQN network. The action set is an array consisting of multiple detection intervals. The traffic detection scheduling model includes a decision network and a target network with the same structure. The decision network is used to take the current state as input and select the next detection interval from the action set as an action. The target network is used to optimize the parameters of the decision network based on the current state, action, action reward and the state at the next moment. The agent module is used to perform the following:

[0074] (1) Action evaluation: Taking the current state as input, the decision network predicts and outputs the Q value of each action under the current state, selects an action based on the Q value, returns the action to the environment and stores it in the data storage module, obtains the state, action and reward corresponding to the current moment and the state at the next moment from the environment as four-element sample data, and stores the sample data in the cache pool;

[0075] (2) Parameter optimization: Obtain sample data from the cache pool, take the state at the next moment as input, predict the Q value of each action at the next moment through the target network, and select a Q value as output. Based on the Q value predicted by the decision network and the Q value predicted by the target network, optimize the parameters of the decision network. During the parameter optimization process, when the number of iterations reaches the predetermined step size, periodically copy the network parameters of the decision network to the target network until the network state is wrong, and reset the detection interval.

[0076] During the parameter optimization process, network state errors are understood as follows: the network environment is dynamic and complex. After the intelligent agent module takes the most valuable action in the current state, the next state may be significantly different from the envisioned state due to a sudden change in the network environment. When the current state is significantly different from the next state when the most valuable action is taken, the network environment mutates, and this round of training ends.

[0077] The reward value setting target in this embodiment is that when the overhead of the SDN controller to execute monitoring increases after the new detection scheme is executed, the reward value is negative, otherwise it is positive; when the monitored switch resources are used more after the new detection scheme is executed, the reward value is negative, otherwise it is negative; when the channel overhead between the controller and the data plane increases after the new detection scheme is executed, the reward value is negative, otherwise it is positive; when the traffic rate changes more after the new detection scheme is executed, the reward value is negative, otherwise it is positive; other scheme adjustments are subject to specific circumstances.

[0078] The reward calculation formula is as follows:

[0079]

[0080] in, as well as represents the weight, ΔV represents the difference between the CPU usage of the SDN controller at the previous moment and the current moment, ΔC represents the difference between the channel overhead between the controller and the data plane at the previous moment and the current moment, ΔS represents the difference between the resource overhead generated by the monitored switch at the previous moment and the current moment, and ΔU represents the difference between the traffic rate of the monitored port of the switch at the previous moment and the current moment;

[0081] The channel overhead calculation formula between the SDN controller and the data plane is as follows:

[0082] C=(g rq +g rp )*|V|+(h rq +h rp )*|θ|,

[0083] g rq Indicates port detection request message g rp Indicates the port detection response message, h rq Refers to the flow table entry detection request message sent by the SDN controller to the data plane. rp is the flow entry detection response message, V refers to the number of switches to be detected, and θ is the number of surviving flow entries of the port to be detected;

[0084] The formula for the SDN controller CPU usage is:

[0085]

[0086] Among them, the constant x is used to calculate the CPU consumption of the SDN controller;

[0087] The switch cost is expressed as:

[0088]

[0089] Where y is a constant value used to calculate the switch resource overhead, which refers to the number of survivable flow entries in the current switch;

[0090] The calculation formula of the traffic rate V of the monitored port is:

[0091]

[0092] V=|sendspeed+receivingspeed|,

[0093] sendspeed indicates the sending byte rate, sendbytes c Refers to the amount of bytes sent in the current state, sendbytes p It refers to the amount of bytes sent in the previous state;

[0094] ReceivingSpeed ​​indicates the receiving byte rate, receivingbytes c Refers to the amount of received bytes in the current state, receivingbytes p Refers to the number of bytes sent in the previous state.

[0095] The action set is represented by {0,1,2,…i,…,9,10}, where 0 indicates that no probe request message is sent to the port, 1 to 10 indicate the detection interval for sending probe request messages to the port, 1 indicates that the detection interval is 1 second, and i indicates that the detection interval is i seconds.

[0096] When evaluating an action, an action is selected based on the Q value through the ε-greedy strategy. The steps are as follows:

[0097] (1) Set an ε between 0 and 1;

[0098] (2) Randomly select a random number. When the random number is in the range of 0 to ε, randomly select an action in the action set to execute. When the random number is in the range of ε to 1, select the most valuable action in the action set. The Q value corresponding to the action is expressed as argmax a Q π (s,a), where Q π (s,a) represents the expected return of executing action a in state s and following strategy π, and finally executing action a * It is expressed as:

[0099]

[0100] When optimizing parameters, the error between the Q value predicted by the decision network and the Q value predicted by the target network is calculated, and the policy network is updated using the reverse gradient descent method based on the square of the error value.

[0101] The calculation formula of the error value TD is as follows:

[0102] TD=R t+1 +γmax a Q(S t+1 ,a)-Q(S t ,A t ),

[0103] Among them, R t+1 R represents the immediate reward after executing the action, γ represents the discount factor, max a Q(S t+1 ,a) represents the Q value of the state-action pair at the next moment output by the target network, Q(S t ,A t) represents the Q value of the current state-action pair output by the decision network;

[0104] The policy network update formula is:

[0105] Q(S t ,A t )←Q(S t ,A t )+α*TD,

[0106] Among them, α represents the learning rate.

[0107] In this embodiment, the intelligent body module can learn to promptly derive the optimal detection plan in the face of changes in the network environment. Under the constraints of limited controller load, it can monitor network traffic at a very low cost. When a sudden change occurs in the network, it can adjust the detection interval in real time, providing good support for network management and allowing network maintenance personnel to handle anomalies in the network well.

[0108] The system of this embodiment introduces an intelligent body module into the SDN controller, which continuously interacts with the network environment for the SDN controller load and the traffic information data in the network. While ensuring the normal operation of the network, it flexibly monitors the traffic in the network and balances the monitoring accuracy and the detection interval size, so as to monitor the network traffic at a very low cost.

[0109] Embodiment 2:

[0110] The present invention provides a flow detection method based on a software defined network, which monitors flow through the system disclosed in Example 1, and includes the following steps:

[0111] S100, configuring an initialization detection interval in a data storage module;

[0112] S200, the monitoring manager reads the detection interval from the data storage module, sends a detection request message to all ports of each switch in the data plane based on the detection interval, receives the detection response message returned by all ports of each switch in the data plane, extracts the network traffic information from the detection response message, merges the network traffic information corresponding to all ports of each switch, and uploads it to the statistics module;

[0113] S300, extracting network traffic features from network traffic information as the state of the environment at a corresponding time through a statistical module, and sending the state to a data storage module for storage, wherein the state includes a CPU utilization rate of the SDN controller, a channel overhead between the controller and the data plane, a resource utilization rate of the monitored switch, and a traffic rate of the monitored switch;

[0114] S400, the intelligent agent module interacts with the environment to perform action evaluation and parameter optimization. During action evaluation, the current state is used as input, and the Q value of each action under the current state is predicted and output through the decision network, and an action is selected based on the Q value, and the action is returned to the environment and stored in the data storage module, and the state, action and reward corresponding to the current moment and the state of the next moment are obtained from the environment as four-element sample data, and the sample data is stored in the cache pool; during parameter optimization, sample data is obtained from the cache pool, and the state of the next moment is used as input, and the Q value of each action under the state of the next moment is predicted through the target network, and a Q value is selected as output, and the parameters of the decision network are optimized based on the Q value predicted by the decision network and the Q value predicted by the target network. During the parameter optimization process, when the number of iterations reaches a predetermined step length, the network parameters of the decision network are periodically copied to the target network until the network state is wrong, and the detection interval is reset;

[0115] Correspondingly, the monitoring manager reads the updated detection interval from the data storage module, sends a detection request message to all ports of each switch in the data plane based on the new detection interval, receives the detection response message returned by all ports of each switch in the data plane, extracts the network traffic information from the detection response message, merges the network traffic information corresponding to all ports of each switch, and uploads it to the statistics module;

[0116] Correspondingly, the statistical module extracts network traffic features from the network traffic information as the state of the environment at the corresponding moment, and sends the state to the data storage module for storage;

[0117] S500, executing step S400 repeatedly until a predetermined number of cycles is reached.

[0118] The above is a detailed introduction to the traffic detection system and method based on software-defined network provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A traffic detection system based on software defined network, characterized in that: It includes a monitoring manager, a statistics module, a data storage module, a state conversion module and an intelligent agent module configured in the SDN controller; The data storage module stores the initialized detection interval; The monitoring manager is used to obtain the detection interval from the data storage module, send a detection request message to all ports of each switch in the data plane based on the detection interval, receive the detection response message returned by all ports of each switch in the data plane, and extract the network flow information from the data response message, and merge the network flow information corresponding to all ports of each switch and upload it to the statistics module; The statistical module is used to perform data preprocessing on the network traffic information, and to perform data cleaning and format conversion through data preprocessing to obtain the preprocessed network traffic information; The state conversion module is used to obtain the pre-processed network traffic information from the statistical module, extract the network traffic characteristics from the network traffic information as the state of the environment at the corresponding time, and send the state to the data storage module for storage, wherein the state includes the CPU utilization rate of the SDN controller, the channel overhead between the controller and the data plane, the resource utilization rate of the monitored switch, and the traffic rate of the monitored switch; The agent module is provided with an action set and a traffic detection scheduling model based on a DQN network. The action set is an array consisting of multiple detection intervals. The traffic detection scheduling model includes a decision network and a target network with the same structure. The decision network is used to take the current state as input and select the next detection interval from the action set as an action. The target network is used to optimize the parameters of the decision network based on the current state, action, action reward and the state at the next moment. The agent module is used to perform the following: Action evaluation: Taking the current state as input, the decision network predicts and outputs the Q value of each action under the current state, selects an action based on the Q value, returns the action to the environment and stores it in the data storage module, obtains the state, action and reward corresponding to the current moment and the state at the next moment from the environment as four-element sample data, and stores the sample data in the cache pool; Parameter optimization: Get sample data from the cache pool, take the state at the next moment as input, predict the Q value of each action at the next moment through the target network, and select a Q value as output. Based on the Q value predicted by the decision network and the Q value predicted by the target network, optimize the parameters of the decision network. During the parameter optimization process, when the number of iterations reaches the predetermined step size, periodically copy the network parameters of the decision network to the target network until the network state is wrong, and reset the detection interval.

2. A traffic monitoring system based on software defined network according to claim 1, characterized in that: The reward calculation formula is as follows: in, as well as represents the weight, ΔV represents the difference between the CPU usage of the SDN controller at the previous moment and the current moment, ΔC represents the difference between the channel overhead between the controller and the data plane at the previous moment and the current moment, ΔS represents the difference between the resource overhead generated by the monitored switch at the previous moment and the current moment, and ΔU represents the difference between the traffic rate of the monitored port of the switch at the previous moment and the current moment; The channel overhead calculation formula between the SDN controller and the data plane is as follows: C=(g rq +g rp )*|V|+(h rq +h rp )*|θ|, g rq Indicates port detection request message g rp Indicates the port detection response message, h rq Refers to the flow table entry detection request message sent by the SDN controller to the data plane. rp is the flow entry detection response message, V refers to the number of switches to be detected, and θ is the number of surviving flow entries of the port to be detected; The formula for the SDN controller CPU usage is: Among them, the constant x is used to calculate the CPU consumption of the SDN controller; The switch cost is expressed as: Where y is a constant value used to calculate the switch resource overhead, which refers to the number of live flow entries in the current switch; The calculation formula of the traffic rate V of the monitored port is: V=|sendspeed+receivingspeed|, sendspeed indicates the sending byte rate, sendbytes c Refers to the amount of bytes sent in the current state, sendbytes p It refers to the amount of bytes sent in the previous state; ReceivingSpeed ​​indicates the receiving byte rate, receivingbytes c Refers to the amount of received bytes in the current state, receivingbytes p It refers to the amount of bytes sent in the previous state; The action set is represented by {0,1,2,…i,…,9,10}, where 0 indicates that no probe request message is sent to the port, 1 to 10 indicate the detection interval for sending probe request messages to the port, 1 indicates that the detection interval is 1 second, and i indicates that the detection interval is i seconds.

3. A traffic monitoring system based on software defined network according to claim 1, characterized in that: When evaluating an action, an action is selected based on the Q value through the ε-greedy strategy. The steps are as follows: Set an ε between 0 and 1; Randomly select a random number. When the random number is in the range of 0 to ε, randomly select an action in the action set to execute. When the random number is in the range of ε to 1, select the most valuable action in the action set. The Q value corresponding to the action is expressed as argmax a Q π (s,a), where Q π (s,a) represents the expected return of executing action a in state s and following strategy π, and finally executing action a * It is expressed as:

4. A traffic monitoring system based on software defined network according to claim 1, characterized in that: When optimizing parameters, the error between the Q value predicted by the decision network and the Q value predicted by the target network is calculated, and the policy network is updated using the reverse gradient descent method based on the square of the error value. The calculation formula of the error value TD is as follows: TD=R t+1 +γmax a Q(S t+1 ,a)-Q(S t ,A t ), Among them, R t+1 R represents the immediate reward after executing the action, γ represents the discount factor, max a Q(S t+1 ,a) represents the Q value of the state-action pair at the next moment output by the target network, Q(S t ,A t ) represents the Q value of the current state-action pair output by the decision network; The policy network update formula is: Q(S t ,A t )←Q(S t ,A t )+α*TD, Among them, α represents the learning rate.

5. A traffic detection method based on software defined network, characterized in that: Traffic monitoring is performed by a traffic monitoring system based on a software defined network as described in any one of claims 1 to 4, comprising the following steps: S100, configuring an initialization detection interval in a data storage module; S200, the monitoring manager reads the detection interval from the data storage module, sends a detection request message to all ports of each switch in the data plane based on the detection interval, receives the detection response message returned by all ports of each switch in the data plane, extracts the network traffic information from the detection response message, merges the network traffic information corresponding to all ports of each switch, and uploads it to the statistics module; S300, extracting network traffic features from network traffic information as the state of the environment at a corresponding time through a statistical module, and sending the state to a data storage module for storage, wherein the state includes a CPU utilization rate of the SDN controller, a channel overhead between the controller and the data plane, a resource utilization rate of the monitored switch, and a traffic rate of the monitored switch; S400, the intelligent agent module interacts with the environment to perform action evaluation and parameter optimization. During action evaluation, the current state is used as input, and the Q value of each action under the current state is predicted and output through the decision network, and an action is selected based on the Q value, and the action is returned to the environment and stored in the data storage module, and the state, action and reward corresponding to the current moment and the state of the next moment are obtained from the environment as four-element sample data, and the sample data is stored in the cache pool; during parameter optimization, sample data is obtained from the cache pool, and the state of the next moment is used as input, and the Q value of each action under the state of the next moment is predicted through the target network, and a Q value is selected as output, and the parameters of the decision network are optimized based on the Q value predicted by the decision network and the Q value predicted by the target network. During the parameter optimization process, when the number of iterations reaches a predetermined step length, the network parameters of the decision network are periodically copied to the target network until the network state is wrong, and the detection interval is reset; Correspondingly, the monitoring manager reads the updated detection interval from the data storage module, sends a detection request message to all ports of each switch in the data plane based on the new detection interval, receives the detection response message returned by all ports of each switch in the data plane, extracts the network traffic information from the detection response message, merges the network traffic information corresponding to all ports of each switch, and uploads it to the statistics module; Correspondingly, the statistical module extracts network traffic features from the network traffic information as the state of the environment at the corresponding moment, and sends the state to the data storage module for storage; S500, executing step S400 repeatedly until a predetermined number of cycles is reached.

Images