A method for optimizing network traffic scheduling based on proximal strategy optimization algorithm

By using the ASTPPO system and feature extraction module in the SDN network to capture timing and spatial correlations, and generate an optimized SDN flow table, the problem of failure to effectively optimize the SDN network in the prior art is solved, and load balancing and performance improvement are achieved.

CN115550268BActive Publication Date: 2024-11-19GUILIN UNIV OF ELECTRONIC TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211042072.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-29
Publication Date
2024-11-19
Estimated Expiration
2042-08-29

AI Technical Summary

Technical Problem

Existing reinforcement learning methods fail to effectively consider network timing and spatial correlation, resulting in poor performance optimization in software-defined networks (SDNs).

Method used

The ASTPPO system architecture based on the near-end strategy optimization algorithm is adopted, combining the gated cycle unit, self-attention mechanism and graph attention network to extract the timing and spatial correlations in the network state information, and generate an optimized SDN flow table for traffic scheduling.

Benefits of technology

It realizes high availability and good performance in load balancing and performance optimization, makes full use of network resources, and improves the global optimization capabilities of the SDN network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115550268B_ABST
    Figure CN115550268B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of deep learning technology, and specifically to a method for optimizing network traffic scheduling based on a proximal strategy optimization algorithm. The data plane of a software-defined network has time characteristics and spatial characteristics. In order to globally consider the time and space correlation of the network, it is proposed to use a gated recurrent unit and a graph attention network method to extract the temporal correlation and spatial correlation in state information, and provide implicit information containing more environments for the decision-making of deep reinforcement learning, thereby solving network traffic engineering problems, making full use of network resources to dynamically adjust network loads, effectively alleviating network link congestion, and optimizing network performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method for optimizing network traffic scheduling based on a proximal strategy optimization algorithm. Background Art

[0002] In recent years, with the rapid development of the Internet, the scale of networks has become increasingly large. The popularity of social media, high-definition movies, online games, and 5G has led to a rapid increase in network traffic, and the pressure on traditional forwarding networks has increased. To adapt to the continued growth of network scale, a more flexible, intelligent, and high-traffic-carrying network architecture, namely Software Defined Networking (SDN), has emerged. Among the optimization goals for intelligent SDN networks, traffic engineering (TE) optimization is an important area. For TE optimization goals in various network scenarios, there are a large number of solutions based on flow routing, link state protocols, or overlay networks. A basic TE problem solved in the relevant literature is intra-domain TE, that is, networks in the same autonomous domain, such as operator networks, data center networks, or campus networks. The classic optimization goal is to minimize the maximum link load within the self-managed network domain, which has been proven to be an NP-hard problem.

[0003] Over the past few years, researchers have applied machine learning (ML) to complex network control and management problems, particularly deep reinforcement learning (DRL) technology, achieving remarkable results in network control. However, existing research on DRL applications in SDN has not yet focused on the characteristics of SDN networks, which limits the performance of DRL algorithms in optimizing SDN network performance. Customized deep reinforcement learning models perform well in specific network scenarios, but their generalization capabilities in other network environments remain to be verified. Furthermore, DRL algorithms optimized using other networks, such as CNNs, RNNs, or GNNs, only consider network temporal or spatial correlations, without fully considering the network's spatiotemporal correlations. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for optimizing network traffic scheduling based on a proximal strategy optimization algorithm, so as to solve the problem that the existing reinforcement learning method does not consider the network timing correlation or network space correlation and has a poor effect on SDN network performance optimization.

[0005] To achieve the above object, the present invention provides a method for optimizing network traffic scheduling based on a proximal policy optimization algorithm, comprising the following steps:

[0006] Build the ASTPPO system architecture;

[0007] Collect network link information and network topology information;

[0008] Generate status information and reward values ​​respectively;

[0009] Taking the state information and the reward value as input, obtaining an action value and forwarding it;

[0010] An SDN flow table is generated based on the action value, and the SDN flow table is issued for traffic scheduling.

[0011] Among them, the ASTPPO system architecture includes a data exchange layer, a control layer and an intelligent decision-making layer. The data exchange layer is composed of an SDN switch that supports the SDN working protocol. The control layer is connected to the data exchange layer through a southbound interface and to the intelligent decision-making layer through a northbound interface. The intelligent decision-making layer outputs link weights for routing policy updates.

[0012] Among them, in the process of collecting network link information and network topology information, the information collector of the control layer collects network link information from the SDN switch of the data exchange layer through the southbound interface at regular intervals, and the topology discoverer of the control layer collects network topology information from the SDN switch of the data exchange layer through the southbound interface at regular intervals.

[0013] Among them, the process of generating status information and reward value respectively is that the information collector of the control layer processes the network link information and generates the status information of the intelligent agent in the intelligent decision-making layer. The reward function module of the control layer processes the network link information collected by the information collector and generates the reward value of the intelligent agent in the intelligent decision-making layer. Both the status information and the reward value are transmitted to the intelligent agent through the northbound interface.

[0014] Among them, the process of taking the state information and the reward value as input, obtaining the action value and forwarding it is specifically that the intelligent agent receives the state information and reward value as input of the intelligent agent algorithm, uses the intelligent agent algorithm to generate the link weight in the network topology as the action value of the intelligent agent decision, which is used for network traffic scheduling strategy optimization, and then transmits the action value to the flow table issuer of the control layer through the northbound interface.

[0015] Among them, the intelligent agent algorithm designs a gated recurrent unit module, a self-attention mechanism module and a graph attention network module for feature extraction, wherein the gated recurrent unit module is used to capture the temporal related information of the input state, and the self-attention mechanism module is used to calculate the weight, and the graph attention network module is used to capture the spatial related information of the input state, so that the output action value finally has better optimization performance and generalization ability.

[0016] Among them, in the process of generating an SDN flow table based on the action value and sending the SDN flow table for traffic scheduling, the flow table sender of the control layer receives the action value sent by the intelligent body, and generates an SDN flow table in combination with the network topology information collected by the topology discoverer, and sends the SDN flow table to the SDN switch of the data exchange layer through the southbound interface.

[0017] The present invention provides a method for optimizing network traffic scheduling based on a proximal policy optimization algorithm. The data plane of a software-defined network has temporal and spatial characteristics. In order to globally consider the spatiotemporal correlation of the network, it is proposed to use a gated recurrent unit and a graph attention network method to extract the temporal correlation and spatial correlation in state information, so that deep reinforcement learning can pay attention to the spatial and temporal information of the network, and provide implicit information containing more environments for reinforcement learning decisions, thereby solving the network TE problem, making full use of network resources, and achieving high availability and good performance in load balancing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a schematic diagram of the ASTPPO system architecture of the present invention, which is a method for optimizing network traffic scheduling based on a proximal strategy optimization algorithm.

[0020] Figure 2 It is the intelligent agent architecture diagram in the ASTPPO basic model of the present invention.

[0021] Figure 3 Schematic diagram of input and output of the gated recurrent unit module of the present invention. DETAILED DESCRIPTION

[0022] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0023] The following are the corresponding abbreviations in the manual:

[0024] Proximal Policy Optimization based on Attention mechanism and Spatio-Temporal correlation (ASTPPO)

[0025] Gated Recurrent Unit (GRU):

[0026] Self-attention mechanism module (self-attention);

[0027] Graph Attention Network (GAT)

[0028] Multi-Layer Perceptron (MLP);

[0029] Deep Reinforcement Learning (DRL).

[0030] The present invention provides a method for optimizing network traffic scheduling based on a proximal policy optimization algorithm, comprising the following steps:

[0031] S1: Build the ASTPPO system architecture;

[0032] S2: Collect network link information and network topology information;

[0033] S3: Generate state information and reward value respectively;

[0034] S4: Taking the state information and the reward value as input, obtaining an action value and forwarding it;

[0035] S5: Generate an SDN flow table based on the action value, and send the SDN flow table to perform traffic scheduling.

[0036] Specifically, the ASTPPO system structure consists of three planes: data exchange layer, control layer and intelligent decision layer, such as Figure 1 As shown in the figure, the data exchange layer primarily consists of SDN switches that support SDN protocols (such as OpenFlow and P4) to forward network traffic data. The control layer, comprised of servers running SDN controller programs, implements network status collection, agent reward generation, network topology discovery, and SDN flow table delivery. The intelligent decision-making layer, comprised of servers running agent programs, makes network traffic scheduling decisions. The intermediate control layer connects to the data exchange layer via a southbound interface and to the intelligent decision-making layer via a northbound interface.

[0037] The present invention is further described in detail below in conjunction with the system structure and specific implementation steps:

[0038] S2. Collect network link information and network topology information. Specifically, the information collector at the control layer periodically collects network link information from the SDN switches at the data exchange layer via the southbound interface. The topology discoverer at the control layer periodically collects network topology information from the SDN switches at the data exchange layer via the southbound interface.

[0039] The network link information in step S2 includes the current network link flow rate T τ , network link utilization U τ , network link traffic demand P τ and action A in the previous time slice τ-1 , as shown in Formula 1.

[0040] S τ =[T τ ,U τ ,P τ ,A τ-1 ] 4×m (1)

[0041] Where m is the number of links between switches in the network; T τ is the set of actual traffic passing through each link in the network at time τ, as shown in Equation 2; U τ is the set of utilization rates of each link in the network at time τ, as shown in Equation 3. Link utilization is the ratio of the traffic passing through the link to the link bandwidth, as shown in Equation 4, where c i is the link bandwidth of the i-th link; P τ is the set of traffic that needs to pass through each link in the network at time τ, as shown in Equation 5.

[0042]

[0043]

[0044]

[0045]

[0046] S3. The information collector in the control layer processes network link information, generates state information for the intelligent decision-making layer agent, and transmits it to the agent via the northbound interface. The reward function module in the control layer processes the network link information collected by the information collector, generates a reward value for the intelligent decision-making layer agent, and transmits it to the agent via the northbound interface.

[0047] The state information of the intelligent agent in the intelligent decision-making layer in step S3 above includes the past l input sequences using the sliding window method. The agent input at time τ is formula 6, where x τ is the network link information at time τ, as shown in Equation 7.

[0048] X τ =[x τ-l ,x τ-l+1 ,…,x τ ] l×d×m (6)

[0049] x τ =S τ (7

[0050] The reward function described in step S3 is designed based on the two optimization goals of load balancing and congestion avoidance. Load balancing is to minimize the link utilization rate F τ ∈[0,1], the link utilization distance is the difference between the maximum and minimum utilization values ​​of all links, as shown in Equation 8; congestion avoidance is to minimize the congestion coefficient Ω τ ∈[0,1], the congestion coefficient is the ratio of the number of fully loaded links (i.e., the link utilization value is 1) to the total number of network links m, as shown in Equation 9.

[0051] F τ =max(U τ )-min(U τ ) (8)

[0052]

[0053] In the present invention, the full range of link utilization is set as a positive reward, while the congestion coefficient is set as a negative reward. For load balancing, the smaller the full range of link utilization, the more rewards the agent should get, so the relationship between the full range of link utilization and the reward is negatively correlated. The present invention designs the full range of link utilization to be set as a positive reward, which can motivate the agent to optimize load balancing. For congestion avoidance, the smaller the congestion coefficient, the more rewards the agent should get, so the optimization of the congestion coefficient and the reward are also negatively correlated. Since the tolerance for network congestion is low in reality, the present invention designs the congestion coefficient as a negative reward. The calculation formula of the reward function is shown in Formula 10, where α1, α2, α3, α4 and α5 are adjustable coefficients, which can adjust the range of the reward value R and the weights of the two optimization objectives.

[0054] R=α1-log((F τ +α2) α3 )-α4*(Ω τ ) α5 (10)

[0055] S4. The agent in the intelligent decision-making layer receives state information and reward values ​​as input to the agent algorithm. It uses the agent algorithm to generate link weights in the network topology as action values ​​for the agent's decision, which are used to optimize network traffic scheduling policies. The action values ​​are then transmitted to the flow table issuer in the control layer via the northbound interface.

[0056] The action value decided by the agent in step S4 above assigns a weight to each link, as shown in formula 11.

[0057]

[0058] The agent algorithm described in step S4 above adopts the actor-critic network architecture of the proximal policy optimization algorithm. Taking into account the temporal and spatial correlations of network information, three main components are designed: a gated recurrent unit module, a self-attention mechanism module, and a graph attention network module. These three components are responsible for different tasks in feature extraction and output implicit information with temporal and spatial correlations. The algorithm captures temporal information of the input state through a GRU. A self-attention mechanism is then used to calculate the weight of each element of the GRU output. This is then used through a fully connected (FC) network to obtain the input of a GAT. Finally, the GAT captures spatial information of the input state and outputs implicit information with temporal and spatial correlations. Simultaneously, a skip-connect method is used to feed the key raw information of the input state and the implicit information extracted through the feature extraction into an MLP. The attention mechanism is used to calculate the weights of the implicit information and the key raw information, balancing the weights of the implicit and key raw information. Finally, the actor network outputs the action value of the intelligent decision layer through the MLP layer, and the critic network outputs the estimated value required by the policy gradient algorithm through the MLP layer. The policy gradient algorithm is then used to update the learnable parameters of the actor and critic networks. The Actor and Critic networks use the same structure but do not share network parameters. The overall architecture is shown in the figure below. Figure 2 shown.

[0059] S4.1, use the GRU module to capture the temporal information of the state information. First, the state information X τ Input to the gated recurrent unit module to calculate and update the gate control signal z t and reset gate control signal r t , as shown in Equation 12 and Equation 13, where W z 、U z 、W r and U r is a learnable parameter, h t-1 is the output of the previous GRU unit, x t For X τThe tth time series signal. Then calculate the hidden layer output h of the current time step t′ , as shown in Equation 14, where W and U are learnable parameters. Finally, calculate the current GRU unit output h t , as shown in formula 15.

[0060] z t =σ(W z x t +U z h t-1 ) (12)

[0061] r t =σ(W r x t +U r h t-1 ) (13)

[0062] h t' =tanh(Wx t +r t ⊙Uh t-1 ) (14)

[0063] h t =z t ⊙h t-1 +(1-z t )⊙h t' (15)

[0064] The state information X in the present invention τ It contains a total of l network link information, with 2 layers of cyclic GRU units, such as Figure 3 As shown. Therefore, the output of the GRU module is the implicit sequence H∈R 2×l×d×m , the sequence contains time-related hidden information, as shown in Equation 16.

[0065] H=[h1,h2,…,h 2l ] 2×l×d×m (16)

[0066] S4.2. Use the self-attention mechanism module to calculate the weight of each element of the implicit sequence output by GRU. Input the implicit sequence H output in the previous step into the self-attention mechanism module, and first calculate the Q, K and V matrices, as shown in Equations 17, 18, and 19, where W Q 、W K and W V is a learnable parameter. Then calculate the attention of each matrix, as shown in Equation 20, and obtain the attention matrix H as shown in Equation 21 a Finally, H a Do Hadamard product with the implicit sequence H output by GRU to get the self-attention module output H A , as shown in Equation 22, where WA is the learnable parameter matrix.

[0067] Q=W Q H (17)

[0068] K=W K H (18)

[0069] V=W V H (19)

[0070]

[0071]

[0072] H A =W A (H⊙H a ) (twenty two)

[0073] S4.3, use the GAT module to capture the spatial correlation information of the state information. After the output H of the self-attention module A ∈R d×m Transpose to get GAT input H A' , as shown in Equation 23. Then calculate e ij , as shown in Equation 24, e ij Indicates the importance of the feature of node j to node i. Then e ij The attention coefficient α is obtained by leakyReLU activation function and softmax normalization ij , as shown in Equation 25, where W e is a learnable parameter. Finally, the attention coefficient is linearly transformed with the corresponding node feature, and then transformed by the σ activation function as the output of the node feature As shown in formula 26, where W h is a learnable parameter. All node features are the output H of GAT G , as shown in formula 27. G It is the implicit information with temporal and spatial correlation after feature extraction.

[0074]

[0075]

[0076] α ij =Softmax j (leakyReLU(e ij )) (25)

[0077]

[0078]

[0079] S4.4. Use the skip-connect method to capture the key raw information of the status information (i.e. link utilization U τ ), input it and the implicit information extracted by features into MLP, and calculate the weight of implicit information and key original information through attention mechanism to balance the proportion of implicit information and key original information. G Link utilization U with status information τ After concatenation, the attention mechanism is used to make the model pay more attention to a part of the direct features or implicit features, as shown in Equation 29, where H a' H τ The self-attention output is H τ Calculate the attention value H of the key original information and implicit information A' Then, it is input into the MLP layer, as shown in Equation 30, to obtain Y τ .

[0080] H τ =[H G ||U τ ] m×(d+1) (28)

[0081] H A' =W A (H τ ⊙H a' ) (29)

[0082]

[0083] S4.5, the Actor network takes the output Y of the MLP layer in the previous step τ Split into the input parameter mean matrix μ of the multidimensional normal distribution τ and the standard deviation matrix logstd τ , as shown in Equation 31. Then, actions are sampled from the generated distribution, as shown in Equation 32. The sampled output is then linearly mapped as the action of the agent's decision, as shown in Equation 33, with the mapping range being [α, β].

[0084]

[0085] A τ' =sample(NormalDistributions(μ τ , logstd τ )) (32)

[0086]

[0087] S4.6. Use the policy gradient of the proximal policy optimization algorithm to update the learnable parameters. The learning parameter update of the actor network is shown in Equation 34, where θ is the learning parameter of the actor network, α θ is an adjustable parameter, L CLIP (θ) is the loss function of the actor network, as shown in Equation 35, where E represents the expected cumulative return; r(θ) is the probability ratio between the new and old policies, as shown in Equation 36, where θ is the learning parameter of the actor network; A is the estimated advantage function at time t, which is estimated using the generalized advantage estimation (GAE) method, as shown in Equation 37, where λ and γ are adjustable parameters, N is the number of steps within the association range to be calculated, V(·) is the estimated value of the critic output, ω is the learning parameter of the critic network, and R is the reward at time t+1; ε is a hyperparameter that limits the range of allowed updates. When the probability ratio between the new and old policies falls outside the range of (1-ε) and (1+ε), the advantage function is clipped.

[0088]

[0089]

[0090]

[0091]

[0092] The learning parameter update of the Critic network is shown in Equation 38, where α ω is an adjustable parameter, and J(ω) is the loss function of the Critic network, as shown in Equation 39.

[0093]

[0094]

[0095] In step S5, the flow table dispatcher at the control layer receives the action value issued by the agent and, combined with the network topology information collected by the topology discoverer, generates a network traffic forwarding path. The forwarding path is then converted into an SDN flow table and dispatched to the SDN switch at the data exchange layer via the southbound interface. The SDN switch forwards network traffic based on the flow table information, dynamically adjusting network load, effectively alleviating network link congestion, and optimizing network performance.

[0096] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for optimizing network traffic scheduling based on a proximal strategy optimization algorithm, characterized in that: The following steps are involved: Build the ASTPPO system architecture; The ASTPPO system architecture includes a data exchange layer, a control layer and an intelligent decision layer. The data exchange layer is composed of an SDN switch that supports the SDN working protocol. The control layer is connected to the data exchange layer through a southbound interface and to the intelligent decision layer through a northbound interface. The intelligent decision layer outputs link weights for routing strategy updates. Collect network link information and network topology information; Generate status information and reward value respectively; The process of generating state information and reward value respectively is as follows: the information collector of the control layer processes the network link information to generate the state information of the intelligent agent in the intelligent decision layer; the reward function module of the control layer processes the network link information collected by the information collector to generate the reward value of the intelligent agent in the intelligent decision layer; both the state information and the reward value are transmitted to the intelligent agent through the northbound interface; Taking the state information and the reward value as input, obtaining an action value and forwarding it; The process of taking the state information and the reward value as input, obtaining the action value and forwarding it, specifically, the agent receives the state information and the reward value as input of the agent algorithm, uses the agent algorithm to generate the link weight in the network topology as the action value of the agent decision, which is used for network traffic scheduling strategy optimization, and then transmits the action value to the flow table sender of the control layer through the northbound interface; The agent algorithm designs a gated recurrent unit module, a self-attention mechanism module, and a graph attention network module for feature extraction, wherein the gated recurrent unit module is used to capture the temporal related information of the input state, the self-attention mechanism module is used to calculate the weight, and the graph attention network module is used to capture the spatial related information of the input state; An SDN flow table is generated based on the action value, and the SDN flow table is issued for traffic scheduling.

2. The method for optimizing network traffic scheduling based on a proximal strategy optimization algorithm as claimed in claim 1, characterized in that: In the process of collecting network link information and network topology information, the information collector of the control layer collects network link information from the SDN switch of the data exchange layer through the southbound interface at regular intervals, and the topology discoverer of the control layer collects network topology information from the SDN switch of the data exchange layer through the southbound interface at regular intervals.

3. The method for optimizing network traffic scheduling based on a proximal strategy optimization algorithm as claimed in claim 2, characterized in that: In the process of generating an SDN flow table based on the action value and sending the SDN flow table for traffic scheduling, the flow table sender of the control layer receives the action value sent by the intelligent agent, and generates an SDN flow table in combination with the network topology information collected by the topology discoverer, and sends the SDN flow table to the SDN switch of the data exchange layer through the southbound interface.