SDN routing method and architecture based on graph attention mechanism and dynamic priority replay

Abstract

The present invention discloses an SDN routing method and architecture based on a graph attention mechanism and dynamic priority replay, including real-time collection of network device and link status; generation of a set of alternative paths based on a shortest path algorithm; construction of a GAT-DPR model structure, training the model structure using the status information of the alternative paths as input, integrating the graph attention mechanism with the dynamic priority replay technology to optimize the training efficiency of the policy network and obtain the optimal routing strategy; converting the optimal routing strategy into a flow table and sending it to the data layer to complete traffic forwarding; and completing the corresponding operation steps by a network topology perception module, a network status detection module, an initial path calculation module, a path optimization module, and a path installation module in the control layer. Compared with existing algorithms, the GAT-DPR algorithm provided by the present invention has obvious advantages in improving throughput, reducing latency, and packet loss rate, and provides an innovative solution for efficient routing decision-making in tactical communication networks.

Description

Technical Field

[0001] The present invention belongs to the field of information engineering, and in particular relates to an SDN routing method and architecture based on a graph attention mechanism and dynamic priority playback. Background Art

[0002] As information warfare accelerates its evolution from network-centric warfare to intelligent warfare, higher demands are placed on the efficient transmission and reliable sharing of battlefield information. Routing decisions are a key technology for optimizing this goal. By improving routing strategies to optimize the transmission paths of battlefield information and achieve stable and reliable transmission of battlefield information, we not only ensure efficient coordination between combat units but also enhance commanders' comprehensive perception and dynamic control of the battlefield situation. Currently, routing algorithms are mainly categorized into traditional routing algorithms, heuristic routing algorithms, and deep reinforcement learning routing algorithms.

[0003] Traditional routing algorithms, such as Shortest Path First (SPF), Bellman-Ford (BF), and Equal Cost Multipath (ECMP), are suitable for static and stable civilian communication networks. However, they struggle to adapt to the high dynamics and complexity of tactical communication networks, resulting in rigid resource allocation and large fluctuations in end-to-end latency.

[0004] To address dynamic network demands, heuristic algorithms such as the greedy algorithm (GA), simulated annealing (SA), and ant colony optimization (ACO) generate near-optimal solutions through local search. However, these heuristic algorithms rely on a deterministic network state and are prone to being trapped in local optimality when faced with complex, high-dimensional tactical communication networks characterized by strong uncertainty. They also lack the ability to perceive global topological characteristics.

[0005] In recent years, deep reinforcement learning routing algorithms, such as the Deep Q Network (DQN), Proximal Policy Optimization (PPO), and Deep Deterministic Policy Gradient (DDPG), have emerged. By training intelligent agents to continuously interact with the environment, they accumulate operational experience and store it in an experience pool, making them well suited for dynamically changing tactical communication networks. While deep learning routing algorithms have shown potential in dynamic networks, their main drawbacks are: Traditional neural networks struggle to effectively model non-Euclidean topological relationships, resulting in poor generalization of node and link characteristics; and the experience pool utilizes random experience replay, which neglects the prioritization of critical operational experience.

[0006] Therefore, the challenges faced by existing routing algorithms in tactical communication networks are: it is difficult to cope with frequent changes in network topology; deep reinforcement learning routing algorithms have poor ability to represent unstructured data of network topology, resulting in uneven communication delays between nodes; insufficient priority adjustment in experience replay leads to inefficient experience replay. Summary of the Invention

[0007] Aiming at the high dynamics of node traffic load and the strong time-varying nature of link status in tactical communication networks, there are problems such as high routing computation overhead, long forwarding strategy cycle, and poor timeliness of replay pool samples. In this paper, an SDN (Software-Defined Networking) routing method and architecture based on Graph Attention Networks with Dynamic Prioritization Replay (GAT-DPR) is proposed.

[0008] In order to solve the above problems, the present invention adopts the following technical solutions:

[0009] An SDN routing method based on graph attention mechanism and dynamic priority replay is used for information transmission in tactical communication networks, including the following steps:

[0010] Step 1: Use network simulation tools to build a virtual tactical communication network topology, including defining terminal nodes and forwarding nodes, establishing logical connection links between node devices, and configuring parameters of the links between node devices, including bandwidth and latency.

[0011] Step 2: Based on the event-driven topology status monitoring mechanism, the network topology data of the data plane is periodically collected, including the number of nodes, the number of links, and the connection relationship, to build a network topology graph;

[0012] Step 3: Periodically collect and calculate data on each forwarding node device port in the data plane based on the southbound interface protocol, including the sending rate and receiving rate of the forwarding node device port, to obtain network status information;

[0013] Step 4: Based on the network topology and network status information, a shortest path algorithm is used to generate a set of candidate forwarding paths for each pair of source-destination nodes in the tactical communication network. The performance indicators of each forwarding path, including link remaining bandwidth, delay, packet loss rate, and feasible path node load variance, are calculated, and a structured path description file is output.

[0014] Step 5. Construct a deep reinforcement learning model structure based on a graph neural network (GAT-DPR model structure), including a graph attention network, a policy network, a target network, and an experience replay pool. Taking the structured path description file as input, the node and link features in the network topology graph are aggregated based on the graph attention network to obtain the global feature vector of the network topology graph. The global feature vector of the network topology graph is input into the policy network, and the action value of the network state at the current moment is output. The action with the highest value corresponding to the network state at the current moment is calculated, and the network environment is interacted with to obtain the network state at the next moment and the reward value at the current moment. A four-tuple is formed by the current network state, the action with the highest value corresponding to the network state at the current moment, the network state at the next moment, and the reward value at the current moment, and is stored in the experience replay pool as a sample. According to the dynamic priority replay and sampling mechanism, the policy network parameters are iteratively optimized, and the policy network parameters are periodically synchronized to the target network until the reward value reaches the maximum. The optimal routing strategy under the current network state is obtained and output.

[0015] Taking the structured path description file as input and the optimized routing strategy as output, the GAT-DPR model structure is trained to obtain the trained GAT-DPR model;

[0016] Step 6: Using the structured path description file of the tactical communication network to be optimized as input, apply the trained GAT-DPR model to obtain the optimized routing strategy; generate a flow table based on the optimized routing strategy and send it to the forwarding node device of the data plane to finally complete the traffic forwarding.

[0017] Furthermore, the graph attention network encodes the state information of the input alternative forwarding path into a node feature vector through linear transformation, calculates the attention weights of the node and its neighboring nodes through the self-attention mechanism, and then generates each node feature after weighted aggregation of neighboring node features. Finally, all node features are aggregated to obtain the global feature vector of the network topology graph as the input of the policy network;

[0018] Based on the global feature vector of the network topology, the policy network outputs the action value under the current network state through the fully connected layer of the policy network. This is used to calculate the action with the highest value corresponding to the current network state. The policy network then interacts with the network environment to obtain the network state at the next moment and the reward value at the current moment. A four-tuple is formed by the current network state, the action with the highest value, the network state at the next moment, and the reward value at the current moment, and is stored as a sample in the experience replay pool.

[0019] Until the number of stored samples reaches the capacity threshold of the experience replay pool, samples of a preset batch size are randomly extracted and used as the input of the policy network and the target network. The policy network and the target network are combined to calculate the temporal difference (TD) error of each sample, introduce an attenuation factor to dynamically adjust the priority of samples in the experience replay pool, adjust the sample sampling probability based on the priority weight, and update the policy network parameters based on the mean square error loss; the policy network parameters are periodically synchronized to the target network until the model structure converges, that is, the reward value reaches the maximum, and the optimal routing strategy is output.

[0020] The present invention also protects an architecture for implementing the above-mentioned SDN routing method, including an application layer, a control layer and a data layer; the application layer is connected to the control layer through a northbound interface; the control layer mainly includes a network topology perception module, a network status detection module, an initial path calculation module, a path optimization module and a path installation module, and the network topology perception module, the network status detection module, the initial path calculation module, the path optimization module and the path installation module respectively execute step 2, step 3, step 4, step 5 and step 6 in the SDN routing method; the data layer constructs a virtual network topology through a network simulation tool, and is connected to the control layer through a southbound interface to receive various control policies issued by the control layer.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The SDN routing method based on graph attention mechanism and dynamic priority replay provided by the present invention first uses a graph attention neural network to replace the traditional neural network in the reinforcement learning model, thereby enhancing the model's ability to learn and characterize node and edge features in the network; secondly, dynamic priority experience replay and priority weighted sampling mechanism are used to weaken the correlation between data in the experience pool and eliminate the deviation caused by uneven sampling. This improvement can optimize the problem of insufficient sample priority adjustment in the experience replay pool and ensure that the algorithm can converge faster and make routing decisions efficiently in complex and changeable tactical communication networks; compared with OSPF, DQN and Dueling DQN algorithms, the GAT-DPR algorithm of the present invention improves throughput by 24.28%, 13.12% and 7.17% respectively, reduces end-to-end delay by 37.21%, 26.49% and 23.02% respectively, and reduces packet loss rate by 13.16%, 8.08% and 7.62% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the method provided by the present invention.

[0024] Figure 2 This is the overall architecture diagram of SDN.

[0025] Figure 3 This is the principle diagram of the GAT-DPR algorithm.

[0026] Figure 4 This is a schematic diagram of the GAT module extracting features of the network topology graph.

[0027] Figure 5 This is the ablation experiment diagram of the GAT-DPR algorithm.

[0028] Figure 6 This is the GAT-DPR algorithm hyperparameter training experiment diagram.

[0029] Figure 7 It is a real tactical communication network topology diagram simulated by the present invention.

[0030] Figure 8 This is the performance analysis diagram of the GAT-DPR algorithm. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] like Figure 1 As shown, the SDN routing method based on graph attention mechanism and dynamic priority playback provided by the present invention is used for transmitting information in a tactical communication network, comprising the following steps:

[0033] Step 1: Use the mininet network simulation tool to build a tactical communication virtual network topology, including defining host nodes, switch nodes, and router nodes, establishing logical connection links between node devices, and configuring the parameters of the links between node devices such as bandwidth and latency; represent the constructed network topology as a graph. ,in is a collection of nodes, representing switches and routers in the network, It is a set of edges, representing links in the network topology;

[0034] Step 2: Use event listeners such as EventSwitchEnter and EventLinkAdd to periodically call the get_switch() and get_link() functions to collect network topology data of the data plane, including the number of nodes, the number of links, and the connection relationship, and build a network topology graph.

[0035] Step 3: Based on the port statistics request of the OpenFlow protocol, the data of each switch port in the data plane is periodically collected and calculated through the OFPPortStatsRequest message and the response event EventOFPPortStatsReply to obtain network status information;

[0036] Step 4: Based on the network topology and network status information, the Dijskra shortest path algorithm is used to calculate a set of K alternative forwarding paths for each pair of source-destination nodes in the tactical communication network. , and calculate the performance indicators of each alternative path including link remaining bandwidth, delay, packet loss rate and feasible path node load variance, and save them in the form of a dictionary as a JSON file;

[0037] Step 5: Construct a deep reinforcement learning model structure based on graph neural network, namely the GAT-DPR model structure, including graph attention network, policy network, target network, and experience replay pool; use the JSON file as input and the optimal routing strategy as output to train the constructed GAT-DPR model structure and obtain the trained GAT-DPR model; Figure 3 and Figure 4 As shown in Figure 2, the specific training process is as follows:

[0038] Step 5.1 Based on the status information of the input alternative path , using the graph attention network to the input state Perform linear changes to obtain re-encoded node features ; Calculate node features through self-attention mechanism Attention weights with neighbor node features ; Then use the attention weight to perform weighted summation on the features of neighboring nodes to obtain the aggregated node feature information ; Fuse the neighbor aggregation information with the current node feature information and execute T times to obtain the node feature after message transmission ; Aggregate the features of all nodes and output them as the feature vector of the entire network topology The specific calculation formula is as follows:

[0039] ,

[0040] ,

[0041] ,

[0042] ,

[0043] ,

[0044] ,

[0045] ,

[0046] ,

[0047] ,

[0048] ,

[0049] ,

[0050] ,

[0051] ,

[0052] ,

[0053] ,

[0054] ,

[0055] in Indicates source to destination node A set of candidate paths; Represents the structure of each candidate path, consisting of the source node ,go through Several intermediate links to reach the destination node ; 、 and Respectively represent the storage The state matrix of the remaining bandwidth, delay and packet loss rate of each path; Indicates the network Traffic load weight matrix of each node; Represents the adjacency matrix of all nodes between the source-destination node pair; Indicates the remaining bandwidth of the link, which is the bandwidth of all links in the path The minimum value of Represents the link delay, which is the delay of all links in the path The sum of is calculated; represents the link packet loss rate, which is calculated by subtracting the product of the successful transmission probabilities on all links from 1; Indicates the The traffic load weight of a node, the current traffic through the node The highest traffic in history The ratio of is calculated; Represents the variance of node load on the feasible path. The smaller the variance, the more evenly distributed the node load is, and the lower the probability of congestion on the feasible path is. is the initial feature of the input, Respectively nodes and The feature representation of a node, For the The set of neighbor nodes of a node, represents the weight matrix, Represents the relu activation function, represents the feature concatenation operation, Represents the attention weight of the neighboring nodes after the T-th message aggregation;

[0056] Step 5.2 Extract the feature vector of the network topology Input into the policy network and output the current network status The action value under ,pass Calculate the current network status The corresponding highest value action , interact with the network environment to obtain the network status at the next moment , the reward value at the current moment ,Depend on A quadruple is formed and stored as a sample in the experience replay pool; when the number of stored samples reaches the capacity threshold of the experience replay pool, N samples are randomly drawn from the experience replay pool. ; Take the extracted samples as the input of the policy network and the target network, and use the temporal difference TD error function to obtain the error value of each sample , introduce a decay factor to dynamically adjust the priority of samples ; Then use priority weighting to calculate the sample sampling probability , combined with the mean square error loss function to calculate the descent gradient And through the back propagation process of the neural network, the policy network parameters are updated according to the descending gradient; the policy network parameters are periodically synchronized to the target network until the model structure converges, that is, the reward value When the value reaches the maximum, the calculated path is the optimal forwarding path. The specific calculation formula is as follows:

[0057] ,

[0058] ,

[0059] ,

[0060] ,

[0061] ,

[0062] ,

[0063] in represents the state value function, represents the advantage function, represents the average of the advantage functions of all possible actions, represents the number of all possible actions, Represents the parameters of the policy network; represents the TD error, Indicates the current time Instant rewards; Represents the discount factor, which measures the importance of the current moment's immediate reward and future rewards; represents the initial priority of the sample, 、 Represents the modified sample ,sample Priority weight; represents the attenuation factor, Indicates the number of storage time steps of the sample in the experience replay pool, Indicates the sample priority adjustment coefficient corresponding to the storage time step; represents the sampling probability of the priority-weighted sample; represents the Q value calculated by the policy function, Represents the Q value calculated by the objective function; express The remaining bandwidth, delay, packet loss rate of the link and the variance of the node load on the feasible path at the moment, represents the weight matrix.

[0064] Step 6: Using the structured path description file of the tactical communication network to be optimized as input, apply the trained GAT-DPR model to obtain the optimized routing strategy; generate the OpenFlow flow table based on the optimized routing strategy and send it to each switch in the data plane to complete traffic forwarding.

[0065] like Figure 2 As shown in the figure, the architecture for implementing the above-mentioned SDN routing method includes an application layer, a control layer and a data layer; the application layer is connected to the control layer through a northbound interface; the control layer mainly includes a network topology perception module, a network status detection module, an initial path calculation module, a path optimization module and a path installation module, and the network topology perception module, the network status detection module, the initial path calculation module, the path optimization module and the path installation module respectively execute step 2, step 3, step 4, step 5 and step 6 in the above-mentioned SDN routing method; the data layer constructs a virtual tactical communication network topology through the mininet network simulation tool, and is connected to the control layer through a southbound interface to receive and execute various control policies issued by the control layer.

[0066] Figure 5 This is an ablation experiment on the attention mechanism designed by the present invention. The experiment shows that adding the attention mechanism to the model structure can ensure that the model converges faster and has a greater reduction when other experimental conditions are consistent.

[0067] like Figure 6 The hyperparameter training results of the GAT-DPR algorithm shown in (6-1), (6-2), (6-3), (6-4), (6-5), (6-6), and (6-7) are shown in Table 1. The optimal hyperparameters of the GAT-DPR algorithm are shown in Table 1:

[0068] Table 1 Optimal hyperparameters of the GAT-DPR algorithm

[0069]

[0070] Figure 7 This is the real tactical communication network topology diagram used in this invention. It is a complex tactical communication network composed of multiple link types including optical fiber, wired Ethernet, microwave links and regional broadband networks. The main difference between each link is the communication bandwidth. By setting differentiated bandwidth parameters for different links in the mininet network simulation software, a simulation of the real network environment is achieved.

[0071] Figure 8 The present invention is based on Figure 7 The network topology structure is used to compare the performance of four different routing algorithms under different traffic demand scenarios; Figure 8As shown in (8-1), (8-2) and (8-3) in the figure, the GAT-DPR algorithm of the present invention outperforms the other three algorithms in core indicators including delay, average throughput and packet loss rate, verifying its robustness and efficiency under dynamic traffic load.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made by any person skilled in the art within the technical scope disclosed in the present invention and based on the technical solution and inventive concept of the present invention shall be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An SDN routing method based on graph attention mechanism and dynamic priority replay for information transmission in tactical communication networks, characterized by: The following steps are involved: Step 1: Use network simulation tools to build a virtual tactical communication network topology, including defining terminal nodes and forwarding nodes, establishing logical connection links between node devices, and configuring parameters of the links between node devices, including bandwidth and latency. Step 2: Based on the event-driven topology status monitoring mechanism, the network topology data of the data plane is periodically collected, including the number of nodes, the number of links, and the connection relationship, to build a network topology graph; Step 3: Periodically collect and calculate data on each forwarding node device port in the data plane based on the southbound interface protocol, including the sending rate and receiving rate of the forwarding node device port, to obtain network status information; Step 4: Based on the network topology and network status information, a shortest path algorithm is used to generate a set of candidate forwarding paths for each source-destination node pair in the tactical communication network. The performance indicators of each forwarding path, including link remaining bandwidth, delay, packet loss rate, and feasible path node load variance, are calculated, and a structured path description file is output. Step 5: Construct the GAT-DPR model structure, including the graph attention network, policy network, target network, and experience replay pool; using the structured path description file as input, the graph attention network aggregates the node and link features in the network topology graph to obtain the global feature vector of the network topology graph; The global feature vector of the network topology is input into the policy network, and the action value of the current network state is output. This is used to calculate the action with the highest value corresponding to the current network state. The network state at the next moment and the reward value at the current moment are obtained by interacting with the network environment. A four-tuple is formed by the current network state, the action with the highest value corresponding to the current network state, the next network state, and the current reward value, and is stored in the experience replay pool as a sample. Based on the dynamic priority playback and sampling mechanism, the policy network parameters are iteratively optimized and periodically synchronized to the target network until the reward value reaches the maximum. The optimal routing strategy under the current network state is obtained and output. Taking the structured path description file as input and the optimized routing strategy as output, the GAT-DPR model structure is trained to obtain the trained GAT-DPR model; Step 6: Using the structured path description file of the tactical communication network to be optimized as input, apply the trained GAT-DPR model to obtain the optimized routing strategy; generate a flow table based on the optimized routing strategy and send it to the forwarding node device of the data plane to finally complete the traffic forwarding.

2. The SDN routing method based on graph attention mechanism and dynamic priority playback according to claim 1 is characterized in that The graph attention network encodes the state information of the input alternative forwarding path into a node feature vector through linear transformation, calculates the attention weight of the node and its neighboring nodes through the self-attention mechanism, and then generates each node feature after weighted aggregation of neighboring node features. Finally, all node features are aggregated to obtain the global feature vector of the network topology graph as the input of the policy network; The policy network is based on the global feature vector of the network topology graph. It outputs the action value under the current network state through the policy network's fully connected layer. It calculates the action with the highest value corresponding to the current network state and interacts with the network environment to obtain the network state at the next moment and the reward value at the current moment. A four-tuple is formed by the current network state, the action with the highest value, the next network state, and the current reward value, and is stored in the experience replay pool as a sample; Until the number of stored samples reaches the capacity threshold of the experience replay pool, samples of a preset batch size are randomly extracted and used as the input of the policy network and the target network. The policy network and the target network are combined to calculate the temporal difference (TD) error of each sample, introduce an attenuation factor to dynamically adjust the priority of samples in the experience replay pool, adjust the sample sampling probability based on the priority weight, and update the policy network parameters based on the mean square error loss; the policy network parameters are periodically synchronized to the target network until the model structure converges, that is, the reward value reaches the maximum, and the optimal routing strategy is output.

3. The architecture of the SDN routing method based on graph attention mechanism and dynamic priority playback according to claim 1 is characterized in that: It includes application layer, control layer and data layer; the application layer is connected to the control layer through the northbound interface; the control layer mainly includes network topology perception module, network status detection module, initial path calculation module, path optimization module and path installation module, and the network topology perception module, network status detection module, initial path calculation module, path optimization module and path installation module respectively execute step 2, step 3, step 4, step 5 and step 6 in the SDN routing method; the data layer builds a virtual network topology through a network simulation tool, and is connected to the control layer through a southbound interface to receive various control policies issued by the control layer.

Images