Intelligent software defined wireless network routing method based on network situation awareness

Abstract

The application discloses an intelligent software-defined wireless network routing method based on network situation awareness, comprehensively perceives various network state information by using an SDWN architecture, predicts a traffic state of a next stage through a GCN-GRU prediction model, designs a next-hop adjacent node as an action space of reinforcement learning, designs different reward functions according to link information and a reward and punishment mechanism, and updates network parameters by using a loss function based on importance sampling and gradient clipping. The application can more comprehensively meet QoS service requirements, and has the characteristics of fast convergence speed, high reliability and strong stability.

Description

Technical Field

[0001] This invention relates to the field of software-defined wireless network (SDWN) technology, and specifically to an intelligent software-defined wireless network routing method based on network situation awareness. Background Technology

[0002] In recent years, Wireless Local Area Networks (WLANs) have developed rapidly, and the number of mobile terminals has grown exponentially. More and more places are providing wireless access points for people to connect to the internet, and the demand for wireless network resources from mobile terminals is increasing. Traditional WLAN architectures face numerous challenges. In traditional wireless network frameworks, the tight coupling of data forwarding and control management limits network scalability, requiring network devices to integrate multiple network functions, resulting in a bloated network architecture that is detrimental to network management and maintenance. As the number of users increases and the network scale expands, existing network bandwidth resources are increasingly unable to meet user demands. Simply expanding and speeding up network hardware not only incurs extremely high costs but also struggles to meet the rapidly growing network requirements. Simultaneously, traditional distributed network architectures have limitations in management and deployment, and deploying new network services can easily lead to network complexity and heterogeneity. Sudden surges in consumer traffic and rapid mobility cause significant latency in wireless networks. Considering these factors, designing cost-effective wireless solutions with efficient resource utilization and manageability is extremely challenging.

[0003] Software-defined wireless networks (SDWNs) are considered a reliable solution. SDN is an emerging wireless network framework that extends beyond traditional software-defined networking (SDN). It breaks away from the vertical structure of traditional wireless network models, separating complex control functions from traditional wireless network devices and centralizing them logically on the controller. This decouples data forwarding and control management, achieving direct programmability of centralized network logic control and abstraction of the underlying wireless infrastructure, thus enabling software-managed networks. The SDWN controller performs fine-grained scheduling of network resources from a global perspective. It uses the OpenFlow protocol on the southbound interface for programmable control of the data plane, enabling flow table entry distribution and network environment status information collection. The northbound interface provides an open interface for the network application layer. These advantages allow for the decoupling of data forwarding and control management in SDWN networks, enabling routing algorithms to reschedule and efficiently utilize network resources.

[0004] Routing algorithms aim to find the optimal routing strategy from source nodes to destination nodes in a network topology. Their performance determines the quality of service (QoS) of the network. Therefore, designing an efficient and stable routing algorithm is crucial for improving the overall performance and resource utilization of software-defined wireless networks (SDWNs). In the development of traditional network architectures, traditional routing algorithms such as Open Shortest Path First (OSPF), Routing Information Protocol (RIP), and Valiant Load Balancing (VLB) have been successfully applied in many fields. Furthermore, some scholars have modeled the routing optimization problem as a linear programming (LP) or nonlinear programming (NLP) problem and proposed using classic routing optimization algorithms to find the optimal routing strategy. However, as network scale continues to increase and network traffic exhibits exponential growth, traditional network architecture routing algorithms, utilizing only limited network information, cannot generate the optimal routing strategy based on the global state of the SDWN network. Secondly, traditional network architecture routing algorithms suffer from slow convergence speed, long response time, and poor adaptability. These problems become more pronounced when the SDWN network is large-scale, clearly indicating that traditional network architecture routing algorithms are no longer suitable for SDWN network architecture. Furthermore, traditional routing algorithms are designed for distributed network processing units, which contradicts the centralized network control of SDWN. Summary of the Invention

[0005] The present invention addresses the problem that traditional network architecture routing algorithms are unable to fully meet QoS service requirements and have slow convergence speed, and provides an intelligent software-defined wireless network routing method based on network situation awareness.

[0006] To solve the above problems, the present invention is achieved through the following technical solution:

[0007] The intelligent software-defined wireless network routing method based on network situation awareness includes the following steps:

[0008] Step 1: Initialize the parameters θ of the Actor network and ψ of the Critic network, set the maximum number of iterations M, and clear the current experience buffer; set the initial node of the agent to the source node N. s ;

[0009] Step 2: First, obtain the network topology information and the current and historical network state information of SDWN; then, convert the network topology information into a network adjacency matrix Ag, and simultaneously convert the current and historical network state information into a network state matrix. The network state matrix at the current and historical moments. The data is fed into the GCN-GRU prediction model to predict the network state matrix at the next time step. Then, the network adjacency matrix Ag and the network state matrix are... An environment is created for interaction with the intelligent agent; where n is the set number of historical moments.

[0010] Step 3: The agent interacts with the environment to obtain the current state s. t ; where the current state s t The network state matrix at the current moment and the routing matrix at the current time composition;

[0011] Step 4: The agent sets the current state s t Inputting into the current Actor network yields the current action 'a'. t And the policy probability distribution π(a) of the current action. t |s t );

[0012] Step 5: The agent executes the current action *at* and interacts with the environment to obtain the current reward value *R*. t and the next state s t+1 The next state s t+1 The network state matrix at the next time step and the routing matrix at the next time step composition;

[0013] Step 6: The agent sets the current state s t Current action a t Current reward value R t The policy probability distribution π(a) of the current action t |s t ) and the next state s t+1 The formed sample trajectory (s) t a t R t ,π(a t |s t ), s t+1 Place it in the current experience cache;

[0014] Step 7: The agent samples the sample trajectory from the current experience buffer and selects the current state s from the sampled sample trajectory. t and the next state s t+1 The input is given to the current Critic network to obtain the current state value function V(s). t ) and the next state value function V(s) t+1 );

[0015] Step 8: Construct the advantage function and update the parameters θ of the current Actor network by maximizing the advantage function; where the advantage function A t for:

[0016] A t =R t +γV(s t+1 )-V(s t )

[0017] Step 9: Clear the current experience cache and repeat steps 4-6;

[0018] Step 10: Construct a loss function, and update the parameters θ of the current Actor network and the parameters ψ of the current Critic network by minimizing the loss function; where the loss function L total (θ, ψ) are:

[0019]

[0020] Step 11: The agent first selects the policy probability distribution π(a) from the current experience buffer. t |s t The largest current action a t and execute the current action a. t Then determine whether the agent should perform the current action a. t Is the selected next node the destination node N? d If yes, it means the current iteration is complete, and proceed to step 12; otherwise, it means the current iteration is not complete, and the agent will execute the current action a. t The selected next node is used as the agent's current node. After clearing the current experience cache, proceed to step 3.

[0021] Step 12: Determine if the number of iterations has reached the set maximum number of iterations M: If so, construct the routing matrix for the current time step obtained in the last round. Output the optimal path; otherwise, proceed to step 2.

[0022] In the above formula, A t R is the dominance function. t V(s) represents the current reward value, γ represents the given state value weight, and V(s) represents the current reward value. t V(s) is the current state value function. t+1 ) is the next state value function; L total (θ, ψ) is the loss function, E[■] represents the expectation function, and min(■) represents the minimum value function. denoted as the weight ratio for importance sampling, ∈ as a given clipping factor, and c1 as a given value factor.

[0023] In step 2 above, the network status information includes the remaining bandwidth of the link, the link delay, the link packet error rate, the link packet drop rate, and the link distance.

[0024] In step 5 above, the agent executes the current action a. t Subsequently, the following three situations may occur during the interaction with the environment:

[0025] ① When the agent executes the current action a t If the next node selected is the adjacent node of the current node, and the path traversed by the agent has no loops, the current reward value R is... t for:

[0026] R t =β1bw ij -β2delay ij -β3 errors ij -β4 drop ij -β5 distance ij

[0027] ②When the agent executes the current action a t If the next node selected is an adjacent node of a safe node, and the path traversed by the agent contains a loop, the current reward value R is... t for:

[0028] R t =-ξ1R s

[0029] ③ When the agent executes the current action a t When the selected next node is not an adjacent node of the current node, the current reward value R is... t for:

[0030] R t =-ξ2R s

[0031] In the formula, bw ij β1 represents the remaining bandwidth of the link between node i and node j, and delay represents the remaining bandwidth weight. ij β2 represents the link delay between node i and node j, and β2 represents the link delay weight. Errors ij β3 represents the packet error rate of the link between node i and node j, and drop represents the packet error rate weight. ij β4 represents the packet loss rate of the link between node i and node j, and distance represents the link packet loss rate weight. ijR represents the link distance between node i and node j, and β5 represents the link distance weight; s Given a standard reward, ξ1 is a given first discount factor, ξ2 is a given second discount factor, and ξ1, ξ2, R s ∈[0,1].

[0032] In step 10 above, the weight ratio of importance sampling for:

[0033]

[0034] In the formula, π new (a t |s t Let π be the updated policy probability distribution of the Actor network. old (a t |s t ) represents the policy probability distribution before the Actor network is updated.

[0035] Compared with the prior art, the present invention has the following characteristics:

[0036] 1) Unlike traditional algorithms that consider a single and limited set of constraints, this invention utilizes the SDWN architecture to achieve comprehensive perception of various network state information and uses this network state information as constraints. This enables a more comprehensive fulfillment of QoS service requirements, lays the foundation for efficient and flexible data transmission, and effectively reduces network congestion.

[0037] 2) Unlike traditional algorithms that determine routes based on network state information measured at fixed intervals, this invention designs and implements a network situation awareness mechanism based on the SDWN architecture through the GCN-GRU prediction model. This mechanism comprehensively considers the spatiotemporal characteristics of wireless network traffic and uses current and historical traffic state data of data streams to predict the traffic state of the next stage. It can make full use of the hidden state information in the wireless network to improve the reliability and stability of the intelligent routing algorithm.

[0038] 3) Unlike k-path action design and single reward function design, this invention enhances the interaction between the agent and the network environment by designing a novel next-hop neighbor node as the action space for reinforcement learning. This makes the routing algorithm more intelligent to adapt to the changes of dynamic and complex networks. Different reward functions are designed according to link information and reward and punishment mechanisms to reduce the situation where the agent gets stuck in local optima, thereby improving the reliability and effectiveness of the intelligent routing algorithm.

[0039] 4) To address the issues of mismatch between the sampled data probability distribution and the current agent's strategy (resulting in low sample validity) and network gradient explosion, this invention replaces the Priority Experience Replay (PER) and KL divergence loss functions with an importance (IS) sampling and gradient clipping loss function during the network parameter update process, thereby improving the convergence speed and stability of the intelligent algorithm. Attached Figure Description

[0040] Figure 1 This is a system architecture diagram of the SDWN system of the present invention.

[0041] Figure 2 This is a schematic diagram of the routing algorithm of the present invention.

[0042] Figure 3 This is a schematic diagram of the network state matrix.

[0043] Figure 4 This is a schematic diagram of the routing matrix.

[0044] Figure 5 This is a flowchart of the routing algorithm of the present invention.

[0045] Figure 6 A diagram illustrating action selection for deep reinforcement learning. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific examples.

[0047] The system architecture of the intelligent software-defined wireless network routing method based on network situation awareness is as follows: Figure 1 As shown, it consists of an infrastructure data plane, a logical control plane, a knowledge plane, and an application layer.

[0048] 1) Infrastructure Data Plane:

[0049] The infrastructure data plane mainly consists of wireless network devices, such as Wireless Access Points (WAPs) and Stations (STAs). The infrastructure data plane primarily performs packet lookup and forwarding, and completes packet parsing and flow table matching. These devices respond to requests sent from the logical control plane via the southbound interface protocol OpenFlow.

[0050] The data plane connects to the control plane via the southbound interface using the OpenFlow protocol. The controller obtains network topology information by sending Link Layer Discovery Protocol (LLDP) messages to the data plane. The controller periodically sends LLDP packet-out messages to connected SDWN radio access points and uses the returned packet-in messages to detect the information of the radio access points, thereby completing the detection of the radio access point's working status and updating the network topology view.

[0051] 2) Logic control plane:

[0052] Since SDWN is a method for centralized control of wireless networks using software-defined networking technology, the rules defined by these programs (often called controllers) determine the network's behavior. SDWN decouples the wireless control plane and data plane, including simplified wireless access devices and forwarding devices, which operate according to the rules scheduled by the logically centralized control plane. The controllers in the control plane periodically collect information from the data plane via the southbound interface protocol, such as network topology, link bandwidth, link round-trip time, link packet loss rate, and distances between wireless access points.

[0053] The controller's main functions include link discovery, topology management, policy formulation, and entry distribution. Link discovery and topology management involve the controller using the uplink channel to uniformly monitor and statistically analyze information reported by the underlying wireless access point devices. Policy formulation and entry distribution involve the controller using the downlink channel to uniformly control network devices. Link discovery aims to obtain global SDWN network information and is the foundation for network address learning, VLANs, and route forwarding. Unlike traditional networks where link discovery is performed autonomously by individual network elements, in SDWN networks, link discovery is uniformly completed by the controller through the Link Layer Discovery Protocol (LLDP). Topology management involves continuously collecting and monitoring information from SDWN wireless access points and providing feedback on their operational status and link connectivity. The controller periodically sends LLDP packet-out messages to connected SDWN wireless access points and uses the returned packet-in messages to detect access point information, thereby updating the network topology view and detecting the operational status of wireless access points. The controller formulates corresponding forwarding policies and generates corresponding flow entries based on the transmission requirements at different layers. Through deep reinforcement learning designed in the knowledge plane, the optimal routing strategy is learned based on the current network state information, and finally, flow table entries are distributed to generate optimized SDWN forwarding routes.

[0054] Unlike other inventions that only consider a single network state information, this invention considers the network state information to include more than one type of network state information determined according to QoS service requirements, such as link remaining bandwidth, link latency, link error rate, link packet rejection rate, and link distance.

[0055] 3) Knowledge plane:

[0056] The knowledge plane is a combination of policy gradient-based reinforcement learning and traffic prediction algorithms. Loading the knowledge plane into the SDWN framework makes the controller more intelligent in policy decision-making. Training the knowledge plane model requires pre-acquiring data from an information pool and transforming network state information into a traffic matrix. This matrix is ​​then used for prediction by a traffic prediction module, which consists of two parts: a Graph Convolutional Neural Network (GCN) and a Gated Recurrent Unit (GRU). First, the GCN acquires the spatial features of the wireless network topology. Second, the time series data with spatial features is input into the GRU network to extract the temporal features. Finally, the prediction is filtered through a fully connected layer to output the predicted results. The acquired predicted traffic data is synchronized to the reinforcement learning network for training. The agent optimizes the executed policy based on the currently acquired network state information, aiming to maximize the reward, and dynamically adjusts the optimal routing path until the model converges and stabilizes. Finally, the trained model is deployed to the SDWN controller to obtain the optimal routing policy and then responds to the data plane through the southbound interface, thus completing the data flow table forwarding.

[0057] 4) Application plane

[0058] The application plane manages all business-related applications and features programmable APIs. The controller interacts with the application plane via a northbound interface protocol to enable the development and deployment of various network applications, such as load balancing, fast switching, and interference management.

[0059] To better optimize network performance and improve the stability and effectiveness of intelligent routing algorithms, this invention designs a network situational awareness architecture based on the SDWN intelligent routing algorithm. This architecture monitors and acquires traffic information and network topology in the wireless network, predicts future network traffic trends, and analyzes the network situation from a global perspective, providing accurate data support for the routing algorithm. Simultaneously, deep reinforcement learning methods are combined to find the optimal wireless routing forwarding path, improving network bandwidth utilization and reducing network latency, thereby ensuring the overall communication quality of the network. Figure 2 As shown.

[0060] Situational awareness refers to the perception of environmental elements and events in time or space, the prediction of their future state, and the search for the optimal solution on a specific timeline by processing existing information. The application of network situational awareness in intelligent routing mainly involves sensing and analyzing real-time data from wireless networks to optimize network topology and wireless network traffic scheduling, thereby improving network throughput and response speed, and providing crucial protection for network security. The network situational awareness architecture of this invention comprises three layers: a perception layer, an understanding layer, and a prediction layer. The perception layer primarily collects and processes data, corresponding to the data plane of SDWN. The understanding layer corresponds to the control plane in the SDWN framework, where the Ryu controller understands and projects the data from the perception layer and assesses the current network situation. The prediction layer, as the highest layer of situational awareness, is responsible for perceiving and understanding various elements of the network environment and predicting future traffic trends, corresponding to the traffic prediction model of this invention.

[0061] The wireless network topology in the infrastructure data plane is abstracted into an undirected graph with relevant link parameter weights, denoted by G = (V, E, W). Here, V is the set of wireless access points, E is the set of links between nodes, W is the set of link parameter information, and each edge e represents a link, e ∈ E. The routing algorithm of this invention is designed based on the undirected graph. First, the link information of the network, including parameters such as bandwidth, latency, and packet loss rate, is obtained according to the SDWN topology and converted into a multi-dimensional matrix. Then, a predicted traffic matrix is ​​obtained through a traffic prediction model, and finally, the predicted traffic matrix is ​​used as input to train the DRL algorithm. The DRL algorithm consists of two major networks: an Actor Network and a Critic Network. The former optimizes the policy by maximizing the expected reward, while the latter is used to evaluate the policy value in the current state. They cooperate to achieve efficient policy optimization. The AC network architecture ensures that the agent always makes decisions in the direction of maximizing the cumulative reward, while dynamically optimizing parameters such as network latency, jitter, bandwidth, and throughput, achieving real-time control of the current network and effectively reducing network load.

[0062] The network traffic composition in SDWN is complex and multifaceted, making single-method traffic tracking and statistics too costly and ineffective. Most studies on SDWN network traffic focus on the traffic matrix. This invention combines GCN and GRU, simultaneously considering the spatiotemporal characteristics of the data and enhancing the agent's ability to perceive future moments. First, network weights are randomly initialized, and the traffic matrix is ​​standardized. Next, the adjacency matrix is ​​written into the topology graph, and the degree matrix parameters in the prediction network convolution are calculated. Finally, the network training iteratively generates hidden layer H parameters based on the GCN-GRU network, and the final prediction result is selected as the output through a fully connected layer. During training, the network weight parameters are updated using the minimum quantized true value and the squared difference between the traffic prediction and the actual value. The predicted traffic matrix output by the GCN-GRU network traffic prediction algorithm serves as the input to the reinforcement learning algorithm.

[0063] The key to optimizing wireless network routing problems using deep reinforcement learning (DRL) algorithms lies in designing a state space, reward function, and action space that perfectly match the current environment. The state space represents the current state of the DRL agent as observed by the environment. The agent then learns and updates its state S by perceiving the current network state information, guided by the reward function R. t Perform action A t Then it will enter the next state S t+1 According to S t+1 and A t Obtain the corresponding reward value R t+1 When the agent obtains the maximum reward value, that is, the path decided by the agent is the optimal path.

[0064] 1) State space S:

[0065] The state space is the current state of the network environment as perceived by the agent, including the network state matrix M. info and routing matrix M l .

[0066] In this embodiment, the network state matrix M info It consists of a traffic matrix related to the remaining bandwidth of the link, a traffic matrix related to the link delay, a traffic matrix related to the link error rate, a traffic matrix related to the link packet rejection rate, and a traffic matrix related to the link distance.

[0067] See Figure 3In each traffic matrix, row number i and column number j represent the AP node number. When i ≠ j, the element in the i-th row and j-th column of the matrix represents the link information between node i and node j (i.e., link remaining bandwidth, link delay, link error rate, link packet rejection rate, and link distance). When i = j, the element in the i-th row and j-th column is denoted as an empty nan.

[0068] See Figure 4 Routing matrix M l The row number i and column number j represent the AP node number. It's a diagonal matrix; when i ≠ j, the element in the i-th row and j-th column is 0. When i = j, the element in the i-th row and j-th column represents the node's position in the routing path. `start` represents the source node of the path, `end` represents the destination node of the path, and `dst`... i This represents the i-th hop node of the path.

[0069] 2) Action Space A:

[0070] The action space is a set of actions that an agent needs to choose during its interaction with the environment. After performing an action based on the current state, the agent will enter the next state. In this invention, the state space is designed to consist of location information and network information. This design aims to enable the agent to quickly find the optimal path in the environment, improve the agent's interactivity with the environment, and reduce the dimensionality of actions in the action space. When making action selections, the agent needs to consider the following three scenarios, and corresponding decision-making schemes are also provided: ① When the agent performs the current action a... t The selected next node is the adjacent node of the current node, and the path traversed by the agent has no loops; ② When the agent executes the current action a t The selected next node is an adjacent node of the safe node, and the path traversed by the agent contains a loop; ③ When the agent executes the current action a t The selected next node is not an adjacent node of the current node.

[0071] 3) Reward function R:

[0072] The reward function is used to optimize the learning direction of the agent. In the algorithm design of this invention, the agent always makes routing decisions in the direction of maximizing the reward value, in order to comprehensively consider the overall performance of the network and thus achieve the goal of multi-objective routing optimization. During the interaction with the environment, the agent will encounter three situations, and the next-hop reward value is also split into three different rewards. Therefore, the agent will accumulate and learn by matching the corresponding reward value according to the current environmental situation.

[0073] ① When the agent executes the current action a t If the next node selected is the adjacent node of the current node, and the path traversed by the agent has no loops, the current reward value R is... t for:

[0074] R t =β1bw ij -β2 delay ij -β3 errors ij -β4 drop ij -β5 distance ij

[0075] ②When the agent executes the current action a t If the next node selected is an adjacent node of a safe node, and the path traversed by the agent contains a loop, the current reward value R is... t for:

[0076] R t =-ξ1R s

[0077] ③ When the agent executes the current action a t When the selected next node is not an adjacent node of the current node, the current reward value R is... t for:

[0078] R t =-ξ2R s

[0079] In the formula, bw ij β1 represents the remaining bandwidth of the link between node i and node j, and delay represents the remaining bandwidth weight. ij β2 represents the link delay between node i and node j, and β2 represents the link delay weight. Errors ij β3 represents the packet error rate of the link between node i and node j, and drop represents the packet error rate weight. ij β4 represents the packet loss rate of the link between node i and node j, and distance represents the link packet loss rate weight. ij R represents the link distance between node i and node j, and β5 represents the link distance weight; s Given a standard reward, ξ1 is a given first discount factor, ξ2 is a given second discount factor, and ξ1, ξ2, R s ∈[0,1].

[0080] 4) PPO update strategy:

[0081] In policy evaluation, we employ importance sampling to estimate the objective function for policy updates. By adjusting the weights of older samples, we can more accurately estimate policy performance during policy updates. Compared to reinforcement learning methods that do not use importance sampling, this approach allows us to estimate policy performance using data from past sampling trajectories and avoids weight mismatch issues. Furthermore, gradient pruning is used in both the old and new policy updates.

[0082] Considering that the gradient descent update algorithm commonly used in deep learning to update network parameters is not entirely accurate and is subject to data bias, the advantage evaluation function estimates values ​​that are not perfect. If the data acquired by the agent is biased, and the policy deviates significantly from the actual value after one policy execution, the next sampling will also deviate completely from the estimate, leading to a policy that completely deviates from the predicted policy. Furthermore, due to the varying distribution of training data, the Actor network parameters θ will change after learning, becoming θ′. Therefore, this invention employs an importance sampling method to adjust the data distribution, avoiding problems caused by data estimation bias, thereby improving policy update efficiency and sample utilization. The weight ratio of importance sampling is... for:

[0083]

[0084] In the formula, π new (a t |s t Let π be the updated policy probability distribution of the Actor network. old (a t |s t Let ) represent the policy probability distribution of the Actor network before the update, and a t and s t Then, these represent the action and state at time t, respectively.

[0085] In importance sampling, in order to avoid the original data distribution p(x) and the sampleable data distribution q(x) being too different, a constraint term σ needs to be added to the distance between them -- KL divergence. Because the loss function of the reinforcement learning PPO algorithm is with relevant condition constraints and is calculated using the conjugate gradient method, while the KL divergence constraint, as an additional constraint condition, does not participate in the update of the convolutional neural network parameters, it is difficult to dynamically adjust the parameters in the KL divergence to adapt to different data distributions. Therefore, KL divergence is put into the loss function as a regularization term for optimization. However, since the KL divergence formula is relatively complex to calculate and it is difficult to choose a suitable penalty factor σ to adjust the similarity between the policies before and after, in order to better adapt to dynamic network topology changes and uneven data distribution, and at the same time effectively solve the problem of gradient explosion or gradient vanishing, the algorithm of this invention uses gradient clipping (Clip optimization) to replace KL divergence, which not only effectively limits the policy update amplitude, but also improves the convergence speed and performance of the algorithm model. The specific gradient clipping operation is as follows: (1) If Then the pruning result is set to 1-∈. This is done to limit the policy update from deviating too far from the old policy and to avoid making excessive policy updates. (2) If Then the pruning result is set to 1+∈, which also prevents the strategy from making excessive updates. (3) If Then keep the cropping result within this range. The value will be unrestricted and can be directly used to calculate the loss function. During training, the pruning factor ∈ is set to 0.2, which effectively limits the policy update magnitude when selecting a new policy and improves the convergence speed and stability of the algorithm model.

[0086] In this invention, the update of the Actor network parameter θ and the Critic network parameter ψ in deep learning is achieved by minimizing the loss function, that is, using a loss function based on importance (IS) sampling and gradient clipping. The update formula maximizes the upper limit of the expected cumulative reward while limiting the magnitude of policy update, so as to ensure the convergence and stability of the algorithm.

[0087] Loss function L total (θ, ψ) are:

[0088]

[0089] In the formula, A t R is the dominance function. t V(s) represents the current reward value, γ represents the given state value weight, and V(s) represents the current reward value. t V(s) is the current state value function. t+1 ) is the next state value function; Ltotal (θ, ψ) is the loss function, E[■] represents the expectation function, and min(■) represents the minimum value function. Let be the weight ratio for importance sampling, ∈ be the pruning factor, ∈ ∈ [0, 1], and c1 be the value factor, c1 ∈ [0, 1]. In this loss function, and A t Related to the parameter θ update of the Actor network, V(s) t ) and V(s t+1 This is related to the parameter ψ update of the Critic network.

[0090] Based on the above analysis, the intelligent software-defined wireless network routing method based on network situation awareness proposed in this invention, such as... Figure 5 As shown, the steps are as follows:

[0091] Step 1: Initialize the parameters θ of the Actor network and ψ of the Critic network, set the maximum number of iterations M, and clear the current experience buffer; set the initial node of the agent to the source node N. s .

[0092] Step 2: First, obtain the network topology information and the current and historical network state information of SDWN; then, convert the network topology information into a network adjacency matrix Ag, and simultaneously convert the current and historical network state information into a network state matrix. The network state matrix at the current and historical moments. The data is fed into the GCN-GRU prediction model to predict the network state matrix at the next time step. Then, the network adjacency matrix Ag and the network state matrix are... An environment for interaction with the intelligent agent is formed; where n is the set number of historical moments; network status information includes link remaining bandwidth, link delay, link error rate, link packet loss rate, and link distance.

[0093] Step 3: The agent interacts with the environment to obtain the current state s. t ; where the current state s t The network state matrix at the current moment and the routing matrix at the current time composition.

[0094] Step 4: The agent sets the current state s t Inputting into the current Actor network yields the current action 'a'. t And the policy probability distribution π(a) of the current action. t |s t ).

[0095] Step 5: The agent executes the current action a. t It interacts with the environment to obtain the current reward value R. t and the next state s t+1 The next state s t+1 The network state matrix at the next time step and the routing matrix at the next time step composition.

[0096] Figure 6 A schematic diagram illustrating action selection in deep reinforcement learning. The agent executes the current action a. t Subsequently, the following three situations may occur during the interaction with the environment:

[0097] ① When the agent executes the current action a t If the next node selected is the adjacent node of the current node, and the path traversed by the agent has no loops, the current reward value R is... t for:

[0098] R t =β1bw ij -β2 delay ij -β3 errors ij -β4 drop ij -β5 distance ij

[0099] ②When the agent executes the current action a t If the next node selected is an adjacent node of a safe node, and the path traversed by the agent contains a loop, the current reward value R is... t for:

[0100] R t =-ξ1R s

[0101] ③ When the agent executes the current action a t When the selected next node is not an adjacent node of the current node, the current reward value R is... t for:

[0102] R t =-ξ2R s

[0103] In the formula, bw ij β1 represents the remaining bandwidth of the link between node i and node j, and delay represents the remaining bandwidth weight. ij β2 represents the link delay between node i and node j, and β2 represents the link delay weight. Errors ij β3 represents the packet error rate of the link between node i and node j, and drop represents the packet error rate weight. ijβ4 represents the packet loss rate of the link between node i and node j, and distance represents the link packet loss rate weight. ij R represents the link distance between node i and node j, and β5 represents the link distance weight; s Given a standard reward, ξ1 is a given first discount factor, ξ2 is a given second discount factor, and ξ1, ξ2, R s ∈[0,1].

[0104] Step 6: The agent sets the current state s t Current action a t Current reward value R t The policy probability distribution π(a) of the current action t |s t ) and the next state s t+1 The formed sample trajectory (s) t a t R t ,π(a t |s t ), s t+1 Place it in the current experience cache.

[0105] Step 7: The agent samples the sample trajectory from the current experience buffer and selects the current state s from the sampled sample trajectory. t and the next state s t+1 The input is given to the current Critic network to obtain the current state value function V(s). t ) and the next state value function V(s) t+1 ).

[0106] Step 8: Construct the advantage function and update the parameters θ of the current Actor network by maximizing the advantage function; where the advantage function A t for:

[0107] A t =R t +γV(s t+1 )-V(s t )

[0108] In the formula, A t R is the dominance function. t V(s) represents the current reward value, γ represents the given state value weight, and V(s) represents the current reward value. t V(s) is the current state value function. t+1 ) is the value function for the next state.

[0109] Step 9: Clear the current experience cache and repeat steps 4-6.

[0110] Step 10: Construct a loss function and update the parameters θ of the current Actor network and the parameters ψ of the current Critic network by minimizing the loss function; where the loss function L total (θ, ψ) are:

[0111]

[0112] In the formula, L total (θ, ψ) is the loss function, E[■] represents the expectation function, min(■) represents the minimum value function, and A t Let V(s) be the dominance function. t V(s) is the current state value function. t+1 ) is the next state value function. The weight ratio for importance sampling. π new (a t |s t Let π be the updated policy probability distribution of the Actor network. old (a t |s t ) represents the policy probability distribution before the Actor network update, ∈ represents the given pruning factor, and c1 represents the given value factor.

[0113] Step 11: The agent first selects the policy probability distribution π(a) from the current experience buffer. t |s t The largest current action a t and execute the current action a. t Then determine whether the agent should perform the current action a. t Is the selected next node the destination node N? d If yes, it means the current iteration is complete, and proceed to step 12; otherwise, it means the current iteration is not complete, and the agent will execute the current action a. t The next selected node is used as the agent's current node. After clearing the current experience cache, proceed to step 3.

[0114] Step 12: Determine if the number of iterations has reached the set maximum number of iterations M: If so, construct the routing matrix for the current time step obtained in the last round. Output the optimal path; otherwise, proceed to step 2.

[0115] It should be noted that although the embodiments described above are illustrative, they are not intended to limit the invention. Therefore, the invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. A smart software-defined wireless network routing method based on network situational awareness, characterized by: The steps include the following: Step 1: Initialize the parameters of the Actor network Parameters of the Critic network Set the maximum number of iterations. Clear the current experience cache; set the agent's initial node as the source node. ; Step 2: First, obtain the network topology information and the current and historical network state information of SDWN; then, convert the network topology information into a network adjacency matrix. Simultaneously, the current and historical network state information is converted into a network state matrix. And the network state matrix at the current and historical moments. The data is fed into the GCN-GRU prediction model to predict the network state matrix at the next time step. Then the network adjacency matrix and network state matrix Forming an environment for interaction with intelligent agents; among which The number of historical moments set; Step 3: The agent interacts with the environment to obtain the current state. ; where the current state The network state matrix at the current moment and the routing matrix at the current time composition; Step 4: The agent displays the current state. Input into the current Actor network to obtain the current action. and the policy probability distribution of the current action. ; Step 5: The agent executes the current action. It interacts with the environment to obtain the current reward value. and the next state The next state The network state matrix at the next time step and the routing matrix at the next time step composition; Step 6: The agent sets the current state. Current action Current reward value The probability distribution of the current action's strategy and the next state The sample trajectory formed ( Place it in the current experience cache; Step 7: The agent samples the trajectory from the current experience buffer and selects the current state from the sampled trajectory. and the next state The input is fed into the current Critic network to obtain the current state value function. and the next state value function ; Step 8: Construct the advantage function and adjust the parameters of the current Actor network by maximizing the advantage function. Update; among which the dominant function for: Step 9: Clear the current experience cache and repeat steps 4-6; Step 10: Construct the loss function and minimize the loss function to adjust the parameters of the current Actor network. And the parameters of the current Critic network Update; where the loss function is... for: Step 11: The agent first selects a policy probability distribution from the current experience cache. Maximum current action and perform the current action. ; Then determine whether the agent should perform the current action. Is the selected next node the destination node? If yes, it means the current iteration is complete, and proceed to step 12; otherwise, it means the current iteration is not complete, and the agent will execute the current action. The selected next node is used as the agent's current node. After clearing the current experience cache, proceed to step 3. Step 12: Determine if the number of iterations has reached the set maximum number of iterations. If so, then the routing matrix constructed in the last round for the current time step will be used. Output the optimal path; otherwise, proceed to step 2. in, For the dominant function, This is the current reward value. For a given state value weight, The current state value function, The next state value function; For loss function, Represents the expectation function, Describes the minimum value function. , The weight ratio for importance sampling. Given a clipping factor, For a given value factor.

2. The intelligent software-defined wireless network routing method based on network situation awareness according to claim 1, characterized in that, In step 2, the network status information includes the remaining bandwidth of the link, the link delay, the link packet error rate, the link packet drop rate, and the link distance.

3. The intelligent software-defined wireless network routing method based on network situation awareness according to claim 1 or 2, characterized in that, In step 5, the agent executes the current action. Subsequently, the following three situations may occur during the interaction with the environment: When the agent performs the current action The current reward value is calculated when the next selected node is the adjacent node of the current node, and the path traversed by the agent does not contain loops. for: When the agent performs the current action The current reward value is calculated when the next selected node is an adjacent node of a safe node, and the path traversed by the agent contains a loop. for: When the agent performs the current action When the selected next node is not an adjacent node of the current node, the current reward value is... for: In the formula, Represents a node With nodes Remaining bandwidth of inter-link connections Indicates the weight of remaining bandwidth. Represents a node With nodes Inter-link latency, Indicates the link delay weight. Represents a node With nodes Inter-link packet error rate, Indicates the link error rate weight. Represents a node With nodes Inter-link packet drop rate Indicates the link packet drop rate weight. Represents a node With nodes Inter-link distance, Indicates the link distance weight; For the given standard reward, Given the first discount factor, Given a second discount factor, .

4. The intelligent software-defined wireless network routing method based on network situation awareness according to claim 1, characterized in that, In step 10, the weight ratio of importance sampling for: In the formula, This represents the updated policy probability distribution of the Actor network. This represents the policy probability distribution before the Actor network update.

Images