Mesh routing method based on graph neural network and reinforcement learning

Abstract

The invention discloses a Mesh routing selection method based on a graph neural network and reinforcement learning. The Mesh routing selection method comprises the following steps: step S1, constructing a Mesh network graph structure model; s2, extracting an embedded representation of each router node by using a graph neural network; s3, designing a routing decision mechanism based on Q-learning, and generating a Q value in combination with the current router node state and neighbor information; s4, introducing a Masking mechanism to process the dynamic action space, and shielding illegal next-hop action; and step S5, defining a multi-target reward function which is used for guiding a Q-learning training process. The invention provides a Mesh routing selection method based on a graph neural network and reinforcement learning, and aims to solve the problems of poor adaptive capacity to a dynamic network environment, low routing efficiency and lack of intelligent decision support in the prior art and improve the data transmission performance of a Mesh network.

Description

TECHNICAL FIELD

[0001] The application relates to a Mesh routing method based on a graph neural network and reinforcement learning and belongs to the technical field of wireless communication and artificial intelligence. BACKGROUND

[0002] At present, with the development of the Internet of Things, edge computing and wireless ad hoc network technology, the Mesh network is widely used in smart cities, industrial automation and military communication due to its high flexibility, strong fault tolerance and good scalability. However, due to the large number of nodes, complex connection relationship and frequent dynamic changes in the Mesh network, the traditional static routing algorithm (such as AODV, DSR, etc.) is difficult to adapt to network congestion, unstable link and other problems, resulting in a decline in the overall network performance.

[0003] In recent years, deep reinforcement learning (DRL) has shown good application prospects in path planning, resource scheduling and other aspects. Among them, Q-learning, as a classic model-free reinforcement learning method, can learn the optimal strategy through the trial-and-error mechanism, but it has limitations in dealing with high-dimensional state space and dynamic action space. At the same time, the graph neural network (GNN) has the ability to naturally process graph structure data and can effectively extract network topology features and perform embedding representation.

[0004] Therefore, how to combine GNN and Q-learning for dynamic routing selection in the Mesh network, especially in the face of key problems such as dynamic change of next hop selection, load balancing, low delay transmission, etc., is an urgent problem to be solved. SUMMARY

[0005] The technical problem to be solved by the application is to overcome the shortcomings of the prior art and provide a Mesh routing method based on a graph neural network and reinforcement learning, aiming to solve the problems of poor adaptability to dynamic network environment, low routing efficiency and lack of intelligent decision support in the prior art, and improve the data transmission performance of the Mesh network.

[0006] To solve the above technical problems, the technical scheme of the application is as follows:

[0007] A Mesh routing method based on a graph neural network and reinforcement learning, comprising the following steps:

[0008] Step S1, constructing a Mesh network graph structure model;

[0009] Step S2, using a graph neural network to extract the embedding representation of each router node;

[0010] Step S3, a routing decision mechanism based on Q-learning is designed to generate Q values in combination with the current router node state and neighbor information;

[0011] Step S4, a Masking mechanism is introduced to process the dynamic action space to shield illegal next-hop actions;

[0012] Step S5, a multi-objective reward function is defined to guide the Q-learning training process.

[0013] Further, in the step S1, the Mesh network graph structure model is constructed, specifically including the following steps:

[0014] The Mesh network graph structure model is established, and the network topology of the Mesh network graph structure model is represented as a graph G, G=(V, E), wherein V represents a set of router nodes, and E represents wireless connection edges between the router nodes;

[0015] The feature vector of each router node includes the current queue length, channel quality, remaining power, and neighbor number;

[0016] The feature vector of each wireless connection edge includes the bandwidth, delay, and packet loss rate.

[0017] Further, in the step S2, a graph neural network is used to extract the embedding representation of each router node, specifically including the following steps:

[0018] Step S21, a graph convolutional network or a graph attention network is constructed to perform multi-layer message propagation and aggregation on the Mesh network graph structure;

[0019] Step S22, a low-dimensional embedding vector hi∈Rd of each router node is output, and the low-dimensional embedding vector is taken as the state representation hi of the router node;

[0020] Step S23, the state representation fuses local topological structure and global network state information.

[0021] Further, in the step S3, a routing decision mechanism based on Q-learning is designed to generate Q values in combination with the current router node state and neighbor information, specifically including the following steps:

[0022] Step S31, for the current router node i, a Q value function Q(hi, a) is constructed according to the state representation hi of the router node i and the neighbor list N(i), wherein a∈N(i) represents the selectable next-hop action;

[0023] Step S32, a neural network is used to approximate the Q function, taking the state representation hi as the input and the Q values of each legal next-hop action as the output;

[0024] Step S33, in the training process of the graph neural network, a-greedy strategy is adopted to balance exploration and utilization, and the Q value is updated by using the Bellman equation.

[0025] Further, in the step S4, a masking mechanism is introduced to process the dynamic action space, and illegal next-hop actions are shielded. Specifically, the following steps are included:

[0026] Step S41, for an illegal next-hop action at a certain time, the Q value corresponding to the illegal next-hop action is shielded;

[0027] Step S42, the action space is dynamically adjusted according to the network topology.

[0028] Further, the objectives in the multi-objective reward function include packet transmission delay delay, link available bandwidth bandwidth, node queue length queue, routing path hop count hops and node energy consumption energy.

[0029] Further, the expression of the multi-objective reward function is as follows:

[0030]

[0031] where w1, w2, w3, w4 and w5 are weight systems, delay t is the packet transmission delay at time t, bandwidth t is the link available bandwidth at time t, queue t is the node queue length at time t, hops t is the routing path hop count at time t, and energy t is the node energy consumption at time t.

[0032] By adopting the above technical solutions, the present application has the following beneficial effects:

[0033] 1. The present application extracts graph structure features through a graph neural network, so that the routing selection has the ability to perceive changes in network topology and realizes intelligent perception of network state.

[0034] 2. The present application combines Q-learning and the masking mechanism, can cope with the challenge of dynamic changes in the number of next-hop, and realizes dynamic adaptation of routing strategy.

[0035] 3. The present application designs a multi-dimensional reward function, taking into account multiple network performance indicators such as delay, bandwidth and energy consumption, and has the ability of multi-objective optimization.

[0036] 4. The model of the present application can maintain stable performance under different topological structures and traffic patterns, and has strong generalization ability.

[0037] 5. The present invention is compatible with the existing Mesh protocol stack and is easy to deploy in actual devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of the mesh routing method based on graph neural network and reinforcement learning of the present invention;

[0039] Figure 2 This is a flow chart of step S2 of the present invention;

[0040] Figure 3 This is a flow chart of step S3 of the present invention;

[0041] Figure 4 This is a flow chart of step S4 of the present invention. DETAILED DESCRIPTION

[0042] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0043] like Figure 1 As shown, this embodiment provides a mesh routing method based on graph neural network and reinforcement learning, which includes the following steps:

[0044] Step S1: Construct a Mesh network graph structure model. Specifically, the following steps are included:

[0045] A Mesh network graph structure model is established, and the network topology of the Mesh network graph structure model is represented as a graph G, G=(V, E), where V represents a set of router nodes and E represents a wireless connection edge between each router node.

[0046] The feature vector of each router node includes but is not limited to parameter information such as current queue length, channel quality, remaining power, and number of neighbors.

[0047] The feature vector of each wireless connection edge includes but is not limited to link state information such as bandwidth, delay, and packet loss rate.

[0048] Step S2: Use a graph neural network to extract the embedded representation of each router node. The graph neural network in this embodiment is a graph convolutional network (GCN) or a graph attention network (GAT). By extracting graph structural features through the graph neural network, the routing selection has the ability to perceive changes in network topology. Figure 2 As shown, the specific steps include:

[0049] Step S21: Build a graph convolutional network (GCN) or a graph attention network (GAT) to perform multi-layer message propagation and aggregation on the Mesh network graph structure.

[0050] Step S22, output the low-dimensional embedding vector hi∈R d , as the state representation of the router node hi.

[0051] Step S23, the state representation fuses local topology and global network state information.

[0052] Step S3, design a routing decision mechanism based on Q-learning, generate Q value combining current router node state and neighbor information. The Q-learning of the embodiment uses neural network to approximate Q function, and combines -greedy policy for action selection. As shown in Figure 3 , it specifically includes the following steps:

[0053] Step S31, for the current router node i, construct the Q value function Q(h i , a) according to the state representation h i of the router node i and the neighbor node list N(i).

[0054] Step S32, use neural network to approximate Q function, take the state representation h i as input, and take the Q value of each legal next hop action as output.

[0055] Step S33, use ∈-greedy policy to balance exploration and utilization in the training process of graph neural network (GNN), and update Q value using Bellman equation.

[0056] Step S4, introduce Masking mechanism to process dynamic action space, and shield illegal next hop actions. As shown in Figure 4 , it specifically includes the following steps:

[0057] Step S41, for an illegal next hop action at a certain time (such as a disconnected neighbor), shield the Q value corresponding to the illegal next hop action (set to negative infinity), to ensure that the Q value corresponding to the illegal next hop action will not be selected.

[0058] Step S42, the action space of the next hop is adjusted dynamically with the network topology, thereby improving the robustness of the algorithm.

[0059] Step S5, define a multi-objective reward function to guide the Q-learning training process. The objectives in the multi-objective reward function include but are not limited to network performance indicators such as data packet transmission delay delay, link available bandwidth bandwidth, node queue length queue, routing path hop count hops, and node energy consumption energy.

[0060] The expression of the multi-objective reward function is as follows:

[0061]

[0062] Wherein w1, w2, w3, w4 and w5 are weight systems, and the values can be adjusted according to application scenarios. t is the data packet transmission delay at time t, bandwidth t is the link available bandwidth at time t, queue t is the node queue length at time t, hops t is the routing path hop count at time t, energy t is the node energy consumption at time t.

[0063] The embodiment realizes online learning and offline deployment through the above method steps, and completes dynamic routing selection.

[0064] The above-described specific embodiments further specifically describe the technical problems solved by the present application, technical solutions and beneficial effects. It should be understood that the above-described specific embodiments are merely specific embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A mesh routing method based on graph neural network and reinforcement learning, characterized in that: It includes the following steps: Step S1, constructing a Mesh network graph structure model; Step S2: Use graph neural network to extract the embedding representation of each router node; Step S3: Design a routing decision mechanism based on Q-learning, and generate a Q value by combining the current router node status and neighbor information; Step S4: Introduce the Masking mechanism to process the dynamic action space and mask illegal next-hop actions; Step S5: Define a multi-objective reward function to guide the Q-learning training process.

2. The mesh routing method based on graph neural network and reinforcement learning according to claim 1, characterized in that: In step S1, a Mesh network graph structure model is constructed, which specifically includes the following steps: Establish a Mesh network graph structure model, and represent the network topology of the Mesh network graph structure model as a graph G, G = (V, E), where V represents the set of router nodes and E represents the wireless connection edges between each router node; The feature vector of each router node includes the current queue length, channel quality, remaining power, and number of neighbors; The feature vector of each wireless connection edge includes bandwidth, delay and packet loss rate.

3. The mesh routing method based on graph neural network and reinforcement learning according to claim 1, characterized in that: In step S2, a graph neural network is used to extract an embedded representation of each router node, specifically including the following steps: Step S21: Build a graph convolutional network or a graph attention network to perform multi-layer message propagation and aggregation on the Mesh network graph structure; Step S22: Output the low-dimensional embedding vector hi∈Rd of each router node, and use the low-dimensional embedding vector as the state representation hi of the router node; Step S23: The state representation integrates the local topology structure and the global network state information.

4. The mesh routing method based on graph neural network and reinforcement learning according to claim 1, characterized in that: In step S3, a routing decision mechanism based on Q-learning is designed to generate a Q value by combining the current router node status and neighbor information. Specifically, the following steps are included: Step S31: For the current router node i, construct a Q-value function Q(hi,a) based on the state representation hi of the router node i and the neighbor list N(i), where a∈N(i) represents an optional next-hop action. Step S32: Use a neural network to approximate the Q function, take the state representation hi as input, and take the Q value of each legal next-hop action as output; Step S33: During the training of the graph neural network, a -greedy strategy is adopted to balance exploration and utilization, and the Bellman equation is used to update the Q value.

5. The mesh routing method based on graph neural network and reinforcement learning according to claim 1, characterized in that: In step S4, a masking mechanism is introduced to process the dynamic action space and mask illegal next-hop actions, which specifically includes the following steps: Step S41: For an illegal next-hop action at a certain moment, mask the Q value corresponding to the illegal next-hop action; Step S42: The action space is dynamically adjusted according to the network topology.

6. The mesh routing method based on graph neural network and reinforcement learning according to claim 1, characterized in that: The objectives in the multi-objective reward function include packet transmission delay, link available bandwidth, node queue length, routing path hops and node energy consumption.

7. The mesh routing method based on graph neural network and reinforcement learning according to claim 6, characterized in that: The expression of the multi-objective reward function is as follows: Among them, w1, w2, w3, w4 and w5 are weight systems, delay t is the data packet transmission delay at time t, bandwidth t is the link available bandwidth at time t, queue t is the node queue length at time t, hops t is the number of routing path hops at time t, energy t is the node energy consumption at time t.

Images