A topology routing joint optimization method based on deep reinforcement learning

Abstract

This invention discloses a joint optimization method for topology routing based on deep reinforcement learning, belonging to the field of UAV communication networks and routing. Specifically, UAVs are deployed within the test airspace to serve M mobile users. The test airspace is then gridded, and users publish task requests within a time step. UAVs, acting as nodes, are randomly deployed on the grid. For each UAV node, a Markov decision process is modeled and solved using a multi-agent reinforcement learning algorithm to obtain the movement decision for the next time step. This Markov decision process is repeated to obtain the final movement path for each UAV. For each task request, the corresponding UAV calculates the multi-index link cost function of its neighboring nodes and selects the neighboring node with the lowest cost value as the next hop for task transmission until the target node is reached. This invention improves the data transmission efficiency and network stability of ad hoc flying networks in complex dynamic environments.

Description

Technical Field

[0001] This invention belongs to the field of UAV communication networks and routing, and relates to a joint optimization method for topology routing based on deep reinforcement learning. Background Technology

[0002] With the rapid development of the low-altitude economy, Flying AdHoc Networks (FANETs), as a critical infrastructure for information transmission, need to achieve stable and efficient data transmission in a highly dynamic and volatile communication environment. However, due to the high-speed movement of nodes and frequent changes in network topology, traditional static or distance-based routing mechanisms are insufficient to meet the dual requirements of real-time performance and reliability for low-altitude networks. How to design topology and routing mechanisms with environmental awareness, autonomous coordination, and dynamic adaptability has become a core issue in current FANET research.

[0003] Existing routing technologies in ad hoc in-flight networks fall into two categories: topology-based routing mechanisms and location-based routing mechanisms.

[0004] Topology-based routing mechanisms typically select paths by maintaining the connectivity relationships between nodes in the network. This type of method is applicable to small-scale networks or networks with slow topology changes. However, in scenarios where drones fly at high speeds and network topology changes frequently, such routing mechanisms often require frequent updates to routing tables and link-state information, which can easily introduce significant control overhead. Furthermore, the routing convergence time is long, making it difficult to meet the real-time and reliability requirements of low-altitude flight networks.

[0005] Location-based routing mechanisms typically utilize node location information for greedy forwarding or area forwarding to reduce routing maintenance overhead and, to some extent, adapt to the dynamic characteristics of ad hoc networks. However, this type of method still has significant limitations in practical applications: for example, in cases of uneven node distribution or sparse network, routing holes can easily occur, causing data packets to be unable to continue forwarding to the target node, thereby reducing overall routing efficiency and transmission reliability.

[0006] With the development of intelligent algorithms, existing technologies attempt to introduce reinforcement learning into the routing design of ad hoc flight networks. By learning the mapping relationship between network states and routing behaviors, adaptive routing selection can be achieved in complex and dynamic environments. This type of method can improve the responsiveness of routing to topology changes to a certain extent, enabling UAV nodes to have a certain degree of autonomous decision-making ability.

[0007] However, most existing routing schemes based on reinforcement learning only regard the routing process as a passive adaptation to a given network topology, focusing mainly on selecting the "optimal path" under the current topology conditions, without fully considering improving the overall communication environment by actively adjusting node trajectories or network topology.

[0008] Furthermore, existing technologies generally treat UAV trajectory planning and data routing design as independent problems, lacking system modeling and joint optimization of the coupling relationship between the two. In highly dynamic flight ad hoc networks, UAV flight trajectories not only directly affect the network topology but also further impact link stability, routing availability, and network load distribution. Relying solely on passive routing adjustments while neglecting trajectory-level collaborative optimization often fails to improve the long-term stability and transmission efficiency of the network from a global perspective.

[0009] In summary, while existing flight ad hoc network routing technologies have improved communication performance in dynamic environments to some extent, they still suffer from problems such as high routing overhead, limited topology adaptability, and lack of routing and trajectory co-optimization, and cannot yet achieve efficient global control of highly dynamic flight ad hoc networks. Summary of the Invention

[0010] To address the aforementioned issues, this invention proposes a joint optimization method for topology routing based on deep reinforcement learning. This method comprehensively considers network topology evolution and routing decision-making processes to improve data transmission efficiency and network stability of ad hoc flying networks in complex dynamic environments.

[0011] The specific steps of the topology routing joint optimization method based on deep reinforcement learning are as follows:

[0012] Step 1: Within the airspace to be tested, A FANET consisting of drones is deployed in a mobile communication network to provide relay network services for M mobile users;

[0013] Step 2: Grid the airspace to be measured at fixed intervals, and then... Within the grid, mobile users each publish task requests, which are then assigned to their respective grids.

[0014] The distance for gridding the airspace to be measured is set manually according to the actual situation.

[0015] Step 3: Initially, each drone is treated as a node and randomly deployed on a grid.

[0016] The drone moves and covers according to the task priority. A single drone can only cover one grid and receive task requests within that grid; a single grid can only correspond to one drone.

[0017] Step 4: For each drone node, at the time step... Model Markov decision processes separately;

[0018] For drones Its state space Represented as: ;

[0019] in, Indicates drone The two-dimensional position of the covering grid. Indicates drone The proportion of remaining energy, Indicates the current drone Network connectivity contribution of neighboring drones Indicates drone The contribution of regional coverage, Indicates drone Average link stability with neighboring drones: ;in, For drones The number of neighbors. Indicates drone and neighbor's drone Link stability between;

[0020] Action space Represented as: ; respectively represent drones Discrete motion actions: up, down, left, right, and hover.

[0021] reward function Represented as: * ;

[0022] in, The weights are the corresponding weights for the four factors. This indicates the network connectivity between the drone and its neighboring drones; Indicates drone The energy consumption of each movement action in the motion space; Indicates drone Area coverage for grid tasks:

[0023]

[0024] Indicates time step Inside, grid The amount of data requested by the task within; Represents a grid Whether it is covered by drones. If it is covered, the value is 1; otherwise, the value is 0.

[0025] Step 5: Solve the Marr decision process using a multi-agent reinforcement learning algorithm to obtain the next step for each UAV. The movement decision is made by repeating the Markov decision process, continuously adjusting the position until the next coverage grid of the drone is obtained; thus obtaining the final movement path of each drone.

[0026] Step 6: For each task request, the corresponding UAV calculates the multi-index link cost function of each neighbor node, selects the neighbor node with the lowest cost value as the next hop for task transmission, and continues until the task request is sent to the target node.

[0027] For drones Its next-hop neighbor node drone Multi-metric link cost function for:

[0028]

[0029] These are the weight values, To prevent fixed values ​​with a divisor of 0, For relay drone nodes The length of the unprocessed task queue. For drones and drones The probability risk value of the link causing end-to-end delay:

[0030]

[0031] This indicates an indicator function that takes the value 1 when the function is true, and 0 otherwise. Indicates the first Next from drone Jump to the drone The time delay. This indicates the preset QoS threshold; For drones Jump to the drone The total number of times.

[0032] The advantages of this invention are:

[0033] 1) This invention performs joint modeling and collaborative optimization of UAV topology control and routing selection. Instead of passively adapting to a given topology, it actively adjusts the spatial distribution of UAVs and network connectivity based on task-driven principles, thereby extending the duration of stable topology and reducing rerouting and retransmission caused by frequent link breaks. This is beneficial for improving the continuity of data transmission and overall network stability in high-speed maneuvering and rapidly changing flight ad hoc networks.

[0034] 2) This invention proposes a multi-index link cost function and a latency risk assessment mechanism during the routing phase. It comprehensively considers factors such as link lifetime, transmission rate, node queue length, and the probability risk of end-to-end latency exceeding the threshold to select the next hop. This can improve the reliability of routing decisions and QoS guarantee capabilities under dynamic link conditions, reduce latency fluctuations and packet loss risks caused by congestion or unstable links, and avoid the additional control overhead caused by frequent maintenance of the global routing table. It is suitable for multi-hop transmission scenarios in highly dynamic flight ad hoc networks. Attached Figure Description

[0035] Figure 1 This is a flowchart of a topology routing joint optimization method based on deep reinforcement learning according to the present invention;

[0036] Figure 2 This is a diagram illustrating the application scenario of the drone as described in this invention.

[0037] Figure 3 This is a structural diagram of the QMIX algorithm model used in this invention;

[0038] Figure 4 This is a flowchart illustrating the overall strategy employed in this invention.

[0039] Figure 5 This is a comparison chart of the connectivity and coverage between the joint optimization algorithm of this invention and the benchmark algorithm;

[0040] Figure 6 This is a diagram showing the deployment effect of the joint optimization algorithm and the benchmark algorithm of this invention;

[0041] Figure 7 This is a performance comparison chart of the present invention with different numbers of drones;

[0042] Figure 8 This is a performance comparison chart of the present invention at different drone speeds. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0044] This invention proposes a joint optimization method for topology routing based on deep reinforcement learning, comprising the following two parts: UAV trajectory planning and route selection; such as... Figure 1 As shown, the specific steps are as follows:

[0045] Step 1: Within the airspace to be tested, A FANET consisting of multiple hop drones is deployed in a mobile communication network to provide relay network services for M randomly distributed mobile users on the ground.

[0046] like Figure 2 As shown.

[0047] Step 2: Grid the airspace to be measured at fixed intervals, and then... Within the grid, mobile users each publish task requests, which are then assigned to their respective grids.

[0048] The distance of the gridded airspace to be tested is set manually according to the actual situation; due to the changes in the location of ground mobile users and task requirements, the distribution of ground tasks is dynamically changing, so the task distribution is gridded.

[0049] Because the movement of ground users within the network is uncontrolled, and the communication requirements of ground missions change, it is necessary to adjust the UAV network location distribution and routing in real time. After acquiring information about ground communication needs, UAVs, in conjunction with the communication status with neighboring UAV nodes, select appropriate movement strategies to adjust their distribution, adapting it to changes in ground missions.

[0050] Step 3: Initially, each drone is treated as a node and randomly deployed on a grid.

[0051] The drone moves and covers according to the task priority. A single drone can only cover one grid and receive task requests within that grid; a single grid can only correspond to one drone.

[0052] Step 4: For each drone node, at the time step... Model Markov decision processes separately;

[0053] Drones need to move constantly to meet the service demands of ground users. Due to the dynamic changes in drone positions and limited communication range, topology interruptions between drones occur frequently, affecting routing performance. Therefore, maximizing the duration of a stable topology while meeting business requirements is key to solving the routing problem.

[0054] This invention also grids the distribution of ground tasks, models the decision-making process of UAV movement as a Markov decision process, and uses reinforcement learning methods to determine the UAV's movement direction in the next moment in a dynamic network scenario, so as to extend the topology duration and meet dynamic business needs.

[0055] Each UAV node can self-optimize and learn based on environmental feedback, solving the strategy evaluation problem through continuous trial and error, thereby achieving the goal of self-selection of movement direction. This invention abstracts the UAV motion and topology control process into a Markov decision process, including state space, action space, and reward function.

[0056] For drones ,state Representing the intelligent agent in time step The observed environmental conditions characterize the current network topology and resource distribution of the system, including all The location of each drone, link lifetime, network connectivity, coverage, and remaining energy of each drone. State space. Represented as:

[0057] ;

[0058] in, Indicates drone The two-dimensional position of the covering grid. Indicates drone The percentage of remaining energy; the remaining energy of the drone Represented as: ; This represents the energy consumption of each movement action performed by the drone within its action space. Normalization is performed, and the value is taken as follows: .

[0059] Indicates the current drone Network connectivity contribution of neighboring drones Indicates drone The contribution of regional coverage, Indicates drone Average link stability with neighboring drones: ;in, For drones The number of neighbors; Indicates drone and neighbor's drone The stability of the link between them.

[0060] Because the distance between the source drone and the target drone may exceed the drone's maximum communication range. Therefore, each data packet will be transmitted via multiple drones over multiple hops. Since congestion in the sky is negligible, the communication link between drones can be assumed to be a line-of-sight (LOS) link. The channel gain between drones follows a free-space path model as follows:

[0061]

[0062] in, Indicates drone Location;

[0063] This represents the channel gain at a reference distance of 1m. All drones have the same transmission power. drones For drones Received power Represented as:

[0064]

[0065] drones For drones The signal-to-noise ratio is:

[0066]

[0067] in, This indicates the carrier bandwidth of the drone; each drone has the same bandwidth. This represents the one-sided power spectral density of noise.

[0068] According to the Shannon formula, the transmission rate between drones... Represented as:

[0069]

[0070] Link stability between drones Represented as:

[0071]

[0072] in:

[0073]

[0074]

[0075]

[0076]

[0077] Indicates the communication radius of the node, ( )and( ), and , and Let represent the two-dimensional coordinates, speed, and angle between the direction of motion and the horizontal direction of the UAV and the UAV, respectively. ).

[0078] At each time step t, the discrete motion actions that the UAV can take include: hovering, moving in four directions (up, down, left, and right), and the motion space. Represented as: ;

[0079] reward function The algorithm, calculated using a weighted average of multiple factors, guides drone movement while optimizing topology stability and network performance. The first factor is link duration. Link duration is used to measure the topological stability of the drone relative to surrounding drones. The second factor is network connectivity. This represents the number of connections between the drone and its surrounding nodes. Higher connectivity means the drone has more potential next nodes to choose from, reducing the likelihood of it becoming isolated. The third factor is area coverage. This factor indicates the coverage of drones for ground missions. The larger the value, the greater the workload in that area, making it more attractive to drones. The fourth factor... This refers to the energy consumed by the drone when performing movement; hovering is generally considered to consume less energy than moving. Reward function. The following formula can be used to calculate it.

[0080] * ;

[0081] in, These are the weights corresponding to the four factors; the weights satisfy:

[0082]

[0083] The specific weight values ​​can be adjusted according to the application scenario.

[0084] This indicates the network connectivity between a UAV and its neighboring UAVs; it is used to measure whether UAVs have formed a connected network and to prevent UAVs from becoming isolated. ;in, For drones The number of neighbors, This represents the total number of drones.

[0085] Indicates drone The energy consumption of each movement action within the action space; the energy consumption of a drone mainly consists of two parts: flight energy consumption and transmission energy consumption. Transmission energy consumption is very small compared to flight energy consumption, therefore we do not consider transmission energy consumption. Drone Energy Consumption The main factors determining this are flight status, which are represented as follows:

[0086]

[0087] in, Indicates energy consumption during movement. This indicates the energy consumed during hovering. The symbol indicates flight mode.

[0088]

[0089] Indicates drone Area coverage for grid tasks:

[0090]

[0091] Indicates time step Inside, grid The size of the data requested by the task; assume that the arrival of task requests follows a Poisson distribution: ; Represents a grid Whether it is covered by drones. If it is covered, the value is 1; otherwise, the value is 0.

[0092] Step 5: Solve the Marr decision process using a multi-agent reinforcement learning algorithm to obtain the next step for each UAV. The movement decision is made by repeating the Markov decision process, continuously adjusting the position until the next coverage grid of the drone is obtained; thus obtaining the final movement path of each drone.

[0093] like Figure 3 As shown, QMIX is a multi-agent reinforcement learning algorithm that learns the joint action value function of all drones in a centralized hybrid network by training in a centralized manner and executing in a decentralized manner, thereby optimizing the global performance of the network.

[0094] The core of QMIX is the design of a global hybrid network that receives... With global state Non-negative weights are generated through a hypernetwork, and the Q-values ​​of single agents are weighted and combined to form the global value.

[0095]

[0096] in ensure It satisfies the decomposability constraint.

[0097] After completing the topology optimization based on the QMIX algorithm, this invention designs a link-state routing calculation mechanism based on multi-index weighting to further improve the stability and efficiency of data forwarding. This mechanism dynamically calculates the optimal next-hop route after each round of topology updates, based on factors such as the node's current location information, link lifetime, transmission rate, and relay queue length.

[0098] This article introduces risk values. Indicates link The probability risk value that may cause high-end end-to-end latency, for each link. Maintenance latest The actual one-hop delay of the next communication is calculated using a sliding window method to determine the frequency of high-latency events. The calculation formula is as follows:

[0099]

[0100] in, This indicates an indicator function that takes the value 1 when the function is true, and 0 otherwise. Indicates the first Next from drone Jump to the drone The time delay. This indicates the preset QoS threshold.

[0101] Step 6: For each task request, the corresponding UAV calculates the multi-index link cost function of each neighbor node, selects the neighbor node with the lowest cost value as the next hop for task transmission, and continues until the task request is sent to the target node.

[0102] For drones Its next-hop neighbor node drone Multi-metric link cost function for:

[0103]

[0104] These are the weight values, To prevent fixed values ​​with a divisor of 0, For relay drone nodes The length of the unprocessed task queue.

[0105] The overall strategy flowchart is as follows Figure 4 As shown, after the UAV initial deployment is complete, it acquires environmental status information through information exchange. The topology control module uses Markov decision-making to calculate the optimal action based on the environmental status. The UAV updates its own position based on the optimal action, and simultaneously receives task requests from the coverage area, executing routing algorithms based on the environmental status to obtain the next-hop node. The UAV transmits data packets and receives link latency feedback from the next hop, calculating and updating the risk value based on the latency feedback.

[0106] The performance of the proposed deep reinforcement learning-driven joint optimization strategy for topology routing was verified through simulation experiments. The number of UAV nodes, u, was set to 16, the node speed to [10, 25] m / s, the communication carrier bandwidth to 15 MHz, and the transmission power to 0.03 W. Other simulation parameters for the UAV are shown in Table 1. The performance and advantages of the proposed algorithm were evaluated by comparing it with the following routing strategies:

[0107] Ad-hoc On-Demand Distance Vector (AODV) routing protocol for wireless ad hoc networks: Under this strategy, drones broadcast nodes on demand and establish and maintain routing tables.

[0108] The greedy, stateless, perimeter-based routing algorithm uses geographic location information to select the next-hop route based on a greedy algorithm.

[0109] Table 1

[0110] Simulation scene range L*L 2000m*2000m Number of grids n*n 20*20 Number of drone nodes u 16 Drone node speed v 0-25m / s Communication radius R 400m Coverage radius r 100m Carrier bandwidth B 15MHz Transmission power P 0.03W Simulation scene range L*L 2000m*2000m Number of grids n*n 20*20 Number of drone nodes u 16 Drone node speed v 0-25m / s

[0111] The simulation results are as follows:

[0112] The effectiveness of the JTRO solution proposed in this invention was verified through comparative experiments. Figure 5 The comparison results show that, under the same simulation conditions, when using the JTRO of this invention, the network connectivity and task coverage metrics remain at a higher level overall. Compared to the random movement strategy, the connectivity improvement of the JTRO scheme of this invention is more stable and less volatile, while the coverage shows a more significant advantage in most periods. This JTRO invention, by introducing reinforcement learning-based topology control, makes the spatial distribution of UAVs more conducive to maintaining network connections and improving coverage of ground tasks, thereby providing a better network foundation for subsequent reliable routing and stable forwarding.

[0113] Figure 6 The presentation showcases the spatial deployment and networking patterns of UAVs under different strategies. From a distribution perspective, when employing the JTRO invention, UAV nodes exhibit a spatial trend of "clustering around mission hotspot areas," while the network backbone still maintains connectivity across all areas. The overall topology exhibits a "hotspot enhancement + global connectivity" pattern. The JTRO invention, while ensuring network connectivity, guides nodes to more effectively converge towards high-demand areas, achieving coordinated optimization of deployment and networking, thereby enhancing mission-oriented service capabilities and network stability.

[0114] Figure 7 The results demonstrate that the JTRO solution of this invention consistently maintains the highest throughput and lowest latency across different numbers of drones, and exhibits good scalability as the network scale increases. Simultaneously, the packet loss rate decreases significantly with increasing number of drones, and the link reliability is significantly better than GPSR and AODV.

[0115] Figure 8 The results demonstrate that the JTRO invention consistently maintains higher throughput and lower latency at different speeds, with particularly significant advantages in high-speed mobile environments. The packet loss rate generally increases with increasing speed, but the single increase is minimal, exhibiting the strongest dynamic topology adaptability.

[0116] The above examples are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art should understand that any modifications, variations, or equivalent substitutions to the invention without departing from its spirit and scope should be included in the claims of the invention to make the objectives, technical solutions, and advantages of the invention clearer. The embodiments of the invention will be further described in detail below with reference to the accompanying drawings.

Claims

1. A joint optimization method for topology routing based on deep reinforcement learning, characterized in that, The specific steps are as follows: Step 1: Within the airspace to be tested, A FANET consisting of drones is deployed in a mobile communication network to provide relay network services for M mobile users; Step 2: Grid the airspace to be measured at fixed intervals, and then... Within the grid, mobile users each publish task requests, which are then assigned to their respective grids. Step 3: Initially, each drone is treated as a node and randomly deployed on a grid. Step 4: For each drone node, at the time step... Model Markov decision processes separately; For drones Its state space Represented as: ; in, Indicates drone The two-dimensional position of the covering grid. Indicates drone The proportion of remaining energy, Indicates the current drone Network connectivity contribution of neighboring drones Indicates drone The contribution of regional coverage, Indicates drone Average link stability with neighboring drones: ;in, For drones The number of neighbors; Indicates drone and neighbor's drone Link stability between; Action space Represented as: ; respectively represent drones Discrete motion actions: up, down, left, right, and hover; reward function Represented as: * ; in, The weights are the corresponding weights for the four factors. This indicates the network connectivity between the drone and its neighboring drones; Indicates drone The energy consumption of each movement action in the motion space; Indicates drone Area coverage for grid tasks: Indicates time step Inside, grid The amount of data requested by the task within; Represents a grid Whether it is covered by drones. If it is covered, the value is 1; otherwise, the value is 0. Step 5: Solve the Marr decision process using a multi-agent reinforcement learning algorithm to obtain the next step for each UAV. The movement decision is made by repeating the Markov decision process, continuously adjusting the position until the next coverage grid of the UAV is obtained; thus, the final movement path of each UAV is obtained. Step 6: For each task request, the corresponding UAV calculates the multi-index link cost function of each neighbor node, selects the neighbor node with the lowest cost value as the next hop for task transmission, and continues until the task request is sent to the target node. For drones Its next-hop neighbor node drone Multi-metric link cost function for: These are the weight values, To prevent fixed values ​​with a divisor of 0, For relay drone nodes The length of the unprocessed task queue. For drones and drones The probability risk value of the link causing end-to-end delay: This indicates an indicator function that takes the value 1 when the internal function is true, and 0 otherwise. Indicates the first Next from drone Jump to the drone The time delay; This indicates the preset QoS threshold; For drones Jump to the drone The total number of times.

2. The method as described in claim 1, characterized in that, In step two, the distance of the gridded airspace to be measured is set manually according to the actual situation.

3. The method as described in claim 1, characterized in that, In step three, the drone moves and covers according to the task priority. A single drone can only cover one grid and receive task requests within that grid. Each grid can only correspond to one drone.

Images