Unmanned aerial vehicle network routing and power joint optimization method based on graph reinforcement learning
By combining graph reinforcement learning with graph attention networks and reinforcement learning, we have achieved joint optimization of routing and power in UAV networks under dynamic environments. This solves the problems of routing instability and low transmission efficiency in UAV networks under high mobility and topology changes, and is suitable for applications such as disaster response and precision agriculture.
Patent Information
- Application Number
- CN202511882780.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-02-10
AI Technical Summary
Unmanned aerial vehicle (UAV) networks face challenges such as unstable routing and low transmission efficiency in dynamic environments. Traditional routing algorithms struggle to adapt to high mobility and topology changes, and existing methods fail to effectively combine routing and power control, resulting in suboptimal network performance.
A graph reinforcement learning approach is adopted, which captures dynamic topology features through a graph attention network (GAT) and combines it with reinforcement learning (RL) to achieve joint decision-making on routing and power. The graph reinforcement learning decision model is trained in a simulation environment and deployed on real UAV nodes to perform joint optimization of routing and power.
It enables dynamic adaptation of drone networks in highly mobile environments, reducing energy consumption and improving communication quality, and is suitable for applications such as disaster response and precision agriculture.
Smart Images
Figure CN121510191A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) network technology, specifically to a method for joint optimization of routing and power in UAV networks based on graph reinforcement learning. This method is applicable to efficient communication resource management in dynamic UAV networks, such as disaster response, precision agriculture, and emergency communication scenarios. Background Technology
[0002] In recent years, unmanned aerial vehicle (UAV) networks have been increasingly widely used in disaster response, intelligent transportation, and reconnaissance, creating an urgent need for robust and efficient communication protocols. UAV networks inherently possess characteristics such as high node mobility, intermittent link interruptions, and dynamic topology changes. Traditional static routing algorithms (such as RIP and OSPF) struggle to adapt to such dynamic environments, generally exhibiting slow convergence speeds, high overhead, and delayed responses to topology changes. For example, the convergence speed of the OSPF protocol is limited by the protocol timer; shortening the detection interval easily leads to false alarms, while extending the interval exacerbates convergence delays. In fault scenarios, the asynchronous detection mechanism may also trigger continuous route calculations, further degrading performance.
[0003] To address these issues, researchers have attempted to introduce machine learning methods, with Graph Reinforcement Learning (GRL) becoming a research hotspot due to its ability to capture network topology relationships and dynamic features. While existing schemes based on Graph Neural Networks (GNNs) and Deep Reinforcement Learning (DRLs) have shown potential in route optimization, they often treat routing and power control as independent problems, failing to fully utilize their synergistic relationship. The choice of routing path directly affects the power required by the link, and power allocation determines link quality and stability; optimizing them separately can easily lead to suboptimal overall network performance. Furthermore, in time-varying scenarios of UAV networks, the random relocation of node positions makes it difficult for traditional GNNs to efficiently capture the spatiotemporal correlations during movement, thus limiting the algorithm's generalization ability.
[0004] Therefore, in order to address the problems of unstable routing and low transmission efficiency in the highly dynamic topology of UAV networks, there is an urgent need for an intelligent decision-making method that can jointly optimize routing and power control and has strong generalization capabilities, so as to adapt to rapidly changing link conditions, balance transmission quality and energy consumption, and meet the real-time and reliability requirements in practical applications. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for joint optimization of routing and power in UAV networks based on graph reinforcement learning. This invention captures dynamic topology features using a graph attention network (GAT) and combines it with reinforcement learning (RL) to achieve joint decision-making on routing and power, thus solving the problems of routing instability and inefficient transmission caused by the high mobility of UAV networks.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] A graph reinforcement learning-based method for joint optimization of routing and power in UAV networks includes the following steps:
[0008] S1. A virtual UAV network is constructed through simulation. The raw information of each UAV node in the UAV network is collected with a fixed discrete time step k, including UAV node location, neighbor information, channel gain, and interference power.
[0009] S2, for each UAV node, at each time step k, based on the collected raw information, calculate its own cumulative power consumption, neighbor validity identifier, relative distance with each neighbor, relative distance with the destination node, channel gain, and aggregated interference of all neighbors, to form the state information of the current time step. ;
[0010] S3. The state s of the drone node at time step k, the action a it performs, the reward r from the environment feedback, and the state s' at the next time step k+1 are combined into a quadruple (s, a, r, s'), which serves as a training sample. Multiple training samples are collected to form a training sample set. The reward r is the minimum link capacity among all links on the multi-hop routing path. Action 'a' is the next-hop neighbor node and its transmit power.
[0011] S4. Construct a graph reinforcement learning decision model. The input of the model is a quadruple of the drone node, and the output is the next-hop neighbor node of the drone node and the transmission power.
[0012] S5, Use the training sample set to train the graph reinforcement learning decision model;
[0013] S6. The trained graph reinforcement learning decision model is deployed on real drone nodes. The real drone nodes collect their own raw information, including location, neighbor information, channel gain, and interference power. The raw information is processed into state information, and then the state information is input into the graph reinforcement learning decision model to obtain the next-hop neighbor node and the transmission power.
[0014] Furthermore, in step 1, the specific method for constructing a virtual drone network through simulation is as follows:
[0015] S11, set up U drone nodes, all randomly distributed within a D×D square area, and each equipped with a single omnidirectional antenna; set the communication transmission distance of the drone nodes to R; set the maximum transmit power of each drone node to be [value missing]. ;
[0016] S12, Set the position of the drone node to change with time snapshot k, specifically by generating a displacement vector based on the movement rules of random waypoints or Gaussian Markov movement models. , the displacement vector Overlaying the current position gives the position of the next time snapshot;
[0017] S13, the UAV network is abstracted as a dynamic directed graph, where vertices correspond to UAV nodes. A directed edge representing a feasible communication link is established only when the distance between two UAV nodes does not exceed the transmission distance R. The data flow is forwarded from the randomly selected source node to the destination node and transmitted in the UAV network through multi-hop routing.
[0018] S14 uses the ITU-R P.1411 standard to model the LosS line-of-sight channel gain, where the channel gain... It's about distance. The piecewise function is expressed as follows:
[0019]
[0020] in, For basic transmission loss, Distance from breakpoint Antenna gain;
[0021] S15. During the simulation, for each data transmission, the signal-to-noise ratio received by the receiving node is calculated. The noise power in the signal-to-noise ratio is the sum of the aggregated interference power generated by all other nodes transmitting data and the system background noise power.
[0022] S16, using Shannon's formula to calculate link capacity. Determine the link capacity, where B is the bandwidth. This refers to the signal-to-noise ratio.
[0023] Furthermore, the graph reinforcement learning decision model includes a graph attention network, a gated recurrent unit, and an action branch Q-network; wherein:
[0024] Graph Attention Networks employ a multi-head attention mechanism to handle dynamically changing heterogeneous graph inputs, generating node embedding vectors through a "message passing-aggregation-update" processing method.
[0025] The gated recurrent unit receives the node embedding vectors and historical states output by the graph attention network, and outputs the hidden state representation to capture temporal dependencies;
[0026] The action branch Q-network employs a shared encoder and branch evaluation structure, and sets up two online Q-networks with identical structures but independent parameters. and The action branch Q-network works as follows:
[0027] S421, Hidden state based on the output of the gated loop unit Generate a [number] for each candidate neighbor j. dimensional score vector ;
[0028] S422, compare the maximum value among all candidate neighbor score vectors, and select the neighbor corresponding to the maximum value as the next hop node;
[0029] S423, in the score vector corresponding to the selected next-hop node, select the dimension k with the largest value. The discrete power level corresponding to this dimension is the optimal transmission power level.
[0030] Furthermore, the specific method of step S5 is as follows:
[0031] S51 sets the maximum number of iterations, batch size, discount factor, soft update rate, learning rate, and initial exploration rate. , where 0 < <1;
[0032] S52, the graph reinforcement learning decision model selects a joint action at each time step using an ϵ-greedy policy. Specifically, it performs random exploration with probability ϵ and selects the joint action with the highest current Q-value with probability 1−ϵ. By routing action and power action Together they constitute the following; when the number of samples in the buffer reaches the batch sampling size, gradient updates are performed, specifically including:
[0033] S521 employs a Double Q-learning mechanism to calculate the temporal difference target value, uses an online Q-network to select the optimal action for the next state, and uses a target Q-network to evaluate the value of the optimal action, thereby calculating the learning target; wherein, the parameters of the target Q-network are periodically copied from the online Q-network;
[0034] S522 uses the mean squared error to calculate the loss function, minimizes the temporal difference error through gradient descent, and updates the parameters of the online Q-network. ;
[0035] S523, for target network parameters Perform a soft update every N steps;
[0036] S53: When the number of training iterations reaches the maximum number of iterations, training is terminated.
[0037] The beneficial effects of this invention are as follows:
[0038] 1. This invention achieves joint optimization of routing and power control through graph reinforcement learning technology, enabling dynamic adaptation to the high mobility environment of UAV networks. Furthermore, it employs a weighted index of link quality and energy consumption as the optimization objective, effectively reducing energy consumption while ensuring communication quality.
[0039] 2. This invention innovatively uses Graph Attention Network (GAT) to process dynamic heterogeneous graph input, and combines it with Gated Recurrent Unit (GRU) to capture temporal dependencies. It generates node embedding vectors through a "message passing-aggregation-update" mechanism, accurately capturing network topology features and effectively solving the limitations of traditional routing algorithms in UAV networks.
[0040] 3. This invention combines pre-training in a simulated environment with actual deployment to pre-train graph reinforcement learning decision models in a virtual drone network, significantly reducing the trial-and-error costs of actual deployment. It is applicable to various application scenarios such as disaster response and precision agriculture. Attached Figure Description
[0041] Figure 1 This is a flowchart of the UAV network routing and power joint optimization method based on graph reinforcement learning in an embodiment of the present invention;
[0042] Figure 2 This is a schematic diagram illustrating the application scenario of the UAV network routing and power joint optimization method based on graph reinforcement learning in this embodiment of the invention;
[0043] Figure 3 This is a schematic diagram illustrating the principle of the UAV network routing and power joint optimization method based on graph reinforcement learning in this embodiment of the invention.
[0044] Figure 4 This is a structural block diagram of the UAV network routing and power joint optimization system based on graph reinforcement learning in an embodiment of the present invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0046] A method for joint optimization of routing and power in UAV networks based on graph reinforcement learning, referring to... Figure 1 This includes the following steps:
[0047] S1. Construct a high-fidelity virtual drone network in a simulation environment to simulate the dynamic behavior of a real network, and collect the original network information at fixed time steps to provide a data foundation for subsequent model training.
[0048] S2. After acquiring the raw information, each UAV node needs to process it into a state representation required for reinforcement learning decision-making. This step calculates the decision state for each candidate forwarding node (i.e., the node where the data packet is currently located) in the simulation environment;
[0049] S3. Drive the agent to interact with the environment in the simulation environment, and store the interaction process in the form of (s,a,r,s') to build the experience playback buffer required for model training;
[0050] S4. Construct a core neural network model for learning from state information and making joint decisions, the structure of which is shown in the attached figure. Figure 3 As shown;
[0051] S5. Using the data in the experience replay buffer generated in step S3, train the graph reinforcement learning decision model constructed in step S4 so that it learns the optimal joint decision strategy.
[0052] S6. Model Deployment and Practical Application: After the trained and stable graph reinforcement learning decision model is lightweighted, it is deployed to real drone nodes to achieve online, distributed joint optimization of routing and power.
[0053] The specific scenario in this embodiment is as follows: Figure 2 As shown, step S1 specifically involves:
[0054] S11. Define U UAV mobile nodes randomly distributed within a D×D area, forming a set U={1,2,...,U}. Each node is equipped with a single omnidirectional antenna, operates within a transmission range R, and its transmit power P must satisfy 0 ≤ P ≤ The simulation is carried out using a discrete-time model, with a fixed time step k set as the basic unit for network state updates.
[0055] To simulate the high maneuverability of the drone, a movement model is configured for each node, such as a Gaussian Markov movement model or a random waypoint model. The node position is denoted as... Generate random displacement vectors based on the movement model. The location update rule is + Simulates the dynamic changes in the topology of unmanned aerial vehicles (UAVs);
[0056] At each discrete time step k, the simulation system collects the following raw information for each UAV node in the network:
[0057] (1) Node location: node coordinates .
[0058] (2) Neighbor information: Obtain a list of all neighboring nodes of node i within the current communication radius R through periodic Hello beacon exchanges or distance calculations. .
[0059] (3) Channel gain: For each neighbor j ∈ The channel gain is calculated based on the preset channel model (ITU-R P.1411 LoS model). The gain is a function of distance and can be expressed as:
[0060] ,
[0061] in, It is the distance between nodes i and j.
[0062] (4) Interference power: For node i, measure or calculate the total interference power currently received by its receiver. (k). This includes aggregated interference power from other nodes communicating within its communication range. Typically, , where T(k) is the set of nodes that are transmitting within time step k.
[0063] Step S2 is as follows:
[0064] At time step k, for the current node i that needs to make a forwarding decision, its state is... It is a feature vector containing the following calculated features:
[0065] (1) Cumulative power consumption: The total transmit power consumed by all hops used along the path from the source node to the current node i.
[0066] (2) Neighbor Validity Identifier Vector: Used to determine whether each of the current drone node's neighboring nodes is a valid candidate for data forwarding. This vector is a binary number with a length equal to the total number of neighbors of node i. For the neighbor list... For each neighbor j in the algorithm, determine whether it is a valid candidate forwarding node. The specific criteria for determination include: j is not the previous hop node (to prevent loops), j is closer to the destination node than the current node i (greedy approach towards the destination), and the estimated signal-to-noise ratio is higher than a certain threshold. If the conditions are met, it is marked as 1; otherwise, it is marked as 0.
[0067] (3) Relative distance vectors with each neighbor: Record the Euclidean distance from the current node i to each neighbor j within its one-hop range. .
[0068] (4) Relative distance to the destination node: Record the Euclidean distance from the current node i to the final destination node.
[0069] (5) Channel gain vector: Records the channel gain from the current node i to each neighbor j within one hop range. .
[0070] (5) Aggregated interference vector: Records the aggregated interference power received by each neighbor j of the current node i. The estimated value.
[0071] Step S3 is as follows:
[0072] S31. At time step k, the agent (current node i) observes the current state s(k). According to its policy, which is random exploration in the early stage of training and model-based decision-making in the later stage, the agent chooses a joint action a(k) = .in, It selects a node from the set of effective neighbors as the next hop. It is a transmit power level selected from a preset set of discrete power levels.
[0073] S32. After executing action a(k), the data packet is sent from node i to the selected next-hop node j. The simulation environment calculates the reward r(k) based on the result of this transmission. In this embodiment, the reward is defined as the minimum link capacity among all links on the multi-hop routing path. Specifically, after the complete path from the source node to the destination node is established, the instantaneous capacity of each link on the path is calculated. =B ,in This is the signal-to-noise ratio of the link. (Take all) The minimum value in the result is used as the reward for this end-to-end transmission, directly optimizing the bottleneck of network end-to-end throughput.
[0074] S33. After the action is executed, the data packet arrives at the new node (next-hop node), or the transmission fails. If it successfully reaches the next hop, the new state s′ is the state calculated by the next-hop node at time step k+1. If it fails (link interruption, reaching the maximum hop count), it will transition to the terminated state.
[0075] S34. Combine a complete interactive tuple<s(k),a(k),r(k),s’> This serves as a training sample. During the simulation, a large number of such interactions are run to collect enough training samples to form the experience replay buffer D.
[0076] In step S4, the structure for constructing the core neural network model used to learn from state information and make joint decisions is as follows:
[0077] The GAT encoder receives a constructed local topology graph and uses a multi-head attention mechanism to calculate the mutual importance between nodes. Through a "message passing-aggregation-update" mechanism, GAT can generate an embedding vector for each node (especially the current agent node). This vector integrates the node's own features, neighbor features, and the connections between them, effectively capturing the dynamically changing network topology.
[0078] The Gated Recurrent Unit (GRU), acting as a temporal memory module, receives the current node embedding vector output by the GAT and combines it with its own stored hidden state from the previous time step to output a new hidden state. This enables the model to remember the contextual information of historical decisions, understand the sequential dependencies of multi-hop routes, and thus make forward-looking decisions, avoiding short-sighted behavior.
[0079] The action branch Q-network, acting as a joint decision-making module, receives the hidden states containing temporal information from the GRU output. The shared encoder first processes the hidden states to extract high-level features.
[0080] The specific method for step S4 is as follows:
[0081] S41. For each candidate neighbor j of the current node, combine the high-level feature with the feature of neighbor j and input it into an evaluation branch. This branch outputs an L-dimensional vector. Where L is the power level number, The expected long-term return Q value represents the joint action of selecting neighbor j and using the l-th power.
[0082] S42. Compare the maximum value of the Q-vector corresponding to all candidate neighbor j, i.e. Choose the neighbor that maximizes this value as the next hop.
[0083] S43. In the selected next-hop neighbor The corresponding Q vector In the middle, select the power level corresponding to the dimension with the largest value. .
[0084] Step S5 is as follows:
[0085] S51. Initialize training parameters: Number of training iterations =300000, batch sampling quantity =32, discount factor γ=0.99, soft update rate η=0.005, learning rate 0.0005; initialize GAT parameters, including the number of attention heads n_heads, hidden layer size hidden_size, and GRU parameters. Q. Network parameters optim_eps, initial exploration rate ϵ; S52. Construct experience replay buffer D to store the agent's transition experience. ;
[0086] S53. Perform E rounds of training, each episode including:
[0087] S531. Environment initialization, obtaining initial observations. This can be represented as a graph structure;
[0088] S532. Perform experience collection, executing the following steps for each time step t:
[0089] via GAT encoder Observation Encoding as node embedding Selecting joint actions using an ε-greedy strategy Explore randomly with probability ε, and select the action with the highest Q value with probability 1-ε; execute within the environment. Receive rewards and the next state ; transfer experience Store in buffer D;
[0090] S533. When the number of samples in buffer D reaches the batch size, execute: randomly sample a batch of empirical data from D; calculate the TD error. ,in The optimal action for the next state; calculate the mean squared error loss L= Update the GAT parameters θ and Q network parameters using gradient descent. ; Soft update the target network, execute every N steps. ,in This refers to the soft update rate.
[0091] S54. Output the trained model. After training, output the optimized GAT encoder, GRU layer and Q network parameters.
[0092] Step S6 is as follows:
[0093] S61. Deploy the trained GAT encoder, GRU layer, and action branch Q network to the UAV node;
[0094] S62. Obtain local observation information from UAV nodes in real time. This includes its own cumulative power and the characteristic information of its one-hop neighbors;
[0095] S63, Generating state representation, observing The input is a heterogeneous graph; node embeddings are generated using a GAT encoder; the node embeddings are input into a GRU layer, and the current hidden state is generated by combining the historical states. ;
[0096] S64, Based on Hidden State Each candidate neighbor is generated through an action-branch Q-network. The rating vector is used to perform routing decisions, selecting the neighbor with the highest rating as the next-hop node; power decisions are also performed, selecting the power level corresponding to the highest rating from the rating vectors of the selected neighbors.
[0097] S65. Execute the action: forward the data packet according to the selected next-hop node and transmission power;
[0098] S66. Iterative forwarding: The next hop node repeats steps S42-S45 until the data packet reaches the destination node or the maximum number of hops is reached.
[0099] This method uses dynamic modeling, intelligent decision-making, simulation training, and actual deployment as its core framework to construct an end-to-end UAV network routing and power joint optimization system. The specific principles are as follows:
[0100] I. Dynamic Network Modeling
[0101] A network representation system adapted to the high mobility of UAVs is constructed, abstracting the UAV network into a dynamic directed graph, and using discrete-time snapshots to synchronize topology changes such as node positions and communication links. Channel characteristics and interference models are integrated, and a LoS line-of-sight channel gain calculation model is constructed based on the ITU-RP.1411 standard. The link capacity is quantified by combining the Shannon formula. At the same time, node mobility models (random waypoints / Gauss-Markov) and power constraints (maximum transmit power, discrete power levels) are integrated to provide accurate network environment input for subsequent decision-making.
[0102] II. Graph-based reinforcement learning decision-making
[0103] Construct an intelligent decision-making architecture that combines topology awareness, temporal memory, and joint decision-making:
[0104] Topology Aware Module (GAT): Processes heterogeneous graph input through a multi-head attention mechanism, extracts dynamic features of nodes and links in a "message passing-aggregation-update" mode, and generates node embedding vectors containing topological relationships, solving the problem that traditional algorithms have difficulty capturing dynamic topology;
[0105] Temporal Memory Module (GRU): It integrates node embedding vectors and historical decision states to output a contextual representation containing temporal dependencies, thereby alleviating the limitations of local observations and adapting to the temporal characteristics of multi-hop transmission.
[0106] Joint Decision Module (Action Branch Q Network): Employing a branch structure of dual online Q network and dual objective Q network, it generates multi-dimensional score vectors for candidate neighbors, simultaneously completing next-hop route selection and optimal power level decision, achieving coordinated optimization of routing and power; at the same time, it models the decision process as a partially observable Markov decision process (POMDP), guiding the model to learn the optimal strategy through a multi-dimensional reward mechanism (power consumption penalty, throughput reward, timeout penalty).
[0107] III. Model Training and Deployment
[0108] A low-cost deployment model combining simulation pre-training and actual deployment is adopted. Multi-scenario training tasks are constructed in the simulation environment. Stable model training is achieved through experience playback and ε-greedy exploration strategies. Training efficiency and convergence are optimized by combining Double Q-learning and soft update mechanism. After the training is mature, the model is deployed to real drone nodes in a lightweight manner. The nodes quickly output decision results based on the local real-time collected network status (location, channel, interference, etc.).
[0109] Reference Figure 4 This embodiment provides a joint optimization system for UAV network routing and power based on graph reinforcement learning. The system adopts a hierarchical and modular design, with core modules including an information acquisition module, a graph encoding module, a temporal modeling module, a decision evaluation module, an experience management module, a training control module, and a decision execution module. These modules are tightly coupled through data flow, collectively forming an end-to-end joint optimization system. Specifically:
[0110] The information acquisition module, as the foundation of the system, is responsible for acquiring raw observation information from the UAV network environment in real time.
[0111] The graph encoding module is connected to the information acquisition module, receives the observation information input from it, and processes it using a graph neural network to generate embedding vectors that represent the network topology and node relationships.
[0112] The temporal modeling module is connected to the graph coding module, receives the embedded vectors output by the module, and outputs a system state representation containing temporal context information by fusing historical state information.
[0113] The decision evaluation module, connected to the temporal modeling module, is the core of the system's intelligent decision-making. This module includes a dual online Q-network and a dual-objective Q-network. The dual online Q-network, based on a deep reinforcement learning algorithm, receives the state representation output by the temporal modeling module and, using its internal shared encoder, edge scoring sub-network, routing unit, and power selection unit, collaboratively calculates and outputs a joint optimization strategy, namely, the next-hop routing selection and transmit power level.
[0114] The decision execution module is connected to the decision evaluation module. It receives the joint optimization strategy generated by the module and, through its internal observation and acquisition, graph construction, state encoding, action generation, command execution and state update sub-units, parses the strategy into specific actions and applies them to the UAV network environment, while completing the state transition.
[0115] The experience management module is connected to the time series modeling module and the decision evaluation module respectively, and is used to store historical experience data generated by the interaction between the system and the environment, including status, actions, and rewards.
[0116] The training control module is responsible for the system's model training and optimization. This module includes a simulation environment unit, a parameter update unit, an exploration control unit, a network synchronization unit, and a performance monitoring unit. Utilizing data stored in the experience management module, it continuously trains and improves the performance of the neural network models in components such as the decision evaluation module by calculating loss, updating network parameters, controlling exploration strategies, and synchronizing network weights.
[0117] This invention uses a weighted index of link quality and energy consumption as the optimization target, which can quickly adapt to the drastic topology changes caused by the high mobility of UAVs and reduce deployment costs and communication overhead.
[0118] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A method for joint optimization of routing and power in UAV networks based on graph reinforcement learning, characterized in that, Includes the following steps: S1. A virtual UAV network is constructed through simulation. The raw information of each UAV node in the UAV network is collected with a fixed discrete time step k, including UAV node location, neighbor information, channel gain, and interference power. S2, for each UAV node, at each time step k, based on the collected raw information, calculate its own cumulative power consumption, neighbor validity identifier, relative distance with each neighbor, relative distance with the destination node, channel gain, and aggregated interference of all neighbors, to form the state information of the current time step. ; S3. The state s of the drone node at time step k, the action a it performs, the reward r it receives from the environment, and the state s' of the next time step k+1 are combined into a quadruple (s, a, r, s') as a training sample. Multiple training samples are collected to form a training sample set; where the reward r is the minimum link capacity among all links on the multi-hop routing path. Action 'a' is the next-hop neighbor node and its transmit power. S4. Construct a graph reinforcement learning decision model. The input of the model is a quadruple of the drone node, and the output is the next-hop neighbor node of the drone node and the transmission power. S5, Use the training sample set to train the graph reinforcement learning decision model; S6. The trained graph reinforcement learning decision model is deployed on real drone nodes. The real drone nodes collect their own raw information, including location, neighbor information, channel gain, and interference power. The raw information is processed into state information, and then the state information is input into the graph reinforcement learning decision model to obtain the next-hop neighbor node and the transmission power.
2. The method for joint optimization of UAV network routing and power based on graph reinforcement learning according to claim 1, characterized in that, In step 1, the specific method for constructing a virtual drone network through simulation is as follows: S11, set up U drone nodes, all randomly distributed within a D×D square area, and each equipped with a single omnidirectional antenna; set the communication transmission distance of the drone nodes to R; set the maximum transmit power of each drone node to be [value missing]. ; S12, Set the position of the drone node to change with time snapshot k, specifically by generating a displacement vector based on the movement rules of random waypoints or Gaussian Markov movement models. , the displacement vector Overlaying the current position gives the position of the next time snapshot; S13, abstract the UAV network as a dynamic directed graph, where vertices correspond to UAV nodes. Only when the distance between two UAV nodes does not exceed the transmission distance R, a directed edge representing a feasible communication link is established. Data flows from a randomly selected source node to a destination node, and is transmitted in the drone network via multi-hop routing; S14 uses the ITU-R P.1411 standard to model the LosS line-of-sight channel gain, where the channel gain... It's about distance. The piecewise function is expressed as follows: in, For basic transmission loss, Distance from breakpoint Antenna gain; S15. During the simulation, for each data transmission, the signal-to-noise ratio received by the receiving node is calculated. The noise power in the signal-to-noise ratio is the sum of the aggregated interference power generated by all other nodes transmitting data and the system background noise power. S16, using Shannon's formula to calculate link capacity. Determine the link capacity, where B is the bandwidth. This refers to the signal-to-noise ratio.
3. The method for joint optimization of UAV network routing and power based on graph reinforcement learning according to claim 1, characterized in that, The graph reinforcement learning decision model includes a graph attention network, a gated recurrent unit, and an action branch Q-network; wherein: Graph Attention Networks employ a multi-head attention mechanism to handle dynamically changing heterogeneous graph inputs, generating node embedding vectors through a "message passing-aggregation-update" processing method. The gated recurrent unit receives the node embedding vectors and historical states output by the graph attention network, and outputs the hidden state representation to capture temporal dependencies; The action branch Q-network employs a shared encoder and branch evaluation structure, and sets up two online Q-networks with identical structures but independent parameters. and The action branch Q-network works as follows: S421, Hidden state based on the output of the gated loop unit Generate a [number] for each candidate neighbor j. dimensional score vector ; S422, compare the maximum value among all candidate neighbor score vectors, and select the neighbor corresponding to the maximum value as the next hop node; S423, in the score vector corresponding to the selected next-hop node, select the dimension k with the largest value. The discrete power level corresponding to this dimension is the optimal transmission power level.
4. The method for joint optimization of UAV network routing and power based on graph reinforcement learning according to claim 1, characterized in that, The specific method of step S5 is as follows: S51 sets the maximum number of iterations, batch size, discount factor, soft update rate, learning rate, and initial exploration rate. , where 0 < <1; S52, the graph reinforcement learning decision model selects a joint action at each time step using an ϵ-greedy policy. Specifically, it performs random exploration with probability ϵ and selects the joint action with the highest current Q-value with probability 1−ϵ. By routing action and power action Together they constitute the following; when the number of samples in the buffer reaches the batch sampling size, gradient updates are performed, specifically including: S521 employs a Double Q-learning mechanism to calculate the temporal difference target value, uses an online Q-network to select the optimal action for the next state, and uses a target Q-network to evaluate the value of the optimal action, thereby calculating the learning target; wherein, the parameters of the target Q-network are periodically copied from the online Q-network; S522 uses the mean squared error to calculate the loss function, minimizes the temporal difference error through gradient descent, and updates the parameters of the online Q-network. ; S523, for target network parameters Perform a soft update every N steps; S53: When the number of training iterations reaches the maximum number of iterations, training is terminated.