An unmanned aerial vehicle ad hoc network double-layer Q learning dynamic routing method based on multi-dimensional state perception fusion

Abstract

The application discloses a kind of unmanned aerial vehicle self-organizing network double-layer Q learning dynamic routing method based on multi-dimensional state perception fusion, to solve the problem that unmanned aerial vehicle node high-speed movement leads to topology dramatic change in prior art, unmanned aerial vehicle battery energy is limited, global routing decision perception performance weak problem, realize the collaborative optimization of unmanned aerial vehicle self-organizing network dynamic routing reliability, energy efficiency and task adaptability.The method includes that node is based on two-hop neighbor perception mechanism to realize topology comprehensive perception, and the interval of Hello information package is sent by dynamic time slot allocation adjustment;Construct "link-topology-energy-task" multidimensional fusion state model;Based on the double-layer decision framework of Q (λ) learning, combine routing hole dynamic avoidance mechanism to optimize routing selection;Balancing multi-optimization target using multi-objective reward function.The application has good environmental adaptability and generalization ability, can adapt to different network size and node moving speed scene, significantly improve packet delivery rate, reduce energy consumption and shorten end-to-end delay.

Description

Technical Field

[0001] This invention relates to a two-layer Q-learning dynamic routing method for UAV ad hoc networks based on multi-dimensional state awareness fusion, belonging to the field of UAV communication and routing technology. Background Technology

[0002] With the deep integration of drone technology and communication technology, drone self-organizing networks have been widely applied in several key areas such as disaster emergency response, environmental monitoring, intelligent transportation, and precision agriculture, thanks to their advantages of not relying on fixed infrastructure, flexible deployment, and wide coverage. In disaster relief scenarios, drone swarms can quickly build temporary communication networks to transmit rescue instructions and contact affected people, buying valuable time for disaster relief. In environmental monitoring tasks, multi-drone collaborative networking can complete tasks such as meteorological data collection and ecological environment surveys over large areas. In the field of intelligent transportation, drone self-organizing networks can assist in functions such as vehicle dynamic tracking and traffic flow scheduling, significantly improving traffic management efficiency.

[0003] The core requirement of UAV ad hoc networks is to achieve efficient, reliable, and low-latency data transmission. Routing protocols, as a key technology for ensuring network communication performance, directly determine the stability of data transmission and the efficiency of resource utilization. However, the inherent characteristics of UAV ad hoc networks present numerous challenges to routing protocol design: Firstly, the high-speed mobility of UAV nodes leads to frequent changes in network topology and continuous dynamic switching of link connections. Traditional static routing protocols struggle to adapt quickly to these topology changes, easily resulting in link interruptions and data loss. Secondly, UAV nodes are limited by battery capacity, have scarce energy resources, and limited computing and communication capabilities. Routing protocols must ensure communication quality while minimizing energy consumption and extending network lifetime. Furthermore, in complex application scenarios, data transmission often has explicit Quality of Service (QoS) requirements. For example, command transmission in emergency rescue requires millisecond-level latency, and environmental monitoring data requires high reliability. This places higher demands on the multi-objective optimization capabilities of routing protocols.

[0004] To address these issues, various routing protocols for UAV ad hoc networks have been proposed, primarily categorized into topology-based and geolocation-based protocols. Topology-based protocols construct routing tables by maintaining global topology information; however, in highly dynamic networks, topology updates incur significant overhead, network convergence is slow, and they struggle to adapt to frequently changing network environments. Geolocation-based protocols, on the other hand, only require obtaining the location information of neighboring nodes to achieve data forwarding, offering advantages such as low control overhead and strong scalability, making them more suitable for UAV ad hoc networks. Among these, the GPSR protocol employs a combination of greedy forwarding and perimeter forwarding. In the greedy forwarding phase, the nearest neighbor to the destination node is selected as the next hop, while in the perimeter forwarding phase, routing holes are bypassed using right-hand rules. However, traditional GPSR protocols still have significant shortcomings in practical applications: greedy forwarding is prone to local minimization problems, and perimeter forwarding may result in path redundancy and detours, leading to increased transmission latency. Furthermore, these protocols mostly consider only the single factor of node geolocation, failing to fully integrate multi-dimensional state information such as link quality, node energy, and task priority, making it difficult to achieve coordinated optimization of communication reliability, energy efficiency, and QoS requirements.

[0005] In recent years, reinforcement learning technology has received widespread attention in the field of route optimization due to its strong environmental adaptability and self-learning capabilities. Compared with traditional path planning algorithms such as A* and Dijkstra, reinforcement learning can learn environmental patterns autonomously through trial and error mechanisms and dynamically adjust decision-making strategies, making it more suitable for handling the dynamic and uncertain problems of UAV self-organizing networks. Existing reinforcement learning-based routing methods mostly adopt a single Q-learning framework, which optimizes route decisions by constructing a state space, action space, and reward function. However, it still has many limitations in practical applications: some methods only focus on single state dimensions such as link quality or node energy, lacking a comprehensive perception and fusion of the multi-dimensional states of the network, resulting in insufficient decision accuracy; some methods adopt a centralized learning architecture, which is difficult to adapt to the distributed collaboration requirements of large-scale UAV swarms, and suffers from slow convergence speed and untimely response to dynamic topology changes during the learning process; in addition, for topology anomalies such as route holes, the avoidance mechanisms of existing methods are mostly relatively simple, making it difficult to quickly construct the optimal detour path in complex scenarios, affecting the continuity and reliability of data transmission.

[0006] Therefore, designing a routing decision method that can fully integrate multi-dimensional state information, quickly adapt to dynamic topology changes, and balance energy efficiency and QoS requirements has become a key technical problem that urgently needs to be solved in the field of UAV ad hoc networks. This is of great significance for improving the communication performance and practical application value of UAV ad hoc networks. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings and deficiencies of existing technologies by proposing a two-layer Q-learning dynamic routing method for UAV ad hoc networks based on multi-dimensional state awareness fusion. This method aims to solve problems such as drastic topology changes caused by high-speed movement of UAV nodes, limited UAV battery energy, and weak global routing decision-making perception performance in existing technologies, achieving coordinated optimization of dynamic routing reliability, energy efficiency, and task adaptability in UAV ad hoc networks. The method includes: nodes achieving comprehensive topology awareness based on a two-hop neighbor awareness mechanism; adjusting the Hello packet sending interval through dynamic time slot allocation; constructing a multi-dimensional fused state model of "link-topology-energy-task"; designing a two-layer decision framework based on Q(λ) learning, combined with a dynamic routing hole avoidance mechanism to optimize route selection; and employing a multi-objective reward function to balance multiple optimization objectives. This invention possesses good environmental adaptability and generalization ability, can adapt to different network sizes and node movement speeds, significantly improves packet delivery rate, reduces energy consumption, and shortens end-to-end latency.

[0008] The technical solution adopted by this invention to solve its technical problem is: a two-layer Q-learning dynamic routing method for UAV ad hoc networks based on multi-dimensional state-aware fusion, which includes the following steps: Step 1: Using each drone in the drone ad hoc network as a node, all nodes send Hello packets based on a two-hop neighbor awareness mechanism. The packets carry the drone's own state, the state of its one-hop neighbors, and a topology summary of its two-hop neighbors. A dynamic time slot allocation mechanism is used to adjust the Hello packet sending interval. After receiving a Hello packet, any node constructs a neighbor table containing a node-link dual-layer state and updates information such as the node's three-dimensional position, movement vector, remaining battery power, link quality level, and two-hop reachability in real time. Step 2: Construct a multi-dimensional state space for reinforcement learning, based on the dynamic topology of the UAV ad hoc network, channel state information, remaining battery energy of the UAV, source and destination node locations, data volume, latency requirements, QoS requirements, and task priority requirements (data transmission urgency). Define the selection of the next-hop node for the UAV as the action space. Each node obtains a multi-dimensional fused state through Hello packets, including local link state, two-hop topology association state, energy balance state, and task priority state. Then, it selects the next-hop node in the action space. After the node completes the routing decision, the system environment provides a reward value based on multi-objective optimization criteria. Step 3: The node adopts a "distributed collaborative + local independent learning" architecture. Based on the Q (λ) learning framework, a two-layer decision model is built. The upper layer filters the candidate next-hop set through global Q value evaluation, and the lower layer dynamically adjusts the routing weight based on local instant rewards. The learning parameters are adaptively updated by combining experience replay and qualification trace mechanism. At the same time, the topology anomaly is handled through the dynamic avoidance mechanism of routing holes. Finally, the optimal next-hop node is selected to complete the multi-hop routing of data packets. In step 2 above, the multi-dimensional state space S t for: State dimension (where m is the number of one-hop neighbors of node i), and the dimensions are defined as follows: Lᵢ represents the local link state set. Let m be the number of one-hop neighbors of node i, where each (3 sub-dimensions) The expected connection time of the link. For the link packet error rate, The link signal interference ratio; It represents a two-hop topological association state, which includes two sub-dimensions: the number of two-hop neighbor reachables and the topological stability coefficient. The energy balance state includes two sub-dimensions: its own remaining power and the average remaining power of neighboring nodes. Assign urgency weights to task priority status (1 sub-dimension) based on data type; The action space At is: (There are m actions in total), aⱼ (j=1~m) is the action of selecting the j-th one-hop neighbor as the next hop node; the action selection adopts the ε-greedy strategy (ε∈[0.1,0.3], which decays linearly with iteration); The reward function Rt for system environment feedback is: When neighbor node j is the target node and the link quality meets the QoS requirements. (Rmax is the preset maximum reward value, and Mᵢ is the current task priority weight). When all neighboring nodes fail to meet two-hop reachability or the drone's remaining battery power falls below a threshold. ; otherwise, ,in , The maximum link duration threshold. The average remaining power of neighboring nodes. The topological stability coefficient, Let be the remaining battery power of neighboring node j. This represents the initial charge level of the node. This is the signal-to-interference ratio threshold.

[0009] Furthermore, in step 1 of the present invention, the two-hop neighbor topology summary is a two-hop topology description information generated by a node based on its own one-hop neighbor information and nested forwarded neighbor feedback information, which is used to quickly construct the two-hop topology association state; 1) The summary contains the following information: basic identification information, topology association information, and key status information. The basic identification information refers to the ID list of the one-hop neighbor node and the ID list of the two-hop neighbor node corresponding to each hop neighbor. The topology association information refers to the association relationship identifier of each "self-one neighbor-two neighbors" and the reachability mark of the two-hop neighbor. The key status information refers to the remaining power status and link quality status of the one-hop neighbor. 2) Abstract generation process; Nodes collect the IDs, remaining battery power, and link quality levels of their own one-hop neighbors to form a basic information set of one-hop neighbors; The node receives the Hello message packet forwarded by the one-hop neighbor, which contains information such as the one-hop neighbor ID and link quality level of the neighbor. The node extracts and filters out neighbor nodes that are not itself as candidates for two-hop neighbors. Deduplication is performed on the two-hop neighbor candidates, removing duplicate IDs and their own IDs to determine the final set of two-hop neighbor IDs; Establish an association mapping table of "self-one-hop neighbor-two-hop neighbor" and mark the reachability state of each two-hop neighbor; Status information such as link quality and remaining power is quantified by level (high=3, medium=2, low=1). The above information is encapsulated in the structure of "identification information - association information - status digest" to form a two-hop neighbor topology digest, which is then embedded in the Hello message packet and sent.

[0010] Furthermore, the method for calculating the Hello message packet transmission interval in the dynamic time slot allocation mechanism of step 1 of the present invention is as follows: in, and These are the preset shortest and longest time intervals, respectively. Let i be the maximum moving speed of node i. Let i be the average moving speed of node i. Let be the duration of the link between node i and its neighbor node j. Let be the two-hop topological correlation coefficient (α∈[0.5,1.5]). is the topological stability coefficient.

[0011] Furthermore, the two-hop neighbor perception mechanism in step 1 of the present invention is implemented through nested forwarding of Hello information packets: the Hello information packet sent by the node contains its own ID, state parameters and a summary of the one-hop neighbor list. After receiving the packet, the neighboring node supplements its own one-hop neighbor information and forwards it to its adjacent nodes. The receiving node constructs a two-hop topology graph based on the nested information and calculates the two-hop reachability and topology stability coefficient.

[0012] Furthermore, the parameter update method for the improved Q(λ) learning bi-layer decision model in step 3 of this invention is as follows: Upper-level global Q-value update: Lower-level local Q-value update: Final decision Q value: in, The global learning rate ( ), Local learning rate ( ), Global discount factor ( , (Number of two-hop neighbors) For local discount factor ( ); and These are global and local eligibility traces, respectively. For localized, immediate rewards; ω represents the decision weights (ω∈[0.6,0.8]). , , , M represents the total number of nodes in the network.

[0013] Furthermore, step 3 of the present invention also includes a dynamic routing hole avoidance mechanism: when a routing hole is detected at the current node, emergency scheduling of two-hop neighbors is triggered, the optimal two-hop relay node is selected based on Q-value sorting, a cross-hole forwarding path is constructed, and nodes on this path are given additional rewards. This accelerates path learning.

[0014] Furthermore, the experience replay mechanism in step 3 of this invention adopts a priority experience replay strategy: experience priorities are allocated according to the absolute value of the TD error of the routing decision, and the sampling probability is positively correlated with the priority; after each update of the Q network parameters, the experience priorities are recalculated and the sampling distribution is dynamically adjusted to improve the utilization efficiency of high-value experience.

[0015] Beneficial effects: 1. This invention effectively solves the problems of rapid topology changes caused by high-speed movement of UAV nodes, limited battery energy of UAVs, and weak global routing decision-making perception performance in the prior art. It achieves coordinated optimization of the reliability, energy efficiency and task adaptability of dynamic routing in UAV ad hoc networks.

[0016] 2. This invention has excellent environmental adaptability and generalization ability, and can be adapted to different network scales and node movement speeds, significantly improving data packet delivery rate, reducing energy consumption and shortening end-to-end latency. Attached Figure Description

[0017] Figure 1 This is a scene diagram of the self-organizing network of the UAV according to the present invention.

[0018] Figure 2 This is a flowchart of the method of the present invention.

[0019] Figure 3 This is a diagram of the two-layer Q-learning network structure of the present invention. Detailed Implementation

[0020] The invention will now be described in further detail with reference to the accompanying drawings.

[0021] like Figures 1 to 3 As shown, this invention provides a two-layer Q-learning dynamic routing method for UAV ad hoc networks based on multi-dimensional state-aware fusion, which includes the following steps: Step 1: Using each drone in the drone ad hoc network as a node, all nodes send Hello packets based on a two-hop neighbor awareness mechanism. The packets carry the drone's own state, the state of its one-hop neighbors, and a topology summary of its two-hop neighbors. A dynamic time slot allocation mechanism is used to adjust the Hello packet sending interval. After receiving a Hello packet, any node constructs a neighbor table containing a node-link dual-layer state and updates information such as the node's three-dimensional position, movement vector, remaining battery power, link quality level, and two-hop reachability in real time. Step 2: Construct a multi-dimensional state space for reinforcement learning, based on the dynamic topology of the UAV ad hoc network, channel state information, remaining battery energy of the UAV, source and destination node locations, data volume, latency requirements, QoS requirements, and task priority requirements (data transmission urgency). Define the selection of the next-hop node for the UAV as the action space. Each node obtains a multi-dimensional fused state through Hello packets, including local link state, two-hop topology association state, energy balance state, and task priority state. Then, it selects the next-hop node in the action space. After the node completes the routing decision, the system environment provides a reward value based on multi-objective optimization criteria. Step 3: The node adopts a "distributed collaborative + local independent learning" architecture. Based on the Q (λ) learning framework, a two-layer decision model is built. The upper layer filters the candidate next-hop set through global Q value evaluation, and the lower layer dynamically adjusts the routing weight based on local instant rewards. The learning parameters are adaptively updated by combining experience replay and qualification trace mechanism. At the same time, the topology anomaly is handled through the dynamic avoidance mechanism of routing holes. Finally, the optimal next-hop node is selected to complete the multi-hop routing of data packets. The multi-dimensional state space St in step 2 above is: State dimension (where m is the number of one-hop neighbors of node i), and the dimensions are defined as follows: Lᵢ represents the local link state set. Let m be the number of one-hop neighbors of node i, where each (3 sub-dimensions) The expected connection time of the link. For the link packet error rate, The link signal interference ratio; It represents a two-hop topological association state, which includes two sub-dimensions: the number of two-hop neighbor reachables and the topological stability coefficient. The energy balance state includes two sub-dimensions: its own remaining power and the average remaining power of neighboring nodes. Assign urgency weights to task priority status (1 sub-dimension) based on data type; The action space At is: (There are m actions in total), aⱼ (j=1~m) is the action of selecting the j-th one-hop neighbor as the next hop node; the action selection adopts the ε-greedy strategy (ε∈[0.1,0.3], which decays linearly with iteration); The reward function Rt for system environment feedback is: When neighbor node j is the target node and the link quality meets the QoS requirements. (Rmax is the preset maximum reward value, and Mᵢ is the current task priority weight). When all neighboring nodes fail to meet two-hop reachability or the drone's remaining battery power falls below a threshold. ; otherwise, ,in , The maximum link duration threshold. The average remaining power of neighboring nodes. The topological stability coefficient, Let be the remaining battery power of neighboring node j. This represents the initial charge level of the node. This is the signal-to-interference ratio threshold.

[0022] In step 1 of this invention, the two-hop neighbor topology summary is a two-hop topology description information generated by a node based on its own one-hop neighbor information and nested forwarded neighbor feedback information, which is used to quickly construct the two-hop topology association state. 1) The summary contains the following information: basic identification information, topology association information, and key status information. The basic identification information refers to the ID list of the one-hop neighbor node and the ID list of the two-hop neighbor node corresponding to each hop neighbor. The topology association information refers to the association relationship identifier of each "self-one neighbor-two neighbors" and the reachability mark of the two-hop neighbor. The key status information refers to the remaining power status and link quality status of the one-hop neighbor. 2) Abstract generation process; Nodes collect the IDs, remaining battery power, and link quality levels of their own one-hop neighbors to form a basic information set of one-hop neighbors; The node receives the Hello message packet forwarded by the one-hop neighbor, which contains information such as the one-hop neighbor ID and link quality level of the neighbor. The node extracts and filters out neighbor nodes that are not itself as candidates for two-hop neighbors. Deduplication is performed on the two-hop neighbor candidates, removing duplicate IDs and their own IDs to determine the final set of two-hop neighbor IDs; Establish an association mapping table of "self-one-hop neighbor-two-hop neighbor" and mark the reachability state of each two-hop neighbor; Status information such as link quality and remaining power is quantified by level (high=3, medium=2, low=1). The above information is encapsulated in the structure of "identification information - association information - status digest" to form a two-hop neighbor topology digest, which is then embedded in the Hello message packet and sent.

[0023] The method for calculating the Hello packet transmission interval in the dynamic time slot allocation mechanism of step 1 of this invention is as follows: in, and These are the preset shortest and longest time intervals, respectively. Let i be the maximum moving speed of node i. Let i be the average moving speed of node i. Let be the duration of the link between node i and its neighbor node j. Let be the two-hop topological correlation coefficient (α∈[0.5,1.5]). is the topological stability coefficient.

[0024] In step 1 of this invention, the two-hop neighbor perception mechanism is implemented through nested forwarding of Hello information packets: the Hello information packet sent by a node contains its own ID, state parameters and a summary of the list of one-hop neighbors. After receiving the packet, the neighboring node supplements its own one-hop neighbor information and forwards it to its adjacent nodes. The receiving node constructs a two-hop topology graph based on the nested information and calculates the two-hop reachability and topology stability coefficient.

[0025] The parameter update method for the improved Q(λ) learning two-layer decision model in step 3 of this invention is as follows: Upper-level global Q-value update: Lower-level local Q-value update: Final decision Q value: in, The global learning rate ( ), Local learning rate ( ), Global discount factor ( , (Number of two-hop neighbors) For local discount factor ( ); and These are global and local eligibility traces, respectively. For localized, immediate rewards; ω represents the decision weights (ω∈[0.6,0.8]). , , , M represents the total number of nodes in the network.

[0026] Step 3 of this invention also includes a dynamic routing hole avoidance mechanism: when a routing hole is detected at the current node, emergency scheduling of two-hop neighbors is triggered, the optimal two-hop relay node is selected based on Q-value sorting, a cross-hole forwarding path is constructed, and nodes on this path are given additional rewards. This accelerates path learning.

[0027] In step 3 of this invention, the experience replay mechanism adopts a priority experience replay strategy: experience priorities are allocated according to the absolute value of the TD error of the routing decision, and the sampling probability is positively correlated with the priority; after each update of the Q network parameters, the experience priority is recalculated and the sampling distribution is dynamically adjusted to improve the utilization efficiency of high-value experience.

[0028] In summary, this invention details a two-layer Q-learning dynamic routing decision-making method for UAV ad hoc networks based on multi-dimensional state-aware fusion. First, a neighbor table is constructed by exchanging information with neighboring nodes using Hello message packets. Next, a multi-dimensional state space is built and action selection is performed. Then, a reward function is triggered to update the node state. Next, non-compliant nodes are avoided through routing holes. Finally, a two-layer Q-learning decision model completes parameter updates and experience replay, thereby achieving the optimal routing strategy output for UAV ad hoc networks under dynamic conditions. The pseudocode of the method is as follows: Algorithm 1: Unmanned Aerial Vehicle (UAV) Multi-hop Routing Algorithm Based on Improved Q(λ) Learning Input: Total number of nodes in the network K, maximum number of iterations E, experience replay batch size B, initial policy parameters θ 0 , Initial value function parameters Cutting range Discount factor , Qualification trace parameters , Maximum reward Energy threshold Output: Optimal routing strategy parameters 1: Initialization: Each node Independent initialization - Neighbor table Two hops can reach a number - Global Q-table Local Q-table - Experience replay buffer Exploration rate - Local policy parameters Local value parameters 2: for global iteration count to E do 3: for each node to K, execute do in parallel 4: / / Step 1: Hello message exchange and neighbor table update 5: Calculate the dynamic Hello interval : 6: Broadcast Hello message packet (self-state + one-hop neighbor summary) 7: Receive Hello packets from neighbors and update. and 8: Calculate the topological stability coefficient and energy equilibrium state 9: / / Step 2: Multidimensional State Construction and Reward Calculation 10: Collect environmental conditions : 11: Action selection based on ε-greedy strategy (Next hop node index): if then else end if 12: Perform the action Obtain environmental feedback rewards : if the neighbor corresponding to the action is the target node and then elif All neighbors are unreachable or then Else end if 13: Node movement update status , trajectory deposit 14: / / Exploration rate decays linearly 15: / / Step 3: Route hole avoidance 16: Determine a route hole: If all neighbors are more distant from the target node than the current node, and... then Filter the set of two-hop transit nodes that meet the criteria u∈two-hop neighbors and Determine the one-hop neighbor corresponding to the transit Supplementary Rewards end if 17: / / Experience replay and parameter update 18: if | |≥B then from Randomly sampled batch data for each do / / Global Q-value update (with eligibility criteria) / / Local Q-value update (with eligibility criteria) end for end if 19: end for 20: / / Distributed collaboration: synchronizing optimal parameters (based on global reward sorting) 21: Calculate the cumulative reward for each node k. 22: Select the parameters of the Top-10% nodes as the globally optimal parameters. , 23: All nodes are updated synchronously. , 24: end for 25: Output the optimal policy parameters for all nodes. The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

[0029] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A two-layer Q-learning dynamic routing method for UAV ad hoc networks based on multi-dimensional state-aware fusion, characterized in that, The method includes the following steps: Step 1: Using each drone in the drone ad hoc network as a node, all nodes send Hello packets based on a two-hop neighbor awareness mechanism. The packets carry the drone's own state, the state of its one-hop neighbors, and a topology summary of its two-hop neighbors. A dynamic time slot allocation mechanism is used to adjust the Hello packet sending interval. After receiving a Hello packet, any node constructs a neighbor table containing a node-link dual-layer state and updates information such as the node's three-dimensional position, movement vector, remaining battery power, link quality level, and two-hop reachability in real time. Step 2: Construct a multi-dimensional state space for reinforcement learning, based on the dynamic topology of the UAV ad hoc network, channel state information, remaining battery energy of the UAV, source and destination node locations, data volume, latency requirements, QoS requirements, and task priority requirements (data transmission urgency). Define the selection of the next-hop node for the UAV as the action space. Each node obtains a multi-dimensional fused state through Hello packets, including local link state, two-hop topology association state, energy balance state, and task priority state. Then, it selects the next-hop node in the action space. After the node completes the routing decision, the system environment provides a reward value based on multi-objective optimization criteria. Step 3: The node adopts a "distributed collaborative + local independent learning" architecture. Based on the Q (λ) learning framework, a two-layer decision model is built. The upper layer evaluates and selects the candidate next-hop set through global Q value evaluation, and the lower layer dynamically adjusts the routing weight based on local instant rewards. The learning parameters are adaptively updated by combining experience playback and qualification trace mechanism. At the same time, the topology anomaly is handled through the dynamic avoidance mechanism of routing holes. Finally, the optimal next-hop node is selected to complete the multi-hop routing of data packets. The multi-dimensional state space St in step 2 above is: State dimension (where m is the number of one-hop neighbors of node i), and the dimensions are defined as follows: Lᵢ represents the set of local link states. Let m be the number of one-hop neighbors of node i, where each (3 sub-dimensions) The expected connection time of the link. For the link packet error rate, The link signal interference ratio; It represents a two-hop topological association state, which includes two sub-dimensions: the number of two-hop neighbor reachables and the topological stability coefficient. The energy balance state includes two sub-dimensions: its own remaining power and the average remaining power of neighboring nodes. Assign urgency weights to task priority status (1 sub-dimension) based on data type; The action space At is: (There are m actions in total), aⱼ (j=1~m) is the action of selecting the j-th one-hop neighbor as the next hop node; the action selection adopts the ε-greedy strategy (ε∈[0.1,0.3], which decays linearly with iteration); The reward function Rt for system environment feedback is: When neighbor node j is the target node and the link quality meets the QoS requirements. (Rmax is the preset maximum reward value, and Mᵢ is the current task priority weight). When all neighboring nodes fail to meet two-hop reachability or the drone's remaining battery power falls below a threshold. ; otherwise, ,in , The maximum link duration threshold. The average remaining power of neighboring nodes. The topological stability coefficient, Let be the remaining battery power of neighboring node j. This represents the initial charge level of the node. This is the signal-to-interference ratio threshold.

2. The method for dynamic routing of UAV ad hoc networks based on multi-dimensional state-aware fusion using two-layer Q-learning, as described in claim 1, is characterized in that... In step 1, the two-hop neighbor topology summary is a two-hop topology description information generated by a node based on its own one-hop neighbor information and nested forwarded neighbor feedback information, which is used to quickly construct the two-hop topology association state. 1) The information contained in the summary includes: basic identification information, topological association information and key status information. The basic identification information refers to the list of IDs of its own one-hop neighbor nodes and the list of IDs of the two-hop neighbor nodes corresponding to each hop neighbor. Topological association information refers to the association identifier of each group of "self-one neighbor-two neighbors" and the reachability marker of the two-hop neighbors; Key status information refers to the remaining battery status and link quality status of a one-hop neighbor; 2) Abstract generation process; Nodes collect the IDs, remaining battery power, and link quality levels of their own one-hop neighbors to form a basic information set of one-hop neighbors; The node receives the Hello message packet forwarded by the one-hop neighbor, which contains information such as the one-hop neighbor ID and link quality level of the neighbor. The node extracts and filters out neighbor nodes that are not itself as candidates for two-hop neighbors. Deduplication is performed on the two-hop neighbor candidates, removing duplicate IDs and their own IDs to determine the final set of two-hop neighbor IDs; Establish an association mapping table of "self-one-hop neighbor-two-hop neighbor" and mark the reachability state of each two-hop neighbor; Status information such as link quality and remaining power is quantified by level (high=3, medium=2, low=1). The above information is encapsulated in the structure of "identification information - association information - status digest" to form a two-hop neighbor topology digest, which is then embedded in the Hello message packet and sent.

3. The UAV ad hoc network two-layer Q-learning dynamic routing method based on multi-dimensional state-aware fusion according to claim 1, characterized in that, The calculation method for the Hello packet transmission interval in the dynamic time slot allocation mechanism in step 1 is as follows: in, and These are the preset shortest and longest time intervals, respectively. Let i be the maximum moving speed of node i. Let i be the average moving speed of node i. Let be the duration of the link between node i and its neighbor node j. Let be the two-hop topological correlation coefficient (α∈[0.5,1.5]). is the topological stability coefficient.

4. The two-layer Q-learning dynamic routing method for UAV ad hoc networks based on multi-dimensional state-aware fusion according to claim 1, characterized in that, In step 1, the two-hop neighbor perception mechanism is implemented through nested forwarding of Hello packets: the Hello packet sent by a node contains its own ID, state parameters and a summary of the list of one-hop neighbors. After receiving the packet, the neighboring node supplements its own one-hop neighbor information and forwards it to its adjacent nodes. The receiving node constructs a two-hop topology graph based on the nested information and calculates the two-hop reachability and topology stability coefficient.

5. The UAV ad hoc network two-layer Q-learning dynamic routing method based on multi-dimensional state-aware fusion according to claim 1, characterized in that, The parameter update method for the improved Q(λ) learning two-level decision model in step 3 is as follows: Upper-level global Q-value update: Lower-level local Q-value update: Final decision Q value: in, The global learning rate ( ), Local learning rate ( ), Global discount factor ( , (Number of two-hop neighbors) For local discount factor ( ); and These are global and local eligibility traces, respectively. For localized, immediate rewards; ω represents the decision weights (ω∈[0.6,0.8]). , , , M represents the total number of nodes in the network.

6. The UAV ad hoc network two-layer Q-learning dynamic routing method based on multi-dimensional state-aware fusion according to claim 1, characterized in that, Step 3 also includes a dynamic routing hole avoidance mechanism: when a routing hole is detected at the current node, emergency scheduling of two-hop neighbors is triggered, the optimal two-hop relay node is selected based on Q-value sorting, a cross-hole forwarding path is constructed, and nodes on this path are given additional rewards. This accelerates path learning.

7. The UAV ad hoc network two-layer Q-learning dynamic routing method based on multi-dimensional state-aware fusion according to claim 1, characterized in that, In step 3, the experience replay mechanism adopts a priority experience replay strategy: the experience priority is allocated according to the absolute value of the TD error of the routing decision, and the sampling probability is positively correlated with the priority. Each time the Q network parameters are updated, the experience priority is recalculated and the sampling distribution is dynamically adjusted to improve the utilization efficiency of high-value experience.

Images