A flight ad hoc network cooperative routing method based on geographic beacon and reinforcement learning

Abstract

The application discloses a kind of flight ad hoc network cooperation routing method based on geographic beacon and reinforcement learning, specifically: each unmanned aerial vehicle node is periodically sent to neighbor with geographical position, establishes respective neighbor table;And trigger geographic beacon according to period;Then, source node generates data packet, obtains the position of all neighbor nodes and calculates the actual speed of each neighbor node, the neighbor node that is not lower than request speed is kept to candidate neighbor set, otherwise, carry out the avoidance mechanism of routing hole;Then, the link utility value of candidate neighbor is calculated, and the decision score of each other is calculated by fusing historical experience Q value, select the neighbor node of maximum score as next hop forwarding data packet, wait for ACK feedback;Next hop node is taken as current node, and next hop node is repeatedly selected until data packet is successfully delivered to destination node or reaches maximum hop number;The application improves link stability, reduces path failure rate and reduces detour redundancy.

Description

Technical Field

[0001] This invention belongs to the field of UAV communication networks and flight ad hoc network (FANET) routing, specifically involving a flight ad hoc network cooperative routing method based on geographic beacons and reinforcement learning. Background Technology

[0002] In recent years, with the rapid development of the low-altitude economy and the application of drones, drones have taken on more and more tasks in scenarios such as disaster response, forest fire prevention, and emergency rescue. Complex tasks usually require multiple drones to work together, relying on real-time information sharing and distributed collaboration capabilities among drones.

[0003] A flight ad hoc network (FANET) composed of multiple drones can achieve multi-hop communication and collaborative operation among drones in the absence of ground infrastructure or when the infrastructure is damaged. Compared with a single-drone system, it has advantages in task sharing, coverage and fault tolerance.

[0004] However, due to the characteristics of FANET, such as high-speed movement, three-dimensional maneuvering, frequent topology changes, link volatility, and network sparsity, routing protocol design faces more stringent challenges than traditional MANET / VANET. Especially under the conditions of high-speed node movement and rapid link interruption, routing needs to balance real-time performance, reliability, and control overhead: frequent maintenance of routes or broadcasting of global information can easily lead to control signaling congestion and occupy data transmission resources; insufficient maintenance or reliance on local observation can easily lead to path failure and increased probability of data loss.

[0005] The existing FANET routing protocol is divided into two categories: topology-based routing and geographic information-based routing.

[0006] Topology-based routing includes proactive, reactive, and hybrid methods. Proactive routing achieves lower forwarding latency by periodically maintaining the topology, but maintenance overhead is significant in highly dynamic scenarios. Reactive routing reduces maintenance overhead by discovering paths on demand, but there is route discovery delay and paths are prone to rapid failure after establishment. Hybrid routing attempts to compromise between the two, but it is still difficult to simultaneously guarantee maintenance efficiency and real-time performance in highly dynamic and sparse networks.

[0007] In contrast, geographic routing typically utilizes node locations for greedy forwarding, reducing routing table maintenance and control signaling overhead, thus offering advantages in highly dynamic scenarios. However, geographic routing faces the typical "routing hole / local minimum" problem: when there are no neighboring nodes within the current node's communication range that can contribute to forwarding progress, greedy forwarding will cease, potentially leading to detours, path redundancy, or even transmission failure. Furthermore, when the destination node's location changes rapidly or the network is sparse, relying solely on local location for selection can easily result in suboptimal paths or frequent rerouting.

[0008] To improve routing quality, existing technologies typically introduce end-to-end quality parameters or multi-index weighted routing concepts. These methods comprehensively consider factors such as path remaining time, carrying capacity, latency, and load utilization, and adaptively weight the routes to select those with higher overall scores. Other approaches balance coverage and control overhead by announcing routes and dynamically adjusting the broadcast range (e.g., adaptive broadcasting based on hop count or TTL). However, these solutions are usually geared towards a "node-to-gateway / base station" communication paradigm, relying on the announcement and management mechanisms of infrastructure or central nodes. Their assumptions are not entirely consistent with the "arbitrary pairwise communication / multiple mobile destination node communication" scenario of pure UAV ad hoc networks. Furthermore, end-to-end path parameters are prone to rapid obsolescence in highly dynamic topologies. Without coordinated modeling of local transient changes in links and the global situation, issues such as path instability, redundant detours, or insufficient coverage may still arise.

[0009] In recent years, reinforcement learning has been used for routing decisions in FANET, modeling routing as a Markov decision process and gradually optimizing forwarding strategies through interactive learning. Reinforcement learning methods possess a certain degree of environmental adaptability, but still have the following limitations:

[0010] First, routing decisions may rely too much on a single or few indicators, lacking a unified characterization of multi-dimensional states such as link stability, forwarding progress, and neighborhood density.

[0011] Secondly, the learning process is mostly based on local observations, lacking effective perception of the overall evolution of the network situation, and is prone to local optima;

[0012] Third, in extreme scenarios such as routing gaps and sparse disconnections, the lack of robust processing mechanisms causes the strategy to fail or converge slowly at critical moments.

[0013] In summary, existing FANET routing technologies generally face the following contradictions and challenges:

[0014] 1) In highly dynamic topologies, the more comprehensive the global information, the greater the control overhead; however, relying solely on local information can easily lead to suboptimal paths and failed forwarding.

[0015] 2) Although geographical routing has lower overhead, it suffers from problems such as routing holes and path redundancy, making it difficult to guarantee reliability;

[0016] 3) Although multi-index weighting and reinforcement learning methods can improve adaptability, they still have problems with insufficient applicability and robustness under conditions such as multiple moving destination nodes / arbitrary pairwise communication and sparse disconnection.

[0017] Therefore, there is an urgent need for a routing technology that can enhance network situational awareness, improve routing coordination and decision robustness, and adapt to the high dynamics, multiple mobile destination nodes, and arbitrary pairwise communication needs of flight ad hoc networks, while keeping signaling overhead under control. Summary of the Invention

[0018] This invention addresses the problems of high-speed node movement, frequent topology changes, link fragility, and network sparsity leading to routing instability, routing holes, and path redundancy in FANETs by providing a cooperative routing method based on geographic beacons and reinforcement learning.

[0019] The specific steps are as follows:

[0020] Step 1: Setup A flight ad hoc network composed of drone nodes, where each drone node obtains its own position and speed information, periodically sends messages to its neighbors, and each neighbor establishes its own neighbor table;

[0021] In a flight ad hoc network, each node can act as a source node, relay node, or destination node; the node's position and speed change over time.

[0022] drone nodes Periodically package its own motion state information into a HELLO message and broadcast it; neighboring nodes After receiving HELLO, a local neighbor table is created and maintained, recording information such as the identifier, location, speed, timestamp, and number of neighbors for each neighbor node.

[0023] Step 2: The drone node packages its own information and the information in its local neighbor table into its own initial local global information table; and triggers the geographic beacon at a set fixed time period;

[0024] The beacon initially carries a set of geographic information about the node: node identifier, location, speed, and timestamp; as well as the geographic information of each neighbor node in each node's local neighbor table;

[0025] Step 3: The source node generates data packets and retrieves the destination node from the global information table. The positions of all neighboring nodes of the source node.

[0026] Step 4: Based on the request speed, determine whether the actual speed of each neighbor node is not lower than the request speed. If so, keep the neighbor node in the candidate neighbor set and proceed to Step 5; otherwise, proceed to Step 8.

[0027] Request speed is the ratio of the remaining distance from the current node to the destination node to the remaining deadline for the data packet; the initial value is the distance from the source node to the destination node. The ratio of distance to data packet transmission time.

[0028] Actual speed refers to the ratio of the distance between the neighboring node and the destination node to the time it takes for the data packet to travel from the neighboring node to the destination node.

[0029] Step 5: For each neighbor in the candidate neighbor set, calculate the link utility value, and combine the historical experience Q value to calculate the decision score of each neighbor. Select the neighbor node with the highest score as the next hop.

[0030] Current node Passing through neighboring nodes to the destination node Link utility value:

[0031]

[0032] in For reference delay, These are the weighting coefficients; For nodes Jump to neighbor node The time delay; This is the actual link stability index; For nodes to the destination node The distance to the one-hop neighbor node to the destination node The difference in distance; The communication radius that a node can reach in one hop. For nodes The current number of neighbors.

[0033] Candidate Neighbors The overall decision score is:

[0034]

[0035] in Based on historical experience values, As the amplification factor, These are adaptive weighting coefficients.

[0036] Step 6: The current node forwards the data packet to the selected next hop and waits for an ACK response; it checks whether an ACK has been received within the maximum waiting time. If so, it returns to Step 4, sets the next hop node as the current node, and repeats the process of selecting candidate neighbors and selecting the next hop node until the data packet is successfully delivered to the destination node. The process may terminate upon reaching the maximum number of jumps or exceeding the timeout limit; otherwise, proceed to step 7.

[0037] The ACK message contains the actual transmission parameters for this hop: the actual sending time. Reception time Single-hop delay The destination node in the current state is The maximum Q value, etc.

[0038] Step 7: If forwarding the data packet to the next hop node fails, a negative reward is given, and the historical experience Q value is updated. Return to step 5, and use the updated historical experience Q value to reselect the neighbor node with the highest score as the next hop.

[0039] Step 8: If the candidate neighbor set is empty, a routing hole occurs. The following mechanism is used to avoid this:

[0040] The innovative process for handling routing holes is as follows:

[0041] 1) Current node If the candidate neighbor nodes are empty, move the current node to the next node. Set it as a hole node and broadcast a HOLE message to spread the hole experience to other nodes. A one-hop neighbor node prompts surrounding nodes to access the node with that node. The relevant link reward for "the next hop" is set to minimum and the corresponding Q value is updated;

[0042] 2) The recovery scoring criterion of "node degree + destination distance progress" is applied to the current data packet, starting from the current node. Among the neighboring nodes, the one with higher priority and closer to the destination node is selected. The neighboring node is used as the new next-hop node.

[0043] The advantages of this invention are:

[0044] (1) Creation Point 1: Geographic beacon sparse triggering + path enhancement + cached flood suppression + maximum hop count limit enables nodes to obtain global location prior with low control overhead, thereby improving routing availability and scenario adaptability in scenarios that support any pair of communication and multiple mobile destination nodes.

[0045] (2) Creation Point 2: The joint action decision of multidimensional link utility measurement and historical experience Q value, combined with adaptive fusion weight, enables the route selection to both utilize long-term experience and respond to transient link changes in a timely manner, thereby improving link stability, reducing path failure rate and reducing detour redundancy.

[0046] (3) Creation Point 3: Hole penalty update based on HOLE broadcast and recovery forwarding criterion of "node degree + distance progress" enable hole experience to spread locally and quickly correct forwarding selection, thereby effectively mitigating forwarding failure caused by routing holes and improving network robustness and delivery reliability. Attached Figure Description

[0047] Figure 1This is a flowchart of a collaborative routing method for ad hoc networks based on geographic beacons and reinforcement learning, according to the present invention.

[0048] Figure 2 This is a flowchart illustrating the specific steps used in the embodiments of the present invention;

[0049] Figure 3 This is a schematic diagram illustrating the process of generating geographic beacons (GBeacon), path enhancement, multi-hop propagation, and buffered flood suppression used in embodiments of the present invention.

[0050] Figure 4 This is a schematic diagram of the joint action decision-making process that integrates multidimensional link utility measurement and Q-learning historical experience in an embodiment of the present invention.

[0051] Figure 5 This is a flowchart illustrating the routing hole detection, HOLE message broadcast penalty, and recovery forwarding criteria used in embodiments of the present invention.

[0052] Figure 6 This is a comparison curve of packet delivery rate (PDR) under different node movement speed conditions used in the embodiments of the present invention;

[0053] Figure 7 This is a comparison curve of network throughput under different node movement speeds used in the embodiments of the present invention;

[0054] Figure 8 This is a comparison curve of the average end-to-end delay under different node movement speeds used in the embodiments of the present invention;

[0055] Figure 9 This is a comparison of the ablation experiment results using the geographic beacon mechanism employed in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0057] This implementation provides a cooperative routing method for ad hoc flight networks (FANETs) based on geographic beacons and reinforcement learning, abbreviated as GBMCR, which addresses the highly dynamic, sparse topology and multi-mobile destination node communication requirements of FANETs without relying on ground infrastructure / central nodes. Under the premise of controllable signaling overhead, it can enhance network situational awareness, improve routing cooperation and robustness, and support routing methods for multiple mobile destination nodes and arbitrary pairwise communication.

[0058] This invention employs a technical solution of "geographic beacon update + multi-dimensional link utility measurement + reinforcement learning joint decision-making + route hole penalty and recovery". It achieves global location prior awareness by sparsely triggering geographic beacons in the network, characterizes real-time link quality through multi-dimensional link utility measurement, and adaptively fuses link utility with historical experience obtained through Q-learning for decision-making. Simultaneously, it performs broadcast penalty and recovery forwarding when route holes occur, transforming route decision-making from "relying solely on local greedy / single-index learning" to "collaborative intelligent routing that considers global priors, real-time links, and historical experience". This achieves the technical effects of reducing path failure probability, alleviating route holes, reducing detour redundancy, and improving delivery reliability and real-time performance. It has the advantages and significance of being adaptable to highly dynamic FANET scenarios, having controllable control overhead, strong robustness, and scalability to support multiple mobile destination nodes.

[0059] like Figure 1 As shown, the specific steps are as follows:

[0060] Step 1: Setup A flight ad hoc network composed of drone nodes, where each drone node obtains its own position and speed information, periodically sends messages to its neighbors, and each neighbor establishes its own neighbor table;

[0061] In a flight ad hoc network, each node can act as a source node, relay node, or destination node; node position and velocity change over time. The communication link uses a wireless channel model, and the communication radius that a node can reach in one hop is [missing information]. .

[0062] drone nodes The system obtains its own position and speed information through the positioning module, periodically packages its own motion state information into HELLO messages, and broadcasts them; neighboring nodes Upon receiving a "HELLO," a local neighbor table is created and maintained, recording information such as the identifier, location, speed, timestamp, and number of neighbors for each neighbor node; if a neighbor node... If the information is not refreshed within the preset time limit, then the neighboring nodes... Delete the neighbor node from the neighbor table. .

[0063] Step 2: Each drone node packages its own information and the information from its local neighbor table into its own initial local global information table; and triggers the geographic beacon at a set fixed time period;

[0064] Before forwarding a data packet, each node obtains the location information of the destination node based on its local global information table;

[0065] Each node triggers a geographic beacon at a fixed set time interval; the beacon initially carries the geographic information set of each node: node identifier, location, speed, timestamp; as well as the geographic information set of each neighbor node in each node's local neighbor table;

[0066] In the network, geographic beacon (GBeacon) updates are sparsely triggered with a relatively long period. During the multi-hop propagation process, the relay node adds its own geographic information group and forwards it, thereby enabling the receiving node to update its local global information table and form a global location prior to the spatial distribution characteristics of the network. At the same time, the beacon propagation process is suppressed and constrained to control overhead.

[0067] The innovative process for updating geographic beacons is as follows:

[0068] 1) The use of sparsely triggered geographic beacon update cycles significantly reduces control signaling overhead compared to frequent network-wide broadcasts;

[0069] 2) During the multi-hop forwarding process, the beacon continuously adds geographic information groups of the nodes it passes through, forming path enhancement, which enables the receiver to obtain richer global location priors;

[0070] 3) Only allow neighbors that have not appeared in the beacon to continue forwarding, and cache the processed beacon identifiers, adopting a cached flood suppression strategy to avoid repeated propagation;

[0071] 4) Set a maximum hop count / propagation range limit, and combine it with the update cycle to constrain the beacon propagation cost, so as to achieve a controllable trade-off between "global awareness capability and signaling overhead".

[0072] Step 3: The source node generates data packets and retrieves the destination node from the global information table. The positions of all neighboring nodes of the source node.

[0073] When the source node generates a data packet, it determines the destination node identifier of the data packet and obtains the latest estimated location of the destination node from the global information table to provide a basis for subsequent forwarding based on geographical progress; for data packets with real-time requirements, a deadline or time limit parameter is introduced.

[0074] Step 4: Based on the request speed, determine whether the actual speed of each neighbor node is not lower than the request speed. If so, keep the neighbor node in the candidate neighbor set and proceed to Step 5; otherwise, proceed to Step 8.

[0075] The current forwarding node calculates the requested speed (the minimum average transmission speed required) based on the remaining deadline of the data packet, and predicts the arrival time and location of the neighbor based on the movement status of the neighbor, and calculates the actual transmission speed after being forwarded by the neighbor; only neighbor nodes whose actual speed is not lower than the requested speed are retained in the candidate neighbor set.

[0076] Request speed is the ratio of the remaining distance from the current node to the destination node to the remaining deadline for the data packet; the initial value is the distance from the source node to the destination node. The speed is the ratio of the distance traveled to the data packet transmission time. This speed is dynamically updated as the forwarding process progresses.

[0077] Actual speed refers to the expected effective transmission speed that a data packet can achieve by forwarding through a neighbor after considering single-hop delay and the neighbor's motion state, if a neighbor is selected as the next hop; the actual calculation is the ratio of the distance between the neighbor node and the destination node to the time it takes for the data packet to be transmitted from the neighbor node to the destination node.

[0078] Step 5: For each neighbor in the candidate neighbor set, calculate the link utility value based on the local neighbor table and link estimation information, and calculate the decision score of each neighbor by integrating historical experience Q value, and select the next hop according to the maximum score;

[0079] The link utility takes into account at least the following factors: destination distance progress, link stability, single-hop forwarding delay, and neighbor density. The combined utility is obtained by normalizing and weighting each feature to characterize the "real-time link quality".

[0080] The current node establishes an action space for the candidate neighbor set and maintains a Q-value table with "current node identifier + destination node identifier" as the state. When selecting the next hop, the "historical experience Q-value" and "real-time link utility" are fused and scored: a decision score is calculated for each candidate neighbor, and the optimal next hop is selected based on the maximum score. The fusion weight coefficient is adaptively adjusted with the network dynamics, so that when the network is stable, more long-term experience with Q-values ​​is used, and when the network changes drastically, more real-time link utility is relied on for exploration and correction.

[0081] Current node Passing through neighboring nodes to the destination node Link utility value:

[0082]

[0083] in For reference delay, These are the weighting coefficients; For nodes Jump to neighbor node The time delay; This is the actual link stability index; For nodes to the destination node The distance to the one-hop neighbor node to the destination node The difference in distance; The communication radius that a node can reach in one hop. For nodes The current number of neighbors.

[0084] Candidate Neighbors The overall decision score is:

[0085]

[0086] in Based on historical experience values, As the amplification factor, These are adaptive weighting coefficients.

[0087] The innovative process for joint action decision-making is as follows:

[0088] 1) Construct a multi-dimensional link utility metric to visualize the reliability and efficiency of candidate links in real time;

[0089] 2) Construct the Q learning state as (current node ID, destination node ID) to accumulate historical forwarding experience without needing to maintain the global topology;

[0090] 3) The Q-value and link utility are integrated and scored, and an adaptive weight coefficient is introduced to enable the decision to adaptively switch between "utilizing historical experience" and "exploring the optimal link in real time" according to the network dynamics.

[0091] Step 6: The current node forwards the data packet to the selected next hop and waits for an ACK response; it checks whether an ACK has been received within the maximum waiting time. If so, it returns to Step 4, sets the next hop node as the current node, and repeats the process of selecting candidate neighbors and selecting the next hop node until the data packet is successfully delivered to the destination node. The process may terminate upon reaching the maximum number of jumps or exceeding the timeout limit; otherwise, proceed to step 7.

[0092] The ACK message contains the actual transmission parameters for this hop: the actual sending time. Reception time Single-hop delay The destination node in the current state is The maximum Q value, etc.

[0093] The next-hop node returns the actual transmission parameters of this hop along with the ACK, so that the current node can calculate the immediate reward accordingly; if the ACK is not received within the maximum waiting time, the forwarding is deemed to have failed and a negative reward is given; the current node updates the Q value online based on the reward and the expected Q value of the next state.

[0094] The innovation process for collaborative feedback rewards is as follows:

[0095] 1) The next-hop node returns the actual transmission parameters of this hop along with the ACK, realizing local information exchange and collaborative feedback;

[0096] 2) The reward function considers multiple factors such as single-hop latency and link stability, and guides the learning of stable, low-latency forwarding strategies through positive / negative incentives;

[0097] 3) ACK timeout triggers failure judgment and penalty, prompting the Q value to reflect link failure and topology changes in a timely manner.

[0098] Step 7: If forwarding the data packet to the next hop node fails, a negative reward is given, and the historical experience Q value is updated. Return to step 5, and use the updated historical experience Q value to reselect the neighbor node with the highest score as the next hop.

[0099] The updated Q value is the Q value table of the current node, triggered by an ACK message or a timeout in which no ACK message is received. For example, if node A forwards data packet p to node B, and node A does not receive an ACK message for data packet p from node B within the timeout period, node A updates its own Q value table by updating the Q value of the entry in the Q value table that indicates the destination node d of data packet p and the next node is B.

[0100] The learning rate and discount factor of Q-learning are adaptively adjusted based on the link quality and the degree of change in the neighbor set; at the same time, the HELLO sending interval is adaptively adjusted according to network stability: the interval is extended when the network is stable to reduce overhead, and the interval is shortened when the topology changes frequently to ensure timely state updates.

[0101] Step 8: If the candidate neighbor set is empty, a routing hole occurs. The following mechanism is used to avoid this:

[0102] The innovative process for handling routing holes is as follows:

[0103] 1) Current node If the candidate neighbor node is empty or there is no neighbor that can provide positive forwarding progress, the current node will be... The node is identified as a hole and a HOLE message is broadcast to spread the hole experience to other nodes. A one-hop neighbor node prompts surrounding nodes to access the node with that node. The reward for the relevant link "as the next hop" is set to the minimum and the corresponding Q value is updated to avoid selecting the hole node again in a short period of time, thereby reducing the probability of repeatedly falling into the hole.

[0104] 2) The recovery scoring criterion of "node degree + destination distance progress" is applied to the current data packet, starting from the current node. Among the neighboring nodes, the one with higher priority and closer to the destination node is selected. The neighboring node is used as the new next-hop node to increase the probability of successfully navigating the hole while also taking into account routing efficiency.

[0105] The method described in this invention, based on the corresponding routing device / system, can be composed of the following modules:

[0106] 1) Location and Status Acquisition Module: Acquires the location, speed, timestamp, etc. of this node;

[0107] 2) Neighbor Discovery and Neighbor Table Maintenance Module: Sends / receives HELLO and maintains the neighbor table;

[0108] 3) Geographic beacon generation and update module: triggering GBeacon, adding information groups, flooding suppression, hop count limit, and updating the global information table;

[0109] 4) Real-time business constraint processing module: Maintains data packet deadlines and calculates request rates;

[0110] 5) Candidate Neighbor Filtering Module: Filters the candidate set based on request speed and actual speed;

[0111] 6) Link Utility Calculation Module: Calculates multi-dimensional link utility and outputs utility values;

[0112] 7) Q-learning and parameter adaptation module: Maintains the Q-table and calculates the adaptive learning rate and discount factor;

[0113] 8) Joint Decision Module: Combines Q-value and link utility to calculate the decision score and select the next hop;

[0114] 9) Collaboration Feedback and Reward Calculation Module: Processes ACK feedback, calculates rewards, and triggers Q-value updates;

[0115] 10) Routing hole detection and recovery module: detects holes, broadcasts HOLE penalties, performs recovery forwarding scoring and routing.

[0116] Example:

[0117] (a) Implementation environment and key parameters

[0118] 1. Network environment

[0119] In this embodiment, FANET is composed of The system consists of a fleet of UAV nodes, all with equal status, capable of serving as source, relay, or destination nodes. Each node possesses wireless transceiver and positioning capabilities (such as GPS / INS fusion), and its position and velocity change over time. The communication link employs a wireless channel model, allowing a node to reach a communication radius of [missing information - likely a radius or value]. .

[0120] 2. Control Messages and Data Structures

[0121] To integrate "global situational awareness—local intelligent decision-making—collaborative feedback updates," this invention adopts the following control messages and data structures:

[0122] (1) HELLO message: carries node ID and timestamp ,Location ,speed and number of neighbors Fields such as these are used for neighbor discovery and neighbor table maintenance;

[0123] (2) HACK message: Acknowledgment / acknowledgment of HELLO, used to count link quality and packet loss;

[0124] (3) GBeacon message: carries beacon ID and hop count Geographic Information Group List ( Supports path enhancement (adding information groups along the route);

[0125] (4) DATA data packet: carries source ID, destination ID, deadline / remaining deadline time Business fields, etc.

[0126] (5) ACK message: Confirmation of the DATA forwarding result, carrying the actual transmission parameters of this hop (such as sending time, receiving time, single-hop delay). Maximum Q value (etc.), used to calculate the reward for the previous jump and update the Q value;

[0127] (6) HOLE message: Route hole notification, carrying the hole node ID and associated destination node ID, used to spread hole experience and trigger penalty update.

[0128] The corresponding data structures should include at least:

[0129] NeighborTable: Records neighbor IDs and locations. ,speed timestamp Number / density of neighbors Link statistics (HELLO / HACK count, ACK statistics, etc.);

[0130] GlobalInfoTable: Records the node geographic information groups learned by GBeacon. ;

[0131] QTable: by state As an index, by action For indexing, store .

[0132] 3. Simulation Platform and Parameter Examples

[0133] In one simulation embodiment, the performance of the method of the present invention can be verified on the FANET simulation platform built in Python: number of nodes N=30, three-dimensional region X: 0–800m, Y: 0–800m, Z: 0–300m; the movement model is Gaussian-Markov; the propagation loss model is free space path loss; the simulation duration is 50s; the node speed is 10–40m / s; the service type is CBR; the packet size is 1024 bytes; the data generation rate is 40960bps; multiple mobile destination nodes are set (e.g., 5 sinks); the comparison protocol can be QGeo or other similar reinforcement learning geographic routing, and other parameters should be kept consistent except for the routing mechanism to ensure fairness.

[0134] (II) Method Implementation Process

[0135] like Figure 2 As shown, the specific steps are as follows:

[0136] Step 1: Node Localization and Adaptive Neighbor Discovery

[0137] Each UAV node uses an onboard positioning module (such as GPS / inertial navigation) to obtain its own motion state information, including its three-dimensional position. and velocity vector To build a local network view, each node periodically broadcasts a HELLO message containing its node identifier. timestamp Current location ,speed and the current number of neighbors Neighbor nodes Upon receiving "HELLO", update the local neighbor table to record the node. The above information, and record the receiving time. This is used for subsequent link quality estimation.

[0138] HELLO messages are fundamental to neighbor discovery, but a fixed interval is ill-suited to the drastic topology changes in FANETs: an excessively long interval leads to outdated information and routing errors, while an excessively short interval causes a surge in signaling overhead. Therefore, this invention introduces an adaptive HELLO interval adjustment mechanism, dynamically adjusting the transmission period based on network stability. Specifically, define nodes. rate of change of neighbor set (As shown in equation (14)), then Determined by the following formula:

[0139] in As the reference time constant, This is the adaptive discount factor (calculated in step 8), which reflects the stability of the neighbor set. When the network topology is stable, Larger Extending the timeframe reduces control overhead; however, when the topology changes drastically... Decrease This shortens the timeframe and ensures the real-time nature of neighbor information. This adaptive mechanism achieves a dynamic balance between overhead and freshness, significantly improving the protocol's scalability.

[0140] Step 2: Global Collaborative Geographic Beacon Update

[0141] To obtain global location priors with low overhead, this invention designs a geographic beacon (GBeacon) mechanism, the process of which is as follows: Figure 3 As shown. Each node has a longer period. (generally This triggers a GBeacon update. The initial beacon only contains the geographic information group of the source node. The beacon also includes the geographic information groups of each neighboring node in the source node's local neighbor table. The beacon propagates through the network via multi-hop broadcast, with each relay node appending its own geographic information group to the beacon payload before forwarding, creating "path enhancement." Thus, when the beacon reaches the remote node, it carries the latest status of multiple nodes along the way, allowing the receiving node to update its local GlobalInfoTable accordingly.

[0142] To suppress flooding storms, a cached flooding suppression strategy is adopted: each node maintains a cache of the most recently processed beacon IDs within a time window, and forwards the beacon only if the beacon ID is not in the cache and the current node does not appear in the beacon path, and sets a maximum hop count (TTL) to limit the propagation range.

[0143] Relying solely on HELLO only allows for local neighbor awareness, while global location information is crucial for routing to any destination node. Geographic beacons, through sparse triggering and path enhancement, distribute the global topology across the entire network with controllable overhead, enabling each node to obtain the approximate locations of all nodes, thus supporting arbitrary pairwise communication. Cache suppression and TTL prevent redundant broadcasts and infinite diffusion, further reducing signaling load. Compared to periodic network-wide flooding, this invention significantly reduces control overhead while ensuring information freshness, achieving an effective trade-off between global situational awareness and communication overhead.

[0144] Step 3: Business Triggering and Prior Acquisition of Destination Node

[0145] When the source node has data packets to send, it first determines the destination node identifier. If the data packet has real-time requirements, it carries a deadline, and the remaining deadline is maintained at each forwarding node. The node searches for the target node in the global information table. The latest records, including location and speed This serves as the basis for subsequent geographic forwarding.

[0146] By obtaining the destination node's location information in advance, each hop can be made based on geographical progress, avoiding blind forwarding. Simultaneously, the introduction of a deadline parameter provides a basis for subsequent speed constraint filtering, thereby ensuring the quality of service for real-time operations and enabling the protocol to meet diverse business needs.

[0147] Step 4: Candidate Neighbor Filtering

[0148] To ensure that data packets arrive before the deadline, this invention introduces a transmission speed constraint, eliminates neighbors that cannot meet the timeliness requirements, and constructs a candidate neighbor set. The specific process is as follows:

[0149] Request speed calculation: for the current node , defined from to the destination node Minimum average speed required :

[0150]

[0151] in For nodes to the destination node Euclidean distance, This represents the remaining deadline for the data packet.

[0152] Actual speed prediction: for each neighbor If the prediction is made, As the next hop, the data packet from through The actual effective speed that can be obtained by forwarding First, estimate from arrive Single-hop delay Then predict the arrival of data packets. The moment According to the neighbors Records (locations) in the neighbor table ,speed ,direction ), predictable time Location:

[0153]

[0154]

[0155] Calculate the predicted location to the destination node distance The actual speed is then defined as:

[0156]

[0157] Collection construction: only when At that time, the neighbor Only then were they included in the candidate neighbor set. .

[0158] This step eliminates links that might introduce excessive latency at the physical layer, ensuring that each hop meets end-to-end latency constraints and preventing subsequent routing decisions from being wasted on invalid links. Simultaneously, by predicting the movement status of neighbors, it improves the accuracy of speed estimation, adapting to highly dynamic environments. Its direct effect is to increase the success rate of real-time services and reduce network end-to-end latency.

[0159] Step 5: Calculation of Multidimensional Link Utility Metric

[0160] To comprehensively evaluate the real-time link quality of candidate neighbors, this invention constructs a multi-dimensional link utility metric. This metric integrates multiple key features, all of which are normalized to... Interval:

[0161] Single-hop forwarding latency It consists of MAC delay, queuing delay, and transmission delay, and the specific calculation formula is as follows:

[0162]

[0163] in Estimated from ACK messages: ; , For the current node The number of packets preceding the main data packet. Processing time for each group; , For bag size, For transmission rate. To smooth out fluctuations, multiple measurements are averaged.

[0164] Link stability index : You can choose between link remaining time or relative speed stability. Link remaining time of neighboring nodes. The calculation formula is as follows:

[0165]

[0166] in Indicates the communication range of the node. Representing nodes respectively with neighboring nodes The location coordinates. Similarly, Then the corresponding node with neighboring nodes speed value, It should be a very small number to prevent errors when the speeds are equal. The larger the value, the longer the neighboring node is within the communication range; conversely, the smaller the value, the shorter the time.

[0167] To avoid drastic data fluctuations, the following smoothing method is used:

[0168]

[0169] in Indicates the remaining time of the link at the current moment. Take the nearest The mean at each time point.

[0170] Link stability The calculation formula is as follows:

[0171]

[0172] in and Representing nodes respectively With neighboring nodes The speed. This is the maximum speed limit value. When... The larger the value, the stronger the node. The closer the relative speeds of the links are to their neighboring nodes, the higher the link stability; conversely, the further apart they are, the less stable the link becomes. A smoothing method similar to that in formula (8) is also used.

[0173] Link Mobility Index It is composed of both the remaining link time and link stability, and the specific expression is as follows:

[0174]

[0175] in When the remaining link time is long and the stability is high, The larger it is, the smaller it is, and vice versa.

[0176] Distance progress towards destination node : Reflecting the choice The distance shortened in the target direction. Normalized and divided by the communication radius. ,Right now .

[0177] Neighbor density The more neighbors a node has, the better its connectivity and the greater the likelihood of it navigating through a hole. Normalization divided by the total number of network nodes. ,Right now .

[0178] Overall utility is defined as:

[0179]

[0180] in For reference delay, These are weighting coefficients with a sum of 1, and can be adjusted according to business needs.

[0181] A single metric can easily lead to routing holes or the selection of unstable links. This invention uses multi-dimensional fusion, simultaneously considering latency, stability, progress, and density, to make routing more comprehensive and robust. Link utility is calculated in real time, reflecting the current network state and providing immediate awareness for subsequent fusion with Q-values, thereby improving the accuracy of decision-making.

[0182] Step 6: Joint Action Decision Based on Reinforcement Learning

[0183] This step is the core of the routing decision, such as... Figure 4 As shown. Node Maintain a Q table, categorized by state. For action, movement Store as columns This value reflects historical experience from via arrive The expected cumulative reward. When forwarding data packets, the node... First, from the candidate neighbor set The system selects the optimal next hop based on the fusion score.

[0184] Decision score calculation: For each candidate neighbor Calculate the overall decision score:

[0185]

[0186] in Based on historical experience values, The real-time link utility calculated in step 5, This is the amplification factor (usually taken as 1~10, used to balance dimensions). These are adaptive weighting coefficients.

[0187] Adaptive weight coefficients : Based on the network's dynamic adaptive adjustment, the following definition is made:

[0188]

[0189] in The rate of change of the neighbor set (see Equation (14)) reflects the degree of topological fluctuation; and Preset boundaries (e.g., 0.2 and 0.8). To adjust the slope. The computation is based on nodes The set of neighbors within two consecutive HELLO cycles:

[0190]

[0191] When the topology changes drastically ( When (large), As the number of cases increases, decision-making becomes more reliant on real-time utility. Encourage the exploration of new paths; when the topology is stable ( Hour, The number of Q values ​​decreases, and decision-making relies more on historical Q values, utilizing past experience.

[0192] Next hop selection: Choose the neighbor with the highest decision score as the next hop.

[0193]

[0194] Relying solely on Q-values ​​(in reinforcement learning) can lead to slow convergence and insensitivity to instantaneous changes; relying solely on heuristic utility can result in getting trapped in local optima. This invention achieves a dynamic balance between "utilizing historical experience" and "exploring real-time optima" through adaptive weighted fusion, preserving the long-term optimization capabilities of reinforcement learning while enhancing its rapid response to dynamic scenarios. Adaptive weight coefficients It automatically adjusts according to network conditions, avoiding the performance degradation of fixed weights when the environment changes, and significantly improving the robustness and efficiency of routing decisions. Step 7: Collaborative Feedback, Reward Calculation, and Online Updates

[0195] When node To the selected next hop After sending the data packet, the collaborative feedback and learning phase begins:

[0196] ACK feedback: Node After successfully receiving a data packet, an ACK packet is generated, which contains the actual transmission parameters for this hop: the actual sending time. Reception time Single-hop delay The destination node in the current state is Maximum Q value, etc. Nodes unicast ACK back to the node .

[0197] Instant reward calculation: Node Upon receiving the ACK, calculate the immediate reward for this action based on its parameters. The reward function is designed as follows:

[0198] If the data packet reaches the destination node, the maximum positive reward is given. .

[0199] If the data packet is successfully forwarded to the next hop, a reward will be given. Calculated by the following formula:

[0200]

[0201] in , The weights are 1. This represents the actual single-hop delay. The actual link stability index (by nodes) (Based on historical statistics). This reward encourages low-latency, high-stability forwarding.

[0202] If node Maximum waiting time If no ACK is received, the forwarding is considered a failure, and a negative reward will be given. (e.g., -10) to penalize invalid or lost forwards.

[0203] Q-value update: Based on the Bellman equation learned from Q, the node... Update the corresponding entry in the Q table:

[0204]

[0205] in , , , For learning rate, This is the discount factor. Represents a node The expected maximum future Q value at a given point can be obtained from the node. Piggybacking in ACK (i.e., nodes) Set its current destination node as Maximum Q value tells This enables collaborative updates.

[0206] This step implements distributed cooperative reinforcement learning. By using ACK feedback to provide real transmission parameters, the reward function accurately reflects the forwarding effect, guiding the Q-value to converge towards the optimal policy. The setting of positive and negative rewards enables nodes to quickly avoid failed links. The online update mechanism allows the Q-value to continuously adapt to network changes without offline training. Cooperative feedback (piggybacking on the maximum Q-value) reduces additional signaling and improves learning efficiency.

[0207] Step 8: Adaptive learning parameters and signaling frequency adjustment

[0208] To ensure the learning process matches network dynamics, this invention adaptively adjusts the key parameters of Q-learning and the HELLO interval:

[0209] Adaptive learning rate The learning rate controls how much new information updates the Q-value. In FANET, a larger learning rate should be used when link quality fluctuates drastically to quickly adapt to changes. (Define link quality.) Statistics based on HELLO and HACK:

[0210]

[0211] in and These represent the number of HELLO messages sent by node i within a certain period and the number of HACK messages replied by node j to node i, respectively; similarly, and These represent the number of messages on the reverse link, respectively. This ratio reflects the success rate of bidirectional communication, and its value ranges from [value range missing]. Then the learning rate Defined as:

[0212]

[0213] When the link quality is poor ( Hour, Larger, faster learning of new states; when the link quality is good, Small, preserving historical experience.

[0214] Adaptive discount factor The discount factor reflects the importance placed on future returns. If the neighbor set changes frequently and the uncertainty of future states is high, it should be reduced. . Based on neighbor change rate (Equation (14)) Calculate:

[0215]

[0216] Right now Positively correlated with network stability, when the topology is stable Approaching 1, emphasizing long-term returns; during topological upheavals They are reducing their focus and paying more attention to immediate gains.

[0217] HELLO interval adaptation: As described in step 1, the HELLO interval... Similarly, To adjust, because When topological changes are drastic ( When (large), Decrease This shortens the transmission frequency, thereby achieving a match between the HELLO transmission frequency and network dynamism.

[0218] Fixed parameters are difficult to optimize in dynamic environments. This invention dynamically adjusts learning parameters and signaling frequency by sensing link quality and topology changes in real time, so that the protocol always works in the best configuration to adapt to the current state, thereby improving the learning convergence speed and decision accuracy.

[0219] Step 9: Route hole detection, penalty and recovery forwarding

[0220] like Figure 5 As shown, when node During routing decisions, if the candidate neighbor set The range is empty (i.e., there are no neighbors that satisfy the velocity constraint), or all neighbors cannot provide positive distance progress (e.g., all neighbors are less than the required range). If the distance to the destination node is further, it is determined that the route is trapped in a routing hole. In this case, the following steps are taken:

[0221] HOLE broadcast penalty: node Broadcast a HOLE message to all one-hop neighbors, containing its own ID and the destination node. Neighbor nodes After receiving HOLE, search for it in the local Q table. The state and the next jump are Entries (i.e.) Set its corresponding reward value to the minimum. And immediately update the Q value. In this way, the neighboring node's future forwarding destination is... When dealing with data packets, they tend to avoid selecting... This serves as the next jump, thus reducing the probability of falling into the void again.

[0222] Resumption of forwarding criteria: For nodes For data packets that need to be forwarded, they must immediately bypass the hole. This invention employs a recovery scoring criterion to select the best next hop from all neighbors. The scoring formula comprehensively considers node degree and distance progress:

[0223]

[0224] in For the current hole node, For neighboring nodes, For the destination node, For the neighbors degree of nodes, for arrive distance, The score represents the communication radius. This score encourages selecting nodes with a large number of neighbors (higher probability of bypassing the hole) and closer to the destination node (maintaining routing efficiency). The neighbor with the highest score is selected as the next hop.

[0225] Loop execution: After resuming forwarding, the new node continues to execute steps 4-9 until the data packet reaches the destination node or times out.

[0226] Routing holes are an inherent problem in geographic routing. This invention uses HOLE broadcasting to rapidly propagate hole experience to surrounding nodes, forming a collaborative obstacle avoidance mechanism to prevent subsequent data packets from repeatedly getting trapped. A recovery scoring criterion provides an effective escape path when a hole occurs, balancing success rate and efficiency. Compared to traditional hole handling (such as boundary forwarding), this invention is more flexible and does not require maintaining a network plan, adapting to highly dynamic environments. This mechanism significantly enhances the network's robustness in sparse scenarios.

[0227] Step 10: Forward in a loop until it arrives or terminates.

[0228] For each hop, steps 4 through 9 are repeated until the data packet is successfully delivered to the destination node, or is discarded due to exceeding the hop limit or the remaining deadline. During the forwarding process, each node makes the above decisions independently, without the need for central control, thus achieving fully distributed intelligent routing.

[0229] (III) Presentation of Results

[0230] To verify the effectiveness of the "Cooperative Routing Method for Flight Ad hoc Networks Based on Geographic Beacons and Reinforcement Learning" described in this invention under highly dynamic conditions with multiple mobile destination nodes, this embodiment compares the method of this invention with the contrast routing protocol (QGeo) and the version that removes joint action decision (GBMCR-NJ) in a simulation environment. While ensuring consistency between physical layer and service load parameters, only the routing mechanism is changed to ensure the fairness and interpretability of the comparison results.

[0231] 1) Evaluation indicators and diagrams

[0232] The main evaluation metrics in this embodiment include: Packet Delivery Success Rate (PDR), network throughput, and average end-to-end transmission delay. PDR measures the probability that a packet will successfully reach its destination node to reflect network reliability; network throughput measures the amount of data successfully transmitted per unit time to reflect transmission efficiency; and average end-to-end transmission delay characterizes the average transmission time of a packet from the source node to the destination node to reflect latency characteristics.

[0233] Figure 6 The PDR comparison curves under different node velocity conditions are shown; Figure 7 The curves showing the network throughput under different node speed conditions are illustrated. Figure 8 The curves showing the average end-to-end transmission delay under different node speed conditions are illustrated. Figure 9 The performance comparison of the geographic beacon mechanism in terms of geographic accuracy, information freshness, network overhead and coverage is shown (including ablation comparison).

[0234] 2) Figure 6 (PDR) Results Description and Analysis

[0235] like Figure 6 As shown, the overall PDR decreases with increasing node movement speed. This is due to the reduced effective forwarding opportunities caused by increased link discontinuity and frequent changes in the neighbor set. However, under the same speed conditions, the PDR curve of the GBMCR of this invention is generally higher than that of the comparative protocol, and its advantage is more obvious in the medium and high speed range (e.g., ≥30m / s).

[0236] The reason for this is that: before forwarding, the present invention filters out neighbors that do not meet the transmission speed / time limit constraints through "candidate neighbor filtering," reducing invalid attempts; at the same time, it adopts a joint action decision of "multi-dimensional link utility + Q-value experience," which can prioritize the selection of a more reliable next hop when link quality fluctuates rapidly, thereby reducing the probability of packet loss; when routing holes occur, it guides users out of the hole area through HOLE notification and recovery criteria, reducing forwarding failures caused by local minima. This demonstrates the technical effectiveness of the present invention in improving network reliability under highly dynamic topologies.

[0237] 3) Figure 7 (Throughput) Result Description and Analysis

[0238] like Figure 7 As shown, as node speed increases, the throughput of the comparison protocol decreases more significantly, while the GBMCR of this invention maintains a higher effective throughput level, indicating that it can transmit more effective data under the same network load.

[0239] The results show that the present invention improves the forwarding success rate and reduces detour redundancy through joint decision-making, so that more network resources can be used for effective data transmission; at the same time, the adaptive learning parameters enable the policy to be updated in a timely manner with topology changes, avoiding policy lag caused by fixed parameters, thereby maintaining high transmission efficiency in dynamic environments.

[0240] 4) Figure 8 (End-to-end delay) Result Description and Analysis

[0241] like Figure 8 As shown, under the same speed conditions, the average end-to-end transmission delay of the GBMCR of the present invention is lower than that of the comparative protocol, indicating that the present invention can complete data transmission in a shorter time.

[0242] The reason is that this invention comprehensively considers factors such as single-hop latency, link stability, forwarding progress and neighbor density when selecting forwarding paths, avoiding the selection of paths that "seem reachable in the short term but are unstable in the link"; at the same time, it reduces retries and detours through time-limited constraint filtering and hole recovery mechanisms, thereby reducing end-to-end latency and improving the reachability of real-time services.

[0243] 5) An ablation comparison of joint action decision-making mechanisms

[0244] To verify the contribution of the joint action decision-making mechanism, an ablation experiment can be conducted, comparing the version of GBMCR-NJ without the joint action decision-making mechanism with the complete GBMCR. Figures 6-8 It is evident that the complete GBMCR outperforms GBMCR-NJ in terms of PDR, throughput, and latency, indicating that the "fusion decision based on multi-dimensional link utility and Q-value experience" can effectively improve the adaptability and robustness to dynamic topologies, thereby improving overall routing performance.

[0245] 6) Explanation of the ablation comparison of the geographic beacon mechanism

[0246] like Figure 9 As shown, comparing the geographic beacon mechanism of this invention with versions that remove path enhancement mechanisms (such as GBeacon-NPE) and versions that remove cached flood suppression mechanisms (such as GBeacon-NCFS), it can be observed that this invention maintains a superior level in terms of geographic accuracy and information freshness, while achieving a more reasonable trade-off between network overhead and coverage. In particular, the cached flood suppression mechanism is more effective in reducing duplicate propagation and improving information maintenance; the path enhancement mechanism enables the beacon to carry more complete path information, thereby enhancing the global information maintenance capability.

[0247] The results demonstrate that this invention, through a combination of "sparse triggering + path enhancement + cache suppression + hop count limitation," achieves more effective global situation updates while keeping controllable overhead, providing more reliable location priors for subsequent routing decisions.

[0248] 7) Overall Conclusion

[0249] comprehensive Figures 6 to 9 It is understood that the GBMCR of this invention, under the condition of supporting multiple mobile destination nodes, can simultaneously improve delivery reliability and transmission efficiency in highly dynamic topologies, and reduce latency and detour redundancy; at the same time, it enhances global situational awareness through the geographical beacon mechanism and enhances routing robustness through joint decision-making and hole processing mechanisms, thereby providing more stable and reliable routing guarantees for multi-UAV collaborative communication.

[0250] The above examples are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art should understand that any modifications, variations, or equivalent substitutions to the invention without departing from its spirit and scope should be included in the claims of the invention to make the objectives, technical solutions, and advantages of the invention clearer. The embodiments of the invention will be further described in detail below with reference to the accompanying drawings.

Claims

1. A cooperative routing method for ad hoc flight networks based on geographic beacons and reinforcement learning, characterized in that, The specific steps are as follows: Step 1: Setup A flight ad hoc network composed of drone nodes, where each drone node obtains its own position and speed information, periodically sends messages to its neighbors, and each neighbor establishes its own neighbor table; Step 2: The drone node packages its own information and the information in its local neighbor table into its own initial local global information table; and triggers the geographic beacon at a set fixed time period; The beacon initially carries a set of geographic information about the node: node identifier, location, speed, and timestamp; as well as the geographic information of each neighbor node in each node's local neighbor table; Step 3: The source node generates data packets and retrieves the destination node from the global information table. The positions of all neighboring nodes of the source node; Step 4: Based on the request speed, determine whether the actual speed of each neighbor node is not lower than the request speed. If so, keep the neighbor node in the candidate neighbor set and proceed to Step 5. Otherwise, proceed to step 8; Step 5: For each neighbor in the candidate neighbor set, calculate the link utility value, and combine the historical experience Q value to calculate the decision score of each neighbor. Select the neighbor node with the highest score as the next hop. Current node Passing through neighboring nodes to the destination node Link utility value: in For reference delay, These are the weighting coefficients; For nodes Jump to neighbor node The time delay; This is the actual link stability index; For nodes to the destination node The distance to the one-hop neighbor node to the destination node The difference in distance; The communication radius that a node can reach in one hop; For nodes The number of current neighbors; Candidate Neighbors The overall decision score is: in Based on historical experience values, As the amplification factor, These are adaptive weighting coefficients; Step 6: The current node forwards the data packet to the selected next hop and waits for an ACK response; it checks whether an ACK has been received within the maximum waiting time. If so, it returns to Step 4, sets the next hop node as the current node, and repeats the process of selecting candidate neighbors and choosing a next hop node until the data packet is successfully delivered to the destination node. The process may terminate upon reaching the maximum number of jumps or exceeding the timeout limit; otherwise, proceed to step 7. Step 7: If forwarding the data packet to the next hop node fails, a negative reward is given, and the historical experience Q value is updated. Return to step 5, and use the updated historical experience Q value to reselect the neighbor node with the highest score as the next hop. Step 8: If the candidate neighbor set is empty, a routing hole occurs. The following mechanism is used to avoid this: The innovative process for handling routing holes is as follows: 1) Current node If the candidate neighbor nodes are empty, move the current node to the next node. Set it as a hole node and broadcast a HOLE message to spread the hole experience to other nodes. A one-hop neighbor node prompts surrounding nodes to access "with that node". The relevant link reward for "the next hop" is set to minimum and the corresponding Q value is updated; 2) The recovery scoring criterion of "node degree + destination distance progress" is applied to the current data packet, starting from the current node. Among the neighboring nodes, the one with higher priority and closer to the destination node is selected. The neighboring node is used as the new next-hop node.

2. The cooperative routing method for ad hoc networks based on geographic beacons and reinforcement learning as described in claim 1, characterized in that, In a flight ad hoc network, each node can act as a source node, relay node, or destination node; the node's position and speed change over time. drone nodes Periodically package its own motion state information into a HELLO message and broadcast it; neighboring nodes After receiving HELLO, a local neighbor table is created and maintained, recording the identifier, location, speed, timestamp, and number of neighbors for each neighbor node.

3. The flight ad hoc network cooperative routing method based on geographic beacons and reinforcement learning as described in claim 1, characterized in that, In step 4, each node in the flight ad hoc network can act as a source node, relay node, or destination node; the node position and speed change over time; the requested speed is the ratio of the remaining distance from the current node to the destination node to the remaining deadline of the data packet. The initial value is: from the source node to the destination node. The ratio of the distance to the data packet transmission time; Actual speed refers to the ratio of the distance between the neighboring node and the destination node to the time it takes for the data packet to travel from the neighboring node to the destination node.

4. The flight ad hoc network cooperative routing method based on geographic beacons and reinforcement learning as described in claim 1, characterized in that, In step 6, the ACK message contains the actual transmission parameters for this hop: the actual sending time. Reception time Single-hop delay The destination node in the current state is The maximum Q value.

5. The flight ad hoc network cooperative routing method based on geographic beacons and reinforcement learning as described in claim 1, characterized in that, In the recovery scoring criteria of step 8, the current node Passing through neighboring nodes to the destination node Void restoration scoring formula: in, Neighboring nodes to the destination node The distance.

Images