A geographic position-based offline double Q-learning routing method
By combining Q-learning and double Q-learning algorithms, the Euclidean distance and link quality of neighboring nodes are comprehensively evaluated, solving the routing hole problem in highly dynamic UAV ad hoc networks and improving network performance and data transmission reliability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2023-05-08
- Publication Date
- 2026-07-21
AI Technical Summary
In highly dynamic drone ad hoc network environments, existing routing protocols struggle to effectively avoid routing holes, leading to decreased network performance and increased latency.
By combining the Q-learning algorithm and the double-Q learning concept in reinforcement learning, and comprehensively evaluating the Euclidean distance, link quality, and node degree of neighboring nodes through a reward mechanism, an offline double-Q learning routing algorithm is established to find the globally optimal route and avoid routing holes.
It improves the data transmission success rate of drone networks, reduces end-to-end latency, and improves the convergence of Q-value tables through double Q learning, thereby reducing network congestion and routing holes.
Smart Images
Figure CN116456416B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of wireless ad hoc network communication technology, specifically to a routing protocol based on geographic location information assistance and a double Q-learning (DQGR) algorithm, which is mainly used to solve the reliable routing problem in highly dynamic UAV ad hoc network environments. Background Technology
[0002] As a subset of mobile ad hoc networks, Flying Ad hoc Networks (FANETs) are networks built by drone swarms in a self-organizing manner. Unlike other subsets such as vehicular ad hoc networks, FANETs are characterized by highly dynamic network topology, high node mobility, and proneness to link interruptions. Drone services typically have high quality-of-service requirements; therefore, it is necessary to design targeted routing protocols to ensure the reliability and effectiveness of data transmission within the network.
[0003] Classic routing protocols have varying applicability in different scenarios. For example, the proactive protocol OLSR requires establishing routing information for all nodes in the network before data transmission and continuously updating the routing table as the topology changes. The overhead of establishing and maintaining routes is significant, making it suitable for large-scale, dense, static scenarios. The reactive protocol AODV only needs to establish and maintain routes when data needs to be sent, saving resources and making it suitable for ad hoc networks with some dynamic changes where only some nodes have communication needs. However, the initial data transmission initiates a route discovery process, causing high latency, and the route repair process in case of link failure also affects the real-time performance of network communication. In highly dynamic FANET scenarios, the applicability of all the above protocols is limited.
[0004] Greedy Perimeter Stateless Routing (GPSR) is a stateless protocol that uses location information to assist in routing decisions. Routes do not need to be pre-established; they are calculated in real-time using both greedy forwarding and perimeter forwarding modes, making it suitable for highly dynamic network environments. Drones can use GPS modules to achieve real-time positioning and obtain location information, making GPSR well-suited for FANET scenarios. However, GPSR has limitations. The greedy forwarding mode frequently forwards packets to routing holes, leading to a decline in overall network performance. The perimeter forwarding mode requires a large number of hops to bypass holes, significantly increasing packet loss and latency.
[0005] Reinforcement learning algorithms describe the problem of maximizing rewards by learning strategies during the interaction between an agent and its environment. They are typically used to find the optimal decisions for achieving a specific goal. To further enhance the applicability of GPSR in FANET and address the routing hole problem, this invention combines the Q-learning algorithm and the double-Q learning concept from reinforcement learning to propose a location-based offline double-Q learning routing algorithm. This algorithm uses a reward and penalty mechanism to find the globally optimal route in a highly dynamic network environment, thus avoiding routing holes. Summary of the Invention
[0006] Purpose of the invention:
[0007] The purpose of this invention is to provide a reliable routing method for highly dynamic unmanned aerial vehicle (UAV) network environments, maximizing data transmission success rate and minimizing end-to-end latency. The technical solution of this invention is based on the classic GPSR (Global Positioning Reference Registry) protocol, combining Q-learning algorithms and double-Q learning from reinforcement learning to avoid frequent routing hole effects. The proposed location-based offline double-Q learning routing algorithm uses metrics such as Euclidean distance, link quality, and node degree to comprehensively evaluate the reward values of neighboring nodes. It seeks the real-time globally optimal route through Q-learning exploration and addresses the overestimation problem of Q-values through double-Q learning, ultimately effectively improving network performance and service quality.
[0008] Technical solution:
[0009] The technical solution of the present invention specifically includes the following steps:
[0010] Step 1: Model the UAV Ad hoc network environment as a reinforcement learning problem. Reinforcement learning tasks are typically described using Markov decision processes. An agent exists in environment E, its state space is X, its set of actions is action space A, the state transition function is P, and the reward obtained by the agent during state transitions is R. Modeling the ad hoc network environment as a reinforcement learning problem requires mapping the Markov process quadruple to specific objects in the network environment. When the entire network is considered as the environment, only data packets are continuously forwarded between nodes. Therefore, the data packet being forwarded by a node can be considered as an agent, its current node being the agent's current state x, and the action 'a' taken by the agent representing the choice of the next hop. When the data packet is forwarded to the next hop, the agent's state transitions and it receives a reward, with a fixed state transition probability of 1. Since the reward obtained by the agent is unknown, the action space depends on the node's neighbors, and the network environment is constantly changing, the quadruple needs to be obtained through the interaction between the agent and the environment. Therefore, the reinforcement learning task of the network can be classified as model-free learning. Consider using the heterogeneous policy Q-learning algorithm to maximize the cumulative reward and find the optimal path.
[0011] Step 2, Establishment of each table and algorithm initialization: Each node needs to maintain two Q-value tables, one reward table, and two neighbor maximum Q-value tables. Since the operation and decision-making of each node in an Ad hoc network are distributed, a node can only know the reward value of its neighbor node's state transition, but cannot know the reward information of the entire network. Therefore, the reward table is set as shown in Table 1, which includes the reward value obtained by forwarding data packets destined for different nodes from the current node to neighbor nodes.
[0012] Table 1. Reward Table Structure
[0013]
[0014] The design of the node Q-value table is similar to that of the reward table. Each node only needs to contain the Q-value corresponding to the state transition of the agent pointing to different destination nodes.
[0015] The neighbor maximum Q-value table contains the maximum Q-value that a node can obtain by choosing any next action for each of its neighbors, as shown in Table 2.
[0016] Table 2. Structure of the Neighbor Maximum Q Value Table
[0017]
[0018] Before the algorithm starts, each node initializes its Q-value table, reward table, and neighbor maximum Q-value table to 0. After receiving the application layer data packet, it records the location information of the destination node and inserts the corresponding table entry for the destination node into each table. The subsequent algorithm process includes updating and calculating the corresponding table entries for all destination nodes. During this process, it is necessary to ensure that the location information of all destination nodes is available. This information will be used during the update process.
[0019] Step 3, Reward Calculation: Choosing an appropriate reward and penalty mechanism allows the agent to correctly accumulate and maximize rewards during exploration, learning the optimal path to the destination node. First, the route is destination-oriented; all actions that forward to the destination node in one hop will receive the maximum reward. Second, the reward setting for relay nodes comprehensively considers link quality, node degree, route holes, and Euclidean distance. Based on the classic GPSR greedy forwarding mode, forwarding to nodes closer to the destination node should yield a larger reward. Therefore, the reward value corresponding to the Euclidean distance metric is defined as:
[0020]
[0021] Where x i x n x d C represents the x-coordinates of the current node, neighboring nodes, and destination node, respectively. The y and z directions are similar. iThis represents the communication range of the current node. The location information of neighboring nodes, like in classic GPSR, relies on the exchange of Hello message packets.
[0022] The reward value corresponding to the node degree is defined as follows:
[0023] Where n is the number of neighbors of a node, K is the average degree of each node in the network, and N is the total number of nodes in the network.
[0024] The reward value corresponding to link quality is defined as the product of the node's success rate in sending to and receiving data from its neighbors:
[0025] l q =p t ×p r
[0026] The success rate of node transmission and reception also depends on the interaction of Hello packets. Nodes continuously count the number of Hello packets received from each neighbor within a period, obtain the reception success rate of each link and write it into the corresponding field in the packet header. At the same time, they read the field in the Hello packet showing the success rate of the neighbor receiving its own Hello packet, which is the transmission success rate.
[0027] The reward value for a relay node is defined as a weighted average of the above metrics:
[0028] R re =σ1l q +σ2k de +σ3d o
[0029] The coefficients σ1, σ2, and σ3 represent the weights of the three reward measures, and their sum is 1.
[0030] When a routing hole appears in the relay during an agent's state transition, a penalty should be applied on top of the original reward to prevent data packets from being forwarded to the holed node again. Ultimately, the reward value for the agent transitioning from state i to state j is as follows:
[0031]
[0032] Where R max R is the maximum reward value obtained by forwarding to the destination node. p The penalty value for forwarding to a hole node.
[0033] Step 4, Experience Acquisition: Agents are generated at a certain frequency in the network. To balance exploration and utilization, an exponentially decaying ε-greedy strategy is used to select actions for exploration. In the early stages of the algorithm, agents are allowed to fully explore the entire network. After the algorithm stabilizes, the maximum Q-value path is learned and reinforced. The probability of random exploration is:
[0034]
[0035] Where ε0 is the initial random probability, s e d represents the total number of explorations. r The decay rate is calculated per day. s Each exploration decays the ε value once. After exploration, record the current state, the selected action, and the next state. Calculate the reward value obtained from the state transition according to step three, and add the above information as an exploration experience to the experience pool.
[0036] Step 5, Q-value table update: Extract experience from the experience pool at a certain frequency, select a Q-value table to be updated, obtain the maximum Q-value of the next state from the corresponding neighbor maximum Q-value table, and update the table.
[0037] Step 6, Synchronize the neighbor's maximum Q-value table: This step is also performed during Hello packet interaction. Before sending the Hello packet, find the Q-value table Q. A The action corresponding to the maximum Q value in the next state will be Q. B The Q value corresponding to this action is added to the neighbor's maximum Q value array Q. nmaxarrA Find Q B The action corresponding to the maximum Q value in the next state will be Q. A Add the Q value corresponding to this action to Q. nmaxarrB In the code, two arrays are added to the body of the Hello package and then broadcast.
[0038] When a node receives a Hello packet, it retrieves the two maximum Q-value arrays of its neighbors from the packet body and updates its neighbor maximum Q-value table Q accordingly. nmaxA and Q nmaxB In the middle, the synchronization process is completed.
[0039] Beneficial effects:
[0040] Compared with the prior art, the present invention has the following advantages:
[0041] First, this invention uses the Q-learning algorithm to find the globally optimal route, improving the real-time performance and reliability of routing in highly dynamic UAV network scenarios. Simultaneously, it employs a dual-Q learning method to address the overestimation problem of the value function in traditional Q-learning, ensuring fast and effective convergence of the Q-value table.
[0042] Second, this invention comprehensively considers Euclidean distance, node degree, and link quality metrics. While ensuring that Q-learning converges to an effective path to the destination node, it selects neighbors with higher node degree to avoid routing holes and avoids sending data to neighbor nodes with poor link quality, thereby reducing network congestion. Attached Figure Description
[0043] Figure 1 This invention provides the framework for each module of a location-based offline dual-Q learning routing algorithm.
[0044] Figure 2 This is a flowchart of the experience acquisition and Q-value table training and update process in the routing algorithm of this invention;
[0045] Figure 3 This paper compares the packet delivery rate performance of the routing algorithm of this invention with that of GPSR at different node speeds.
[0046] Figure 4 This paper compares the end-to-end latency performance of the routing algorithm of this invention with that of GPSR at different node speeds. Detailed Implementation
[0047] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and examples.
[0048] like Figure 1 As shown, the specific steps to start the complete offline dual-Q learning routing algorithm mainly include:
[0049] Step 1: Initialize each table entry.
[0050] (1) Each node generates two Q-value tables Q A and Q B The vertical dimension represents the total number of nodes in the network, and the horizontal dimension represents the number of neighbors of the current node. The number of columns in the table expands or shrinks as Hello packets are interacted with and neighbor information is updated.
[0051] (2) Initialization and the maximum Q-value table of two neighbors Q nmaxA and Q nmaxB A reward table R, whose dimensions are the same as those of the Q-value table.
[0052] (3) Initialize the value of the above entries to 0.
[0053] Step 2: Calculate the reward value.
[0054] After receiving the Hello packet, the node updates the location information of its neighbors and performs the following steps for different destination nodes:
[0055] (1) Calculate the Euclidean distance from each neighbor to the destination node. This metric is a key factor in guiding the destination node and has a weight of 0.5.
[0056] (2) Calculate the node degree with a weight of 0.25.
[0057] (3) Calculate the link quality of the neighboring links, with a weight of 0.25.
[0058] (4) Calculate the weighted reward value of the relay node according to the relay node reward calculation formula, and then calculate the final reward for forwarding to each neighbor node. In order to learn a forwarding strategy to reach the destination, the maximum reward of the destination node is set to 100, and the hole penalty value is set to about -5 to -10 to achieve a better hole avoidance effect.
[0059] (5) Update the calculated reward value to the entry corresponding to the target node in the reward table.
[0060] Step 3: Update the Q-value table.
[0061] like Figure 2 As shown, the experience acquisition process for each node is as follows:
[0062] (1) Each node generates an intelligent agent at a fixed frequency.
[0063] (2) Calculate the exploration probability based on the current number of iterations of the network, where the decay rate is 0.9. The probability decays once every 1000 iterations. This allows the agent to fully explore the optimal routing path of the entire network in the early stage of the algorithm. After the algorithm stabilizes, it consolidates the learning of the path with the maximum Q value, thus ensuring the convergence of the algorithm.
[0064] (3) The agent selects an action according to the ε-greedy policy based on the above exploration probability.
[0065] (4) Query the reward table for transitioning from the current state to the next state.
[0066] (5) Add the current state, the next state, the action and the reward as an experience to the experience pool.
[0067] At the same time, the Q-value table is updated using the following procedure:
[0068] (1) Each node retrieves experience from the experience pool at a fixed frequency.
[0069] (2) Obtain the current state, next state, action and reward.
[0070] (3) Randomly select to update Q A Or Q B .
[0071] (4) Obtain the maximum Q value of the next state from the corresponding neighbor maximum Q value table.
[0072] (5) Update the Q-value table, where α is set to 0.4-0.5 to balance learned knowledge and new knowledge, and γ is set to 0.95 so that the reward value of the target node can be diffused throughout the network:
[0073] Q(s i ,a ij )←Q(si ,a ij )+α[r ij +γmax a′ Q(s j ,a′)-Q(s i ,a ij )]
[0074] (6) Q A and Q B The Q-values are averaged to update the decision Q-value table.
[0075] Step 4: Synchronize the maximum Q value of the neighboring units.
[0076] Before a node prepares to send a Hello packet, fill in the Hello packet body as follows:
[0077] (1) In Q A Query action a A1 ,a A2 …make Q A (s,a A1 ),Q A (s,a A2 ...is the maximum value among all the next state Q values corresponding to the destination nodes d1, d2... respectively.
[0078] (2) In Q B Query action a B1 ,a B2 …make Q B (s,a B1 ),Q B (s,a B2 ...is the maximum value among all the next state Q values corresponding to the destination nodes d1, d2... respectively.
[0079] (3) Add Q B (s,a A1 ),Q B (s,a A2 ...to the neighbor's maximum Q-value array Q nmaxarrA middle.
[0080] (4) Add Q A (s,a B1 ),Q A (s,a B2 ...to the neighbor's maximum Q-value array Q nmaxarrB middle.
[0081] (5) Add Q nmaxarrA and Q nmaxarrB Enter the Hello package body and broadcast the Hello package.
[0082] After receiving the Hello packet, the node updates its local neighbor maximum Q-value table as follows to complete the synchronization process of neighbor maximum Q-values:
[0083] (1) Remove Q from the package body nmaxarrA Update to Q nmaxA middle.
[0084] (2) Remove Q from the package body nmaxarrB Update to Q nmaxB middle.
[0085] Step 5, routing decision.
[0086] (1) When the algorithm just starts, the Q-value table has not yet converged. The classic GPSR is used to complete the routing and forwarding to ensure network performance, while waiting for beacon interaction and Q-value table updates.
[0087] (2) After the algorithm starts for a certain period of time, each node queries the decision Q value table for the next state with the largest Q value, and uses this as the basis for routing decisions.
[0088] like Figure 3 Figure 4 shows the simulation performance test of the routing protocol DQGR and GPSR of this invention in a scenario with 50 UAV nodes. As can be seen from the figure, compared with the classic GPSR protocol, the packet delivery rate and end-to-end latency performance of the proposed protocol are improved to varying degrees as the node moving speed increases, and the performance improvement is more significant when the node speed is lower.
[0089] In summary, the routing protocol DQGR of this invention comprehensively considers Euclidean distance, node degree, and link quality metrics, and establishes a better route to the destination node in real time through the double-Q learning algorithm. At the same time, it reduces network congestion and the occurrence of routing holes. Simulation experiments show that DQGR improves network performance in highly dynamic UAV swarm scenarios.
[0090] The above specific embodiments are merely specific examples of the present invention and further illustrations of the technical solutions and beneficial effects of the present invention. They are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A location-based offline dual-Q learning routing method, characterized in that, Step 1: Model the UAV ad hoc network environment as a reinforcement learning problem and use the heterogeneous policy Q-learning method for offline learning. In this method, data packets are regarded as agents, nodes are regarded as states, choosing the next hop is regarded as an action, and a reward is obtained after the data packet is forwarded. Step 2: Initialize the reward table R and the two Q-value tables Q maintained by each node. A With Q B Table of maximum Q values between two neighbors nmaxA and Q nmaxB ; Step 3: Each node exchanges real-time location information via Hello message packets and calculates the reward value obtained by the agent for forwarding a neighbor as the next hop; the reward value is calculated as follows: Euclidean distance metric reward value ;in , , These are the x-coordinates of the current node, neighboring nodes, and the destination node, respectively. The same applies to the y and z directions. This defines the communication range of the current node. Node Degree Reward Value It is determined by a piecewise function based on the number of neighboring nodes n, the average node degree K, and the total number of nodes N; Link quality is the success rate of a node sending data to its neighbors. With reception success rate The product of: ; Relay node weighted reward value , where the coefficient , , These represent the weights of the three reward metrics, and their sum is 1. If the next hop is the destination node, the reward is set to the maximum value. ; If it is a hole node, apply a penalty value. ; Step 4: Each node generates agents at a fixed frequency, selects actions using an exponentially decaying ε-greedy strategy, and records the current state, action, next state, and reward, storing them as experience in the experience pool. Step 5: Each node draws experience from the experience pool at a fixed frequency and randomly selects Q. A Or Q B Update the Q-value table as follows: The maximum Q value of the next state is obtained from the corresponding neighbor's maximum Q value table; Step Six: Synchronize the maximum Q-value of neighbors, which specifically includes: Before sending the Hello packet, from Q A Find the action corresponding to the maximum Q value under each destination node, and set Q... B The Q value of this action is stored in array Q. nmaxarrA From Q B Find the action corresponding to the maximum Q value under each destination node, and set Q... A The Q value of this action is stored in array Q. nmaxarrB Q nmaxarrA and Q nmaxarrB Add the Hello package and broadcast it; After receiving the Hello packet, Q nmaxarrA Update Q to local nmaxA In the middle, Q nmaxarrB Update Q to local nmaxB middle; Step 7: Once the Q-value table converges, query the decision Q-value table, i.e., Q... A With Q B The next hop corresponding to the largest Q value among the average values is used as the routing decision.
2. The location-based offline dual-Q learning routing method as described in claim 1, characterized in that, The node degree reward value The specific calculation method is as follows: 。 3. The location-based offline dual-Q learning routing method as described in claim 1, characterized in that, The exploration probability ε of the ε-greedy strategy is updated in an exponentially decaying manner as follows: ， in The initial random probability, Total number of explorations For the attenuation rate, each This exploration The value decays once.
4. The location-based offline dual-Q learning routing method as described in claim 1, characterized in that, The learning factor α for updating the Q-value table is set to 0.4 to 0.5, and the future reward factor γ is set to 0.95.