Adaptive interference-aware routing method and system for dynamic unmanned aerial vehicle network

Abstract

The application provides a dynamic unmanned aerial vehicle network-oriented adaptive interference-aware routing method and system, comprising: establishing a routing protocol framework based on a Q-learning algorithm; constructing a multi-index fusion reward function; based on an iterative model and three indexes reflecting quantifiable targets, updating a learning rate alpha in a negative exponential function of link duration, adjusting a discount factor gamma through a complementary function of link stability, adopting a Softmax strategy with a temperature coefficient tau attenuation for action selection, and forming a closed-loop parameter optimization. The application enables unmanned aerial vehicles to perform routing decisions in a distributed and autonomous manner, and comprehensively considers the influence of link interference, link stability and other factors on unmanned aerial vehicle networks, thereby realizing efficient, low-delay and low-overhead optimal routing.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) network communication technology, specifically relating to an adaptive interference-aware routing method and system for dynamic UAV networks. Background Technology

[0002] Currently, drone swarm technology has become an important development trend in the drone field. This involves networking multiple drones carrying various devices to significantly improve payload, perception, and information processing capabilities, thereby achieving large-scale, high-efficiency, and multi-tasking operational goals.

[0003] When a single drone expands into a swarm, efficient and reliable end-to-end communication is crucial for achieving collaborative drone swarm operations. A routing protocol is a network protocol that ensures data packets are transmitted from their source address to their destination address. Currently, routing protocols suitable for drone networks are mainly divided into two categories: topology-based routing and location-based routing, such as traditional routing protocols like On-Demand Distance Vector Routing (AODV), Optimized Link-State Routing (OLSR), and Greedy Perimeter Stateless Routing (GPSR). Based on traditional protocols, some papers have proposed improved methods, such as Location-Based Optimized Link-State Routing (P-OLSR) and Mobility Prediction-Based Virtual Routing (MPVR) algorithms. However, existing routing protocols for drone networks still have the following problems:

[0004] (1) The open nature of UAV communication makes it more susceptible to interference than traditional communication systems. Once a UAV is attacked by interference signals, its communication quality may be significantly reduced, thereby affecting the normal execution of the mission.

[0005] (2) The high dynamism and spatial freedom of UAVs will lead to frequent changes in communication links, which in turn will affect the overall network performance. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive interference-aware routing method and system for dynamic unmanned aerial vehicle (UAV) networks to solve the above-mentioned problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides an adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks, comprising:

[0009] A routing protocol framework based on the Q-learning algorithm is established, with data packets as intelligent agents and UAV networks as the environment. Through the state transition mechanism, data packet forwarding decisions are mapped to Q-learning state-action pairs, and a Q-value iterative model containing learning rate α and discount factor γ is constructed.

[0010] A reward function integrating multiple indicators is constructed. The dynamic characteristics of UAV networks are transformed into quantifiable targets for reinforcement learning by three indicators: the expected number of transmissions (ETX) based on the signal-to-interference-plus-noise ratio (SINR), the link duration predicted by the node motion trajectory, and the distance difference reflecting the proximity of the destination node.

[0011] Based on the iterative model and the quantifiable objectives reflected by three indicators, the learning rate α is updated by a negative exponential function of link duration, the discount factor γ is adjusted by a complementary function of link stability, and a softmax strategy with temperature coefficient τ decay is used for action selection, forming a closed-loop parameter optimization.

[0012] Furthermore, the construction of the Q-value iterative model, which includes a learning rate α and a discount factor γ, includes:

[0013] By representing data packets as agents, the entire network is considered the environment, and the state space consists of all drones used to forward data packets; the choice of the next hop is regarded as an action, and the action space includes all available neighbor nodes; state transitions are equivalent to data packet forwarding decisions between drones; that is, any data packet in the network environment has a state of S. t , representing the current drone node; the set of all actions that a data packet can take is the action space A, which consists of all available neighbor nodes; if the data packet is in S t A certain action a is taken in state A. t That is, if a neighboring node is selected for forwarding, the current state will transition to the next state S. t+1 During the state transition, the environment will provide a reward R(S) based on the reward mechanism. t ,a t Ultimately, the algorithm aims to continuously learn the optimal policy π in the environment, maximizing the cumulative reward obtained by executing this policy over a long period, and obtaining the state S based on this policy. t The optimal action to be performed is a t =π(S) t );

[0014] The object estimated for the policy π evaluation is the value function Q, which is calculated by the following formula:

[0015]

[0016] Q t (s t ,a t ) is in state s t Take action a t Q value at time, Q t+1Let be the Q value at time t+1; α∈(0,1) represents the learning rate, and γ∈(0,1) represents the discount factor; Indicates all possible actions 'a' in the next state. ′ ∈A t+1 The maximum Q value, where A t+1 It is state s t+1 The action space within the context; for different QoS metrics, the reward R(s) is defined and calculated using information embedded in the Hello message. t ,a t All drone nodes will maintain a neighbor table and a Q-value table by periodically exchanging Hello messages with their neighbors; the former is used to update the action space and calculate rewards, and the latter is used to determine routing decisions.

[0017] Furthermore, the three indicators—the expected number of transmissions (ETX) calculated based on the signal-to-interference-plus-noise ratio (SINR), the link duration predicted by the node's motion trajectory, and the distance difference reflecting the proximity of the destination node—include:

[0018] First, the bit error rate (BER) is calculated using the mapping relationship between the bit error rate and the signal-to-interference-plus-noise ratio (SINR).

[0019]

[0020] Where SINR represents the signal-to-interference-plus-noise ratio at the receiving node, γ is a bandwidth-related constant, and Q(z) represents the probability that the random variable X in the Gaussian distribution X ~ N(0,1) is greater than the threshold z.

[0021]

[0022] Therefore, the success rate (PDR) for transmitting a data packet of size 1 lbits can be calculated as follows:

[0023]

[0024] Among them, BER i This represents the bit error rate of the i-th bit at the receiving node;

[0025] Finally, the expected number of data packet transmissions (ETX) is obtained.

[0026]

[0027] Among them, PDR f PDR r These represent the success rates of forward and backward data packet transmission, respectively.

[0028] After obtaining its own and its neighbors' location information, the node estimates the link duration; d ij(t) represents the relative distance between nodes i and j at time t, and R is the communication coverage radius of the nodes; when nodes i and j receive each other's Hello message at time t2, they will extract the corresponding node location information from the message; then, nodes i and j will estimate the link duration between themselves and the other party based on the information.

[0029] First, calculate the distance between nodes i and j at time t using the Euclidean distance formula.

[0030]

[0031] Secondly, the change in the relative distance between nodes i and j is calculated using the following formula.

[0032] Δd=d ij (t2)-d ij (t1)(7)

[0033] Finally, the node estimates the link duration as follows:

[0034]

[0035] Among them, T max =R / V min V min This represents the minimum speed of the drone node;

[0036] The sign of Δd reflects the trend of the relative distance between nodes; when Δd is positive, the relative distance between the two nodes gradually increases; from time t1 to t2, the relative velocity between the nodes is Δd / (t2-t1); therefore, the expected time for the relative distance between the two nodes to exceed the communication coverage range is... That is, the link duration; when Δd is negative, the two nodes are moving towards each other; the two nodes move in opposite directions at a speed of V. min The time required for relative velocity to travel a relative distance R is recorded as T. max At time t2, d ij The ratio of (t2) to R multiplied by T max Used as a predicted value for link duration;

[0037] The distance metric in the reward function is,

[0038] Δd ij =d iD -d jD (9)

[0039] Where d iD d jD These represent the Euclidean distances between nodes i and j and the destination node, respectively.

[0040] Furthermore, the reward function includes:

[0041]

[0042] Here, ETX is the estimated expected number of transmissions, used to measure the degree of link interference between nodes; t LD (i,j) represents the duration of the link between nodes, Δd ij It is the difference in distance between the node and the destination node, indicating that choosing the node closer to the destination node as a relay will yield a greater reward; A, B, and C are the weight parameters of various routing metrics, which satisfy A+B+C=1.

[0043] Furthermore, two special cases of reward function design are shown in formula (11).

[0044]

[0045] The current node will receive the maximum reward R when the next hop node is the destination. max This ensures that the route eventually leads to the destination node; when all of a node's neighboring nodes are farther from the destination node, the routing process gets stuck in a local optimum; at this point, the minimum reward R that the current node should receive is reached. min .

[0046] Furthermore, the learning rate α, based on the iterative model and quantifiable targets reflected by three indicators, is updated using a negative exponential function of link duration. The discount factor γ is adjusted through a complementary function of link stability. A softmax strategy with temperature coefficient τ decay is employed for action selection, forming a closed-loop parameter optimization, including:

[0047] An adaptive learning mechanism was introduced to dynamically adjust the values ​​of α and γ by utilizing link stability;

[0048] When the links between nodes are more unstable, i.e., the link duration is shorter, the Q-value needs to be updated more quickly, requiring a larger learning rate. The update formula for the learning rate α is defined as follows:

[0049]

[0050] Among them, t LD (i,j) is calculated by formula (8), where i and j represent the values ​​in the current state s. t And select action a t The corresponding pair of adjacent nodes;

[0051] When the links between nodes are more unstable, a smaller γ needs to be applied; the update formula for the discount factor γ is defined as follows.

[0052]

[0053] Among them, t LD (j,k) is calculated by formula (8), where j and k represent the next state s. t+1 And the action that maximizes Q in the next state, i.e. The corresponding pair of adjacent nodes;

[0054] The Softmax method was used.

[0055]

[0056] Where τ is a positive constant; during the iteration process, τ is updated by τ(t) = τ(0) / log2(1+t); initially, since τ is large, the policy probability π(s) corresponding to different actions is... t ,a t The fact that τ is equal to the time step size indicates that the strategy is more inclined to choose actions randomly; as the time step increases, τ decreases, and the policy probability π(s) corresponding to different actions increases. t ,a t The gap between them is gradually widening, which means that they tend to choose the more stable link.

[0057] Furthermore, the protocol process executed by each drone is as follows:

[0058] First, drones maintain their neighbor tables and interaction status information through Hello messages. When a Hello message is received from another node, if the sender is newly joined, it is added to the neighbor list and the corresponding status information is updated. Then, routing metrics such as expected number of transmissions and link duration are calculated. If no Hello message is received from any other drone, neighbor nodes whose links have failed are removed based on the estimated remaining link duration.

[0059] Secondly, after each drone node updates its neighbor information via a Hello message, it will execute a distributed Q-learning process. When the target node is a neighbor of the current node, the current node will receive the maximum reward and update its Q value. When the target node is not included in the node's neighbor table, the node will gain experience based on its behavioral strategy, calculate the reward value, and then the node will adaptively adjust the learning rate and discount factor, and iteratively update the Q-table.

[0060] Finally, when the drone generates or receives a data packet, it queries the Q-table at the current time and selects the neighbor node with the largest Q value as the next-hop relay node.

[0061] In a second aspect, the present invention provides an adaptive interference-aware routing system for dynamic unmanned aerial vehicle (UAV) networks, comprising:

[0062] The framework building module is used to establish a routing protocol framework based on the Q-learning algorithm. It treats data packets as agents and UAV networks as environments. Through a state transition mechanism, it maps data packet forwarding decisions to Q-learning state-action pairs and builds a Q-value iterative model that includes a learning rate α and a discount factor γ.

[0063] The reward function construction module is used to construct a reward function that integrates multiple indicators. It transforms the dynamic characteristics of UAV networks into quantifiable targets for reinforcement learning by using three indicators: the expected number of transmissions (ETX) based on the signal-to-interference-plus-noise ratio (SINR), the link duration predicted by the node's motion trajectory, and the distance difference reflecting the proximity of the destination node.

[0064] The optimized output module is used to update the learning rate α as a negative exponential function of link duration based on the iterative model and quantifiable targets reflected by three indicators. The discount factor γ is adjusted through the complementary function of link stability, and the action selection is performed using a softmax strategy with temperature coefficient τ decay, thus forming a closed-loop parameter optimization.

[0065] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the adaptive interference-aware routing method for dynamic unmanned aerial vehicle networks.

[0066] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the adaptive interference-aware routing method for dynamic unmanned aerial vehicle networks.

[0067] Compared with the prior art, the present invention has the following technical effects:

[0068] This invention models the routing process of UAV networks using a Markov decision process, analyzes a routing protocol framework based on the Q-learning algorithm, and optimizes the algorithm flow for the characteristics of swarm UAV networks. This enables UAVs to perform routing decisions in a distributed and autonomous manner, improving the algorithm's convergence speed and stability. Consequently, UAVs can select the optimal relay node more quickly and accurately, effectively improving packet transmission rate and reducing end-to-end latency.

[0069] This invention quantitatively analyzes and evaluates factors such as link interference and link stability based on UAV perception information, and designs a reward function that takes all factors into consideration. This improves the adaptability of the UAV network in interference environments, enabling routing paths to effectively avoid areas with strong interference. Furthermore, comparisons show that the path established by this invention is more efficient, resulting in better overall performance, such as lower latency and lower energy consumption.

[0070] This invention utilizes the link stability parameters obtained from analysis to design an adaptive learning mechanism suitable for dynamic topologies. This mechanism can adaptively adjust behavioral strategies and learning parameters, thereby effectively improving the data packet transmission success rate and enhancing the adaptability of routing protocols in high-speed environments. Attached Figure Description

[0071] Figure 1 This is a diagram of the routing protocol framework of the present invention.

[0072] Figure 2 This is a flowchart of the present invention.

[0073] Figure 3 This is a routing example diagram established for this invention.

[0074] Figure 4 This is a comparison chart of the packet delivery rates of the present invention and the comparative routing protocols.

[0075] Figure 5 This is a comparison chart of the end-to-end latency of the present invention and the comparative routing protocol.

[0076] Figure 6 This is a comparison chart of the energy consumption of the present invention and the comparative routing protocol. Detailed Implementation

[0077] The present invention will be further described below with reference to the accompanying drawings:

[0078] Example 1, please refer to Figure 1 This invention provides an adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks, comprising:

[0079] A routing protocol framework based on the Q-learning algorithm is established, with data packets as intelligent agents and UAV networks as the environment. Through the state transition mechanism, data packet forwarding decisions are mapped to Q-learning state-action pairs, and a Q-value iterative model containing learning rate α and discount factor γ is constructed.

[0080] A reward function integrating multiple indicators is constructed. The dynamic characteristics of UAV networks are transformed into quantifiable targets for reinforcement learning by three indicators: the expected number of transmissions (ETX) based on the signal-to-interference-plus-noise ratio (SINR), the link duration predicted by the node motion trajectory, and the distance difference reflecting the proximity of the destination node.

[0081] Based on the iterative model and the quantifiable objectives reflected by three indicators, the learning rate α is updated by a negative exponential function of link duration, the discount factor γ is adjusted by a complementary function of link stability, and a softmax strategy with temperature coefficient τ decay is used for action selection, forming a closed-loop parameter optimization.

[0082] The purpose of this invention is to address the shortcomings of existing methods by proposing a fast adaptive interference-aware routing protocol for dynamic unmanned aerial vehicle (UAV) networks. This protocol enables UAVs to perform routing decisions in a distributed and autonomous manner, and comprehensively considers the impact of factors such as link interference and link stability on the UAV network, thereby achieving optimal routing with high efficiency, low latency, and low overhead.

[0083] Example 2: This invention provides an adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks, comprising:

[0084] S1: Establish a routing protocol framework based on the Q-learning algorithm

[0085] S2: Design the reward function based on actual needs.

[0086] S3: Design an adaptive learning mechanism based on UAV network characteristics

[0087] Specifically, in step S1, by representing data packets as agents, the entire network can be considered an environment, and the state space consists of all drones used to forward data packets. The choice of the next hop is considered an action, and the action space includes all available neighbor nodes. Thus, the state transition is equivalent to the data packet forwarding decision between drones. That is, any data packet in the network environment has a state of S. t , representing the current drone node. The set of all actions a data packet can take is the action space A, consisting of all available neighbor nodes. If a data packet takes an action a from A in state St... t That is, if a neighboring node is selected for forwarding, the current state will transition to the next state S. t+1 During the state transition, the environment will provide a reward R(S) according to the reward mechanism. t ,a t Ultimately, the algorithm aims to learn an optimal policy π in the environment through continuous attempts, maximizing the cumulative reward obtained by executing this policy over a long period. The state S can then be obtained based on this policy. t The optimal action to be performed is a t =π(S) t ).

[0088] The object estimated for the policy π evaluation is the value function Q, which is calculated by the following formula:

[0089]

[0090] Q t (s t ,a t) is in state s t Take action a t Q value at time, Q t+1 Let α be the Q value at time t+1. α∈(0,1) represents the learning rate, and γ∈(0,1) represents the discount factor. Indicates all possible actions 'a' in the next state. ′ ∈A t+1 The maximum Q value, where A t+1 It is state s t+1 The action space within the message. For different QoS metrics, the reward R(s) can be defined and calculated using information embedded in the Hello message. t ,a t All drone nodes maintain a neighbor table and a Q-table by periodically exchanging Hello messages with their neighbors. The former is used to update the action space and calculate rewards, while the latter is used to determine routing decisions.

[0091] Specifically, in step S2, the design of the reward function takes into account the effects of factors such as link quality, stability, and distance.

[0092] To better characterize link quality, this invention uses the signal-to-interference-plus-noise rate (SINR) to calculate the expected number of transmissions (ETX) of data packets.

[0093] First, the BER is calculated using the mapping relationship between Bit Error Rate (BER) and Signal-to-Interference-Ratio (SIR).

[0094]

[0095] Where SINR represents the signal-to-interference-plus-noise ratio at the receiving node, and γ is a bandwidth-related constant. Q(z) represents the probability that the random variable X in the Gaussian distribution X ~ N(0,1) is greater than the threshold z, i.e.

[0096]

[0097] Therefore, the success rate (PDR) for transmitting a data packet of size 1 lbits can be calculated as follows:

[0098]

[0099] Among them, BER i This represents the bit error rate of the i-th bit at the receiving node.

[0100] Finally, the expected number of data packet transfers (ETX) can be obtained.

[0101]

[0102] Among them, PDR f PDR r These represent the success rates of forward and backward data packet transmission, respectively.

[0103] Because drones are highly maneuverable aircraft, drone networks face the challenge of frequent link changes. This invention employs a link duration estimation method to measure the stability of links between nodes.

[0104] After obtaining its own and its neighbors' location information, a node can estimate the link duration. For example, nodes i and j are a pair of neighboring nodes. ij (t) represents the relative distance between nodes i and j at time t, and R is the communication coverage radius of the nodes. When nodes i and j receive each other's Hello message at time t2, they extract the corresponding node location information from the message. Then, nodes i and j can estimate the link duration between themselves and each other based on the information.

[0105] First, the distance between nodes i and j at time t can be calculated using the Euclidean distance formula.

[0106]

[0107] Secondly, the change in the relative distance between nodes i and j can be calculated using the following formula.

[0108] Δd=d ij (t2)-d ij (t1).#(7)

[0109] Finally, the node can estimate the link duration as follows:

[0110]

[0111] Among them, T max =R / V min V min This represents the minimum speed of the drone node.

[0112] The sign of Δd reflects the trend of the relative distance between nodes. When Δd is positive, the relative distance between the two nodes gradually increases. From time t1 to t2, the relative velocity between the nodes is Δd / (t2-t1). Therefore, the time it takes for the relative distance between the two nodes to exceed the expected communication coverage range is... This refers to the link duration. When Δd is negative, the two nodes are moving towards each other. The two nodes move in opposite directions at a speed of V. min The time required for relative velocity to travel a relative distance R is recorded as T. maxAt time t2, d ij The ratio of (t2) to R multiplied by T max Used as a predicted value for link duration.

[0113] The distance metric in the reward function is,

[0114] Δd ij =d iD -d jD #(9)

[0115] Where d iD d jD Let represent the Euclidean distances between nodes i and j and the destination node, respectively. If node j has a larger Δd... ij This means that if node j is closer to the destination node, it will generate a larger reward.

[0116] Taking into account the impact of the above factors on routing decisions, the reward function designed in this invention is as follows:

[0117]

[0118] Here, ETX is the estimated expected number of transmissions, used to measure the degree of link interference between nodes; t LD (i,j) represents the duration of the link between nodes, Δd ij This is the difference in distance between the node and the destination node, indicating that choosing the node closer to the destination node as a relay will yield a greater reward. A, B, and C are the weight parameters for various routing metrics, which satisfy A+B+C=1.

[0119] Furthermore, in order to ensure that the routing strategy is goal-oriented and can effectively avoid getting trapped in local optima, the design of the reward function also needs to consider two special cases, as shown in formula (11).

[0120]

[0121] The current node will receive the maximum reward R when the next hop node is the destination. max This is to ensure that the route eventually leads to the destination node. When all of a node's neighbors are further away from the destination node, the routing process gets stuck in a local optimum. At this point, the current node should receive the minimum reward R. min This is to avoid routing gaps. Under normal circumstances, the reward will be calculated using formula (10).

[0122] Specifically, in step S3, due to the high dynamism of the swarm drone network, the location, status, and neighbor information of the drones will change frequently. In order to better adapt to the dynamism of the swarm drone network and improve network performance, this invention introduces an adaptive learning mechanism on the traditional Q-learning algorithm, using link stability to dynamically adjust the values ​​of α and γ in formula (1).

[0123] When the links between nodes are more unstable, i.e., the link duration is shorter, the Q-value needs to be updated more quickly, requiring a larger learning rate. Therefore, the update formula for the learning rate α is defined as follows:

[0124]

[0125] Among them, t LD (i,j) is calculated by formula (8), where i and j represent the values ​​in the current state s. t And select action a t The corresponding pair of adjacent nodes.

[0126] And because of formula (1) Indicates all possible actions 'a' in the next state. ′ ∈A t+1 The maximum Q value is determined by the discount factor γ, meaning a larger discount factor γ implies a greater weight for future rewards in the iteration. Therefore, when the links between nodes are more unstable, a smaller γ needs to be applied to weaken the impact of future rewards. The update formula for the discount factor γ is defined as follows:

[0127]

[0128] Among them, t LD (j,k) is calculated by formula (8), where j and k represent the next state s. t+1 And the action that maximizes Q in the next state, i.e. The corresponding pair of adjacent nodes.

[0129] In addition, the choice of behavioral strategy also needs to consider the impact of node dynamics. When generating experience samples, the behavioral strategy can choose the next action by randomly selecting a new path or greedily selecting the action with the highest Q value. If only a greedy strategy is used, it may get stuck in a local optimum and fail to discover a better strategy. On the other hand, if only a random strategy is used, a lot of time may be spent exploring useless actions without obtaining sufficient cumulative rewards. In swarm drone networks, due to the frequent updates of the action space, conventional behavioral strategies cannot fully fit the dynamic characteristics of the network. Therefore, this invention adopts the Softmax method.

[0130]

[0131] Here, τ is a positive constant. During the iteration process, τ is updated by τ(t) = τ(0) / log2(1+t). Initially, since τ is relatively large, the policy probabilities π(s) corresponding to different actions are... t ,a t The probability π(s) is almost equal to the probability τ, meaning the policy tends to choose actions randomly; as the time step increases, τ decreases, and the policy probability π(s) corresponding to different actions increases. t ,a t The widening gap between the two means that the strategy will reduce exploration and tend to choose more stable links.

[0132] Specifically, the protocol process executed by each drone is as follows:

[0133] First, drones maintain their neighbor tables and interaction state information through Hello messages. When a Hello message is received from another node, if the sender is newly joined, it is added to the neighbor list, and the corresponding state information is updated. Then, routing metrics such as expected number of transmissions and link duration are calculated. If no Hello messages are received from any other drones, neighbor nodes whose links have failed are removed based on the estimated remaining link duration.

[0134] Secondly, each drone node will perform a distributed Q-learning process after updating its neighbor information via a Hello message. When the target node is a neighbor of the current node, the current node will receive the maximum reward and update its Q-value according to formula (1). When the target node is not included in the node's neighbor table, the node will gain experience based on its behavioral strategy, calculate the reward value using formula (11), and then the node will adaptively adjust its learning rate and discount factor, and iteratively update the Q-table. It is worth noting that the iteration will stop when any of the following conditions are met: 1) The maximum number of iterations k is reached. max ;2)

[0135] Finally, when the drone generates or receives a data packet, it queries the Q-table at the current time and selects the neighbor node with the largest Q value as the next-hop relay node.

[0136] In another embodiment of the present invention, an adaptive interference-aware routing system for dynamic unmanned aerial vehicle (UAV) networks is provided, which can be used to implement the above-described adaptive interference-aware routing method for dynamic UAV networks. Specifically, the system includes:

[0137] The framework building module is used to establish a routing protocol framework based on the Q-learning algorithm. It treats data packets as agents and UAV networks as environments. Through a state transition mechanism, it maps data packet forwarding decisions to Q-learning state-action pairs and builds a Q-value iterative model that includes a learning rate α and a discount factor γ.

[0138] The reward function construction module is used to construct a reward function that integrates multiple indicators. It transforms the dynamic characteristics of UAV networks into quantifiable targets for reinforcement learning by using three indicators: the expected number of transmissions (ETX) based on the signal-to-interference-plus-noise ratio (SINR), the link duration predicted by the node's motion trajectory, and the distance difference reflecting the proximity of the destination node.

[0139] The optimized output module is used to update the learning rate α as a negative exponential function of link duration based on the iterative model and quantifiable targets reflected by three indicators. The discount factor γ is adjusted through the complementary function of link stability, and the action selection is performed using a softmax strategy with temperature coefficient τ decay, thus forming a closed-loop parameter optimization.

[0140] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0141] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of an adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks.

[0142] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the adaptive interference-aware routing method for dynamic UAV networks in the above embodiments.

[0143] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks, characterized in that, include: A routing protocol framework based on the Q-learning algorithm is established, with data packets as intelligent agents and UAV networks as the environment. Through the state transition mechanism, data packet forwarding decisions are mapped to Q-learning state-action pairs, and a Q-value iterative model containing learning rate α and discount factor γ is constructed. A reward function integrating multiple indicators is constructed. The dynamic characteristics of UAV networks are transformed into quantifiable targets for reinforcement learning by three indicators: the expected number of transmissions (ETX) based on the signal-to-interference-plus-noise ratio (SINR), the link duration predicted by the node motion trajectory, and the distance difference reflecting the proximity of the destination node. Based on the iterative model and the quantifiable targets reflected by three indicators, the learning rate α is updated by a negative exponential function of the link duration, the discount factor γ is adjusted by a complementary function of the link stability, and the action selection is carried out by a softmax strategy with temperature coefficient τ decay, thus forming a closed-loop parameter optimization. The three metrics—Expected Transmission Count (ETX) calculated based on the signal-to-interference-plus-noise ratio (SINR), Link Duration predicted by node motion trajectories, and Distance Difference reflecting the proximity of the destination node—include: First, the bit error rate (BER) is calculated using the mapping relationship between the bit error rate and the signal-to-interference-plus-noise ratio (SINR). SINR represents the signal-to-interference-plus-noise ratio at the receiving node. A constant related to bandwidth; Represents Gaussian distribution random variables Greater than the threshold The probability, i.e. Thus, a transmission of a size of The success rate (PDR) of a data packet of bits can be calculated as follows: in, Indicates the first node at the receiving node Bit error rate per bit; Finally, the expected number of data packet transmissions (ETX) is obtained. in , These represent the success rates of forward and backward data packet transmission, respectively. After obtaining its own and its neighbors' location information, the node estimates the link duration; Represents a node and exist Relative distance at any given moment The communication coverage radius of the node; when the node and exist When a node receives a Hello message from another node, it extracts the corresponding node location information from that message; then, the node... and Estimate the duration of the link between yourself and the other party based on the information; First, calculate the time using the Euclidean distance formula. Time node and Distance between Secondly, nodes and The change in relative distance between them is calculated using the following formula: Finally, the node estimates the link duration as follows: in, , This represents the minimum speed of the drone node; The positive or negative sign reflects the trend of the relative distance between nodes; when When the time is positive, the relative distance between the two nodes gradually increases; from the perspective of time... arrive The relative speed between nodes is Therefore, the time it takes for the relative distance between two nodes to exceed the expected communication coverage range is... That is, the link duration; when When the value is negative, the two nodes are moving towards each other; the two nodes are moving towards each other. The relative speed and the relative distance traveled are The required time is recorded as follows ;exist At that moment, and The ratio multiplied by Used as a predicted value for link duration; The distance metric in the reward function is, in , Representing nodes respectively and The Euclidean distance between the destination node and the target node.

2. The adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks according to claim 1, characterized in that, The construction of the Q-value iterative model, which includes a learning rate α and a discount factor γ, includes: By representing data packets as agents, the entire network is considered the environment, and the state space consists of all drones used to forward data packets; the choice of the next hop is regarded as an action, and the action space includes all available neighbor nodes; state transitions are equivalent to data packet forwarding decisions between drones; that is, any data packet in the network environment has a state of S. t This represents the current drone node; the set of all actions that a data packet can take is called the action space. It consists of all available neighbor nodes; if the data packet is in S t The measures taken in the state A certain action in t That is, if a neighboring node is selected for forwarding, the current state will transition to the next state S. t+1 During the state transition, the environment will provide a reward based on the reward mechanism. (S t , t Ultimately, the algorithm aims to learn the optimal strategy by continuously trying different approaches within the environment. To maximize the cumulative reward obtained by executing this strategy over a long period of time, the state S is obtained according to this strategy. t The optimal action to be performed t = (S t ); strategy The object being evaluated is a value function. The value function is calculated by the following formula: in It is in state Take action time value, for The Q value at time t; α∈(0,1) represents the learning rate, and γ∈(0,1) represents the discount factor; Indicates all possible actions in the next state. The maximum Q value, where It is a state The action space within the message; for different QoS metrics, rewards are defined and calculated using information embedded in the Hello message. All drone nodes will maintain a neighbor table and a Q-value table by periodically exchanging Hello messages with their neighbors; the former is used to update the action space and calculate rewards, and the latter is used to determine routing decisions.

3. The adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks according to claim 1, characterized in that, The reward function includes: in, It is an estimated expected number of transmissions, used to measure the degree of link interference between nodes; Indicates the duration of the link between nodes. is the difference in distance between the node and the destination node, indicating that choosing the node closer to the destination node as a relay will yield a greater reward; A, B, and C are the weight parameters for various routing metrics, which satisfy... .

4. The adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks according to claim 3, characterized in that, Two special cases of reward function design are shown in formula (11). The current node will receive the maximum reward when the next hop node is the destination. This is to ensure that the route eventually leads to the destination node; When all of a node's neighboring nodes are farther from the destination node, the routing process gets stuck in a local optimum; at this point, the current node should receive the minimum reward. .

5. The adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks according to claim 4, characterized in that, The iterative model and quantifiable objectives reflected by three metrics are used to update the learning rate α as a negative exponential function of link duration, adjust the discount factor γ through a complementary function of link stability, and employ a softmax strategy with temperature coefficient τ decay for action selection, forming a closed-loop parameter optimization, including: An adaptive learning mechanism was introduced to dynamically adjust the values ​​of α and γ by utilizing link stability; The more unstable the links between nodes, i.e., the shorter the link duration, the faster the Q-value needs to be updated, requiring a larger learning rate; define the learning rate. The update formula is in, Calculated by formula (8), , Indicates the current state and select actions The corresponding pair of adjacent nodes; When the links between nodes are more unstable, a smaller γ needs to be applied; the update formula for the discount factor γ is defined as follows. in, Calculated by formula (8), , Indicates the next state And the action that maximizes Q in the next state, i.e. , the corresponding pair of adjacent nodes; The Softmax method was used. in, It is a positive constant; during the iteration process, Depend on To update; initially, due to Larger, the probability of the strategy corresponding to different actions Equal values ​​mean that the strategy is more inclined to select actions randomly; as the time step increases, Reduce the probability of strategies corresponding to different actions. The widening gap between them indicates a tendency to choose the more stable link.

6. The adaptive interference-aware routing method for dynamic unmanned aerial vehicle (UAV) networks according to claim 5, characterized in that, The protocol process executed by each drone is as follows: First, drones maintain neighbor tables and interaction status information through Hello messages. When a Hello message is received from another node, if the sender is newly joined, it is added to the neighbor list and the corresponding status information is updated. Then, the expected number of transmissions and the link duration routing metric are calculated respectively. If no Hello messages are received from any other drones, the neighboring nodes whose links have failed are removed based on the estimated remaining link duration. Secondly, each drone node will execute a distributed Q-learning process after updating its neighbor information via a Hello message; When the destination node is a neighbor of the current node, the current node will receive the maximum reward and update its Q value; When the node's neighbor table does not contain the target node, the node will gain experience based on the behavioral strategy, calculate the reward value, and then the node will adaptively adjust the learning rate and discount factor, and iteratively update the Q-table. Finally, when the drone generates or receives a data packet, it queries the Q-table at the current time and selects the neighbor node with the largest Q value as the next-hop relay node.

7. An adaptive interference-aware routing system for dynamic unmanned aerial vehicle (UAV) networks, characterized in that, include: The framework building module is used to establish a routing protocol framework based on the Q-learning algorithm. It treats data packets as agents and UAV networks as environments. Through a state transition mechanism, it maps data packet forwarding decisions to Q-learning state-action pairs and builds a Q-value iterative model that includes a learning rate α and a discount factor γ. The reward function construction module is used to construct a reward function that integrates multiple indicators. It transforms the dynamic characteristics of UAV networks into quantifiable targets for reinforcement learning by using three indicators: the expected number of transmissions (ETX) based on the signal-to-interference-plus-noise ratio (SINR), the link duration predicted by the node's motion trajectory, and the distance difference reflecting the proximity of the destination node. The optimized output module is used to update the learning rate α as a negative exponential function of link duration based on the iterative model and quantifiable targets reflected by three indicators, adjust the discount factor γ through the complementary function of link stability, and use the Softmax strategy with temperature coefficient τ decay for action selection to form closed-loop parameter optimization. The three metrics—Expected Transmission Count (ETX) calculated based on the signal-to-interference-plus-noise ratio (SINR), Link Duration predicted by node motion trajectories, and Distance Difference reflecting the proximity of the destination node—include: First, the bit error rate (BER) is calculated using the mapping relationship between the bit error rate and the signal-to-interference-plus-noise ratio (SINR). SINR represents the signal-to-interference-plus-noise ratio at the receiving node. A constant related to bandwidth; Represents Gaussian distribution random variables Greater than the threshold The probability, i.e. Thus, a transmission of a size of The success rate (PDR) of a data packet of bits can be calculated as follows: in, Indicates the first node at the receiving node Bit error rate per bit; Finally, the expected number of data packet transmissions (ETX) is obtained. in , These represent the success rates of forward and backward data packet transmission, respectively. After obtaining its own and its neighbors' location information, the node estimates the link duration; Represents a node and exist Relative distance at any given moment The communication coverage radius of the node; when the node and exist When a node receives a Hello message from another node, it extracts the corresponding node location information from that message; then, the node... and Estimate the duration of the link between yourself and the other party based on the information; First, calculate the time using the Euclidean distance formula. Time node and Distance between Secondly, nodes and The change in relative distance between them is calculated using the following formula: Finally, the node estimates the link duration as follows: in, , This represents the minimum speed of the drone node; The positive or negative sign reflects the trend of the relative distance between nodes; when When the time is positive, the relative distance between the two nodes gradually increases; from the perspective of time... arrive The relative speed between nodes is Therefore, the time it takes for the relative distance between two nodes to exceed the expected communication coverage range is... That is, the link duration; when When the value is negative, the two nodes are moving towards each other; the two nodes are moving towards each other. The relative speed and the relative distance traveled are The required time is recorded as follows ;exist At that moment, and The ratio multiplied by Used as a predicted value for link duration; The distance metric in the reward function is, in , Representing nodes respectively and The Euclidean distance between the destination node and the target node.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the adaptive interference-aware routing method for dynamic unmanned aerial vehicle networks as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the adaptive interference-aware routing method for dynamic unmanned aerial vehicle networks as described in any one of claims 1 to 6.

Images