An adaptive routing method for UAV ad hoc networks based on Q-Learning

Abstract

The present invention belongs to the field of unmanned aerial vehicle self-organizing network, and relates to a self-adaptive routing method for unmanned aerial vehicle self-organizing network based on Q-Learning; the method includes a routing discovery phase and a routing maintenance phase, in which a routing decision model based on a Q-Learning algorithm is respectively called to obtain a route from a source node to a destination node; in the routing discovery phase, the unmanned aerial vehicle node calculates the link quality from a received RREQ message or RREP message, and updates a Q table based on the link quality, and uses the Q table to establish a route from the source node to the destination node; in the routing maintenance phase, the network topology change is perceived through a HELLO message, the Q table is dynamically updated, and the forward route from the source node to the destination node is updated according to the Q table. The present invention can dynamically adapt to changes in network topology, reduce data packet transmission delay, and improve network life, and the complexity of the present invention is low.

Description

Technical Field

[0001] The invention belongs to the field of unmanned aerial vehicle self-organizing networks, and relates to a self-adaptive routing method for unmanned aerial vehicle self-organizing networks based on Q-Learning. Background Art

[0002] With the remarkable advancement of modern technology, such as artificial intelligence, sensors, batteries, radio and global positioning system (GPS), unmanned aerial vehicles (UAVs) have been used in many different applications. Due to the characteristics of fast deployment, high flexibility, strong adaptability and strong functionality, UAVs are also widely used in various civil and military applications, such as detection, monitoring, search and rescue. One of the most critical challenges in the application of UAVs is the mission area coverage. However, a single UAV cannot complete complex tasks, and a multi-UAV network can more effectively complete various complex tasks. In a multi-UAV network, each UAV node can act as a router, and all UAV nodes establish a fully distributed wireless network, which forms a UAV self-organizing network (Flying Ad-hoc Network, FANET). There are huge challenges in cooperative communication and networking in FANET, among which routing technology has become one of the hottest research areas in multi-UAV network communication.

[0003] The transmission paths in drone networks are mainly supported by multi-hop communications, where drones move freely in the mission area at a speed of 20–300 km / h. The fast-moving drones cause the network topology to change very frequently. In addition, the joining and leaving of nodes in FANET also makes routing maintenance more complicated. The application of FANET requires reliable and robust data transmission. Therefore, routing protocols have a crucial impact on the stability and performance of the network. Traditional topology-based routing methods include active, reactive and hybrid routing. Among them, active routing brings huge overhead to maintaining the routing table, and reactive routing faces the problem of high latency caused by frequently starting the routing discovery process in highly dynamic FANET.

[0004] Therefore, in order to better improve the communication capability of UAV networking, how to design a simple, efficient and robust routing protocol based on network status information is the problem to be solved by the present invention. Summary of the invention

[0005] In order to solve the above problems and effectively improve the communication performance of unmanned aerial vehicle ad hoc networks, the present invention provides an adaptive routing protocol algorithm based on Q-Learning. The present invention first establishes a routing decision model based on the Q-Learning algorithm. The unmanned aerial vehicle in the network learns and maintains the Q table to the destination node in a distributed manner, thereby obtaining the best route to the destination node. Then, a link quality evaluation model between nodes is proposed. The unmanned aerial vehicle node interacts with the local environment through control messages such as RREQ (Route Request), RREP (Route Reply), and HELLO, collects the status information of each neighbor node, and calculates the link quality between the neighbor nodes according to the link quality evaluation model. Finally, the Q-Learning algorithm in reinforcement learning is adopted in the route discovery and route maintenance stages. The unmanned aerial vehicle in the network learns and maintains the Q table to the destination node in a distributed manner, realizes dynamic switching of routes, and thereby obtains the best route to the destination node. In addition, in the route maintenance stage, an adaptive HELLO interval is also adopted to improve the real-time performance of Q table update and enhance the robustness of the protocol.

[0006] The present invention discloses a UAV self-organizing network routing method based on Q-Learning, which includes a routing discovery phase and a routing maintenance phase. In the routing discovery phase and the routing maintenance phase, a routing decision model based on the Q-Learning algorithm is respectively called to obtain a route from a source node to a destination node, specifically including:

[0007] In the route discovery phase, before the source node and the destination node communicate for the first time in the UAV ad hoc network, route discovery is started and the Q tables of all UAV nodes are initialized;

[0008] The source node broadcasts the RREQ message, and the drone node calculates the link quality based on the link evaluation model based on the received RREQ message, and updates the Q table of the source node based on the link quality, and uses the Q table to establish a reverse route from the source node to the destination node;

[0009] After receiving the RREQ message, the destination node sends a RREP message to the source node, calculates the link quality according to the link evaluation model, updates the Q table to the destination node based on the link quality, and uses the Q table to establish a forward route from the source node to the destination node;

[0010] In the routing maintenance phase, the network topology changes are perceived through HELLO messages, the Q table is dynamically updated according to the UAV self-organizing network environment, and the route from the source node to the destination node is updated according to the Q table.

[0011] Beneficial effects of the present invention:

[0012] The present invention provides an adaptive routing protocol based on Q-Learning in a drone self-organizing network. In view of the problems of fast node speed, limited energy, frequent topology changes, etc. in the drone self-organizing network, it is proposed to use the Q-Learning algorithm to learn the optimal path to the destination node in a distributed manner, and comprehensively consider multiple indicators such as energy, mobility, bandwidth, etc. in terms of link quality. In addition, the adaptive HELLO interval is combined in the routing maintenance stage to improve the real-time performance of the Q table update. In summary, the present invention can be applied to the drone self-organizing network. It can dynamically adapt to changes in network topology, reduce data packet transmission delays, and increase network life. At the same time, the complexity of the present invention is relatively low. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of a UAV self-organizing network routing method based on Q-Learning in an embodiment of the present invention;

[0014] Figure 2 It is a flow chart of a UAV self-organizing network routing method based on Q-Learning in a preferred embodiment of the present invention;

[0015] Figure 3 It is a routing decision model based on Q-Learning in an embodiment of the present invention;

[0016] Figure 4 It is a schematic diagram of the UAV node maintenance Q in an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of the route discovery phase in an embodiment of the present invention;

[0018] Figure 6 is a relative movement model diagram of drone nodes in an embodiment of the present invention;

[0019] Figure 7 is a graph showing the relationship between the number of nodes and the packet delivery rate in an embodiment of the present invention;

[0020] Figure 8 is a graph showing the relationship between the number of nodes and the average end-to-end delay in an embodiment of the present invention;

[0021] Fig. 9 is a graph showing the relationship between the number of nodes and the network life in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Figure 1 A flow chart of a UAV self-organizing network routing method based on Q-Learning in an embodiment of the present invention is given. Figure 1 As shown, the method includes a route discovery phase and a route maintenance phase. In the route discovery phase and the route maintenance phase, a route decision model based on the Q-Learning algorithm is respectively called to obtain a route from a source node to a destination node. The route decision model based on the Q-Learning algorithm includes all drone nodes in the drone ad hoc network interacting with the drone ad hoc network environment through RREQ messages, RREP messages, and HELLO messages, and collecting status information and link-related information of each neighbor node; calculating the link quality according to the link quality evaluation model, bringing the link quality into the Q value function, and calculating the Q value of the drone node in combination with the status information of the neighbor node.

[0024] The routing method specifically includes a routing discovery phase of steps 101 to 103 and a routing maintenance phase of step 104:

[0025] 101. In the route discovery phase, before the source node and the destination node communicate for the first time in the drone ad hoc network, the route discovery is started and the Q tables of all drone nodes are initialized;

[0026] In an embodiment of the present invention, in the routing protocol proposed in the present invention, when two drone nodes communicate for the first time, they will first start the routing discovery process and update the Q table from the node in the network to the destination node. Initially, each Q table is initialized to 0.

[0027] 102. The source node broadcasts the RREQ message, and the drone node calculates the link quality based on the link evaluation model based on the received RREQ message, and updates the Q table of the source node based on the link quality, and uses the Q table to establish a reverse route from the source node to the destination node;

[0028] In an embodiment of the present invention, before broadcasting the RREQ message, the drone node will attach its own link-related information and the maximum Q value to the destination node to the RREQ message. That is to say, the RREQ message of the present invention has made certain adaptability improvements compared to the traditional RREQ message. The link-related information of the present invention is used by the next-hop node to evaluate the current link quality, and the maximum Q value is used to update the Q value of the next-hop node. During the RREQ propagation process, the node will update the Q value to the source node, that is, a reverse route from the source node to the destination node is established.

[0029] In an embodiment of the present invention, the reverse route from the source node to the destination node established by using the Q table actually starts from the source node, passes through the intermediate nodes and then to the final destination node. Therefore, in addition to the source node broadcasting the RREQ message, this process also requires the intermediate nodes to broadcast the RREQ message. These intermediate nodes need to receive the RREQ message from the source node and broadcast the RREQ message to their next-hop nodes. When the node receives the RREQ message, the current node will first extract the neighbor node information carried in the RREQ and calculate the current link quality lq(c,x), where c represents the node receiving the RREQ message, that is, the current node, and x represents the node sending the RREQ message. Node c will update the corresponding Q value according to the Q value function. The present invention introduces a link quality factor in the Q value function, as shown below:

[0030]

[0031] Among them, Q c (d,x) represents the Q value of drone node c selecting neighbor node x as the next hop node and forwarding the packet to the destination node d; ← represents the update symbol; α represents the learning rate; lq(c,x) represents the link quality between drone node c and drone node x; r represents the reward; γ represents the attenuation coefficient; It represents the expected maximum Q value from drone node x to destination node d, i.e. Q x (d,y) indicates the Q value of drone node x selecting neighbor node y as the next hop node and forwarding the packet to destination node d; N x Represents the set of neighbor nodes of drone node x.

[0032] Among them, the link quality evaluation model includes weighted summation of the energy factor, stability factor and bandwidth factor of the drone nodes, and the sum value is used as the link quality between the drone nodes, which is expressed as:

[0033] lq(c,x)=a1×EF(c,x)+a2×SF(c,x)+a3×BF(c,x)

[0034] Among them, lq(c,x) represents the link quality between UAV node c and UAV node x; EF(c,x) represents the energy factor between UAV node c and UAV node x, SF(c,x) represents the stability factor between UAV node c and UAV node x, BF(c,x) represents the bandwidth factor between UAV node c and UAV node x, a1+a2+a3=1, a1, a2 and a3 represent different weight factors respectively.

[0035] 103. After receiving the RREQ message, the destination node sends a RREP message to the source node, calculates the link quality according to the link evaluation model, updates the Q table of the destination node based on the link quality, and uses the Q table to establish a forward route from the source node to the destination node;

[0036] In an embodiment of the present invention, after the destination node receives the RREQ message, the node selects the neighbor node with the maximum Q value as the next hop, sends a RREP message to the source node, establishes a forward route from the source node to the destination node, and the nodes in the path update the Q table of the destination node according to the link-related information and the maximum Q value attached to the RREP message. Like the RREQ message, the present invention also makes certain adaptive improvements to the RREP message relative to the traditional RREP message, so that the message carries link-related information and the maximum Q value, which is convenient for scheduling the routing decision model based on the Q-Learning algorithm to complete the routing decision.

[0037] In an embodiment of the present invention, the forward route from the source node to the destination node established by using the Q table actually starts from the destination node, passes through the intermediate nodes and then to the final source node. Therefore, in addition to requiring the destination node to send a RREP message to the intermediate node, this process also requires the intermediate node to broadcast the RREP message. These intermediate nodes need to receive the RREP message from the destination node and broadcast the RREP message to their next-hop nodes. Similarly, when a node receives a RREP message, the current node will first extract the neighbor node information carried in the RREQ and calculate the current link quality lq(c,x), where c represents the node receiving the RREQ message, that is, the current node, and x represents the node sending the RREQ message. Node c will update the corresponding Q value according to the Q value function. The present invention introduces a link quality factor in the Q value function, as shown below:

[0038] 104. In the routing maintenance phase, the network topology change is sensed through HELLO messages, the Q table is dynamically updated according to the drone self-organizing network environment, and the forward route from the source node to the destination node is updated according to the Q table.

[0039] In an embodiment of the present invention, in order to avoid the problem of routing protocols frequently starting route discovery in a highly dynamic network, the present invention uses HELLO messages to dynamically update the Q table according to the network topology during the route maintenance phase, thereby achieving dynamic route switching, reducing the waste of resources caused by route reconstruction, and reducing data packet transmission delay.

[0040] In a preferred embodiment of the present invention, the present invention also sets the HELLO message to an adaptive HELLO message interval, that is, based on the network topology information, using link duration estimation, the link duration is calculated according to the relative motion state between the drone nodes, and the minimum link duration from each drone node to its neighbor node is calculated, and the HELLO message interval is determined according to the minimum link duration.

[0041] Figure 2 This is a flow chart of a UAV self-organizing network routing method based on Q-Learning in a preferred embodiment of the present invention. The present invention is applicable to the scenario of multi-UAV self-organizing network data transmission. The method includes several core steps, namely:

[0042] Step 1: Establish a routing decision model based on Q-Learning.

[0043] In the embodiment of the present invention, it is assumed that there are N drone nodes in the drone network, and each drone is deployed in the airspace of the same altitude to perform tasks, that is, each drone moves randomly in a 2D plane, and the collision problem of drones in the same plane is not considered. The drones are equipped with GPS, cameras, sensors, wireless communication interfaces and other equipment, and the communication distance is R. Each drone can use GPS to obtain its location information P i =(x i ,y i ).

[0044] The drone network is a dynamic network, and it is difficult to use simple rules to determine the packet forwarding strategy. In addition, frequent topology changes also require dynamic adjustment of the packet forwarding strategy. In the face of dynamic and complex problems, the present invention uses reinforcement learning to deal with the routing problem in FANET. The reinforcement learning algorithm attempts to find a strategy that maps the state of the system to the actions that the agent should take when these states occur. In reinforcement learning, the evaluation of the system can usually be carried out simultaneously with learning, which can well solve the packet forwarding problem of the dynamic drone self-organizing network.

[0045] Reinforcement learning relies on four core elements S, A, P and R, where S is the state set, A is the action set, P is the state transition probability, and R is the reward function. In FANET, the node forwards the data packet to the next hop node through the wireless link, which involves the probability problem of selecting the node among the neighboring nodes. Therefore, the routing problem in FANET is considered to be a Markov decision process (MDP) problem. Although the optimal state value can be calculated by using MDP, this method is not suitable for drone nodes due to the large amount of calculation. Therefore, the present invention adopts the Q-Learning algorithm to find the optimal behavior selection strategy for limited MDP.

[0046] The goal of the Q-Learning algorithm is to devise a strategy that maximizes the observed rewards during agent interactions. In order to solve the routing problem in FANET, the Q-Learning algorithm works in a distributed manner, and the state transfer equation is: Each drone in the network maintains a Q table, and the Q table is updated using state transfer as follows:

[0047] Q(s,a)←(1-α)×Q(s,a)+α×{r+γ×maxQ(s′,a′)}

[0048] Where α∈(0,1), γ∈(0,1). The learning rate α and the decay coefficient γ are important parameters of the Q-Learning algorithm. The learning rate parameter limits how fast learning occurs. It controls how fast the Q value changes with each state-action. The decay coefficient controls the impact of the estimate on the current state. If this value is low, more emphasis is placed on immediate rewards in the Q value update, while a larger decay coefficient causes the learning algorithm to focus more on future rewards.

[0049] In the network, each drone maintains a Q table, and each drone formulates a data packet forwarding strategy based on the Q table.

[0050] Step 2: Establish an inter-node link evaluation model.

[0051] In FANET, due to multiple factors such as poor wireless links, limited bandwidth, short routing life, and limited energy, the shortest path is not necessarily the best route. Therefore, in the UAV routing process, we must first consider how to evaluate the link quality. The present invention collects neighbor node status information through information exchange between nodes, and comprehensively considers indicators such as remaining energy, stability, and available bandwidth to establish a link evaluation model.

[0052] Step 3: In the route discovery phase, RREQ and RREP messages are used to update the Q table.

[0053] The routing protocol proposed in the present invention absorbs the advantages of traditional reactive routing protocols. Only when there is data to be sent in the network will the best route to the destination node in the network be found, which helps to reduce the waste of resources in the network. When the source node in the network communicates with the destination node for the first time, the Q table of each network node is 0, the route discovery process is started, and the first learning is performed using the broadcast RREQ message, and the best route to the destination node is created at the same time.

[0054] Step 4: During the routing maintenance phase, the Q table is updated using the HELLO message.

[0055] In traditional reactive routing protocols, when a route is broken, it is necessary to rebroadcast the RREQ message to reconstruct the route. Therefore, in order to avoid the problem of frequent route discovery initiated by the routing protocol in a highly dynamic network, the present invention uses the HELLO message to dynamically update the Q table according to the network topology during the route maintenance phase, realizes dynamic route switching, reduces the waste of resources caused by route reconstruction, and reduces the delay of data packet transmission.

[0056] Step 5: Use adaptive HELLO message interval to improve the real-time performance of Q table update.

[0057] In FANET, drones move at high speeds and node links are frequently broken, so the routing algorithm applied to FANET must be able to adapt to the frequent changes in the network topology of the nodes. The Q table update in the routing maintenance phase depends on the HELLO message. If the HELLO message is sent at a fixed interval, it will cause the lag of the Q table update, which will affect the transmission of data packets. In order to update the Q value faster and adapt to changes in network topology, the present invention uses an adaptive HELLO message interval. The sending interval of the HELLO message is dynamically updated according to the local environment of the drone, which helps to improve the robustness of the protocol. In addition, the adaptive HEELO message interval can also take into account the network overhead and neighbor information interaction.

[0058] Among them, the routing decision model based on the Q-Learning algorithm in the embodiment of the present invention is based on the Q-Learning algorithm idea and adopts distributed learning to model the routing problem of the UAV self-organizing network. Figure 3 As shown, it includes the following:

[0059] 1) Environment: The entire UAV network serves as the environment, which includes each mobile node, the links between nodes, and the packets transmitted in the network.

[0060] 2) Agent: The packets transmitted in the UAV network are considered as an agent, and the knowledge learned by the agent is stored in the corresponding UAV.

[0061] 3) Status: The node where the packet is located indicates the current status of the packet. All nodes in the network are the state space of the packet. Forwarding the packet from the current node to the next hop node means that the status of the packet has changed.

[0062] 4) Action: All one-hop neighbor nodes of a node are the action set of the node state.

[0063] In the drone network, the Q value of the destination node is updated through packet forwarding. Each node maintains a Q table, and each Q table maintains the Q value of each destination node, expressed as Q c (d, x), where d represents the destination node, x represents the next hop neighbor node to the destination node, and Qc (d,x) indicates that the packet is currently at node c and selects node x as the next hop node to reach the destination node d. Therefore, the size of each node's Q table is determined by the number of destination nodes and neighbor nodes. When a node selects a next hop node for a packet, it selects the next hop node corresponding to the highest Q value. Figure 4 As shown in the figure, the Q table of the source node S maintains the Q values ​​of the two next hop nodes N1 and N2 to the destination node D, which are 0.6 and 0.5 respectively. At this time, if the source node needs to send data to the destination node, the source node should select the neighbor node N1 with the largest Q value as the next hop. During the forwarding process, the relay node also selects the neighbor node with the largest Q value as the next hop until the data packet reaches the destination node D.

[0064] In the embodiment of the present invention, the link evaluation model between nodes can be calculated by the following factors, which are as follows:

[0065] 1) Energy Factor (EF). Small drones usually use batteries with limited energy for power supply and communication. The remaining energy of the node directly determines the survival time of the node. On the one hand, the exhaustion of node energy will cause node failure, resulting in communication link breakage, network packet loss and additional routing reconstruction overhead. On the other hand, the failure of a large number of drones will reduce the network life and affect task execution. In the process of route establishment, it is necessary to avoid the exhaustion of energy of some nodes as much as possible, which will reduce the network life. The energy factor is used to represent the remaining energy level of the node, which is defined as follows:

[0066]

[0067] Where E0 is the total initial communication energy of the node; E r The remaining energy for the drone.

[0068] 2) Stability Factor (SF). The faster the UAV self-organizing network node moves, the more drastic the network topology changes. Selecting nodes with relatively small movements to forward data can extend the link lifetime and improve the stability of the transmission path. Therefore, the stability factor of node x relative to node c at time i can be expressed as follows:

[0069]

[0070] Among them, d t (c,x) is the distance between the current node c and node x at time t, and R is the transmission range of the drone. In order to eliminate some errors, the exponential moving average is used to calculate the stability factor as follows:

[0071] SF(c,x)=(1-β)SFi-1 (c, x) + β(c, x)

[0072] Among them, SF i-1 (c, x) represents the stability factor of the UAV node x relative to the neighbor node c at time i - 1; SF i (c, x) represents the stability factor of the UAV node x relative to the neighbor node c at time i; β represents the control factor, which is actually a weight parameter used to balance the stability at the current moment and the previous moment. When |d i (c, x) - d i-1 (c, x)| < R, When |d i (c, x) - d i-1 (c, x)| ≥ R, SF i (c, x) = 0; d i (c, x) is the distance between the UAV node c and the neighbor node x at time i, d i-1 (c, x) is the distance between the UAV node c and the neighbor node x at time i - 1, and R is the maximum transmission range of the UAV. In this embodiment, β is set to 0.7 and SF is initialized to 0.

[0073] 3) Bandwidth Factor (BF). UAV nodes with better link quality have a higher probability of being selected as the next-hop forwarding node, which is likely to cause excessive traffic passing through this node and result in network congestion. Therefore, selecting UAVs with low or even idle traffic can effectively improve communication performance. The UAV obtains its own Channel Idle Time Ratio (CITR) from the MAC layer. After using weighted exponential moving average, the current Channel Idle Time Ratio is defined as follows:

[0074] CITR = (1 - δ) × CITR i-1 + δ × CITR i

[0075] Among them, CITR i-1 and CITR i represent the CITR value at the previous moment and the current CITR value respectively. The coefficient δ is set to 0.7.

[0076] Each node attaches its own CITR to the HELLO message and the RREQ message. After receiving the HELLO or RREQ message, the neighbor node calculates the bandwidth factor, and the bandwidth factor is calculated as follows:

[0077] BF(c, x) = min(CIRT(c), CITR(x))

[0078] Wherein, CIRT(c) represents the CITR at the current node c, and CITR(x) represents the CITR of the node x that sends the HELLO message or the RREQ message.

[0079] In summary, multiple factors in the network need to be considered comprehensively when selecting relay nodes. Therefore, the definition of link quality is as follows:

[0080] lq(c,x)=a1×EF(x)+a2×SF(c,x)+a3×BF(c,x)

[0081] Wherein, a1+a2+a3=1, a1, a2 and a3 represent different weight factors respectively; lq(c,x) represents the link quality between node c and node x.

[0082] It can be understood that this EF(x) refers to the energy of drone node x. Node x sends its own energy to node c through some control messages, and node c knows the energy consumption of node x at this time, which is expressed by the energy factor. Then the larger the energy factor, the smaller the consumption of node x at this time. Therefore, EF(x) represents the energy consumption of x. Node c uses these factors to evaluate whether x is suitable to be selected as the next hop for transmission.

[0083] In an embodiment of the present invention, the Q table can be updated by using RREQ and RREP messages in the route discovery phase. In the routing protocol proposed by the present invention, when two nodes communicate for the first time, they will first start the route discovery process and update the Q table from the node in the network to the destination node. When the two nodes communicate for the first time, they will generate data at the application layer and then reach the network layer. At this time, there is no routing information. They will cache the data first, find the route first, that is, perform the route discovery process, and then send specific data. Since the specific data transmission is not the focus of the present invention, the present invention does not make specific restrictions on this. Before broadcasting the RREQ message, the node attaches its own link-related information and the maximum Q value to the destination node in the RREQ. The link-related information is used by the next-hop node to evaluate the current link quality, and the maximum Q value is used to update the Q value of the next-hop node. During the RREQ propagation process, the node will update the Q value to the source node, that is, establish a reverse route from the source node to the destination node. After the destination node receives the RREQ message, the node selects the neighbor node with the maximum Q value as the next hop, sends a RREP message to the source node, establishes a forward route from the source node to the destination node, and the nodes in the path update the Q table of the destination node according to the link-related information and the maximum Q value attached to the RREP message.

[0084] Initially, each Q table is initialized to 0. When a node receives a RREQ message, the current node first extracts the neighbor node information carried in the RREQ and calculates the current link quality lq(c,x) according to step 2, where c represents the node receiving the RREQ message, that is, the current node, and x represents the node sending the RREQ message. Node c will update the corresponding Q value according to the Q value function. The present invention introduces a link quality factor in the Q value function, as shown below:

[0085]

[0086] Among them, the learning rate α and the decay coefficient γ are taken as 0.8 and 0.9 respectively according to the empirical values, Q c (d,x) indicates the Q value of node c selecting neighbor node x as the next hop node and forwarding the packet to destination node d. It is transmitted through RREQ message, indicating the expected maximum Q value from node x to destination node d. x represents the one-hop neighbor node of node x. r represents the reward, which is defined as follows:

[0087]

[0088] Among them, N d represents the neighbor set of node d. When node c is a neighbor of node d, the reward is 1, otherwise it is 0.

[0089] The specific process of the routing discovery phase is as follows Figure 5 As shown in the figure, the source node first broadcasts the RREQ message. When receiving the RREQ message from the source node S, the node N3 first calculates the link quality lq(N3,S) between the nodes, assuming it is 0.9. Because this is the reverse route establishment process from the source node to the destination node, the node S is the destination address in the Q value function, and the reward is 1. Therefore, Q(S,S) is 0.72. N4 performs the same calculation after receiving the RREQ message from N3. After the destination node D receives the RREQ message, it updates its Q table. At this time, the reverse route from the source node to the destination node has been established in the network. The destination node D establishes a forward route by sending a RREP message to the source node. The RREP message also contains the relevant information of this node and the maximum Q value to the destination node. At this time, the neighbor node N4 with the maximum Q value will be selected as the next hop node, and N4 will update the Q value Q(D,D) to the destination node D; after the intermediate node receives the RREP data packet, each node uses the same method to select the next hop and updates the Q value to the destination node D until it reaches the source node. The path that the RREP data packet passes through will be the optimal data transmission route for the current network status. At this time, the optimal route "S->N3->N4->D" for the current network status is established.

[0090] In a preferred embodiment of the present invention, the Q table is updated during the routing maintenance phase using HELLO messages. During the routing maintenance phase, network topology changes are sensed through HELLO messages, and the Q table is dynamically updated according to the environment. In the HELLO message, the node status information and the maximum Q value to the destination node are added. After receiving the HELLO message, the node updates the Q table in a similar manner to the Q table update process during the routing establishment phase. First, the link quality is calculated, and then the corresponding Q value is updated according to the Q value function to complete the Q table update. Therefore, if there is a destination node d in the network, the remaining nodes maintain the Q value of each neighbor node as the next hop. Therefore, when node c needs to send or forward a packet to node d, it only needs to check its own Q table and select the Q value that is the next hop. c The largest x in (d,x) is used as the next hop node.

[0091] In a preferred embodiment of the present invention, the present invention also uses an adaptive HELLO message interval to improve the real-time performance of the Q table update. The present invention first uses link duration estimation based on topology information, and the HELLO interval and link maintenance duration are automatically adjusted according to the value of the link duration. After obtaining the location information of the node itself and its neighbors, the node can estimate the link duration.

[0092] like Figure 6 As shown, the drone node U i and U j are their respective one-hop neighbors, and the transmission distance of the drone is R. i and U j The relative distance at t1 is d ij (t1), at t2, the drone U i Received UAV U j The HELLO message sent.

[0093] First, the relative distance d between two nodes at time t is calculated according to the Euclidean distance formula ij (t), as follows:

[0094]

[0095] Calculate the node U at time t1 and t2, i.e., time period (t2-t1) i and U j The relative displacement between them is as follows:

[0096] Δd=d ij (t2)-d ij (t1)

[0097] When Δd is a positive value, it means that the two drone nodes are moving away from each other during this time period; otherwise, it means that the drone nodes are moving closer to each other.

[0098] The link duration between nodes can be estimated based on the relative motion state of the two drones as follows:

[0099]

[0100] Among them, Δd / (t2-t1) represents the relative speed of the two UAV nodes in this time period. V min Indicates the minimum transmission rate of the drone node.

[0101] Each drone node measures the minimum link duration of all its neighboring nodes. Based on the neighbor node link duration, the HELLO interval is dynamically set according to the minimum link duration.

[0102]

[0103] Among them, neighbor(i) represents the neighbor node set of drone node i.

[0104] HELLO interval =β×LD min ,0<β<1

[0105] Here, β is an empirical value, which may be set to 0.8 by default in this embodiment. Of course, this empirical value may be set by a person skilled in the art.

[0106] Figure 7 The relationship between the number of nodes in the network and the packet delivery rate is given, and the adaptive routing protocol based on Q-Learning proposed in the present invention is compared with the traditional AODV routing protocol. It can be seen from the figure that compared with the AODV routing protocol, the routing protocol proposed in the present invention has a significant improvement in packet delivery rate. Initially, due to the low density of drone nodes and frequent network disconnections, the packet delivery rates of both protocols were low. As the number of drone nodes increases and the network connectivity increases, the protocol proposed in the present invention has a greater improvement in packet transmission rate. This is because the protocol proposed in the present invention comprehensively considers multiple factors that affect communication performance. At the same time, the Q-Learning method is used in the routing maintenance stage to achieve dynamic switching of routes and reduce data packet loss caused by routing reconstruction.

[0107] Figure 8The relationship between the number of nodes in the network and the average end-to-end delay is given. The routing protocol proposed in the present invention has a significant improvement in end-to-end delay compared to the AODV protocol. This is because the Q-Learning algorithm is introduced in the present invention. The Q table mechanism is updated through control messages in the network, so that the routing can dynamically adapt to changes in network topology and reduce the delay caused by route reconstruction. In addition, the stability of the drone nodes is considered in the path evaluation, so that a more stable path is selected. When the number of nodes is small, the average end-to-end delay is lower. This is because the average end-to-end delay does not calculate the delay of lost data packets. When the density of drone nodes is low, only data packets with a closer distance or fewer hops can be transmitted normally, so the average delay at this time is lower.

[0108] Fig. 9 The relationship between the number of nodes in the network and the network life is given. Since the routing protocol proposed in the present invention takes the remaining energy of the node into consideration when selecting the path, the problem of energy exhaustion of a single node in the network can be avoided as much as possible. At the same time, in the routing maintenance stage, the present invention uses the Q-Learning algorithm to realize dynamic routing switching, reducing the energy consumption caused by broadcasting RREQ. In addition, the adaptive HELLO interval also reduces the redundant messages sent by the node. Therefore, compared with AODV, the present invention significantly improves the network life.

[0109] The present invention studies the routing protocol in the unmanned aerial vehicle self-organizing network. In the unmanned aerial vehicle self-organizing network, each node acts as a router, and the communication of the unmanned aerial vehicles in the network depends on multi-hop transmission. How to select the best forwarding path is the main problem in the routing protocol. The present invention proposes an adaptive routing method based on Q-Learning. First, the link quality evaluation problem in the network is considered; then the Q-Learning algorithm is introduced in the routing discovery and maintenance stage to realize dynamic routing switching; finally, the real-time performance of the Q table update is improved through the adaptive HELLO interval in the routing maintenance stage. Simulation results show that the present invention has a higher packet delivery rate and lower latency, while also ensuring a good network life.

[0110] In the description of the present invention, it is necessary to understand that the terms "coaxial", "bottom", "one end", "top", "middle", "the other end", "upper", "one side", "top", "inside", "outside", "front", "center", "both ends" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0111] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0112] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A UAV self-organizing network routing method based on Q-Learning, It is characterized in that The method includes a route discovery phase and a route maintenance phase. In the route discovery phase and the route maintenance phase, a route decision model based on a Q-Learning algorithm is respectively called to obtain a route from a source node to a destination node, specifically including: In the route discovery phase, before the first communication between the source node and the destination node in the UAV ad hoc network, route discovery is started and the Q tables of all UAV nodes are initialized; The source node broadcasts the RREQ message, and the drone node calculates the link quality based on the link evaluation model based on the received RREQ message, and updates the Q table of the source node based on the link quality, and uses the Q table to establish a reverse route from the source node to the destination node; The link quality evaluation model includes weighted summation of the energy factor, stability factor and bandwidth factor of the drone nodes, and the sum value is used as the link quality between the drone nodes, which is expressed as: lq(c,x)=a 1 ×EF(x)+a 2 ×SF(c,x)+a 3 ×BF(c,x) Where lq(c,x) represents the link quality between drone node c and drone node x; EF(x) represents the energy factor of drone node x, SF(c,x) represents the stability factor between drone node c and drone node x, BF(c,x) represents the bandwidth factor between drone node c and drone node x, and a 1 +a 2 +a 3 =1,a 1 、a 2 and a 3 In turn, they represent different weight factors; The calculation formula of the stability factor of the drone node is expressed as: SF(c,x)=(1-β)SF i-1 (c,x)+βSF i (c,x) Among them, SF i-1 (c, x) represents the stability factor of the UAV node x relative to the neighbor node c at time i - 1; SF i (c, x) represents the stability factor of the UAV node x relative to the neighbor node c at time i; β represents the control factor. When |d i (c, x) - d i-1 (c, x)| < R, When |d i (c, x) - d i-1 (c, x)| ≥ R, SF i (c, x) = 0; d i (c, x) is the distance between the UAV node c and the neighbor node x at time i, and d i-1 (c, x) is the distance between the UAV node c and the neighbor node x at time i - 1, and R is the maximum transmission range of the UAV; After receiving the RREQ message, the destination node sends a RREP message to the source node, calculates the link quality according to the link evaluation model, updates the Q table to the destination node based on the link quality, and uses the Q table to establish a forward route from the source node to the destination node; In the routing maintenance phase, the network topology change is perceived through HELLO messages, the Q table is dynamically updated according to the UAV ad hoc network environment, and the forward route from the source node to the destination node is updated according to the Q table; Among them, the RREQ message, the RREP message and the HELLO message are attached with link-related information of the drone node and the maximum Q value to the destination node, and the link-related information includes remaining energy, node location and channel idle time ratio.

2. According to claim 1, a UAV self-organizing network routing method based on Q-Learning, It is characterized in that The routing decision model based on the Q-Learning algorithm regards the packets in the UAV self-organizing network as intelligent agents, and the transmission of packets from one UAV to another UAV as the transition of the intelligent agent state, and maintains a Q table on each UAV; therefore, in the UAV self-organizing network, the environment is explored through routing control messages including at least RREQ messages, RREP messages, and HELLO messages, and the status information and link-related information of each neighbor node are collected; the link quality is calculated according to the link quality evaluation model, the link quality is brought into the Q value function, and the Q value of the UAV node is calculated in combination with the status information of the neighbor nodes; When forwarding data packets in the network, the Q table maintained by the drone node is used to provide a forwarding strategy.

3. A UAV self-organizing network routing method based on Q-Learning according to claim 1 or 2, It is characterized in that The calculation formula of the Q table updated based on the link quality is expressed as: Among them, Q c (d,x) represents the Q value of drone node c selecting neighbor node x as the next hop node and forwarding the packet to the destination node d; ← represents the update symbol; α represents the learning rate; lq(c,x) represents the link quality between drone node c and drone node x; r represents the reward; γ represents the attenuation coefficient; It represents the expected maximum Q value from drone node x to destination node d, i.e. Q x (d,y) indicates the Q value of drone node x selecting neighbor node y as the next hop node and forwarding the packet to destination node d; N x Represents the set of neighbor nodes of drone node x.

4. According to claim 1, a UAV self-organizing network routing method based on Q-Learning, It is characterized in that The method of perceiving network topology changes through HELLO messages, dynamically updating the Q table according to the drone self-organizing network environment, and updating the route from the source node to the destination node according to the Q table includes that after the drone node receives the HELLO message, the drone node calculates the link quality according to the link evaluation model, and updates the Q table of the source node based on the link quality, and uses the Q table to establish the route from the source node to the destination node.

5. A UAV self-organizing network routing method based on Q-Learning according to claim 1 or 4, It is characterized in that The HELLO message is an adaptive HELLO message interval, that is, based on network topology information, using link duration estimation, calculating the link duration according to the relative motion state between drone nodes, calculating the minimum link duration from each drone node to its neighbor node, and determining the HELLO message interval according to the minimum link duration.

6. A UAV self-organizing network routing method based on Q-Learning according to claim 5, It is characterized in that The calculation formula of link duration is expressed as: Where LD(i,j) represents the link duration between drone node i and drone node j, R represents the maximum transmission distance of the drone, and d ij (t 2 ) indicates that at time t 2 The relative distance between drone node i and drone node j, Δd / (t 2 -t 1 ) represents the two UAV nodes in the time period (t 2 -t 1 ) relative speed, Δd represents the relative speed in the time period (t 2 -t 1 )’s relative displacement between drone nodes, V min Indicates the minimum transmission rate of the drone node.

Images