Unmanned aerial vehicle time delay and position optimization method for air-ground heterogeneous network

Abstract

The invention provides an unmanned aerial vehicle time delay and position optimization method for an air-ground heterogeneous network. The method comprises the following steps: calculating a signal to interference plus noise ratio between unmanned aerial vehicles, and calculating transmission time delay and queuing time delay of the unmanned aerial vehicles according to the data packet size, the information propagation speed, the number of to-be-processed data packets of the unmanned aerial vehicles and the data processing speed of the unmanned aerial vehicles; calculating the domain degrees of all the previous-hop unmanned aerial vehicles generating the congested public relay unmanned aerial vehicles, and taking the unmanned aerial vehicle with the maximum adaptive value as a scheduling unmanned aerial vehicle; the method comprises the following steps: modeling a position optimization problem of a dispatched unmanned aerial vehicle as an MDP model, solving the MDP model through a DDQN-based unmanned aerial vehicle position and time delay optimization algorithm by using a state space, an action space and a reward function, and obtaining a dispatched unmanned aerial vehicle position capable of realizing the lowest report loss rate and system time delay. According to the method, the maneuverability of the unmanned aerial vehicle in the network can be fully utilized, the queuing time delay and the transmission time delay in the network are reduced, the data transmission quality can be ensured, and the packet loss rate is stabilized below 8%.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a method for optimizing the delay and position of a drone in an air-to-ground heterogeneous network. Background Art

[0002] With the continuous development of drone technology, the air-ground heterogeneous network built based on drone self-organizing network technology can meet the diversified, high-capacity and low-latency network communication needs in the integrated computing scenario. However, with the increase of devices in the network and the continuous growth of network traffic, the data of multiple source nodes in the air-ground heterogeneous network often pass through one or more public relay drones during transmission, which increases the queuing delay of information transmission in the network, and even causes information loss in severe cases. Therefore, how to effectively deploy the position of relay drones in the air-ground heterogeneous network, reduce the queuing delay and transmission delay of information transmission in the air-ground heterogeneous network, and improve the effectiveness of the network and the efficiency of information transmission is crucial to the performance and stability of the air-ground heterogeneous network.

[0003] At present, some research has been conducted on delay optimization methods for relay drones. For the optimization method of queuing delay in the network, a scheme introduces a distributed time division multiple access method to dynamically allocate time slots by monitoring the data transmission time slots, improve time slot utilization and reduce network delay. In view of the serious channel conflicts and poor access reliability of services with different priorities in the current SPMA (Statistic Priority-based Multiple Access) protocol, a scheme proposes a dynamic SPMA protocol based on Q-learning, which dynamically adjusts the threshold and updates the backoff time to provide stable and low-latency communication services for drones under different service quality requirements. Another scheme proposes a new multi-priority queuing mechanism and designs a priority queuing scheme for different priority types to meet the high reliability and low-latency transmission of the highest priority service under limited channel resources. The above methods only consider the optimization of queuing delay in the network, and do not consider the transmission path and transmission delay of data in the network. To this end, there are also solutions that further consider the network architecture and design a drone network architecture based on software-defined networks. A directed particle swarm optimization method is used to optimize the transmission of data in the network, thereby improving the network throughput and packet delivery rate and reducing the transmission delay of data in the network. There are also solutions that consider the problem of low network communication efficiency caused by the high mobility of relay drones, cluster the drone network, and improve the stability of the clusters. By reducing the probability of data link interruption in the network, the network delay is reduced. There are also solutions that use the DQN (Deep Q-Network) algorithm based on the user reference signal receiving power to solve the downlink throughput optimization problem of system users under the condition of co-frequency interference in the ground network, reducing the delay and packet loss rate of data transmission in the network. There are also solutions that design a hole node detection mechanism to address the problems of short link survival time between nodes and the high number of routing holes encountered by nodes in high-dynamic drone self-organizing networks. By avoiding the appearance of hole areas during information transmission, communication interruptions in the network are reduced, and the end-to-end delay of information transmission in the network is reduced.

[0004] At present, the existing technology reduces the queuing delay and transmission delay in the network by allocating channels, setting data transmission priorities, or selecting a data transmission path with low transmission delay or queuing delay. The disadvantages of this method include: it does not take into account the coordination of relay drones in the air-ground heterogeneous network and the data loss problem caused by the excessive load of public relay drones in the network. Summary of the invention

[0005] An embodiment of the present invention provides a method for optimizing the latency and position of a UAV in an air-ground heterogeneous network, so as to effectively reduce the data transmission latency in the air-ground heterogeneous network.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme.

[0007] A method for optimizing the delay and position of unmanned aerial vehicles in an air-ground heterogeneous network, comprising:

[0008] Calculate the signal-to-interference-to-noise ratio between drones, and calculate the transmission delay and queuing delay of drones based on the packet size, information propagation speed, the number of drone packets to be processed, and the drone data processing speed;

[0009] Calculate the territoriality of all previous-hop drones that generate the congested public relay drone. Use the fuzzy logic method to calculate the drone with the largest fitness value among the previous-hop drones based on the territoriality, signal-to-interference-noise ratio, and transmission delay between drones. Use the drone with the largest fitness value as the dispatching drone.

[0010] The position optimization problem of scheduling the UAV is modeled as a Markov decision process MDP model, the state space is set according to the UAV position and the amount of data to be processed, the three-dimensional spatial position variable of the UAV is set as the action space, and the reward function is set according to the signal-to-interference-to-noise ratio, the queuing delay and the transmission delay;

[0011] By using the state space, action space and reward function, the MDP model is solved through the DDQN-based drone position and delay optimization algorithm to obtain the scheduling drone position that can achieve the lowest report loss rate and system delay.

[0012] Preferably, the signal-to-interference-to-noise ratio between drones is calculated, and the transmission delay and queuing delay of the drones are calculated according to the data packet size, information propagation speed, the number of data packets to be processed by the drones, and the data processing speed of the drones, including:

[0013] Set up an air-ground heterogeneous UAV self-organizing network based on UAV self-organizing network, including N UAVs and M ground base stations, each UAV has a unique ID n , S i represents the source node in the network, D j Represents the ground base station in the network, UAV p Represents a public relay drone, UAV in the network F Indicates the idle relay drones in the network, with (x i ,y i ,z i )、(x D ,y D ,z D )、(x s ,y s ,z s) represent the spatial position of the UAV, the position of the ground base station and the spatial position of the source node respectively. The network includes two communication modes: UAV-to-UAV communication U2U and UAV-to-ground base station communication U2G;

[0014] UAV i and u j The signal to interference plus noise ratio SINR is defined as:

[0015]

[0016] The channel power gain between two UAVs is expressed as Where g is the channel power gain when the reference distance is 1m, D i,j is the current distance between the two UAVs, represents the path loss index, σ 2 represents the environmental noise, and the interference of the neighboring drone to the communication link is expressed as Where ρ represents the drone u j The proportion of information sent by other previous hop nodes, U j Indicates drone u j The set of all previous hop drones;

[0017] The single-hop delay in the air-ground heterogeneous network s Defined as:

[0018] Delay s =t ACK -t send (3)

[0019] t ACK and t send Respectively i Receive u j The returned ACK confirms the character data and u i The time when the packet is sent.

[0020] Transmission delay p Defined as:

[0021]

[0022] V p is the propagation speed of electromagnetic waves in free space;

[0023] Queue delay q Defined as:

[0024]

[0025] Where P t is the number of packets to be processed, qt is the speed at which the current public relay drone processes data packets.

[0026] The total delay in the air-ground heterogeneous network is expressed as:

[0027] Delay=Delay s +Delay p +Delay q (6).

[0028] Preferably, the calculation generates the territoriality of all previous-hop drones that congest the public relay drone, and uses a fuzzy logic method to calculate the drone with the largest fitness value among the previous-hop drones according to the territoriality, the signal-to-interference-noise ratio and the transmission delay between the drones, and uses the drone with the largest fitness value as the dispatching drone, including:

[0029] The Mamdani fuzzy inference mechanism is used to design a fuzzy system in which the previous hop drone of the public relay drone is a UAV. last , D norm To be able to work with UAV last The number of one-hop drones that perform normal communication is calculated using formula (14) norm Perform normalization processing;

[0030]

[0031] D k For UAV last The neighborhood degree of one of the neighboring drones, D max and D min For UAV last The points with the largest and smallest neighborhood degrees in the neighbor drone set.

[0032] UAV last SINR between the drones and their neighbors norm Defined as:

[0033]

[0034] SINR last,k For UAV last The SINR between its kth neighbor drone, SINR min and SINR max UAV last The minimum and maximum SINR among all neighboring drones;

[0035] UAV last The transmission delay between the drone and its neighbor p-norm It is expressed as:

[0036]

[0037] Delay p-last,k For UAV last Delay between its kth neighbor drone p , Delay p-min and Delay p-max UAV last The smallest delay among all neighbor drones p and the maximum Delay p ;

[0038] According to D norm 、SINR norm and Delay p-norm Using fuzzy system to calculate UAV last The fuzzy value F of each neighboring drone k ,k=(1,2,...K), K is UAV last The number of reachable neighbor drones is selected with the largest F k The drone is used as a dispatch drone;

[0039] In the fuzzy system, the rules of the fuzzy system are set according to the fuzzy input and fuzzy output. According to the input and output conditions, there are 27 rules. The specific rules are shown in Table 1. The fuzzy module converts the original input into fuzzy input and determines the membership of each fuzzy module. Then the fuzzy engine calculates the fuzzy value of the input, recombines it through the operation of rules and fuzzy logic, and finally uses the weighted average method to defuzzify, that is:

[0040]

[0041] f i is the fuzzy output of the previous step, w i represents the corresponding weight;

[0042] Table 1

[0043] Table 4-1 Rules of fuzzy system

[0044]

[0045] Preferably, the location optimization problem of scheduling the drone is modeled as a Markov decision process MDP model, the state space is set according to the drone position and the amount of data to be processed, the three-dimensional spatial position variable of the drone is set as the action space, and the reward function is set according to the signal-to-interference-to-noise ratio, queuing delay and transmission delay, including:

[0046] The location optimization problem of scheduling drones is modeled as an MDP model, and the MDP model is defined using a five-tuple (S, A, P, R, π), where S represents the state space, A represents the action space, P represents the state transition probability, R represents the reward function for the current state to enter the next state after an action, and π represents the strategy of the agent to select actions. The process of optimizing the location of drones in the air-ground heterogeneous network is to select action a(t) and obtain reward r(t) under the current system state s(t), and then enter the next state s(t+1);

[0047] The state space S(t) is expressed as:

[0048]

[0049] L p (t) = [(x p ,y p ,z p ) t ] T represents the 3D spatial position of the public relay drone at step t, L last (t) represents the position of the last hop drone at step t, L d represents the location of the dispatched UAV at step t, L nest represents the position of the next hop drone at step t, represents the amount of data to be processed by the public relay drone at step t, It represents the amount of data to be processed by the previous drone at step t. When the dispatched drone selects action a(t) in state s(t), it will enter the next state s(t+1);

[0050] The flight speed of the dispatched drone within the decision time t is set to v, and the position change caused by an action is c = v*t. The action space of the drone is expressed as:

[0051]

[0052] Where [c,0,0] T , [-c,0,0] T , [0,c,0] T , [0,-c,0] T , [0,0,c] T , [0,0,-c] T Respectively represent the distances that the dispatched drone moves in the three-dimensional space along the x-axis direction, y-axis direction, z-axis direction, negative x-axis direction, negative y-axis direction, and negative z-axis direction, [0,0,0] T Indicates that the dispatched drone does not move, and the action a(t) of the dispatched drone is restricted by the state s(t) of each step;

[0053] The reward function is designed based on the SINR, queuing delay and transmission delay between drones. The total reward function is expressed as:

[0054]

[0055] Among them, r SINR (t) is the reward function related to the SINR of all communication links of the scheduled UAV, To schedule the reward function related to the delay of receiving and transmitting data to the UAV, is the reward function related to the queuing delay of the public relay drone, and w1, w2, and w3 are the weight values ​​of the above three reward functions respectively.

[0056] Preferably, the method of using the state space, action space and reward function to solve the MDP model through a DDQN-based drone position and delay optimization algorithm to obtain a dispatched drone position that can achieve the lowest report loss rate and system delay includes:

[0057] Set up a DDQN-based drone position and delay optimization algorithm. The DDQN uses a decoupling method to separate the selection action and the evaluation action, designs the current state strategy and the current state value evaluation as two value functions, and randomly updates the two value functions. - is the parameter of the current state policy value function, θ is the parameter of the evaluation value function, and the objective function of DDQN is expressed as:

[0058]

[0059] The probability of random action selection ε is expressed as:

[0060]

[0061] Where I is the number of iterations of the algorithm;

[0062] The input data of the relay drone position and delay optimization algorithm based on DDQN are the position of the public relay drone, the position of the previous hop drone of the public relay drone, the position of the next hop drone of the dispatch drone, the position of the dispatch drone, the amount of data to be processed by the public relay node and the amount of data to be processed by the previous hop drone node of the public relay drone. The state space is composed of the above five variables, and the action space is the three-dimensional spatial position of the drone. The drone communication link signal-to-noise ratio reward function, the dispatch drone receiving and transmitting data delay reward function and the public relay drone queuing delay reward function are respectively constructed, and the total reward function is composed of a weighted sum. The relay drone position and delay optimization algorithm based on DDQN is trained by the objective function of formula (16), and the dispatch drone position is output;

[0063] The processing flow of the DDQN-based relay drone position and delay optimization algorithm is as follows:

[0064]

[0065]

[0066] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the method of the present invention can make full use of the maneuverability of drones in the network, while reducing the queuing delay and transmission delay in the network and reducing the packet loss rate.

[0067] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0069] Figure 1 A processing flow chart of a method for optimizing the delay and position of a UAV in an air-ground heterogeneous network provided by an embodiment of the present invention;

[0070] Figure 2 A structural diagram of an air-to-ground heterogeneous UAV self-organizing network provided by an embodiment of the present invention;

[0071] Figure 3 A high, medium and low level fuzzy membership graph provided by an embodiment of the present invention;

[0072] Figure 4 A DDQN-based relay drone position and delay optimization algorithm framework diagram is provided for an embodiment of the present invention.

[0073] Figure 5 It is a schematic diagram comparing the changes in Q values ​​of a DQN algorithm and a DDQN algorithm during a training process provided by an embodiment of the present invention.

[0074] Figure 6 A three-dimensional position map of a drone in an initial state and three different network conditions provided by an embodiment of the present invention;

[0075] Figure 7 A schematic diagram of optimizing the queuing delay of data in a network using different algorithms provided in an embodiment of the present invention.

[0076] Figure 8 A schematic diagram of optimizing data transmission delay using different algorithms provided in an embodiment of the present invention.

[0077] Fig. 9 The figure is a schematic diagram of optimizing data retransmission rate and packet loss rate using different algorithms provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0079] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0080] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0081] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0082] The embodiment of the present invention proposes a method for optimizing the delay and position of drones in air-ground heterogeneous networks. Idle drones with high neighborhood degree, high signal-to-noise ratio and low delay are selected as scheduling drones. A double deep Q-network (DDQN) algorithm is used to optimize the position of the scheduling drones so that they share the transmission tasks of public relay drones.

[0083] The processing flow chart of a method for optimizing the delay and position of a UAV in an air-ground heterogeneous network provided by an embodiment of the present invention is as follows: Figure 1 As shown, the process includes the following steps:

[0084] Step S10: Calculate the signal-to-interference-to-noise ratio between drones, and calculate the transmission delay and queuing delay of the drones based on the data packet size, information propagation speed, the number of data packets to be processed by the drones, and the data processing speed of the drones.

[0085] Step S20: Calculate the domain degree of all previous-hop drones that generate the congested public relay drone. According to the domain degree, the signal-to-interference-noise ratio and the transmission delay between drones, use the fuzzy logic method to calculate the drone with the largest fitness value among the previous-hop drones, and use the drone with the largest fitness value as the scheduling drone.

[0086] Step S30: Model the location optimization problem of scheduling drones as a Markov decision process MDP model, set the state space according to the drone position and the amount of data to be processed, set the drone's three-dimensional spatial position variable as the action space, and set the reward function according to the signal-to-interference-noise ratio, queuing delay, and transmission delay.

[0087] Step S40: Using the state space, action space and reward function, the MDP model is solved by the DDQN-based drone position and delay optimization algorithm to obtain the scheduling drone position that can achieve the lowest report loss rate and system delay.

[0088] The above step S10 includes: the structure of an air-to-ground heterogeneous UAV self-organizing network proposed in the embodiment of the present invention is as follows Figure 2 As shown, there are N drones and M ground base stations in the network, and each drone has a unique ID. n ,The motion of all UAVs is three ,dimensional.,All UAVs in the network have the same initial energy, processing ,capability, communication range and memory.,There are multiple source nodes, relay UAVs and public relay UAVs in the network.

[0089] Use S i To represent the source node in the network, use D j To represent the ground base stations in the network, UAV p To represent public relay drones, UAV FTo indicate an idle relay drone. i ,y i ,z i )、(x D ,y D ,z D )、(x s ,y s ,z s ) represent the spatial position of the UAV, the position of the ground base station and the spatial position of the source node respectively. There are two main communication modes in the network, namely UAV-to-UAV communication (U2U) and UAV-to-ground base station communication (U2G).

[0090] There are multiple source nodes in the network that communicate with the ground base station at the same time, and at least one or a small number of drones in the network use U2G communication with the ground base station. When multiple source nodes transmit information, there is a situation where the queuing delay of the public relay node is large. In order to solve this problem, the embodiment of the present invention uses a dispatching drone to share the transmission pressure of the public relay drone, and optimizes the position of the dispatching drone, thereby reducing the delay of data transmission in the network while ensuring the quality and efficiency of network information transmission.

[0091] Since drones fly at a high altitude and are not easily blocked by obstacles, the present invention only considers line-of-sight transmission between drones and adopts a free space loss model. i and u j The SINR (Signal to Interference plus Noise Ratio) between is defined as:

[0092]

[0093] The channel power gain between two UAVs can be expressed as Where g is the channel power gain when the reference distance is 1m, D i,j is the current distance between the two UAVs, representing the path loss index. 2 represents the environmental noise, and the interference of the neighboring drone to the communication link can be expressed as Where ρ represents the drone u j The proportion of information sent by other previous hop nodes, U j Indicates drone u j The set of all previous hop drones. i and u j The SINR between can be expressed as:

[0094]

[0095] In the air-ground heterogeneous network based on UAV ad hoc network, the delay is mainly composed of the single-hop delay Delay s , Queue delay q and transmission delay p Composition, single-hop delay Delay s It can be defined as:

[0096] Delay s =t ACK -t send (3)

[0097] t ACK and t send Respectively i Receive u j The returned ACK (Acknowledgement) data and u i The time when the packet is sent.

[0098] Transmission delay p It can be defined as:

[0099]

[0100] V p is the propagation speed of electromagnetic waves in free space.

[0101] When multiple source nodes transmit information in an air-ground heterogeneous network, data packets queue up on the public relay drone for processing, which will cause queuing delay. The heavier the load of the public relay drone, that is, the longer the length of the data packet waiting to be sent by the public relay drone, the longer the queuing delay will be, and data loss may also occur. q Defined as:

[0102]

[0103] Where P t is the number of packets to be processed, q t is the speed at which the current public relay drone processes data packets.

[0104] The total delay of the network is expressed as:

[0105] Delay=Delay s +Delay p +Delay q (6)

[0106] The above step S20 includes: In order to reduce the queuing delay in the network and ensure the quality of data transmission, a scheduling drone selection algorithm is designed based on fuzzy logic. The neighborhood degree, SINR and transmission delay of the drone are used as the input fuzzy set, and the fuzzy logic is used for weight processing, and the drone with the largest fuzzy value is selected as the scheduling drone.

[0107] In the selection process of dispatching UAVs, a fuzzy system is used to calculate the UAV on the previous hop of the public relay UAV. last The fuzzy value F of each neighboring drone k ,k=(1,2,...K), K is UAV last The number of reachable neighboring drones. The fuzzy system is designed using the Mamdani fuzzy inference mechanism. last The neighbor drone neighborhood degree D norm , SINR norm And its transmission delay Delay p-norm composition.

[0108] D norm It can be used with UAV last The number of one-hop drones that perform normal communication. The larger the neighborhood degree, the smaller the probability of disconnection. Therefore, the neighborhood degree of drones is a very important factor in the selection of scheduling drones. Formula (14) is used to normalize it.

[0109]

[0110] D k For UAV last The neighborhood degree of one of the neighboring drones, D max and D min For UAV last The points with the largest and smallest neighborhood degrees in the neighbor drone set.

[0111] UAV last SINR between the drones and their neighbors norm The choice of dispatching drones is equally important and directly determines the communication quality of the network. It is defined as:

[0112]

[0113] SINR last,k For UAV last The SINR between its kth neighbor drone, SINR min and SINR max UAV last The minimum and maximum SINR among all neighboring drones.

[0114] Transmission delay p-norm It can be normalized as:

[0115]

[0116] Delay p-last,k For UAV last Delay between its kth neighbor drone p , Delay p-min and Delay p-max UAV last The smallest delay among all neighbor drones p and the maximum Delay p .

[0117] D norm 、SINR norm and Delay p-norm The same fuzzy membership input method consists of three modes, including low, medium and high. The corresponding fuzzy membership diagram is as follows Figure 3 shown.

[0118] In the fuzzy system, the rules of the fuzzy system are set according to the fuzzy input and fuzzy output. According to the input and output conditions, there are 27 rules. The specific rules are shown in Table 4-1. The fuzzy module converts the original input into fuzzy input and determines the membership of each fuzzy module. Then the fuzzy engine calculates the fuzzy value of the input, recombines it through the operation of rules and fuzzy logic, and finally uses the weighted average method for defuzzification, that is:

[0119]

[0120] f i is the fuzzy output of the previous step, w i Indicates the corresponding weight;

[0121] Table 4-1 Rules of fuzzy system

[0122]

[0123] In the scheduling drone selection algorithm, the queuing delay of the public relay drone is first calculated according to formula (7). When the queuing delay is greater than the threshold, it means that the public relay drone is heavily loaded, and data transmission may be slow or even lost. At this time, an alarm message is sent to all previous-hop drones, and the previous-hop drone obtains the status of all one-hop neighbor drones except the public relay drone, and calculates D norm 、SINR norm and Delay p-norm, based on fuzzy logic calculation F k , select the one with the largest F k The drone is used as a dispatch drone.

[0124] The above-mentioned step S30 includes: an embodiment of the present invention proposes a relay drone position and delay optimization algorithm, which first models the position optimization process of the scheduling drone as a Markov decision process, and uses the DDQN algorithm to optimize the position of the scheduling drone to obtain the maximum reward function, thereby reducing the data transmission delay in the network and the queuing delay of the public relay drone.

[0125] The embodiment of the present invention models the location optimization problem of scheduling drones as an MDP (Markov Decision Process), which is defined using a five-tuple (S, A, P, R, π), where S represents the state space, A represents the action space, P represents the state transition probability, R represents the reward function for the current state to enter the next state after an action, and π represents the strategy of the agent to select actions. The process of optimizing the location of drones in an air-ground heterogeneous network is to select an action a(t) and obtain a reward r(t) under the current system state s(t), and then enter the next state s(t+1).

[0126] (1) State space

[0127] The state space of the algorithm proposed in this invention mainly considers the location of the drone and the amount of data to be processed. The location parameters of the drone include the location L of the public relay drone p , the position of the drone in the previous jump L last , dispatch the position of the drone L d And the next hop drone's location L nest The position of the drone will affect the SINR and transmission delay. The variables of data to be processed include the amount of data to be processed by the public relay drone. p and the amount of data to be processed by the drone in the previous hop Date last , the amount of data to be processed will affect the queuing delay. p When it is larger, the optimization algorithm will make the dispatching drone closer to the public relay drone to alleviate the queuing delay of the data in the public relay drone. last When t is larger, the dispatched drone is closer to the previous hop drone, reducing data transmission in the network and single-hop delay. Therefore, the state space S(t) can be expressed as:

[0128]

[0129] L p (t) = [(x p ,y p,z p ) t ] T represents the three-dimensional spatial position of the public relay drone at step t; L last (t), L nest (t), L d (t)Same. represents the amount of data to be processed by the public relay drone at step t, Similarly, when the dispatched drone selects action a(t) in state s(t), it will enter the next state s(t+1).

[0130] (2) Action Space

[0131] The flight speed of the dispatched drone within the decision time t is set to v, so the position change caused by one action is c = v*t. The action space of the drone can be expressed as:

[0132]

[0133] Where [c,0,0] T , [-c,0,0] T , [0,c,0] T , [0,-c,0] T , [0,0,c] T , [0,0,-c] T Respectively represent the distances that the dispatched drone moves in the three-dimensional space along the x-axis direction, y-axis direction, z-axis direction, negative x-axis direction, negative y-axis direction, and negative z-axis direction. [0,0,0] T It means that the dispatched drone does not move. The action a(t) of the dispatched drone is restricted by the state s(t) of each step.

[0134] (3) State transition probability

[0135] In the air-ground heterogeneous network, the position of the drone is constantly changing, and the amount of data in the network also changes accordingly. The drones that have established communication can exchange their own state information. After completing an action a(t), the network state of the next step can be obtained, so there is no need to consider the state transition probability P.

[0136] (4) Reward Function

[0137] The reward function is designed based on the SINR, queuing delay and transmission delay between drones. The total reward function can be expressed as:

[0138]

[0139] Among them, r SINR(t) is the reward function related to the SINR of all communication links of the scheduled UAV, To schedule the reward function related to the delay of receiving and transmitting data to the UAV, is the reward function related to the queuing delay of the public relay drone. w1, w2, and w3 are the weight values ​​of the above three reward functions respectively. Different weight values ​​can be set according to the proportion of each factor.

[0140] The above step S40 includes: after modeling the relay drone position optimization problem as an MDP model, designing a drone position and delay optimization algorithm based on DDQN, and optimizing the position of the drone in the three-dimensional space to reduce the queuing delay in the air-ground heterogeneous network and ensure the transmission quality of data in the network.

[0141] In the Q-learning algorithm, after calculating the Q values ​​of all actions in the corresponding state, the action with the largest Q value is usually directly selected as the action of the optimal strategy, which will lead to the situation where the estimated value is higher than the actual value in the result. In order to make the Q-learning algorithm more widely used and solve more complex problems, the Q value in the Q-learning algorithm is expressed as:

[0142]

[0143] in is the objective function, expressed as:

[0144]

[0145] The DQN algorithm is based on the Q-learning algorithm and uses the parameters θ of the target network. - Therefore, the objective function in DQN is It is expressed as:

[0146]

[0147] The standard Q-learning algorithm and DQN algorithm both use the operation of maximizing the value of the objective function, which makes the evaluation and selection of an action value overestimated. To solve this problem, DDQN uses a decoupling method to separate the selection of actions and the evaluation of actions, and designs the current state strategy (used to select actions) and the value evaluation of the current state as two value functions, and randomly updates the two value functions. - is the parameter of the current state policy value function (target network), and θ is the parameter of the evaluation value function (evaluation network). The objective function of DDQN can be expressed as:

[0148]

[0149] DDQN adopts the idea of ​​dual Q learning. On the basis of DQN, it maximizes the decomposition of action selection and action evaluation, and calculates the Q value of the action selected by the evaluation network through the target network to avoid the overestimation problem.

[0150] The embodiment of the present invention provides a DDQN-based relay drone position and delay optimization algorithm framework structure as follows: Figure 4 As shown. By adding the experience replay pool to the DDQN-based relay drone position and delay optimization algorithm, the strong correlation of the observation results can be reduced. At the same time, the ε-greedy algorithm is added in the training stage to ensure that the drone can make more random action selections in the early stage of algorithm training to enrich the experience replay pool; in the later stage of algorithm training, the optimal action selection is made according to the optimal Q value. The probability of random action selection ε can be expressed as:

[0151]

[0152] Where I is the number of iterations of the algorithm.

[0153] The input data of the relay drone position and delay optimization algorithm based on DDQN are the position of the public relay drone, the position of the previous hop drone of the public relay drone, the position of the next hop drone of the dispatch drone, the position of the dispatch drone, the amount of data to be processed by the public relay node and the amount of data to be processed by the previous hop drone node of the public relay drone. The state space is composed of the above five variables, and the action space is the three-dimensional spatial position of the drone. The drone communication link signal-to-noise ratio reward function, the dispatch drone receiving and transmitting data delay reward function and the public relay drone queuing delay reward function are respectively constructed, and the total reward function is composed of a weighted sum. The relay drone position and delay optimization algorithm based on DDQN is trained by the objective function of formula (16), and the dispatch drone position is output;

[0154] The algorithm flow of the relay drone position and delay optimization algorithm based on DDQN is shown in Algorithm 1.

[0155] Algorithm 1 Relay UAV Position and Delay Optimization Algorithm Based on DDQN

[0156]

[0157] To verify the performance of the proposed algorithm, Python is used for simulation verification, and the main parameters are shown in Table 1. To verify the performance of the proposed algorithm in optimizing transmission delay, queuing delay and packet loss rate, the OLSR algorithm based on fuzzy strategy (FP-OLSR) and the GPSR algorithm based on greedy strategy (GPSR-NS) are selected as the control group for comparative analysis.

[0158] Table 1 Main simulation parameters

[0159]

[0160] Figure 5 This is a schematic diagram comparing the changes in Q values ​​of the DQN algorithm and the DDQN algorithm during training provided by an embodiment of the present invention. It can be seen that during the iteration process, the DDQN algorithm is more stable than the DQN algorithm, and the Q value after stabilization is smaller. This is because the method of selecting the maximum Q value each time in the DQN algorithm is prone to overestimation problems, and the sudden decrease in the Q value is due to the decrease in the reward value caused by the selection of a small probability action.

[0161] Figure 6 A three-dimensional position map of a drone in an initial state and three different network conditions is provided in an embodiment of the present invention. Figure 6 b) It can be seen that when Date p and Date last When the two drones are close, the dispatching drone receives the data sent by the two drones at the same time and forwards it to the next-hop drone. At this time, the dispatching drone is located at the center of the three drones. While reducing the queuing delay of the public relay drone, it also reduces the data transmission delay and improves the signal-to-noise ratio of data transmission; Figure 6 c) and d) are Date respectively last Larger and Date p From the three-dimensional position map of drones when the distance is larger, it can be seen that the dispatched drones will be closer to drones with more data to be transmitted to ensure the quality of data transmission and reduce the delay during network transmission.

[0162] Figure 7 A schematic diagram of the optimization of the queuing delay of data in the network by a different algorithm provided in an embodiment of the present invention. By comparison, it can be seen that the proposed algorithm can reduce the queuing delay in the network faster than the GPSR-NS method and the FP-OLSR method, and has a better delay optimization effect. When the queuing delay is too large, both the GPSR-NS method and the FP-OLSR method avoid data congestion in the public relay drone by re-formulating the routing plan. The proposed DDQN-based algorithm reduces the burden of the public relay drone by optimizing the location of the dispatched drone, which can more directly and effectively reduce the amount of data transmitted by the public relay drone, thereby reducing the queuing delay in the network.

[0163] Figure 8A schematic diagram of the optimization of data transmission delay by different algorithms provided in the embodiment of the present invention. As time goes by, the GPSR-NS method and the FP-OLSR method reduce the data transmission delay in the network by finding a low-latency routing solution. The proposed algorithm will increase the transmission delay at the beginning of the operation, but then it will decrease rapidly and stabilize. The final result is better than the GPSR-NS method and the FP-OLSR method. This is because the proposed optimization algorithm will be in the UAV p When the delay is too large, the UAV last Select a dispatch drone with a high signal-to-interference-to-noise ratio to assist in data transmission, as the dispatch drone also needs to assist the UAV p Completing the transmission of the data to be transmitted will result in an increase in the data transmission delay when the dispatched drone begins to assist in data transmission.

[0164] Fig. 9 The following is a schematic diagram of the optimization of data retransmission rate and packet loss rate by different algorithms given in the embodiment of the present invention. As time goes by, the three algorithms all reduce the data retransmission rate and packet loss rate of the network, but the proposed DDQN-based optimization algorithm optimizes the position of the drone based on the communication quality, which can ensure reliable data transmission, avoid the problem of data loss caused by communication interruption and multi-hop path change, and alleviate the data loss caused by excessive queuing delay. Compared with the GPSR-NS method and the FP-OLSR method, a lower packet loss rate can be achieved.

[0165] In summary, the embodiment of the present invention aims at the problem of excessive communication delay caused by high-traffic data in the air-ground heterogeneous network based on drone ad hoc network, and designs a relay drone position and delay optimization algorithm based on DDQN. The fuzzy logic method is used to select idle drones with high receiving signal-to-noise ratio and low transmission delay as scheduling drones, and the DDQN method is used to optimize the position of the scheduling drone in the network, thereby assisting the public relay drone to complete the data transmission. The simulation results show that the proposed algorithm can effectively reduce the data transmission delay and queuing delay, achieving a delay optimization effect that is 18% lower than the existing method, and can ensure the quality of data transmission, stabilizing the packet loss rate below 8%.

[0166] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0167] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0168] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0169] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for optimizing the delay and position of unmanned aerial vehicles in an air-ground heterogeneous network, characterized in that: include: Calculate the signal-to-interference-to-noise ratio between drones, and calculate the transmission delay and queuing delay of drones based on the packet size, information propagation speed, the number of drone packets to be processed, and the drone data processing speed; Calculate the territoriality of all previous-hop drones that generate the congested public relay drone. Use the fuzzy logic method to calculate the drone with the largest fitness value among the previous-hop drones based on the territoriality, signal-to-interference-noise ratio, and transmission delay between drones. Use the drone with the largest fitness value as the dispatching drone. The position optimization problem of scheduling the UAV is modeled as a Markov decision process MDP model, the state space is set according to the UAV position and the amount of data to be processed, the three-dimensional spatial position variable of the UAV is set as the action space, and the reward function is set according to the signal-to-interference-to-noise ratio, the queuing delay and the transmission delay; By using the state space, action space and reward function, the MDP model is solved through the DDQN-based drone position and delay optimization algorithm to obtain the scheduling drone position that can achieve the lowest report loss rate and system delay.

2. The method according to claim 1, characterized in that The signal-to-interference-to-noise ratio between drones is calculated, and the transmission delay and queuing delay of the drones are calculated according to the data packet size, information propagation speed, the number of data packets to be processed by the drones, and the data processing speed of the drones, including: Set up an air-ground heterogeneous UAV self-organizing network based on UAV self-organizing network, including N UAVs and M ground base stations, each UAV has a unique ID n , S i represents the source node in the network, D j Represents the ground base station in the network, UAV p Represents a public relay drone, UAV in the network F Indicates the idle relay drones in the network, with (x i ,y i ,z i )、(x D ,y D ,z D )、(x s ,y s ,z s ) represent the spatial position of the UAV, the position of the ground base station and the spatial position of the source node respectively. The network includes two communication modes: UAV-to-UAV communication U2U and UAV-to-ground base station communication U2G; UAV i and u j The signal to interference plus noise ratio SINR is defined as: The channel power gain between two UAVs is expressed as Where g is the channel power gain when the reference distance is 1m, D i,j is the current distance between the two UAVs, represents the path loss index, σ 2 represents the environmental noise, and the interference of the neighboring drone to the communication link is expressed as Where ρ represents the drone u j The proportion of information sent by other previous hop nodes, U j Indicates drone u j The set of all previous hop drones; The single-hop delay in the air-ground heterogeneous network s Defined as: Delay s =t ACK -t send (3) t ACK and t send Respectively i Receive u j The returned ACK confirms the character data and u i The time when the data packet is sent; Transmission delay p Defined as: V p is the propagation speed of electromagnetic waves in free space; Queue delay q Defined as: Where P t is the number of packets to be processed, q t is the speed at which the current public relay drone processes data packets; The total delay in the air-ground heterogeneous network is expressed as: Delay=Delay s +Delay p +Delay q (6)。 3. The method according to claim 2, characterized in that The calculation generates the territoriality of all previous-hop drones of the congested public relay drone, and uses the fuzzy logic method to calculate the drone with the largest fitness value among the previous-hop drones according to the territoriality, the signal-to-noise ratio and the transmission delay between the drones, and uses the drone with the largest fitness value as the dispatching drone, including: The Mamdani fuzzy inference mechanism is used to design a fuzzy system in which the previous hop drone of the public relay drone is a UAV. last , D norm To be able to work with UAV last The number of one-hop drones that perform normal communication is calculated using formula (14) norm Perform normalization processing; D k For UAV last The neighborhood degree of one of the neighboring drones, D max and D min For UAV last The points with the largest and smallest neighborhood degrees in the neighbor drone set; UAV last SINR between the drones and their neighbors norm Defined as: SINR last,k For UAV last The SINR between its kth neighbor drone, SINR min and SINR max UAV last The minimum and maximum SINR among all neighboring drones; UAV last The transmission delay between the drone and its neighbor p-norm It is expressed as: Delay p-last,k For UAV last Delay between its kth neighbor drone p , Delay p-min and Delay p-max UAV last The smallest delay among all neighbor drones p and the maximum Delay p ; According to D norm 、SINR norm and Delay p-norm Using fuzzy system to calculate UAV last The fuzzy value F of each neighboring drone k ,k=(1,2,...K), K is UAV last The number of reachable neighbor drones is selected with the largest F k The drone is used as a dispatch drone; In the fuzzy system, the rules of the fuzzy system are set according to the fuzzy input and fuzzy output. According to the input and output conditions, there are 27 rules. The specific rules are shown in Table 4-1. The fuzzy module converts the original input into fuzzy input and determines the membership of each fuzzy module. Then the fuzzy engine calculates the fuzzy value of the input, recombines it through the operation of rules and fuzzy logic, and finally uses the weighted average method for defuzzification, that is: f i is the fuzzy output of the previous step, w i Indicates the corresponding weight; Table 4-1 Rules of fuzzy system 4. The method according to claim 3, characterized in that The location optimization problem of scheduling the UAV is modeled as a Markov decision process MDP model, the state space is set according to the UAV position and the amount of data to be processed, the three-dimensional spatial position variable of the UAV is set as the action space, and the reward function is set according to the signal-to-interference-noise ratio, queuing delay and transmission delay, including: The location optimization problem of scheduling drones is modeled as an MDP model, and the MDP model is defined using a five-tuple (S, A, P, R, π), where S represents the state space, A represents the action space, P represents the state transition probability, R represents the reward function for the current state to enter the next state after an action, and π represents the strategy of the agent to select actions. The process of optimizing the location of drones in the air-ground heterogeneous network is to select action a(t) and obtain reward r(t) under the current system state s(t), and then enter the next state s(t+1); The state space S(t) is expressed as: L p (t) = [(x p ,y p ,z p ) t ] T represents the 3D spatial position of the public relay drone at step t, L last (t) represents the position of the last hop drone at step t, L d represents the location of the dispatched UAV at step t, L nest represents the position of the next hop drone at step t, represents the amount of data to be processed by the public relay drone at step t, It represents the amount of data to be processed by the previous drone at step t. When the dispatched drone selects action a(t) in state s(t), it will enter the next state s(t+1); Assume that the flight speed of the dispatched drone within the decision time t is v, and the position change caused by an action is c = v*t. The action space of the drone is expressed as: Where [c,0,0] T , [-c,0,0] T , [0,c,0] T , [0,-c,0] T , [0,0,c] T , [0,0,-c] T Respectively represent the distances that the dispatched drone moves in the three-dimensional space along the x-axis direction, y-axis direction, z-axis direction, negative x-axis direction, negative y-axis direction, and negative z-axis direction, [0,0,0] T Indicates that the dispatched drone does not move, and the action a(t) of the dispatched drone is restricted by the state s(t) of each step; The reward function is designed based on the SINR, queuing delay and transmission delay between drones. The total reward function is expressed as: Among them, r SINR (t) is the reward function related to the SINR of all communication links of the scheduled UAV, To schedule the reward function related to the delay of receiving and transmitting data to the UAV, is the reward function related to the queuing delay of the public relay drone, and w1, w2, and w3 are the weight values ​​of the above three reward functions respectively.

5. The method according to claim 4, characterized in that The method of using the state space, action space and reward function to solve the MDP model through the DDQN-based drone position and delay optimization algorithm to obtain the dispatched drone position that can achieve the lowest report loss rate and system delay includes: Set up a DDQN-based drone position and delay optimization algorithm. The DDQN uses a decoupling method to separate the selection action and the evaluation action, designs the current state strategy and the current state value evaluation as two value functions, and randomly updates the two value functions. - is the parameter of the current state policy value function, θ is the parameter of the evaluation value function, and the objective function of DDQN is expressed as: The probability of random action selection ε is expressed as: Where I is the number of iterations of the algorithm; The input data of the relay drone position and delay optimization algorithm based on DDQN are the position of the public relay drone, the position of the previous hop drone of the public relay drone, the position of the next hop drone of the dispatch drone, the position of the dispatch drone, the amount of data to be processed by the public relay node and the amount of data to be processed by the previous hop drone node of the public relay drone. The state space is composed of the above five variables, and the action space is the three-dimensional spatial position of the drone. The drone communication link signal-to-noise ratio reward function, the dispatch drone receiving and transmitting data delay reward function and the public relay drone queuing delay reward function are respectively constructed, and the total reward function is composed of a weighted sum. The relay drone position and delay optimization algorithm based on DDQN is trained by the objective function of formula (16), and the dispatch drone position is output; The processing flow of the DDQN-based relay drone position and delay optimization algorithm is as follows:

Images