Multi-unmanned aerial vehicle trajectory optimization method for user mobile communication in heterogeneous network

Abstract

The invention discloses a multi-unmanned aerial vehicle (UAV) trajectory optimization method for user mobile communication in a heterogeneous network, and aims to improve the security of a considered communication system and suppress the eavesdropping capability of an air mobile eavesdropper by transmitting an interference signal from a friend UAV. The invention provides an optimization problem of a plurality of unmanned aerial vehicles under three-dimensional flight path flight conditions, ground mobile user scheduling and limited energy consumption constraints, and aims to maximize the total fair secrecy rate of ground mobile users. According to the invention, a multi-agent 3D flight path and power control algorithm is designed by using DRL, and the algorithm has a centralized training distributed execution framework. And a heterogeneous MADDPG is adopted to carry out reasonable 3D trajectory design and power control on the communication unmanned aerial vehicle and the interference unmanned aerial vehicle, so that the communication unmanned aerial vehicle and the interference unmanned aerial vehicle are used as different types of intelligent agents to independently interact with the environment. According to the method, due to the diversity of action variables between the intelligent agents, different types of intelligent agents need to utilize unique information of the intelligent agents and depend on the speciality of other intelligent agents to collaborate to find an effective strategy. The total fair secrecy rate in a communication system is maximized by jointly optimizing the 3D trajectories and transmitting powers of the communication unmanned aerial vehicle and the interference unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention mainly relates to the field of UAV communication, specifically, a heterogeneous MADDPG is used to reasonably design 3D trajectories and power control for communication UAVs and jamming UAVs, so that communication UAVs and jamming UAVs can interact with the environment independently as different types of intelligent agents. Through theoretical derivation and simulation verification, the design jointly optimizes the 3D trajectory and transmission power of communication UAVs and jamming UAVs to maximize the total fair confidentiality rate in the communication system. Background Art

[0002] Unmanned aerial vehicles (UAVs) will play an important role in future wireless communication networks due to their high mobility, low cost and on-demand deployment [Y. Zeng, R. Zhang, and a TJ Lim. Wireless communications with unmanned aerial vehicles: opportunities and challenges [J]. IEEE Commun. Mag., 2016, 54(5): 36-42.], [Wu Q, Xu J, Zeng Y, et al. A comprehensive overview on 5G-and-beyond networks with UAVs: From communications to sensing and Intelligence [J]. IEEE J. Sel. Areas Commun., 2021, 39(10): 2912-2945.]. In air-to-ground (A2G) links, UAVs are widely used to improve the performance of wireless communication systems due to the high probability of the existence of line-of-sight (LoS) links [Li B, Fei Z, Zhang Y UAV communications for 5G and beyond: Recent advances and future trends [J]. IEEE Internet Things J., 2019, 6(2): 2241-2263.]. UAVs have high maneuverability and can be used in real time through trajectory design or position optimization to improve the quality of A2G channels and provide higher quality communication services. Therefore, trajectory design of UAVs plays a decisive role in improving the performance of UAV communication systems [Wu Q, Liu L, Zhang R Fundamental trade-offs in communication and trajectory design for UAV-enabled wireless network [J]. IEEE Wireless Commun., 2019, 26(1): 36-44.].

[0003] Physical layer security is a supplement to traditional information theory-based encryption technology, and can utilize the time-varying characteristics of wireless channels to improve information security. In order to improve the security of information transmission, many works combine UAV communication technology with physical layer security technology [Wang HM, Zhang X, Jiang J-CUAV-involved wireless physical layer secure communications: Overview and research directions [J]. IEEE Wireless Commun., 2019, 26 (5): 32-39.], [Wu Q, Mei W, Zhang R Safeguarding wireless network with UAVs: a physical layer security perspective [J]. IEEE Wireless Commun., 2019, 26 (5): 12-18.], [J. Wang, X. Wang, R. Gao, et al. Physical layer security for UAV communications: a comprehensive survey [J]. China Commun., 2022, 19 (9): 77-115.]. By reasonably designing the flight path of the UAV, it can be as close to the legitimate user as possible and away from potential eavesdroppers, thereby improving the security of the communication system. References [Gao Y, Tang H, Li B, et al.Securing energy-constrained UAV communications against both internal and external eavesdropping[J]. IEEE Commun. Lett., 2021, 25(3): 749-753.] studied the security issues of auxiliary communication systems of rotary-wing UAVs with internal and external eavesdropping under limited energy consumption. An iterative algorithm based on SCA and BCD is proposed to solve this non-convex problem by jointly optimizing the UAV's 2D flight trajectory, transmission power, and user scheduling.The paper [Duo B, Luo J, LiY, et al. Joint trajectory and power optimization for securing UAV communications against active eavesdropping [J]. China Commun., 2021] studied an active eavesdropping UAV system and proposed an effective iterative algorithm. It obtained high-quality suboptimal solutions through SCA and BCD methods, jointly optimized the 2D trajectory and transmission power of the UAV, and effectively improved the average secrecy rate (ASR) performance of the uplink and downlink.

[0004] In aviation communication systems, fairness among users is also a key issue that needs to be considered. In the paper [Ding R, Gao F, Shen X 3D UAV trajectory design and frequency band allocation for energy-efficient and fair communication: A deep reinforcement learning approach [J]. IEEE Trans. Wireless Commun., 2020, 19 (12): 7796-7809.], the authors proposed a new dynamic frequency band allocation strategy, which uses the DDPG algorithm to solve the optimization problem while considering the random movement of users. This strategy considers both the flight energy consumption and fairness of the UAV. The uplink fair throughput is maximized by jointly optimizing the 3D trajectory and frequency band allocation. The literature [Zhang Z, Xu C, Li Z, et al. Deep reinforcement learning foraerial data collection in hybrid powered NOMA-IoT networks[J]. IEEE InternetThings J., 2023, 10(2): 1761-1774.] designed the 3D trajectory of the drone and the time allocation of data collection, proposed a DDPG-based algorithm to solve the problem, and studied the drone-assisted communication system using non-orthogonal multiple access (NOMA) technology, considering the effective energy consumption constraints, service quality requirements and flight conditions, to maximize the total fair throughput. The literature [Khodaparast SS, Lu X, Wang P, et al. Deep reinforcement learning based energy-efficient multi-UAV data collection for IoT networks[J]. IEEE Open J. Veh. Technol., 2021, 2 249-260.] considered the UAV flight energy consumption, sensor node energy consumption and aerial obstacle constraints, and proposed an optimization problem. The DDPG algorithm and multi-agent DQN algorithm were used to respectively implement the UAV trajectory design and energy consumption control problems and the multi-UAV task allocation problem to minimize the total energy consumption of the considered IoT system.The paper [Yin S, Yu FR Resource allocation and trajectory design in UAV-aided cellular networks based on multiagent reinforcement learning[J].IEEE InternetThings J., 2022, 9(4):2933-2943.] studied a multi-UAV-aided downlink cellular network, in which multiple UAVs act as aerial base stations (BSs) to provide services to ground users based on FDMA technology. The DQN-based multi-agent reinforcement learning algorithm is used to solve the problem. The total rate and fair throughput of the system are maximized by jointly optimizing the user scheduling, power, and trajectory of the UAVs. Summary of the invention

[0005] Aiming at the problem of fair and secure communication of multiple drones assisting ground mobile users in the scenario of aerial mobile eavesdropping by drones, the present invention proposes to suppress the eavesdropping ability of aerial mobile eavesdroppers by transmitting interference signals from friendly drones. The present invention proposes the optimization problem of multiple drones under the flight conditions of 3D flight trajectories, ground mobile user scheduling and limited energy consumption constraints, in order to maximize the total fair confidentiality rate of ground mobile users.

[0006] To this end, the technical solution adopted by the present invention is: multi-UAV trajectory design for user mobile communication in a heterogeneous network, comprising the following steps:

[0007] (1) Constructing the communication system model: In the system, a drone S acts as an aerial base station BS, which can communicate with a single-antenna downlink ground user U k To communicate, a potential eavesdropping drone E from the starting point Move along a straight line at a constant speed to the end point Attempt to independently eavesdrop on ground user U k Confidential information, another drone J acts as a friendly jammer, suppressing the eavesdropping capability of the eavesdropping drone E by emitting artificial noise; all drones operate at different altitudes and share the same frequency band, and each drone provides communication services to only one ground user in each time slot. Define {U i , i=1,2,...,K} is the downlink ground user set, K represents the number of ground mobile users, and the users move within a certain area. Considering the user random mobility model, the user's moving speed V is uniformly distributed in θ~U(0,5), and the user's moving angle is uniformly distributed in θ~U(0,2π);

[0008] (2) With the goal of maximizing the total fair confidentiality rate in the communication system, an optimization mathematical model is constructed by jointly optimizing the 3D trajectories and transmission powers of the communication UAV and the jamming UAV, with this as a constraint;

[0009] (3) For the constructed optimization mathematical model, when a large number of drones and ground users are involved, the complexity of the traditional convex optimization solution increases dramatically and may fall into a local optimal solution. Therefore, alternative methods are needed to effectively solve this problem. Since the proposed problem involves optimizing the 3D trajectories of all drones and the optimization variables are continuous, a multi-agent deep reinforcement learning (MADRL) algorithm, referred to as HFSCTPD, is adopted. The algorithm has a centralized training and distributed execution framework. At the same time, heterogeneous MADDPG is further used to perform reasonable 3D trajectory design and power control for communication drones and jamming drones. The algorithm uses the powerful learning ability of deep reinforcement learning (DRL) to find the optimal solution by exploring a vast policy space.

[0010] The present invention is aimed at the actual situation of the fair and secure communication problem of multiple drones assisting ground mobile users in the aerial mobile eavesdropping scenario, with the goal of maximizing the total fair confidentiality rate in the communication system, by jointly optimizing the 3D trajectory and transmission power of the communication drone and the jamming drone. Among them, fair scheduling for ground mobile users and maximizing the total fair confidentiality rate of the system are the ultimate expected effects and goals of the present invention.

[0011] Most existing studies focus on homogeneous multi-agent reinforcement learning, where multiple agents share the same state and action space. The proposed method has heterogeneous composition of multi-agents including communication drones and jamming drones due to the diversity of action variables between agents. Different categories of agents need to use their unique information and rely on the expertise of other agents to collaboratively find effective strategies. By jointly optimizing the 3D trajectory and transmission power of the communication drone and the jamming drone, the total fair confidentiality rate in the communication system is maximized.

[0012] Step 1: First, based on the basic theories of wireless communication systems and PLoS channel models, we build energy consumption models, flight mobility models, and wireless communication system models for drone-assisted ground mobile users. The optimization objective in is a non-convex multivariate coupling problem with respect to Q and P. The energy constraint C1 and the multi-UAV distance constraint C4 are also non-convex with respect to Q. In addition, a probabilistic LoS link is considered, where the channel gain depends not only on the 3D trajectory but also on the distance between the UAV and the user, the flight altitude, and the probability of the LoS link. The above factors increase the complexity of the total throughput between the UAV and the user.

[0013] Steps 2 and 3 improve the mathematical theory derivation and problem-solving analysis in the process of model building. In view of the non-convexity and high coupling of the optimization problem corresponding to the system model, a multi-agent deep reinforcement learning (MADRL) algorithm is proposed, referred to as HFSCTPD. The algorithm has a centralized training distributed execution framework. At the same time, heterogeneous MADDPG is further used to perform reasonable 3D trajectory design and power control for communication drones and jamming drones. The algorithm uses the powerful learning ability of DRL to find the optimal solution by exploring a broad strategy space. (For the specific process, see the specific implementation of the present invention).

[0014] The present invention also adopts the POMDP reconstruction optimization problem, designs the state space, action space and reward design, and each agent has its own Actor network, Critic network and corresponding Target network. The Actor network of each agent uses independent observations as input to generate actions, while the Critic network of each agent uses the observations and actions of all agents as input. In this way, the Critic network can more accurately evaluate the joint behavior of multiple agents, thereby helping the agents make more accurate decisions to obtain the optimal 3D trajectory and transmission power. This optimal solution is the maximum system total fair confidentiality rate required by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A communication system model of the present invention;

[0016] Figure 2 is the average cumulative reward graph of the drone of the present invention;

[0017] Figure 3 The cumulative confidentiality rate graph of the UAV of the present invention;

[0018] Figure 4 is the average fairness factor graph of the drone of the present invention;

[0019] Figure 5 Accumulate fair confidentiality rate graph for the present invention;

[0020] Figure 6 It is the average speed graph of the training rounds of the UAV of the present invention;

[0021] Figure 7 is a probability diagram of the UAV violating the acceleration constraint of the present invention;

[0022] Figure 8 It is a probability graph of the UAV violating the distance constraint of the present invention;

[0023] Fig. 9 A probability graph of the UAV violating the altitude constraint of the present invention;

[0024] Fig.10 It is a flight cycle speed diagram of the communication UAV of the present invention;

[0025] Fig.11 This is the flight cycle speed diagram of the interference drone of the present invention;

[0026] Fig.12 The flight cycle user scheduling diagram of the present invention;

[0027] Fig.13 A confidential rate diagram for the flight cycle of the present invention;

[0028] Fig.14 A three-dimensional trajectory diagram of the UAV of the present invention;

[0029] Fig.15 It is a two-dimensional trajectory diagram of the UAV of the present invention;

[0030] Fig.16 This is the transmission power diagram of the UAV of the present invention;

[0031] Fig.17 This is the flight cycle fairness index diagram of the present invention. DETAILED DESCRIPTION

[0032] The process results and simulation of the present invention are briefly described below in conjunction with the accompanying drawings.

[0033] Figure 1 The system model diagram of the whole scheme is shown in FIG. 1 . The present invention considers an IoT communication system supported by two UAVs, in which one UAV (S) acts as an aerial base station (BS) and can communicate with a single-antenna downlink ground user U k There is a potential eavesdropping drone (E) from the starting point Move to the end point along a straight line at a constant speed Attempt to eavesdrop on the confidential information of the ground user independently. To improve the security of wireless communication, another UAV (J) acts as a friendly jammer to suppress the eavesdropping ability of the eavesdropping UAV (E) by emitting artificial noise. Assume that all UAVs operate at different altitudes and share the same frequency band, and each UAV only provides communication services to one ground user in each time slot. Define {U i ,i=1,2,...,K}, K represents the number of ground mobile users, is the downlink ground user set, we denote the coordinates of user k as U k In actual situations, users move within a certain area. Considering the user random movement model, the user's movement speed V is uniformly distributed from θ to U (0,5). The user's movement angle is uniformly distributed from θ to U (0,2π), where U represents uniform distribution. In order to facilitate trajectory optimization, we set the drone flight period T t Divided equally into N time slots δ t =Tt / N, where as long as N is large enough, the position of ground user k in each time slot can be guaranteed to be approximately unchanged. Without loss of generality, the Cartesian coordinate system is used to describe the positions of all nodes. The coordinate positions of S and J in the nth time slot are expressed as U k The coordinate positions of and E in the nth time slot are respectively expressed as and x is the x-axis of the Cartesian coordinate system, y is the y-axis of the Cartesian coordinate system, and z is the z-axis of the Cartesian coordinate system.

[0034] The speed and position of S and J at the n+1th time slot are expressed as

[0035]

[0036] In the formula, ξ ai is a binary indicator, ξ ai =1,ξ ai =-1 means the acceleration direction of the drone is the same as or opposite to the speed direction. v i [n] represents the UAV acceleration vector and unit velocity vector respectively. i [n] represents the speed of the drone in the nth time slot, Represents the drone unit acceleration vector.

[0037] We adopt user scheduling for ground users and assume that the communication UAV can only associate and communicate with at most one ground user in each time slot, which results in the following ground user scheduling constraints:

[0038]

[0039] u k [n] represents a user-scheduled binary variable.

[0040] Assuming that the air-to-ground (A2G) link experiences a probabilistic LoS path loss model, the probabilistic LoS of the UAV to the ground user in the nth time slot is expressed as

[0041]

[0042] where η a and η b is a constant related to the type of propagation environment.

[0043] From S to U k The Euclidean distance between S and U k The average path loss expression between

[0044]

[0045] They represent the probabilistic line-of-sight link in the nth time slot between the communication UAV and the ground user, and the non-probabilistic line-of-sight link in the nth time slot between the communication UAV and the ground user, respectively. and Represents S and U respectively k The path loss between LoS and NLoS links, is the free space loss, f c is the carrier frequency, v c is the speed of light, η LoS and η NLoS are the path losses of the LoS and NLoS links respectively. Then S and U in the nth time slot k The channel gain between

[0046]

[0047] Similarly, in the nth time slot, the interference drones J and U k The channel gain between

[0048]

[0049] Indicates J and U k The average path loss between .

[0050] Drone to U k The downlink ergodic capacity of a terrestrial user at the nth time slot is

[0051]

[0052] Where N0 is the noise power spectrum density and B is the bandwidth occupied by ground users. S [n] and p J [n] are the transmission powers of the communication UAV and the jamming UAV respectively.

[0053] The channel gain of the UAV in air-to-air (A2A) is expressed as

[0054]

[0055] Where β0 represents the channel gain at a reference distance of 1 m.

[0056] Then the eavesdropping rate of the system communication link that can be achieved at E is expressed as

[0057]

[0058] We consider the worst communication security situation for ground users, then U k The achievable confidentiality rate is expressed as

[0059]

[0060] In the formula

[0061] Therefore, the total secrecy rate (TST) of the ground user in the nth time slot is expressed as

[0062]

[0063] There are two types of UAV energy consumption, namely communication energy consumption and propulsion energy consumption. The communication energy consumption is usually two orders of magnitude smaller than the propulsion energy consumption, so the communication energy consumption can be ignored. The propulsion power of the UAV is expressed as

[0064]

[0065] Where P B and P I is a constant parameter representing the blade power and induced power in the hovering state, U tip is the rotor tip speed, v0 is the average rotor induced speed in the hovering state, d0, s, ρ and A are the body resistance ratio, rotor stiffness, air density and rotor disk area, respectively.

[0066] The remaining energy E of the drone in the nth time slot i [n] is represented by

[0067]

[0068] Where E max is the initial energy of the drone, that is, E i [0] = E max In order to ensure that the drone can reach the destination normally, it must have

[0069]

[0070] Let v s [n] and v J [n] represents the flight speed of S and J respectively, then v s [n] and v J [n] should meet

[0071]

[0072] v max Indicates the maximum speed of the drone.

[0073] At the same time, the flight height of S and J should satisfy

[0074]

[0075] z min , z max They represent the lowest altitude and the highest altitude of the drone respectively.

[0076] In order to avoid collisions between multiple drones, the flight distance of the drones should meet the following requirements:

[0077]

[0078] D represents the minimum safe flight distance between multiple drones.

[0079] In order to ensure fairness among the ground users served by the drone, we define a binary variable I uk [n]

[0080]

[0081] in represents the Jain fairness index, R cum [n] represents the total safety rate of the nth time slot. f is the target fairness threshold, and They respectively indicate whether the fairness among all ground users is satisfied or not. Depends on the fairness index and the fairness of the n-1th time slot when or Sometimes, there is These two conditions respectively indicate that the fairness constraint is satisfied in the nth time slot and the fairness constraint is satisfied before the nth time slot. The fairness of depends on the fairness of n-1 time slots if but

[0082] It should be noted that if So and When the fairness constraint between ground users is satisfied, the optimization problem is transformed into maximizing the confidentiality rate of ground users. At this time, S should schedule the ground users with the best channel information to maximize the total confidentiality rate of ground users. By defining the above variables, after the UAV is associated with the ground user, it can make a trade-off between the fairness and confidentiality rate of the system during the communication process.

[0083] In the nth time slot, the fairness factor between ground users is defined as

[0084]

[0085] where k gp >0 is the attenuation coefficient, R max is the threshold value. gp Will cause the curve to decay faster, and a smaller k gp This will cause the curve to decay more slowly. When the uploaded information reaches the threshold R max When S It can establish associations and communicate with other ground users at higher levels, thereby improving fairness among ground users.

[0086] The present invention proposes a fair scheduling scheme for ground users. Specifically, S selects the ground user with the largest fair confidentiality rate in each time slot. Establish a communication link to maximize the total confidentiality rate. Define the nth time slot S and U k The scheduling index between

[0087]

[0088] More specifically, in the nth time slot, S first determines whether the fairness constraint is met based on the cumulative confidentiality rate of the ground users in the n-1 time slots. If the fairness constraint in the formula is not met, S will schedule the unassociated ground users to improve the fairness among the ground users. Otherwise, the optimization problem becomes a trade-off between fairness and confidentiality rate among ground users.

[0089] When the UAV is associated with the ground user and communicates, the total fair confidentiality rate during the entire system mission cycle is expressed as

[0090]

[0091] Construct the complete optimization problem:

[0092]

[0093]

[0094] In the above formula, C1 represents the initial and final position constraints of the drone, C2 represents the acceleration constraint of the drone, C3 represents the position constraint of the drone, C4 represents the energy consumption constraint, C5 represents the initial speed constraint, C6 represents the speed constraint, C7 represents the height constraint, C8 represents the safety distance constraint of the drone, and C9 represents the peak power constraint of the transmission power of the drone in each time slot. represents the maximum transmission power of the communication UAV and the jamming UAV, and C10 represents the mobility constraint of the ground user;

[0095] a min , a max They represent the minimum acceleration and maximum acceleration of the drone respectively. i [0], Q i [N], They respectively represent the coordinate position of the drone at the 0th time slot, the coordinate position of the drone at the Nth time slot, the starting coordinate position of the drone, and the ending coordinate position of the drone.

[0096] solve Challenging because The optimization objective in is a non-convex multivariate coupling problem with respect to Q and P. The energy constraint C1 and the multi-UAV distance constraint C4 are also non-convex with respect to Q. In addition, a probabilistic LoS link is considered, where the channel gain depends not only on the 3D trajectory but also on the distance between the UAV and the user, the flight altitude, and the probability of the LoS link. The above factors increase the complexity of the total throughput between the UAV and the user.

[0097] When the optimization problem proposed in the present invention involves a large number of drones and ground users, the complexity of traditional convex optimization solutions will increase dramatically and may fall into local optimal solutions. Therefore, alternative methods are needed to effectively solve this problem. Since the problem proposed involves optimizing the 3D trajectories of all drones and the optimization variables are continuous, a multi-agent deep reinforcement learning (MADRL) algorithm is proposed, referred to as HFSCTPD. The algorithm has a centralized training distributed execution framework. At the same time, heterogeneous MADDPG is further used to perform reasonable 3D trajectory design and power control for communication drones and jamming drones. The algorithm uses the powerful learning ability of DRL to find the optimal solution by exploring a broad strategy space. The solution to this problem will be based on the following steps.

[0098] The specific steps are as follows:

[0099] (1) In a multi-agent environment, each agent can usually only observe part of the environmental information, independently choose its own actions, and cannot access the state of the entire system. Therefore, the partially observable Markov decision process (POMDP) ​​will be used to model the system to improve the understanding of the environment and adaptability. Each drone is regarded as an agent, which is regarded as POMDP and (S, A, R, P, γ), where S is the set of states describing the global environment, A is the joint action set of all agents, R is a set of reward functions that map states and agent behaviors to rewards, and P is the probabilistic state transition function that describes the next state of all agents after taking joint actions in the current state. γ is the reward discount factor. Many Markov decision processes are partially observable, where agents can only observe part of the environmental state. The observation set of all agents is defined as O{O s ,O J}. Let o={o s ,o J} is defined as the observable state of the local environment, a={a s ,a J} is defined as an observable action, r = {r s ,r J} is defined as each agent will receive a corresponding reward, and then the environment transitions to the next state according to the state transition function P. s ,o J , a s ,a J , r s ,r J They represent the observable input states of the communication UAV and the jamming UAV, the output actions of the communication UAV and the jamming UAV, and the reward functions of the communication UAV and the jamming UAV respectively.

[0100] (2) All MADDPG models require state information as input to estimate the Q value of the action. It should be noted that the joint optimization process involves two different types of drones (communication drone and jamming drone), so the drone position coordinates are Q[n] = {Q S [n],Q J [n],Q E [n]}. In this case, the state information includes user channel information. The advantage of using user channel information instead of user position is that the UAV itself knows the user channel information, while collecting the user position each time will incur additional communication overhead. Therefore, user channel information is also the basis for determining the power allocation of all users. The partial observation state space of S consists of the UAV position, speed, remaining energy consumption, Euclidean distance to the end point, fairness index, cumulative confidentiality rate of S relative to all ground users, and user channel information. Therefore, the partial observation state space of S is expressed as

[0101]

[0102] It can be observed that S has a total of 2K+12 state dimensions.

[0103] The state space of J consists of the drone’s position, speed, remaining energy consumption, and Euclidean distance to the destination. Therefore, the partial observed state space of J is expressed as

[0104]

[0105] It can be observed that J has a total of 11 state dimensions.

[0106] It should be noted that communication drones and jamming drones are considered as two types of heterogeneous multi-agents. In the real world, different types of agents must share some information with each other. The state space of each agent contains the location information of all drones. The difference is that the state space of each agent only contains its own flight speed to the destination, its own energy consumption, and its own Euclidean distance to the destination. In order to protect the security of ground users during communication, the fairness index, the cumulative confidentiality rate of all ground users, and the user channel information are not included in J's state space.

[0107] (3) The action space consists of the 3D trajectory of the UAV and the transmission power. Since the 3D trajectory of the UAV can be obtained by iterating the velocity vector, the action is represented by the velocity vector of the UAV, so the trajectory of the UAV can be equivalently represented.

[0108] 1) UAV speed: using spherical coordinates Describe the speed and flight direction of the drone,

[0109] in and denote the polar angle and azimuth respectively. Then the velocity is expressed as The normalized expression of the UAV speed is in, They represent normalized velocity, normalized polar angle, and normalized azimuth, respectively.

[0110] 2) UAV power allocation: UAV transmit power action P,

[0111] Therefore, the action space of S and J is expressed as

[0112]

[0113] They represent the polar angle of the UAV in the nth time slot and the azimuth angle of the UAV in the nth time slot respectively. And it can be observed that action a has a total of 8 action dimensions.

[0114] (4) In DRL, rewards are used to evaluate state actions. Through reward design, the optimization problem can be transformed into maximizing cumulative rewards. In the HFSCTPD algorithm proposed in this paper, the reward function consists of four parts: energy consumption reward, destination reward, fair confidentiality rate reward, and constraint violation reward.

[0115] 1) Energy consumption reward: Energy consumption reward is used to encourage drones to control energy consumption by optimizing speed during flight. Therefore, the total fair confidentiality rate can be maximized under limited energy consumption. The energy consumption reward is defined as

[0116]

[0117] The subscript “ec” represents energy consumption, κ ec is a normal quantity used to adjust the reward size of the energy consumption part.

[0118] 2) Reward for reaching the end point: In order to solve the problem of sparse rewards during the return to the end point, the present invention adopts a reward reshaping technique, which provides more continuous and informative reward feedback, improves learning efficiency, and makes it easier for the drone to reach the end point.

[0119] To describe the state where the drone is about to run out of energy and must return to the end point, define a binary indicator

[0120]

[0121] The formula gives E i [n],E i [n] represents the energy consumption of the nth time slot, where E i,min is the energy threshold. In order to enable the UAV to maximize the fair confidentiality rate, based on the relationship between the Euclidean distance to the end point and the energy threshold, the adaptive energy threshold is defined as

[0122]

[0123] In the formula represents the optimal speed when the energy consumption per unit distance is the minimum, and E0 is the energy compensation constant. Therefore, the continuous reward endpoints of multiple drones are defined. represents the difference in distance between the drone and the end point after it moves for one time slot, κ rd1 , κ rd2 and κ rd3 , are constants used to adjust the incremental distance reward, distance reward, and the slope of the reward with distance. i,rd When [n] = 1, the continuous reward associated with the end point is given by κ rd1 Δdi and The former represents the incremental distance reward for flying to the end point, and the latter represents the distance reward of the drone relative to the end point.

[0124]

[0125] Δd i It represents the difference between the coordinate positions of the drone in two adjacent time slots.

[0126] In order to indicate whether the drone reaches the destination when the energy is exhausted, the discontinuous arrival reward (r i,ar [n],i∈{S,J}) is

[0127]

[0128] Among them, ξ i,ar =1 means that the drone reaches the destination in the last time slot, otherwise, ξ i,ar =0,κ ar is a normal quantity, motivating the drone to reach the end point, κ nar is a negative constant that penalizes drones that fail to reach the end point.

[0129] 3) Fair confidentiality rate reward: S communicates with ground users and needs to optimize the transmission power to maximize the total fair confidentiality rate. J's goal is to suppress airborne mobile eavesdropping to improve the fair confidentiality rate of ground users. Therefore, in the nth time slot, the fair confidentiality rate reward is defined as

[0130]

[0131] Among them, κ th represents the reward when the energy is sufficient, κ nth It indicates that the flight energy is about to run out and the reward needs to return to the end point. It is worth noting that when κ th >κ nth , because when the drone’s energy is about to run out, the reward decreases, making the drone more inclined to return to the end point. i,rd [n], R ins [n] represents the binary variable when the energy is sufficient or the energy is about to be exhausted, and the fair confidentiality rate hyperparameter.

[0132] 4) Constraint violation reward: To punish the drone for violating the constraints given by distance, height and acceleration, the following reward is defined:

[0133]

[0134] Among them, r i,L =1, r i,H =1 and r i,A=1, indicating that the distance constraint, height constraint and acceleration constraint are not satisfied. Negative constant κ L , κ H and κ A denote the reward constants for violating the distance constraint, height constraint, and acceleration constraint, respectively. ξ i,L [n],ξ i,H [n],ξ i,A [n] represents the distance hyperparameter, height hyperparameter, and acceleration hyperparameter respectively.

[0135] In summary, in the HFSCTPD algorithm, the total reward function of S and J is expressed as

[0136]

[0137] In the HFSCTPD algorithm, S and J are considered as heterogeneous agents and the communication system under consideration as the environment. The goal of each agent is to select the best action based on its experience to maximize the reward accumulated over time. The training process of MADDPG is stabilized mainly through the dimension expansion in the HFSCTPD algorithm, as well as its implementation in the Actor network, the Critic network, and the respective Target network.

[0138] 1) Actor network: In the HFSCTPD algorithm, the actions of each agent are first normalized to in, In neural networks, the Tanh activation function is used to output In the state space of S, The dimensional expansion is N q ,N v ,N e , N d , N f ,N r ,N g , and then these expanded dimensions are combined as the input to the actor and critic networks.

[0139] 2) Critic network: The input of the Critic network is the extended state of all agents and joint actions. The Critic network evaluates the quality of the subject's actions through global states and actions. In the Actor network, the input is a partially observable state. In the Critic network, the output is the action, which is extended to N a Dimension. Centralized training and distributed execution training methods can significantly improve the learning efficiency of multi-agents. Critic and Actor networks have the same hidden layer structure. The output layer of the Critic network does not need to use any activation function, but directly outputs the original value.

[0140] 3) Target network: In the MADDPG algorithm, the Target network calculates the loss of the Actor and Critic networks to stabilize their updates. The neural network structure of the Target network is the same as that of the Actor network. The Critic network and the Target network use soft updates to update the network parameters, which can make the parameters of the Target network follow the changes of the current network more smoothly, which helps to reduce instability and oscillation during training and improve the stability of training.

[0141] The HFSCTPD algorithm proposed in this invention is mainly divided into two parts: training phase and implementation phase. In the training phase, each agent starts from the starting position until the energy is exhausted or reaches its end position. The maximum number of training rounds is M. ep Before starting training, initialize the neural network parameters and the random process of all agents. At the beginning of each round, the starting position, speed and energy consumption of the multiple drones are initialized, where the initial energy of the multiple drones is E max In the nth time slot of each round, each agent obtains a joint action by feeding its partial observed state back to its own Actor policy network. We use a normally distributed noise with a mean of 0 and a standard deviation of 0.1. To prevent falling into the local optimal strategy. All agents perform this joint action to obtain the new state o[n+1] and reward r[n], and the transition data {o[n], a[n], r[n], o[n+1]} is stored in the experience replay buffer If the number of samples exceeds the storage space of the experience replay buffer, the oldest sample is replaced with the latest sample. During network update, each agent extracts the sample from the experience replay buffer. A fixed batch of experience data is randomly sampled from the network to obtain the losses of the Actor network and the Critic network, and the network parameters are updated through the back-propagation algorithm. Then the Target network is slowly updated through the soft update technique. In the implementation stage, each agent inputs the current partial observation state into its own Actor policy network to obtain the 3D trajectory and emission power.

[0142] The following is a pseudo code of the algorithm proposed by the present invention for solving the optimization problem, and its detailed process corresponds to the specific solution steps of the above-mentioned optimization problem.

[0143]

[0144] Figure 2This is a simulation verification based on the proposed solution. It can be seen that as the number of training rounds increases, the reward curve of the drone will converge. However, due to exploration noise, the reward will fluctuate within a certain range.

[0145] Figure 3 This is a simulation verification based on the proposed solution of the present invention. It can be seen that with the increase of the number of training rounds, the confidentiality rate curve of the drone will converge. However, due to the exploration noise, the reward will fluctuate within a certain range.

[0146] Figure 4 This is a simulation verification based on the proposed solution of the present invention. It can be seen that with the increase of the number of training rounds, the average fairness factor will converge within the flight cycle of the drone. However, due to the exploration noise, the reward will fluctuate within a certain range.

[0147] Figure 5 This is a simulation verification based on the proposed solution of the present invention. It can be seen that with the increase of the number of training rounds, the fair confidentiality rate curve of the drone will converge. However, due to the exploration noise, the reward will fluctuate within a certain range.

[0148] Figure 6 The figure shows the changing trend of the average speed of the drone as the number of training rounds increases. It can be seen that as the number of training rounds increases, the average speed first increases and then decreases, and finally converges to about 11m / s. This is because under the model that considers energy consumption, the communication drone has the lowest energy consumption per time slot at a speed of 10m / s, and can fly the farthest distance at a speed of 16m / s. From the beginning, communication drones prioritize learning to meet fairness among service users. Therefore, they tend to visit all ground users and fly frequently between ground users, so the average speed tends to 16m / s. As the number of training rounds increases, when fairness is met, communication drones tend to fly to nearby users to maximize the total confidentiality rate. Therefore, communication drones will fly at a speed of about 11m / s to ensure that they can fly longer and thus obtain greater throughput.

[0149] Figure 7 , Figure 8 and Fig. 9 The constraint violation probability curves of the communication UAV and the jamming UAV are given as the number of training rounds increases. Figure 8 It can be observed that at the beginning of training, the probability of violating the acceleration constraint is around 0.6-0.4. As the number of training rounds increases, the probability of violating the acceleration constraint converges around 600 rounds and converges to between 0 and 0.1. This is because multiple drones will receive a discrete penalty reward if they violate the constraint in each flight time slot, so that the policy network of each agent learns how to output the appropriate flight speed and angle to meet the constraints and avoid being punished. Fig. 9It can be observed that the probability of the communication UAV violating the distance constraint tends to 0, while the probability of the interference UAV violating the distance constraint first increases and then decreases, and finally converges. Fig.10 It can be observed that the probability of violating the height constraint is around 0.5 to 0.2. As the number of training rounds increases, the probability of violating the acceleration constraint converges around 600 rounds and converges to between 0 and 0.1. The drone is punished after violating the height constraint, thus ensuring that the drone flies at a normal altitude.

[0150] Fig.10 and Fig.11 The simulation verifies the convergence of the solution of the present invention and plots the speed change curves of the communication UAV and the jamming UAV during flight. Fig.11 and Fig.12 It can be seen that the communication drone's flight speed tends to be 18m / s when it is far away from the user, and it flies at a speed of about 10m / s when it is close to the user. This is because, under the energy consumption model, the drone has the lowest energy consumption per time slot at 10m / s, and the longest flight distance when it tends to 18m / s. Therefore, by flying at the above speeds, it can fly as long as possible and fly over the ground users as quickly as possible to maximize the cumulative fair confidentiality rate. Fig.11 It can be observed that the speed of the jammer drone increases first and then tends to fly at a speed of about 10 m / s. This is because, first, the jammer drone needs to fly near E as quickly as possible to reduce the eavesdropping rate. Then, it flies around E at a speed of about 10 m / s, and finally moves away and returns to the end point.

[0151] Fig.12 and Fig.13 The user scheduling and confidentiality rate of the communication UAV during the flight cycle are verified. It can be seen that all ground users associated with the communication UAV are awakened in turn, and the user with the highest fair confidentiality rate is scheduled in each time period to ensure fairness among users. Fig.13 and Fig.14 It can be seen that the confidentiality rate of the communication drone always increases first and then decreases in each time period due to the communication drone approaching and moving away from the user.

[0152] Fig.14 and Fig.15The three-dimensional and two-dimensional trajectories of the communication and jamming drones. It can be observed that the ground users are served by a dedicated communication drone. The communication drone starts from the starting point and flies around the ground users in turn, and tries to fly as far away from the eavesdropping drone as possible to provide confidentiality and fair service to the ground users. When the energy is about to be exhausted, it returns to the destination. The jamming drone flies from the starting point, sends jamming signals around the eavesdropping drone, suppresses the eavesdropping ability, and tends to follow the flight trajectory of the eavesdropping drone, maintaining a certain safe distance. Finally, if the energy consumption is about to be exhausted, the jamming drone will return to the end point.

[0153] Fig.16 The simulation shows the transmit power of all drones. The communication drone communicates with the ground user at approximately maximum power, and adjusts its trajectory to ensure the maximum confidentiality rate. At the same time, the jamming drone transmits artificial noise at maximum power. When close to the eavesdropping drone, the jamming drone operates at a lower power to reduce interference. An interesting finding is that drones are more inclined to adjust their trajectory to improve communication performance than to adjust their transmit power.

[0154] Fig.17 The simulation shows the fairness index of the communication drone during the flight. It can be seen that the fairness index first increases and then decreases to a constant. This is because the communication drone needs to visit each user first to ensure the fairness constraint. When the communication drone provides services to more users, the fairness index of the considered system will increase with the increase in the number of users in the service, reflecting the fairness of the communication service. When the fairness constraint is met, the communication drone will continue to communicate around certain users to maximize the total confidentiality rate, resulting in a decrease in the fairness index.

Claims

1. A multi-UAV trajectory optimization method for user mobile communication in heterogeneous networks, characterized in that: The following steps are involved: (1) Constructing the communication system model: In the system, a drone S acts as an aerial base station BS, which can communicate with a single-antenna downlink ground user U k To communicate, a potential eavesdropping drone E from the starting point Move along a straight line at a constant speed to the end point Attempt to independently eavesdrop on ground user U k Confidential information, another drone J acts as a friendly jammer, suppressing the eavesdropping capability of the eavesdropping drone E by emitting artificial noise; all drones operate at different altitudes and share the same frequency band, and each drone provides communication services to only one ground user in each time slot. Define {U i , i=1,2,...,K} is the downlink ground user set, K represents the number of ground mobile users, the users move within a certain area, the user's moving speed V is uniformly distributed θ~U(0,5), and the user's moving angle is uniformly distributed θ~U(0,2π); (2) With the goal of maximizing the total fair confidentiality rate in the communication system, an optimization mathematical model is constructed by jointly optimizing the 3D trajectories and transmission powers of the communication UAV and the jamming UAV, with this as a constraint; (3) The centralized training distributed execution framework in the multi-agent deep reinforcement learning algorithm and the heterogeneous MADDPG are used to perform 3D trajectory design and power control for communication UAVs and jamming UAVs.

2. The multi-UAV trajectory optimization method for user mobile communication in a heterogeneous network according to claim 1 is characterized by: The optimization mathematical model is: In the above formula, C1 represents the initial and final position constraints of the drone, C2 represents the acceleration constraint of the drone, C3 represents the position constraint of the drone, C4 represents the energy consumption constraint, C5 represents the initial speed constraint, C6 represents the speed constraint, C7 represents the height constraint, C8 represents the safety distance constraint of the drone, C9 represents the peak power constraint of the drone's transmission power in each time slot, and P i max represents the maximum transmission power of the communication UAV and the jamming UAV, and C10 represents the mobility constraint of the ground user; In the optimization problem P1, Q represents the coordinate position of the UAV, P represents the transmission power of the UAV, and R sum It represents the total fair confidentiality rate in the whole system task cycle, Q i [0], Q i [N], They represent the coordinate position of the drone at the 0th time slot, the coordinate position of the drone at the Nth time slot, the starting coordinate position of the drone, and the ending coordinate position of the drone. min , a max They represent the minimum acceleration and maximum acceleration of the drone respectively. v i [n] represents the UAV acceleration vector and unit velocity vector, Q i [n] represents the coordinate position of the UAV at the nth time slot, v i [n] represents the speed of the UAV at the nth time slot, δ t Indicates time slot, E max is the initial energy of the drone, E i [N] represents the remaining energy of the drone in the Nth time slot, v max Indicates the maximum speed of the drone. i [n] represents the flight altitude of the drone, z min , z max They represent the minimum flight altitude and the maximum flight altitude of the drone, respectively. D represents the minimum flight safety distance between drones. i [n], They represent the transmission power of the UAV in the nth time slot and the maximum transmission power of the UAV, respectively. V represents the coordinate position of the Kth ground mobile user in the nth time slot, the coordinate position of the Kth ground mobile user in the (n+1)th time slot, and the moving speed of the ground user.

3. The multi-UAV trajectory optimization method for user mobile communication in a heterogeneous network according to claim 2 is characterized by: In the optimization mathematical model Indicates the nth time slot S and U k The scheduling indicators between represents the fairness factor among terrestrial users in the nth time slot, It represents the total confidentiality rate of the ground user in the nth time slot, N and K represent the Nth time slot and the Kth user respectively.

4. The multi-UAV trajectory optimization method for user mobile communication in a heterogeneous network according to claim 1, 2 or 3, characterized in that: In step (3), POMDP is used to model the system. Each drone is considered as an agent, which is regarded as POMDP and (S, A, R, P, γ), where S is the state set describing the global environment, A is the joint action set of all agents, R is a set of reward functions that map states and agent behaviors to rewards, P is the probabilistic state transition function that describes the next state of all agents after taking joint actions in the current state, γ is the reward discount factor, and the observation set of all agents is defined as O{O s ,O J }, let o={o s ,o J } is defined as the observable state of the local environment, a={a s ,a J } is defined as an observable action, r = {r s ,r J } is defined as each agent will receive a corresponding reward, and then the environment will transition to the next state according to the state transfer function P, o s ,o J , a s ,a J , r s ,r J They represent the observable input states of the communication UAV and the jamming UAV, the output actions of the communication UAV and the jamming UAV, and the reward functions of the communication UAV and the jamming UAV respectively.

5. The multi-UAV trajectory optimization method for user mobile communication in a heterogeneous network according to claim 4 is characterized by: In step (3), the MADDPG model requires state information as input to estimate the Q value of the action. The joint optimization process involves two types of drones with different functions, namely communication drones and jamming drones, so the drone position coordinates are Q S [n],Q J [n],Q E [n] represents the coordinate position of the communication drone, the coordinate position of the jamming drone, and the coordinate position of the eavesdropping drone. In this case, the state information includes the user channel information. Therefore, the user channel information is also the basis for determining the power allocation of all users. The partial observation state space of S consists of the drone position, speed, remaining energy consumption, Euclidean distance to the end point, fairness index, cumulative confidentiality rate of S relative to all ground users, and user channel information. Therefore, the partial observation state space of S is expressed as It can be observed that S has a total of 2K+12 state dimensions; v S [n],E S [n], They represent the communication UAV speed, the communication UAV energy consumption, the fairness index, the total safety rate between the communication UAV and the ground mobile users, and the channel gain between the communication UAV and the ground mobile users respectively. Indicates the final position of the communication drone. The state space of J consists of the position, speed, remaining energy consumption, and Euclidean distance to the end point of the drone. Therefore, the partial observation state space of J is expressed as It can be observed that J has a total of 11 state dimensions, v J [n],E J [n], They represent the interference with the speed of the UAV, the interference with the energy consumption of the UAV, and the interference with the final position of the UAV respectively. It should be noted that communication drones and jamming drones are regarded as two types of heterogeneous multi-agents. The state space of each agent contains the location information of all drones. The state space of each agent only contains its own flight speed to the destination, its own energy consumption and its Euclidean distance to the destination. In order to protect the security of ground users during communication, the state space of J does not include the fairness index, the cumulative confidentiality rate of all ground users and the user channel information.

6. The multi-UAV trajectory optimization method for user mobile communication in a heterogeneous network according to claim 5, characterized in that: The action space consists of the 3D trajectory of the UAV and the transmission power. Since the 3D trajectory of the UAV can be obtained by iterating the velocity vector, the velocity vector of the UAV is used to represent the action, which is equivalent to representing the trajectory of the UAV. 1) UAV speed: using spherical coordinates Describes the speed and flight direction of the drone, where and denote the polar angle and azimuth respectively, then the velocity can be expressed as The normalized expression of the UAV speed is in, λ v , They represent normalized velocity, normalized polar angle, and normalized azimuth, respectively. 2) UAV power allocation: UAV transmit power action P, Therefore, the action space of S and J is expressed as Observe that action a has a total of 8 action dimensions, They represent the polar angle of the UAV in the nth time slot and the azimuth angle of the UAV in the nth time slot respectively.

7. The multi-UAV trajectory optimization method for user mobile communication in a heterogeneous network according to claim 1 or 6, characterized in that: The reward function in the multi-agent deep reinforcement learning algorithm in step (3) consists of four parts: energy consumption reward, end point arrival reward, fair confidentiality rate reward and constraint violation reward; 1) Energy consumption reward is defined as The subscript "ec" represents energy consumption, κ ec is a normal quantity used to adjust the reward size of the energy consumption part; 2) Reward for reaching the end point: In order to describe the state where the drone is about to run out of energy and must return to the end point, a binary indicator is defined E is found in the formula i [n] represents the energy consumption of the nth time slot, E i,min is the energy threshold. In order to maximize the fair confidentiality rate of the UAV, based on the relationship between the Euclidean distance to the end point and the energy threshold, the adaptive energy threshold is defined as Where v mr It represents the optimal speed when the energy consumption per unit distance is the minimum, and E0 is the energy compensation constant; Therefore, the continuous reward endpoints of multiple drones are defined, when represents the difference in distance between the drone and the end point after it moves for one time slot, κ rd1 , κ rd2 and κ rd3 are constants used to adjust the incremental distance reward, distance reward, and the slope of reward with distance, respectively; i,rd When [n] = 1, the continuous reward associated with the end point is given by κ rd1 Δd i and The former represents the incremental distance reward for flying to the end point, and the latter represents the distance reward of the drone relative to the end point. Δd i It represents the difference between the coordinate positions of the drone in two adjacent time slots. In order to indicate whether the drone reaches the destination when the energy is exhausted, the discontinuous arrival reward (r iar [n],i∈{S,J}) is Among them, ξ i,ar =1 means that the drone reaches the destination in the last time slot, otherwise, ξ i,ar =0,κ ar is a normal quantity, motivating the drone to reach the end point, κ nar is a negative constant that penalizes drones that fail to reach the end point; 3) Fair confidentiality rate reward: In the nth time slot, the fair confidentiality rate reward is defined as Among them, κ th represents the reward when the energy is sufficient, κ nth indicates that the flight energy is about to run out and the reward is needed to return to the end point; it is worth noting that when κ th >κ nth , because when the drone's energy is about to run out, the reward decreases, making the drone more inclined to return to the end point; i,rd [n],R ins [n] represents the binary variable when the energy is sufficient or the energy is about to be exhausted, and the fair confidentiality rate hyperparameter. 4) Constraint violation reward: To punish the drone for violating the constraints given by distance, height and acceleration, the following reward is defined: Among them, r i,L =1, r i,H =1 and r i,A =1, indicating that the distance constraint, height constraint and acceleration constraint are not satisfied, and the negative constant κ L , κ H and κ A They represent the reward constants for violating the distance constraint, height constraint, and acceleration constraint respectively; ξ i,L [n],ξ i,H [n],ξ i,A [n] represents the distance hyperparameter, height hyperparameter, and acceleration hyperparameter respectively. In summary, the total reward function of S and J is expressed as 8. The method for optimizing multi-UAV trajectories for user mobile communications in a heterogeneous network according to claim 7, characterized in that: In the multi-agent deep reinforcement learning algorithm, S and J are regarded as heterogeneous agents, and the communication system under consideration is regarded as the environment. The goal of each agent is to select the best action based on its experience to maximize the reward accumulated over time. Each agent has its own Actor network, Critic network and corresponding Target network. The Actor network of each agent uses independent observations as input to generate actions, and the Critic network of each agent uses the observations and actions of all agents as input. The Target network calculates the loss of the Actor and Critic networks.

Images