Unmanned aerial vehicle cluster relay system optimization method

Abstract

The invention discloses an unmanned aerial vehicle cluster relay system optimization method, and aims to significantly reduce the overall outage probability of the system by performing adaptive optimization on cooperative position deployment and flight trajectories of multiple unmanned aerial vehicles under multiple actual constraint conditions of flight speed, safety distance, flight boundary and the like. Therefore, the reliability and the service quality of the multi-unmanned aerial vehicle relay communication network are improved. According to the method, a Markov decision process (MDP) model is constructed, a multi-unmanned aerial vehicle trajectory optimization problem is converted into a multi-agent reinforcement learning problem, and efficient optimization of unmanned aerial vehicle trajectories is realized by using a centralized training-distributed execution framework, so that the outage probability of a communication system is minimized.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, specifically relating to an optimization method for a drone swarm relay system. Background Technology

[0002] With the rapid development of drone technology, drone swarm collaborative communication systems have shown great potential in fields such as emergency communication, the Internet of Things, and military reconnaissance due to their wide coverage, high mission flexibility, and strong fault tolerance. However, the dynamic deployment and trajectory optimization of multi-drone systems face many challenges, including the complexity of resource allocation, the difficulty of coordination among drones, and high requirements for environmental adaptability.

[0003] Existing technologies often employ deep reinforcement learning to optimize UAV swarm deployment and user association, or focus on researching collaborative detection systems for UAV swarms based on joint radar communication, or optimize multi-edge cloud collaborative mobile edge computing systems for UAV swarms. However, none of these methods adequately consider the problem of optimizing the probability of interruption in lossy communication scenarios. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides an optimization method for a UAV swarm relay system. The aim is to significantly reduce the overall outage probability of the system by adaptively optimizing the coordinated deployment and flight trajectories of multiple UAVs under multiple practical constraints such as flight speed, safe distance, and flight boundaries, thereby improving the reliability and quality of service of the multi-UAV relay communication network. This method transforms the multi-UAV trajectory optimization problem into a multi-agent reinforcement learning problem by constructing a Markov Decision Process (MDP) model. Utilizing a centralized training-distributed execution framework, it achieves efficient optimization of UAV trajectories, thereby minimizing the outage probability of the communication system.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows:

[0006] Step 1: Construct a lossy cooperative multi-UAV relay communication system model, including a base station, multiple UAVs and multiple ground users, and define the flight constraints of the UAVs and the communication interruption probability calculation model.

[0007] Step 2: Model the multi-UAV trajectory optimization problem as a Markov decision process, defining the state space, action space, and reward function. The state space is the set of UAV position coordinates, the action space is the change in UAV position, and the reward function is designed based on the communication interruption probability and constraint violation penalty term.

[0008] Step 3: Employ the MADDPG (Multi-Agent Deep Reinforcement Learning) algorithm for centralized training and distributed execution; each UAV acts as an agent, equipped with a local policy network and evaluation network, learning the optimal trajectory policy through interaction with the environment, ultimately minimizing the probability of interruption.

[0009] Furthermore, step 1 specifically includes:

[0010] Step 1-1: Construct a dual-timeslot cooperative communication system model that includes ground base stations, several relay UAVs, and multiple ground users, using an air-to-ground channel model that considers line-of-sight probability;

[0011] For a lossy cooperative UAV relay communication system consisting of one base station, J UAVs, and K users, where the UAVs act as aerial relays to assist communication between the base station and the users; in the first time slot, the base station sends the original information sequence to the users and the UAVs, and in the second time slot, the UAVs send the auxiliary information sequence, which is related to the original information, to the users; it is assumed that each user is served by at most one UAV, and different users served by the same UAV use FDMA multiple access mode; each UAV shares information with each other through a central controller and reuses the entire bandwidth resource;

[0012] In a three-dimensional Cartesian coordinate system, the position of ground user k is represented as... The horizontal coordinate of the j-th UAV is represented as: The drone altitude is fixed; the initial position of the j-th drone is... The constraints that each drone needs to meet are as follows:

[0013]

[0014] Where v max The maximum flight speed of the drone in the horizontal direction is represented by Δq[t] = (Δx) which limits the coordinate change of the drone in the horizontal position. q [t],Δy q [t]) to achieve; q j [t] represents the horizontal coordinate of the j-th UAV at time t; d min This represents the minimum distance between drones to prevent collisions. It requires that at any time t, the square of the horizontal distance between any two drones at the next time t+1 must be greater than or equal to... x min x max y min and y max This represents the upper and lower limits of the drone's flight range in horizontal coordinates.

[0015] Air-to-ground channel modeling uses an elevation-angle-based probabilistic line-of-sight channel model for description, then the line-of-sight link probability... Non-line-of-sight link probability Line-of-sight link path loss Non-line-of-sight link path loss They can be represented as:

[0016]

[0017]

[0018] in, These are environmental parameters, set according to the actual communication scenario. ; θ j,k η represents the elevation angle between the transmitting and receiving ends of the link. LoS η NLos f represents the additional path loss. k Indicates the carrier frequency, d j,k represents the distance between the sending and receiving ends of the link, and c represents the speed of light in a vacuum;

[0019] The average path loss between drone j and the k-th user it serves can be expressed as:

[0020]

[0021] Given the transmit power P of the UAV UAV Then the signal power received by the k-th user from the j-th drone is... Represented as:

[0022]

[0023] Among them, |g| 2 Indicates channel fading gain;

[0024] The instantaneous signal-to-noise ratio of the link between the k-th user and the j-th drone can be expressed as:

[0025]

[0026] Where N o This represents the noise power spectral density of additive Gaussian noise;

[0027] Therefore, the expression for the interruption probability of the k-th user is:

[0028]

[0029] in Representation function ρ(γ) represents the probability that the SNR in the channel does not meet the requirements, m represents the channel fading parameter, and ρ(γ) represents the probability that the SNR in the channel does not meet the requirements. 2,k) represents the crossover probability between the compressed information sequence relayed by the UAV to the user link and its lossy compressed sequence, Dk represents the distortion contained in the information sequence finally recovered by the user, and Hb(·) represents the binary entropy function;

[0030] Steps 1-2: Construct a multi-UAV relay location deployment and trajectory optimization problem with the goal of minimizing the system outage probability;

[0031] The optimization problem is defined as follows:

[0032]

[0033] Where u j,k The value constraint for the binary variable is that a user can be served by at most one drone, and N represents the final communication time.

[0034] Furthermore, step 2 is specifically as follows:

[0035] The optimization of the outage probability of a multi-UAV relay system is abstracted into a Markov decision process, which includes the following three parts:

[0036] (1) State Space: For the j-th UAV relay, its position information has a decisive impact on the performance of the communication system. Therefore, the state space of the agent is represented as the set of position coordinates of all UAV relays, i.e. in This indicates the location information of the j-th UAV;

[0037] (2) Action Space: Taking the j-th UAV as an example, the action space is defined as the change in position of the UAV, i.e. in By limiting To constrain the flight speed of drones, These represent the range of lateral and longitudinal movement of the j-th UAV in the horizontal coordinate system, respectively.

[0038] (3) Reward Function: The reward function consists of three parts; the first part, the reward function R(t), is set based on the interruption probability of the communication system; the second part... Part Three The reward is a negative number; when the drone flies out of the boundary or collides, it is penalized as a penalty. Ultimately, the reward function is defined as:

[0039]

[0040] Where b1, b2, and b3 represent the weighting coefficients for different rewards;

[0041] Let the total number of users served by the j-th drone be K. j Specifically, R(t) is defined as:

[0042]

[0043] Furthermore, step 3 specifically includes:

[0044] Step 3-1: Equip each UAV with an Actor policy network and a Critic evaluation network, and perform centralized training and distributed execution using the MADDPG algorithm;

[0045] Each agent is equipped with a local policy network and a local value network, that is, the j-th agent has an Actor policy network, a Critic evaluation network, and corresponding Actor target policy network and Critic target evaluation network.

[0046] Actor Policy Network: This network takes the current state vector of the j-th agent as input and outputs the optimal action decision; Actor Target Policy Network takes the next state of the agent as input and outputs the optimal action of the j-th agent in the next state, which is used for subsequent network training and updates.

[0047] The Critic evaluation network takes not only the state and action of the j-th agent as input, but also, in multi-agent learning, the state and action of all agents as input, and outputs the Q-value of the state-action pair to evaluate the expected reward of making the action decision in the current state. The Critic target policy network takes the next state and action of all agents as input, and outputs the Q-value of the state-action pair in the next state to calculate the target Q-value.

[0048] Set up an experience pool to store historical sample data for network training and updates;

[0049] Step 3-2: Update the network and optimize network parameters through empirical replay and gradient descent;

[0050] First, a batch of samples is drawn from the experience pool. as well as It contains the states of all agents; 'a' represents the action performed by the agent, and 'r' represents the reward the agent receives for performing action 'a' in state s.

[0051] The online update method is as follows:

[0052] Actor Policy Network: For the Actor policy network of the j-th agent, firstly, extract the state s corresponding to the j-th agent from the sample. J Input the current Actor policy network to obtain the deterministic continuous action μ output by the network.j (a j |s j ), a j This represents the action of the j-th agent; to simplify the writing, each formula will include the parameter θ. j Actor policy network Abbreviated as μ j Simultaneously, the centralized Critic evaluation network receives the states and actions of all agents, i.e., s, a1, ..., a J As input, the state-action Q-value is obtained. The gradient of the loss function required for updating the Actor policy network. Represented as:

[0053]

[0054] in, This represents the mathematical expectation based on randomly sampled data. The output μ of the Actor policy network represents... j Regarding network parameter θ j gradient, This indicates that the Q-value of the Critic network output is related to action a. j The gradient, μ j (s j ) indicates that the Actor policy network is in state s j Deterministic actions that output the next action;

[0055] Critic evaluation network: For the j-th agent, the target value for network update needs to be calculated first, and the sampled values ​​will be used as the basis for the network update. The target actions of each agent are obtained by inputting them into the Actor target policy network. After that Inputting the target evaluation network into the Critic network yields the target value function. The obtained objective function is compared with the reward r of the j-th agent. j Construct the objective function y j as follows:

[0056]

[0057] Where η represents the discount factor;

[0058] Use the obtained state to act on the Q value and objective function y j The loss function needed to construct and update the Critic evaluation network.

[0059]

[0060] in, This represents the mathematical expectation based on randomly sampled data;

[0061] Target network: Updates to the target network still use a soft update method; the update method is as follows:

[0062]

[0063] Where τ represents the soft update coefficient, which controls the magnitude of the target network parameter update; This represents the target network parameters for the j-th agent.

[0064] Preferably, the flight constraints of the UAV include initial position, maximum flight speed, flight range, and collision avoidance constraints.

[0065] An electronic device includes: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described relay system interruption probability optimization method.

[0066] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described relay system interruption probability optimization method.

[0067] A chip includes a processor for retrieving and running a computer program from a memory, causing a device equipped with the chip to perform the aforementioned relay system interruption probability optimization method.

[0068] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described relay system interruption probability optimization method.

[0069] The beneficial effects of this invention are as follows:

[0070] This invention models the trajectory optimization problem of multiple UAVs as a Markov decision process and designs a multi-agent deep reinforcement learning (MADDPG)-based multi-UAV trajectory optimization scheme. This scheme can calculate a reasonable flight trajectory and optimal deployment location for each UAV, ensuring that the total probability of user downtime is effectively reduced while satisfying all constraints. Attached Figure Description

[0071] Figure 1 It is a lossy collaborative multi-UAV relay system model.

[0072] Figure 2This is a graph showing the impact of different learning rate parameter values ​​on the algorithm.

[0073] Figure 3 It is a graph showing the change in the probability of interruption.

[0074] Figure 4 It is a multi-drone trajectory map. Detailed Implementation

[0075] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0076] This invention provides an optimization method for a drone swarm relay system. Under multiple practical constraints such as flight speed, safe distance, and flight boundaries, the method adaptively optimizes the coordinated deployment and flight trajectories of multiple drones to significantly reduce the overall outage probability, thereby improving the reliability and quality of service of the multi-drone relay communication network. This method transforms the multi-drone trajectory optimization problem into a multi-agent reinforcement learning problem by constructing a Markov Decision Process (MDP) model. Utilizing a centralized training-distributed execution framework, it achieves efficient optimization of drone trajectories, thereby minimizing the outage probability of the communication system.

[0077] Step S1: Construct a lossy cooperative multi-UAV relay communication system model, including a base station, multiple UAVs and multiple ground users, and define the flight constraints of the UAVs (such as initial position, maximum flight speed, flight range and collision avoidance constraints) and the communication interruption probability calculation model.

[0078] Step S101: Construct a dual-timeslot cooperative communication system model that includes ground base stations, several relay UAVs, and multiple ground users, using an air-to-ground channel model that considers line-of-sight probability.

[0079] For a lossy cooperative UAV relay communication system consisting of one base station, J UAVs, and K users, where the UAVs act as aerial relays to assist communication between the base station and the users; in the first time slot, the base station sends the original information sequence to the users and the UAVs, and in the second time slot, the UAVs send the auxiliary information sequence, which is related to the original information, to the users; it is assumed that each user is served by at most one UAV, and different users served by the same UAV use FDMA multiple access mode; each UAV shares information with each other through a central controller and reuses the entire bandwidth resource;

[0080] In a three-dimensional Cartesian coordinate system, the position of ground user k can be represented as: The horizontal coordinate of the j-th UAV is represented as: The drone altitude is fixed. The initial position of the j-th drone is... In the communication system model, each UAV needs to satisfy various constraints, such as initial position, maximum flight speed, flight range, and collision avoidance. These constraints are described as follows:

[0081]

[0082] Where v max The maximum flight speed of the drone in the horizontal direction is represented by Δq[t] = (Δx) which limits the coordinate change of the drone in the horizontal position. q [t],Δy q [t]) to achieve this. q j [t] represents the horizontal coordinate of the j-th UAV at time t. d min This represents the minimum distance between drones to prevent collisions. It requires that at any time t, the square of the horizontal distance between any two drones at the next time t+1 must be greater than or equal to... x min x max y min and y max This indicates the upper and lower limits of the drone's flight range in horizontal coordinates.

[0083] Air-to-ground channel modeling uses an elevation-angle-based probabilistic line-of-sight channel model for description, then the line-of-sight link probability... Non-line-of-sight link probability Line-of-sight link path loss Non-line-of-sight link path loss They can be represented as:

[0084]

[0085]

[0086] in, These are environmental parameters, set according to the actual communication scenario. θ j,k η represents the elevation angle between the transmitting and receiving ends of the link. LoS η NLos f represents the additional path loss. k Indicates the carrier frequency, d j,k denoted by , where represents the distance between the sending and receiving ends of the link, and c represents the speed of light in a vacuum.

[0087] The average path loss between drone j and the k-th user it serves can be expressed as:

[0088]

[0089] Given the transmit power P of the UAV UAVThen the signal power received by the k-th user from the j-th drone is... Represented as:

[0090]

[0091] Among them, |g| 2 This indicates the channel fading gain.

[0092] The instantaneous signal-to-noise ratio of the link between the k-th user and the j-th drone can be expressed as:

[0093]

[0094] Where N o This represents the noise power spectral density of additive Gaussian noise.

[0095] Therefore, the expression for the interruption probability of the k-th user is:

[0096]

[0097] in Representation function ρ(γ) represents the probability that the SNR in the channel does not meet the requirements, m represents the channel fading parameter, and ρ(γ) represents the probability that the SNR in the channel does not meet the requirements. 2,k ) represents the crossover probability between the compressed information sequence relayed by the UAV to the user link and its lossy compressed sequence, Dk represents the distortion contained in the information sequence finally recovered by the user, and Hb(·) represents the binary entropy function;

[0098] Step S102: Construct a multi-UAV relay location deployment and trajectory optimization problem with the goal of minimizing the system interruption probability.

[0099] The optimization problem is defined as follows:

[0100]

[0101] Where u j,k The value constraint for the binary variable is that a user can be served by at most one drone, and N represents the final communication time.

[0102] Step S2: Model the multi-UAV trajectory optimization problem as a Markov decision process, defining the state space, action space, and reward function. The state space is the set of UAV position coordinates, the action space is the change in UAV position, and the reward function is designed based on the communication interruption probability and constraint violation penalty term.

[0103] The optimization of the outage probability of a multi-UAV relay system is abstracted into a Markov decision process, which includes the following three parts:

[0104] (1) State Space: For the j-th UAV relay, its position information has a decisive impact on the performance of the communication system. Therefore, the state space of the agent is represented as the set of position coordinates of all UAV relays, i.e. in This indicates the location information of the j-th UAV;

[0105] (2) Action Space: Taking the j-th UAV as an example, the action space is defined as the change in position of the UAV, i.e. in By limiting [Δy min ,Δy max To constrain the flight speed of drones, These represent the range of lateral and longitudinal movement of the j-th UAV in the horizontal coordinate system, respectively.

[0106] (3) Reward Function: The reward function consists of three parts. The first part, the reward function R(y), is set based on the interruption probability of the communication system. The second part... Part Three The reward is negative; when the drone flies out of the boundary or collides, a penalty is imposed. Ultimately, the reward function is defined as:

[0107]

[0108] Where b1, b2, and b3 represent the weighting coefficients for different rewards;

[0109] Let the total number of users served by the j-th drone be K. j Specifically, R(t) is defined as:

[0110]

[0111] Step S3: The MADDPG algorithm is used for centralized training and distributed execution. Each UAV acts as an agent, equipped with a local policy network and an evaluation network. It learns the optimal trajectory policy by interacting with the environment, ultimately minimizing the probability of interruption.

[0112] Step S301: Equip each UAV with an Actor policy network and a Critic evaluation network, and perform centralized training and distributed execution using the MADDPG algorithm.

[0113] The trajectory optimization algorithm based on MADDPG proposed in this invention adopts a centralized training-decentralized decision-making learning architecture to solve optimization problems under continuous multi-agent behavior. The MADDPG-based trajectory optimization algorithm uses an Actor-Critic algorithm architecture, where global information is added to the input of the Critic evaluation network, allowing agents to evaluate the value of their current actions by combining the states and actions of other agents, thus enabling interaction and learning among agents in a shared environment. Therefore, each agent is equipped with a local policy network and a local value network; that is, the j-th agent has an Actor policy network, a Critic evaluation network, and corresponding Actor target policy network and Critic target evaluation network. The target network is mainly used to stabilize the learning process, while the local network is used for actual state-action value estimation and policy selection action decision-making. The following sections will introduce each network:

[0114] Actor Policy Network: This network takes the current state vector of the j-th agent as input and outputs the optimal action decision; Actor Target Policy Network takes the next state of the agent as input and outputs the optimal action of the j-th agent in the next state, which is used for subsequent network training and updates.

[0115] The Critic evaluation network takes not only the state and action of the j-th agent as input, but also, in multi-agent learning, the state and action of all agents as input, and outputs the Q-value of the state-action pair to evaluate the expected reward of making the action decision in the current state. The Critic target policy network takes the next state and action of all agents as input, and outputs the Q-value of the state-action pair in the next state to calculate the target Q-value.

[0116] Set up an experience pool to store historical sample data for network training and updates;

[0117] Step S302: Update the network and optimize the network parameters through empirical replay and gradient descent.

[0118] First, a batch of samples is drawn from the experience pool. as well as It contains the states of all agents; 'a' represents the action performed by the agent, and 'r' represents the reward the agent receives for performing action 'a' in state s.

[0119] The online update method is as follows:

[0120] Actor Policy Network: For the Actor policy network of the j-th agent, firstly, extract the state s corresponding to the j-th agent from the sample. J Input the current Actor policy network to obtain the deterministic continuous action μ output by the network.j (a j |s j ), a j This represents the action of the j-th agent; to simplify the writing, each formula will include the parameter θ. j Actor policy network Abbreviated as μ j Meanwhile, the centralized Critic evaluation network receives the states and actions of all agents, i.e., s, a1, ..., a J As input, the state-action Q-value is obtained. The gradient of the loss function required for updating the Actor policy network. Represented as:

[0121]

[0122] in, This represents the mathematical expectation based on randomly sampled data. The output μ of the Actor policy network represents... j Regarding network parameter θ j gradient, This indicates that the Q-value of the Critic network output is related to action a. j The gradient, μ j (s j ) indicates that the Actor policy network is in state s j Deterministic actions that output the next action;

[0123] Critic evaluation network: For the j-th agent, the target value for network update needs to be calculated first, and the sampled values ​​will be used as the basis for the network update. The target actions of each agent are obtained by inputting them into the Actor target policy network. After that Inputting the target evaluation network into the Critic network yields the target value function. The obtained objective function is compared with the reward r of the j-th agent. j Construct the objective function y j as follows:

[0124]

[0125] Where η represents the discount factor;

[0126] Use the obtained state to act on the Q value and objective function y j The loss function needed to construct and update the Critic evaluation network.

[0127]

[0128] in, This represents the mathematical expectation based on randomly sampled data;

[0129] Target network: Updates to the target network still use a soft update method. The update method is as follows:

[0130]

[0131] Where τ represents the soft update coefficient, which controls the magnitude of the target network parameter update; This represents the target network parameters for the j-th agent.

[0132] Example:

[0133] In the simulation analysis of this invention, a lossy cooperative UAV relay communication network consisting of j=2 UAVs and K=4 users was constructed. The simulation environment of this network was also set within a 20×20km area. The flight altitude of the UAVs was set to a fixed value of 300 meters. To avoid collisions, the initial positions of the two UAVs were set to (1,0) and (0,1) respectively. The specific positions of each user were predetermined and known, and the carrier frequencies were set to f1=1.970Hz, f2=1.985Hz, f3=2.000Hz, and f4=2.015Hz respectively. The UAV transmission power P UAV Set it to 0.01W.

[0134] Table 1 shows the overall process framework.

[0135] Table 1 Method Flow Framework

[0136]

[0137]

[0138] Figure 2 Simulation results show the algorithm's performance under different learning rate configurations. It can be observed that the choice of learning rate has a significant impact on the algorithm's performance. Specifically, when the learning rate of the Critic evaluation network is reduced, the convergence speed of the algorithm slows down significantly. This may be because a learning rate that is too small leads to overly conservative network weight updates, thus slowing down the learning process. On the other hand, when the learning rate of the Actor policy network is increased, the algorithm's performance not only fails to improve but also shows a significant decrease, even causing the algorithm to fail to converge. This may be because a learning rate that is too large leads to overly drastic network weight updates, making it difficult for the algorithm to stably learn effective policies. Furthermore, the learning rate of the Critic evaluation network should be set slightly larger than that of the Actor policy network to ensure policy stability while rapidly responding to environmental changes.

[0139] Similarly, the probability of interruption is observed. Figure 3 This further demonstrates that as the number of training epochs increases, the total user interruption probability gradually decreases and tends to converge. Meanwhile, the user interruption probability still shows a negative correlation with the user's distortion requirement. When different users have different distortion requirements, the interruption probability is mainly affected by the user with the strictest distortion requirement.

[0140] Figure 4 This diagram illustrates the final trajectories of each drone in a multi-drone relay communication system scenario. The diagram shows that the drones start from their respective initial positions and move rapidly and efficiently to their optimal positions that minimize the probability of communication interruption. Throughout the communication process, the two drones maintain a safe distance and do not collide. After reaching their optimal positions, the drones remain nearby to ensure the stability and reliability of the communication.

Claims

1. An optimization method for a drone swarm relay system, characterized in that, The method comprises the following steps: Step 1: constructing a lossy cooperative multi-unmanned aerial vehicle relay communication system model, including a base station, multiple unmanned aerial vehicles and multiple ground users, defining flight constraint conditions of the unmanned aerial vehicles and a communication interruption probability calculation model; Step 2: modeling the multi-unmanned aerial vehicle trajectory optimization problem as a Markov decision process, defining a state space, an action space and a reward function; the state space is a set of position coordinates of the unmanned aerial vehicles, the action space is a position change amount of the unmanned aerial vehicles, and the reward function is designed based on the communication interruption probability and a constraint violation penalty term; Step 3: using a multi-agent deep reinforcement learning MADDPG algorithm for centralized training and distributed execution; each unmanned aerial vehicle is regarded as an agent, equipped with a local policy network and an evaluation network, and learns an optimal trajectory policy through interaction with the environment, finally realizing minimization of the interruption probability. 2.The method of claim 1, wherein, The step 1 specifically comprises: Step 1-1: constructing a two-time-slot cooperative communication system model comprising a ground base station, a plurality of relay unmanned aerial vehicles and a plurality of ground users, and using a line-of-sight probability considered air-to-ground channel model; For a lossy cooperative unmanned aerial vehicle relay communication system comprising one base station, J unmanned aerial vehicles and K users, wherein the unmanned aerial vehicles serve as air relays to assist communication between the base station and the users; in a first time slot, the base station transmits an original information sequence to the users and the unmanned aerial vehicles, and in a second time slot, the unmanned aerial vehicles transmit an auxiliary information sequence having correlation with the original information sequence to the users; one user is served by at most one UAV, and different users served by the same UAV use a FDMA multiple access mode; the unmanned aerial vehicles share information with each other through a central controller and reuse the entire bandwidth resource; The position of the ground user k is expressed as The horizontal coordinate of the jth unmanned aerial vehicle is expressed as The fixed unmanned aerial vehicle height; the initial position of the jth unmanned aerial vehicle is The constraints that each unmanned aerial vehicle needs to meet are as follows: where v max represents the maximum flight speed of the UAV in the horizontal direction, which will be achieved by limiting the coordinate change of the UAV in the horizontal position Δq[t] = (Δx q [t], Δy q [t]) at time t; q j [t] represents the horizontal coordinate of the jth UAV at time t; d min represents the minimum distance between UAVs to prevent collision, which requires that at any time t, the square of the horizontal position distance between any two UAVs at the next time t+1 must be greater than or equal to x min , x max , y min , and y max represent the upper and lower limits of the flight range boundary of the UAV in the horizontal coordinate. The air-to-ground channel modeling adopts an elevation-based probabilistic line-of-sight channel model to describe, then the line-of-sight link probability The non-line-of-sight link probability The line-of-sight link path loss And the non-line-of-sight link path loss Can be represented as: wherein, is an environmental parameter, which is set according to the actual communication scenario; j,k denotes the elevation angle between the transmitting end and the receiving end of the link, η LoS , η NLos denotes the additional path loss, f k denotes the carrier frequency, d j,k denotes the distance between the transmitting end and the receiving end of the link, c denotes the speed of light in a vacuum; The average path loss between the unmanned aerial vehicle j and the kth user served by the unmanned aerial vehicle j can be expressed as: Let P be the transmit power of a given drone UAV Then the kth user receives the signal power from the jth drone is expressed as: where |g| < 1 2 denotes the channel fading gain; The link instantaneous signal-to-noise ratio between the kth user and the jth unmanned aerial vehicle can be expressed as: where N o denotes the noise power spectral density of the additive Gaussian noise; Therefore, the expression of the interruption probability of the kth user is: wherein denotes a function denotes the probability that the SNR in the channel does not meet the requirement, m denotes the channel fading parameter, ρ(γ 2,k ) denotes the cross probability of the compressed information sequence of the UAV relay-to-user link and its lossy compressed sequence, Dk denotes the distortion degree contained in the final recovered information sequence of the user end, Hb(·) denotes a binary entropy function; Step 1-2: constructing a multi-unmanned aerial vehicle relay position deployment and trajectory optimization problem with the optimization objective of minimizing the system interruption probability; The optimization problem is established as follows: where u j,k is a binary variable constraint that indicates that a user can be served by at most one drone, and N denotes the final time of communication. 3.The method of claim 2, wherein, The step 2 specifically comprises: The multi-unmanned aerial vehicle relay system interruption probability optimization is abstracted as a Markov decision process, comprising the following three parts: (1) State space: For the jth UAV relay, its position information has a decisive influence on the performance of the communication system, so the state space of the agent is represented as the set of position coordinates of all UAV relays, that is, wherein represents the position information of the jth UAV. (2) Action space: Taking the jth UAV as an example, the action space is defined as the position change of the UAV, that is Wherein By limiting To constrain the flight speed of the UAV, respectively represent the horizontal coordinate of the jth UAV horizontal and longitudinal movement change range; (3) Reward function: The reward function is divided into three parts; the first part of the reward function R(t) is set based on the outage probability of the communication system, the second part Third part is negative reward, when the UAV flies out of the boundary range or collision occurs, as a punishment item to give punishment; finally, the reward function is defined as: Wherein, b1, b2 and b3 represent weight coefficients of different rewards; Let the total number of users served by the jth drone be K j In particular, R(t) is defined as:

4. The method of claim 3, wherein, Further, the step 3 specifically comprises: Step 3-1: each unmanned aerial vehicle is equipped with an Actor policy network and a Critic evaluation network, and centralized training and distributed execution are performed through the MADDPG algorithm; Each agent is equipped with a local policy network and a local value network, that is, the jth agent has an Actor policy network, a Critic evaluation network and corresponding Actor target policy network and Critic target evaluation network; Actor policy network: the network inputs a state vector of the current jth agent, and outputs a determined optimal action decision; the Actor target policy network inputs a next state of the agent, and outputs an optimal action of the jth agent in the next state, which is used for subsequent network training and updating; Critic evaluation network: the input of the network is not only the state and action of the jth agent, but also the state of all agents and the action of all agents under the learning of multi-agent, and the output is the Q value of the state-action pair in the current state for evaluating the expected return of the decision-making action in the current state; the input of the Critic target policy network is the next state of all agents and the action in the next state, and the output is the Q value of the state-action pair in the next state for calculating the target Q value; An experience pool is set to store historical sample data for network training and updating; Step 3-2: update the network and optimize the network parameters through experience replay and gradient descent; First, a batch of samples is drawn from the experience pool s = (s1,..., s J ), and contains the states of all agents; a represents the action performed by the agent, and r represents the reward obtained by the agent when performing action a in state s. The network updating method is as follows: Actor Policy Network: For the Actor policy network of the j-th agent, firstly, extract the state s corresponding to the j-th agent from the sample. J Input the current Actor policy network to obtain the deterministic continuous action μ output by the network. j (a j |s j ), a j This represents the action of the j-th agent; to simplify the writing, each formula will include the parameter θ. j Actor Policy Network Abbreviated as μ j Simultaneously, the centralized Critic evaluation network receives the states and actions of all agents, i.e., s, a1, ..., a J As input, the state-action Q-value is obtained. The gradient of the loss function required for updating the Actor policy network. Represented as: wherein, denotes the mathematical expectation based on the randomly sampled data, denotes the gradient of the Actor policy network output μ j with respect to the network parameters θ j , denotes the gradient of the Critic network output Q value with respect to the action a j , j (s j ) denotes the deterministic action output by the Actor policy network in state s j ; Critic evaluation network: for the jth agent, first need to calculate the target value of network update, the target value of network update is calculated by the extracted sample Input the actor target policy network to get the output target action of each agent Then input the critic target evaluation network to get the target value function Input the critic target evaluation network to get the target value function The target value function obtained is combined with the reward r j Construct the target function y j As follows: Wherein, η represents a discount factor; Utilizing the state-action Q-value and the target function y j Constructing the loss function needed by the update Critic evaluation network wherein denotes the mathematical expectation based on the randomly sampled data; Target network: the update of the target network still adopts a soft update method; the update method is as follows: Wherein τ represents a soft update coefficient, controlling the amplitude of the target network parameter update; represents the target network parameter of the jth agent.

5. The method of claim 4, wherein, The flight constraint conditions of the unmanned aerial vehicle include an initial position, a maximum flight speed, a flight range and an anti-collision constraint.

6. An electronic device, comprising: Comprise: A processor and a memory; The memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, so that the electronic device executes the method in any one of claims 1 to 5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 5.

8. A chip, characterized by Comprise: A processor is used for calling and running a computer program from a memory, so that a device installed with the chip executes the method in any one of claims 1 to 5.

9. A computer program product, characterised in that, The computer program product comprises a computer storage medium, the computer storage medium stores a computer program, and the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method in any one of claims 1 to 5 is implemented.

Images