Track planning method for auxiliary data transmission of multiple unmanned aerial vehicles

Abstract

The invention relates to a trajectory planning method for auxiliary data transmission of multiple unmanned aerial vehicles, and belongs to the technical field of communication. According to the method, firstly, a scene comprising a plurality of unmanned aerial vehicles, ground users and ground interference sources is built; then, aiming at the defects of a traditional trajectory planning method in a complex unmanned aerial vehicle environment and the problems of insufficient exploration, slow convergence, poor stability and the like faced by deep reinforcement learning in a multi-agent environment, a trajectory planning method is established; the invention provides a pre-training multi-agent double-delay depth deterministic strategy gradient algorithm based on a centralized training distributed execution framework. According to the algorithm, firstly, pre-training is performed in a single-agent environment, so that the exploration difficulty is reduced; then, in a multi-agent training stage, a pre-training action network is imported, and strategy convergence is accelerated and learning stability is improved through centralized training and distributed execution strategies. According to the method, the trajectory planning of the unmanned aerial vehicle group is effectively optimized, and the data transmission efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology and relates to a trajectory planning method for multi-unmanned aerial vehicle assisted data transmission. Background Art

[0002] In the field of wireless communications, multiple unmanned aerial vehicles (UAV) systems are widely used in tasks such as data transmission and network coverage. In recent years, the concept of UAV as a mobile base station has received increasing attention. In this mode, UAV provides services to ground users (GU) through wireless communication technology.

[0003] In practical applications, due to the movement of GU and the presence of ground interference sources, UAVs must constantly adjust their flight trajectories to optimize data transmission efficiency. Therefore, trajectory planning of UAVs is crucial to the data transmission rate. However, traditional trajectory planning algorithms face many challenges, especially in dynamic and heavily interfered environments. Deep Reinforcement Learning (DRL), as an advanced intelligent algorithm, has been applied to trajectory planning problems. However, in multi-agent environments, DRL has problems such as insufficient exploration, slow convergence, and poor stability, which limit its application in complex UAV swarm systems.

[0004] In order to solve these problems, the present invention proposes a pre-trained multi-agent twin-delayed deep deterministic policy gradient algorithm (PRE-MATD3) based on the centralized training with decentralized execution (CTDE) framework. The algorithm provides an effective initial strategy by pre-training and saving the action network in a single-agent environment, and imports the pre-trained action network in the multi-agent training stage. Through centralized training and distributed execution, the convergence speed of training and the stability of the system are significantly improved. This method can better adapt to complex multi-UAV environments, improve the efficiency of trajectory planning and data transmission rate, and overcome many limitations of existing methods. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a trajectory planning method for multi-UAV assisted data transmission. It aims to solve the shortcomings of trajectory planning of multi-UAV systems in dynamic environments in the prior art. This method significantly improves the efficiency of trajectory planning, data transmission rate and system stability by combining a centralized training distributed execution framework and a pre-trained multi-agent dual-delay deep deterministic policy gradient algorithm.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] In the first aspect, the present invention establishes a communication scenario including a drone, a ground user, and an unknown interference source. In this scenario, there are M drones, K ground users, and N ground interference sources. The ground users are clustered into M clusters, each of which is associated with a drone. In addition, the optimal hovering position of the drone is set as the centroid of its associated user cluster.

[0008] In the second aspect, when constructing the communication link between the UAV and the ground user, the wireless channel adopts Line of Sight (LoS) and Non-Line of Sight (NLoS). Each UAV uses Orthogonal Frequency Division Multiple Access to provide services to its associated ground users. In addition, the ground interference source continuously affects the communication between the UAV and the ground user.

[0009] In a third aspect, the present invention proposes a PRE-MATD3 algorithm based on the CTDE framework. The algorithm comprises the following steps:

[0010] Single-agent pre-training: Pre-train the algorithm in a single-agent environment and save the trained action network. This stage uses the TD3 algorithm to train to obtain an effective initial behavior strategy.

[0011] Multi-agent training: Import the saved pre-trained action network into the multi-agent model and use the MATD3 algorithm for multi-agent training. During the multi-agent training phase, the convergence speed of the strategy and the stability of the system are enhanced through centralized training and distributed execution.

[0012] Centralized training and distributed execution framework: Global state information is collected during centralized training, while in the distributed execution phase, each drone makes independent decisions based on the local state, thereby optimizing the trajectory planning of the drone swarm.

[0013] The beneficial effects of the present invention are as follows: on the basis of pre-training, the method of the present invention combines the knowledge of centralized training distributed execution framework and multi-agent deep reinforcement learning, effectively optimizes the trajectory planning of drone swarms, improves data transmission efficiency and system stability, and overcomes many limitations in the prior art. The method is particularly suitable for scenarios where drones act as mobile base stations to provide communication services to ground users, and can automatically adjust the flight trajectory of drones in a dynamic environment to improve the overall performance of the system.

[0014] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0016] Figure 1 This is a model diagram of a multi-UAV-assisted data transmission system in the present invention;

[0017] Figure 2 It is a framework diagram of the PRE-MATD3 algorithm described in the present invention;

[0018] Figure 3 Flowchart of trajectory planning based on multi-UAV assisted data transmission. DETAILED DESCRIPTION

[0019] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0020] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0021] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0022] Figure 1 FIG. 2 shows a schematic diagram of the structure of a drone swarm assisted data transmission system according to an embodiment of the present invention. Figure 1 As shown in the figure, M drones, K ground users and N ground interference sources are deployed in a square area with a side length of L meters. The ground users are divided into M clusters, and each drone is responsible for covering a user cluster and transmitting data for it. In the system, the ground interference source will interfere with the data transmission between the drone and the ground users. To this end, the drone optimizes the flight trajectory by executing a pre-trained multi-agent dual-delay deep deterministic policy gradient algorithm based on a centralized training distributed execution framework, avoiding interference sources while maintaining efficient communication connections with ground users, thereby improving the data transmission rate.

[0023] by Figure 1 The lower left boundary of the middle area is the coordinate axis, and a three-dimensional Cartesian coordinate system is established. M drones are deployed in the area at different speeds V u m ,m∈M flies at a height H, denoted as M={1,2,…,m,…,M}, then the coordinates of UAV m are q m =(x m ,y m ,H). K ground users travel at a fixed speed V g Move, denoted as K = {1, 2, ·…·, k, ·…·, K}, then the coordinates of ground user k are q k =(x k ,y k ,0). In addition, N randomly distributed ground interference sources continuously affect the data transmission of the UAV to the ground user, denoted as N = {1,2,…,n,…,N}, then the coordinates of the interference source n are q n =(x n ,y n ,0).

[0024] Based on the above scenario, the k-means++ algorithm is used to cluster K GUs into M clusters, denoted as K′={1,2,…,m,…,M}, and the UAV flies to the centroid of the nearest cluster and associates with it. Assume that the system works in a time slot manner, that is, T={1,2,…,t,…,T}, and each time slot δ t The duration is short, and UAVs and GUs can be considered to be approximately stationary in a time slot. Then the trajectories of UAV m and user k can be expressed as:

[0025]

[0026] q k (t+1)-q k (t) = V g δ t e k (t),t=1,2...,T (2)

[0027] Among them, e m (t) and e k (t) represent the movement directions of drone m and user k at time slot t. Since the drone trajectory and movement direction are linearly related, the trajectory optimization of the drone is equivalent to optimizing the flight direction of the drone. In addition, the drones need to maintain a safe distance d min , to avoid collisions, namely:

[0028] ||q m1 (t)-q m2 (t)||≤d min (3)

[0029] Introducing a binary variable to represent the connection relationship between drone m and user cluster k′ in time slot t. m,k′ When (t) = 1, it means that the two have established a communication connection at time slot t. Otherwise, no connection is established. In order to maintain efficient data connection, the connection between the drone and the user cluster must meet the following constraints:

[0030]

[0031]

[0032] Among them, formula (4) means that each user cluster has only one UAV transmitting data to it in the same time slot, and formula (5) means that each UAV has only one user cluster receiving its data in the same time slot.

[0033] Consider two types of links, LoS and NLoS, between drones and ground users. The average path loss between drone m and user k in time slot t can be expressed as:

[0034]

[0035] Among them, η LoS and η NLoS are the average additional losses for LoS and NLoS links, respectively, depending on the channel environment; a and b are the parameters of the S-curve, depending on the environment; d m,k (t) represents the distance between drone m and user k at time slot t; f is the carrier frequency; c is the speed of light. Then the channel gain between drone m and user k is expressed as:

[0036]

[0037] In addition, the present invention considers the impact of ground interference sources on the communication between the UAV and the user, and the link between the UAV and the interference source also conforms to the same air-ground channel model. Therefore, the channel gain between the UAV m and the interference source n is:

[0038]

[0039] Assume that the total system bandwidth is B, which is evenly distributed to M drones, and each drone provides services to the ground users it covers in frequency division multiple access mode (FDMA) to avoid mutual interference between UAVs. Let the transmission power of the drone be P, then in time slot t, the signal-to-interference-to-noise ratio of user k receiving drone m can be expressed as:

[0040]

[0041] Among them, P N represents the power of the interference source; n0 is the noise power spectrum density; the effective interference distance of the interference source is d N , that is, when the distance between the UAV and the interference source exceeds d N Therefore, the data transmission rate from UAV m to user k in time slot t is:

[0042]

[0043] The present invention jointly optimizes the flight trajectory and flight speed of the UAV to avoid interference sources and maximize the average data transmission rate of users. The optimization problem can be expressed as:

[0044]

[0045] in, and Represent the upper and lower boundaries of the region respectively. C1 is the speed constraint of the UAV. C2 represents the collision constraint between UAVs. C3 and C4 represent the matching relationship between UAVs and user clusters. C5 indicates that the UAV is restricted to the considered region.

[0046] In this embodiment, the PRE-MATD3 algorithm is proposed. The algorithm is first pre-trained in a single-agent environment to generate an initial exploration strategy and save the trained action network. Then, in the multi-agent training phase, the pre-trained action network is imported into the multi-agent model and learned through a centralized training distributed execution framework, thereby accelerating the convergence of the algorithm and improving the stability of the system. At the same time, the algorithm significantly improves the data transmission rate by optimizing the flight trajectory of the drone swarm.

[0047] Figure 2 The PRE-MATD3 algorithm framework is shown in Figure 1. The algorithm consists of two main stages. First, pre-training is performed in a simplified single-drone scenario and the pre-trained action network parameters are saved. Then, in the multi-agent training phase, the action network parameters are used Initialize the action networks of all agents to provide an effective initial strategy. Finally, learn again through the CTDE framework to achieve efficient trajectory planning in the UAV swarm-assisted data transmission scenario.

[0048] In the multi-agent training phase of the PRE-MATD3 algorithm, since the decision of each drone will affect the subsequent environmental state, and each drone can only observe its own local environmental state, the interaction process between the drone swarm and the environment is expressed as a partially observable Markov decision process (POMDP). Based on the standard reinforcement learning process, the elements are defined as follows:

[0049] State space: Each UAV observes a local state. The observation space of UAV m at time slot t is represented by s m (t) represents the 3D coordinates of the UAV itself, q m (t), the coordinate q of the associated user cluster k′ k′ (t) and the transmission rate R between UAV m and the associated user cluster k′ m,k′ (t). Therefore, the state space of UAV m at time slot t is s m (t) = {q m (t),q k′ (t),R m,k′ (t)}. The present invention normalizes the state space to eliminate the diversity of variables, and the state space can be converted into:

[0050]

[0051] in, and R max Represents the maximum transmission rate between the drone and the user cluster.

[0052] Action Space: The action space of UAV m is composed of a m (t). Each UAV action includes the flight direction of the UAV e m (t) (in the range [0,2π]) and flight speed (The range is [0,V max ]). In the action network output layer, these two decision variables are compressed to [0,1] by the Sigmoid function. Therefore, the action space of UAV m in time slot t is:

[0053]

[0054] Reward function: The reward function of the UAV is given by r m In the UAV swarm-assisted data transmission scenario, all UAVs need to avoid interference sources and maintain communication connections with users while maximizing the data transmission rate. Therefore, the reward of UAV m in time slot t includes the transmission rate reward and penalty:

[0055] r m (t) = λR m,k′ (t)+κ1r1+κ2r2+κ3r3 (14)

[0056] Among them, λ represents the positive adjustment weight, κ1, κ2, and κ3 represent whether the UAV flies out of the considered area; whether it collides with other UAVs; whether communication is interrupted, and when the transmission rate R m,k′ When (t) < 0, communication interruption occurs. In addition, r1, r2, and r3 are all constant penalties.

[0057] Figure 3 The figure shows the trajectory planning flow chart based on multi-UAV assisted data transmission. The specific steps are as follows:

[0058] S301: system initialization;

[0059] S302: construct a communication network scenario including UAVs, GUs, and ground interference sources;

[0060] S303: clustering the GUs into M clusters using the k-means++ clustering algorithm;

[0061] S304: deploy M UAVs to be associated with the M clusters respectively, and fly to the best position, that is, the centroid of the cluster;

[0062] S305: Establish a communication link between the UAV and the GU, consider the air-ground channel, and add the interference source influence to the communication link;

[0063] S306: Establishing a model for maximizing data transmission rate;

[0064] S307: Pre-training a single agent in a simplified environment and saving the pre-trained action network;

[0065] S308: Initialize the multi-agent action network using pre-trained action network parameters;

[0066] S309: Initialize the experience replay buffer and the exploration noise and its annealing factor;

[0067] S310: Initialize the environment state, and each UAV observes its own environment state;

[0068] S311: Each drone performs actions in a distributed manner according to its own strategy;

[0069] S312: Observe the reward of the next state and environmental feedback;

[0070] S313: storing the observation state, action, reward and next observation state into the experience playback buffer;

[0071] S314: Randomly sample small batches of experience from the experience pool to break the correlation between experiences;

[0072] S315: updating network parameters through centralized training;

[0073] S316: Annealing exploration noise, balancing exploration and learning.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A trajectory planning method for multi-UAV assisted data transmission, characterized in that: The following steps are involved: Establish a communication scenario including drones, ground users and unknown interference sources, in which there are M drones, K ground users and N ground interference sources. Cluster the ground users into M clusters, each of which is associated with a drone, and let the best hovering position of the drone be the centroid of its associated user cluster; When building the communication link between the UAV and the ground user, the wireless channel adopts the line of sight link (LoS) and the non-line of sight link (NLoS). Each UAV uses orthogonal frequency division multiple access to provide services to its associated ground users. The ground interference source continuously affects the communication between the UAV and the ground user. A Pre-Trained Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (PRE-MATD3) algorithm based on the Centralized Training with Decentralized Execution (CTDE) framework is proposed, including: Single-agent pre-training: Pre-train the algorithm in a single-agent environment and save the trained action network; Multi-agent training: Import the saved pre-trained action network into the multi-agent model, use the MATD3 algorithm for multi-agent training, and enhance the convergence speed of the strategy and the stability of the system through centralized training and distributed execution; Centralized training and distributed execution framework: Global state information is collected during centralized training, while in the distributed execution phase, each drone makes independent decisions based on the local state, thereby optimizing the trajectory planning of the drone swarm.

2. The trajectory planning method for multi-UAV assisted data transmission according to claim 1, characterized in that: In the communication link between the UAV and the ground user, the ground interference source continuously affects the communication of the UAV to the ground user, and the link between the UAV and the interference source also conforms to the same air-ground channel model.

3. The trajectory planning method for multi-UAV assisted data transmission according to claim 2 is characterized in that: In the multi-agent training phase of the PRE-MATD3 algorithm, the interaction process between the drone swarm and the environment is expressed as a partially observable Markov decision process (POMDP), and learning is performed through centralized training and distributed execution framework to achieve efficient trajectory planning in the drone swarm-assisted data transmission scenario.

4. The trajectory planning method for multi-UAV assisted data transmission according to claim 3 is characterized in that: In the drone swarm-assisted data transmission scenario, all drones avoid interference sources and maintain communication connections with users while maximizing the data transmission rate. The reward of drone m in time slot t includes transmission rate reward and penalty.

Images