Multi-unmanned aerial vehicle path planning and power distribution method based on automatic encoder

Abstract

The invention provides a multi-unmanned aerial vehicle path planning and power distribution method based on an automatic encoder, and relates to the technical field of unmanned aerial vehicles. According to the invention, the problems of power distribution and path planning in a multi-unmanned aerial vehicle scene are researched, and the purpose is to maximize the average data transmission rate of user equipment in a random user environment. Therefore, the invention provides a deep reinforcement learning framework oriented to a distributed multi-agent environment, and provides a solution for autonomous cooperation of the unmanned aerial vehicle. According to the framework, a common communication basis of the unmanned aerial vehicles is learned by using a self-supervised representation learning task based on an automatic encoder, and on the basis, the unmanned aerial vehicles can understand and convey information observed by each other, so that the information acquisition capability of the unmanned aerial vehicles under a limited observation condition is improved. The method is suitable for a completely dispersedly executed system structure, does not need additional auxiliary information, and realizes efficient information transmission and communication between unmanned aerial vehicles, thereby effectively improving the service efficiency of an unmanned aerial vehicle network.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a multi-unmanned aerial vehicle path planning and power allocation method based on an automatic encoder. Background Art

[0002] With the rapid development of Internet of Things technology, the traffic demand of various services is experiencing explosive growth, such as big data, cloud computing, etc. Traditional ground wireless networks have gradually been unable to adapt to the traffic demand of massive applications. In addition, due to the limitations of network capacity and coverage, when outdoor disasters such as forest fires occur, ground base stations cannot be deployed in time to provide network services. Therefore, an easy-to-deploy and easy-to-expand network architecture is needed to adapt to various harsh environments and meet the traffic needs of massive mobile users. Considering the low cost, easy expansion and easy deployment of drones, the use of drones as aerial base stations has attracted widespread attention from academia and industry, and has become an effective solution to choose from.

[0003] Unlike the wireless transmission of traditional ground static base stations, drones are limited by their energy storage and transmission power, making it difficult to deploy them for a long time. They can only serve users in a limited range within a limited time. However, in reality, the location of users often changes all the time, and the network requirements in different areas also change constantly. In traditional drone networks, since a single drone can only observe user information within its service range and lacks cooperation with other drones, a large number of drones often gather and serve the same area, resulting in a small effective coverage range of the drone network and a waste of drone resources. Therefore, designing a reasonable flight trajectory for drones so that they can achieve the greatest possible service coverage within a limited flight time is of great significance to improving the service efficiency and service quality of drones.

[0004] Deep reinforcement learning (DRL) is a technique for controlling agents to autonomously learn and make decisions in a random environment, with the goal of maximizing a certain cumulative reward. Recently, DRL has made significant achievements in some high-dimensional state observation and planning problems, such as video games, Go, and robot control. Inspired by this, the DRL framework has gradually become a popular solution for solving traditional optimization problems in drone networks. Existing methods include the FANET framework for optimizing the resource supply problem of drone networks, combining DRL with FLMs to solve the complex problem of finding the optimal drone trajectory to improve spectrum efficiency, using distributed DRL methods to solve the TSP problem in UAV scenarios, and using multi-agent reinforcement learning methods to solve the path planning problem of drones. Most of the current solutions are based on single-agent scenarios, where each drone makes decisions and trains based on its own independent observation data. Such solutions ignore the collaboration between drones and are prone to problems such as excessive resource concentration. Multi-agent solutions often achieve collaborative decision-making and training stability by directly sharing the drone's field of view. However, due to the physical conditions of drones, such solutions are difficult to apply in reality. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multi-UAV path planning and power allocation method based on an automatic encoder in response to the deficiencies of the above-mentioned prior art, which is used to solve the multi-UAV power allocation and path planning problems in the scenario where UAVs serve as aerial base stations to serve ground users, maximize the average data transmission rate of user devices in a random user environment, provide a solution for autonomous collaboration of UAVs, and enhance the information acquisition capabilities of UAVs under restricted observation conditions.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A multi-UAV path planning and power allocation method based on autoencoder is proposed. In the UAV-assisted wireless network architecture, K ground users in the service area are served by N UAVs and ground base stations. The stable and efficient network service optimization objective is modeled as:

[0008]

[0009] ||l n (t)-l n′ (t)||2≥d′ min (3)

[0010] 0≤v n (t)≤v max (4)

[0011] 0≤p n (t)≤P Nmax (5)

[0012] Where p = {p n (t)|n∈N\{0}}, L={l n (t)|n∈N\{0}}; R kn (t) represents the instantaneous transmission rate between user k and drone n at time t. There are K ground users and N drones, k = {1, 2, 3, ..., K}, n = {0, 1, 2, 3, ..., N}, and n = 0 represents the ground base station; Indicates the return rate of the drone; l n (t), l n′ (t) represents the positions of UAV n and other UAV n′ at time t, n′∈N\{0,n}; d′ min Indicates the minimum safe distance between drones; v n (t) represents the speed of UAV n at time t; v max Indicates the maximum instantaneous speed of the drone during flight; p n (t) represents the transmission power of UAV n at time t; P N max represents the maximum transmission power of the UAV; constraint (2) indicates that the transmission rate between the UAV and the user cannot exceed the return rate of the UAV, constraint (3) limits the minimum safe distance between UAVs, constraint (4) limits the maximum instantaneous speed of the UAV during flight, and constraint (5) is used to ensure that the instantaneous transmission power of each UAV does not exceed the maximum transmission power;

[0013] The path planning and power allocation problem of UAVs is modeled as a partially observable Markov decision process, and a multi-agent reinforcement learning algorithm with autoencoder is proposed to seek approximate optimal solutions. Each UAV is regarded as an intelligent agent, which will make decisions based on the surrounding environmental status and information from other UAVs, and continuously learn and update strategies. This process is carried out using centralized training and distributed execution methods.

[0014] Furthermore, in the UAV-assisted wireless network architecture, basic UAV control and non-payload communication (CNPC) is selected as a communication channel model between UAVs.

[0015] Furthermore, the partially observable Markov decision process is represented by a tuple<S,A,R,P,O,γ> express;

[0016] S={s1,s2,..,s t ,..,s T} represents a set of environmental states, s tRepresents the state of the environment at time t and is defined as follows:

[0017] s t = {l n (t),l k (t),R kn (t),m(t)},n∈N\{0}

[0018] Among them, l k (t) represents the location of user k at time t, and m(t) represents the message of the drone at time t;

[0019] A={a1,a2,…,a t ,…,a T} represents a series of actions of the drone, including the speed, direction and transmission power of the drone. t Represents the action of the drone at time t, expressed as:

[0020]

[0021] in, Indicates the direction in which the UAV is traveling at time t;

[0022] R={r1,r2,…,r t ,…,r T} represents the rewards corresponding to a series of actions of the drone, r t represents the reward corresponding to the drone's action at time t, and the reward function is expressed as:

[0023]

[0024] Among them, r t n represents the reward corresponding to the action of drone n at time t; represents the action of drone n at time t;

[0025] represents a set of observation states, It represents the observation state of UAV n at time t, expressed as

[0026]

[0027] Among them, m n′ (t) represents the message of other UAV n′ at time t;

[0028] P represents the probability that a drone in one environmental state will transfer to another environmental state after taking an action; γ represents the cumulative discount rate.

[0029] Furthermore, each of the intelligent agents has a common architecture, which includes an encoding network and a decoding network; the encoding network observes the current state value o t Generate a communication message φ ξ (o t ): o t →φ ξ (o t )∈R comm , ξ is the encoder parameter, the decoding network will receive the message m t Decode into status information ψ is the decoder parameter; the encoder network attempts to reconstruct the message into observations after encoding the observations using the decoder: The goal is to minimize the following reconstruction loss function:

[0030]

[0031] Among them, c i Represents the encoded message, Represents the decoded message.

[0032] Furthermore, in the centralized training and distributed execution, each drone has a policy network and an evaluation network. The policy network makes a decision based on the observed environmental information, and the evaluation network scores the decision based on the environmental information. Both use a deep neural network as a function approximator.

[0033] In the training phase, the experience replay mechanism is used; the observation data of all agents are stored in an experience replay pool D, which is implemented on the main base station MBS. Each observation data is stored as an information tuple {s, a, r, s′}, where s, a, r, s′ represent the state, action, reward and the state at the next moment respectively; in each training process, a small batch of observation data is randomly sampled from the experience replay pool D to update the neural network;

[0034] The policy network and evaluation network are trained by the PPO algorithm; the loss function of the policy network is expressed as:

[0035]

[0036] in, is the ratio of the new and old strategies, θ is the parameter of the strategy network; clip is the clipping function, which is used to limit γ(θ) to the range of [1-ε,1+ε] to prevent the gap between the new and old strategies from being too large; ε is a fixed hyperparameter; is the advantage function, which is used to evaluate the advantage of action a compared to general actions, Q(s,a) represents the action value function, and V(s) represents the state value function;

[0037] The loss function of the evaluation network is expressed as:

[0038]

[0039] in, is the parameter of the value network, R = ∑γ t-1 r t To accumulate discount rewards;

[0040] In the execution phase, each drone selects a suitable ground-associated user by cooperating with surrounding drones, and determines the position and transmission power at the next moment; the drone first collects surrounding user information and communication messages of other drones, and then decodes the messages of other drones into understandable environmental information through a decoder, and inputs them into the policy network together with the collected user information to generate an action. Finally, the drone executes the action and sends the encoded observation information to the environment.

[0041] Furthermore, the specific steps of the multi-agent reinforcement learning algorithm with autoencoder to seek the approximate optimal solution are as follows:

[0042] Step 1: Initialize the policy network, evaluation network, encoding network, decoding network, experience replay pool, number of samples, maximum training rounds, and training cycle;

[0043] Step 2: In each training round, initialize the environment and obtain the initial state;

[0044] Step 3: For each time t, each UAV first obtains the message and environment observation of its neighbor UAV, and then passes the message to the decoder to obtain the observation of the neighbor UAV; then the joint observation is passed as input to the policy network to obtain the current action, and the joint observation is passed as input to the encoding network to generate the message;

[0045] Step 4: The encoding network is updated by combining the output of the decoding network with the mean square error loss function;

[0046] Step 5: The environment executes the drone action and generates rewards for each drone, and then proceeds to the next time t+1;

[0047] Step 6: Put the drone’s actions, observations, rewards at time t and the state at the next moment into the experience replay pool;

[0048] Step 7: Repeat steps 2 to 6 until the training round reaches the training cycle. For each network training, perform steps 8 and 9.

[0049] Step 8: Take a sample number of training data from the experience replay pool, and update the evaluation network through the loss function formula of the evaluation network, and update the policy network through the loss function formula of the policy network;

[0050] Step 9: Repeat steps 2 to 8 until the maximum number of training rounds is reached.

[0051] Furthermore, the encoding network and the decoding network both have l1 hidden layers, and the encoding network input layer contains nodes, and the hidden layer contains nodes, and the output layer contains nodes, in contrast, the number of nodes in the input layer, hidden layer, and output layer of the decoding network are as well as The number of hidden layers in the policy network and the evaluation network is l2, and the number of hidden layer nodes is N. hidden , the number of nodes in the input layer and output layer of the policy network and the evaluation network are as well as In the execution phase, the time complexity of decoding and encoding a message is The time complexity of generating execution actions is During the training phase, the time complexity of updating the policy network and the evaluation network is

[0052] The beneficial effects of adopting the above technical solution are: the multi-UAV path planning and power allocation method based on autoencoder provided by the present invention discusses the power allocation and path planning problems of multiple UAVs in the scenario where UAVs serve ground users as aerial base stations, aiming to improve the service efficiency of the UAV network. Considering the collaboration of UAVs, the present invention proposes a multi-agent reinforcement learning framework based on autoencoder, which learns the compact representation of UAV observations in an unsupervised manner based on autoencoders, realizes efficient information transmission and communication between UAVs, improves the information acquisition ability of UAVs in a distributed execution environment, and thus effectively improves the service efficiency of the UAV network. The effectiveness of the communication module proposed by the present invention is fully demonstrated through ablation experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a wireless network architecture assisted by a drone provided in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of the structure of an automatic encoder provided by an embodiment of the present invention;

[0055] Figure 3A flow chart of a multi-UAV path planning and power allocation method based on an autoencoder provided in an embodiment of the present invention;

[0056] Figure 4 The experimental effect diagram provided for the embodiment of the present invention; wherein Figure (4a) is the effect diagram of the traditional multi-agent reinforcement learning method, Figure (4b) is the effect diagram of the TRPO algorithm, and Figure (4c) is the effect diagram of the PG algorithm. DETAILED DESCRIPTION

[0057] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0058] like Figure 1 The wireless network architecture assisted by drones is shown in the figure. The network consists of aerial base stations composed of drones and macro base stations located on the ground. It is assumed that the network serves a square area with a total of K ground users in the area, k = {1, 2, 3, .., K}. Ground users are served by N drones and ground base stations, n = {0, 1, 2, 3, .., N}, n = 0 represents the ground base station. Users near the ground base station are directly served by the ground base station, and those outside the service range of the ground base station or in areas where the ground base station signal is blocked are served by drones. All drones will adjust their positions in real time according to the user's location, with the aim of providing stable and efficient network services for all user devices.

[0059] During the service process, it is assumed that the ground base station is always located at the center of the region, that is, the origin. For the convenience of description, the time is divided into multiple equal time slots δ. Assuming that the user moves randomly in the region, the position of user k at time t is expressed as Assuming that the drone always flies at a fixed height h0, the position of drone n at time t is expressed as The distance between user k and drone n at time t is calculated by the following formula:

[0060]

[0061] For drone n, we only need to know its position l at time t n (t) = (x n (t),y n (t),h0), the moving direction α∈[0,2π], and the velocity v∈[0,v max ], we can calculate its position at time t+1 as follows:

[0062] l n (t+1)=(x n (t)+vn (t)δcos(α n (t)),y n (t)+v n (t)δcos(α n (t)),h0) (2)

[0063] So given the initial position l of UAV n n (0), its movement trajectory within a period T can be obtained through its actions in each time slot t.

[0064] In a multi-UAV network, air-to-ground communication links are usually considered to be line-of-sight (LoS) or non-line-of-sight (NLoS), which is highly dependent on the network environment. If the link from the user device to the UAV is visually blocked, it is considered to be LoS, otherwise it is NLoS. Assuming that LoS and NLoS are encountered randomly in the environment, the probability of LoS of the link between the user device k and the UAV n at height h0 is expressed as:

[0065]

[0066] Among them, μ1 and μ2 are constants related to the environment. kn (t) = 1-PLoS kn (t). According to the free space path loss model, the channel power gain between user equipment k and drone n is expressed as:

[0067]

[0068] in, f c is the carrier frequency, c is the speed of light, χ is the path loss exponent, β LoS and β NLoS are the attenuation factors of LoS link and NLoS link respectively.

[0069] The drone is limited by its transmission power and can only serve users within a certain range around itself. Assuming that the service range of the drone is a circular range with a radius of R, the drone can sense the information of user devices and other drones within this range. At the same time, the drone can broadcast the observed information to other drones within its service range in each time slot. t , and can also receive observation information broadcast by other drones. t', so that the best decision can be made in the next time slot. Considering that the UAV needs to transmit messages to the surrounding UAVs in a timely manner, and the size of the encoded message is small, this embodiment selects the basic Control and Non-Payload Communication (CNPC) as the communication channel model between UAVs.

[0070] Let p n (t) represents the transmission power of UAV n at time t, then the signal-to-noise ratio between user k and base station n at time t is expressed as:

[0071]

[0072] where σ 2 is the additive noise power, assumed to be the same for all user devices. k is the interference imposed on user k by other drones except drone n, which is calculated by the following formula:

[0073] I k =∑ n′∈N\{n} .p n′ (t)g kn′ (t) (6)

[0074] Given the channel bandwidth W between user device k and drone n, the instantaneous transmission rate between user k and drone n at time t is expressed as:

[0075] R kn (t) = Wlog2(1 + SNR kn (t)) (7)

[0076] Accordingly, the average rate and speed of users in the area within a period T are expressed as:

[0077]

[0078] Under the premise of the above model, the optimization target model of the problem in this embodiment is:

[0079]

[0080] Where p = {p n (t)|n∈N\{0}}, L={l n (t)|n∈N\{0}}. Constraint (9b) indicates that the transmission rate between the drone and the user cannot exceed the return rate of the drone. Constraint (9c) limits the minimum safe distance d′ between UAVs min , constraint (9d) limits the maximum instantaneous speed v of the UAV during flight max, constraint (9e) is used to ensure that the instantaneous transmission power of each UAV does not exceed the maximum transmission power P N max .

[0081] It can be seen that the objective function (9a) is a non-convex function. In theory, the optimal solution to the problem can be obtained by exhaustive method, but its high time complexity is unacceptable in reality. Therefore, this embodiment adopts MADRL method to solve the above joint optimization problem, and uses PPO algorithm to seek approximate optimal solution.

[0082] This embodiment models the path planning and power allocation problem of drones as a partially observable Markov decision process, and proposes a multi-agent reinforcement learning algorithm with an autoencoder to solve it. Each drone is regarded as an agent, which will make decisions based on the surrounding environment and information from other drones, and continuously learn and update strategies. Considering the energy consumption of drones and the problem that independent learning may cause unstable model learning, this process adopts centralized training and distributed execution methods.

[0083] The Markov decision process is a discrete-time random process that describes the probability distribution of future decisions and state transitions of a system given its current state and a sequence of observed events. It is modeled based on a Markov chain, in which each state is related to its previous state and currently observed information. In the Markov decision process, the decision maker faces a series of decision problems, each of which is based on the current state and past observations. In the above model, the agent acts as a decision maker, and each decision it makes is related to the state of the observed environment. However, the agent is limited by its own physical conditions and can only observe a limited environment. Therefore, this embodiment models the above-mentioned joint optimization problem as a partially observable Markov decision process, using a tuple<S,A,R,P,O,γ> express.

[0084] The environment of each time slot has a specific state. Let st represent the state of the environment at time t. Then {s1,s2,..,s t ,..,s T} represents a series of states. The state of the environment includes the location of the drone, the location of the user, the transmission rate between the drone and the user, and the communication message of the drone. Therefore, the state of the environment at time t is defined as:

[0085] s t = {l n (t),l k (t),R kn (t),m(t)},n∈N\{0} (10)

[0086] Among them, m(t) represents the message of the drone at time t.

[0087] During the action phase, drones can only observe the surrounding environment. For drone n, its observation range includes the locations of surrounding drones and users, the transmission rate between users directly connected to drone n, and the messages of surrounding drones. Represents a set of observation states. The observation state of drone n at time t is expressed as:

[0088]

[0089] Among them, m n′ (t) represents the message of other UAV n′ at time t.

[0090] In each time slot t, the drone needs to take corresponding actions based on the observed state to enable the UE to obtain the maximum transmission rate. t ,…,a T} represents a series of actions, including the speed, direction and transmission power of the drone. The action of the drone at time t is expressed as:

[0091]

[0092] in, Indicates the direction the drone is traveling at time t.

[0093] The environment will give a reward to the agent n according to its action at time t. This reward is represented by r, so R = {r1, r2, …, r t ,…,r T} represents the rewards corresponding to a series of actions of the drone. The overall reward function of the system in this embodiment is expressed as:

[0094]

[0095] Among them, r t n represents the reward corresponding to the action of drone n at time t; Represents the action of drone n at time t.

[0096] At a certain time t, in state s t After the agent takes action, the environment will be transferred to s with a certain probability. t+1 , this probability can be represented by P. γ represents the cumulative discount rate.

[0097] In a decentralized multi-agent communication environment, the biggest difficulty in communication between agents is that they lack a common communication foundation, which usually leads to asynchrony between agents and affects the stability of the training process. To solve this problem, this embodiment introduces a self-supervised representation learning task, the purpose of which is to learn a communication foundation that all agents can understand, so that agents can autonomously transmit and interpret the information they observe. Specifically, each agent has a common architecture, such as Figure 2 shown.

[0098] The structure consists of an encoding network and a decoding network. The encoding network generates a communication message o according to the current state observation value. t →φ ξ (o t )∈R comm , ξ is the encoder parameter, and the decoding network decodes the received message into state information ψ is the decoder parameter. After encoding the observation information, the encoder network will try to reconstruct the message into the observation value using the decoder The goal of this embodiment is to minimize the following reconstruction loss function:

[0099]

[0100] On the one hand, drones have limited computing power and computing resources due to their own size. On the other hand, drones can only perform independent training based on the local information received during training, which is likely to lead to model uncertainty. Therefore, this embodiment uses a centralized training and distributed execution framework to train drones. Specifically, each drone has a policy network and an evaluation network. The policy network makes decisions based on the observed environmental information, and the evaluation network scores the decision based on the environmental information. Both use a deep neural network as a function approximator.

[0101] During the training phase, this embodiment uses an experience replay mechanism to improve the efficiency of learning the optimal strategy. The observation data of all agents are stored in an experience replay pool D, which is implemented on the main base station (MBS), and each observation data is stored as an information tuple such as {s, a, r, s′}. During each training process, the neural network is updated by randomly sampling a small batch of observation data from the experience replay pool D to reduce the mutual dependence between the observation data and improve learning efficiency. In subsequent experiments, this embodiment compared the performance of several popular DRL algorithms, and finally selected the PPO algorithm to train the policy network and the evaluation network. The loss function of the policy network is expressed as:

[0102]

[0103] in, is the ratio of the new and old strategies, θ is the parameter of the strategy network. Clip is the clipping function, which limits γ(θ) to the range of [1-ε,1+ε] to prevent the gap between the new and old strategies from being too large. ε is a fixed hyperparameter. is the advantage function, which is used to evaluate the advantage of action a compared to general actions. Q(s,a) represents the action value function and V(s) represents the state value function.

[0104] The loss function of the evaluation network is expressed as:

[0105]

[0106] in, is the parameter of the value network, R = ∑γ t-1 r t For cumulative discount rewards.

[0107] In the execution phase, each drone needs to select the appropriate ground-associated user by cooperating with the surrounding drones, and determine the position and transmission power at the next moment. Specifically, the drone will first collect the surrounding user information and communication messages of other drones. Then the decoder decodes the messages of other drones into understandable environmental information, and inputs them into the policy network together with the collected user information to generate an action. Finally, the drone executes the action and sends the encoded observation information to the environment.

[0108] The above-mentioned path design method based on multi-agent reinforcement learning with autoencoder, such as Figure 3 As shown, it mainly includes the following steps:

[0109] Step 1: Initialize the policy network, evaluation network, encoding network, decoding network, experience replay pool, number of samples, maximum training rounds, and training cycle.

[0110] Step 2: In each training round, initialize the environment and obtain the initial state;

[0111] Step 3: For each time t, each UAV first obtains the message and environment observation of its neighbor UAV, and then passes the message to the decoder to obtain the observation of the neighbor UAV; then the joint observation is passed as input to the policy network to obtain the current action, and the joint observation is passed as input to the encoding network to generate the message;

[0112] Step 4: The encoding network is updated by combining the output of the decoding network with the mean square error loss function;

[0113] Step 5: The environment executes the drone action and generates rewards for each drone, and then proceeds to the next time t+1;

[0114] Step 6: Put the drone’s actions, observations, rewards at time t and the state at the next moment into the experience replay pool;

[0115] Step 7: Repeat steps 2 to 6 until the training round reaches the training cycle. For each network training, perform steps 8 and 9.

[0116] Step 8: Take a sample number of training data from the experience replay pool, and update the evaluation network through the loss function formula of the evaluation network, and update the policy network through the loss function formula of the policy network;

[0117] Step 9: Repeat steps 2 to 8 until the maximum number of training rounds is reached.

[0118] Assume that both the encoding network and the decoding network have l1 hidden layers, and the encoding network input layer contains nodes, and the hidden layer contains nodes, and the output layer contains nodes, in contrast, the number of nodes in the input layer, hidden layer, and output layer of the decoding network are as well as The number of hidden layers in the policy network (actor) and the evaluation network (critic) is l2, and the number of hidden layer nodes is N. hidden , the number of nodes in the input layer and the output layer are as well as Then in the execution phase, the time complexity of decoding and encoding the message is The time complexity of generating execution actions is During the training phase, the time complexity of updating the policy network and the evaluation network is

[0119] In order to verify the effectiveness of the communication module, this embodiment conducts three groups of ablation experiments, such as Figure 4As shown. Undoubtedly, the algorithm with the communication module shows better performance than the traditional algorithm. Taking Figure (4a) as an example, PPO withoutcomm adopts the traditional multi-agent reinforcement learning method. In the execution stage, the UAV only relies on its own observation information to make decisions, and the training algorithm of the ac network is PPO. The algorithm proposed in the present invention provides a communication method for the UAV. The UAV embeds its own observed state information into a low-dimensional communication message through an automatic encoder, and then transmits the information to the surrounding UAVs in a timely manner through the CNPC channel, thereby completing information sharing. Compared with the traditional PPO multi-agent reinforcement learning method, the performance of the algorithm proposed in the present invention is improved by about 22.4%. The performance of the TRPO algorithm in Figure (4b) and the PG algorithm in Figure (4c) are also improved by about 9% and 4.4% respectively after adding the communication module. The experimental results fully demonstrate the effectiveness of the communication module proposed in the present invention.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multi-UAV path planning and power allocation method based on an autoencoder, characterized in that: In the UAV-assisted wireless network architecture, K ground users in the service area are served by N UAVs and ground base stations. The optimization objective of stable and efficient network service is modeled as: ||l n (t)-l n′ (t)|||2≥d′ min (3) 0≤v n (t)≤v max (4) 0≤p n (t)≤p N max (5) Where p=(p n (t)|n∈N\{0}}, L={l n (t)|n∈N\{0}}; R kn (t) represents the instantaneous transmission rate between user k and drone n at time t. There are K ground users and N drones, k = {1, 2, 3, ..., K}, n = {0, 1, 2, 3, ..., N}, and n = 0 represents the ground base station; Indicates the return rate of the drone; l n (t), l n′ (t) represents the positions of UAV n and other UAV n′ at time t, n′∈N\{0,n}; d′ min Indicates the minimum safe distance between drones; v n (t) represents the speed of UAV n at time t; v max Indicates the maximum instantaneous speed of the drone during flight; p n (t) represents the transmission power of UAV n at time t; P N max represents the maximum transmission power of the UAV; constraint (2) indicates that the transmission rate between the UAV and the user cannot exceed the return rate of the UAV, constraint (3) limits the minimum safe distance between UAVs, constraint (4) limits the maximum instantaneous speed of the UAV during flight, and constraint (5) is used to ensure that the instantaneous transmission power of each UAV does not exceed the maximum transmission power; The path planning and power allocation problem of UAVs is modeled as a partially observable Markov decision process, and a multi-agent reinforcement learning algorithm with autoencoder is proposed to seek the approximate optimal solution; each UAV is regarded as an agent, which will make decisions based on the surrounding environment state and information from other UAVs, and continuously learn and update strategies; This process is carried out using centralized training and distributed execution methods.

2. The multi-UAV path planning and power allocation method based on automatic encoder according to claim 1 is characterized in that: In the UAV-assisted wireless network architecture, basic UAV control and non-payload communication (CNPC) is selected as the communication channel model between UAVs.

3. The multi-UAV path planning and power allocation method based on autoencoder according to claim 1 is characterized in that: The partially observable Markov decision process is represented by a tuple<S,A,R,P,O,γ> express; S={s1,s2,..,s t , ..., s T } represents a set of environmental states, s t Represents the state of the environment at time t and is defined as follows: s t ={l n (t),l k (t),R kn (t),m(t)},n∈N\{0} Among them, l k (t) represents the location of user k at time t, and m(t) represents the message of the drone at time t; A={a1,a2,…,a t , …, a T } represents a series of actions of the drone, including the speed, direction and transmission power of the drone. t Represents the action of the drone at time t, expressed as: in, Indicates the direction in which the UAV is traveling at time t; R={r1,r2,…,r t , ..., r T } represents the rewards corresponding to a series of actions of the drone, r t represents the reward corresponding to the drone's action at time t, and the reward function is expressed as: Among them, r t n represents the reward corresponding to the action of drone n at time t; represents the action of drone n at time t; represents a set of observation states, It represents the observation state of UAV n at time t, expressed as Among them, m n′ (t) represents the message of other UAV n′ at time t; P represents the probability that a drone in one environmental state will transfer to another environmental state after taking an action; γ represents the cumulative discount rate.

4. The multi-UAV path planning and power allocation method based on autoencoder according to claim 3 is characterized in that: Each of the agents has a common architecture that includes an encoder network and a decoder network; the encoder network observes the current state o t Generate a communication message φ ξ (o t ): o t →φ ξ (o t )∈R comm , ξ is the encoder parameter, the decoding network will receive the message m t Decode into status information ψ is the decoder parameter; the encoder network attempts to reconstruct the message into observations using the decoder after encoding the observations: The goal is to minimize the following reconstruction loss function: Among them, c i Represents the encoded message, Represents the decoded message.

5. The multi-UAV path planning and power allocation method based on autoencoder according to claim 4 is characterized in that: In the centralized training and distributed execution, each drone has a policy network and an evaluation network. The policy network makes decisions based on the observed environmental information, and the evaluation network scores the decision based on the environmental information. Both use a deep neural network as a function approximator. In the training phase, the experience replay mechanism is used; the observation data of all agents are stored in an experience replay pool D, which is implemented on the main base station MBS. Each observation data is stored as an information tuple {s, a, r, s′}, where s, a, r, s′ represent the state, action, reward and the state at the next moment respectively; in each training process, a small batch of observation data is randomly sampled from the experience replay pool D to update the neural network; The policy network and evaluation network are trained by the PPO algorithm; the loss function of the policy network is expressed as: in, is the ratio of the new and old strategies, θ is the parameter of the strategy network; clip is the clipping function, which is used to limit γ(θ) to the range of [1-ε, 1+ε] to prevent the gap between the new and old strategies from being too large; ε is a fixed hyperparameter; is the advantage function, which is used to evaluate the advantage of action a compared to general actions, Q(s, a) represents the action value function, and V(s) represents the state value function; The loss function of the evaluation network is expressed as: in, is the parameter of the value network, R = ∑γ t-1 r t To accumulate discount rewards; In the execution phase, each drone selects a suitable ground-associated user by cooperating with surrounding drones, and determines the position and transmission power at the next moment; the drone first collects surrounding user information and communication messages of other drones, and then decodes the messages of other drones into understandable environmental information through a decoder, and inputs them into the policy network together with the collected user information to generate an action. Finally, the drone executes the action and sends the encoded observation information to the environment.

6. The multi-UAV path planning and power allocation method based on autoencoder according to claim 5 is characterized in that: The specific steps of the multi-agent reinforcement learning algorithm with autoencoder to seek the approximate optimal solution are as follows: Step 1: Initialize the policy network, evaluation network, encoding network, decoding network, experience replay pool, number of samples, maximum training rounds, and training cycle; Step 2: In each training round, initialize the environment and obtain the initial state; Step 3: For each time t, each UAV first obtains the message and environment observation of its neighbor UAV, and then passes the message to the decoder to obtain the observation of the neighbor UAV; then the joint observation is passed as input to the policy network to obtain the current action, and the joint observation is passed as input to the encoding network to generate the message; Step 4: The encoding network is updated by combining the output of the decoding network with the mean square error loss function; Step 5: The environment executes the drone action and generates rewards for each drone, and then proceeds to the next time t+1; Step 6: Put the drone’s actions, observations, rewards at time t and the state at the next moment into the experience replay pool; Step 7: Repeat steps 2 to 6 until the training round reaches the training cycle. For each network training, perform steps 8 and 9. Step 8: Take a sample number of training data from the experience replay pool, and update the evaluation network through the loss function formula of the evaluation network, and update the policy network through the loss function formula of the policy network; Step 9: Repeat steps 2 to 8 until the maximum number of training rounds is reached.

7. The multi-UAV path planning and power allocation method based on autoencoder according to claim 6 is characterized in that: The encoding network and decoding network both have l1 hidden layers, and the encoding network input layer contains nodes, and the hidden layer contains nodes, and the output layer contains nodes, in contrast, the number of nodes in the input layer, hidden layer, and output layer of the decoding network are as well as The number of hidden layers in the policy network and the evaluation network is l2, and the number of hidden layer nodes is N. hidden , the number of nodes in the input layer and output layer of the policy network and the evaluation network are as well as In the execution phase, the time complexity of decoding and encoding a message is The time complexity of generating execution actions is During the training phase, the time complexity of updating the policy network and the evaluation network is

Images