A Multi-UAV Path Planning and Power Allocation Method Based on Autoencoders

Abstract

This invention provides a multi-UAV path planning and power allocation method based on an autoencoder, relating to the field of UAV technology. This invention studies the power allocation and path planning problem in multi-UAV scenarios, aiming to maximize the average data transmission rate of user devices in a stochastic user environment. To this end, this invention proposes a deep reinforcement learning framework for distributed multi-agent environments, providing a solution for autonomous UAV collaboration. This framework uses a self-supervised representation learning task based on an autoencoder to learn the common communication basis of UAVs. On this basis, UAVs can understand and convey information observed by each other, improving their information acquisition capabilities under constrained observation conditions. This method is applicable to fully distributed architectures and does not require additional auxiliary information, achieving efficient information transmission and communication between UAVs, thereby effectively improving the service efficiency of UAV networks.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a method for multi-UAV path planning and power allocation based on an automatic encoder. Background Technology

[0002] With the rapid development of IoT technology, the traffic demands of various services, such as big data and cloud computing, are experiencing explosive growth. Traditional terrestrial wireless networks are gradually becoming unable to meet the massive traffic demands of these applications. Furthermore, limited by network capacity and coverage, ground base stations cannot be deployed in a timely manner to provide network services during wilderness disasters such as forest fires. Therefore, an easily deployable and scalable network architecture is needed to adapt to various harsh environments and meet the traffic demands of massive mobile users. Considering the low cost, scalability, and ease of deployment of drones, using drones as aerial base stations has attracted widespread attention from academia and industry and has become an effective and viable solution.

[0003] Unlike traditional ground-based static base stations, drones are limited by their energy storage and transmission power, making long-term deployment difficult and limiting their service range to a confined area within a limited timeframe. However, in reality, user locations are constantly changing, and network demands in different areas evolve accordingly. In traditional drone networks, because individual drones can only observe user information within their own service area and lack collaboration with other drones, large numbers of drones often cluster together to serve the same area, resulting in limited effective network coverage and wasted drone resources. Therefore, designing reasonable flight paths for drones to achieve the maximum possible service coverage within a limited flight time is crucial for improving drone service efficiency and quality.

[0004] Deep reinforcement learning (DRL) is a technique for controlling agents to learn and make decisions autonomously in stochastic environments, with the goal of maximizing some cumulative reward. Recently, DRL has achieved significant success in some high-dimensional state observation and planning problems, such as video games, Go, and robot control. Inspired by this, DRL frameworks have gradually become a popular solution for solving traditional optimization problems in UAV networks. Existing methods include the FANET framework for optimizing resource provisioning in UAV networks, combining DRL with FLMs to solve the complex problem of finding the optimal UAV trajectory to improve spectral efficiency, using distributed DRL methods to solve the TSP problem in UAV scenarios, and employing multi-agent reinforcement learning methods to solve UAV path planning problems. Most current solutions are based on single-agent scenarios, where each UAV makes decisions and trains based on its own independent observation data. These solutions neglect the cooperation between UAVs, easily leading to problems such as excessive resource concentration. Multi-agent solutions often achieve collaborative decision-making and training stability by directly sharing the UAV's field of view. However, due to the physical limitations of UAVs, these solutions are difficult to apply in reality. Summary of the Invention

[0005] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a multi-UAV path planning and power allocation method based on an autoencoder. This method is used to solve the problem of power allocation and path planning for multiple UAVs in scenarios where UAVs serve as airborne base stations for ground users, maximize the average data transmission rate of user equipment in random user environments, provide a solution for autonomous UAV collaboration, and improve the information acquisition capability of UAVs under limited observation conditions.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A multi-UAV path planning and power allocation method based on an autoencoder is proposed. In a UAV-assisted wireless network architecture, K ground users in the service area are served by N UAVs and ground base stations. The stable and efficient network service optimization objective is modeled as follows:

[0008]

[0009] ||l n (t)-l n′ (t)||2≥d′ min (3)

[0010] 0≤v n (t)≤v max (4)

[0011] 0≤p n (t)≤P Nmax (5)

[0012] Where, p = {p n (t)|n∈N\{0}},L={l n (t)|n∈N\{0}};R kn (t) represents the instantaneous transmission rate between user k and drone n at time t. There are K ground users and N drones, k = {1, 2, 3, ..., K}, n = {0, 1, 2, 3, ..., N}, and n = 0 represents the ground base station. Indicates the return trip rate of the drone; l n (t), l n′ (t) represents the positions of drone n and other drones n′ at time t, where n′∈N{0,n}; d′ min Indicates the minimum safe distance between drones; v n (t) represents the velocity of the drone n at time t; v max p represents the maximum instantaneous speed of the drone during flight. n (t) represents the transmit power of UAV n at time t; P N max The maximum transmit power of the drone is indicated by constraint (2); constraint (3) indicates that the transmission rate between the drone and the user cannot exceed the return rate of the drone; constraint (4) limits the minimum safe distance between drones; constraint (5) limits the maximum instantaneous speed of the drone during flight; and constraint (6) ensures that the instantaneous transmit power of each drone does not exceed the maximum transmit power.

[0013] The path planning and power allocation problem of UAVs is modeled as a partially observable Markov decision process, and a multi-agent reinforcement learning algorithm with an autoencoder is proposed to seek an approximate optimal solution. Each UAV is regarded as an agent, which makes decisions based on the surrounding environmental state and information from other UAVs, and continuously learns and updates its strategy. This process is carried out using a centralized training and distributed execution approach.

[0014] Furthermore, the basic drone control and non-payload communication (CNPC) is selected as the communication channel model between drones in the drone-assisted wireless network architecture.

[0015] Furthermore, the partially observable Markov decision process is represented by a tuple.<S,A,R,P,O,γ> express;

[0016] S={s1,s2,..,s t ,..,s T} represents a set of environmental states, s tThe state of the environment at time t is defined as follows:

[0017] s t ={l n (t),l k (t),R kn (t),m(t)},n∈N\{0}

[0018] Among them, l k m(t) represents the position of user k at time t, and m(t) represents the message of the drone at time t;

[0019] A = {a1, a2, ..., a} t ,…,a T} represents a series of actions of the drone, including its speed, direction, and transmission power. t The action of the drone at time t is represented as:

[0020]

[0021] in, Indicates the direction the drone is traveling at time t;

[0022] R = {r1, r2, ..., r} t ,…,r T} represents the reward corresponding to a series of actions of the drone, r t The reward function represents the action of the drone at time t.

[0023]

[0024] Where, r t n This represents the reward corresponding to the action of drone n at time t; This represents the action of drone n at time t;

[0025] Represents a set of observed states. The observation state of UAV n at time t is represented as:

[0026]

[0027] Where, m n′ (t) represents the message from other drones n′ at time t;

[0028] P represents the probability that a drone in one environmental state will move to another environmental state after taking an action; γ represents the cumulative discount rate.

[0029] Furthermore, each of the aforementioned agents shares a common architecture, which includes an encoding network and a decoding network; the encoding network is based on the current state observation value o. t Generate a communication message φ ξ (o t ):o t →φ ξ (o t )∈R comm ξ represents the encoder parameters, and the decoding network will receive the message m. t Decoded into status information ψ represents the decoder parameters; after encoding the observation information, the encoding network attempts to reconstruct the message into observations using the decoder. The goal is to minimize the following reconstruction loss function:

[0030]

[0031] Among them, c i This represents the encoded message. This indicates the decoded message.

[0032] Furthermore, in the centralized training and distributed execution, each UAV has a policy network and an evaluation network. The policy network makes decisions based on observed environmental information, and the evaluation network scores the decisions based on environmental information. Both use a deep neural network as a function approximator.

[0033] During the training phase, an experience replay mechanism is used; all agent observation data are stored in an experience replay pool D, which is implemented on the main base station MBS. Each observation data is stored as an information tuple {s, a, r, s′}, where s, a, r, and s′ represent the state, action, reward, and the state at the next moment, respectively. During each training process, the neural network is updated by randomly sampling a small batch of observation data from the experience replay pool D.

[0034] The policy network and evaluation network are trained using the PPO algorithm; the loss function of the policy network is expressed as:

[0035]

[0036] in, γ(θ) represents the ratio of the new and old policies, θ is a parameter of the policy network, clip is a clipping function that limits γ(θ) to the range of [1-ε, 1+ε] to prevent excessive differences between the new and old policies, and ε is a fixed hyperparameter. The advantage function is used to evaluate the advantage of action a compared to a normal action. Q(s,a) represents the action value function and V(s) represents the state value function.

[0037] The loss function for evaluating a network is expressed as:

[0038]

[0039] in, For the parameters of the value network, R = ∑γ t-1 r t For cumulative discount rewards;

[0040] During the execution phase, each UAV collaborates with surrounding UAVs to select appropriate ground-associated users and determine its position and transmission power for the next moment. The UAV first collects information about surrounding users and communication messages from other UAVs, then decodes the messages from other UAVs into understandable environmental information using a decoder, and inputs it along with the collected user information into the policy network to generate an action. Finally, the UAV executes the action and sends the encoded observation information to the environment.

[0041] Furthermore, the specific steps of the multi-agent reinforcement learning algorithm with autoencoder to seek an approximate optimal solution are as follows:

[0042] Step 1: Initialize the policy network, evaluation network, encoder network, decoder network, experience replay pool, number of samples, maximum number of training epochs, and training period;

[0043] Step 2: In each training round, initialize the environment and obtain the initial state;

[0044] Step 3: For each time t, each UAV first acquires the messages and environmental observations of its neighboring UAVs, then passes the messages to the decoder to acquire the observations of its neighboring UAVs; then it passes the joint observations as input to the policy network to acquire the current action, and passes the joint observations as input to the encoding network to generate messages;

[0045] Step 4: The encoding network updates itself by combining the output of the decoding network with the mean squared error loss function;

[0046] Step 5: The environment executes drone actions and generates rewards for each drone, then proceeds to the next time step t+1;

[0047] Step 6: Put the drone's actions, observations, rewards, and state at time t into the experience replay pool;

[0048] Step 7: Repeat steps 2 to 6 until the training rounds reach the training cycle. For each network training session, execute steps 8 and 9.

[0049] Step 8: Take a number of training data samples from the experience replay pool, update the evaluation network using the loss function formula of the evaluation network, and update the policy network using the loss function formula of the policy network;

[0050] Step 9: Repeat steps 2 through 8 until the maximum number of training rounds is reached.

[0051] Furthermore, both the encoding and decoding networks have 11 hidden layers, and the input layer of the encoding network contains... There are 1 node, and the hidden layer contains 1 node. The output layer contains [number] nodes. The number of nodes in the input layer, hidden layer, and output layer of the decoding network are respectively [number of nodes]. as well as Both the policy network and the evaluation network have l2 hidden layers and N hidden layer nodes. hidden The number of nodes in the input and output layers of the policy network and the evaluation network are respectively... as well as During the execution phase, the time complexity of decoding and encoding the message is O(n). The time complexity of generating the action is During the training phase, the time complexity of updating the policy network and the evaluation network is O(n).

[0052] The beneficial effects of adopting the above technical solution are as follows: The multi-UAV path planning and power allocation method based on autoencoders provided by this invention explores the power allocation and path planning problem of multiple UAVs in scenarios where UAVs serve as airborne base stations for ground users, aiming to improve the service efficiency of UAV networks. Considering UAV cooperation, this invention proposes a multi-agent reinforcement learning framework based on autoencoders. This framework learns compact representations of UAV observations in an unsupervised manner using autoencoders, realizing efficient information transmission and communication between UAVs, improving the information acquisition capability of UAVs in distributed execution environments, and thus effectively improving the service efficiency of UAV networks. Ablation experiments fully demonstrate the effectiveness of the communication module proposed in this invention. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of a drone-assisted wireless network architecture provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of an automatic encoder structure provided in an embodiment of the present invention;

[0055] Figure 3A flowchart of a multi-UAV path planning and power allocation method based on an automatic encoder provided in an embodiment of the present invention;

[0056] Figure 4 Figure 4a shows the experimental results of the traditional multi-agent reinforcement learning method, Figure 4b shows the results of the TRPO algorithm, and Figure 4c shows the results of the PG algorithm. Detailed Implementation

[0057] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0058] like Figure 1 The illustrated drone-assisted wireless network architecture consists of aerial base stations composed of drones and ground-based macro base stations. Assume the network serves a square area with K ground users, k = {1, 2, 3, ..., K}. Ground users are served by N drones and ground base stations, n = {0, 1, 2, 3, ..., N}, where n = 0 represents a ground base station. Users near ground base stations are served directly by the base stations, while users outside the service range of ground base stations or in areas where ground base station signals are blocked are served by drones. All drones adjust their positions in real time based on user locations to provide stable and efficient network service to all user devices.

[0059] During the service process, it is assumed that the ground base station is always located at the center of the area, i.e., the origin. For ease of description, time is divided into multiple equal time slots δ. Assuming the user moves randomly within the area, the position of user k at time t is represented as... Assuming the drone always flies at a fixed altitude h0, the position of drone n at time t is represented as follows: The distance between user k and drone n at time t is calculated using the following formula:

[0060]

[0061] For a drone n, it is only necessary to know its position l at time t. n (t)=(x n (t),y n (t),h0), the direction of movement α∈[0,2π], and the velocity v∈[0,v max Then, its position at time t+1 can be calculated, as follows:

[0062] l n (t+1)=(x n (t)+vn (t)δcos(α n (t)),y n (t)+v n (t)δcos(α n (t)),h0) (2)

[0063] Thus, at a given initial position l of the drone n n In the case of (0), its movement trajectory within a period T can be obtained by the actions it takes in each time slot t.

[0064] In multi-drone networks, air-to-ground communication links are typically considered either line-of-sight (LoS) or non-line-of-sight (NLoS), highly dependent on the network environment. A link is considered LoS if it is visually blocked from the user device to the drone; otherwise, it is NLoS. Assuming random encounters with both LoS and NLoS scenarios in the environment, the probability of LoS at altitude h0 for the link between user device k and drone n is expressed as:

[0065]

[0066] Where μ1 and μ2 are constants related to the environment. Correspondingly, PNLOS kn (t)=1-PLoS kn (t). According to the free-space path loss model, the channel power gain between user equipment k and UAV n is expressed as:

[0067]

[0068] in, f c Let be the carrier frequency, c be the speed of light, χ be the path loss exponent, and β be the carrier frequency. LoS and β NLoS These are the attenuation factors for LoS and NLoS links, respectively.

[0069] Due to limitations in its transmission power, a drone can only serve users within a certain radius. Assuming the drone's service range is a circular area with radius R, the drone can detect information about user devices and other drones within this range. Simultaneously, the drone can broadcast observed information to other drones within its service range in each time slot. t It can also receive observation information broadcast by other drones. tThis allows for optimal decision-making in the next time slot. Considering that drones need to transmit messages to surrounding drones in a timely manner, and that the encoded message size is relatively small, this embodiment selects the basic Control and Non-Payload Communication (CNPC) as the communication channel model between drones.

[0070] Let p n (t) represents the transmit power of drone n at time t. Then the signal-to-noise ratio of user k and base station n at time t is expressed as:

[0071]

[0072] Where σ 2 This is additive noise power, assumed to be the same for all user equipment. k The interference exerted on user k by drones other than drone n is calculated using the following formula:

[0073] I k =∑ n′∈N\{n} .p n′ (t)g kn′ (t) (6)

[0074] Given the channel bandwidth W between user equipment k and drone n, the instantaneous transmission rate between user k and drone n at time t is expressed as:

[0075] R kn (t)=Wlog2(1+SNR kn (t)) (7)

[0076] Accordingly, the average rate of users within a region over a period T is expressed as:

[0077]

[0078] Under the premise of the above model, the optimization objective of the problem in this embodiment is modeled as follows:

[0079]

[0080] Where, p = {p n (t)|n∈N\{0}},L={l n (t)|n∈N\{0}}. Constraint (9b) indicates that the transmission rate between the drone and the user cannot exceed the drone's return rate. Constraint (9c) limits the minimum safe distance d′ between drones. min The constraint (9d) limits the maximum instantaneous speed v of the UAV during flight. maxThe constraint (9e) is used to ensure that the instantaneous transmit power of each UAV does not exceed the maximum transmit power P. N max .

[0081] It can be seen that the objective function (9a) is a non-convex function. Theoretically, the optimal solution to the problem can be found by exhaustive search, but its high time complexity is unacceptable in reality. Therefore, this embodiment adopts the MADRL method to solve the above joint optimization problem and uses the PPO algorithm to find an approximate optimal solution.

[0082] This embodiment models the path planning and power allocation problem of UAVs as a partially observable Markov decision process and proposes a multi-agent reinforcement learning algorithm with an autoencoder to solve it. Each UAV is treated as an agent, making decisions based on the surrounding environmental state and information from other UAVs, and continuously learning and updating its strategy. Considering the energy consumption of UAVs and the potential instability of model learning due to independent learning, this process employs a centralized training and distributed execution approach.

[0083] A Markov decision process (MDF) is a discrete-time stochastic process used to describe the probability distribution of future decisions and state transitions of a system given a current state and a sequence of observed events. It is modeled based on Markov chains, where each state is related to its previous state and currently observed information. In a Markov decision process, the decision-maker faces a series of decision problems, each based on the current state and past observations. In the above model, the agent acts as the decision-maker, and each decision it makes is related to the observed state of the environment. However, the agent is limited by its own physical conditions, and the environment it can observe is finite. Therefore, this embodiment models the above joint optimization problem as a partially observable Markov decision process, using a tuple...<S,A,R,P,O,γ> express.

[0084] Each time slot's environment has a specific state, denoted by st, where st represents the state of the environment at time t. Then {s1, s2, ..., s...} t ,..,s T Let} represent a series of states. The state of the environment includes the drone's position, the user's position, the transmission rates of the drone and the user, and the drone's communication messages. Therefore, the state of the environment at time t is defined as:

[0085] s t ={l n (t),l k (t),R kn (t),m(t)},n∈N\{0} (10)

[0086] Where m(t) represents the message from the drone at time t.

[0087] During the operational phase, a drone can only observe its surrounding environment. For drone n, its observation range includes the positions of surrounding drones and users, the transmission rate between users directly connected to drone n, and messages from surrounding drones. Therefore, using... Let represent a set of observation states. The observation state of UAV n at time t is represented as:

[0088]

[0089] Where, m n′ (t) represents the message from other drones n′ at time t.

[0090] In each time slot t, the UAV needs to take corresponding actions based on the observed state to enable the UE to obtain the maximum transmission rate. A = {a1, a2, ..., a...} t ,…,a T Let} represent a series of actions, including the drone's speed, direction, and transmission power. Then, the drone's actions at time t can be expressed as:

[0091]

[0092] in, This indicates the direction the drone is traveling at time t.

[0093] The environment will give agent n a reward based on the action of agent n at time t. This reward is denoted as r, so R = {r1, r2, ..., r} t ,…,r T} represents the reward corresponding to a series of actions of the drone. The overall reward function of the system in this embodiment is expressed as:

[0094]

[0095] Where, r t n This represents the reward corresponding to the action of drone n at time t; This represents the action of drone n at time t.

[0096] At some moment t, the state is s. t The environment will shift to s with a certain probability after the agent takes an action. t+1 This probability can be represented by P. γ represents the cumulative discount rate.

[0097] In decentralized multi-agent communication environments, the biggest challenge to communication between agents lies in their lack of a common communication foundation. This often leads to asynchrony among agents, thus affecting the stability of the training process. To address this issue, this embodiment introduces a self-supervised representation learning task. The goal is to learn a communication foundation that all agents can understand, enabling them to autonomously transmit and interpret the information they observe. Specifically, each agent shares a common architecture, such as... Figure 2 As shown.

[0098] The structure includes an encoding network and a decoding network. The encoding network generates a communication message based on the current state observation. t →φ ξ (o t )∈R comm ξ represents the encoder parameters, and the decoding network decodes the received messages into state information. ψ represents the decoder parameters. After encoding the observed information, the encoding network attempts to reconstruct the message into observed values ​​using the decoder. The objective of this embodiment is to minimize the following reconstruction loss function:

[0099]

[0100] On the one hand, drones are limited by their size and have limited computing power and resources. On the other hand, drones can only train independently based on received local information during the training process, which may lead to model uncertainty. Therefore, this embodiment adopts a framework of centralized training and distributed execution to train drones. Specifically, each drone has a policy network and an evaluation network. The policy network makes decisions based on observed environmental information, and the evaluation network scores the decisions based on environmental information. Both use a deep neural network as a function approximator.

[0101] During the training phase, this embodiment uses an experience replay mechanism to improve the efficiency of learning the optimal policy. All agent observation data are stored in an experience replay pool D, implemented on the master base station (MBS). Each observation is stored as an information tuple like {s, a, r, s′}. During each training iteration, a small batch of observation data is randomly sampled from the experience replay pool D to update the neural network, thereby reducing the interdependence between observations and improving learning efficiency. In subsequent experiments, this embodiment compares the performance of several popular DRL algorithms and ultimately selects the PPO algorithm to train the policy network and evaluation network. The loss function of the policy network is expressed as:

[0102]

[0103] in, γ(θ) represents the ratio of the new to the old policies, and θ is a parameter of the policy network. Clip is a clipping function that limits γ(θ) to the range [1-ε, 1+ε] to prevent excessive differences between the new and old policies. ε is a fixed hyperparameter. The advantage function is used to evaluate the advantage of action 'a' over a normal action. Q(s,a) represents the action value function, and V(s) represents the state value function.

[0104] The loss function for evaluating a network is expressed as:

[0105]

[0106] in, For the parameters of the value network, R = ∑γ t-1 r t This is a cumulative discount reward.

[0107] During the execution phase, each UAV needs to collaborate with surrounding UAVs to select appropriate ground-based associated users and determine its next position and transmission power. Specifically, the UAV first collects information about surrounding users and communication messages from other UAVs. Then, a decoder decodes the messages from other UAVs into understandable environmental information, and this information, along with the collected user information, is input into a policy network to generate an action. Finally, the UAV executes the action and sends the encoded observation information back to the environment.

[0108] The aforementioned path design method based on multi-agent reinforcement learning with autoencoders, such as... Figure 3 As shown, the main steps include:

[0109] Step 1: Initialize the policy network, evaluation network, encoding network, decoding network, experience replay pool, number of samples, maximum number of training rounds, and training period.

[0110] Step 2: In each training round, initialize the environment and obtain the initial state;

[0111] Step 3: For each time t, each UAV first acquires the messages and environmental observations of its neighboring UAVs, then passes the messages to the decoder to acquire the observations of its neighboring UAVs; then it passes the joint observations as input to the policy network to acquire the current action, and passes the joint observations as input to the encoding network to generate messages;

[0112] Step 4: The encoding network updates itself by combining the output of the decoding network with the mean squared error loss function;

[0113] Step 5: The environment executes drone actions and generates rewards for each drone, then proceeds to the next time step t+1;

[0114] Step 6: Put the drone's actions, observations, rewards, and state at time t into the experience replay pool;

[0115] Step 7: Repeat steps 2 to 6 until the training rounds reach the training cycle. For each network training session, execute steps 8 and 9.

[0116] Step 8: Take a number of training data samples from the experience replay pool, update the evaluation network using the loss function formula of the evaluation network, and update the policy network using the loss function formula of the policy network;

[0117] Step 9: Repeat steps 2 through 8 until the maximum number of training rounds is reached.

[0118] Assume that both the encoding and decoding networks have l1 hidden layers, and the input layer of the encoding network contains l1 hidden layers. There are 1 node, and the hidden layer contains 1 node. The output layer contains [number] nodes. The number of nodes in the input layer, hidden layer, and output layer of the decoding network are respectively [number of nodes]. as well as Both the policy network (actor) and the evaluation network (critic) have L2 hidden layers and N hidden layer nodes. hidden The number of nodes in the input layer and the output layer are respectively as well as Therefore, during the execution phase, the time complexity of decoding and encoding the message is... The time complexity of generating the action is During the training phase, the time complexity of updating the policy network and the evaluation network is O(n).

[0119] To verify the effectiveness of the communication module, this embodiment conducted three sets of ablation experiments, such as... Figure 4As shown in Figure (4a), the algorithm with a communication module undoubtedly demonstrates better performance than traditional algorithms. Taking Figure (4a) as an example, PPO without communication employs a traditional multi-agent reinforcement learning method. During the execution phase, the UAV makes decisions solely based on its own observation information, and the training algorithm for the AC network is PPO. The algorithm proposed in this invention provides a communication method for the UAV. The UAV embeds its observed state information into a low-dimensional communication message through an autoencoder, and then transmits the information to surrounding UAVs in a timely manner through the CNPC channel, thereby achieving information sharing. Compared to the traditional PPO multi-agent reinforcement learning method, the algorithm proposed in this invention improves performance by approximately 22.4%. The TRPO algorithm in Figure (4b) and the PG algorithm in Figure (4c) also show performance improvements of approximately 9% and 4.4%, respectively, after the addition of the communication module. The experimental results fully demonstrate the effectiveness of the communication module proposed in this invention.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for multi-UAV path planning and power allocation based on an automatic encoder, characterized in that: In a drone-assisted wireless network architecture, K ground users in the service area are served by N drones and ground base stations. The goal of optimizing stable and efficient network services is modeled as follows: ||l n (t)-l n′ (t)|||2≥d′ min (3) 0≤v n (t)≤v max (4) 0≤p n (t)≤p N max (5) Where, p = (p n (t)|n∈N\{0}},L={l n (t)|n∈N\{0}};R kn (t) represents the instantaneous transmission rate between user k and drone n at time t. There are K ground users and N drones, k = {1, 2, 3, ..., K}, n = {0, 1, 2, 3, ..., N}, and n = 0 represents the ground base station. Indicates the return trip rate of the drone; l n (t), l n′ (t) represents the positions of drone n and other drones n′ at time t, where n′∈N{0,n}; d′ min Indicates the minimum safe distance between drones; v n (t) represents the velocity of the drone n at time t; v max p represents the maximum instantaneous speed of the drone during flight. n (t) represents the transmit power of UAV n at time t; P N max The maximum transmit power of the drone is indicated by constraint (2); constraint (3) indicates that the transmission rate between the drone and the user cannot exceed the return rate of the drone; constraint (4) limits the minimum safe distance between drones; constraint (5) limits the maximum instantaneous speed of the drone during flight; and constraint (6) ensures that the instantaneous transmit power of each drone does not exceed the maximum transmit power. The path planning and power allocation problem of UAVs is modeled as a partially observable Markov decision process, and a multi-agent reinforcement learning algorithm with an autoencoder is proposed to seek an approximate optimal solution. Each UAV is regarded as an agent, which will make decisions based on the surrounding environmental state and information from other UAVs, and continuously learn and update its strategy. This process employs a centralized training and distributed execution approach.

2. The multi-UAV path planning and power allocation method based on an autoencoder according to claim 1, characterized in that: The unmanned aerial vehicle (UAV) assisted wireless network architecture selects basic unmanned aerial vehicle (UAV) control and non-payload communication (CNPC) as the communication channel model between UAVs.

3. The multi-UAV path planning and power allocation method based on an autoencoder according to claim 1, characterized in that: The partially observable Markov decision process is represented by a tuple.<S,A,R,P,O,γ> express; S = {s1, s2, ..., s} t , ..., s T } represents a set of environmental states, s t The state of the environment at time t is defined as follows: s t ={l n (t),l k (t),R kn (t),m(t)},n∈N\{0} Among them, l k m(t) represents the position of user k at time t, and m(t) represents the message of the drone at time t; A = {a1, a2, ..., a} t , ..., a T } represents a series of actions of the drone, including its speed, direction, and transmission power. t The action of the drone at time t is represented as: in, Indicates the direction the drone is traveling at time t; R = {r1, r2, ..., r} t ,...,r T } represents the reward corresponding to a series of actions of the drone, r t The reward function represents the action of the drone at time t. Where, r t n This represents the reward corresponding to the action of drone n at time t; This represents the action of drone n at time t; Represents a set of observed states. The observation state of UAV n at time t is represented as: Where, m n′ (t) represents the message from other drones n′ at time t; P represents the probability that a drone in one environmental state will move to another environmental state after taking an action; γ represents the cumulative discount rate.

4. The multi-UAV path planning and power allocation method based on an automatic encoder according to claim 3, characterized in that: Each of the aforementioned agents shares a common architecture, which includes an encoding network and a decoding network; the encoding network is based on the current state observation value o. t Generate a communication message φ ξ (o t ):o t →φ ξ (o t )∈R comm ξ represents the encoder parameters, and the decoding network will receive the message m. t Decoded into status information ψ represents the decoder parameters; after encoding the observation information, the encoding network attempts to reconstruct the message into observations using the decoder. The goal is to minimize the following reconstruction loss function: Among them, c i This represents the encoded message. This indicates the decoded message.

5. The multi-UAV path planning and power allocation method based on an autoencoder according to claim 4, characterized in that: In the centralized training and distributed execution, each UAV has a policy network and an evaluation network. The policy network makes decisions based on observed environmental information, and the evaluation network scores the decisions based on environmental information. Both use a deep neural network as a function approximator. During the training phase, an experience replay mechanism is used; all agent observation data are stored in an experience replay pool D, which is implemented on the main base station MBS. Each observation data is stored as an information tuple {s, a, r, s′}, where s, a, r, and s′ represent the state, action, reward, and the state at the next moment, respectively. During each training process, the neural network is updated by randomly sampling a small batch of observation data from the experience replay pool D. The policy network and evaluation network are trained using the PPO algorithm; the loss function of the policy network is expressed as: in, γ(θ) represents the ratio of the new and old policies, θ is a parameter of the policy network, clip is a clipping function that limits γ(θ) to the range of [1-ε, 1+ε] to prevent excessive differences between the new and old policies, and ε is a fixed hyperparameter. The advantage function is used to evaluate the advantage of action a compared to a normal action. Q(s, a) represents the action value function and V(s) represents the state value function. The loss function for evaluating a network is expressed as: in, For the parameters of the value network, R = ∑γ t-1 r t For cumulative discount rewards; During the execution phase, each UAV collaborates with surrounding UAVs to select appropriate ground-associated users and determine its position and transmission power for the next moment. The UAV first collects information about surrounding users and communication messages from other UAVs, then decodes the messages from other UAVs into understandable environmental information using a decoder, and inputs it along with the collected user information into the policy network to generate an action. Finally, the UAV executes the action and sends the encoded observation information to the environment.

6. The multi-UAV path planning and power allocation method based on an autoencoder according to claim 5, characterized in that: The specific steps of the multi-agent reinforcement learning algorithm with autoencoder to seek an approximate optimal solution are as follows: Step 1: Initialize the policy network, evaluation network, encoder network, decoder network, experience replay pool, number of samples, maximum number of training epochs, and training cycle; Step 2: In each training round, initialize the environment and obtain the initial state; Step 3: For each time t, each UAV first acquires the messages and environmental observations of its neighboring UAVs, then passes the messages to the decoder to acquire the observations of its neighboring UAVs; then it passes the joint observations as input to the policy network to acquire the current action, and passes the joint observations as input to the encoding network to generate messages; Step 4: The encoding network updates itself by combining the output of the decoding network with the mean squared error loss function; Step 5: The environment executes drone actions and generates rewards for each drone, then proceeds to the next time step t+1; Step 6: Put the drone's actions, observations, rewards, and state at time t into the experience replay pool; Step 7: Repeat steps 2 to 6 until the training rounds reach the training cycle. For each network training session, execute steps 8 and 9. Step 8: Take a number of training data samples from the experience replay pool, update the evaluation network using the loss function formula of the evaluation network, and update the policy network using the loss function formula of the policy network; Step 9: Repeat steps 2 through 8 until the maximum number of training rounds is reached.

7. The multi-UAV path planning and power allocation method based on an autoencoder according to claim 6, characterized in that: Both the encoding and decoding networks have l1 hidden layers. The input layer of the encoding network contains l1 hidden layers. There are 1 node, and the hidden layer contains 1 node. The output layer contains [number] nodes. The number of nodes in the input layer, hidden layer, and output layer of the decoding network are respectively [number of nodes]. as well as Both the policy network and the evaluation network have l2 hidden layers and N hidden layer nodes. hidden The number of nodes in the input and output layers of the policy network and the evaluation network are respectively... as well as During the execution phase, the time complexity of decoding and encoding the message is O(n). The time complexity of generating the action is During the training phase, the time complexity of updating the policy network and the evaluation network is O(n).

Images