An unmanned aerial vehicle path planning method, device and storage medium

Abstract

The application provides a UAV path planning method, device and storage medium, and relates to the technical field of wireless communication. The method comprises the following steps: determining the fair throughput of a wireless sensor network (WSN) based on a channel model of the WSN, wherein the fair throughput indicates the fairness of data transmission of sensor nodes occupying a wireless channel; establishing a Markov game model of the WSN based on the fair throughput and a UAV energy consumption model; iteratively training a multi-agent deep deterministic policy gradient (MADDPG) algorithm neural network based on the Markov game model, so that the fair throughput and the energy consumption of the UAV meet an optimization target; and planning a flight path of the UAV by using the trained MADDPG algorithm neural network. Based on this, the flight path of the UAV for executing a data collection task can be reasonably planned, the total energy consumption of the UAV is effectively reduced while the fairness among the sensor nodes is ensured, and the execution efficiency of the data collection task is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication, in particular to a method, device and storage medium for unmanned aerial vehicle path planning. BACKGROUND

[0002] Internet of Things (IoT) can realize transparent and seamless integration between a large number of different and heterogeneous terminal systems, and is considered as a key technology for future network development. As an important part of the Internet of Things, the wireless sensor network (WSN) is mainly responsible for connecting the sensor nodes deployed in the monitoring area through the wireless network, and performing data aggregation on the data collected by the sensor nodes to realize the monitoring of the environmental conditions. Unmanned aerial vehicle (UAV) has high mobility and good line-of-sight transmission characteristics, and is gradually applied to the field of wireless communication. The combination of unmanned aerial vehicle technology and communication network can provide long-distance and high-reliability line-of-sight transmission for WSN, so the unmanned aerial vehicle can reliably assist WSN to complete the data collection and transmission task.

[0003] However, in the scenario where multiple sensor nodes share limited channel resources, the fairness of data collection for each sensor node and the energy consumption of the unmanned aerial vehicle become key factors that limit the efficiency of performing the data collection task. Therefore, how to reasonably plan the flight path of the unmanned aerial vehicle so as to ensure the fairness among the sensor nodes while reducing the total energy consumption of the unmanned aerial vehicle is a problem to be solved at present. SUMMARY

[0004] The present application provides a method, device, apparatus and storage medium for unmanned aerial vehicle path planning, which can reasonably plan the flight path of the unmanned aerial vehicle, ensure the fairness among the sensor nodes, and effectively reduce the total energy consumption of the unmanned aerial vehicle.

[0005] In order to achieve the above-mentioned purpose, in a first aspect, the present application provides a method for unmanned aerial vehicle path planning based on multi-agent reinforcement learning, which comprises the following steps:

[0006] Step S1: determining the fair throughput of the sensor nodes in the WSN based on the channel model of the wireless sensor network (WSN), wherein the channel model is used to represent the wireless channel between the unmanned aerial vehicle and the sensor nodes in the WSN, and the fair throughput indicates the fairness of the sensor nodes occupying the wireless channel to transmit data;

[0007] Step S2: establishing a Markov game model of the WSN based on the fair throughput and an unmanned aerial vehicle energy consumption model;

[0008] Step S3: iteratively training a multi-agent deep deterministic policy gradient (MADDPG) algorithm neural network based on the Markov game model, so that the fair throughput and the energy consumption of the UAVs meet an optimization target;

[0009] Step S4: planning a flight path of the UAVs based on the trained MADDPG algorithm neural network.

[0010] In a possible implementation, the step S1 includes:

[0011] determining, based on the channel model, an uplink throughput of the sensor nodes;

[0012] determining, based on the uplink throughput and a fairness index of the WSN, the fair throughput.

[0013] In a possible implementation, the WSN includes M UAVs and N sensor nodes, where a UAV m establishes a communication link with a sensor node n, and the channel model is represented as a wireless communication link channel power gain between the UAV m and the sensor node n. ;

[0014] The determination of the uplink throughput of the sensor nodes based on the channel model is implemented by the following formula:

[0015]

[0016] wherein, SINR is a signal-to-interference-plus-noise ratio between the UAV m and the sensor node n, Pn(t) is a transmission power of the sensor node n at the t time slot, indicates whether the UAV m and the sensor node n establish a communication connection at the t time slot; Gn(t) is an additive white Gaussian noise, indicates an interference amount of the sensor node n when the UAV m and the node n establish the communication link; Rn(t) is a data transmission rate of the sensor node n for uploading data to the UAV m at the t time slot; indicates a bandwidth of the wireless channel; Un(t) is the uplink throughput of the sensor node n at the t time slot, and M and N are positive integers.

[0017] In a possible implementation, the WSN includes M UAVs and N sensor nodes, where a UAV m establishes a communication link with a sensor node n;

[0018] The determination of the fair throughput based on the uplink throughput and the fairness index of the WSN is implemented by the following formula:

[0019]

[0020] in, Let n be the uplink throughput of sensor node n in time slot t; Jain's fairness index; For fair throughput.

[0021] In one possible implementation, step S2 includes:

[0022] Treat each drone as an intelligent agent, and define the state space, action space, and reward function of each agent based on the fair throughput and drone energy consumption model.

[0023] Wherein, the state space of the intelligent agent m in time slot t At least includes: the current horizontal position of agent m and the energy consumed up to time slot t The Determined based on the aforementioned UAV energy consumption model;

[0024] The action space of the intelligent agent m in time slot t Including: the flight speed of the drone m and flight yaw angle ;

[0025] The reward function of agent m in time slot t It should include at least: a fair throughput maximization reward and a minimum energy consumption reward.

[0026] In one possible implementation, the state space Also includes: the remaining flight time of agent m Node communication scheduling ;

[0027] The reward function It also includes at least one of the following: a reward for guiding the drone to its destination as quickly as possible, a boundary penalty, and a collision penalty;

[0028] The node communication scheduling is defined as follows: ;

[0029] in, The available radius for sensor node communication. This represents the distance between the drone m and the sensor node n. H It is the fixed flight altitude of the drone.

[0030] In one possible implementation, the mission flight duration of the UAV is T, and the total flight time T of the UAV is discretely divided into K time slots; the optimization objective is expressed as:

[0031]

[0032] wherein, is the flight speed of the drone m; is the flight yaw angle of the drone m; t indicates the tth time slot; is the fair throughput; is the energy consumption of the drone up to the tth time slot.

[0033] In a possible implementation, each sensor node in the WSN simultaneously establishes a communication connection with at most one drone.

[0034] In a possible implementation, the MADDPG algorithm neural network includes a policy network, a target policy network, a value network, and a target value network. During the training process, each drone is regarded as an agent, and n agents are trained. The step S3 includes:

[0035] Step S31: for each training round, initializing an exploration noise distribution to obtain the initial state of each agent;

[0036] Step S32: taking the initial state as the input of the policy network, for each agent , obtaining the output action according to the policy network and the exploration noise distribution, wherein, represents the policy of the gth agent;

[0037] Step S33: performing the action to obtain the reward and the next state observation ;

[0038] Step S34: putting the pre-sampling data composed of the joint state observation of the n agents , the joint action , the joint reward , and the joint next state observation into an experience replay buffer;

[0039] wherein represents the state observation value of the n agents, represents the action of the n agents;

[0040] Based on the randomly sampled data in , the policy parameters of the policy network are updated: ;

[0041] updating the value parameters of the value network based on the loss function:

[0042]

[0043] wherein, is the loss function; is the policy parameter of the policy network, represents the value function, is the value parameter of the value network;

[0044] Step S35: for each agent g, update the parameters of the target policy network and the target value network:

[0045]

[0046] wherein, is a soft update coefficient, is the parameter of the target policy network; is the parameter of the target value network.

[0047] In one possible implementation, before performing step S1, the method further comprises:

[0048] establishing a channel model of the WSN based on a system model and a line-of-sight (LoS) model of the WSN;

[0049] The WSN includes M drones and N sensor nodes, and the channel power gain of a wireless communication link between a drone m and a sensor node n represented by the channel model is:

[0050]

[0051] wherein, represents the probability that there is a LoS link between the drone m and the sensor node n at time slot t, represents the elevation angle of the sensor node n to the drone m, is the channel power when the reference distance is 1 meter, represents the distance between the drone m and the sensor node n, is the path loss exponent, and M and N are both positive integers.

[0052] In a second aspect, a device for unmanned aerial vehicle path planning based on multi-agent reinforcement learning is provided, and the device comprises:

[0053] ​a fair throughput module configured to determine a fair throughput of sensor nodes in a wireless sensor network (WSN) based on a channel model of the WSN, the channel model being used to represent a wireless channel between a UAV and the sensor nodes in the WSN, the fair throughput indicating a fairness of the sensor nodes occupying the wireless channel to transmit data;

[0054] a modeling module configured to establish a Markov game model of the WSN based on the fair throughput and a UAV energy consumption model;

[0055] a training module configured to iteratively train a multi-agent deep deterministic policy gradient (MADDPG) algorithm neural network based on the Markov game model, so that the fair throughput and energy consumption of the UAV satisfy an optimization target;

[0056] a path planning module configured to plan a flight path of the UAV based on the trained MADDPG algorithm neural network.

[0057] In a third aspect, an electronic device is provided, which includes a memory and a processor, the memory storing at least one program, and the at least one program being executed by the processor to implement the multi-agent reinforcement learning based UAV path planning method according to the first aspect.

[0058] In a fourth aspect, a computer readable storage medium is provided, which stores at least one program, and the at least one program is executed by a processor to implement the multi-agent reinforcement learning based UAV path planning method according to the first aspect.

[0059] The technical solutions provided in the present application have at least the following technical effects:

[0060] By taking the fair throughput and the UAV energy consumption as the strategy optimization target of the UAV trajectory planning, and using the MADDPG algorithm architecture to train the neural network for planning the UAV cluster path in the mode of multi-agent reinforcement learning, a flight path that can balance the fairness and low energy consumption is obtained. Therefore, the technical solutions of the present application can reasonably plan the flight path of the UAV for performing the data collection task, effectively reduce the total energy consumption of the UAV while ensuring the fairness among the sensor nodes, and greatly improve the execution efficiency of the data collection task. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a schematic diagram of a system model of a wireless sensor network provided by an embodiment of the present application;

[0062] Figure 2 is a flowchart of a multi-agent reinforcement learning based UAV path planning method provided by an embodiment of the present application;

[0063] Figure 3 is an architecture diagram of a DDPG algorithm provided by an embodiment of the present application;

[0064] Figure 4 is an architecture diagram of a MADDPG algorithm provided by an embodiment of the present application;

[0065] Figure 5 is a schematic diagram of a flight path of a UAV determined by using the algorithm provided by the present application;

[0066] Figure 6 is a comparison diagram of the influence of the communication range of a node on the fair throughput under the algorithm provided by the present application and related algorithms;

[0067] Figure 7 is a comparison diagram of the influence of the communication range of a node on the energy consumption of a UAV under the algorithm provided by the present application and related algorithms;

[0068] Figure 8 is a schematic diagram of a UAV path planning device based on multi-agent reinforcement learning provided by an embodiment of the present application;

[0069] Figure 9 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0070] To further illustrate the embodiments, the present application provides accompanying drawings. These accompanying drawings are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can be used to explain the operating principles of the embodiments in conjunction with the related descriptions in the specification. Those of ordinary skill in the art should understand other possible implementations and advantages of the present application by referring to these contents. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0071] The present application will be further described in conjunction with the accompanying drawings and specific embodiments.

[0072] Compared with traditional ground base stations and static relays, UAVs have higher mobility as relay forwarding information, which can realize reliable data transmission over long distances in an environment with poor environment, poor communication quality and heavy transmission pressure. UAVs use the "storage-carrying-forwarding" mode, which can effectively and reliably assist wireless sensor networks to complete data collection and transmission tasks. In this way, not only the distance of data transmission is expanded, but also the interference and attenuation problems caused by traditional multi-hop networks are relieved.

[0073] The unmanned aerial vehicle assisted Internet of Things wireless sensor network is used for data collection, which effectively overcomes the problem of limited transmission range of the traditional wireless sensor network, but also brings new problems: in the scene of multiple unmanned aerial vehicles performing data collection tasks on multiple sensor nodes in the region, the scheduling and flight path planning of the unmanned aerial vehicles need to consider the fairness problem of data collection among multiple sensor nodes, but the energy of the unmanned aerial vehicles is limited, and the flight path or scheduling strategy formulated to consider fairness must also consider the energy consumption of the unmanned aerial vehicles. Therefore, how to balance the two and determine a reasonable flight path of the unmanned aerial vehicles is a problem to be solved at present.

[0074] Therefore, the present application provides a kind of unmanned aerial vehicle path planning method based on multi-agent reinforcement learning, which can effectively solve the above problems.

[0075] Firstly, the implementation environment involved in the present application is introduced.

[0076] Figure 1 A schematic diagram of a wireless sensor network system model provided by the present application is shown. Figure 1 A multi-unmanned aerial vehicle assisted Internet of Things wireless sensor network (WSN) 10 is shown, which includes M unmanned aerial vehicles 11 and N sensor nodes 12 (SNs). Each sensor node 12 is distributed in the illustrated area and is used to sense environmental information.

[0077] Optionally, each sensor node is equipped with an omnidirectional antenna for uploading the cached and / or collected data to the unmanned aerial vehicle.

[0078] In some optional embodiments, the unmanned aerial vehicles fly at a fixed height The flight time of the unmanned aerial vehicles is limited, and each unmanned aerial vehicle takes off from the same or different initial position, flies to the nearest charging station 13 after completing the data collection task.

[0079] The technical solutions of the present application will be described in detail below in combination with the wireless sensor network shown in the above Figure 1 The technical solutions of the present application will be described in detail below in combination with the wireless sensor network shown in the above Figure 2 is a flowchart of a method for planning the flight path of an unmanned aerial vehicle based on multi-agent reinforcement learning provided by an embodiment of the present application, and with reference to Figure 2 The technical solutions provided by the present application include the following steps 201 to 205.

[0080] Step 201: based on the system model of the wireless sensor network WSN and the line of sight LoS model, a channel model of the WSN is established.

[0081] In this embodiment, first, based on the initial position of the unmanned aerial vehicle in the WSN, the position of the sensor node and the charging station, a mathematical modeling is performed to obtain a system model of the WSN. The system model can represent the device deployment of the wireless sensor network and the execution environment of the data collection task.

[0082] For example, the task flight time of the unmanned aerial vehicle is set as T, and the total flight time T of the unmanned aerial vehicle is discretized into K time slots with a length of , that is, When the unmanned aerial vehicle enters the public communication range of the sensor node, the unmanned aerial vehicle can communicate with the sensor node in a time division multiple access (TDMA) manner.

[0083] The unmanned aerial vehicle is denoted as , and the set of sensor nodes is denoted as . For example, in a three-dimensional Cartesian coordinate system, the position of the unmanned aerial vehicle m at the t time slot is denoted as , and the coordinates of the sensor node n are denoted as . is the fixed flight height of the unmanned aerial vehicle.

[0084] The communication range of the sensor node n can be represented by a circular area with a radius of (refer to the circular area represented by a dashed line in Figure 1 ), and the starting position of the unmanned aerial vehicle m and the position of the charging station are denoted as and , respectively.

[0085] The embodiments of the present application take the collection of node data by the unmanned aerial vehicle during flight as an example for illustration. When the unmanned aerial vehicle passes through the public communication area of certain nodes, the unmanned aerial vehicle can establish a communication connection with the sensor node. In some possible implementation manners, the technical solutions of the embodiments of the present application can also be applied to flight tasks including hovering scenarios, which are not limited in the present application.

[0086] Further, based on the system model and the probability LoS model, an air-ground channel model of the WSN can be established. The air-ground channel model is used to represent the wireless channel between the unmanned aerial vehicle and the sensor node in the WSN. In the present application, the channel model is represented as: the wireless communication link channel power gain between the unmanned aerial vehicle m and the node n. The process of channel modeling is described in detail below.

[0087] The urban communication scenario can be divided into two cases of line-of-sight (LoS) communication and non-line-of-sight (NLoS) communication. The probabilities of the two cases can be represented by a probabilistic LoS channel model. The present application adopts the probabilistic LoS model to model the channel of the WSN, so that the factors such as obstacle shielding can be fully considered, and the technical solution of the present application has a wider application scenario.

[0088] In the probabilistic LoS channel model, the large-scale channel gain is represented as:

[0089] (1)

[0090] wherein, is the channel power gain at a reference distance of 1 meter (m) in the LoS case, is an additional attenuation factor in the NLoS case, is a path loss exponent. The probability that the node and the UAV are in the LoS link can be represented as a logistics function of the elevation angle:

[0091] (2)

[0092] wherein a and b are parameters related to the communication environment, denotes the elevation angle of the UAV and the ground node. Therefore, the probability of the NLoS link can be obtained by

[0093] Therefore, the channel model can be represented as: the wireless communication link channel power gain between the UAV m and the node n:

[0094] (3)

[0095] wherein, denotes the probability of the LoS link between the UAV m and the node n at time slot t, denotes the elevation angle of the node n to the UAV m, is the channel power at a reference distance of 1 m, denotes the distance between the UAV m and the node n, denotes the path loss exponent. Further, according to the basic formula of a triangle, , and the calculation formula is as follows.

[0096] (4)

[0097] (5)

[0098] ​ (6)

[0099] where H is a fixed flight height of the UAV, is a carrier frequency, is a light speed.

[0100] Step 202: determining a fair throughput of a sensor node in a wireless sensor network (WSN) based on a channel model of the WSN.

[0101] The channel model is used to represent a wireless channel between the UAV and the sensor node in the WSN, and the fair throughput indicates the fairness of the sensor node occupying the wireless channel to transmit data.

[0102] In some embodiments, the step 202 includes the following steps 2-1 and 2-2.

[0103] Step 2-1: determining an uplink throughput of the sensor node based on the channel model.

[0104] In order to better reflect the cooperative relationship between the UAVs, the present application sets that each node can be accessed by at most one UAV. Exemplarily, the present application adopts a binary variable to represent the connection relationship between the UAV m and the node n at time t, when , it means that the UAV m is in the communication range of the sensor node n, and the UAV m is providing data collection service for the sensor node n, otherwise .

[0105] Based on this, if a sensor node n is accessed by a UAV m, other UAVs no longer provide collection service for it. The connection relationship between the UAV and the ground node is strongly related to the flight path of the UAV, therefore, when multiple UAVs simultaneously enter the communication range of a node which is not served, the present application can ensure that the UAV closest to the node provides data collection service for it first. The constraint condition can be expressed as follows:

[0106] (7)

[0107] Further, the present application also considers the collision problem between the UAVs in the multi-UAV scenario, and uses to represent the minimum safe distance between the UAVs. Exemplarily, the distance constraint between the UAVs can be expressed as:

[0108] (8)

[0109] wherein, is the position of a UAV, is the position of another UAV.

[0110] The application also considers that the communication connection between the UAV and the sensor node is interfered by other UAVs in the multi-UAV scenario. In some embodiments, the UAV can establish communication connections with multiple sensor nodes simultaneously in a time-division multiple access manner when the UAV is within the common communication range of the sensor nodes. Assuming that the transmission power of the sensor node n in the t time slot is denoted as Pn(t), if the UAV m establishes a communication link with the sensor node n at this time, the signal to interference plus noise ratio (SINR) between the UAV m and the sensor node n can be expressed as:

[0111] (9)

[0112] wherein, is an additive white Gaussian noise, denotes the interference of other nodes when the UAV m establishes a communication link with the node n. The wireless channel bandwidth is denoted as B.

[0113] Therefore, the data transmission rate of the sensor node n to the UAV m in the t time slot can be expressed as:

[0114] (10)

[0115] The uplink throughput of the sensor node n in the t time slot can be expressed as:

[0116] (11) Step 2-2, based on the uplink throughput and the fairness index of the WSN, the fairness throughput of the sensor node is determined.

[0117] The application considers the fairness among the sensor nodes and introduces the Jain fairness index to measure the fairness among the nodes, which is expressed as follows:

[0118]

[0119] (12) According to the Cauchy-Schwarz inequality, it can be obtained that

[0120] The greater the difference in the throughputs among the nodes, the smaller the fairness index , that is, the greater the fairness index indicates the more fair the data collection task of the node. The application introduces the concept of fairness throughput to balance the total throughput of the system and the fairness among the individual sensor nodes, and the fairness throughput is defined as:

[0121] ​​​​

[0122] (13)

[0123] In the related art, much attention is focused on the total throughput of the system, resulting in frequent access of some nodes by the UAV and very small throughput of other nodes. The fair throughput proposed in the present application combines the uplink throughput of the sensor node and the fairness index of the system as a whole to calculate, thereby avoiding the above situation and achieving more accurate fairness measurement.

[0124] In some embodiments, before step 203 is performed, the energy consumption of the UAV is modeled to obtain a UAV energy consumption model. In one specific implementation, a prior art UAV energy consumption model can be used.

[0125] In another specific implementation, for example, the present application takes the scenario of a rotor-wing UAV assisting the data collection of an Internet of Things system as an example (other types of UAVs can also be used, and the present application is not limited thereto). The flight energy consumption of a rotor-wing UAV is generally related to the flight state of the UAV, air, and other factors. The flight speed of the UAV is V The propulsion power consumption of the UAV can be modeled as:

[0126] (14)

[0127] The propulsion power consumption of the UAV includes blade profile power, induced power, and parasitic power, which correspond to the three parameters from left to right in the above formula (13), respectively. Among them, and are two constants representing the blade profile power and induced power of the UAV in the hovering state, represents the tip speed of the rotor blade, represents the average moving blade induced speed in hovering, represents the fuselage drag ratio of the UAV, represents the rigidity of the rotor, represents the air density, represents the rotor disc volume.

[0128] Therefore, the energy consumed by the UAV m up to time slot t can be represented as (15)

[0129] wherein is the flight speed of the UAV m at time slot t, is the time slot length.

[0130] Step 203: Based on the fair throughput and the UAV energy consumption model, a Markov game model of the WSN is established.

[0131] The present application will be optimized for the flight path of the UAV, modeled as a multi-agent extension of Markov decision process, namely, Markov game model. Specifically, each UAV is regarded as an agent, and the state space, action space and reward function of each agent are defined as follows:

[0132] (1) State space: the state space of agent m at time slot t is composed of the current horizontal position of the agent , the energy consumed by the agent up to time slot t , the remaining flight time and the node communication schedule , and is specifically defined as:

[0133] (16)

[0134] wherein is the communication schedule of the first sensor node, is the communication schedule of the Nth sensor node. It should be noted that is not equal to , is the node communication schedule when n is N.

[0135] (2) Action space: the action space of agent m at time slot t is composed of the flight speed of UAV m and the flight yaw angle , and is specifically defined as:

[0136] (17)

[0137] (3) Reward function: the reward function of agent m at time slot t is composed of five parts: the reward of maximizing the fair throughput, the reward of guiding the UAV to reach the destination as soon as possible, the reward of minimizing the energy consumption, the boundary penalty and the collision penalty, and can be expressed as:

[0138] (18)

[0139] wherein represents the horizontal distance of UAV m to the nearest charging station, represents the remaining time of the task, represents the number of times that UAV m flies out of the boundary, represents the collision penalty of the UAV, , , and are weight parameters, which are used to adjust the influence of each value on the reward value, as well as to adjust the order of magnitude of each value, for example, is a weight parameter of is a weight parameter of is of the order of is and and

[0140] The optimization objective of the present application is to jointly optimize the flight speed and yaw angle (parameters constituting the flight path) of multiple unmanned aerial vehicles to complete the data collection task of all sensor nodes within a specified time, maximize the total fair throughput and minimize the energy consumption of the unmanned aerial vehicles. The optimization objective can be modeled as a multi-objective optimization problem, expressed as:

[0141]

[0142] wherein formula (19a) represents the initial state and the final state of the unmanned aerial vehicles, formula (19b) and (19c) represent the flight speed and yaw angle limits of the unmanned aerial vehicles respectively, formula (19d) represents that each node is provided with data collection service by at most one unmanned aerial vehicle, and formula (19e) represents the safety distance constraint between the unmanned aerial vehicles.

[0143] Step 204: Based on the Markov game model, the neural network of the multi-agent deep deterministic policy gradient MADDPG algorithm is iteratively trained to make the fair throughput and the energy consumption of the unmanned aerial vehicles satisfy the optimization objective.

[0144] After determining the Markov game process, the present application adopts the method of multi-agent deep reinforcement learning to realize the planning of the flight path of the unmanned aerial vehicles.

[0145] Specifically, the present application adopts the MADDPG algorithm to adaptively adjust the flight path of the unmanned aerial vehicles. The MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm is an extension of the DDPG algorithm in the multi-agent, and is used to solve related problems in the multi-agent environment in the continuous action space.

[0146] Figure 3 is a DDPG algorithm architecture provided by an embodiment of the present application. Refer to Figure 3 ​​​​​​, the DDPG (deep deterministic policy gradient) algorithm is a deep reinforcement learning algorithm based on an Actor-Critic framework, and is an algorithm proposed to solve continuous action control problems. The structure of DDPG can be divided into two parts of a policy network and a value network. In order to break the correlation of the training data and increase the stability of the training, the DDPG algorithm introduces the experience replay and double network structure of the deep Q network (DQN), wherein the experience replay means that a plurality of experience data samples produced by the interaction between the policy network and the environment are stored in an experience replay buffer, and a small batch of data (Mini-Batch) samples are extracted for training; the double network structure means that the policy network and the value network have their own current network and target network respectively.

[0147] Figure 4 is an architecture diagram of a MADDPG algorithm provided by an embodiment of the present application. Referring to Figure 4 Similar to DDPG, each agent in MADDPG also has four neural networks, and the training of each agent is similar to the training process of a single DDPG algorithm. The input of the Actor network is the state observation value of the agent, and the output is the action performed by the agent in the current state. Unlike DDPG, MADDPG adopts a centralized training and distributed execution method. In the DDPG algorithm, the input of the Critic is a state-action pair information to calculate the Q value (value of the policy), but in MADDPG, the Critic of each agent inputs not only its own state-action pair, but also the actions of other agents to determine the Q value of the current decision. Therefore, by using the MADDPG algorithm, the cooperation and conflict relationship between multiple unmanned aerial vehicles can be effectively simulated to formulate a reasonable unmanned aerial vehicle flight path.

[0148] In the embodiment, the neural network of the MADDPG algorithm includes a policy network, a target policy network, a value network, and a target value network. In the training process, each unmanned aerial vehicle is regarded as an agent. The following describes the principle of the iterative training process of the neural network of the MADDPG algorithm, which includes the following steps a to e.

[0149] Step a: For each training round, initialize an exploration noise distribution , obtain the initial state of each agent .

[0150] Step b: Take the initial state as the input of the policy network, and for each agent , according to the policy network and the exploration noise distribution, obtain the output action ,in, Let represent the policy of the g-th agent.

[0151] Step c: Perform the action Receive reward and the next state observation .

[0152] Step d: Observe the joint state of the n agents. Joint actions Joint awards and joint next state observation Presampled data consisting of "state-action-reward" tuples Place it into the experience replay buffer; where, Represents the state observations of n agents. This represents the actions of the n agents.

[0153] The neural network is trained based on data randomly sampled from D, and the policy parameters of the policy network are updated. This process can be represented as:

[0154]

[0155] The value parameters of the value network are updated based on the loss function. This process can be represented as:

[0156]

[0157] in, It is a loss function; These are the policy parameters of the policy network. Represents the value function. It is the value parameter of this value network.

[0158] Step e: For each agent g, update the parameters of the target policy network and the target value network:

[0159]

[0160] in, It is a soft update coefficient. These are the parameters of the target policy network; These are the parameters of the target value network.

[0161] Repeat the training process described in steps a-e above to train the network until the maximum number of steps T is reached in each round, or the agent reaches the specified position, or the training reaches the maximum number of rounds M, then end the training.

[0162] Step 205: Based on the trained MADDPG algorithm neural network, plan the flight path of the UAV.

[0163] In the embodiments of the present application, the parameters indicating the flight path include the aforementioned action parameters (flight speed and yaw angle) in modeling. In other embodiments, the parameters indicating the flight path can also be other parameters in different modeling coordinate systems, and the present application is not limited thereto.

[0164] In the embodiments of the present application, the planning of the UAV path can be achieved in the following two ways:

[0165] 1. Online planning: load the trained MADDPG algorithm neural network into the UAV cluster; the UAV inputs the neural network according to the real-time acquired environmental parameters, converts the action parameters (such as the flight speed and flight yaw angle output by the neural network into corresponding UAV control instructions, and controls the UAV trajectory in real time. Of course, the UAV can communicate with other UAVs to obtain the required parameters, such as the positions of other UAVs, the task execution status, etc., and the present application is not limited thereto.

[0166] 2. Offline planning: load the planned flight path related parameters into the UAV, and the UAV directly flies according to the planned path.

[0167] Figure 5 is a schematic diagram of a UAV flight path determined by the algorithm provided in the embodiments of the present application. Referring to Figure 5 , the flight paths of the two UAVs with initial positions of (0, 0) and (400, 0) respectively pass through the sensor nodes along the way uniformly, and finally fly to the charging station.

[0168] The present application takes the fair throughput and the energy consumption of the UAV as the strategy optimization target of the UAV trajectory planning, and uses the MADDPG algorithm architecture to train the neural network for planning the flight path of the UAV cluster in the mode of multi-agent reinforcement learning, so as to obtain a flight path that can balance the fairness and low energy consumption. Therefore, the technical solution of the present application can reasonably plan the flight path of the UAV for executing the data collection task, while ensuring the fairness among the sensor nodes and effectively reducing the total energy consumption of the UAV, thereby greatly improving the execution efficiency of the data collection task.

[0169] The following describes the significant progress achieved by the algorithm of the present application relative to related technologies through some drawings. The related algorithms include: DDPG, Greedy (greedy algorithm) and FSMADDPG (fixed speed MADDPG).

[0170] Figure 6is a comparison chart of the influence of the node communication range on the fair throughput under the algorithm and the related algorithm provided by the embodiment of the application. As shown in Figure 6 , the fair throughput achieved by the algorithm is greater than that of the related algorithm, and significantly increases as the communication range becomes larger.

[0171] Figure 7 is a comparison chart of the influence of the node communication range on the energy consumption of the unmanned aerial vehicle under the algorithm and the related algorithm provided by the embodiment of the application. As shown in Figure 7 , the energy consumption achieved by the algorithm is less than the average performance of the related algorithm in the general trend.

[0172] The application also provides a schematic diagram of an unmanned aerial vehicle path planning device based on multi-agent reinforcement learning, referring to Figure 8 , the device comprises:

[0173] a fair throughput module 801 configured to determine the fair throughput of a sensor node in a wireless sensor network (WSN) based on a channel model of the WSN, the channel model being used to represent a wireless channel between an unmanned aerial vehicle and the sensor node in the WSN, and the fair throughput indicating the fairness of the sensor node occupying the wireless channel to transmit data;

[0174] a modeling module 802 configured to establish a Markov game model of the WSN based on the fair throughput and an energy consumption model of the unmanned aerial vehicle;

[0175] a training module 803 configured to iteratively train a multi-agent deep deterministic policy gradient (MADDPG) algorithm neural network based on the Markov game model, so that the fair throughput and the energy consumption of the unmanned aerial vehicle satisfy an optimization target;

[0176] a path planning module 804 configured to plan a flight path of the unmanned aerial vehicle based on the trained MADDPG algorithm neural network.

[0177] In a possible implementation manner, the device further comprises:

[0178] a channel modeling module configured to establish a channel model of the WSN based on a system model of the WSN and a line-of-sight (LoS) model;

[0179] The WSN comprises M unmanned aerial vehicles and N sensor nodes, and the wireless communication link channel power gain between an unmanned aerial vehicle m and a sensor node n represented by the channel model is:

[0180] ;

[0181] wherein, denotes the probability that there is a LoS link between the UAV m and the sensor node n at time slot t, denotes the elevation angle of the sensor node n to the UAV m, is the channel power when the reference distance is 1 meter, denotes the distance between the UAV m and the sensor node n, is the path loss exponent, and M and N are both positive integers.

[0182] In a possible implementation, the mission flight time of the UAV is T, and the total flight time T of the UAV is discretized into K time slots; the optimization objective is denoted as:

[0183]

[0184] wherein, is the flight speed of the UAV m; is the flight yaw angle of the UAV m; t indicates the t-th time slot; is the fair throughput; is the energy consumption of the UAV up to the t-th time slot.

[0185] It should be noted that the UAV path planning apparatus based on multi-agent reinforcement learning provided in the above embodiments is only used as an example for dividing the above functional modules when implementing the corresponding steps, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. In addition, the UAV path planning apparatus based on multi-agent reinforcement learning provided in the above embodiments and the UAV path planning method based on multi-agent reinforcement learning described above belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0186] The term "at least one" in the present application means one or more, and the term "multiple" in the present application means two or more, for example, multiple nodes refer to two or more nodes.

[0187] The UAV path planning method based on multi-agent reinforcement learning provided in the present application can be executed by an electronic device. Figure 9 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application, such as Figure 9As shown, the electronic device includes a processor 901, a memory 902, a bus 903, and a computer program stored in the memory 902 and executable on the processor 901, the processor 901 includes one or more processing cores, the memory 902 is connected to the processor 901 through the bus 903, the memory 902 is configured to store program instructions, and the processor executes the computer program to implement all or part of the steps in the above method embodiments.

[0188] Further, as an executable solution, the electronic device described above can be a computer unit, which can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer unit can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above-mentioned constituent structure of the computer unit is merely an example of the computer unit, and does not constitute a limitation on the computer unit, and can include more or fewer components than the above, or combine certain components, or different components. For example, the computer unit can also include an input / output device, a network access device, a bus, and the like, and the embodiments of the present application do not limit this.

[0189] Further, as an executable solution, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like, and the processor is the control center of the computer unit, and is connected to each part of the computer unit through various interfaces and lines.

[0190] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the computer unit by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store data created according to the use of the mobile phone and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0191] The application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to realize the steps of the method provided in the embodiments of the application.

[0192] The modules / units integrated in the computer unit, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. The computer program is executed by a processor to realize the steps of the above-mentioned various method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording media, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM) and software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of legislation and patent practice in the jurisdiction.

[0193] Although the application is specifically shown and described in combination with the preferred embodiments, it should be understood by those skilled in the art that various changes can be made in form and details without departing from the spirit and scope of the application as defined in the appended claims, and all such changes are intended to be within the protection scope of the application.

Claims

1. A method for unmanned aerial vehicle path planning, characterized in that, The method includes: Step S1: Based on the channel model of the Wireless Sensor Network (WSN), determine the fair throughput of the sensor nodes in the WSN. The channel model is used to represent the wireless channel between the UAV and the sensor nodes in the WSN, and the fair throughput indicates the fairness of the sensor nodes occupying the wireless channel to transmit data. Step S2: Based on the fair throughput and UAV energy consumption model, establish the Markov game model of the WSN; Step S3: Based on the Markov game model, iteratively train the multi-agent deep deterministic policy gradient (MADDPG) algorithm neural network to ensure that the fair throughput and the energy consumption of the UAV meet the optimization objectives. Step S4: Based on the trained MADDPG algorithm neural network, plan the flight path of the UAV; Step S2 includes: Treat each drone as an intelligent agent, and define the state space, action space, and reward function of each agent based on the fair throughput and drone energy consumption model. Wherein, the state space of the intelligent agent m in time slot t At least includes: the current horizontal position of agent m and the energy consumed up to time slot t The Determined based on the aforementioned UAV energy consumption model; The action space of the intelligent agent m in time slot t Including: the flight speed of the drone m and flight yaw angle ; The reward function of agent m in time slot t It should include at least: a fair throughput maximization reward and a minimum energy consumption reward.

2. The method according to claim 1, characterized in that, Step S1 includes: Based on the channel model, the uplink throughput of the sensor node is determined; The fair throughput is determined based on the uplink throughput and the fairness metric of the WSN.

3. The method according to claim 2, characterized in that, The WSN comprises M unmanned aerial vehicles (UAVs) and N sensor nodes, wherein UAV m establishes a communication link with sensor node n. The channel model is expressed as: the channel power gain of the wireless communication link between UAV m and sensor node n. ; The uplink throughput of the sensor node, based on the channel model, is determined using the following formula: ; in, The signal interference between the UAV m and the sensor node n is increased by the SINR (Signal-to-Noise Ratio). Let n be the transmission power of sensor node n in time slot t. Indicates whether the UAV m in time slot t has established a communication connection with the sensor node n; It is additive white Gaussian noise. This represents the amount of interference from other sensor nodes that occurs when the drone m establishes a communication link with node n; Let be the data transmission rate of sensor node n uploading data to UAV m in time slot t; B represents the bandwidth of the wireless channel. Let M be the uplink throughput of sensor node n in time slot t, where M and N are positive integers.

4. The method according to claim 2, characterized in that, The WSN includes M drones and N sensor nodes, wherein drone m establishes a communication link with sensor node n; The fair throughput, determined based on the uplink throughput and the fairness metric of the WSN, is achieved through the following formula: ; in, Let n be the uplink throughput of sensor node n in time slot t; Jain's fairness index; For fair throughput.

5. The method according to claim 1, characterized in that, The state space Also includes: the remaining flight time of agent m Node communication scheduling ; The reward function It also includes at least one of the following: a reward for guiding the drone to its destination as quickly as possible, a boundary penalty, and a collision penalty; The node communication scheduling is defined as follows: ; in, The available radius for sensor node communication. H represents the distance between the drone m and the sensor node n, and H is the fixed flight altitude of the drone.

6. The method according to claim 1, characterized in that, The mission flight time of the UAV is T, and the total flight time T of the UAV is discretized into K time slots; the optimization objective is expressed as: ; in, That is the flight speed of the drone, m. The yaw angle of drone m is the flight yaw angle; t indicates the t-th time slot. For fair throughput; The energy consumption of the drone up to time slot t.

7. The method according to claim 1, characterized in that, Each sensor node in the WSN can establish a communication connection with at most one UAV at a time.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the method as claimed in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is executed by a processor to implement the method as described in any one of claims 1 to 7.

Images