An unmanned aerial vehicle path planning method based on a deep deterministic policy gradient algorithm

Abstract

The application discloses a kind of unmanned plane path planning methods based on deep deterministic policy gradient algorithm, which comprises the following steps: constructing unmanned plane auxiliary communication sensing integrated system model, including ground sensor model, unmanned plane model, channel model and radar detection model;Based on the optimization target of unmanned plane task, the multi-objective optimization function of unmanned plane task is constructed;Based on deep deterministic policy gradient algorithm training unmanned plane, the state space, action space and reward function of DDPG algorithm are constructed according to the multi-objective optimization function of unmanned plane task;According to the state space, action space and reward function constructed, the optimal unmanned plane path planning result is output by training unmanned plane. The application improves the autonomy of unmanned plane flight, and can make unmanned plane serve more target ground sensors with lower energy consumption.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) path planning technology, and specifically to a UAV path planning method based on a deep deterministic policy gradient algorithm. Background Technology

[0002] With the development of wireless communication technology, the data transmission rate of next-generation mobile communications is constantly improving. The future sixth-generation mobile communication network will integrate technologies such as the Internet of Things, artificial intelligence, big data, and blockchain, enabling deep integration of communication and sensing. Furthermore, future integrated air-space-ground communication requires communication networks to support high-speed and ultra-reliable low-latency communication for all users, and to provide powerful environmental sensing and computing capabilities. Therefore, higher demands are placed on the information transmission and data processing capabilities of next-generation mobile communication technologies.

[0003] Integrated communication and sensing technology uses the same wireless transmission signal for both communication and sensing. This allows the system to perform end-to-end communication while simultaneously sensing target objects using feedback information from the wireless signal, thus improving the system's spectral efficiency. Furthermore, by sharing spectrum and hardware platform resources and implementing unified signal waveform and constellation design, integrated communication and sensing systems can reduce hardware costs, decrease size and power consumption, and improve overall system efficiency. Therefore, integrated communication and sensing technology has gradually become a research hotspot in both academia and industry, with research directions primarily including integrated communication and sensing architecture research, signal waveform design, interference cancellation, and network convergence technologies.

[0004] Due to the high mobility of drones and the random deployment characteristics of IoT platforms, drone path planning has become a key research area in drone-assisted communication networks. Drones should possess autonomous decision-making capabilities when facing complex environments to adjust and control their flight paths in real time. However, traditional decision optimization algorithms suffer from low efficiency, poor adaptability, and high computational cost in complex environments, significantly impacting the autonomy of drone path planning. Summary of the Invention

[0005] To overcome the defects and shortcomings of existing technologies, this invention provides a UAV path planning method based on the Deep Deterministic Policy Gradient Algorithm (DDPG). This invention trains the UAV using the DDPG algorithm to detect the most ground sensor devices and collect data from the sensors while consuming the least amount of energy.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a UAV path planning method based on a deep deterministic policy gradient algorithm, comprising the following steps:

[0008] Construct a UAV-assisted integrated communication and sensing system model, including a ground sensor model, a UAV model, a channel model, and a radar detection model;

[0009] The construction of the ground sensor model includes: randomly setting up multiple sensor devices on the ground, the sensor devices monitoring and collecting the latest environmental status data in real time, and storing the data in the sensor buffer for collection by the UAV;

[0010] The drone model includes a flight dynamics model of the drone and a power consumption model of the drone during flight. The coordinate position of the drone is obtained based on the flight dynamics model of the drone.

[0011] The construction of the channel model includes: constructing an air-to-ground channel model based on NLoS links and LoS links;

[0012] The construction of the radar detection model includes: constructing a radar detection model based on radar detection information, which includes radar transmit power, antenna gain, effective receiving area, radar cross-section and minimum detectable signal; obtaining the specific coordinate position of the sensor device based on the radar detection model and the coordinate position of the UAV; and adding it to the known information set of the UAV as a candidate target sensor for information collection.

[0013] Based on the optimization objectives of UAV missions, a multi-objective optimization function for UAV missions is constructed;

[0014] The UAV is trained based on the Deep Deterministic Policy Gradient Algorithm (DDPG), and the state space, action space and reward function of the DDPG algorithm are constructed according to the multi-objective optimization function of the UAV mission.

[0015] The drone selects a target sensor device and moves to hover directly above it to collect data. After the data collection is completed, it selects a new target sensor device and continues to perform the data collection task. During the entire mission, the drone detects new ground sensor devices and adds them to the drone's known information set as candidate targets for data collection.

[0016] The drone is trained based on the constructed state space, action space, and reward function, and the optimal drone path planning result is output.

[0017] As a preferred technical solution, the sensor device uses a time-division multiple access protocol for transmission, and the data transmission priority of different sensor devices is represented as follows:

[0018]

[0019] Among them, f k(t) represents the data waiting to be collected in the sensor data buffer at time t, a k a represents the data generation rate of sensor k at time t. k The sensor follows a Poisson distribution, but the Poisson distribution parameters are different for different sensor devices. max This represents the maximum capacity of the data cache.

[0020] As a preferred technical solution, the flight dynamics model of the UAV is expressed as follows:

[0021]

[0022] Among them, (x u y u , z u The ) represents the drone's coordinates, and v represents the drone's flight speed. The yaw angle represents the drone's angle of flight, and h represents the drone's fixed flight altitude.

[0023] As a preferred technical solution, the power consumption model of the UAV during flight is expressed as follows:

[0024] P(V)=P b (V)+P i (V)+P a (V)

[0025]

[0026]

[0027]

[0028] Among them, P b P0 is the blade profile power, P0 is the blade profile power when the UAV is hovering, U is the tip speed of the UAV rotor, and P0 is the blade tip speed. i (V) represents the induced power of the UAV, P d P is the induced power when the drone is hovering, v0 is the average rotor air-induced velocity when the drone is hovering, and P is the induced power when the drone is hovering. a (V) represents the parasitic power of the UAV, and b0, ρ, m, and S represent the fuselage drag coefficient, air density, rotor solidity, and rotor disk area, respectively.

[0029] As a preferred technical solution, the loss model of the air-to-ground channel model is expressed as follows:

[0030]

[0031] Where η represents the channel power gain, Let μ represent the propagation distance between the drone and sensor device k, β represent the path loss coefficient, and μ represent the propagation distance between the drone and sensor device k. NLoS It is the attenuation coefficient of the NLoS link.

[0032] As a preferred technical solution, for sensor device k, the Loss probability at time t is expressed as:

[0033]

[0034]

[0035] Where c and d are two constants that depend on the carrier frequency and the environment, θ k d represents the elevation angle between the drone and the sensor equipment. k H represents the distance between the drone and the sensor device, and H represents the altitude of the drone.

[0036] The channel power gain between the drone and sensor device k is:

[0037]

[0038] As a preferred technical solution, the radar detection model is represented as follows:

[0039]

[0040] Among them, P t Let G be the radar transmit power, G be the antenna gain, and A be the antenna gain. e The effective receiving area is σ, where σ is the radar cross-section and s is the effective receiving area. min It is the smallest detectable signal.

[0041] As a preferred technical solution, the multi-objective optimization function for the UAV mission is expressed as:

[0042] max(R total D totat -E total )

[0043] Among them, R total D represents the total data transmission rate of the drone collecting data from ground sensor devices. total E represents the number of ground sensor devices detected by the drone during its mission. total This indicates the total energy consumption of the drone during the mission.

[0044] As a preferred technical solution, the state space is constructed as a quintuple S = {X} u X d X f X l X t}, where Xu The coordinates of the drone are represented by X. d The coordinates of the target sensor device, X f X represents the number of times the drone went outside the map. l X represents the number of sensors detected by the drone that caused data overflow. t This indicates the number of sensor devices detected by the drone;

[0045] The action space is represented as follows: Where v represents the flight speed of the drone, The yaw angle representing the drone;

[0046] The reward function is expressed as: R = 10D total -PD f -D l +D t +D d ;

[0047] Among them, D total Let P be the total information transmission rate, and D be the energy consumption of the UAV. f D is the penalty for a drone flying outside the map area. l D is the penalty for data overflow in ground sensor equipment. t For the reward item of the drone detecting ground sensor equipment, D d The reward item indicates the distance between the drone and the target sensor device.

[0048] As a preferred technical solution, the deep deterministic policy gradient algorithm is based on the Actor-Critic framework. The Actor network, responsible for fitting the policy function, and the Critic network, responsible for fitting the value function, are represented by μ networks and Q networks, respectively. A fixed network technique is employed, with a target neural network added to both the Actor and Critic networks. New parameters are updated to the target neural network using a soft update method, which is expressed as follows:

[0049] τθ μ +(1-τ)θ μ′ →θ μ′

[0050] τθ Q +(1-τ)θ Q′ →θ Q′

[0051] Where, θ μ θ μ′ The parameters of the evaluation neural network and the target neural network, θ, represent the parameters of the Actor neural network. Q θ Q′The parameters represent the evaluation neural network parameters and the target neural network parameters of the Critic neural network, and τ represents the update rate.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] This invention constructs a communication and sensing integrated system model assisted by unmanned aerial vehicles (UAVs), models the tasks of the UAVs in the system and analyzes the optimization objectives. By analyzing the energy consumption of the UAVs, the number of ground sensor targets detected by the UAVs and the data transmission rate of the ground sensor target data collected, the optimization objective function is obtained, and a reward function for the algorithm is designed to guide the training of the algorithm.

[0054] Compared to traditional path planning methods, the method proposed in this invention is highly flexible and portable, and has a strong ability to adapt to environmental changes. Compared to ordinary reinforcement learning methods, the method proposed in this invention can optimize for multiple different optimization objectives. Furthermore, by designing a reasonable reward function and introducing techniques such as distributed rewards and smooth updates during training, the algorithm can converge faster and achieve better performance metrics than traditional methods. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the UAV path planning method based on the deep deterministic policy gradient algorithm.

[0056] Figure 2 This is a schematic diagram of the architecture of the UAV-assisted integrated communication and sensing system model of the present invention;

[0057] Figure 3 This is a schematic diagram of the implementation process framework of the DDPG algorithm of this invention;

[0058] Figure 4 This is a schematic diagram showing the increase in the total number of data acquisition sensor devices of the UAV of this invention as a function of training rounds;

[0059] Figure 5 This is a schematic diagram showing the increase in the total number of sensor devices detected by the UAV of this invention as a function of training rounds;

[0060] Figure 6 This is a schematic diagram showing the increase in the total data transmission rate of the UAV performing the data acquisition task as a function of training rounds.

[0061] Figure 7 This is a schematic diagram showing the increase in energy consumption of the UAV of the present invention with the number of training rounds. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0063] Example

[0064] like Figure 1 As shown, this embodiment provides a UAV path planning method based on a deep deterministic policy gradient algorithm, including the following steps:

[0065] S1: Construct an integrated communication and sensing system model assisted by unmanned aerial vehicles (UAVs), including a ground sensor model, an UAV model, a channel model, and a radar detection model;

[0066] Assuming several sensors are distributed within a certain area to collect environmental information, the information collected by the sensors needs to be collected, summarized, and further processed and analyzed. Due to environmental factors such as natural conditions and convenience considerations, drones are used to detect the sensor locations and collect the information from the sensors.

[0067] like Figure 2 As shown, Python 3.6 is used for modeling to simulate a drone-assisted integrated communication and sensing system environment. This embodiment includes a drone and 100 ground sensor devices randomly distributed within a 500×500 meter area. The drone performs its mission at a fixed altitude of 15 meters. The drone's communication coverage radius is 15 meters, and its sensing coverage radius is 30 meters. Data in the ground sensor devices is updated once per second. The maximum capacity of the sensor devices' data buffer is 5000 data packets, and each data packet is 10 megabytes in size.

[0068] S11: The steps for constructing a ground sensor model are as follows:

[0069] A number of sensor devices (preferably 100 in this embodiment) are randomly established on the ground, and the position of sensor k is denoted by . The sensors can monitor and collect the latest environmental status data in real time, storing it in a sensor buffer for collection by the UAV. The sensor devices use a time-division multiple access protocol for transmission. The data transmission priority of different sensor devices is represented as follows:

[0070]

[0071] Among them, f k (t) represents the data waiting to be collected in the sensor data buffer at time t, a k a represents the data generation rate of sensor k at time t. k The sensor follows a Poisson distribution, but the Poisson distribution parameters are different for different sensor devices. maxThis represents the maximum capacity of the data buffer. It is assumed that the data buffer capacity is the same for each sensor device. When the data buffer reaches its maximum capacity, new data will directly overwrite the old data.

[0072] S12: The steps for building a drone model are as follows:

[0073] Modeling the system in a three-dimensional coordinate system, with the ground modeled as the OXY plane, and the UAV and ground sensor equipment equated to point masses, and the UAV performing its mission at a fixed altitude, the flight dynamics model of the UAV is represented as follows:

[0074]

[0075] Among them, (x u y u , z u The ) represents the drone's coordinates, and v represents the drone's flight speed. The yaw angle represents the drone's angle, and h represents the drone's fixed flight altitude. v is affected by the maximum flight speed. max =20m / s limit, yaw angle The range is [-π, π].

[0076] The power consumption model of a drone during flight is represented as follows:

[0077] P(V)=P b (V)+P i (V)+P a (V)

[0078] in, P0 is the blade profile power, and U is the blade profile power when the UAV is hovering. Let P be the induced power of the UAV. d v0 represents the induced power of the drone while it is hovering, and v0 represents the average rotor air-induced velocity of the drone while it is hovering. Let represent the parasitic power of the UAV, where b0, ρ, m, and S represent the fuselage drag coefficient, air density, rotor solidity, and rotor disk area, respectively.

[0079] S13: The steps for constructing the channel model are as follows:

[0080] The loss model for air-to-ground channels is expressed as follows:

[0081]

[0082] Where η represents the channel power gain when the reference distance d0 = 1m, μ represents the propagation distance between the drone and the sensor device k, and β represents the path loss coefficient.NLoS It is the attenuation coefficient of the NLoS link.

[0083] For sensor device k, the Loss probability at time t is expressed as:

[0084]

[0085] Here, c and d are two constants that depend on the carrier frequency and the environment, primarily the carrier frequency and the type of environment in which the UAV is located. d represents the elevation angle between the drone and the sensor equipment. k H represents the distance between the drone and the sensor device, and H represents the altitude of the drone.

[0086] The channel power gain between the drone and sensor device k is:

[0087]

[0088] S14: The steps for constructing a radar detection model are as follows:

[0089] The drone is equipped with radar to detect ground-based sensor equipment. The radar detection model is as follows:

[0090]

[0091] Among them, R max P is the maximum detectable range of the radar. t Let G be the radar transmit power, G be the antenna gain, and A be the antenna gain. e For the effective receiving area, σ is the radar cross-section, which is determined by the specific target size, and S min This is the minimum detectable signal. According to R... max The specific coordinates of sensor device k are obtained from the coordinates of the UAV and added to the UAV's known information set as a candidate target sensor for information collection.

[0092] S2: Unmanned Aerial Vehicle (UAV) Mission Modeling and Analysis;

[0093] Based on the optimization objective of the UAV mission, the multi-objective optimization function for the UAV mission can be expressed as:

[0094] max(R total D total -E total )

[0095] Among them, R total D represents the total data transmission rate of the drone collecting data from ground sensor devices. total E represents the number of ground sensor devices detected by the drone during its mission. totalThis indicates the total energy consumption of the drone during the mission.

[0096] S3: Train the UAV using the Deep Deterministic Policy Gradient Algorithm (DDPG);

[0097] Design the state space, action space, and reward function of the DDPG algorithm based on the multi-objective optimization function of the UAV mission;

[0098] The drone selects a target sensor device, moves to a position directly above it, hovers, and collects data. After data collection is complete, it selects a new target sensor device and continues the data collection task. Throughout the mission, the drone continuously detects new ground sensor devices and adds them to its known information set as candidate targets for data collection.

[0099] The state space is designed as a quintuple S = {X} u X d X f X l X t}, where X u The coordinates of the drone are represented by X. d The coordinates of the target sensor device, X f This indicates the number of times the drone went beyond the map's boundaries. When the drone exceeded the map's defined boundaries, the reward function would give a negative reward. l X represents the number of sensors detected by the drone that caused data overflow. t This indicates the number of sensor devices detected by the drone, preventing the reinforcement learning algorithm from getting stuck in localized flight operations.

[0100] The continuous motion space of a drone in three-dimensional space is defined as Where v represents the flight speed of the drone, This represents the yaw angle of the drone.

[0101] The reward function is designed as follows:

[0102] R = 10D total -PD f -D l +D t +D d

[0103] Among them, D total The sum of information transmission rates is calculated using the formula D. total =∑D i D i By Shannon's theorem Therefore, P represents the energy consumption of the drone. When the drone is in flight, P is a variable that varies with speed. When the drone is hovering and communicating with ground sensor equipment, P is a constant. f D is the penalty for a drone flying outside the map area. l D is the penalty for data overflow in ground sensor equipment. t For the reward item of the drone detecting ground sensor equipment, D d The reward item indicates the distance between the drone and the target sensor device.

[0104] The UAV was trained based on the designed state space, action space, and reward function. Training compensation was adjusted through adaptive learning rate and dynamic exploration factor adjustments, and the randomness of action selection was increased to prevent the algorithm from getting trapped in local optima, further accelerating the algorithm's convergence speed. Ultimately, the UAV trained by the algorithm was able to complete the target mission.

[0105] The DDPG algorithm used in this embodiment is a reinforcement learning algorithm based on the Actor-Critic framework, capable of solving decision-making problems in continuous action spaces in many scenarios. DDPG is an offline policy-based, model-free deep reinforcement learning algorithm that can significantly improve sample utilization. Figure 3 The diagram illustrates the framework of the DDPG algorithm, using μ and Q networks to represent the Actor network responsible for fitting the policy function and the Critic network responsible for fitting the value function, respectively. Since the updates of the two neural networks in the Actor-Critic framework are interdependent, this can lead to convergence difficulties. Therefore, this embodiment of the DDPG algorithm employs a fixed-network technique, where each of the two neural networks is supplemented with a target neural network. This allows the network parameters to be fixed during algorithm updates, and new parameters are then applied to the target network using a soft update method. The formula for the soft update is as follows:

[0106] τθ μ +(1-τ)θ μ′ →θ μ′

[0107] τθ Q +(1-τ)θ Q′ →θ Q′

[0108] Where, θ μ θ μ′ θ represents the parameters of the evaluation neural network and the target neural network of the Actor neural network. Q θ Q′This represents the parameters of the evaluation neural network and the target neural network. τ represents the update rate, which ensures that updates are performed gradually, guaranteeing that the parameters of the neural network do not change significantly.

[0109] In this embodiment, the specific process of the UAV path planning algorithm is as follows:

[0110] (1) Preset the number of iterations M, randomly initialize the evaluation network and the target network, and initialize the network parameters θ. Q θ μ ;

[0111] (2) Initialize the replay buffer and the exploration factor σ;

[0112] (3) Select the next action to be performed based on the strategy and exploration factors. t ;

[0113] (4) Execute action a t Receive reward r t and the next state s t+1 ;

[0114] (5) (s) t a t r t s t+1 Stored in the replay buffer;

[0115] (6) Select a batch of data from the replay buffer for updating;

[0116] (7) Calculate r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ );

[0117] (8) Minimize Update the Critic network;

[0118] (9) Through policy gradient Update the Actor network;

[0119] (10) Soft update target network:

[0120] τθ μ +(1-τ)θ μ′ →θ μ′

[0121] τθ Q +(1-τ)θ Q′ →θ Q′

[0122] Decrease the exploration factor and iterate repeatedly until the preset number of iterations M is met;

[0123] In the algorithm, the replay buffer is an experience replay buffer used to store information about the interactions between the agent and the environment. This reduces the number of interactions between the agent and the environment, allowing the algorithm to converge faster. i a i r i s i+1 This represents the current state, action, reward, and next state of sample i sampled from the replay buffer. N represents the number of samples sampled from the replay buffer, and γ represents the discount factor used to adjust the importance of future time-to-time rewards to the total cumulative reward.

[0124] This embodiment also provides simulation parameter settings and analysis of some simulation results, as detailed below:

[0125] This embodiment uses Python 3.6 and PyCharm to build the simulation environment, and TensorFlow to complete the algorithm part. The specific simulation parameters are shown in the table below:

[0126] Table 1 Simulation Parameters

[0127]

[0128]

[0129] like Figure 4 , Figure 5 The graphs show the increase in the number of sensors receiving data collection services and the increase in the number of sensors detected by the UAV as a result of training rounds. As can be seen from the graphs, with the increase in the number of training rounds, both the total number of sensor devices collecting data and the total number of sensor devices detected by the UAV gradually increase and eventually stabilize. After training, the UAV can detect more than 70% of the sensor devices and provide data collection services for more than 40% of them.

[0130] like Figure 6 , Figure 7The graph shows the total data transmission rate and total energy consumption of the UAV during mission execution as the number of training rounds increases. As can be seen from the graph, the total data transmission rate gradually increases with the number of training rounds and eventually stabilizes. The UAV's energy consumption initially decreases, then increases, and finally stabilizes with the number of training rounds. This demonstrates that the reward function can effectively guide the agent to learn towards lower energy consumption. To adapt to communication and perception tasks, after a period of training, the UAV's energy consumption gradually increases and then stabilizes. This further proves the effectiveness of the designed reward function.

[0131] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A UAV path planning method based on a deep deterministic policy gradient algorithm, characterized in that, Includes the following steps: Construct a UAV-assisted integrated communication and sensing system model, including a ground sensor model, a UAV model, a channel model, and a radar detection model; The construction of the ground sensor model includes: randomly setting up multiple sensor devices on the ground, the sensor devices monitoring and collecting the latest environmental status data in real time, and storing the data in the sensor buffer for collection by the UAV; The drone model includes a flight dynamics model of the drone and a power consumption model of the drone during flight. The coordinate position of the drone is obtained based on the flight dynamics model of the drone. The construction of the channel model includes: constructing an air-to-ground channel model based on NLoS links and LoS links; The construction of the radar detection model includes: constructing a radar detection model based on radar detection information, which includes radar transmit power, antenna gain, effective receiving area, radar cross-section and minimum detectable signal; obtaining the specific coordinate position of the sensor device based on the radar detection model and the coordinate position of the UAV; and adding it to the known information set of the UAV as a candidate target sensor for information collection. Based on the optimization objectives of the UAV mission, a multi-objective optimization function for the UAV mission is constructed, expressed as: max(R total ,D total ,-E total ) Among them, R total D represents the total data transmission rate of the drone collecting data from ground sensor devices. total E represents the number of ground sensor devices detected by the drone during its mission. total This indicates the total energy consumption of the drone during the mission. The UAV is trained based on the Deep Deterministic Policy Gradient Algorithm (DDPG), and the state space, action space and reward function of the DDPG algorithm are constructed according to the multi-objective optimization function of the UAV mission. The state space is constructed as a quintuple S = {X} u ,X d ,X f ,x l ,x t }, where X u The coordinates of the drone are represented by X. d The coordinates of the target sensor device, X f X represents the number of times the drone went outside the map. l X represents the number of sensors detected by the drone that caused data overflow. t This indicates the number of sensor devices detected by the drone; The action space is represented as follows: Where v represents the flight speed of the drone, The yaw angle representing the drone; The reward function is expressed as: R = 10D total -PD f -D l +D t +D d ; Among them, D total Let P be the total information transmission rate, and D be the energy consumption of the UAV. f D is the penalty for a drone flying outside the map area. l D is the penalty for data overflow in ground sensor equipment. t For the reward item of the drone detecting ground sensor equipment, D d A reward item indicating the distance between the drone and the target sensor device; The drone selects a target sensor device and moves to hover directly above it to collect data. After the data collection is completed, it selects a new target sensor device and continues to perform the data collection task. During the entire mission, the drone detects new ground sensor devices and adds them to the drone's known information set as candidate targets for data collection. The drone is trained based on the constructed state space, action space, and reward function, and the optimal drone path planning result is output.

2. The UAV path planning method based on deep deterministic policy gradient algorithm according to claim 1, characterized in that, The sensor devices use a time-division multiple access protocol for transmission, and the data transmission priority of different sensor devices is represented as follows: Among them, f k (t) represents the data waiting to be collected in the sensor data buffer at time t, a k a represents the data generation rate of sensor k at time t. k The sensor follows a Poisson distribution, but the Poisson distribution parameters are different for different sensor devices. max This represents the maximum capacity of the data cache.

3. The UAV path planning method based on deep deterministic policy gradient algorithm according to claim 1, characterized in that, The flight dynamics model of the UAV is represented as follows: Among them, (x u ,y u ,z u The ) represents the drone's coordinates, and v represents the drone's flight speed. The yaw angle represents the drone's angle of flight, and h represents the drone's fixed flight altitude.

4. The UAV path planning method based on the deep deterministic policy gradient algorithm according to claim 1, characterized in that, The power consumption model of the UAV during flight is expressed as follows: P(V)=P b (V)+P i (V)+P a (V) Among them, P b (V) represents the blade profile power, P0 represents the blade profile power of the UAV when hovering, U represents the tip speed of the UAV rotor, and P i (V) represents the induced power of the UAV, P d P is the induced power when the drone is hovering, v0 is the average rotor air-induced velocity when the drone is hovering, and P is the induced power when the drone is hovering. a (V) represents the parasitic power of the UAV, and b0, ρ, m, and S represent the fuselage drag coefficient, air density, rotor solidity, and rotor disk area, respectively.

5. The UAV path planning method based on the deep deterministic policy gradient algorithm according to claim 1, characterized in that, The loss model of the air-to-ground channel model is expressed as follows: Where η represents the channel power gain, Let μ represent the propagation distance between the drone and sensor device k, β represent the path loss coefficient, and μ represent the propagation distance between the drone and sensor device k. NLoS It is the attenuation coefficient of the NLoS link.

6. The UAV path planning method based on the deep deterministic policy gradient algorithm according to claim 5, characterized in that, For sensor device k, the Loss probability at time t is expressed as: Where c and d are two constants that depend on the carrier frequency and the environment, θ k The elevation angle d represents the distance between the drone and the sensor equipment. k H represents the distance between the drone and the sensor device, and H represents the altitude of the drone. The channel power gain between the drone and sensor device k is:

7. The UAV path planning method based on the deep deterministic policy gradient algorithm according to claim 1, characterized in that, The radar detection model is represented as follows: Among them, P t Let G be the radar transmit power, G be the antenna gain, and A be the antenna gain. e The effective receiving area is σ, where σ is the radar cross-section and S is the effective receiving area. min It is the smallest detectable signal.

8. The UAV path planning method based on the deep deterministic policy gradient algorithm according to claim 1, characterized in that, The deep deterministic policy gradient algorithm is based on the Actor-Critic framework. The Actor network, responsible for fitting the policy function, and the Critic network, responsible for fitting the value function, are represented by μ-networks and Q-networks, respectively. A fixed-network technique is employed, with a target neural network added to both the Actor and Critic networks. New parameters are updated to the target neural network using a soft-update method, which is expressed as follows: tth μ +(1-τ)θ μ′ →θ μ′ tth Q +(1-τ)θ Q' →θ Q' Where, θ μ θ μ′ The parameters of the evaluation neural network and the target neural network, θ, represent the parameters of the Actor neural network. Q θ Q′ The parameters represent the evaluation neural network parameters and the target neural network parameters of the Critic neural network, and τ represents the update rate.

Images