Energy sensing unmanned aerial vehicle data acquisition path planning method

Abstract

The invention discloses a method for planning a data acquisition path of an energy-aware unmanned aerial vehicle. The method comprises the following steps: 1) constructing a system model; 2) unmanned aerial vehicle flight constraint limitation and optimization problem generation; 3) constructing a reward function of the unmanned aerial vehicle; and 4) training the neural network by adopting a deep reinforcement learning DDQN algorithm with a priority experience playback mechanism, and finally obtaining a group of parameters capable of maximizing accumulated rewards by the neural network, thereby obtaining a data collection path for maximizing the data collection rate and the safe landing rate of the unmanned aerial vehicle in a given environment. According to the method, the unmanned aerial vehicle can find a data collection path with the maximum data collection rate and safe landing rate in a given environment.

Description

Technical Field

[0001] The present invention relates to drone technology, and in particular to a scenario in which a drone swarm collects data from distributed Internet of Things devices in an urban environment, and specifically to an energy-aware drone data collection path planning method. Background Art

[0002] With the rapid development of drone technology, drones are easy to deploy, have low maintenance costs, high maneuverability and strong hovering capabilities, making them suitable for many applications. For example, in disaster and search and rescue situations, or in the absence of fixed coverage, drones have shown particular application prospects in collecting data from distributed IoT sensor nodes: in the absence of cellular infrastructure, drones, as mobile base stations, can establish efficient line-of-sight connections with IoT devices and collect data. In drone data collection missions, the efficiency of data collection is crucial. Effective mission planning and drone swarm coordination can optimize the flight path of drones, reduce repeated paths and drone energy waste, and thus improve the efficiency of data collection.

[0003] Research on path planning technology for drones to collect data from distributed sensor networks has made significant progress in recent years. However, most existing path planning technologies focus on the path optimization of a single drone, or are based on traditional heuristic algorithms such as ant colony algorithms and genetic algorithms. Although these methods can optimize the flight path of drones to a certain extent, their limitations are: first, they cannot dynamically adapt to changes in complex environments. In urban environments, the complex location distribution of buildings affects the communication quality between drones and sensor devices. At the same time, buildings of varying heights become obstacles on the drone's movement path, limiting the drone's movement space; second, these methods lack effective management of drone energy consumption, and usually ignore the key factor of limited drone battery capacity, which may lead to the drone being unable to complete the data collection task or land safely during the mission. How to find an efficient drone trajectory under the dual influence of buildings and the drone's own energy is the key to drones completing data collection tasks in urban environments. Summary of the invention

[0004] The purpose of the present invention is to provide an energy-aware UAV data collection path planning method to address the deficiencies of existing technologies. This method enables the UAV to find a data collection path that maximizes the data collection rate and safe landing rate in a given environment.

[0005] The technical solution for achieving the purpose of the present invention is:

[0006] A method for energy-aware UAV data collection path planning includes the following steps:

[0007] 1) Constructing the system model: The system model includes the environment model, the UAV movement model, the building model and the communication channel model, specifically:

[0008] 1-1) Constructing the environment model: Assume that the urban environment is of size M×M∈N 2 A square grid, where the size of each small cell is c. In the assumed square grid N 2 There are three characteristic environments: drone take-off and landing areas, areas that drones cannot reach, and building areas. Let P be all possible locations [x, y] T The set is , I is the number of drones, and the drone take-off and landing areas are represented by the set S, which contains L specified drone take-off or landing locations, expressed as:

[0009]

[0010] The Z areas that drones cannot reach are represented by set D, which includes no-fly zones and high-rise buildings. Set D is represented as:

[0011]

[0012] The B buildings that hinder the wireless connection between drones and IoT sensor devices are represented by a set F:

[0013]

[0014] 1-2) Deploy a set of drones U in a given grid P and build a drone mobility model, including:

[0015] 1-2-1) The position p of the i-th drone i (t) = [x i (t),y i (t),z i (t)] T ∈R 3 , where z i (t)∈{0,h}, indicating that the drone is on the ground or at a constant height h, and the height h∈(0,h max ],h max represents the maximum flight altitude of each drone, and the flight altitude of drone i is expressed as Running status i (t)∈{0,1}, 0 represents inactive, 1 represents active, battery energy level b i (t)∈N;

[0016] 1-2-2) The data collection task of the drone ends after the time step T. The time task step T is divided into task time slots of equal length t∈[0,T], and the length of each time slot is set to ω tSecond;

[0017] 1-2-3) Define the action space A of each drone as shown in formula (1):

[0018]

[0019] In formula (1), the action space A has six action states, represented by a i (t) represents the action of the drone at time t: ai(t) = [0 0 0] T Indicates that the drone performs a hovering action. i (t) = [c 0 0] T Indicates that the drone moves along the positive direction of the x-axis, a i (t) = [-c 0 0] T Indicates that the drone moves along the negative direction of the x-axis, a i (t) = [0 c 0] T Indicates that the drone moves along the positive y direction, a i (t) = [0 - c 0] T Indicates that the drone moves along the negative y direction, a i (t) = [0 0 -h] T It means that the drone lands on the ground from a height of h;

[0020] 1-3) Construct building model: On the square grid N 2 In the figure, it is divided into building area F and non-building area. The height h of the non-building area is non Set to 0, the building area F is divided into buildings that the drone can fly over and buildings that cannot fly over. Set G represents the set of buildings that cannot be flown over. The height of the buildings in set G is denoted by h g , satisfying h g >h max , as shown in formula (2):

[0021] G={g∈F|h g >h max} (2),

[0022] The set of buildings that the drone can fly over is denoted as set H, and the height of the buildings in set H is denoted as h b ,h b ∈(0,h max ), H is as shown in formula (3):

[0023] H={b∈F|0<h b <h max} (3),

[0024] When the flight altitude of UAV i at time t Greater than the height h of the building that can be jumped at the target location b When , the UAV can reach the target position by executing the action, otherwise the UAV i cannot reach the target position;

[0025] 1-4) Construct a communication channel model: The communication between a single drone and a sensor uses the standard time division multiple access TDMA protocol. The IoT device operates in multi-band mode and can communicate with all drones at all orthogonal frequencies at the same time. It is assumed that there is no communication interference between drones:

[0026] In each communication time slot n∈[0,N], the drone selects the highest signal-to-noise ratio SNR with the remaining data to be uploaded i,k (n) of sensor nodes k∈[1,K], in each time slot, the drone only communicates with the selected sensor node, and drone i selects the scheduling variable q of sensor node k in time slot n. i,k (n)∈{0,1}, the evolution of the data volume of the kth device in a communication time slot is shown in formula (4):

[0027]

[0028] Where D k (n+1) represents the amount of remaining data after device k has completed a task time slot, R i,k (n) represents the data transmission speed between drone i and device k at communication slot n, ω n represents the length of a communication time slot. The achievable throughput of the i-th UAV in a mission time slot t is shown in formula (5), where t consists of λ (λ=4) communication slots, and these λ communication slots are n=λt, n=λt+1, n=λt+2, and n=λt+3 respectively:

[0029]

[0030] 2) Generation of UAV flight constraints and optimization problems: Based on the system model, the movement restriction rules of the UAV are obtained. The data collection path planning is generated as an optimization problem by integrating the system model and restriction rules, including:

[0031] 2-1) The drone mobility model is subject to the following constraints:

[0032]

[0033]

[0034] (6a) indicates that the drone cannot enter the location in set D, including no-fly zones and high-rise buildings. (6b) indicates that the drone needs to complete the data collection task and land under the condition of limited energy. (6c) indicates that the location of any drone at the initial moment is within the set S, and the flight altitude is always h. (6d) indicates that the drone is in a moving state at the initial moment. (6e) indicates that any drone is at a different flight altitude during flight. (6f) indicates that the flight altitude of any drone i at time t must be higher than the height of the building at its location. It also indicates that the flight altitude of the drone is not 0 during the movement.

[0035] 2-2) Generate optimization problem: The central goal of the multi-UAV path planning problem is to maximize the throughput of all deployed UAVs within the entire mission time T. The maximization problem is expressed as shown in formula (7):

[0036]

[0037] × i a i (t) represents the joint action of multiple drones. In each task time slot t within the entire time T, drone i performs action a i (t), to maximize the function in formula (7). Each drone has its own state, action and reward. The optimization problem is expressed as a multi-drone reinforcement problem. The deep reinforcement learning DDQN (Double Deep Q-Network, DDQN for short) algorithm is used to solve the problem, thereby dynamically optimizing and maximizing the objective function formula (7);

[0038] 3) Construct the reward function of the drone: The reward function sets rules and goals for the drone in deep reinforcement learning. By rewarding and punishing the drone's behavior, the reward function can guide the drone to continuously improve its strategy and eventually find the optimal path. The reward function is expressed as: R: The reward function applies a reward value to the drone based on the drone’s previous state S, current state S′, and the drone’s previous action A. The reward function r i (t) is defined as shown in formula (8):

[0039]

[0040] is the collective reward for all the information collected by all drones in time interval t. The parameter μ is used to parameterize the amount of information as a reward, which is the shared part of all drones.

[0041] ρ i (t) is the penalty function when the safety controller rejects the action selected by drone i at time t, and is defined as shown in formula (9):

[0042]

[0043] If the action selected by the drone is inconsistent with the action given by the safety controller, a penalty ρ is imposed on the drone; if the actions are consistent, no penalty is imposed on the drone;

[0044] To impose a penalty on drones that fail to land in a landing zone when running out of energy The definition is shown in formula (10):

[0045]

[0046] ε is the movement penalty constant. Each time the drone performs a movement action, a penalty value ε is imposed, which encourages the drone to reduce the number of movement steps and find an efficient drone data collection path;

[0047] χ i (t) is the mobile reward function. If the energy b of drone i at time t is i (t) and initial energy b i (0) between the conditions The mobile reward function will be triggered, ξ is a constant, and the mobile reward function χ is triggered i (t), the drone takes the safe landing at the landing point as the primary goal from time t, and encourages the drone to move closer to the landing point. The mobile reward function χ i (t) is defined as shown in formula (11):

[0048]

[0049] As shown in formula (11), if the distance d between UAV i and the nearest landing point o at time t+1 is i,o (t+1) is less than the distance d between drone i and the nearest landing point o at the previous time t i,o (t), a constant reward χ is imposed on the drone; if the distance relationship is the opposite, no reward is given to the drone. When selecting the nearest landing point, if there are multiple lift zones, let m be the number of lift zones. At time t+1, the distances from the landing point with the shortest distance to the drone in each lift zone to the drone are d1, d2, ..., d m , if d i =min(d1,d2,...,d m), the UAV selects the landing point in lift area i with the shortest distance to the UAV as the target; if multiple shortest distances are equal, the UAV prioritizes the lift area with the largest number of nearby sensor devices, and takes the nearest landing point in the area as the target; if the amount of sensor equipment data near multiple lift areas is equal, in order to obtain environmental information in the map, the UAV selects a lift area other than the starting lift area for landing.

[0050] 4) Use the deep reinforcement learning DDQN algorithm with a priority experience replay mechanism to train the neural network: In the DDQN algorithm, there are two neural networks: the main network and the target network. The main network is used to estimate the Q value of each action in the current state, and calculate the probability of being selected based on the Q value of each action through the Softmax strategy. The target network is used to calculate the target Q value and help update the parameters of the main network to reduce the problem of overestimation of the Q value. The cooperation of these two networks makes the training process more stable and accurate. The environment map, flight time graph, state graph, device data graph, and mobile reward function are input into the main network, and the Q value is output. The Q value is expressed as Q θ (s,a), where s represents the current state and a represents the current action. The Softmax strategy is used to calculate the probability of the action being selected based on the Q value of each action, so that the action corresponding to the higher Q value has a higher selection probability. In the Softmax strategy, action a i The probability of being selected is expressed as shown in formula (12):

[0051]

[0052] In formula (12), Indicates drone i action a i An exponential function is used to amplify the weight of high-value actions. θ is the main network parameter, Q θ (s,a i ) indicates that drone i performs action a in state s i Q value, β is the temperature parameter, which is used to adjust the balance between exploration and utilization. After the drone performs the selected action, it obtains feedback from the environment, obtains the new state s′ and the reward and punishment corresponding to the action, and stores the current state s, current action a, reward r, and next state s′ in the experience replay buffer. The priority experience replay method is used to extract an experience sample of size m from the experience replay area for training. The main network is used to calculate the Q value of the current state s in each sample, and then the target network is used to calculate the maximum Q value of the next state s′. The reward function is used to calculate the reward value r of the current state, and the target Q value of the current state is obtained by adding them together, which is expressed as shown in formula (13):

[0053]

[0054] In formula (13), r(s,a) represents the reward value obtained by drone j after executing action a in state s, and γ is the discount factor, which indicates the importance of future rewards. represents the maximum Q value of the drone for all actions a′ in the next state s′, θ′ is the target network parameter,

[0055] The mean square error of the Q value calculated by the main network and the target network is used as the loss function, and its gradient is calculated according to the loss function and the parameters are updated along the negative gradient direction, so that the model gradually tends to the optimal state. The loss function is defined as shown in formula (14):

[0056]

[0057] Where m represents the total number of samples drawn, w j is the importance sampling weight, Q j (s,a,θ) is the current Q value of sample j, w j The definition is shown in formula (15):

[0058]

[0059] In formula (15), N represents the number of samples in the experience replay area, ζ represents the sampling weight coefficient, which is used to determine the extent to which the influence of priority experience replay on the convergence result is offset. If ζ = 0, it means that importance sampling is not used at all; ζ = 1 means that the influence is completely offset. P(j) is the probability of sample j being extracted, which is defined as shown in formula (16):

[0060]

[0061] where p j represents the priority of sample j, p j > 0, the index α determines how many priorities to use, α of 0 indicates uniform sampling, and α of 1 indicates greedy sampling;

[0062] For the weight w in formula (15) j Normalization, after normalization, we get the following formula (17):

[0063]

[0064] After calculating the loss function, the gradient back propagation of the neural network is used to update all parameters of the Q network, and the temporal difference error TD-error (Temporal Difference Error, referred to as TD-error) of all samples is recalculated, expressed as δ j , as shown in formula (18):

[0065] δj =Y j (s,a,s′)-Q θ (s,a) (18),

[0066] And update the priority p of the corresponding node j =|δ j |, after updating the main network parameters in each round, the target network parameters θ′ are soft updated. After updating the neural network parameters, if the cumulative reward obtained by the current main network during the training process exceeds the cumulative reward of the best neural network, the parameters of the current neural network are saved as the new best neural network. The soft update of the target network parameters θ′ is shown in formula (19):

[0067] θ′←(1-τ)θ′+τθ (19),

[0068] τ∈(0, 1] is the update factor, which is used to control the speed of network parameter update.

[0069] This technical solution proposes an energy-aware UAV data collection path planning idea based on deep reinforcement learning in scenarios where multiple UAVs are energy-constrained and need to cooperate to collect limited data from ground sensor equipment. This idea designs a more detailed building model and assigns different heights to buildings. The UAV swarm will perform data collection tasks at different heights and adjust the actions to be performed based on the relationship between its own flight altitude and the building height. To ensure that the UAV can land safely under low energy consumption, when the UAV energy is reduced to a threshold, a reward value is applied to the action of the UAV approaching the landing area during the training of the UAV using deep reinforcement learning. To speed up the training speed and improve the effectiveness of the training strategy, the UAV is trained through deep reinforcement learning with a priority experience replay mechanism to obtain the optimal policy network, which provides the UAV with the optimal action decision. By adopting this technical solution, the UAV can find an effective path that balances data collection, safety and navigation constraints in the above-mentioned mission scenarios, ensuring the landing rate of the UAV while maximizing the data collection rate.

[0070] This technical solution constructs a system model based on the distribution of buildings in an urban environment. At the same time, a mobile reward function is added to the reward function of the drone, and a priority experience replay mechanism is introduced in the training process of deep reinforcement learning. This technical solution inputs the environmental map, flight time graph, state graph, device data graph, and mobile reward function into the neural network. The neural network trains the drones to gradually learn how to make the best decisions under different conditions, and finally generates the optimal data collection path for the drone swarm.

[0071] This technical solution has the following advantages:

[0072] This technical solution aims to solve the problem of low data collection rate of drones due to the single flight altitude, and constructs constraints on multi-height building models and drone movement models. By setting buildings of different heights to simulate the actual urban environment, drone swarms are assigned to different flight altitudes according to the building heights to perform data collection tasks, so as to shorten the average distance from the sensor and improve the data collection rate. In addition, a reward function with energy constraints is constructed. When the energy of the drone is reduced to the threshold, the drone will aim to land safely at the landing point. In order to improve learning efficiency, a priority experience replay mechanism is introduced to dynamically adjust the sample sampling probability, ensure that high-value samples are used more, and accelerate drone learning. When the drone swarm performs data collection tasks at different altitudes, it can dynamically adjust the flight action according to the relationship between the flight altitude and the building height, and give priority to landing when the energy threshold is reached, thereby balancing data collection, safety and navigation constraints, maximizing the data collection rate while ensuring the safe landing rate of the drone swarm.

[0073] This approach enables the UAV to find a data collection path that maximizes the data collection rate and safe landing rate in a given environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A schematic diagram of a deep reinforcement learning architecture in an embodiment;

[0075] Figure 2 A schematic diagram of the process of building a system model and generating optimization problems in the embodiment;

[0076] Figure 3 A schematic diagram of the process of constructing a reward function in an embodiment;

[0077] Figure 4 A schematic diagram of a neural network training process in an embodiment;

[0078] Figure 5 Schematic diagram of the change of the product of the data collection rate and the safe landing rate during the training process in the embodiment;

[0079] Figure 6 Schematic diagram of the effect of the number of people and machines on the product of data collection rate and safe landing rate in the embodiment;

[0080] Figure 7 Schematic diagram of the effect of the number of devices on the product of data collection rate and safe landing rate in the embodiment;

[0081] Figure 8 The embodiment is a schematic diagram showing the effect of the amount of equipment data on the product of data collection rate and safe landing rate;

[0082] Fig. 9 Schematic diagram of the effect of maximum flight time on the product of data collection rate and safe landing rate in the embodiment. DETAILED DESCRIPTION

[0083] The content of the present invention is further described below in conjunction with the drawings and embodiments, but the present invention is not limited thereto.

[0084] Example:

[0085] Reference Figure 1 , a method for energy-aware UAV data collection path planning, comprising the following steps:

[0086] 1) Constructing the system model: The system model includes the environment model, the UAV movement model, the building model and the communication channel model, specifically:

[0087] 1-1) Constructing the environment model: Assume that the urban environment is of size M×M∈N 2 A square grid, where the size of each small cell is c. In the assumed square grid N 2 There are three characteristic environments: drone take-off and landing areas, areas that drones cannot reach, and building areas. Let P be all possible locations [x, y] T The set is , I is the number of drones, and the drone take-off and landing areas are represented by the set S, which contains L specified drone take-off or landing locations, expressed as:

[0088]

[0089] The Z areas that drones cannot reach are represented by set D, which includes no-fly zones and high-rise buildings. Set D is represented as:

[0090]

[0091] The B buildings that hinder the wireless connection between drones and IoT sensor devices are represented by a set F:

[0092]

[0093] 1-2) Deploy a set of drones U in a given grid P and build a drone mobility model, including:

[0094] 1-2-1) The position p of the i-th drone i (t) = [x i (t),y i (t),z i (t)] T ∈R 3 , where z i (t)∈{0,h}, indicating that the drone is on the ground or at a constant height h, and the height h∈(0,h max ],h maxrepresents the maximum flight altitude of each drone, and the flight altitude of drone i is expressed as Running status i (t)∈{0,1}, 0 represents inactive, 1 represents active, battery energy level b i (t)∈N;

[0095] 1-2-2) The data collection task of the drone ends after the time step T. The time task step T is divided into task time slots of equal length t∈[0,T], and the length of each time slot is set to ω t Second;

[0096] 1-2-3) Define the action space A of each drone as shown in formula (1):

[0097]

[0098] In formula (1), the action space A has six action states, represented by a i (t) represents the action of the drone at time t: a i (t) = [0 0 0] T Indicates that the drone performs a hovering action. i (t) = [c 0 0] T Indicates that the drone moves along the positive direction of the x-axis, a i (t) = [-c 0 0] T Indicates that the drone moves along the negative direction of the x-axis, a i (t) = [0 c 0] T Indicates that the drone moves along the positive y direction, a i (t) = [0 - c 0] T Indicates that the drone moves along the negative y direction, a i (t) = [0 0 -h] T It means that the drone lands on the ground from a height of h;

[0099] 1-3) Construct building model: On the square grid N 2 In the figure, it is divided into building area F and non-building area. The height h of the non-building area is non Set to 0, the building area F is divided into buildings that the drone can fly over and buildings that cannot fly over. Set G represents the set of buildings that cannot be flown over. The height of the buildings in set G is denoted by h g , satisfying h g >h max , as shown in formula (2):

[0100] G={g∈F|h g >h max} (2),

[0101] The set of buildings that the drone can fly over is denoted as set H, and the height of the buildings in set H is denoted as h b ,h b ∈(0,h max ), H is as shown in formula (3):

[0102] H={b∈F|0<h b <h max} (3) When the flight altitude of UAV i at time t Greater than the height h of the building that can be jumped at the target location b When , the UAV can reach the target position by executing the action, otherwise the UAV i cannot reach the target position;

[0103] 1-4) Construct a communication channel model: The communication between a single drone and a sensor uses the standard time division multiple access TDMA protocol. The IoT device operates in multi-band mode and can communicate with all drones at all orthogonal frequencies at the same time. It is assumed that there is no communication interference between drones:

[0104] In each communication time slot n∈[0,N], the drone selects the highest signal-to-noise ratio SNR with the remaining data to be uploaded i,k (n) of sensor nodes k∈[1,K], in each time slot, the drone only communicates with the selected sensor node, and drone i selects the scheduling variable q of sensor node k in time slot n. i,k (n)∈{0,1}, the evolution of the data volume of the kth device in a communication time slot is shown in formula (4):

[0105]

[0106] Where D k (n+1) represents the amount of remaining data after device k has completed a task time slot, R i,k (n) represents the data transmission speed between drone i and device k at communication slot n, ω n represents the length of a communication time slot. The achievable throughput of the i-th UAV in a mission time slot t is shown in formula (5), where t consists of λ (λ=4) communication slots, and these λ communication slots are n=λt, n=λt+1, n=λt+2, and n=λt+3 respectively:

[0107]

[0108] 2) Generation of UAV flight constraints and optimization problems: Based on the system model, the movement restriction rules of the UAV are obtained. The data collection path planning is generated as an optimization problem by integrating the system model and restriction rules, including:

[0109] 2-1) The drone mobility model is subject to the following constraints:

[0110]

[0111] (6a) indicates that the drone cannot enter the location in set D, including no-fly zones and high-rise buildings. (6b) indicates that the drone needs to complete the data collection task and land under the condition of limited energy. (6c) indicates that the location of any drone at the initial moment is within the set S, and the flight altitude is always h. (6d) indicates that the drone is in a moving state at the initial moment. (6e) indicates that any drone is at a different flight altitude during flight. (6f) indicates that the flight altitude of any drone i at time t must be higher than the height of the building at its location. It also indicates that the flight altitude of the drone is not 0 during the movement.

[0112] 2-2) Generate optimization problem: The central goal of the multi-UAV path planning problem is to maximize the throughput of all deployed UAVs within the entire mission time T. The maximization problem is expressed as shown in formula (7):

[0113]

[0114] × i a i (t) represents the joint action of multiple drones. In each task time slot t within the entire time T, drone i performs action a i (t), to maximize the function in formula (7). Each drone has its own state, action and reward. The optimization problem is formulated as a multi-drone reinforcement problem. The deep reinforcement learning DDQN algorithm is used to solve the problem, thereby dynamically optimizing and maximizing the objective function formula (7);

[0115] 3) Construct the reward function of the drone: The reward function sets rules and goals for the drone in deep reinforcement learning. By rewarding and punishing the drone's behavior, the reward function can guide the drone to continuously improve its strategy and eventually find the optimal path. The reward function is expressed as: R: The reward function applies a reward value to the drone based on the drone’s previous state S, current state S′, and the drone’s previous action A. The reward function r i (t) is defined as shown in formula (8):

[0116]

[0117] is the collective reward for all the information collected by all drones in time interval t. The parameter μ is used to parameterize the amount of information as a reward, which is the shared part of all drones.

[0118] ρ i (t) is the penalty function when the safety controller rejects the action selected by drone i at time t, and is defined as shown in formula (9):

[0119]

[0120] If the action selected by the drone is inconsistent with the action given by the safety controller, a penalty ρ is imposed on the drone; if the actions are consistent, no penalty is imposed on the drone;

[0121] To impose a penalty on drones that fail to land in a landing zone when running out of energy The definition is shown in formula (10):

[0122]

[0123] ε is the movement penalty constant. Each time the drone performs a movement action, a penalty value ε is imposed, which encourages the drone to reduce the number of movement steps and find an efficient drone data collection path;

[0124] χ i (t) is the mobile reward function. If the energy b of drone i at time t is i (t) and initial energy b i (0) between the conditions The mobile reward function will be triggered, ξ is a constant, and the mobile reward function χ is triggered i (t), the drone takes the safe landing at the landing point as the primary goal from time t, and encourages the drone to move closer to the landing point. The mobile reward function χ i (t) is defined as shown in formula (11):

[0125]

[0126] As shown in formula (11), if the distance d between UAV i and the nearest landing point o at time t+1 is i,o (t+1) is less than the distance d between drone i and the nearest landing point o at the previous time t i,o (t), a constant reward χ is imposed on the drone; if the distance relationship is the opposite, no reward is given to the drone. When selecting the nearest landing point, if there are multiple lift zones, let m be the number of lift zones. At time t+1, the distances from the landing point with the shortest distance to the drone in each lift zone to the drone are d1, d2, ..., dm , if d i =min(d1,d2,...,d m ), the UAV selects the landing point in lift area i with the shortest distance to the UAV as the target; if multiple shortest distances are equal, the UAV prioritizes the lift area with the largest number of nearby sensor devices, and takes the nearest landing point in the area as the target; if the amount of sensor equipment data near multiple lift areas is equal, in order to obtain environmental information in the map, the UAV selects a lift area other than the starting lift area for landing.

[0127] 4) Use the deep reinforcement learning DDQN algorithm with a priority experience replay mechanism to train the neural network: In the DDQN algorithm, there are two neural networks: the main network and the target network. The main network is used to estimate the Q value of each action in the current state, and calculates the probability of being selected based on the Q value of each action through the Softmax strategy. The target network is used to calculate the target Q value and help update the parameters of the main network to reduce the problem of overestimation of the Q value. The cooperation of these two networks makes the training process more stable and accurate.

[0128] The environment map, flight time graph, state graph, device data graph, and mobile reward function are input into the main network, and the Q value is output. The Q value is expressed as Q(s,a,θ), where s represents the current state and a represents the action. The Softmax strategy is used to calculate the probability of each action being selected based on the Q value of the action, so that the action corresponding to the higher Q value has a higher selection probability. In the Softmax strategy, action a i The probability of being selected is expressed as shown in formula (12):

[0129]

[0130] In formula (12) Indicates drone i action a i An exponential function is used to amplify the weight of high-value actions. θ is the main network parameter, Q i (s,a i ,θ) indicates that drone i performs action a in state s iQ value, β is the temperature parameter, which is used to adjust the balance between exploration and utilization. After the drone performs the selected action, it obtains feedback from the environment, obtains the new state s′ and the reward and punishment corresponding to the action, and stores the current state s, current action a, reward r, and next state s′ in the experience replay buffer. The priority experience replay method is used to extract an experience sample of size m from the experience replay area for training. The main network is used to calculate the Q value of the current state s in each sample, and then the target network is used to calculate the maximum Q value of the next state s′. The reward function is used to calculate the reward value r of the current state, and the target Q value of the current state is obtained by adding them together, which is expressed as shown in formula (13):

[0131]

[0132] In formula (13), r(s,a) represents the reward value obtained by drone j after executing action a in state s, and γ is the discount factor, which indicates the importance of future rewards. It represents the maximum Q value of drone j for all actions a′ in the next state s′, and θ′ is the target network parameter.

[0133] The mean square error of the Q value calculated by the main network and the target network is used as the loss function, and its gradient is calculated according to the loss function and the parameters are updated along the negative gradient direction, so that the model gradually tends to the optimal state. The loss function is defined as shown in formula (14):

[0134]

[0135] Where m represents the total number of samples drawn, w j is the importance sampling weight, Q j (s,a,θ) is the current Q value of sample j, w j The definition is shown in formula (15):

[0136]

[0137] In formula (15), N represents the number of samples in the experience replay area, ζ represents the sampling weight coefficient, which is used to determine the extent to which the influence of priority experience replay on the convergence result is offset. If ζ = 0, it means that importance sampling is not used at all; ζ = 1 means that the influence is completely offset. P(j) is the probability of sample j being extracted, which is defined as shown in formula (16):

[0138]

[0139] where p j represents the priority of sample j, p j > 0, the index α determines how many priorities to use, α of 0 indicates uniform sampling, and α of 1 indicates greedy sampling;

[0140] For the weight w in formula (15) j Normalization, after normalization, we get the following formula (17):

[0141]

[0142] After calculating the loss function, the gradient back propagation of the neural network is used to update all parameters of the Q network, and the temporal difference error TD-error (Temporal Difference Error, referred to as TD-error) of all samples is recalculated, expressed as δ j , as shown in formula (18):

[0143] δ j =Y j (s,a,s′)-Q j (s,a,θ) (18),

[0144] And update the priority p of the corresponding node j =|δ j |, after updating the main network parameters in each round, the target network parameters θ′ are soft updated. After updating the neural network parameters, if the cumulative reward obtained by the current main network during the training process exceeds the cumulative reward of the best neural network, the parameters of the current neural network are saved as the new best neural network. The soft update of the target network parameters θ′ is shown in formula (19):

[0145] θ′←(1-τ)θ′+τθ (19),

[0146] τ∈(0, 1] is the update factor, which is used to control the speed of network parameter update.

[0147] Specifically, in this example:

[0148] The simulated urban environment map required for the experiment is set in advance. The drone flight budget and sensor data graphs are randomly generated by PyCharm Professional Edition software. This example uses environmental information, drone information, and sensor equipment information to simulate real-life scenarios for experiments.

[0149] This example is implemented in a single NVIDIA GeForce RTX 4060Ti, PyCharm Professional Edition development environment. In the simulation experiment, the simulation site is set to the XY Cartesian coordinate system, the square grid size M is set to 32, the starting position of the drone is randomly selected in the lift zone, the initial energy range of the drone is set to [50,60], the number of drones I is set to 3, the flight height of the drone h∈(0,10], and the height of the building H that the drone can fly over hb ∈(0,9], the height h of the building G that the drone cannot fly over g ∈(10,20], the number of IoT devices K is specified as a random number between [3,10], the location of IoT devices is randomly generated, and the range of device data volume is specified as [5.0,20.0]. When the drone is flying, it flies at a constant speed towards the target position. Each movement will reduce energy. In this example, the drone information and sensor device information are the preparation stage before the simulation experiment. The random generation method increases the diversity of training data, specifically:

[0150] 1) Initialize the system model. The process is as follows Figure 2 As shown, including:

[0151] Step 1 Input file manhattan.json to set parameters;

[0152] Step 2 initializes the grid parameters, inputs the image path "map_path": "res / manhattan32.png", obtains the corresponding map image manhattan.png according to the simulated urban environment map image path in the file, converts the map feature environment information in the plane image manhattan.png into a two-dimensional array of corresponding size according to the grid size, and obtains the drone landing zone map landing_zone, the no-fly zone map no_fly_zone, and the obstacle map obstacles respectively, obtains the location of each feature environment from the feature environment map, and obtains the drone landing zone location set S, the no-fly zone and high-rise building location set D, and the building complex location set F respectively; the number of devices device_count∈[3,10], the amount of device data data_range∈[5,20], generates the number of devices device_count, the amount of data data_range and the device location according to the set range, and generates a two-dimensional array device_map of the same size as the grid to represent the location of the device in the grid and the amount of data;

[0153] Step 3 Initialize the height of the building area and the height of the non-building area, the height of the non-building area h non =0, the drone can fly over buildings h b ∈[3,9] and the height h of the unflyable building g =11;

[0154] Step 4 Initialize the reward function r i (0) = 0, initialization reward parameter, data collection parameter μ = 1.0, penalty constant ρ = 1.0 for safety controller rejecting drone action, penalty constant for drone running out of power and not landing safely Drone movement penalty constant ε = 0.1, energy threshold ξ = 0.3, drone movement reward constant χ = ​​0.1;

[0155] Step 5 Initialize the drone parameters and the initial energy b of the drone i (0)∈[50,60], the number of drones I=3, the flight altitude of each drone in the drone group is set to 4, 7, and 10 respectively, and an element is randomly selected from the drone landing zone position set S as the drone's initial position p i (0);

[0156] Step 6 Initialize the channel parameters. The number of communication slots in a task time slot is λ = 4. Set the data transmission rate R between UAV i and sensor device k. i,k (n) = 0.05;

[0157] Step7: Initialize the training parameters, temperature parameter β = 0.1, discount factor γ = 0.95, update factor τ = 0.005, maximum training step number num_steps = 1000000, experience replay area size N = 50000, small batch sample size m = 128, s represents the state of the drone, which is composed of the position and energy of the drone, and a represents the action of the drone; initialize the experience replay area structure memory used to store the experience group, the data volume of memory data = 0, and the experience group is composed of the structure (s, a, r, s′), which respectively represent the current state of the drone, the current action of the drone, the reward function value and the next state of the drone.

[0158] 2) Train the drone using DDQN with priority experience replay. The specific process is as follows: Figure 3 As shown, including:

[0159] Step 8: Construct the main network Q(s,a,θ) and the target network Q(s,a,θ′), initialize the main network neural network weight θ=1, the target network neural network weight θ′=1, the number of convolutional layers in the neural network structure parameters is set to 2, the size of the convolutional kernel is set to 5, the number of convolutional kernels in each convolutional layer is set to 16, the size of the hidden layer is set to 256, and the number of hidden layers is set to 3;

[0160] Step 9 determines whether the amount of data in the current experience playback area memory satisfies data ≥ 50000; if not, jump to Step 9 after generating the experience group to continue to determine the amount of data; if yes, enter Step 10;

[0161] Step 10 Determine whether the current training step count steps_count satisfies steps_count <1000000>. If so, initialize the drone state s and the reward function r(t), and then jump to step 11. If not, the training process ends;

[0162] Step 11 Determine whether the state s of drone i is the final state. If so, jump to step 10; if not, drone i executes the action a i (t) generated by the softmax policy at t, and obtain the next state s' of the drone;

[0163] Step 12 Use steps 20 - 23 to calculate the reward function value r i (t) obtained after drone i executes the action a i (t). If the drone lands in the landing area or the energy b i (t) ≤ 0, set the drone state s to the final state; otherwise, the drone state s remains the running state. Generate the drone running state map agent_map. If the drone is in the final state, mark the corresponding position in the map as False according to the position of the drone; if otherwise, mark it as True;

[0164] Step 13 Generate an experience group (s, a, r, s') from the current state, action, reward value, and next state of the drone. The priority δ i of the experience group is set to 1, and add the experience group to the experience replay area memory. Let steps_count = steps_count + 1 and t = t + 1;

[0165] Step 14 Sample a small batch of samples m from the experience pool according to the sample priority. Input the no - fly zone map no_fly_zone, obstacle map obstacles, drone landing zone map landing_zone, device remaining data volume map device_map, drone running state map agent_map, the initial energy bj(0) of the drone, and the energy b j (t) of the current state of the drone in the sample experience group into the main network to obtain the main network output Q j (s, a, θ);

[0166] Step 15 Then input the above - mentioned information in the next state of the drone into the main network to obtain the action a' that can maximize the main network Q value;

[0167] Step 16 Input the above - mentioned information in the next state of the drone into the target network to obtain the Q value of the next state of the target network. Select the Q value corresponding to the a' action in the Q values of the next state of the target network as

[0168] Step 17 Combine the reward function value in the experience group with the discount factor γ, according to the formula Calculate the target Q value in the current state;

[0169] Step 18 Use Calculate the loss function value of the sample;

[0170] Step 19 calculates the gradient of the loss function relative to the main network weight, and uses the gradient descent method to update the main network weight θ, and updates the target network parameters according to θ′←(1-τ)θ′+τθ. j =Y j (s,a,s′)-Q θ (s,a) Update sample priority δ j , then jump to Step 11.

[0171] 3) Construct a reward function, the process is as follows Figure 4 As shown, including:

[0172] Step 20: Determine if drone i performs action a i (t) Whether the drone is in an unreachable location or the drone’s flight altitude is lower than the building’s height; if so, the safety controller rejects the action a selected by the Softmax strategy. i (t), the drone performs a hovering action, i.e., a i (t) = [0 0 0] T , and impose a penalty on the drone, let r i (t) = r i (t)-ρ; if not, drone i executes action a selected by the Softmax strategy i (t);

[0173] Step 21 Drone i performs action a i (t), let b i (t) = b i (t)-1, the amount of data collected by the drone is parameterized as a reward, that is,

[0174] Step 22 determines whether the drone is moving. If so, impose a movement penalty r on the drone. i (t) = r i (t)-ε, then go to Step 24; otherwise go to Step 23;

[0175] Step 23 determines whether the drone action is landing. If the action is landing, it continues to determine whether the drone is in the landing area. If it is in the landing area, the landing is completed and then enters Step 24. If it is not in the landing area, the landing is not performed and then enters Step 24. If the action is not landing, it directly enters Step 24.

[0176] Step 24 determines whether the energy of drone i at time t is sufficient If satisfied, go to Step 25; otherwise go directly to Step 26;

[0177] Step 25 determines whether the distance between drone i and the nearest landing point at time t is closer than at time t-1. If so, a reward χ is imposed on the drone, and r i (t) = r i (t)+χ; if not, go directly to Step 26;

[0178] Step 26 Determine whether the drone's energy at time t meets b i (t)≤0 and the drone does not land in the safety zone. If it is satisfied, the drone will be penalized for consuming all its energy without landing in the landing zone. make Then end the process; if not satisfied, end the process directly;

[0179] In order to evaluate the effect of the drone path planning of this method, the drone data collection rate and safe landing rate are used as evaluation indicators of the model. During the experiment, the drone data collection rate and safe landing rate of this round of drone training will be recorded in the log file after each round of training. At the same time, the DDQN algorithm will store the drone's previous time step state, current state, reward value, and execution action into a four-tuple in the experience pool at each time step in each round of training. Subsequently, priority experience replay is used to extract small batches of samples from the experience pool for neural network parameter training. If the current parameter value used to evaluate the performance of the neural network is higher than the parameter value of the previous time, the parameters of the current neural network are saved in the file.

[0180] Simulation verification:

[0181] At present, the main performance indicators for measuring the path planning method of UAV data collection are the product of data collection rate and safe landing rate. The data collection rate represents the ratio of the amount of data actually successfully collected by the UAV to the total amount of data that can be collected. The safe landing rate refers to the proportion of successful and safe landings of the UAV after completing the training. The product of data collection rate and safe landing rate is used to evaluate the performance of the method in different scenario examples.

[0182] According to this example method, we get: Figure 5The simulation results of the change of the product of data collection rate and safe landing rate during the training process of this method, control method 1, and control method 2 are shown in Figure 1. The method of this example builds a building model on the basis of control method 1, adds a mobile reward function to the drone reward function, and introduces a priority experience replay method to optimize the deep reinforcement learning algorithm for drone path planning. On the basis of control method 1, control method 2 builds a system model for multi-drone path planning in adversarial environments, and designs a DDQN-based model for multi-drone path planning in adversarial scenarios.

[0183] In the experiment, the number of trap areas in control method 2 was set to 0 to eliminate the influence of trap areas on the data collection rate and safe landing rate of the drone. When the drone energy is limited, control methods 1 and 2 do not have sufficient energy to support the drone to perform long-term data collection tasks. When the drone is at a low energy level, due to the lack of guidance from the reward function, the drone is more likely to crash due to insufficient power. Setting the drones at the same height to perform data collection tasks during training will cause the drone group to collide with each other, and obstacles will also interfere with the communication signal to a greater extent. Therefore, when the drone energy is limited, this method can achieve a higher data collection rate and safe landing rate product than control methods 1 and 2 during training. Figure 6-Figure 9 The figure shows how the drones performed within the training range of the scenario parameters and how they performed outside the range, showing the impact of a single scenario parameter on the average data collection ratio of successful landings for all drones:

[0184] Figure 6 It shows that when more drones are deployed within the range of [1,4] using control method 1 and control method 2, data collection performance will be improved. When the number of drones is further increased within the range of [4,6], more drones will increase the probability of collision between drones, thereby reducing the safe landing rate and data collection rate of drones. However, when the method in this example increases the number of drones within the range of [4,6], it can improve the data collection rate of successful landing of drones.

[0185] Figure 7 It is shown that, since IoT devices are randomly distributed in the free map space, the increase in the number of devices leads to more complex trajectory requirements, which leads to performance degradation;

[0186] Figure 8 The effect of increasing the amount of initial data per device on the overall collection performance is shown. The amount of initial data per device is favorable until it reaches [10,12.5]. Beyond this range, due to flight time limitations, the drone has to abandon some data, resulting in a slight downward trend in the data collection rate.

[0187] Fig. 9 An increase in available flight time obviously benefits collection performance, however this effect diminishes once most of the data has been collected, at which point the drone begins to prioritize minimizing total flight time and landing safely over continuing to collect the remaining data.

Claims

1. A method for energy-aware UAV data collection path planning, characterized in that: The steps include: 1) Constructing the system model: The system model includes the environment model, the UAV movement model, the building model and the communication channel model, specifically: 1-1) Constructing the environment model: Assume that the urban environment is of size M×M∈N 2 A square grid, where the size of each small cell is c. In the assumed square grid N 2 There are three characteristic environments: drone take-off and landing areas, areas that drones cannot reach, and building areas. Let P be all possible locations [x, y] T The set is , I is the number of drones, and the drone take-off and landing areas are represented by the set S, which contains L specified drone take-off or landing locations, expressed as: The Z areas that drones cannot reach are represented by set D, which includes no-fly zones and high-rise buildings. Set D is represented as: The B buildings that hinder the wireless connection between drones and IoT sensor devices are represented by a set F: 1-2) Deploy a set of drones U in a given grid P and build a drone mobility model, including: 1-2-1) The position p of the i-th drone i (t) = [x i (t),y i (t),z i (t)] T ∈R 3 , where z i (t)∈{0,h}, indicating that the drone is on the ground or at a constant height h, and the height h∈(0,h max ],h max represents the maximum flight altitude of each drone, and the flight altitude of drone i is expressed as Running status i (t)∈{0,1}, 0 represents inactive, 1 represents active, battery energy level b i (t)∈N; 1-2-2) The data collection task of the drone ends after the time step T. The time task step T is divided into task time slots t∈[0,T] of equal length. The length of each time slot is set to ω t Second; 1-2-3) Define the action space A of each drone as shown in formula (1): In formula (1), the action space A has six action states, represented by a i (t) represents the action of the drone at time t: a i (t) = [0 0 0] T Indicates that the drone performs a hovering action. i (t) = [c 0 0] T Indicates that the drone moves along the positive direction of the x-axis, a i (t) = [-c 0 0] T Indicates that the drone moves along the negative direction of the x-axis, a i (t) = [0 c 0] T Indicates that the drone moves along the positive y direction, a i (t) = [0 - c 0] T Indicates that the drone moves along the negative y direction, a i (t) = [0 0 -h] T It means that the drone lands on the ground from a height of h; 1-3) Construct building model: On the square grid N 2 In the figure, it is divided into building area F and non-building area. The height h of the non-building area is non Set to 0, the building area F is divided into buildings that the drone can fly over and buildings that cannot fly over. Set G represents the set of buildings that cannot be flown over. The height of the buildings in set G is denoted by h g , satisfying h g >h max , as shown in formula (2): G={g∈F|h g >h max } (2), The set of buildings that the drone can fly over is denoted as set H, and the height of the buildings in set H is denoted as h b ,h b ∈(0,h max ), H is as shown in formula (3): H={b∈F|0<h b <h max } (3) When the flight altitude of UAV i at time t Greater than the height h of the building that can be jumped at the target location b When , the UAV can reach the target position by executing the action, otherwise the UAV i cannot reach the target position; 1-4) Construct a communication channel model: The communication between a single drone and a sensor uses the standard time division multiple access TDMA protocol. The IoT device operates in multi-band mode and can communicate with all drones at all orthogonal frequencies at the same time. It is assumed that there is no communication interference between drones: In each communication time slot n∈[0,N], the drone selects the highest signal-to-noise ratio SNR with the remaining data to be uploaded i,k (n) of sensor nodes k∈[1,K], in each time slot, the drone only communicates with the selected sensor node, and drone i selects the scheduling variable q of sensor node k in time slot n. i,k (n)∈{0,1}, the evolution of the data volume of the kth device in a communication time slot is shown in formula (4): Where D k (n+1) represents the amount of remaining data after device k has completed a task time slot, R i,k (n) represents the data transmission speed between drone i and device k at communication slot n, ω n represents the length of a communication time slot. The achievable throughput of the i-th UAV in a mission time slot t is shown in formula (5), where t consists of λ communication slots, λ = 4, and these λ communication slots are n = λt, n = λt+1, n = λt+2, and n = λt+3: 2) Generation of UAV flight constraints and optimization problems: Based on the system model, the movement restriction rules of the UAV are obtained. The data collection path planning is generated as an optimization problem by integrating the system model and restriction rules, including: 2-1) The drone mobility model is subject to the following constraints: (6a) indicates that the drone cannot enter the location in set D, including no-fly zones and high-rise buildings. (6b) indicates that the drone needs to complete the data collection task and land under the condition of limited energy. (6c) indicates that the location of any drone at the initial moment is within the set S, and the flight altitude is always h. (6d) indicates that the drone is in a moving state at the initial moment. (6e) indicates that any drone is at a different flight altitude during flight. (6f) indicates that the flight altitude of any drone i at time t must be higher than the height of the building at its location. It also indicates that the flight altitude of the drone is not 0 during the movement. 2-2) Generate optimization problem: The central goal of the multi-UAV path planning problem is to maximize the throughput of all deployed UAVs within the entire mission time T. The maximization problem is expressed as shown in formula (7): × i a i (t) represents the joint action of multiple drones. In each task time slot t within the entire time T, drone i performs action a i (t), to maximize the function in formula (7). Each drone has its own state, action and reward. The optimization problem is expressed as a multi-drone reinforcement problem. The deep reinforcement learning DDQN algorithm is used to solve the problem, thereby dynamically optimizing and maximizing the objective function formula (7); 3) Construct the reward function of the drone: The reward function sets rules and goals for the drone in deep reinforcement learning. By rewarding and punishing the drone's behavior, the reward function can guide the drone to continuously improve its strategy and eventually find the optimal path. The reward function is expressed as: The reward function applies a reward value to the drone based on the drone’s previous state S, current state S′, and the drone’s previous action A. The reward function r i (t) is defined as shown in formula (8): is the collective reward for all the information collected by all drones in time interval t. The parameter μ is used to parameterize the amount of information as a reward, which is the shared part of all drones. ρ i (t) is the penalty function when the safety controller rejects the action selected by drone i at time t, and is defined as shown in formula (9): If the action selected by the drone is inconsistent with the action given by the safety controller, a penalty ρ is imposed on the drone; if the actions are consistent, no penalty is imposed on the drone; To impose a penalty on drones that fail to land in a landing zone when running out of energy The definition is shown in formula (10): ε is the movement penalty constant. Each time the drone performs a movement action, a penalty value ε is imposed, which encourages the drone to reduce the number of movement steps and find an efficient drone data collection path; χ i (t) is the mobile reward function. If the energy b of drone i at time t is i (t) and initial energy b i (0) between the conditions The mobile reward function will be triggered, ξ is a constant, and the mobile reward function χ is triggered i (t), the drone takes the safe landing at the landing point as the primary goal from time t, and encourages the drone to move closer to the landing point. The mobile reward function χ i (t) is defined as shown in formula (11): As shown in formula (11), if the distance d between UAV i and the nearest landing point o at time t+1 is i,o (t+1) is less than the distance d between drone i and the nearest landing point o at the previous time t i,o (t), a constant reward χ is imposed on the drone; if the distance relationship is the opposite, no reward is given to the drone. When selecting the nearest landing point, if there are multiple lift zones, let m be the number of lift zones. At time t+1, the distances from the landing point with the shortest distance to the drone in each lift zone to the drone are d1, d2, ..., d m , if d i =min(d1,d2,...,d m ), the UAV selects the landing point in lift area i with the shortest distance to the UAV as the target; if multiple shortest distances are equal, the UAV prioritizes the lift area with the largest number of nearby sensor devices, and takes the nearest landing point in the area as the target; if the amount of sensor equipment data near multiple lift areas is equal, in order to obtain environmental information in the map, the UAV selects a lift area other than the starting lift area for landing. 4) Use the deep reinforcement learning DDQN algorithm with a priority experience replay mechanism to train the neural network: In the DDQN algorithm, there are two neural networks: the main network and the target network. The main network is used to estimate the Q value of each action in the current state, and calculate the probability of being selected according to the Q value of each action through the Softmax strategy. The target network is used to calculate the target Q value, input the environment map, flight time graph, state graph, device data graph, and mobile reward function into the main network, and output the Q value, which is expressed as Q θ (s,a), where s represents the current state and a represents the current action. The Softmax strategy is used to calculate the probability of the action being selected based on the Q value of each action, so that the action corresponding to the higher Q value has a higher selection probability. In the Softmax strategy, action a i The probability of being selected is expressed as shown in formula (12): In formula (12), Indicates drone i action a i An exponential function is used to amplify the weight of high-value actions. θ is the main network parameter, Q θ (s,a i ) indicates that drone i performs action a in state s i Q value, β is the temperature parameter, which is used to adjust the balance between exploration and utilization. After the drone performs the selected action, it obtains feedback from the environment, obtains the new state s′ and the reward and punishment corresponding to the action, and stores the current state s, current action a, reward r, and next state s′ in the experience replay buffer. The priority experience replay method is used to extract an experience sample of size m from the experience replay area for training. The main network is used to calculate the Q value of the current state s in each sample, and then the target network is used to calculate the maximum Q value of the next state s′. The reward function is used to calculate the reward value r of the current state, and the target Q value of the current state is obtained by adding them together, which is expressed as shown in formula (13): In formula (13), r(s,a) represents the reward value obtained by drone j after executing action a in state s, and γ is the discount factor, which indicates the importance of future rewards. represents the maximum Q value of the drone for all actions a′ in the next state s′, θ′ is the target network parameter, The mean square error of the Q value calculated by the main network and the target network is used as the loss function, and its gradient is calculated according to the loss function and the parameters are updated along the negative gradient direction, so that the model gradually tends to the optimal state. The loss function is defined as shown in formula (14): Where m represents the total number of samples drawn, w j is the importance sampling weight, Q j (s,a,θ) is the current Q value of sample j, w j The definition is shown in formula (15): In formula (15), N represents the number of samples in the experience replay area, ζ represents the sampling weight coefficient, which is used to determine the extent to which the influence of priority experience replay on the convergence result is offset. If ζ = 0, it means that importance sampling is not used at all; ζ = 1 means that the influence is completely offset. P(j) is the probability of sample j being extracted, which is defined as shown in formula (16): where p j represents the priority of sample j, p j > 0, the index α determines how many priorities to use, α of 0 indicates uniform sampling, and α of 1 indicates greedy sampling; For the weight w in formula (15) j Normalization, after normalization, we get the following formula (17): After calculating the loss function, the gradient back propagation of the neural network is used to update all parameters of the Q network, and the temporal difference error TD-error (Temporal Difference Error, referred to as TD-error) of all samples is recalculated, expressed as δ j , as shown in formula (18): δ j =Y j (s,a,s′)-Q θ (s,a) (18), And update the priority p of the corresponding node j =|δ j |, after updating the main network parameters in each round, the target network parameters θ′ are soft updated. After updating the neural network parameters, if the cumulative reward obtained by the current main network during the training process exceeds the cumulative reward of the best neural network, the parameters of the current neural network are saved as the new best neural network. The soft update of the target network parameters θ′ is shown in formula (19): θ′←(1-τ)θ′+τθ (19), τ∈(0, 1] is the update factor, which is used to control the speed of network parameter update.

Images