Multi-unmanned aerial vehicle cooperative data acquisition intelligent flight path planning method driven by curiosity rewards

Abstract

The invention discloses a curiosity award-driven multi-unmanned aerial vehicle cooperative data acquisition intelligent flight path planning method. The invention belongs to the field of unmanned aerial vehicle control, and mainly solves the flight path planning problem that a multi-unmanned aerial vehicle system cooperatively completes Internet of Things sensor data acquisition in a complex and dynamic scene. The method is based on deep reinforcement learning, and data volume collection and task time are taken as a joint optimization target. The method comprises the following steps: firstly, modeling a multi-unmanned aerial vehicle data acquisition route planning problem in a complex and dynamic environment into a partially observable Markov decision process; then establishing a flight path planning strategy based on a deep cycle Q network, fusing historical environment information by using a long-short-term memory network, making an accurate decision by using the fused information, setting a dual-source composite reward function, combining an internal curiosity reward with an external environment guidance reward, encouraging the unmanned aerial vehicle to comprehensively explore a complex environment, and improving the flight path planning efficiency. And the training effect of the track planning strategy in a complex environment is improved. According to the method, it is ensured that the multi-unmanned-aerial-vehicle system achieves data collection of the sensor nodes, meanwhile, the higher flight path planning capacity is obtained, and the effectiveness and robustness of the method are proved through simulation experiments.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) control, and specifically relates to the design of a multi-UAV cooperative trajectory planning method based on deep reinforcement learning. Background Technology

[0002] Due to their high mobility and low cost, drones are ideal for data acquisition in wireless sensor networks. Using drones for sensor data collection overcomes the difficulties in data transmission caused by a lack of communication infrastructure or terrain barriers, making them suitable for natural disaster monitoring, forest environment monitoring, and other applications. Furthermore, drones collect data over sensors, resulting in less channel loss compared to ground-to-ground communication, thus improving communication quality and reducing energy consumption. In drone-assisted data acquisition applications, drone path planning is a crucial step. The drone's flight path affects data acquisition efficiency and energy consumption. In multi-drone data acquisition, drones can interact with each other, leading to collisions and conflicting acquisition tasks. Path planning for multi-drone assisted data acquisition includes planning the sensor node acquisition order, flight path planning, and collaboration between drones. Multi-agent deep reinforcement learning methods can solve such path planning problems with low complexity; common methods include Multi-Agent Deep Q-Network (MADQN) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG). However, when designing a method for UAV-assisted data acquisition trajectory planning using deep reinforcement learning, the complexity and dynamism of the mission environment make it difficult for traditional reinforcement learning algorithms to fully learn the characteristics of the environment. This causes the trajectory planning strategy training to get stuck in local optima, preventing the agent from finding the globally optimal flight path.

[0003] Therefore, this invention, based on deep reinforcement learning, proposes a curiosity-driven cooperative trajectory planning (CCTP) method for multi-UAV assisted data acquisition trajectory planning in complex and dynamically changing environments. The proposed method uses a distributed architecture to make trajectory decisions for each UAV based on real-time environmental conditions. Its strategy integrates historical environmental information, performs efficient trajectory planning based on local environmental information, and combines internal curiosity and external environmental guidance rewards to improve the training effect of the strategy, providing UAVs with stronger trajectory planning capabilities. Summary of the Invention

[0004] The purpose of this invention is to provide a trajectory planning method with high data acquisition performance and strong generalization ability for multiple UAVs performing IoT sensor data acquisition tasks in complex and dynamically changing environments. This method optimizes the flight trajectory of the UAV throughout the entire data acquisition process at sensor nodes. It uses a Deep Recurrent Q-Network (DRQN) trajectory planning strategy that integrates historical environmental information for path planning, and simultaneously employs a dual-source composite reward function that combines internal curiosity rewards and external environment guidance rewards to improve the training effect of the strategy. To achieve this objective, the steps employed in this invention are as follows:

[0005] Step 1: Considering the characteristics of multi-UAV collaborative data acquisition tasks, and based on the relevant theoretical knowledge of reinforcement learning, the multi-UAV collaborative trajectory planning problem is modeled as a partially observable Markov Decision Process (POMDP).

[0006] Step 2: Establish a trajectory planning strategy that integrates historical environmental information, using deep Q-networks and long short-term memory networks (LSTM). - Short Term Memory (LSTM) is used for trajectory planning decisions. LSTM can memorize input information, enabling trajectory planning decisions to be based on fused historical environmental information, thus solving the challenge of trajectory planning under locally observable environmental conditions.

[0007] Step 3: Train the strategy using a dual-source composite reward function, including internal curiosity reward and external environment guidance reward. Internal curiosity reward, based on the unfamiliarity of the environment, encourages the UAV to fully explore complex environments. External environment guidance reward is set according to the optimization objectives of trajectory planning and task environment constraints, guiding the UAV to complete data acquisition tasks in dynamic and complex environments. The combination of internal curiosity reward and external environment guidance reward enhances the training effect of the trajectory planning strategy in dynamic and complex environments.

[0008] The effectiveness of the proposed curiosity-reward-driven deep reinforcement learning-based multi-UAV data acquisition trajectory planning method in dynamic and complex environments has been verified through simulation experiments. The simulation environment area was 1000×1000m. 2 The system includes 3 drones, 5 static obstacles and 5 moving obstacles, and 15 IoT sensor nodes. Figure 3 and attached Figure 4 The average success rate and average task time of the proposed method and the baseline method under different numbers of IoT sensor nodes are shown in the attached figures. Figure 5 and attached Figure 6The average task time results of the proposed method and the baseline method under the condition of changing number of moving obstacles are shown in the figure. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the method architecture of the present invention;

[0010] Figure 2 This is a schematic diagram of a dual-source composite reward structure;

[0011] Figure 3 This is a graph showing the test results of the average success rate of the method of the present invention under different numbers of sensor nodes;

[0012] Figure 4 This is a test result graph showing the average task time of the method of the present invention under different numbers of sensor nodes;

[0013] Figure 5 This is a graph showing the test results of the average success rate of the method of the present invention under varying numbers of moving obstacles;

[0014] Figure 6 This is a graph showing the test results of the average task time of the method of the present invention under the condition of changing the number of moving obstacles; Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and examples.

[0016] Step 1: Model the multi-UAV data acquisition trajectory planning problem as a partially observable Markov decision process.

[0017] To enable UAVs to flexibly plan their trajectories in dynamically changing environments and to make accurate trajectory plans using local environmental information, this invention models the multi-UAV data acquisition trajectory planning problem as a partially observable Markov decision process. This allows the UAVs to plan their trajectories step-by-step based on real-time environmental information during the data acquisition task. In this partially observable Markov decision process model, each UAV is an agent, and the set of all N UAVs is represented as {U1, U2, ..., U...}. N The design of the state space and action space is as follows:

[0018] 1. Observation space

[0019] The observation space is the collection of environmental information that a UAV can acquire at its location. Each UAV independently acquires observation information, and the observation information of all UAVs at time t is represented as follows: The observation space of this invention includes endpoint location information, sensor node detection queue, obstacle observation, and aerial interval information. The endpoint location information is the distance of the UAV from the endpoint and its relative azimuth (based on the heading), i.e. The drone detects the status information of surrounding sensor nodes, including distance. relative azimuth The amount of data stored The information of the five sensor nodes closest to its own location is added to the sensor node detection queue, represented as follows: If a drone detects that a node is simultaneously detected by other drones, it first compares the distance between the node and the drone. If the node is too far away, it will not be added to its sensor node detection queue. The drone uses lidar to detect the distance between the drone's fuselage and the surface of obstacles in 16 equally spaced directions around it. The obstacle observation data is returned by the lidar and is represented as... The airspace information is the distance and relative azimuth angle between the two nearest adjacent UAVs, expressed as... Therefore, the observation information obtained by the i-th UAV at time t is

[0020]

[0021] 2. Motion space

[0022] This invention achieves trajectory control by controlling the turning angle of the UAV; therefore, the agent's actions correspond to the UAV's turning angle. Considering both the accuracy of trajectory planning and the computational complexity of the strategy, this invention defines the action space as 11 discrete values, each corresponding to a turning angle with an equal step size within a 45° range to the left and right. During each trajectory planning step, all UAVs execute actions simultaneously, and the actions of all UAVs at time t are represented as follows:

[0023] Step 2: Deep reinforcement learning trajectory planning method incorporating historical information

[0024] After receiving observation information, the UAV agent selects an action and executes it, thus gradually completing trajectory planning. This invention designs a trajectory planning strategy based on a Deep Recurrent Q-Network (DRQN). By fitting an action selection value function through a deep neural network, the UAV selects the optimal action based on its value. The trajectory planning employs a distributed control architecture, with each UAV maintaining its own trajectory planning strategy. For the i-th UAV agent, the selection value of each available action is estimated through a policy network based on the input observation information. First, a two-layer fully connected network processes the input observation information... Perform feature extraction, i.e.

[0025]

[0026] in, Let be the forward propagation output function of a fully connected network. From The extracted feature vectors.

[0027] Since the observation information acquired by the UAV agent is local environmental state information, to enable the UAV to make more accurate trajectory planning, this invention uses a Long Short-Term Memory (LSTM) network to fuse historical observation inputs, allowing the UAV agent to make trajectory planning decisions using more comprehensive environmental state information. LSTM selectively memorizes input information and the output from the previous time step through a gating mechanism, outputting a memory that incorporates historical input information. Feature vector The output after the LSTM layer is a fused feature vector. Right now

[0028]

[0029] in, This is the output function of the LSTM's forward propagation. Finally, the output layer calculates the output function based on the fused feature vector. Calculate the choice value of all actions, i.e.

[0030]

[0031] in, For the forward propagation output function of the output layer, The action value vector includes the selection value of all actions. The policy network learns the optimal action selection policy through iterative updates. This invention uses temporal difference error to update the policy network, with the target value being...

[0032]

[0033] Where γ is the reduction factor. Let be the reward the agent receives at time t. This refers to the next-state prediction value calculated using the target network. The target network has the same structure as the policy network, and its network parameters are directly copied and updated from the policy network at regular intervals. Its function is to stably calculate the next-state prediction value and reduce the iteration bias of the policy network. This invention uses mean squared error to calculate the error between the updated target value and the actual output value of the policy network, i.e.

[0034]

[0035] The output error of the policy network is propagated to each parameter of the network through backpropagation, and the network is updated through gradient descent.

[0036] Step 3: Dual-source composite reward function

[0037] After each action is performed, the drone receives a reward as feedback. To enable the drone swarm to collect as much data as possible while reducing task time, and considering the constraints of the data collection environment, this invention designs a dual-source composite reward function based on internal curiosity-driven and external environment-guided approaches. This allows the drone agent to perform data collection tasks normally in complex environments while comprehensively exploring the environment and finding the optimal trajectory planning strategy.

[0038] 1. Internal curiosity-driven rewards

[0039] The drone agent employs an Intrinsic Curiosity Module (ICM) to generate curiosity rewards for each decision made. Based on principles of human psychology, the ICM assesses curiosity according to the agent's unfamiliarity with the current state and outputs rewards based on the curiosity value to encourage exploration of the unfamiliar environment. Each agent independently maintains an ICM. For the i-th drone, its ICM uses a feature extraction network to extract features from both the current observation and the next observation during a decision-making process, obtaining feature vectors respectively. Then, the feedforward network is used to predict the next observation based on the current observation and the actions performed.

[0040]

[0041] in, This is the output function of the forward propagation of the feedforward network. The mean squared error is used to calculate the deviation between the predicted and actual values, reflecting the agent's unfamiliarity with the current observation. This prediction deviation also serves as the agent's curiosity reward.

[0042] After the ICM outputs a curiosity reward, it uses the prediction error to update the feedforward network, allowing the ICM to gradually become familiar with the current observation. Subsequent times when the same observation is input into the ICM, the prediction error will decrease, and the curiosity value will decrease accordingly. To update the feature extraction network, the ICM uses a backward network to predict the action to be performed based on the current and next observations, thus evaluating the effectiveness of the features extracted by the feature extraction network.

[0043]

[0044] in, This is the output function of the forward propagation of the inverse network. ICM uses the cross-entropy loss function to calculate the deviation between the predicted action and the actual action, and updates the feature extraction network through backpropagation.

[0045] In the early stages of policy learning for an agent, curiosity rewards encourage the agent to explore unfamiliar environmental states in order to comprehensively find the optimal strategy. As the agent's training process iterates, the output of curiosity rewards gradually decreases after the agent has fully experienced various environmental states, causing the agent to reduce exploration behavior and focus on learning the optimal strategy discovered in the early stages of training.

[0046] 2. External environment-guided rewards

[0047] To enable the UAV agent to complete the data acquisition task, this invention sets an environment-guided reward function based on the optimization objectives of the amount of data collected and the task time, as well as the constraints of the task environment. This function provides feedback to the UAV agent based on the environmental state. The external environment reward includes three parts: target point guidance reward, data acquisition reward, and collision avoidance and multi-UAV spacing maintenance reward. For the i-th UAV, its reward at time t is as follows:

[0048] The goal-point guidance reward is used to guide the drone towards the finish line, and is set based on the distance the agent moves towards the finish line.

[0049]

[0050] in, Let be the distance between the drone and the destination at time t. Let be the set of sensor nodes that are included in the sensor node detection queue by the i-th UAV. This reward guides the UAV as far as possible toward the destination during data collection, and guides the UAV to fly to the destination after it has failed to detect any sensor nodes to be collected.

[0051] A data collection reward guides the drone to fly towards IoT sensor nodes and collect as much data as possible. This reward is based on the distance the drone travels towards the sensor node and the amount of data collected from the node. For the j-th sensor node in the sensor node detection queue of the i-th drone, the reward the drone receives for performing the data collection task is...

[0052]

[0053] in, This indicates that the data from the j-th sensor node has not been fully collected. This indicates that the data for this node has been collected. Let be the distance between the i-th UAV and the j-th sensor node at time t. Let be the amount of data collected by the i-th UAV from the j-th sensor node. To encourage the UAV to rationally plan its data collection order based on the detected node information, the data collection reward for the UAV is the sum of the collection rewards it receives from each individual node, i.e.,

[0054]

[0055] Collision avoidance and multi-drone spacing maintenance rewards guide drones to avoid collisions with obstacles while maintaining distance from neighboring drones to avoid collisions and maintain air-to-air communication. The rewards are based on the distance between the drone and obstacles, as well as its distance from neighboring drones.

[0056]

[0057] in, Let be the distance between the i-th drone and the surface of the m-th obstacle at time t. The shortest safe distance between the drone and the obstacle. Let be the distance between the i-th drone and the n-th neighboring drone. These represent the minimum and maximum distances between drones, respectively.

[0058] Finally, the environmental guidance reward obtained by the drone at each decision step is the sum of the above three parts, namely...

[0059]

[0060] The combination of internal curiosity rewards and external environmental guidance rewards forms the reward for each decision made by the drone.

[0061]

[0062] Wherein, βc is a constant used to control the scale of internal curiosity reward.

[0063] The contents not described in detail in this application are existing technologies known to those skilled in the art.

Claims

1. A curiosity-reward-driven intelligent trajectory planning method for multi-UAV collaborative data acquisition, comprising the following steps: Step 1: Considering the characteristics of multi-UAV collaborative data acquisition tasks, and based on the relevant theoretical knowledge of reinforcement learning, the multi-UAV collaborative trajectory planning problem is modeled as a partially observable Markov decision process. Step 2: Establish a trajectory planning strategy that integrates historical environmental information. Use deep Q-networks and long short-term memory networks to make trajectory planning action decisions. This allows the system to remember the input information and make trajectory planning decisions based on integrated historical environmental information, thus solving the problem of trajectory planning under locally observable environmental conditions. Step 3: Use a dual-source composite reward function to train the strategy, including internal curiosity reward and external environment guidance reward. The internal curiosity reward is based on the unfamiliarity of the environment, encouraging the UAV to fully explore the complex environment; the external environment guidance reward is set according to the optimization goals of the amount of data collected and the task time, as well as the constraints of the task environment, guiding the UAV to complete the data collection task in a dynamic and complex environment.

2. The method according to claim 1, characterized in that... The method for modeling the multi-UAV data acquisition trajectory planning problem as a partially observable Markov decision process is as follows: This invention models the multi-UAV data acquisition trajectory planning problem as a partially observable Markov decision process, allowing UAVs to plan their trajectories step-by-step based on real-time environmental information during the data acquisition task. In this partially observable Markov decision process model, each UAV is an agent, and the observation space is the set of environmental information that the UAV can acquire at its current location, including destination position information, sensor node detection queues, obstacle observations, and aerial interval information. The destination position information consists of the distance and relative azimuth (based on the heading) of the UAV from the destination. The drone detects the status information of surrounding sensor nodes, including distance. relative azimuth The amount of data stored The information of the five sensor nodes closest to its own location is added to the sensor node detection queue, represented as follows: If a drone detects that a node is simultaneously detected by other drones, it first compares the distance between the node and the drone. If the node is too far away, it will not be added to its sensor node detection queue. The drone uses lidar to detect the distance between the drone's body and the surface of obstacles in 16 equally spaced directions around it. The obstacle observation data is returned by the lidar and is represented as follows: The airspace information is the distance and relative azimuth angle between the two nearest adjacent UAVs, expressed as... The observation information obtained by the drone agent at time t is The action space refers to the trajectory control achieved by controlling the turning angle of the UAV in this invention. Therefore, the action of the intelligent agent corresponds to the turning angle of the UAV. This invention defines the action space as 11 discrete values, which correspond to 11 turning angles with equal step sizes within a 45° range to the left and right.

3. The method according to claim 1, characterized in that... The method for designing a deep reinforcement learning trajectory planning strategy that incorporates historical information is as follows: This invention designs a trajectory planning strategy based on a Deep Recurrent Q-Network (DRQN). It uses a deep neural network to fit an action selection value function, enabling the UAV to select the optimal action based on the action selection value. The trajectory planning employs a distributed control architecture, with each UAV maintaining its own trajectory planning strategy. For the i-th UAV agent, based on the input observation information, the selection value of each available action is estimated through the policy network. First, a two-layer fully connected network processes the input observation information... Feature extraction is performed, that is: in, Let be the forward propagation output function of a fully connected network. From The extracted feature vectors are obtained by fusing historical observation inputs using a Long Short-Term Memory (LSTM) network. This allows the UAV agent to make trajectory planning decisions using more comprehensive environmental state information. The LSTM selectively memorizes the input information and the output from the previous time step through a gating mechanism, and outputs the memory that incorporates historical input information, resulting in the feature vectors. The output after the LSTM layer is a fused feature vector. Right now in, The output function of the LSTM forward propagation is used. Finally, the output layer is based on the fused feature vector. Calculate the choice value of all actions, i.e. in, For the forward propagation output function of the output layer, The action value vector includes the selection value of all actions. The policy network learns the optimal action selection policy through iterative updates. This invention uses temporal difference error to update the policy network, with the target value being... Where γ is the reduction factor. Let be the reward the agent receives at time t. The next-state prediction value is calculated using the target network, which has the same structure as the policy network. Its network parameters are directly copied and updated from the policy network at regular intervals. Its function is to stably calculate the next-state prediction value and reduce the iteration bias of the policy network. This invention uses mean squared error to calculate the error between the updated target value and the actual output value of the policy network. The output error of the policy network is propagated to each parameter of the network through backpropagation, and the network is updated through gradient descent.

4. The method according to claim 1, characterized in that... The method for setting the dual-source composite reward function is as follows: This invention designs a dual-source composite reward function based on internal curiosity-driven and external environment-guided approaches. This enables UAV agents to perform data acquisition tasks normally in complex environments while comprehensively exploring the environment and finding the optimal trajectory planning strategy in terms of data acquisition volume and task time. Firstly, an internal curiosity-driven reward is set up. The UAV agent uses an Intrinsic Curiosity Module (ICM) to generate curiosity rewards for each decision step. Each agent independently maintains an ICM. For the i-th UAV, its ICM uses a feature extraction network to extract features from the current observation and the next observation in a decision-making process, respectively, obtaining feature vectors. Then, the feedforward network is used to predict the next observation based on the current observation and the actions performed. in, This is the output function of the forward propagation of the feedforward network. The mean squared error is used to calculate the deviation between the predicted and actual values, reflecting the agent's unfamiliarity with the current observation. This prediction deviation also serves as the agent's curiosity reward. After the ICM outputs a curiosity reward, it uses the prediction error to update the feedforward network, allowing the ICM to gradually become familiar with the current observation. Subsequently, when the same observation is input into the ICM again, the prediction error will decrease, and the curiosity value will decrease accordingly. To update the feature extraction network, the ICM uses a backward network to predict the action to be performed based on the current and next observations, thus evaluating the effectiveness of the features extracted by the feature extraction network. in, This is the output function of the forward propagation of the inverse network. ICM uses the cross-entropy loss function to calculate the deviation between the predicted action and the actual action, and updates the feature extraction network through backpropagation. Secondly, it sets external environment guidance rewards, including three parts: target point guidance reward, data acquisition reward, and collision avoidance and multi-drone spacing maintenance reward. For the i-th drone, the reward it receives at time t includes: a target point guidance reward used to guide the drone towards the destination, set based on the distance the agent moves towards the destination, i.e. in, Let be the distance between the drone and the destination at time t. Let be the set of sensor nodes added to the sensor node detection queue by the i-th drone. This reward guides the drone as far as possible towards the destination during data collection. If the drone fails to detect the sensor node to be collected, it is guided to fly towards the destination. This reward is used to guide the drone to fly towards the IoT sensor node and collect as much data as possible. The data collection reward is set based on the distance the drone travels towards the sensor node and the amount of data collected from the node. For the j-th sensor node in the sensor node detection queue of the i-th drone, the reward obtained by the drone for performing data collection is... in, This indicates that the data from the j-th sensor node has not been fully collected. This indicates that the data for this node has been collected. Let be the distance between the i-th UAV and the j-th sensor node at time t. Let be the amount of data collected by the i-th UAV from the j-th sensor node. To encourage the UAV to rationally plan its data collection order based on the detected node information, the data collection reward received by the UAV is the sum of the collection rewards it receives from each individual node, i.e. And a collision avoidance and multi-drone spacing maintenance reward set based on the distance between the drone and obstacles and the distance to neighboring drones, i.e. in, Let be the distance between the i-th drone and the surface of the m-th obstacle at time t. The shortest safe distance between the drone and the obstacle. Let be the distance between the i-th drone and the n-th neighboring drone. These represent the minimum and maximum distances between drones, respectively. Finally, the environmental guidance reward obtained by a drone for each decision step is the sum of the above three parts, i.e. The combination of internal curiosity rewards and external environmental guidance rewards forms the reward for each decision made by the drone. Where, β c It is a constant used to control the scale of internal curiosity reward.

Images