Unmanned cluster path planning method, device and equipment based on deep reinforcement learning

Abstract

The invention relates to an unmanned cluster path planning method, device and equipment based on deep reinforcement learning, and the method comprises the steps: carrying out the modeling through employing an optimized artificial potential field method according to the urban environment parameters for executing a task, the initial position coordinates of each unmanned plane, and the position of a task target point, and obtaining an urban path planning model; based on the unmanned aerial vehicle dynamics model and the urban path planning model, defining a state space, an action space, a state transition process and a reward function in a reinforcement learning framework, and under the framework, performing reinforcement training on the deep Q neural network to obtain a dynamic path planning model; and utilizing the dynamic path planning model to act the unmanned aerial vehicle at the next moment according to the current moment sensing data of the unmanned aerial vehicle so as to realize the real-time planning of the path of the unmanned aerial vehicle. By adopting the method, the action at the next moment can be accurately predicted in real time according to the current environment state in a complex urban scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle path planning, in particular to an unmanned cluster path planning method, device and equipment based on deep reinforcement learning. BACKGROUND

[0002] With the rapid progress of unmanned aerial vehicle technology, its application has become more and more extensive. Especially in the civil field, such as logistics, agriculture and environmental protection, unmanned aerial vehicles also show great application potential. However, the wide application of unmanned aerial vehicles also brings a series of challenges. First, unmanned aerial vehicles need to navigate and avoid obstacles in a complex and variable environment, which requires unmanned aerial vehicles to have high perception and intelligent decision-making capabilities. Secondly, the energy efficiency of unmanned aerial vehicles is a key issue, because the limitation of battery technology seriously restricts the endurance of unmanned aerial vehicles. In addition, the safety of unmanned aerial vehicles cannot be ignored, how to ensure that unmanned aerial vehicles do not pose a threat to humans or property when performing tasks is a problem that unmanned aerial vehicle technology must face.

[0003] In this context, intelligent planning of unmanned aerial vehicles has become a research hotspot. Intelligent planning not only requires unmanned aerial vehicles to efficiently complete tasks, but also considers the safety of flight, the rational use of energy and the adaptability to the environment. Traditional planning methods often rely on pre-set rules or complex mathematical models, and these methods often perform poorly when faced with dynamic and uncertain environments. Therefore, an intelligent planning method that can adaptively learn and dynamically adjust the planning strategy is of great significance to improve the autonomy and task execution efficiency of unmanned aerial vehicles. SUMMARY

[0004] Therefore, it is necessary to provide an unmanned cluster path planning method, device and equipment based on deep reinforcement learning, which can adjust the planning strategy in real time according to the current complex environment.

[0005] An unmanned cluster path planning method based on deep reinforcement learning, the method comprising: constructing an unmanned aerial vehicle dynamics model, in which the speed, pitch angle and yaw angle of the unmanned aerial vehicle are constrained at each time, and the nearest distance between the unmanned aerial vehicle and the nearest obstacle is also constrained; obtaining related parameters of a task to be performed by a cluster of unmanned aerial vehicles, the related parameters including urban environment parameters for performing the task, initial position coordinates of each unmanned aerial vehicle and target point positions, modeling by using an optimized artificial potential field method according to the related parameters to obtain an urban path planning model; Based on the unmanned aerial vehicle dynamics model and the urban path planning model, a state space, an action space, a state transition process and a reward function in a reinforcement learning framework are constructed; Under the reinforcement learning framework, a deep Q neural network is reinforced to obtain a dynamic path planning model; Current perception data of the unmanned aerial vehicle is obtained, and the dynamic path planning model is used to determine the unmanned aerial vehicle action at the next moment, so as to realize real-time planning of the unmanned aerial vehicle path.

[0006] In one embodiment, an optimized artificial potential field method is used to model according to the related parameters to obtain an urban path planning model, including: The urban environment parameters in the related parameters include building-related data and urban space data; According to the urban space data, a rectangular space of a three-dimensional urban model is determined, and in the three-dimensional urban model, corresponding cuboid building models are generated according to the building-related data; Based on the three-dimensional urban model, by introducing a collision distance, an angle and a distance factor, the attractive force of the target point position on the unmanned aerial vehicle and the repulsive force of the building model on the unmanned aerial vehicle are calculated, thereby obtaining an optimized repulsive force function for guiding the unmanned aerial vehicle path planning, so as to obtain the urban path planning model.

[0007] In one embodiment, the optimized repulsive force function is expressed as: ; Wherein, ; In the above formula, represents the repulsive force of the building model to the unmanned aerial vehicle in the direction from the building model to the unmanned aerial vehicle, represents the attractive force of the target point position to the unmanned aerial vehicle in the direction from the unmanned aerial vehicle to the target point position, represents a distance factor coefficient, represents a collision distance, represents the distance of a straight line connected between each building model and the unmanned aerial vehicle and the target point position, represents the included angle between the building model and the unmanned aerial vehicle, and respectively represent the moving direction angle of the unmanned aerial vehicle, the angle of left and right deviation, is the straight line distance between the unmanned aerial vehicle and the building model, , respectively represent the positions of the unmanned aerial vehicle and the building model.

[0008] In one embodiment, in the reinforcement learning framework, the state space includes: the current moment environment state, the target point state and the unmanned aerial vehicle state; The environment state is the position coordinates, the half-length value and the half-width value in the horizontal and vertical directions, and the height of a single cuboid building model in the city three-dimensional model. The target point state is the position coordinates in the city three-dimensional model. The UAV state is the position coordinates at the current time, the energy consumed, the movement direction, and the crash probability in the city three-dimensional model.

[0009] In one embodiment, the state transition process in the reinforcement learning framework is represented as: updating the position coordinates of the UAV according to the movement direction at the current time; calculating the current distance between the updated position coordinates of the UAV and the target point position; obtaining the movement direction at the next time according to the current distance and the environment situation; updating the energy consumption of the UAV; judging whether the UAV crashes according to the crash probability.

[0010] In one embodiment, the reward function in the reinforcement learning framework includes a target function, an ability reward, and a collision penalty.

[0011] In one embodiment, the deep Q neural network includes a Q network and a Q target network with the same network structure, both of which are stacked by four fully connected layers and ReLU activation functions, and are represented as: . In the above formula, represents the expected return of taking action in state , represents the trainable parameters of the network, represents a function mapping realized by a multi-layer perceptron.

[0012] In one embodiment, when the deep Q neural network is reinforced trained under the reinforcement learning framework: After the UAV interacts with the environment under the reinforcement learning framework, the UAV performs an action and obtains an observation result, which includes a reward and a next state; storing the observation results of multiple interactions between the UAV and the environment under the reinforcement learning framework into an experience pool; randomly sampling a batch of training data in the experience pool, calculating the predicted Q value of the corresponding action in the current state for each observation result sampled using the Q network, and calculating the maximum Q value of the next state as the target Q value using the Q target network; According to the predicted Q value and a target Q value, a loss function is calculated, trainable parameters in the Q network are updated by using the loss function until convergence, and the trained Q network is taken as the dynamic path planning model.

[0013] An unmanned cluster path planning device based on deep reinforcement learning, the device comprising: An unmanned aerial vehicle dynamics model construction module, configured to construct an unmanned aerial vehicle dynamics model, in which unmanned aerial vehicle speed, pitch angle and yaw angle are respectively constrained at each time, and the nearest distance between the unmanned aerial vehicle and the nearest obstacle is also constrained; A city path planning model, configured to obtain relevant parameters of a task to be performed by a cluster of unmanned aerial vehicles, the relevant parameters including city environment parameters for performing the task, initial position coordinates of each unmanned aerial vehicle and a task target point position, and to model by using an optimized artificial potential field method according to the relevant parameters to obtain the city path planning model; A reinforcement learning framework definition module, configured to construct a state space, an action space, a state transition process and a reward function in a reinforcement learning framework based on the unmanned aerial vehicle dynamics model and the city path planning model; A dynamic path planning model obtaining module, configured to reinforce training of a deep Q neural network under the reinforcement learning framework to obtain a dynamic path planning model; A path real-time planning module, configured to obtain current time perception data of an unmanned aerial vehicle, and to use the dynamic path planning model to determine an action of the unmanned aerial vehicle at a next time to realize real-time planning of a path of the unmanned aerial vehicle.

[0014] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing steps in the method for unmanned cluster path planning based on deep reinforcement learning when executing the computer program.

[0015] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement steps in the method for unmanned cluster path planning based on deep reinforcement learning.

[0016] The above-mentioned unmanned swarm path planning method, apparatus, and equipment based on deep reinforcement learning utilize an optimized artificial potential field method to model an urban path planning model based on the urban environment parameters of the mission, the initial position coordinates of each drone, and the location of the mission target point. Based on the drone dynamics model and the urban path planning model, the state space, action space, state transition process, and reward function within the reinforcement learning framework are defined. Within this framework, a deep Q neural network is reinforced and trained to obtain a dynamic path planning model. This dynamic path planning model uses the drone's current perception data to predict the next moment's action, thereby achieving real-time drone path planning. This method can accurately predict the next moment's action based on the current environmental state in complex urban scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 1 is a flowchart of an unmanned swarm path planning method based on deep reinforcement learning in one embodiment; Figure 2 Schematic diagram of pitch angle and yaw angle in one embodiment; Figure 3 Schematic diagram of the reinforcement learning framework proposed in this method in one embodiment; Figure 4 Schematic diagram of a deep Q neural network framework in one embodiment; Figure 5 A schematic diagram of the DQN model training score curve in a simulation experiment; Figure 6 The following is a diagram showing the visual results of urban path planning for a single UAV using this method in a simulation experiment, where: Figure 6 (a) is a two-dimensional visual result diagram. Figure 6 (b) Schematic diagram of three-dimensional visual results; Figure 7 This is a diagram showing the visual results of urban path planning using this method for drone clusters in the four scenarios in Table 3 in a simulation experiment. Figure 7 (a) Figure 7 (b) Figure 7 (c) and Figure 7 (d) Schematic diagram of the results for four scenarios respectively; Figure 8 This is a structural block diagram of an unmanned swarm path planning device based on deep reinforcement learning in one embodiment; Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description will be given below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not to limit the present application.

[0019] In the present application, as shown in Figure 1 A deep reinforcement learning-based unmanned cluster path planning method is provided, which specifically includes the following steps: Step S100, constructing an unmanned aerial vehicle dynamics model, in which the unmanned aerial vehicle speed, pitch angle and yaw angle are respectively constrained at each time, and the nearest distance between the unmanned aerial vehicle and the nearest obstacle is also constrained.

[0020] Step S110, obtaining relevant parameters of the unmanned aerial vehicle cluster to be executed task, the relevant parameters including city environment parameters for executing task, initial position coordinates of each unmanned aerial vehicle and task target point position, modeling according to the relevant parameters by using the optimized artificial potential field method to obtain the city path planning model.

[0021] Step S120, based on the unmanned aerial vehicle dynamics model and the city path planning model, constructing the state space, action space, state transition process and reward function in the reinforcement learning framework.

[0022] Step S130, reinforcing training of the deep Q neural network under the reinforcement learning framework to obtain a dynamic path planning model.

[0023] Step S140, obtaining the current time perception data of the unmanned aerial vehicle, and using the dynamic path planning model to determine the unmanned aerial vehicle action at the next time to realize real-time planning of the unmanned aerial vehicle path.

[0024] The wide use of unmanned aerial vehicles brings a series of challenges. First, unmanned aerial vehicles need to perform autonomous navigation and obstacle avoidance in complex and variable environments, which requires unmanned aerial vehicles to have high perception and intelligent decision-making capabilities. Second, the energy efficiency of unmanned aerial vehicles is a key issue, as the limitations of battery technology severely restrict the endurance of unmanned aerial vehicles. In addition, the safety of unmanned aerial vehicles cannot be ignored, and how to ensure that unmanned aerial vehicles do not pose a threat to humans or property when performing tasks is a problem that unmanned aerial vehicle technology must face. In the present application, many challenges of unmanned aerial vehicles in complex scenarios in the real world are simplified, and it is assumed that the unmanned aerial vehicle flies in a three-dimensional map and there are multiple obstacles in the three-dimensional map. The goal of the unmanned aerial vehicle is to fly from the starting point to the ending point, only considering avoiding collision with obstacles and minimizing flight distance or flight time.

[0025] In step S100, first, the unmanned aerial vehicle in the unmanned aerial vehicle cluster is modeled, and the Definition of the mth UAV, without considering wind and air resistance, the dynamics model of its flight in the environment can be modeled as: In the above formula, is the position coordinate of the mth UAV at time t, is the flight speed of the UAV, and are the pitch angle and yaw angle of the UAV, as shown in Figure 2 and are the horizontal and vertical components of the UAV acceleration, denotes the derivative of . And the UAV should also satisfy the following constraints: In the above formula, are the minimum and maximum values of the UAV flight speed, respectively.

[0026] Further, in a complex three-dimensional environment, define multiple target points D = {D1, D2, …, D m M , the coordinates of the target point is the distance between the UAV and the target point : To achieve the flight target, the distance D between the UAV and the obstacle should satisfy , is the safety distance. At the same time, when the distance between the UAV and the target point , it is called that the UAV reaches the target point (D is the target threshold).

[0027] ​​​​​​​​​​​​​​​​​In step S110, the optimized artificial potential field method is used to model according to the related parameters to obtain the urban path planning model, including: wherein the urban environment parameters in the related parameters include building related data and urban space data, the rectangular space of the urban three-dimensional model is determined according to the urban space data, and the corresponding cuboid building model is generated in the urban three-dimensional model according to the building related data. Based on the urban three-dimensional model, the repulsive force function of guiding the path planning of the unmanned aerial vehicle is obtained by introducing the collision distance, the angle and the distance factor to calculate the attractive force of the target point position on the unmanned aerial vehicle and the repulsive force of the building model on the unmanned aerial vehicle, so as to obtain the urban path planning model.

[0028] Specifically, an example of performing actual urban three-dimensional environment modeling is given to better illustrate the process of actual environment modeling. In the actual application of the method for actual environment modeling, modeling can be performed according to the actual situation.

[0029] In an example of actual environment modeling, the urban environment space is considered to be a rectangular space with a length and width of 500 units and a height of 100 units, and 26 buildings are provided, and the coordinate parameters X, Y and the height parameter H are shown in the following table: Table 1 Obstacle position coordinate table

[0030] Specifically, when generating a simulated environment, the system randomly allocates center coordinates to each building and checks whether the new building will overlap with the existing buildings in the environment or exceed the boundary.

[0031] In this embodiment, the artificial potential field method is also used to model the subsequent path planning based on the urban three-dimensional model. The artificial potential field method is a robot path planning algorithm proposed by Khatib. The algorithm regards the target and the obstacle as objects with attractive force and repulsive force to the robot respectively, and the robot moves along the combined force of the attractive force and the repulsive force. Therefore, in actual application to unmanned aerial vehicle path planning, the target point and the building are regarded as attractive force and repulsive force objects respectively, that is, the artificial potential field method can be used to model the environment for unmanned aerial vehicle path planning.

[0032] Specifically, the main principle of the artificial potential field method is the gradient potential field method. In the algorithm, the coordinates of the target point are , the current position coordinates of the unmanned aerial vehicle are , and the building position coordinates are . Therefore, the attractive potential field to which the unmanned aerial vehicle is subjected when moving in the environment is: ; In the above formula, is the attractive gain constant, is the straight-line distance between the target point and the unmanned aerial vehicle. And the size of the gravitational force is the negative gradient of the gravitational potential field: ; Similarly, the repulsive potential field of the UAV affected by the building can be set as: When : ; When : ; In the above formula, is the repulsive field constant, is the straight-line distance between the UAV and the building, is the influence distance of the building on the UAV. The size of the repulsive force on the UAV is the negative gradient of the repulsive potential field : When : ; When : .

[0033] Further, assuming that in the environment where the UAV performs the task, there are n buildings with a distance less than the influence distance from the UAV, the UAV moves in the resultant potential field composed of the target point and multiple buildings in the entire environment, and the size of the resultant force on the UAV is: ; When the path length of the UAV is set to be , the next time path coordinate point of the UAV under the influence of the resultant force can be obtained as: l ; However, in actual use for environment modeling, the traditional artificial potential field method often falls into local optimum or target unattainable problems, therefore, the collision distance is introduced to screen obstacles, the angle is introduced to define the influence range of obstacles, the distance factor is introduced, and the virtual target point is constructed to solve the above problems.

[0034] Further, the optimized repulsive function is obtained, which is expressed as: ; Wherein, ; In the above formula, represents the repulsive force of the building model pointing to the UAV, ​​Indicates the gravity pointing from the drone to the target point. represents the distance factor coefficient, Indicates the collision distance, Indicates the distance from each building model to the straight line connecting the drone and the target point. Indicates the angle between the building model and the drone, and Represents the motion direction angle of the UAV Offset to the left or right Angle, is the straight-line distance between the drone and the building model, 、 Represent the positions of the drone and building model respectively.

[0035] In this embodiment, real-time path planning is achieved using deep reinforcement learning (DRL), a technology that combines deep learning and reinforcement learning. The Deep Q-Network (DQN) algorithm uses an end-to-end learning approach to enable drones to autonomously learn optimal flight strategies through interaction with their environment. This approach not only adapts to environmental changes but also improves planning efficiency and safety through continuous learning.

[0036] In step S120, based on the urban path planning model constructed above, reinforcement learning is used to perform real-time path planning, wherein the basic framework of reinforcement learning is as follows: Figure 3 As shown, it includes state space, action space, state transition process, and reward function.

[0037] In this embodiment, the state space in the reinforcement learning framework includes the current state of the environment, the state of the target point, and the state of the drone. The environment state is the position coordinates, half-length and half-width in the horizontal and vertical directions, and the height of a single rectangular building model in the 3D city model. The target point state is the position coordinates in the 3D city model. The drone state is the current position coordinates, consumed energy, movement direction, and crash probability in the 3D city model.

[0038] Specifically, in any At this moment, the state space includes the environment state, target state, and drone state. The environment state is a randomly generated square building group, and the state vector of a single building is ,in and Respectively represent the horizontal and vertical coordinates of the building on the horizontal plane, both within the generated map range Inside, and represents the half-length and half-width values of the building in the horizontal and vertical directions, represents the height of the building. For a building, its state does not change over time.

[0039] Specifically, the target state is the position coordinates of the target in the map, and the state vector is , wherein respectively represent the positions in the horizontal, vertical, and vertical axes.

[0040] Specifically, the state of the unmanned aerial vehicle is relatively complex, and the state vector is , which changes over time. respectively represent the horizontal, vertical, and vertical axis coordinates of the unmanned aerial vehicle at time , and and respectively represent the energy consumed by the unmanned aerial vehicle at time and the movement direction, represent the probability of the unmanned aerial vehicle crashing.

[0041] In this embodiment, the state transition process in the reinforcement learning framework is represented as: updating the position coordinates of the unmanned aerial vehicle according to the movement direction at the current time, calculating the current distance between the updated position coordinates of the unmanned aerial vehicle and the position of the target point, obtaining the movement direction at the next time according to the current distance and the environmental conditions, updating the energy consumption of the unmanned aerial vehicle, and at the same time, determining whether the unmanned aerial vehicle crashes according to the probability of crashing.

[0042] In this embodiment, the state transition describes the probability of moving to state after taking action in state . For the unmanned aerial vehicle model in this method, the state can be represented as . Here, represents the coordinates of the unmanned aerial vehicle in three-dimensional space, represents the position coordinates of the task target, is the energy consumption of the unmanned aerial vehicle, and respectively represent the direction of the unmanned aerial vehicle and the environmental conditions, represents the probability of the unmanned aerial vehicle crashing. The state transition process of the unmanned aerial vehicle is relatively complex and difficult to intuitively represent with formulas. The state transition process can be realized by the following steps: First, update the coordinates of the unmanned aerial vehicle according to the time and the movement direction : ; Then, calculate the current distance to the target.

[0043] Next, according to environmental conditions adjust the direction .

[0044] Further, update the energy consumption of the UAV .

[0045] Finally, according to the probability of crashing determine whether the UAV crashes.

[0046] In this embodiment, the reward function quantifies the immediate reward of moving from state to state by taking action . For the UAV model, the reward function can be designed as: ; In the above formula, , and represent the target reward, energy consumption reward, and collision penalty, respectively.

[0047] Specifically, the target reward is a reward function for completing the task, represented as: ; In the above formula, is a weighting factor, and represent the initial distance and the current distance from the target, respectively. The target reward function is set to be inversely proportional to , i.e., the closer to the target, the higher the reward.

[0048] Specifically, the energy consumption reward is a reward function for energy consumption, represented as: ; In the above formula, is a weighting factor, is the initial energy, is the energy consumption, i.e., the remaining energy of the UAV. The more the remaining energy, the higher the reward value.

[0049] Specifically, the collision penalty is a penalty for the possible occurrence of a crash, represented as: ; In the above formula, is a weighting factor, is the probability of crashing.

[0050] In this embodiment, only the process of a UAV reaching the target is considered, so the action space only considers the motion of a single UAV. For a single UAV, its action space can be expressed as .here Representing drones in Possible movement directions on the three coordinate axes.

[0051] During the reinforcement learning training process, each training session uses N drones, each trained to reach a single target point. After iterative reinforcement training, a single drone can complete the task of reaching a single target point with a high probability.

[0052] In this embodiment, the core idea of ​​the DQN (Deep Q Network) is to use a deep neural network to approximate the Q function, that is, the action-value function, which predicts the expected return of taking a specific action in a given state. Its framework is as follows Figure 4 shown.

[0053] In this embodiment, the DQN algorithm relies on a deep neural network to approximate the Q function. The network structure should be complex enough to learn the dynamic characteristics of the environment. The deep Q neural network includes a Q network and a Q target network with the same network structure. The Q network and the Q target network are both composed of four fully connected layers stacked with the ReLU activation function, expressed as: ; In the above formula, Indicates that the status Take action The expected return, represents the trainable parameters of the network, Represents the function mapping implemented by the multilayer perceptron.

[0054] In this embodiment, when reinforcement training is performed on a deep Q neural network under a reinforcement learning framework: under the reinforcement learning framework, after the drone interacts with the environment, it performs an action and obtains an observation result, which includes a reward and a next state. The observation results of multiple interactions between the drone and the environment under the reinforcement learning framework are stored in an experience recycling pool, and a batch of training data is randomly sampled from the experience recycling pool. For each observation result obtained by the sample, the Q network is used to calculate the predicted Q value of executing the corresponding action in the current state, and the Q target network is used to calculate the maximum Q value of the next state as the target Q value. The loss function is calculated based on the predicted Q value and the target Q value, and the loss function is used to update the trainable parameters in the Q network until convergence, and the trained Q network is used as a dynamic path planning model.

[0055] Specifically, first, a deep neural network is initialized as an estimator of Q function, which is referred to as Q network, and a Q target network identical to the Q network is initialized, and an experience replay buffer is initialized. Preferably, the dimension of the neural network model is 42x16x16x16x27, and the model size is 36 KB. Then, the UAV interacts with the environment, performs actions and observes the results (rewards and next states), which are stored in the experience replay buffer. This process is referred to as experience replay, which is a key technique for improving sample efficiency of the DQN algorithm. In the embodiment, a circular buffer is used to store the latest experience transitions, each of which consists of a state, an action, a reward, a next state and a done flag, and is denoted as: ; The mechanism allows the algorithm to break the temporal correlation between data by random sampling, thereby improving the stability of the training process.

[0056] Further, a batch of data is randomly sampled from the experience replay buffer during training to reduce the correlation between data. For each transition in the sampled batch, the maximum Q value of the next state is calculated using the Q target network (i.e., the Q value of the best action selected by greediness). Then, the difference between the Q value predicted by the current network and the target Q value is calculated using the sampled data, and the difference is used as the value of the loss function. The goal of training is to minimize the loss function , which measures the difference between the predicted Q value and the target Q value. In the formula, the loss function is denoted as: ; In the above formula, is a time discount factor, is a target network parameter, which is updated from the source network parameter at a rate of , and mean squared error (MSE) is used as the loss function, which calculates the difference between the predicted Q value and the target Q value. The target Q value is given by the target Q network, which is a delayed update copy of the Q network, and the parameter of the target Q network is updated at a rate of , and the update process is denoted as: ; The soft update strategy helps to stabilize the training process and prevent the difference between the target network and the source network from being too large.

[0057] Further, in order to balance exploration and exploitation, an strategy is used, which randomly selects an action with a probability of and selects the current optimal action with a probability of , and is denoted as: ;

[0058] In this paper, a pseudo code for training the deep Q neural network is also provided, as shown in Table 2:

[0059] In step S140, after the training is completed, the trained Q network is taken as a dynamic path planning model. When the UAV is flying in real time, at a certain time, it is in a specific state, and the state at this time is input into the dynamic path planning model to predict the action needed to obtain the maximum Q value, and the next time action is performed according to the prediction.

[0060] In this paper, the effectiveness of the method is also proved by simulation experiments. In the simulation experiment, the urban path planning problem of the UAV cluster is solved, that is, given the initial position of multiple UAVs, building distribution and target point, the collision-free path within the limited energy and maximum range is solved. For this problem, the method is tested in urban environments with different high building densities, and the test experimental parameters are shown in Table 3.

[0061] Table 3 Parameter setting table

[0062] Under the experimental settings shown in Table 3, the initial points, target points and city maps are randomly generated for testing.

[0063] In the simulation experiment, first, the DQN deep reinforcement learning model is trained for 19 hours in a city environment with a density of 10 for a cluster containing 15 UAVs. In the reward function, the action is first selected according to the Q value, then the state is changed according to the selected action, the revenue is obtained, the total revenue is calculated, and the interaction experience of each time is stored. A positive reward is given to the successful state, the experience is strengthened, and the state change is completed. If the pass rate is large, the difficulty is increased. The score curve is shown in Figure 5 .

[0064] As can be seen from Figure 5 , with the increase of the number of training, the cluster path planning score gradually stabilizes after 600 times of training. Since the collision of the UAV and the exceeding of the maximum number of steps will produce a penalty of -200 and -200 respectively, reaching the vicinity of the target point will produce a reward of +200, therefore, the final score after training fluctuates around 3000, which indicates that the cluster can reach the target position in a dense urban environment.

[0065] Further, first, the model is tested by a single UAV under different building densities, and the results are shown in Figure 6In order to further demonstrate the planning effect of the model, a verification experiment was conducted on the drone cluster paths of the above four scenarios in a simulation environment, and the results are shown as follows: Figure 7 As shown in the figure, the step length of the experiment is 1 second, the target point is represented by red, the building is presented in the form of a cube, and the path of the cluster is presented in the form of colored dots.

[0066] from Figure 7 It can be found that at different starting positions, the drone cluster can reach the vicinity of the target point while avoiding collision with buildings. Figure 7 It can be found that when the cluster reaches the target point, it will first move upward for a distance and then fly towards the target point. This shows that during the training process, the algorithm may fall into a local optimal solution when approaching the end point and has not effectively explored other strategies. There are still certain defects in the design of the reward and penalty schemes.

[0067] In the above-mentioned unmanned swarm path planning method based on deep reinforcement learning, an urban path planning model including gravity and repulsion is constructed by optimizing the artificial potential field method, a reinforcement learning framework with three-dimensional state, action space and multi-objective reward function is defined, and a four-layer fully connected DQN network is trained using experience replay and target network soft update mechanism. This enables drones to autonomously learn obstacle avoidance strategies in dynamic and complex environments, achieving real-time path planning with the shortest distance and low energy consumption. Experimental simulation results have proved that this method can effectively solve path planning problems in complex environments.

[0068] Specifically, by coupling the artificial potential field method with reinforcement learning, the artificial potential field method is used to model the urban environment as the combined force field of the target's gravity and the building's repulsion, providing a simulation environment with clear physical constraints for reinforcement learning. At the same time, the DQN algorithm dynamically optimizes the drone's motion strategy in the gravity-repulsion field by learning environmental interaction data (such as state-action-reward sequences), breaking through the limitation of traditional potential field methods that are prone to falling into local optimality, and increasing the drone's obstacle avoidance success rate in three-dimensional urban environments to more than 80% (high-density scenarios), and shortening the path distance by 15%-20% compared with traditional methods.

[0069] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least one part of the steps in the method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0070] In one embodiment, as shown in Figure 8 An unmanned cluster path planning device based on deep reinforcement learning is provided, comprising: an unmanned aerial vehicle dynamics model construction module 200, a city path planning model 210, a reinforcement learning framework construction module 220, a dynamic path planning model obtaining module 230, and a path real-time planning module 240. The unmanned aerial vehicle dynamics model construction module 200 is configured to construct an unmanned aerial vehicle dynamics model, in which the speed, pitch angle, and yaw angle of the unmanned aerial vehicle are constrained at each time, and the closest distance between the unmanned aerial vehicle and the nearest obstacle is also constrained.

[0071] The city path planning model 210 is configured to obtain relevant parameters of a task to be performed by a cluster of unmanned aerial vehicles, including city environment parameters for performing the task, initial position coordinates of each unmanned aerial vehicle, and a target point position, and to model the city path planning model by using an optimized artificial potential field method based on the relevant parameters.

[0072] The reinforcement learning framework construction module 220 is configured to define a state space, an action space, a state transition process, and a reward function in a reinforcement learning framework based on the unmanned aerial vehicle dynamics model and the city path planning model.

[0073] The dynamic path planning model obtaining module 230 is configured to reinforce the training of a deep Q neural network under the reinforcement learning framework to obtain a dynamic path planning model.

[0074] The path real-time planning module 240 is configured to obtain current perception data of the unmanned aerial vehicle, and to use the dynamic path planning model to determine the action of the unmanned aerial vehicle at the next time to achieve real-time planning of the path of the unmanned aerial vehicle.

[0075] The specific limitations of the unmanned cluster path planning device based on deep reinforcement learning can refer to the limitations of the unmanned cluster path planning method based on deep reinforcement learning in the above, which will not be repeated here. Each module in the unmanned cluster path planning device based on deep reinforcement learning can be realized by software, hardware and their combination in whole or in part. The above-mentioned each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operation corresponding to each module by the processor.

[0076] In one embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 9 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement an unmanned cluster path planning method based on deep reinforcement learning. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0077] Those skilled in the art can understand that Figure 9 The structure shown in the above is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0078] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps: Constructing an unmanned aerial vehicle dynamics model, in which the unmanned aerial vehicle speed, pitch angle and yaw angle are respectively constrained at each time, and the closest distance between the unmanned aerial vehicle and the nearest obstacle is also constrained; Obtain relevant parameters of the UAV cluster's mission to be performed, including the city environment parameters for the mission, the initial position coordinates of each UAV, and the mission target point location. Based on the relevant parameters, an optimized artificial potential field method is used to perform modeling to obtain an urban path planning model. Based on the UAV dynamics model and urban path planning model, construct the state space, action space, state transition process and reward function in the reinforcement learning framework; Under the reinforcement learning framework, the deep Q neural network is reinforced trained to obtain a dynamic path planning model; The current moment perception data of the UAV is obtained, and the UAV action at the next moment is calculated using the dynamic path planning model to achieve real-time planning of the UAV path.

[0079] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Constructing a UAV dynamics model, wherein the UAV speed, pitch angle, and yaw angle are constrained at each moment, and the minimum distance between the UAV and the nearest obstacle is constrained; Obtain relevant parameters of the UAV cluster's mission to be performed, including the city environment parameters for the mission, the initial position coordinates of each UAV, and the mission target point location. Based on the relevant parameters, an optimized artificial potential field method is used to perform modeling to obtain an urban path planning model. Based on the UAV dynamics model and urban path planning model, construct the state space, action space, state transition process and reward function in the reinforcement learning framework; Under the reinforcement learning framework, the deep Q neural network is reinforced trained to obtain a dynamic path planning model; The current moment perception data of the UAV is obtained, and the UAV action at the next moment is calculated using the dynamic path planning model to achieve real-time planning of the UAV path.

[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0081] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0082] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for unmanned swarm path planning based on deep reinforcement learning, characterized in that: The method comprises: Constructing a UAV dynamics model, wherein the UAV speed, pitch angle, and yaw angle are constrained at each moment, and the minimum distance between the UAV and the nearest obstacle is constrained; Obtain relevant parameters of the UAV cluster's mission to be performed, including the city environment parameters for the mission, the initial position coordinates of each UAV, and the mission target point location. Based on the relevant parameters, an optimized artificial potential field method is used to perform modeling to obtain an urban path planning model. Based on the UAV dynamics model and urban path planning model, construct the state space, action space, state transition process and reward function in the reinforcement learning framework; Under the reinforcement learning framework, the deep Q neural network is reinforced trained to obtain a dynamic path planning model; The current moment perception data of the UAV is obtained, and the UAV action at the next moment is calculated using the dynamic path planning model to achieve real-time planning of the UAV path.

2. The unmanned cluster path planning method based on deep reinforcement learning according to claim 1 is characterized in that: Based on the relevant parameters, the optimized artificial potential field method is used to perform modeling to obtain an urban path planning model, including: The urban environment parameters in the relevant parameters include building-related data and urban space data; Determine a rectangular space of a three-dimensional urban model according to the urban space data, and generate a corresponding rectangular parallelepiped building model in the three-dimensional urban model according to the building-related data; Based on the three-dimensional urban model, by introducing collision distance, angle and distance factor, the gravitational force of the target point position on the drone and the repulsive force of the building model on the drone are calculated, thereby obtaining an optimized repulsive force function to guide the drone path planning, thereby obtaining the urban path planning model.

3. The unmanned cluster path planning method based on deep reinforcement learning according to claim 2 is characterized in that: The optimized repulsion function is expressed as: in, In the above formula, Indicates the repulsive force from the building model to the drone, Indicates the gravity pointing from the drone to the target point. represents the distance factor coefficient, Indicates the collision distance, Indicates the distance from each building model to the straight line connecting the drone and the target point. Indicates the angle between the building model and the drone, and Represents the motion direction angle of the UAV Offset to the left or right angle, is the straight-line distance between the drone and the building model, 、 Represent the positions of the drone and building model respectively.

4. The unmanned cluster path planning method based on deep reinforcement learning according to claim 3 is characterized in that: In the reinforcement learning framework, the state space includes: the current state of the environment, the state of the target point, and the state of the drone; The environmental state is the position coordinates, half-length and half-width in the horizontal and vertical directions, and height of a single rectangular building model in the three-dimensional city model; The target point state is the position coordinates in the three-dimensional city model; The drone status is the current position coordinates, consumed energy, movement direction, and crash probability in the three-dimensional city model.

5. The unmanned cluster path planning method based on deep reinforcement learning according to claim 4 is characterized in that: In the reinforcement learning framework, the state transition process is expressed as: Update the drone's position coordinates based on the current direction of motion; Calculate the current distance to the target point based on the updated position coordinates of the drone; According to the current distance and environmental conditions, the movement direction at the next moment is obtained; Update the energy consumption of drones; Determine whether the drone has crashed based on the crash probability.

6. The unmanned cluster path planning method based on deep reinforcement learning according to claim 5 is characterized in that: In the reinforcement learning framework, the reward function includes an objective function, an ability reward, and a collision penalty.

7. The unmanned cluster path planning method based on deep reinforcement learning according to claim 6 is characterized in that: The deep Q neural network includes a Q network and a Q target network with the same network structure. The Q network and the Q target network are both composed of four fully connected layers and a ReLU activation function stacked together, which can be expressed as: In the above formula, Indicates that the status Take action The expected return, represents the trainable parameters of the network, Represents the function mapping implemented by the multilayer perceptron.

8. The unmanned cluster path planning method based on deep reinforcement learning according to claim 7 is characterized in that: Under the reinforcement learning framework, when performing reinforcement training on the deep Q neural network: In the reinforcement learning framework, after the drone interacts with the environment, it performs actions and obtains observations, which include rewards and next states. The observation results of multiple interactions between the drone and the environment under the reinforcement learning framework are stored in the experience recycling pool; Randomly sampling a batch of training data from the experience recycling pool, using the Q network to calculate the predicted Q value of executing the corresponding action in the current state for each observation result obtained by the sampling, and using the Q target network to calculate the maximum Q value of the next state as the target Q value; A loss function is calculated based on the predicted Q value and the target Q value, and the trainable parameters in the Q network are updated using the loss function until convergence, and the trained Q network is used as the dynamic path planning model.

9. An unmanned swarm path planning device based on deep reinforcement learning, characterized in that: The device comprises: A UAV dynamics model construction module is used to construct a UAV dynamics model, in which the UAV speed, pitch angle, and yaw angle are constrained at each moment, and the minimum distance between the UAV and the nearest obstacle is constrained; The urban path planning model is used to obtain relevant parameters of the UAV cluster's mission to be performed, including the urban environment parameters for the mission, the initial position coordinates of each UAV, and the location of the mission target point. The optimized artificial potential field method is used to build a model based on the relevant parameters to obtain the urban path planning model; A reinforcement learning framework construction module is used to construct the state space, action space, state transition process and reward function in the reinforcement learning framework based on the UAV dynamics model and the urban path planning model; A dynamic path planning model obtaining module is used to perform reinforcement training on the deep Q neural network under the reinforcement learning framework to obtain a dynamic path planning model; The real-time path planning module is used to obtain the current perception data of the drone and use the dynamic path planning model to predict the drone's action at the next moment to achieve real-time planning of the drone's path.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images