Unmanned cluster path planning method, device and equipment based on deep reinforcement learning

Abstract

The application relates to an unmanned cluster path planning method, device and equipment based on deep reinforcement learning. A city path planning model is obtained by modeling using an optimized artificial potential field method according to city environment parameters for executing a task, initial position coordinates of each unmanned aerial vehicle and a task target point position. Based on an unmanned aerial vehicle dynamics model and the city path planning model, a state space, an action space, a state transition process and a reward function in a reinforcement learning framework are defined, and a deep Q neural network is reinforced and trained in the framework to obtain a dynamic path planning model. The dynamic path planning model is used to predict the action of the unmanned aerial vehicle at the next moment according to current time perception data of the unmanned aerial vehicle, so that the real-time planning of the path of the unmanned aerial vehicle is realized. The method can accurately predict the action at the next moment according to the current environment state in a complex city scene.

Description

A method, apparatus, and equipment for unmanned swarm path planning based on deep reinforcement learning. Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) path planning technology, and in particular to a method, apparatus and equipment for unmanned swarm path planning based on deep reinforcement learning. Background Technology

[0002] With the rapid advancement of drone technology, its applications are becoming increasingly widespread. Especially in civilian sectors such as logistics, agriculture, and environmental protection, drones have demonstrated immense application potential. However, the widespread use of drones has also brought a series of challenges. First, drones need to navigate and avoid obstacles autonomously in complex and ever-changing environments, requiring them to possess advanced perception and intelligent decision-making capabilities. Second, energy efficiency is a critical issue, as limitations in battery technology severely restrict their flight duration. Furthermore, the safety of drones cannot be ignored; ensuring that drones do not pose a threat to people or property during missions is a problem that drone technology development must address.

[0003] Against this backdrop, intelligent planning for unmanned aerial vehicles (UAVs) has become a research hotspot. Intelligent planning not only requires UAVs to complete tasks efficiently but also needs to consider flight safety, rational energy utilization, and environmental adaptability. Traditional planning methods often rely on pre-defined rules or complex mathematical models, which tend to be ineffective in dynamic and uncertain environments. Therefore, an intelligent planning method capable of adaptive learning and dynamically adjusting planning strategies is of great significance for improving the autonomy and mission execution efficiency of UAVs. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, device, and equipment for unmanned swarm path planning based on deep reinforcement learning, which can dynamically adjust the planning strategy in real time according to the current complex environment, in order to address the above-mentioned technical problems.

[0005] A path planning method for unmanned swarms based on deep reinforcement learning, the method comprising:

[0006] A UAV dynamics model is constructed, in which the UAV speed, pitch angle and yaw angle are constrained at each time point, and the nearest distance between the UAV and the nearest obstacle is also constrained.

[0007] The relevant parameters of the task to be executed by the drone swarm are obtained. The relevant parameters include the urban environmental parameters of the task, the initial position coordinates of each drone and the position of the task target point. Based on the relevant parameters, the optimized artificial potential field method is used to model the urban path planning model.

[0008] Based on the aforementioned UAV dynamics model and urban path planning model, the state space, action space, state transition process, and reward function in the reinforcement learning framework are constructed.

[0009] Under the reinforcement learning framework, a deep Q-neural network is reinforced and trained to obtain a dynamic path planning model;

[0010] The system acquires the current perception data of the UAV and uses the dynamic path planning model to plan the UAV's actions for the next moment, thereby achieving real-time path planning for the UAV.

[0011] In one embodiment, an optimized artificial potential field method is used to model the urban path planning model based on the relevant parameters, resulting in:

[0012] The urban environmental parameters mentioned above include building-related data and urban spatial data;

[0013] The rectangular space of the city's three-dimensional model is determined based on the urban spatial data, and corresponding cuboid building models are generated in the city's three-dimensional model based on the building-related data.

[0014] Based on the aforementioned 3D urban model, by introducing factors including collision distance, angle, and distance factor, the gravitational force of the target point position on the UAV and the repulsive force of the building model on the UAV are calculated, thereby obtaining an optimized repulsive force function to guide the UAV path planning, thus obtaining the urban path planning model.

[0015] In one embodiment, the optimized repulsion function is expressed as:

[0016] ;

[0017] in, ;

[0018] In the above formula, This indicates the repulsive force pointing from the building model towards the drone. The direction of the gravitational force is indicated by the direction from the drone towards the target point. Represents the distance factor coefficient. Indicates the collision distance. This represents the distance from each building model to the straight line connecting the drone and the target point. This indicates the angle between the building model and the drone. and These represent the motion direction angles of the drone. Shift to the left and right Angle, The straight-line distance between the drone and the building model. , These indicate the positions of the drone and the building model, respectively.

[0019] In one embodiment, the state space in the reinforcement learning framework includes: the current environmental state, the target point state, and the drone state;

[0020] The environmental state refers to the position coordinates, half-length and half-width values ​​in the horizontal and vertical directions, and height of a single cuboid building model in the city's three-dimensional model.

[0021] The target point status is its position coordinates in the city's 3D model;

[0022] The drone's status refers to its current position coordinates, consumed energy, direction of movement, and probability of crashing within the city's 3D model.

[0023] In one embodiment, the state transition process in the reinforcement learning framework is represented as follows:

[0024] Update the drone's position coordinates based on its current direction of motion;

[0025] Calculate the current distance between the drone and the target point based on the updated position coordinates of the drone;

[0026] Based on the current distance and environmental conditions, the direction of movement at the next moment is obtained;

[0027] Update the energy consumption of drones;

[0028] Determine whether a drone has crashed based on its crash probability.

[0029] In one embodiment, the reward function in the reinforcement learning framework includes an objective function, an ability reward, and a collision penalty.

[0030] In one embodiment, the deep Q-neural network includes a Q-network and a Q-target network with identical network structures. Both the Q-network and the Q-target network are composed of four fully connected layers stacked with a ReLU activation function, as shown below:

[0031] ;

[0032] In the above formula, Indicates the state Take action below Expected returns Represents the trainable parameters of the network. This represents a function mapping implemented using a multilayer perceptron.

[0033] In one embodiment, when performing reinforcement training on a deep Q-neural network within the reinforcement learning framework:

[0034] Within the reinforcement learning framework, after the drone interacts with the environment, it performs actions and obtains observations, which include rewards and the next state.

[0035] The observations of multiple interactions between the drone and the environment within the reinforcement learning framework will be stored in the experience recycling pool.

[0036] A batch of training data is randomly sampled from the experience recycling pool. For each sampled observation, the Q network is used to calculate the predicted Q value for the corresponding action in the current state. The Q target network is used to calculate the maximum Q value of the next state as the target Q value.

[0037] The loss function is calculated based on the predicted Q value and the target Q value. The trainable parameters in the Q network are updated using the loss function until convergence. The trained Q network is then used as the dynamic path planning model.

[0038] An unmanned swarm path planning device based on deep reinforcement learning, the device comprising:

[0039] The UAV dynamics model construction module is used to construct a UAV dynamics model. In the UAV dynamics model, the speed, pitch angle and yaw angle of the UAV are constrained at each time point, and the shortest distance between the UAV and the nearest obstacle is also constrained.

[0040] An urban path planning model is used to obtain relevant parameters of the task to be performed by the UAV swarm. The relevant parameters include urban environmental parameters of the task, initial position coordinates of each UAV and position of the task target point. Based on the relevant parameters, the optimized artificial potential field method is used to model the urban path planning model.

[0041] The reinforcement learning framework definition module is used to construct the state space, action space, state transition process, and reward function in the reinforcement learning framework based on the UAV dynamics model and the urban path planning model.

[0042] The dynamic path planning model module is used to perform reinforcement training on the deep Q-neural network under the reinforcement learning framework to obtain the dynamic path planning model.

[0043] The real-time path planning module is used to acquire the current perception data of the UAV and use the dynamic path planning model to plan the UAV's actions for the next moment, so as to realize the real-time planning of the UAV's path.

[0044] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement steps in a deep reinforcement learning-based unmanned swarm path planning method.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a deep reinforcement learning-based unmanned swarm path planning method.

[0046] The aforementioned method, apparatus, and equipment for unmanned swarm path planning based on deep reinforcement learning, obtains an urban path planning model by modeling the urban environmental parameters of the task, the initial position coordinates of each UAV, and the position of the task target point using an optimized artificial potential field method. Based on the UAV dynamics model and the urban path planning model, the state space, action space, state transition process, and reward function in the reinforcement learning framework are defined. Within this framework, a deep Q-neural network is reinforced and trained to obtain a dynamic path planning model. This dynamic path planning model uses the UAV's current perception data to predict the UAV's actions in the next moment, achieving real-time path planning. This method can accurately predict the actions in the next moment in complex urban scenarios based on the current environmental state. Attached Figure Description

[0047] Figure 1 is a flowchart illustrating an unmanned swarm path planning method based on deep reinforcement learning in one embodiment;

[0048] Figure 2 is a schematic diagram of pitch angle and yaw angle in one embodiment;

[0049] Figure 3 is a schematic diagram of the reinforcement learning framework proposed in this method in one embodiment;

[0050] Figure 4 is a schematic diagram of the framework of a deep Q-neural network in one embodiment;

[0051] Figure 5 is a schematic diagram of the training score curve of the DQN model in a simulation experiment;

[0052] Figure 6 shows a schematic diagram of the visual results of urban path planning using this method for a single UAV in a simulation experiment. Figure 6(a) shows the two-dimensional visual results, and Figure 6(b) shows the three-dimensional visual results.

[0053] Figure 7 is a schematic diagram of the visual results of urban path planning for UAV swarms using this method in four scenarios in Table 3 of a simulation experiment. Figures 7(a), 7(b), 7(c), and 7(d) represent the results of the four scenarios, respectively.

[0054] Figure 8 is a structural block diagram of an unmanned swarm path planning device based on deep reinforcement learning in one embodiment;

[0055] Figure 9 is an internal structure diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] This application, as shown in Figure 1, provides a path planning method for unmanned swarms based on deep reinforcement learning, specifically including the following steps:

[0058] Step S100: Construct a UAV dynamics model. In the UAV dynamics model, constrain the UAV speed, pitch angle, and yaw angle at each time point, and constrain the closest distance between the UAV and the nearest obstacle.

[0059] Step S110: Obtain relevant parameters of the task to be performed by the UAV cluster. The relevant parameters include urban environmental parameters of the task, initial position coordinates of each UAV and position of the task target point. Based on the relevant parameters, the optimized artificial potential field method is used to model the urban path planning model.

[0060] Step S120: Based on the UAV dynamics model and the urban path planning model, construct the state space, action space, state transition process, and reward function in the reinforcement learning framework.

[0061] Step S130: Under the reinforcement learning framework, the deep Q-neural network is reinforced and trained to obtain a dynamic path planning model.

[0062] Step S140: Obtain the current perception data of the UAV and use the dynamic path planning model to plan the UAV's actions for the next moment in order to achieve real-time planning of the UAV's path.

[0063] The widespread adoption of drones has brought a series of challenges. First, drones need to navigate and avoid obstacles autonomously in complex and ever-changing environments, requiring them to possess advanced perception and intelligent decision-making capabilities. Second, energy efficiency is a critical issue, as limitations in battery technology severely restrict their endurance. Furthermore, drone safety is a significant concern; ensuring that drones do not pose a threat to humans or property during missions is a crucial problem that drone technology development must address. This application simplifies the challenges faced by drones in complex real-world scenarios, assuming the drone flies on a 3D map containing multiple obstacles. The drone's objective is to fly from a starting point to a destination, considering only obstacle avoidance and minimizing flight distance or time.

[0064] In step S100, the dynamics of the drones in the drone swarm are first modeled, and then... Defined as the first A drone, without considering wind and air resistance, is considered in the environment. The dynamics of flight can be modeled as follows:

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] ;

[0070] In the above formula, , , Let be the position coordinates of the m-th UAV at time t. For the flight speed of the drone, and Figure 2 shows the pitch and yaw angles of the UAV. and The horizontal and vertical components of the drone's acceleration. Indicates to Differentiate. Furthermore, the UAV must also satisfy the following constraints:

[0071] ;

[0072] ;

[0073] ;

[0074] In the above formula, , These are the minimum and maximum values ​​of the drone's flight speed, respectively.

[0075] Furthermore, in a complex 3D environment, multiple target points D = {D1, D2, ..., D...} are defined. m D M}, target point The coordinates are drones With the target point The distance is:

[0076] ;

[0077] To achieve the flight objective, the distance D between the UAV and the obstacle should satisfy... , To maintain a safe distance. Also, when the drone is at a safe distance from the target point... When the drone reaches the target point, it is said to have reached the target point. (Target threshold).

[0078] In step S110, an optimized artificial potential field method is used to model the urban path planning model based on relevant parameters, resulting in an urban path planning model. This model includes: urban environmental parameters such as building-related data and urban spatial data; determining the rectangular space of the urban 3D model based on the urban spatial data; and generating corresponding cuboid building models within the urban 3D model based on the building-related data. Based on the urban 3D model, by introducing collision distance, angle, and distance factor, the gravitational force of the target point position on the UAV and the repulsive force of the building models on the UAV are calculated, thereby obtaining an optimized repulsive force function to guide the UAV's path planning, thus completing the urban path planning model.

[0079] Specifically, a real-world example of urban 3D environment modeling is provided to better illustrate the process of real-world environment modeling. In practical applications of this method for real-world environment modeling, the modeling can be performed according to the actual situation.

[0080] In a real-world environment modeling example, consider a rectangular space of 500 units in length and width, and 100 units in height, containing 26 buildings. The coordinate parameters X, Y, and height parameter H are shown in the table below:

[0081] Table 1. Obstacle Location Coordinates

[0082]

[0083] Specifically, when generating the simulation environment, the system randomly assigns center coordinates to each building and checks whether the new building will overlap with existing buildings in the environment or exceed the boundaries.

[0084] In this embodiment, the artificial potential field method is also used to model the subsequent path planning based on the 3D city model. The artificial potential field method is a robot path planning algorithm proposed by Khatib. This algorithm treats the target and obstacles as objects that exert attractive and repulsive forces on the robot, respectively, and the robot moves along the resultant force of these forces. Therefore, in practical applications of UAV path planning, by treating the target point and buildings as objects exerting attractive and repulsive forces, respectively, environmental modeling for UAV path planning can be performed using the artificial potential field method.

[0085] Specifically, the main principle of the artificial potential field method is the gradient potential field method. In the algorithm, let the coordinates of the target point be... The current coordinates of the drone are The building's location coordinates are Therefore, the gravitational potential field experienced by the drone when it moves in the environment is:

[0086] ;

[0087] In the above formula, It is the gravitational gain constant. The straight-line distance between the target point and the drone;

[0088] The magnitude of gravity is the gravitational potential field. Negative gradient:

[0089] ;

[0090] Similarly, the repulsive potential field of the drone affected by the building can be set as follows:

[0091] when hour:

[0092] ;

[0093] when hour:

[0094] ;

[0095] In the above formula, Let the repulsive field constant be , The straight-line distance between the drone and the building. The distance at which the building affects the drone. The magnitude of the repulsive force experienced by the drone is the repulsive potential field. Negative gradient:

[0096] when hour:

[0097] ;

[0098] when hour: .

[0099] Furthermore, assume that in the environment where the drone is performing its mission, there are n buildings whose distance from the drone is less than the influence distance. The drone moves within the combined potential field formed by the target point and multiple buildings in the entire environment. The magnitude of the combined force acting on the drone is... for:

[0100] ;

[0101] When the path length of each step taken by the drone is set to l, the path coordinates of the drone at the next moment under the influence of the resultant force can be obtained as follows:

[0102] ;

[0103] However, when used in actual environmental modeling, the traditional artificial potential field method often gets stuck in local optima or the target is unreachable. Therefore, the above problems are solved by introducing collision distance to filter obstacles, introducing angle to define the range of influence of obstacles, introducing distance factor, and constructing virtual target points.

[0104] Furthermore, the optimized repulsion function is obtained. , is represented as:

[0105] ;

[0106] in, ;

[0107] In the above formula, This indicates the repulsive force pointing from the building model towards the drone. The direction of the gravitational force is indicated by the direction from the drone towards the target point. Represents the distance factor coefficient. Indicates the collision distance. This represents the distance from each building model to the straight line connecting the drone and the target point. This indicates the angle between the building model and the drone. and These represent the motion direction angles of the drone. Shift to the left and right Angle, The straight-line distance between the drone and the building model. , These indicate the positions of the drone and the building model, respectively.

[0108] In this embodiment, real-time path planning is achieved using deep reinforcement learning based on the constructed urban path planning model. Deep reinforcement learning, a technique combining deep learning and reinforcement learning, specifically the Deep Q-Network (DQN) algorithm, enables the UAV to autonomously learn the optimal flight strategy through end-to-end learning and interaction with the environment. This method not only adapts to environmental changes but also continuously improves the efficiency and safety of planning through learning.

[0109] In step S120, based on the urban path planning model constructed above, reinforcement learning is used for real-time path planning. The basic framework of reinforcement learning, as shown in Figure 3, includes state space, action space, state transition process, and reward function.

[0110] In this embodiment, the state space in the reinforcement learning framework includes: the current environment state, the target point state, and the UAV state. The environment state consists of the position coordinates of a single cuboid building model within the city's 3D model, its half-length and half-width values ​​in the horizontal and vertical directions, and its height. The target point state is its position coordinates within the city's 3D model. The UAV state consists of its current position coordinates, consumed energy, direction of movement, and crash probability within the city's 3D model.

[0111] Specifically, in any At any given time, the state space includes the environment state, the target state, and the UAV state. The environment state consists of a randomly generated group of square buildings, and the state vector for a single building is... ,in and These represent the horizontal and vertical coordinates of the building on the horizontal plane, both within the generated map area. Inside, and This represents the half-length and half-width values ​​of the building in the horizontal and vertical directions, respectively. This indicates the height of the building. The building's condition remains unchanged over time.

[0112] Specifically, the target state is the target's position coordinates on the map, and the state vector is... ,in These represent its position on the horizontal, vertical, and center axes, respectively.

[0113] Specifically, the state of a drone is relatively more complex, and its state vector is: The state vector changes over time. They respectively represent their in The horizontal, vertical, and moment coordinates, and and These respectively represent the drones in The energy already consumed and the direction of movement at any given moment. This indicates the probability of a drone crashing.

[0114] In this embodiment, the state transition process in the reinforcement learning framework is represented as follows: update the drone's position coordinates according to the current motion direction; calculate the current distance between the drone and the target point based on the updated position coordinates; obtain the next motion direction based on the current distance and environmental conditions; update the drone's energy consumption; and determine whether the drone will crash based on the crash probability.

[0115] In this embodiment, state transition Describes the state Take action below After transitioning to state The probability of [the state]. For the UAV model in this method, the state [is...]. It can be represented as Here This represents the coordinates of the drone in three-dimensional space. Indicates the location coordinates of the mission objective. For the energy consumption of drones, and These represent the drone's orientation and the environmental conditions, respectively. This represents the probability of a drone crash. The state transition process of a drone is quite complex and difficult to represent intuitively with a formula. The state transition process can be implemented using the following steps:

[0116] First, according to time and direction of motion Update drone coordinates:

[0117] ;

[0118] Then, calculate the current distance to the target. .

[0119] Next, according to With environmental conditions Adjust direction .

[0120] Furthermore, update the energy consumption of drones. .

[0121] Finally, based on the probability of crash Determine if the drone has crashed.

[0122] In this embodiment, the reward function Quantified from state Take action Transition to state The immediate reward. For the drone model, the reward function can be designed as follows:

[0123] ;

[0124] In the above formula, , and These represent the target reward, energy consumption reward, and collision penalty, respectively.

[0125] Specifically, target rewards The reward function for completing the task is expressed as:

[0126] ;

[0127] In the above formula, As a weighting factor, and These represent the initial distance and the current distance from the target, respectively. The target reward function is set to... It is inversely proportional, meaning the closer you are to the goal, the higher the reward.

[0128] Specifically, energy consumption rewards The reward function for energy consumption is expressed as:

[0129] ;

[0130] In the above formula, As a weighting factor, As initial energy, For energy consumption, This refers to the drone's remaining energy. The more energy remaining, the higher the reward value.

[0131] Specifically, collision penalty The penalty for a potential crash is represented as follows:

[0132] ;

[0133] In the above formula, As a weighting factor, This represents the probability of a crash.

[0134] In this embodiment, only the process of a single drone reaching the target is considered; therefore, the motion space only considers the motion of a single drone. For a single drone, its motion space can be represented as... .here Representing drones in Possible directions of movement on the three coordinate axes.

[0135] In the reinforcement learning training process, N drones are trained in each session, and each drone is trained to reach a single target point. After iterative reinforcement training, a single drone can complete the task of reaching a single target point with a relatively high probability.

[0136] In this embodiment, the core idea of ​​the DQN (Deep Q-Network) is to use a deep neural network to approximate the Q function, that is, the action-value function, which predicts the expected reward of taking a specific action in a given state. Its framework is shown in Figure 4.

[0137] In this embodiment, the DQN algorithm relies on a deep neural network to approximate the Q-function. This network structure should possess sufficient complexity to learn the dynamic characteristics of the environment. The deep Q-neural network includes a Q-network and a Q-target network with identical network structures. Both the Q-network and the Q-target network consist of four fully connected layers stacked with a ReLU activation function, as shown below:

[0138] ;

[0139] In the above formula, Indicates the state Take action below Expected returns Represents the trainable parameters of the network. This represents a function mapping implemented using a multilayer perceptron.

[0140] In this embodiment, during reinforcement training of the deep Q-neural network within the reinforcement learning framework: After the UAV interacts with the environment, it performs an action and obtains observation results, including rewards and the next state. The observation results from multiple interactions between the UAV and the environment within the reinforcement learning framework are stored in an experience recycling pool. A batch of training data is randomly sampled from the experience recycling pool. For each sampled observation result, the Q-network is used to calculate the predicted Q-value for the corresponding action in the current state. The Q-target network is used to calculate the maximum Q-value of the next state as the target Q-value. A loss function is calculated based on the predicted Q-value and the target Q-value. The trainable parameters in the Q-network are updated using the loss function until convergence. The trained Q-network is then used as a dynamic path planning model.

[0141] Specifically, first, a deep neural network is initialized as the estimator of the Q function; this network is the Q network shown in the diagram. Then, a Q-target network, identical to the Q network, is initialized, along with an experience recycling pool. Preferably, the neural network model has dimensions of 42x16x16x16x27 and a model size of 36KB. Subsequently, the UAV interacts with the environment, performs actions, and observes the results (reward and next state). These results are stored in the experience recycling pool; this process is called experience replay, and experience replay is a key technique for improving the sample efficiency of the DQN algorithm. In this embodiment, a circular buffer is used to store the most recent experience transitions. Each experience transition consists of a state, action, reward, next state, and completion flag, represented as:

[0142] ;

[0143] This mechanism allows the algorithm to break the temporal correlation between data through random sampling, thereby improving the stability of the training process.

[0144] Furthermore, during training, a batch of data is randomly sampled from the experience pool to reduce correlation between data points. For each sampled transition, the Q-target network is used to compute the maximum Q-value for the next state (i.e., the Q-value for greedily choosing the best action). Then, the difference between the current network's predicted Q-value and the target Q-value is calculated using the sampled data; this difference serves as the value of the loss function. The goal of training is to minimize the loss function. This function measures the difference between the predicted Q-value and the target Q-value. The loss function is... Represented as:

[0145] ;

[0146] In the above formula, It is a time discount factor. These are the target network parameters, which are... For rate-based soft updates of source network parameters The mean squared error (MSE) is used as the loss function, which calculates the difference between the predicted Q-value and the target Q-value. The target Q-value is given by the target Q-network, which is a delayed-update copy of the Q-network, and its parameters... by The software update is performed at a certain rate, and the update process is represented as follows:

[0147] ;

[0148] This soft update strategy helps stabilize the training process and prevents excessive differences between the target network and the source network.

[0149] Furthermore, in order to balance exploration and utilization, the following approach is adopted. The strategy, which is based on The probability of randomly selecting an action, in order to The probability of choosing the current optimal action is expressed as:

[0150] ;

[0151] This paper also provides pseudocode for training a deep Q-neural network, as shown in Table 2:

[0152]

[0153] In step S140, after training is completed, the trained Q-network is used as a dynamic path planning model. During real-time flight of the UAV, it is in a specific state at a certain moment. The current state is input into the dynamic path planning model to predict the action required to obtain the maximum Q value, and the action at the next moment is performed based on this prediction.

[0154] In this paper, the effectiveness of the proposed method is also demonstrated through simulation experiments. The simulation experiments address the urban path planning problem of UAV swarms, i.e., given the initial positions of multiple UAVs, building distribution, and target point, solving for a collision-free path within limited energy and maximum range. For this problem, the proposed method was tested in urban environments with varying building densities, and the experimental parameters are shown in Table 3.

[0155] Table 3 Parameter Setting Table

[0156]

[0157] Under the experimental settings shown in Table 3, an initial point, a target point, and a city map were randomly generated for testing.

[0158] In the simulation experiment, the DQN deep reinforcement learning model was first iteratively trained for 19 hours on a cluster of 15 drones in a city environment with a density of 10. The reward function first selects an action based on the Q-value, then changes the state based on the selected action to obtain a reward, calculates the total reward, and stores the interaction experience for each step. Successful states are positively rewarded to reinforce the experience and complete the state change; if the success rate is high, the difficulty is increased. The resulting score curve is shown in Figure 5.

[0159] As shown in Figure 5, the cluster path planning score gradually stabilizes after 600 training iterations as the number of training iterations increases. Since collisions and exceeding the maximum number of steps incur penalties of -200 and -200 respectively, while reaching the vicinity of the target point incurs a reward of +200, the final score fluctuating around 3000 after training indicates that the cluster can reach the target location in a dense urban environment.

[0160] Furthermore, the model was first tested with a single UAV under different building densities, and the results are shown in Figure 6. To further demonstrate the planning effect of the model, a verification experiment was conducted on the UAV swarm path of the above four scenarios in a simulation environment, and the results are shown in Figure 7. The step size of the experiment was 1 second, the target point was represented in red, the buildings were presented in the form of cubes, and the swarm path was represented by the formation of colored dots.

[0161] Figure 7 shows that, from different starting positions, the drone swarm can reach the vicinity of the target point while avoiding collisions with buildings. Furthermore, Figure 7 also reveals that upon reaching the vicinity of the target point, the swarm initially moves upwards for a short distance before flying towards the target point. This indicates that during training, the algorithm may have gotten stuck in a local optimum near the endpoint, failing to effectively explore other strategies, suggesting certain flaws in the design of the reward and penalty schemes.

[0162] The aforementioned unmanned swarm path planning method based on deep reinforcement learning constructs an urban path planning model incorporating attraction and repulsion by optimizing the artificial potential field method. It defines a reinforcement learning framework containing three-dimensional state, action space, and multi-objective reward function. A four-layer fully connected DQN network is trained using experience playback and target network soft update mechanisms, enabling the UAV to autonomously learn obstacle avoidance strategies in dynamic and complex environments. This achieves real-time path planning with the shortest distance and low energy consumption. Experimental simulation results demonstrate that this method can effectively solve the path planning problem in complex environments.

[0163] Specifically, by coupling the artificial potential field method with reinforcement learning, the urban environment is modeled as a combined force field of target attraction and building repulsion using the artificial potential field method. This provides a simulation environment with well-defined physical constraints for reinforcement learning. At the same time, the DQN algorithm dynamically optimizes the UAV's motion strategy in the attraction-repulsion field by learning environmental interaction data (such as state-action-reward sequences). This overcomes the limitation of the traditional potential field method being prone to local optima, increasing the obstacle avoidance success rate of UAVs in three-dimensional urban environments to over 80% (in high-density scenarios), and shortening the path distance by 15%-20% compared to traditional methods.

[0164] It should be understood that although the steps in the flowchart of Figure 1 are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in Figure 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0165] In one embodiment, as shown in Figure 8, a deep reinforcement learning-based unmanned swarm path planning device is provided, comprising: an unmanned aerial vehicle (UAV) dynamics model construction module 200, an urban path planning model 210, a reinforcement learning framework construction module 220, a dynamic path planning model acquisition module 230, and a real-time path planning module 240, wherein:

[0166] The UAV dynamics model construction module 200 is used to construct a UAV dynamics model. In the UAV dynamics model, the speed, pitch angle and yaw angle of the UAV are constrained at each time point, and the shortest distance between the UAV and the nearest obstacle is also constrained.

[0167] The urban path planning model 210 is used to obtain relevant parameters of the task to be performed by the UAV cluster. The relevant parameters include urban environmental parameters of the task, initial position coordinates of each UAV and position of the task target point. The optimized artificial potential field method is used to model the urban path planning model based on the relevant parameters.

[0168] The reinforcement learning framework construction module 220 is used to define the state space, action space, state transition process, and reward function in the reinforcement learning framework based on the UAV dynamics model and the urban path planning model.

[0169] The dynamic path planning model is obtained by module 230, which is used to perform reinforcement training on the deep Q-neural network under the reinforcement learning framework to obtain the dynamic path planning model.

[0170] The real-time path planning module 240 is used to acquire the current perception data of the UAV and use the dynamic path planning model to plan the UAV's actions in the next moment, so as to realize the real-time planning of the UAV's path.

[0171] Specific limitations regarding the deep reinforcement learning-based unmanned swarm path planning device can be found in the limitations of the deep reinforcement learning-based unmanned swarm path planning method described above, and will not be repeated here. Each module in the aforementioned deep reinforcement learning-based unmanned swarm path planning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0172] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in Figure 9. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a deep reinforcement learning-based unmanned swarm path planning method. The display screen of the computer device may be a liquid crystal display (LCD) or an e-ink display. The input device of the computer device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0173] Those skilled in the art will understand that the structure shown in Figure 9 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0174] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0175] A UAV dynamics model is constructed, in which the UAV speed, pitch angle and yaw angle are constrained at each time point, and the nearest distance between the UAV and the nearest obstacle is also constrained.

[0176] The relevant parameters of the task to be executed by the drone swarm are obtained. The relevant parameters include the urban environmental parameters of the task, the initial position coordinates of each drone and the position of the task target point. Based on the relevant parameters, the optimized artificial potential field method is used to model the urban path planning model.

[0177] Based on the aforementioned UAV dynamics model and urban path planning model, the state space, action space, state transition process, and reward function in the reinforcement learning framework are constructed.

[0178] Under the reinforcement learning framework, a deep Q-neural network is reinforced and trained to obtain a dynamic path planning model;

[0179] The system acquires the current perception data of the UAV and uses the dynamic path planning model to plan the UAV's actions for the next moment, thereby achieving real-time path planning for the UAV.

[0180] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0181] A UAV dynamics model is constructed, in which the UAV speed, pitch angle and yaw angle are constrained at each time point, and the nearest distance between the UAV and the nearest obstacle is also constrained.

[0182] The relevant parameters of the task to be executed by the drone swarm are obtained. The relevant parameters include the urban environmental parameters of the task, the initial position coordinates of each drone and the position of the task target point. Based on the relevant parameters, the optimized artificial potential field method is used to model the urban path planning model.

[0183] Based on the aforementioned UAV dynamics model and urban path planning model, the state space, action space, state transition process, and reward function in the reinforcement learning framework are constructed.

[0184] Under the reinforcement learning framework, a deep Q-neural network is reinforced and trained to obtain a dynamic path planning model;

[0185] The system acquires the current perception data of the UAV and uses the dynamic path planning model to plan the UAV's actions for the next moment, thereby achieving real-time path planning for the UAV.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A path planning method for unmanned swarms based on deep reinforcement learning, characterized in that, The method includes: constructing a UAV dynamics model, constraining the UAV's speed, pitch angle, and yaw angle at various time points, and constraining the closest distance between the UAV and the nearest obstacle; obtaining relevant parameters of the task to be executed by the UAV swarm, including urban environmental parameters, initial position coordinates of each UAV, and the position of the task target point; modeling using an optimized artificial potential field method based on the relevant parameters to obtain an urban path planning model, where the urban environmental parameters include building-related data and urban spatial data; determining the rectangular space of the urban 3D model based on the urban spatial data; generating corresponding cuboid building models in the urban 3D model based on the building-related data; and, based on the urban 3D model, calculating the gravitational force of the target point position on the UAV and the repulsive force of the building models on the UAV by introducing collision distance, angle, and distance factor, thereby obtaining an optimized repulsive force function to guide UAV path planning, thus obtaining the urban path planning model; and constructing a reinforcement chemical... The learning framework includes a state space, action space, state transition process, and reward function. The state space comprises the current environmental state, target point state, and drone state. The environmental state consists of the position coordinates, half-length and half-width values ​​in the horizontal and vertical directions, and height of a single cuboid building model within the city's 3D model. The target point state is its position coordinates within the city's 3D model. The drone state includes its current position coordinates, consumed energy, direction of movement, and crash probability within the city's 3D model. The state transition process involves updating the drone's position coordinates based on its current direction of movement, calculating the current distance to the target point based on the updated position coordinates, determining the next direction of movement based on the current distance and environmental conditions, updating the drone's energy consumption, and determining whether the drone will crash based on the crash probability. Within the reinforcement learning framework, a deep Q-neural network is reinforced and trained to obtain a dynamic path planning model. The drone's current perception data is acquired, and the dynamic path planning model is used to plan the drone's actions for the next moment, achieving real-time path planning.

2. The unmanned swarm path planning method based on deep reinforcement learning according to claim 1, characterized in that, The optimized repulsion function is expressed as: in, In the above formula, This indicates the repulsive force pointing from the building model towards the drone. The direction of the gravitational force is indicated by the direction from the drone towards the target point. Represents the distance factor coefficient. Indicates the collision distance. This represents the distance from each building model to the straight line connecting the drone and the target point. This indicates the angle between the building model and the drone. and These represent the motion direction angles of the drone. Shift to the left and right Angle, The straight-line distance between the drone and the building model. 、 These indicate the positions of the drone and the building model, respectively.

3. The unmanned swarm path planning method based on deep reinforcement learning according to claim 2, characterized in that, In the reinforcement learning framework, the reward function includes the objective function, energy reward, and collision penalty.

4. The unmanned swarm path planning method based on deep reinforcement learning according to claim 3, characterized in that, The deep Q-neural network includes a Q-network and a Q-target network with identical network structures. Both the Q-network and the Q-target network are composed of four fully connected layers stacked with a ReLU activation function, as follows: In the above formula, Indicates the state Take action below Expected returns Represents the trainable parameters of the network. This represents a function mapping implemented using a multilayer perceptron.

5. The unmanned swarm path planning method based on deep reinforcement learning according to claim 4, characterized in that, When training a deep Q-neural network under the reinforcement learning framework: after the UAV interacts with the environment, it performs an action and obtains observation results, including rewards and the next state; the observation results of multiple interactions between the UAV and the environment under the reinforcement learning framework are stored in the experience replay pool. A batch of training data is randomly sampled from the experience replay pool. For each sampled observation, the Q network is used to calculate the predicted Q value for the corresponding action in the current state. The Q target network is used to calculate the maximum Q value of the next state as the target Q value. The loss function is calculated based on the predicted Q value and the target Q value. The trainable parameters in the Q network are updated using the loss function until convergence. The trained Q network is then used as the dynamic path planning model.

6. A path planning device for unmanned swarms based on deep reinforcement learning, characterized in that, The device implements the unmanned swarm path planning method based on deep reinforcement learning as described in any one of claims 1-5. The device includes: an unmanned aerial vehicle (UAV) dynamics model construction module, used to construct an UAV dynamics model, in which the UAV speed, pitch angle, and yaw angle are constrained at various time points, and the shortest distance between the UAV and the nearest obstacle is also constrained; an urban path planning model, used to obtain relevant parameters of the UAV swarm's task to be performed, including urban environmental parameters for the task, the initial position coordinates of each UAV, and the position of the task target point, and to obtain the urban path planning model by modeling using an optimized artificial potential field method based on the relevant parameters; a reinforcement learning framework construction module, used to construct the state space, action space, state transition process, and reward function in the reinforcement learning framework based on the UAV dynamics model and the urban path planning model; a dynamic path planning model acquisition module, used to perform reinforcement training on a deep Q-neural network under the reinforcement learning framework to obtain a dynamic path planning model; and a real-time path planning module, used to acquire the current moment's perception data of the UAV and use the dynamic path planning model to plan the UAV's actions for the next moment, thereby achieving real-time path planning for the UAV.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

Images