Unmanned aerial vehicle cluster target monitoring path planning method based on space-time constraint multi-agent reinforcement learning
By building C-POMDP optimization objective function and Actor-Critic architecture, combined with CNN/LSTM network, the problem of incorporating constraints in drone cluster path planning is solved, and the optimal path planning that meets time and space constraints is realized, reducing communication and computing costs.
Patent Information
- Application Number
- CN202510658406.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-21
- Publication Date
- 2025-08-26
AI Technical Summary
The existing drone cluster path planning algorithm based on reinforcement learning has failed to effectively integrate various constraints, resulting in the planning path being unable to meet actual needs.
Using a method based on space-time constraint reinforcement learning of multi-agents, the C-POMDP optimization objective function is constructed, and the Actor-Critic architecture and CNN/LSTM network are used to extract the feature information of the drone cluster, map it into a time/space Critic network and an energy consumption Critic network, and reverse training is performed to meet the time/space constraints.
It realizes that time and space constraints are effectively integrated into the UAV cluster path planning, ensuring that the optimal path that meets various constraints is found, reducing communication and computing costs, and suitable for practical applications.
Smart Images

Figure CN120538527A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) path planning, and in particular to a UAV cluster target monitoring path planning method based on spatiotemporal constraint multi-agent reinforcement learning. Background Art
[0002] In recent years, drones (UAVs) have gained widespread application in a variety of fields, including disaster relief, aerial photography, and aerial base stations, due to their remarkable flexibility and cost-effectiveness. Compared to a single UAV, swarms of multiple drones offer advantages, as they can efficiently and concurrently handle multiple tasks. A particularly prominent challenge in swarming UAV applications is path planning. This process must comprehensively consider the obstacle avoidance requirements between drones, energy consumption, and specific mission constraints.
[0003] Path planning for drone swarms is based on a reinforcement learning algorithm. Each step in the drone's path planning considers its current state and possible actions, adjusting its action strategy based on rewards and costs. This adaptive capability enables the reinforcement learning algorithm to continuously adjust its strategy based on environmental changes, resulting in strong generalization capabilities.
[0004] However, existing reinforcement learning-based drone swarm path planning algorithms do not explicitly integrate various constraints into the overall reinforcement learning architecture. Instead, they simply add the cost of violating the constraints to the reward value of each training step. This makes it impossible to distinguish situations where the constraints are not met, and cannot ensure that the optimal solution that satisfies all constraints and achieves the goal is found.
[0005] It can be seen that providing a method to explicitly integrate various constraints into the overall reinforcement learning architecture for UAV cluster path planning, so as to find the optimal path planning that meets various constraints and achieves the goal is an urgent problem to be solved. Summary of the Invention
[0006] In view of the above analysis, the present invention aims to provide a drone cluster target monitoring path planning method based on spatiotemporal constraint multi-agent reinforcement learning, so as to solve the problem that the current drone cluster path planning algorithm based on reinforcement learning does not explicitly integrate various constraints into the overall reinforcement learning architecture, and cannot ensure the optimal path planning that meets various constraints and achieves the goal, resulting in the planned path failing to meet actual needs.
[0007] The present invention provides a method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning, the method comprising the following steps:
[0008] Obtain relevant characteristic information of the drone cluster and characteristic information of the environment in which the target monitoring task is to be performed; establish time and space constraints for the drone cluster path planning; and construct a constrained optimization problem with the goal of minimizing the total energy consumption of the drone cluster based on the obtained characteristic information and the time and space constraints.
[0009] Based on the acquired feature information, the constrained optimization problem is converted into the optimization objective function of the C-POMDP. A multi-agent reinforcement learning framework is constructed to map the optimization objective function of the C-POMDP to the actor network and critic network corresponding to each drone, as well as the critic network corresponding to the drone cluster.
[0010] Simulate the drone swarm and store the simulation results and related data in an experience replay pool; perform reverse training on the actor network and the critic network based on the data in the experience replay pool to obtain trained actor network and critic network;
[0011] The feature information obtained at the current moment is input into the trained Actor network corresponding to each UAV, and the path of each UAV is planned to complete the target monitoring task.
[0012] Furthermore, the time constraints of the UAV cluster path planning task include: the shortest revisit time constraint of the target monitoring task; the spatial constraints of the UAV cluster path planning task include: obstacle avoidance constraints between UAVs and obstacle avoidance constraints between UAVs.
[0013] Furthermore, the time cost at the current moment is obtained through the following time cost function:
[0014]
[0015] Among them, c1 represents the drone u i The time cost at the current moment, o i t Indicates drone u i The corresponding observation information obtained from the environment, Indicates drone u i The action at time t, Δ t Indicates the duration of each time slot.
[0016] Furthermore, the space cost at the current moment is obtained through the following space cost function:
[0017]
[0018] Among them, c2 represents the drone u i The space cost at the current moment, Indicates the distance from the drone u at the current moment i The nearest obstacles or other drones, Indicates that at the current moment, drone u i The distance to the nearest obstacle or other drone, D Min Indicates the minimum distance.
[0019] Furthermore, the reward at the current moment is obtained through the following reward function:
[0020]
[0021] Among them, s t represents the global state information of the drone cluster at time t, represents the joint action of the drone cluster at time t, Indicates drone u i The energy consumed in the current time slot.
[0022] Furthermore, based on the acquired feature information, the constrained optimization problem is transformed into the optimization objective function of C-POMDP according to the above-mentioned reward function, time cost function and space cost function; the expression of the optimization objective function of C-POMDP is:
[0023]
[0024] Among them, J R (π) represents the reward value function of global energy consumption, represents the time cost value function, Represents the space cost value function, T Max represents the maximum time specified, W represents the total number of time slots, γ t represents the discount factor at time t, and E[] represents the expected value.
[0025] Furthermore, mapping the optimization objective function of the C-POMDP to the Actor network and Critic network corresponding to each drone and the energy consumption Critic network corresponding to the drone cluster includes:
[0026] The reward value function of global energy consumption is mapped to the energy consumption critic network corresponding to the drone cluster, which is used to evaluate the total energy consumption of the drone cluster;
[0027] Map the time cost value function to the time critic network corresponding to each drone to meet the time constraints of each drone;
[0028] Map the spatial cost value function to the spatial critic network corresponding to each UAV to meet the spatial constraints of each UAV;
[0029] The drone's action strategy is mapped to the Actor network corresponding to each drone, which is used to generate the action of each drone at the next moment.
[0030] Furthermore, through the above multi-agent reinforcement learning framework, expression (2) is converted into expression (3):
[0031]
[0032] in, represents the stationary distribution of the initial state, P() represents the probability transfer function, π k represents the action strategy at the current moment, π k+1 Represents the action strategy updated at the next moment, represents the energy consumption advantage function, represents the time-constrained advantage function, represents the spatial constraint advantage function, γ represents the discount factor, argmax represents the independent variable value corresponding to the maximum value of the function, and D KL represents the KL divergence, and δ represents a constant.
[0033] Furthermore, the reverse training of the Actor network and the Critic network based on the data in the experience replay pool to obtain the trained Actor network and the Critic network includes:
[0034] Step S31: Input the samples in the experience replay pool into the time / space critic network corresponding to the drone and the energy consumption critic network of the drone cluster to obtain the value of the value function and advantage function corresponding to each critic network;
[0035] Step S32: Update the parameters of the Actor network corresponding to the drone based on the value of the advantage function corresponding to each Critic network and the reward value corresponding to the sample;
[0036] Step S33: Update the parameters of the time / space critic network corresponding to the drone based on the value of the value function corresponding to the time / space critic network and the cost value corresponding to the sample; update the parameters of the energy consumption critic network corresponding to the drone cluster based on the value of the value function corresponding to the energy consumption critic network and the reward value corresponding to the sample;
[0037] Step S34: If the training termination condition is met, the trained Actor network and Critic network are obtained; otherwise, return to step S31 to continue execution.
[0038] Furthermore, each drone obtains feature information at the current moment based on its own GPS or Beidou receiver, radar, and inertial measurement unit, and inputs the feature information into the trained Actor network corresponding to each drone. The Actor network outputs the action of each drone according to the action strategy, and each drone plans the path according to its own action to complete the target monitoring task.
[0039] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0040] 1. The present invention fully considers the actual situation when drone clusters perform target detection tasks and performs path planning. It divides the constraints into time constraints and space constraints, reflects the obstacle avoidance constraints between drones and between drones and obstacles, and the shortest revisit time constraints for target monitoring tasks, and provides a basis for subsequently finding the optimal path planning that meets various constraints and achieves the goal.
[0041] 2. The present invention constructs a multi-agent reinforcement learning framework, maps the time cost value function to the time critic network corresponding to each drone, and maps the space cost value function to the space critic network corresponding to each drone, thereby explicitly integrating the time / space constraints into the overall reinforcement learning architecture, providing a basis for ensuring that the optimal path planning that meets various constraints is found.
[0042] 3. During the training phase, the present invention uses a CNN network to extract feature information of the spatial dimension of the observation information, and uses an LSTM network to extract feature information of the temporal dimension of the observation information. Based on the feature information, the present invention evaluates whether the drone's actions at the current moment satisfy the time / space constraints. Based on the global state information, the present invention evaluates the energy consumption of the drone cluster at the current moment, thereby ensuring that the optimal path planning that satisfies various constraints and achieves the goal is found.
[0043] 4. During the execution phase of the present invention, each drone makes action decisions only through its own local information and Actor network, without accessing the status information of other drones or the global status information, which reduces communication and computing costs and is more suitable for practical applications.
[0044] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.
[0046] Figure 1 This is a flow chart of a method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to an embodiment of the present invention;
[0047] Figure 2 Schematic diagram of target monitoring path planning for a drone swarm based on spatiotemporal constrained multi-agent reinforcement learning according to an embodiment of the present invention;
[0048] Figure 3 This is an algorithm architecture diagram for drone cluster target monitoring path planning based on spatiotemporal constrained multi-agent reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION
[0049] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.
[0050] For a specific embodiment of the present invention, please refer to Figure 1-Figure 2 , discloses a method for drone cluster target monitoring path planning based on spatiotemporal constrained multi-agent reinforcement learning. The method comprises the following steps:
[0051] Step S1: Acquire relevant characteristic information of the UAV cluster and characteristic information of the environment in which the target monitoring task is to be performed; establish time constraints and space constraints for the UAV cluster path planning; and construct a constrained optimization problem with the goal of minimizing the total energy consumption of the UAV cluster based on the acquired characteristic information and the time constraints and space constraints.
[0052] Step S2: Based on the acquired feature information, the constrained optimization problem is converted into an optimization objective function of a C-POMDP; a multi-agent reinforcement learning framework is constructed to map the optimization objective function of the C-POMDP to the actor network and critic network corresponding to each drone and the critic network corresponding to the drone cluster;
[0053] Step S3: simulate the drone cluster and store the simulation results and related data in the experience replay pool; perform reverse training on the actor network and the critic network based on the data in the experience replay pool to obtain trained actor network and critic network;
[0054] Step S4: Input the feature information obtained at the current moment into the trained Actor network corresponding to each UAV, and perform path planning for each UAV to complete the target monitoring task.
[0055] Specifically, in step S1, the characteristic information related to the drone cluster includes: the number of drones in the drone cluster and the state parameters and motion parameters of each drone; wherein the drone state parameters include: the drone's position, speed, and heading; the drone's motion parameters include: the drone's acceleration and the angle between the acceleration direction and the horizontal direction. The characteristic information of the environment in which the target monitoring task is to be performed includes: the number of monitoring targets and the location of each monitoring target, the number of obstacles and the location of each obstacle, and the remaining time for each monitoring target to be monitored next time.
[0056] Specifically, each drone in the drone cluster is equipped with a GPS or Beidou receiver, a radar, and an inertial measurement unit (IMU). The GPS or Beidou receiver is used to obtain the position coordinates of each drone; the radar is used to obtain the position coordinates of each monitoring target, the position coordinates of each obstacle, the distance between each drone and each obstacle, the distance between each drone and each monitoring target, and the distance between each drone; the inertial measurement unit includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer is used to obtain the acceleration of each drone, the gyroscope is used to obtain the angular velocity of each drone, and the magnetometer is used to obtain the heading of each drone. The speed of the drone is obtained by integrating the acceleration of the drone; the angle between the acceleration direction of the drone and the horizontal direction is obtained by the acceleration and angular velocity of the drone; and the remaining time until the next monitoring of all monitoring targets is obtained by the target monitoring task to be performed.
[0057] It should be noted that the state parameters of a drone constitute its state space, and the action parameters of a drone constitute its action space. The set of state spaces of all drones constitutes the state space of the drone swarm, and the set of action spaces of all drones constitutes the action space of the drone swarm.
[0058] Specifically, when executing the target monitoring task, the state parameters and action parameters of each UAV and the characteristic information of the environment in which the target monitoring task is to be executed are obtained at certain time intervals.
[0059] Specifically, the execution range of the target monitoring task is represented as a rectangular area. Assuming that the total execution time of the target monitoring task is T, T is divided into W time slots at a fixed time interval, and the duration of each time slot is Where t={1,2,......W}. Assume that there are N U UAVs, N MMonitoring targets, N B obstacles; then the drone cluster is represented as: U={u i |i=1,2,......N U}, where u i Denotes the i-th UAV; all monitoring targets are expressed as: M = {m j |j=1,2,......N M}, where m j Denotes the jth monitoring target; all obstacles are represented as: B = {b k |k=1,2,......N B}, where b k Indicates the kth obstacle. At time t, obtain the drone u i The position coordinates are expressed as Monitoring target m j The position coordinates are expressed as Obstacle b k The position coordinates are expressed as The speed is expressed as: Acceleration and the angle between the acceleration direction and the horizontal direction Then at the next moment t+1, drone u i The speed is expressed as: UAV i Location coordinates Expressed as: Then the drone u at time t and time t+1 i Position changes It can be expressed as: UAV i In the current time slot, that is, from time t to time t+1, the energy consumed is expressed as: Where η represents the drone u i The efficiency coefficient of the drone is m, which represents the efficiency of the drone u. i quality.
[0060] It should be noted that for simplicity, the present invention does not consider changes in the altitude of the drones; that is, the altitude is uniformly represented by h. The actions that each drone can take at time t are represented by the drone's own action parameters, namely the drone's acceleration and the angle between the acceleration direction and the horizontal direction.
[0061] Furthermore, the time constraints of the UAV cluster path planning task include: the shortest revisit time constraint of the target monitoring task; the spatial constraints of the UAV cluster path planning task include: obstacle avoidance constraints between UAVs and obstacle avoidance constraints between UAVs.
[0062] It should be noted that the minimum revisit time for target monitoring missions means that the monitored target must be detected again within a specified time. Obstacle avoidance constraints between drones mean that the distance between drones must be greater than the minimum distance value, and obstacle avoidance constraints between drones and obstacles mean that the distance between drones and obstacles must be greater than the minimum distance value.
[0063] Furthermore, the expression of the constrained optimization problem is:
[0064]
[0065] Among them, the optimization goal is to minimize the total energy consumption of the drone cluster Indicates that at the current time t, the reconnaissance target m j The time interval since the last monitoring; Indicates m j To be detected again within the specified time, thus satisfying the time constraint; T Max Indicates the maximum time specified; function Represents the distance between the drone and the obstacle; function Indicates the distance between drones; Indicates that the distance between the drone and the obstacle must be greater than the minimum distance value D Min , Indicates that the distance between drones must be greater than the minimum distance value D Min , thus satisfying the space constraints.
[0066] It should be noted that the minimum distance is set according to actual needs.
[0067] Specifically, in step S2, the constrained Markov decision process (C-POMDP) consists of a tuple Definition. Among them, Represents the global state information set of the drone cluster, represents the joint action set of the drone cluster, represents the set of observation information obtained by the drone cluster from the environment, represents the set of probability transfer functions, γ∈[0,1) represents the discount factor, represents the set of rewards, represents a set of costs, and c represents a set of constraints that need to be satisfied corresponding to the costs.
[0068] Specifically, It includes the current position coordinates, movement direction, speed, acceleration of all drones in the drone cluster, the position coordinates of all monitored targets, the position coordinates of all obstacles, and the remaining time from the next monitoring of all monitored targets, expressed as S t express. Including the distance between each drone and all obstacles, the distance between each drone and all monitoring targets, the distance between all drones, the current position coordinates of all drones, the position coordinates of all monitoring targets, the position coordinates of all obstacles, the speed and acceleration of all drones; i t Indicates drone u i The corresponding observation information obtained from the environment. Indicates drone u i The action at time t, including u i Acceleration at time t The angle between the acceleration direction and the horizontal direction The UAV cluster interacts with the environment by taking joint actions and obtains the reward r at time t. t and cost c t , the cost includes time cost and space cost. The reward is used to measure the quality of the action in the current state. When the action consumes less energy, the reward is positive, otherwise the reward is zero or negative. The cost includes time cost and space cost, which represents the degree to which the action of the drone at the current moment meets the constraint conditions. When the constraint conditions are met, the cost is 0, otherwise, the cost is greater than 0. At the same time, the environment is based on Transition to a new state, where s t Represents the global state information of the drone cluster at time t. Set the discount factor according to the actual situation to weight the reward.
[0069] Specifically, the reward at the current moment is obtained through the following reward function:
[0070]
[0071] in, Indicates drone u i The energy consumed in the current time slot.
[0072] Specifically, the time cost at the current moment is obtained through the following time cost function:
[0073]
[0074] Among them, c1 represents the drone u iThe time cost at the current moment, o i t Indicates drone u i The corresponding observation information obtained from the environment, Indicates drone u i The action at time t, Δ t Indicates the duration of each time slot.
[0075] It should be noted that if any monitoring target is not monitored within the specified time, the drone closest to the monitoring target shall be held responsible. Therefore, the time cost function of the drone is defined as follows: if no drone monitors a monitoring target in the current time slot, the drone closest to the monitoring target will be penalized.
[0076] Specifically, the space cost at the current moment is obtained through the following space cost function:
[0077]
[0078] Among them, c2 represents the drone u i The space cost at the current moment, Indicates the distance from the drone u at the current moment i The nearest obstacles or other drones, Indicates that at the current moment, drone u i The distance to the nearest obstacle or other drone, D Min Indicates the minimum distance.
[0079] It should be noted that for spatial constraints, it is necessary to ensure that the distance between each drone and the nearest obstacle and other drones is greater than the minimum safety distance.
[0080] It should be noted that at the current time t, the drone u i Generate actions based on its action strategy Similarly, other drones in the drone cluster also generate actions according to their action strategies; the actions of all drones in the drone cluster at time t are multiplied together to obtain the joint action of the drone cluster at time t. Less than the minimum safety distance D Min When , a positive cost is generated, otherwise, the cost is zero. The optimization goal of the constrained optimization problem is not only to maximize the cumulative reward, but to maximize the cumulative reward on the basis that the cumulative cost satisfies the constraint conditions.
[0081] Preferably, by and Discretize and combine to form drone u iAn action strategy space is selected in the action strategy space. and get For example, Discretize into 3 levels, Discretized into 8 levels, the drone u i The dimension of the action space is 24, and one dimension is selected in the 24-dimensional action space and get
[0082] Furthermore, based on the acquired feature information, the constrained optimization problem is transformed into the optimization objective function of C-POMDP according to the above-mentioned reward function, time cost function and space cost function; the expression of the optimization objective function of C-POMDP is:
[0083]
[0084] Among them, J R (π) represents the reward value function of global energy consumption, represents the time cost value function, Represents the space cost value function, T Max represents the maximum time specified, W represents the total number of time slots, γ t represents the discount factor at time t, and E[] represents the expected value.
[0085] Further, if Figure 3 As shown, the Actor-Critic architecture is used to build a multi-agent reinforcement learning framework. The C-POMDP optimization objective function is mapped to the Actor network and Critic network corresponding to each drone and the energy consumption Critic network corresponding to the drone cluster, including:
[0086] The reward value function of global energy consumption is mapped to the energy consumption critic network corresponding to the drone cluster, which is used to evaluate the total energy consumption of the drone cluster;
[0087] Map the time cost value function to the time critic network corresponding to each drone to meet the time constraints of each drone;
[0088] Map the spatial cost value function to the spatial critic network corresponding to each UAV to meet the spatial constraints of each UAV;
[0089] The drone's action strategy is mapped to the Actor network corresponding to each drone, which is used to generate the action of each drone at the next moment.
[0090] Specifically, the energy consumption critic network, time critic network, spatial critic network and actor network all include an input layer, a hidden layer and an output layer; wherein the hidden layer includes multiple fully connected layers, and each fully connected layer is connected in series with a ReLU activation function.
[0091] It should be noted that the parameters of the above-mentioned Actor network and Critic network are initialized based on experience.
[0092] Furthermore, for each drone: a convolutional neural network (CNN) is constructed to extract feature information of the spatial dimension based on the input observation information obtained by the drone from the environment; and a long short-term memory (LSTM) network is constructed to extract feature information of the temporal dimension based on the input observation information obtained by the drone from the environment.
[0093] Specifically, the CNN network comprises an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The convolutional layer extracts local features, the pooling layer retains important features, and the fully connected layer fuses features. The LSTM network includes an input gate, a forget gate, and an output gate. Observational information acquired from the environment, such as the position coordinates, velocity, and acceleration of each drone over the past five moments, is fed into the LSTM network to generate temporal feature information for each drone.
[0094] Furthermore, through the above multi-agent reinforcement learning framework, expression (2) is converted into expression (3):
[0095]
[0096] in, represents the stationary distribution of the initial state, P() represents the probability transfer function, π k represents the action strategy at the current moment, π k+1 Represents the action strategy updated at the next moment, represents the energy consumption advantage function, represents the action-value function, represents the value function, represents the time-constrained advantage function, represents the spatial constraint advantage function, γ represents the discount factor, argmax represents the independent variable value corresponding to the maximum value of the function, and D KL represents the KL divergence, and δ represents a constant.
[0097] Specifically, the energy consumption advantage function is used to evaluate the average energy consumption action value under the current state, and its expression is:
[0098]
[0099] The expression of the energy consumption action value function is:
[0100]
[0101] The expression of the energy consumption value function is:
[0102]
[0103] The time constraint advantage function is used to evaluate the average time constraint action value under the current state, and its expression is:
[0104]
[0105] in, represents the time-constrained action-value function, represents the time-constrained value function;
[0106] The expression of the time-constrained action value function is:
[0107]
[0108] The expression of the time constraint value function is:
[0109]
[0110] The spatial constraint advantage function is used to evaluate the average spatial constraint action value under the current state, and its expression is:
[0111]
[0112] in, represents the spatially constrained action-value function, represents the spatial constraint value function;
[0113] The expression of the spatial constraint action value function is:
[0114]
[0115] The expression of the spatial constraint value function is:
[0116]
[0117] It should be noted that the inequality Limits each change in strategy from π k Update to π k+1 range.
[0118] Specifically, in step S3, the drone cluster is simulated in the environment, the feature information of the spatial dimension is input into the spatial critic network corresponding to each drone, the feature information of the time dimension is input into the time critic network corresponding to each drone, and the feature vector obtained by multiplying the feature information of the spatial dimension and the feature information of the time dimension is input into the actor network corresponding to each drone; the global state information of the drone cluster is input into the energy consumption critic network corresponding to the drone cluster.
[0119] It should be noted that the global state information of a drone swarm includes the current position coordinates, movement direction, speed, acceleration of all drones in the swarm, the position coordinates of all monitored targets, the position coordinates of all obstacles, and the remaining time until all monitored targets are next monitored. The spatial dimension feature information includes the current position coordinates, acceleration, and angular velocity of all drones in the swarm, the distance between each drone and all obstacles, the distance between each drone and all monitored targets, and the distance between all drones. The temporal dimension feature information includes the remaining time until each monitored target needs to be next monitored.
[0120] Specifically, the Actor network corresponding to the drone outputs the action of the drone at the current moment according to its current action strategy; the time / space Critic network corresponding to the drone outputs the value of the time / space constraint value function corresponding to the drone, which is used to evaluate the degree to which the action at the current moment in the current state meets the time / space constraint conditions; the energy consumption Critic network of the drone cluster outputs the value of the energy consumption value function corresponding to the drone cluster, which is used to evaluate the degree to which the total energy consumption required for the joint action of the drone cluster at the current moment in the current state meets the minimum energy consumption. The action of each drone at the current moment is input into the simulation environment for interaction to obtain the reward value and cost value corresponding to the current moment and the global state information and observation information corresponding to the next moment; the global state information s corresponding to the current moment of the drone cluster ... t , observation information Joint Action Reward value r t , cost value And the global state information s corresponding to the next moment t+1 , observation information Stored in the experience replay pool as a sample, it is represented as:
[0121] It can be understood that the samples corresponding to each moment in a period of time obtained by the above method are stored in the experience replay pool.
[0122] Furthermore, the reverse training of the Actor network and the Critic network based on the data in the experience replay pool to obtain the trained Actor network and the Critic network includes:
[0123] Step S31: Input the samples in the experience replay pool into the time / space critic network corresponding to the drone and the energy consumption critic network of the drone cluster to obtain the value of the value function and advantage function corresponding to each critic network;
[0124] Step S32: Update the parameters of the Actor network corresponding to the drone based on the value of the advantage function corresponding to each Critic network and the reward value corresponding to the sample;
[0125] Step S33: Update the parameters of the time / space critic network corresponding to the drone based on the value of the value function corresponding to the time / space critic network and the cost value corresponding to the sample; update the parameters of the energy consumption critic network corresponding to the drone cluster based on the value of the value function corresponding to the energy consumption critic network and the reward value corresponding to the sample;
[0126] Step S34: If the training termination condition is met, the trained Actor network and Critic network are obtained; otherwise, return to step S31 to continue execution.
[0127] Specifically, in step S31, the s in the sample t 、 Input the energy consumption critic network of the drone cluster and obtain the energy consumption value function and energy consumption advantage function corresponding to the energy consumption critic network; Input the time / space critic network corresponding to the drone and obtain the values of the time / space constraint value function and time / space constraint advantage function corresponding to the time / space critic network.
[0128] Specifically, in step S32, the update gradient of the Actor network corresponding to the drone is calculated using the following formula:
[0129]
[0130] Among them, λ T,i ,λ D,i is the Lagrange multiplier, represents the action strategy at the current moment, Indicates the action strategy to be updated at the next moment; clip() is used to limit the amplitude of the action strategy update to ensure that χ(θ i ) is kept within the range of (1-,1+); ∈ is a hyperparameter.
[0131] Preferably, ∈ is 0.2.
[0132] Update the parameters of the Actor network corresponding to the drone using the following formula:
[0133]
[0134] Among them, α θ is the learning rate of the Actor network, Indicates drone u i The parameters of the corresponding Actor network at the current time k.
[0135] Specifically, in step S33, the update gradient of the time / space critic network corresponding to the drone is calculated using the following formula:
[0136]
[0137] Update the parameters of the time / space critic network corresponding to the drone using the following formula:
[0138]
[0139] α φ is the learning rate of each Critic network.
[0140] The updated gradient of the energy consumption critic network corresponding to the drone cluster is calculated using the following formula:
[0141]
[0142] Update the parameters of the energy consumption critic network corresponding to the drone cluster using the following formula:
[0143]
[0144] Specifically, in step S34, the training termination conditions include: reaching a preset number of iterations, that is, the training batch size; or, the update gradients corresponding to the Actor network and each Critic network are less than a threshold.
[0145] It should be noted that, given the complexity of the monitoring mission environment and the frequent interactions between drones and their environment, training the Actor network directly on drones based on real-world scene data is prohibitively expensive. Therefore, this application employs a ground-based simulation system to simulate various real-world environments for network training, thereby optimizing the action strategies of drone swarms. The goal of the Actor network is to learn the optimal action strategy, maximizing the cumulative reward while satisfying the cumulative cost constraints. This is done by minimizing the value of the energy-consuming Critic network's value function. The goal of the time / space critic network is to learn the time / space constrained action value function and the time / space constrained value function. The input of the time / space constrained action value function is the observation information and action of the drone, which indicates that after taking the current action at the current moment, the estimated time / space cumulative reward of the next moment action will be generated according to the current strategy; the input of the time / space constrained value function is the observation information of the drone, which indicates that under the current moment state, the estimated time / space cumulative reward of the next moment action will be generated according to the current strategy; the time / space constrained value function is subtracted from the time / space constrained action value function to obtain the time / space constrained advantage function, which is used to evaluate the time / space constrained average action value at the current moment state, that is, the quality of the current action strategy in satisfying the time / space constraints. The goal of the Energy Critic Network is to learn an energy-action-value function and an energy-value function. The Energy-Action-Value Function takes the observations and joint actions of the drone swarm as input, representing the estimated cumulative energy reward for the next joint action, generated according to the current strategy, after the current joint action is taken at the current moment. The Energy-Value Function takes the observations of the drone swarm as input, representing the estimated cumulative energy reward for the next joint action, generated according to the current strategy, at the current moment. Subtracting the Energy-Value Function from the Energy-Action-Value Function yields an Energy-Advantage Function, which is used to evaluate the average energy-value action value at the current moment—that is, the quality of the current action strategy in minimizing energy consumption. When all drone behaviors meet temporal and spatial constraints, the system relies solely on the Energy Critic Network to train the Actor Network to minimize energy consumption. When a drone violates temporal and spatial constraints, the system introduces a Temporal and Spatial Critic Network to work in conjunction with the Energy Critic Network to train the Actor Network, ensuring that the actions generated by the Actor Network meet temporal and spatial constraints and minimize energy consumption.
[0146] Specifically, in step S4, each drone obtains feature information at the current moment based on its own GPS or Beidou receiver, radar, and inertial measurement unit, and inputs the feature information into the trained Actor network corresponding to each drone. The Actor network outputs the action of each drone according to the action strategy, and each drone plans the path according to its own action to complete the target monitoring task.
[0147] It's important to note that after training, in the actual application phase, the Actor network corresponding to each drone is deployed on the drone's GPU and carried into the air with the drone. When performing path planning to complete the target monitoring mission, each drone can only access its own local information (i.e., feature information acquired by its own sensors) and cannot access the status of other drones or global information. Therefore, each drone relies solely on its own trained Actor network to perform forward reasoning tasks, obtain predicted actions, and execute them.
[0148] Compared with the existing technology, the UAV cluster target monitoring path planning method based on spatiotemporal constraint multi-agent reinforcement learning provided by the present invention has the following beneficial effects:
[0149] 1. The present invention fully considers the actual situation when drone clusters perform target detection tasks and performs path planning. It divides the constraints into time constraints and space constraints, reflects the obstacle avoidance constraints between drones and between drones and obstacles, and the shortest revisit time constraints for target monitoring tasks, and provides a basis for subsequently finding the optimal path planning that meets various constraints and achieves the goal.
[0150] 2. The present invention constructs a multi-agent reinforcement learning framework, maps the time cost value function to the time critic network corresponding to each drone, and maps the space cost value function to the space critic network corresponding to each drone, thereby explicitly integrating the time / space constraints into the overall reinforcement learning architecture, providing a basis for ensuring that the optimal path planning that meets various constraints is found.
[0151] 3. During the training phase, the present invention uses a CNN network to extract feature information of the spatial dimension of the observation information, and uses an LSTM network to extract feature information of the temporal dimension of the observation information. Based on the feature information, the present invention evaluates whether the drone's actions at the current moment satisfy the time / space constraints. Based on the global state information, the present invention evaluates the energy consumption of the drone cluster at the current moment, thereby ensuring that the optimal path planning that satisfies various constraints and achieves the goal is found.
[0152] 4. During the execution phase of the present invention, each drone makes action decisions only through its own local information and Actor network, without accessing the status information of other drones or the global status information, which reduces communication and computing costs and is more suitable for practical applications.
[0153] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.
[0154] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.
Claims
1. A method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning, characterized by: The method comprises the following steps: Obtain relevant characteristic information of the drone cluster and characteristic information of the environment in which the target monitoring task is to be performed; establish time and space constraints for the drone cluster path planning; and construct a constrained optimization problem with the goal of minimizing the total energy consumption of the drone cluster based on the obtained characteristic information and the time and space constraints. Based on the acquired feature information, the constrained optimization problem is converted into the optimization objective function of the C-POMDP. A multi-agent reinforcement learning framework is constructed to map the optimization objective function of the C-POMDP to the actor network and critic network corresponding to each drone, as well as the critic network corresponding to the drone cluster. Simulate the drone swarm and store the simulation results and related data in an experience replay pool; perform reverse training on the actor network and the critic network based on the data in the experience replay pool to obtain trained actor network and critic network; The feature information obtained at the current moment is input into the trained Actor network corresponding to each UAV, and the path of each UAV is planned to complete the target monitoring task.
2. The method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 1 is characterized in that: The time constraints of the UAV cluster path planning task include: the shortest revisit time constraint of the target monitoring task; the spatial constraints of the UAV cluster path planning task include: obstacle avoidance constraints between UAVs and obstacle avoidance constraints between UAVs.
3. The method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 1 is characterized in that: The time cost at the current moment is obtained through the following time cost function: Among them, c1 represents the drone u i The time cost at the current moment, Indicates drone u i The corresponding observation information obtained from the environment, Indicates drone u i The action at time t, Δ t Indicates the duration of each time slot.
4. The method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 3 is characterized in that: The space cost at the current moment is obtained through the following space cost function: Among them, c2 represents the drone u i The space cost at the current moment, Indicates the obstacle or other drone closest to the drone ui at the current moment. Indicates that at the current moment, drone u i The distance to the nearest obstacle or other drone, D Min Indicates the minimum distance.
5. The method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 4 is characterized in that: The reward at the current moment is obtained through the following reward function: Among them, s t represents the global state information of the drone cluster at time t, represents the joint action of the drone cluster at time t, Indicates drone u i The energy consumed in the current time slot.
6. The method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 5 is characterized in that: Based on the acquired feature information, the constrained optimization problem is transformed into the optimization objective function of C-POMDP according to the above-mentioned reward function, time cost function and space cost function. The expression of the optimization objective function of C-POMDP is: Among them, J R (π) represents the reward value function of global energy consumption, represents the time cost value function, Represents the space cost value function, T Max represents the maximum time specified, W represents the total number of time slots, γ t represents the discount factor at time t, and E[] represents the expected value.
7. The method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 6 is characterized in that: Mapping the optimization objective function of the C-POMDP to the Actor network and Critic network corresponding to each drone and the energy consumption Critic network corresponding to the drone cluster includes: The reward value function of global energy consumption is mapped to the energy consumption critic network corresponding to the drone cluster, which is used to evaluate the total energy consumption of the drone cluster; Map the time cost value function to the time critic network corresponding to each drone to meet the time constraints of each drone; Map the spatial cost value function to the spatial critic network corresponding to each UAV to meet the spatial constraints of each UAV; The drone's action strategy is mapped to the Actor network corresponding to each drone, which is used to generate the action of each drone at the next moment.
8. The method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 7 is characterized in that: Through the above multi-agent reinforcement learning framework, expression (2) is converted into expression (3): in, represents the stationary distribution of the initial state, P() represents the probability transfer function, π k represents the action strategy at the current moment, π k+1 Represents the action strategy updated at the next moment, represents the energy consumption advantage function, represents the time-constrained advantage function, represents the spatial constraint advantage function, γ represents the discount factor, argmax represents the independent variable value corresponding to the maximum value of the function, and D KL represents the KL divergence, and δ represents a constant.
9. The method for UAV swarm target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 1 is characterized in that: The reverse training of the Actor network and the Critic network based on the data in the experience replay pool to obtain the trained Actor network and the Critic network includes: Step S31: Input the samples in the experience replay pool into the time / space critic network corresponding to the drone and the energy consumption critic network of the drone cluster to obtain the value of the value function and advantage function corresponding to each critic network; Step S32: Update the parameters of the Actor network corresponding to the drone based on the value of the advantage function corresponding to each Critic network and the reward value corresponding to the sample; Step S33: Update the parameters of the time / space critic network corresponding to the drone based on the value of the value function corresponding to the time / space critic network and the cost value corresponding to the sample; update the parameters of the energy consumption critic network corresponding to the drone cluster based on the value of the value function corresponding to the energy consumption critic network and the reward value corresponding to the sample; Step S34: If the training termination condition is met, the trained Actor network and Critic network are obtained; otherwise, return to step S31 to continue execution.
10. The method for UAV cluster target monitoring path planning based on spatiotemporal constraint multi-agent reinforcement learning according to claim 1 is characterized in that: Each drone obtains feature information at the current moment based on its own GPS or Beidou receiver, radar, and inertial measurement unit, and inputs the feature information into the trained Actor network corresponding to each drone. The Actor network outputs the action of each drone according to the action strategy. Each drone plans the path according to its own action to complete the target monitoring task.
Citation Information
Cited By
Unmanned aerial vehicle cluster dynamic path planning method and device based on non-electrical variable control
CN121677738A
Unmanned aerial vehicle cluster dynamic path planning method and device based on non-electric variable control
CN121677738B
