Coverage path planning method based on DRL multi-UAV dam daily inspection

Abstract

The invention discloses a DRL-based multi-UAV dam daily inspection coverage path planning method, which comprises the following steps: firstly, according to a real environment of dam daily inspection, carrying out formalized definition on a multi-UAV coverage path planning problem, and representing the problem as an optimization problem which comprises two optimization targets; the energy consumption and the coverage rate of the multiple UAVs in the daily inspection process of the dam are respectively calculated. Then, based on the optimization problem, the invention proposes a DRL-based EA-MATD3 model for the coverage path planning of multiple UAVs, and aims to improve the coverage rate of multiple UAVs and reduce the energy consumption of multiple UAVs in the daily inspection of dams.

Description

Technical Field

[0001] The present invention relates to a coverage path planning method based on DRL multi-UAV daily inspection of dams, and belongs to the research field of artificial intelligence and smart water conservancy. Background Art

[0002] In recent years, UAV (Unmanned Aerial Vehicle, UAV for short) as an efficient autonomous mobile platform has played an increasingly important role in many fields such as agricultural monitoring, urban planning, disaster response and environmental protection. Especially when performing dam area coverage tasks, UAV swarms can complete large-scale coverage tasks faster and more efficiently than single UAVs due to their unique collaboration capabilities and flexibility. Dam area coverage tasks usually require UAV swarms to conduct comprehensive searches or monitoring in a specific area to ensure that each area is effectively covered and to avoid repeated coverage as much as possible to save resources. In this process, the challenges faced mainly include how to efficiently allocate the UAV's mission area, how to plan the flight path of each UAV, and how to coordinate the actions between UAVs to avoid resource waste. In addition, the energy limitations of UAVs and the dynamic changes in the environment also add additional complexity to the dam area coverage path planning problem.

[0003] In recent years, artificial intelligence technology has developed rapidly and been widely used. Deep learning and reinforcement learning have achieved great success in many fields. Deep learning has powerful perception capabilities, while reinforcement learning has decision-making capabilities. DRL (Deep Reinforcement Learning, DRL for short) combines the advantages of deep learning and reinforcement learning to provide solutions to the perception and decision-making problems of complex systems. DRL can effectively solve the problems of continuous state space and continuous action space. At present, DRL has been widely used in the field of multi-UAV coverage path planning. For example: Some studies use Actor-Critic neural networks with multi-resolution observations to train multi-UAV trajectory planning strategies, but all UAVs are considered to belong to the same entity and execute the same strategy. Some scholars have studied the trajectory of multi-UAV uplink data collection tasks based on the multi-agent DRL algorithm. Researchers provide an algorithm based on MADDPG (Multi-Agent Deep Deterministic Policy Gradient, MADDPG for short) for independently managing the trajectory of each UAV. Some studies apply MADDPG to trajectory planning in UAV motion control. Some scholars have designed an incentive mechanism to alleviate the problem of sparse rewards. In order to make better navigation and obstacle avoidance decisions, researchers proposed a trajectory planning method based on RNN (Recurrent Neural Network, RNN for short) architecture and temporal attention, ignoring multi-UAV formation obstacle avoidance. Some works express multi-UAV trajectory planning as a dynamic non-cooperative game and use the DRL algorithm based on ESN (Echo State Network, ESN for short) unit to solve the game. Some studies proposed a multi-UAV collision avoidance method based on two-stage reinforcement learning, but did not consider the strategy differences between individual UAVs and the constraints of multi-UAV cooperative formation trajectory planning. Some works use a decentralized DRL method in a two-dimensional grid-based map while ignoring complex environments, and consider multi-UAV trajectory planning in non-cooperative multi-UAV scenarios. In order to avoid invalid action exploration in DRL and improve convergence, researchers use Bayesian optimization to estimate the flight decisions of UAVs. For multi-UAV coverage trajectory planning, some works proposed the DCDQN (Distributed Cooperative DeepQ-Learning, DCDQN for short) algorithm to obtain UAV coverage trajectories online, ignoring the collaborative obstacle avoidance of multiple UAV formations.Some studies have proposed an effective reward mechanism based on DQN (Deep Q-Network, DQN for short) that comprehensively considers the changes in spacecraft orbits, thereby improving the success rate of orbital interception in uncertain environments. Some work uses artificial potential fields to guide anti-V multi-UAV formation navigation and obstacle avoidance, ignoring continuous actions and complex environments. However, the current multi-UAV coverage path planning algorithms encounter a key problem during daily dam inspections, that is, these algorithms do not take into account the coverage and energy consumption of multiple UAVs during daily dam inspections, resulting in the inefficiency of multi-UAV coverage path planning algorithms. Summary of the invention

[0004] Purpose of the invention: In view of the problems and shortcomings of the prior art, the present invention provides a coverage path planning method for daily dam inspection based on DRL multi-UAV. The use of UAV groups for dam inspection not only greatly improves the accuracy and efficiency of the inspection, but also reduces labor costs, which is of great significance for ensuring the safe operation of the dam.

[0005] Technical solution: A coverage path planning method based on DRL multi-UAV daily dam inspection includes the following steps:

[0006] (1) Problem formalization: The multi-UAV coverage path planning problem is defined as an optimization problem, which contains two optimization objectives, namely, the energy consumption and coverage rate of multiple UAVs. The optimization problem is specifically as follows:

[0007]

[0008] in, Represents the total coverage rate, that is, the ratio of the covered grid area to the entire area. is the repeated coverage rate, that is, the ratio of the area of ​​the same area repeatedly covered by different UAVs to the area of ​​the entire area, θ c represents the threshold of repeated coverage, E represents the total energy consumption of the UAV group, β1 and β2 represent weight parameters, T t Represents the total time step.

[0009] (2) Model building: The multi-UAV coverage path planning problem can be modeled as a Markov random process. The Markov random process describes the transition of the system state, and its characteristic is that the probability of the next state depends only on the current state. Based on the characteristics of the Markov random process, the present invention proposes the EA-MATD3 (Energy Aware Multi-Agent Twin Delayed Deep Deterministic Policy Gradient, referred to as EA-MATD3) model to solve the multi-UAV coverage path planning problem. Specifically, EA-MATD3 adopts a centralized training and decentralized execution mechanism based on MATD3 (Multi-Agent Twin Delayed DeepDeterministic Policy Gradient, MATD-3 for short), which fully considers the interactions between multiple UAVs while maintaining flexibility and scalability during execution; by introducing dual Q networks and policy smoothing technology, the stability of learning is significantly improved; stacked LSTM (Long Short-Term Memory, LSTM for short) is used to process time series data, enabling it to better predict and adapt to dynamic changes in the environment, and reduce repeated paths by analyzing time series data, thereby reducing energy consumption.

[0010] The EA-MATD3 model mainly consists of two parts, namely the MATD3 network and the stacked LSTM network. The specific contents are as follows:

[0011] In the MATD3 network, in order to improve the exploration performance and accelerate the convergence speed, Gaussian noise is introduced for action exploration. Specifically, a noise vector is generated by a deterministic strategy, which is then multiplied by a learnable parameter and added to the selected action. The Gaussian noise can be limited to a set range using a clipping function.

[0012] In order to better capture the experience gained by the UAV in its interaction with the environment during the inspection process, a two-layer LSTM network is used for better results. The position, speed, and direction of the UAV are used as the input of the first LSTM layer. Then, the output of the first layer and the reward and action obtained by the UAV in the previous time step are used as the input of the second LSTM layer. In addition, the memory function of LSTM enables it to store and utilize previous state information, which is conducive to solving the path planning problem under partially observable conditions. Finally, the output of LSTM is combined with the observation vector in MATD3 to obtain the state representation of the UAV.

[0013] The Q function update formula in the MATD3 algorithm is: in, represents the loss function of the UAVi critic network, Represents the expected value, tuple (s, a, r i , s′) represents the state, action, reward and state of the UAV at the next moment, represents the experience replay buffer, y i To update the target value of the critic network, Q(s,a;θ) represents the TD target of the MATD3 algorithm. - ) represents the Q value.

[0014] The MATD3 algorithm uses a target critic network to compute the TD target y t =r t +γQ′(s t+1 ,μ′(s t+1 ∣θ μ′ )|θ Q′ ) Among them, r t represents the reward value at time step t, γ represents the discount factor, Q′(s t+1 ,μ′(s t+1 ∣θ μ′ )|θ Q′ ) represents the target Q value. In order to improve the exploration performance and accelerate the convergence speed, Gaussian noise is introduced for action exploration. Specifically, through the deterministic strategy μ(s|θ μ ) generates a noise vector ∈, which is then multiplied by the learnable parameter σ and added to the selected action, i.e.: a = μ(s|θ μ )+∈,∈~N(0,σ 2 ). Therefore, the Q value with noise can be expressed as Q(s t ,a t ∣θ Q )+∈, where θ Q represents the parameters of the critic network, s t and a t denote the state and action at time step t respectively. Therefore, the noise target y t It can be expressed as: t =r t +γQ′(s t+1 ,μ′(s t+1 ∣θ μ′ )|θ Q′ )+clip(∈,-c,c). Where clip(∈,-c,c) represents the clipping function, ∈ represents Gaussian noise, and [-c,c] represents the range of Gaussian noise.

[0015] The actor network update rule in MATD3 is Among them, f i represents the actor network of the i-th UAV, oj represents the observation vector of the jth UAV, and N represents the number of UAVs. Denotes the actor network objective function of UAVi with respect to its parameters θ i The update is approximated using samples from the experience replay buffer, s j is the state observation value of the jth sample. Denotes the Q function of UAVi’s actor network with respect to its parameters θ i gradient.

[0016] In order to better capture the experience gained by the UAV in its interaction with the environment during the inspection process, a two-layer LSTM network is used for better results. The position, speed, and direction of the UAV are used as the input of the first LSTM layer. Then, the output of the first layer and the reward and action obtained by the UAV in the previous time step are used as the input of the second LSTM layer. In addition, the memory function of LSTM enables it to store and utilize previous state information, which is conducive to solving the path planning problem under partially observable conditions. The output of LSTM is represented as h t =LSTM(s t ,h t-1 ). Where s t represents the state at time t, h t Represents the output state of LSTM. Finally, the output of LSTM is combined with the observation vector in MATD3 to obtain the state of UAV as in represents the observation vector of UAVi at time t, Represents the LSTM output state of UAVi at time t.

[0017] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the coverage path planning method based on DRL multi-UAV daily dam inspection is implemented.

[0018] A computer-readable storage medium stores a computer program for executing the coverage path planning method for daily inspection of a dam based on a DRL multi-UAV as described above.

[0019] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0020] In the system modeling part, a regional dynamic weight model was constructed, which can dynamically adjust the coverage area weight according to the actual dam inspection scenario.

[0021] In the problem construction part, the UAV group action selection problem can be modeled as a decentralized partial observation Markov decision process, which is in line with actual application scenarios.

[0022] In the network model establishment part, the stacked LSTM network is integrated into the MATD3 algorithm and the EA-MATD3 method is proposed to maintain high coverage of the target area while keeping the energy consumption of the UAV group low. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a diagram of the implementation of the UAV group coverage path planning framework based on EA-MATD3 in a specific embodiment. DETAILED DESCRIPTION

[0024] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0025] In the process of daily inspection of the dam, the coverage path planning method for daily inspection of the dam based on DRL multi-UAV proposed in the present invention can be divided into two steps, namely, formal definition of the problem and establishment of the model.

[0026] First, the present invention defines the multi-UAV coverage path planning problem as an optimization problem, which includes two optimization objectives, namely, the energy consumption and coverage rate of the multiple UAVs. The optimization problem is specifically as follows:

[0027]

[0028] The meaning of each parameter is the same as above. Secondly, after formalizing the multi-UAV coverage path planning problem, the present invention finds that the problem is a Markov random process. The Markov random process describes the transition of the system state, and its characteristic is that the probability of the next state depends only on the current state. Based on the characteristics of the Markov random process, the present invention proposes the EA-MATD3 model to solve the multi-UAV coverage path planning problem in daily dam inspections. Specifically, EA-MATD3 adopts a centralized training and decentralized execution mechanism based on MATD3, which fully considers the interaction between multiple UAVs while maintaining flexibility and scalability during execution. By introducing dual Q networks and policy smoothing technology, the stability of learning is significantly improved. Then, EA-MATD3 uses a two-layer LSTM to process time series data, so that it can better predict and adapt to the dynamic changes of the environment, and reduce repeated paths by analyzing time series data, thereby reducing energy consumption.

[0029] like Figure 1As shown in the figure, the EA-MATD3 model mainly consists of two parts, namely the MATD3 network and the two-layer LSTM network.

[0030] In the MATD3 network, in order to improve the exploration performance and accelerate the convergence speed, Gaussian noise is introduced for action exploration. Specifically, a noise vector is generated by a deterministic strategy, which is then multiplied by the learnable parameters and added to the selected action. The Gaussian noise can be limited to a set range using a clipping function.

[0031] In order to better capture the experience gained by the UAV in its interaction with the environment during the inspection process, a two-layer LSTM network is used for better results. The position, speed, and direction of the UAV are used as the input of the first LSTM layer. Then, the output of the first layer and the reward and action obtained by the UAV in the previous time step are used as the input of the second LSTM layer. In addition, the memory function of LSTM enables it to store and utilize previous state information, which is conducive to solving the path planning problem under partially observable conditions. Finally, the output of LSTM is combined with the observation vector in MATD3 to obtain the state representation of the UAV.

[0032] The Q function update formula in the MATD3 algorithm is: in, represents the loss function of the UAVi critic network, Represents the expected value, tuple (s, a, r i , s′) represents the state, action, reward and state of the UAV at the next moment, represents the experience replay buffer, y i To update the target value of the critic network, Q(s,a;θ) represents the TD target of the MATD3 algorithm. - ) represents the Q value.

[0033] The MATD3 algorithm uses a target critic network to calculate the TD target y t =r t +γQ′(s t+1 ,μ′(s t+1 ∣θ μ′ )|θ Q′ ). In order to improve the exploration performance and accelerate the convergence speed, Gaussian noise is introduced for action exploration. Specifically, through the deterministic strategy μ(s|θ μ ) generates a noise vector ∈, which is then multiplied by the learnable parameter σ and added to the selected action, i.e.: a = μ(s|θ μ )+∈,∈~N(0,σ 2 ). Therefore, the Q value with noise can be expressed as Q(s t ,a t ∣θ Q)+∈, noise target y t It can be expressed as: t =r t +γQ′(s t+1 ,μ′(s t+1 ∣θ μ′ )|θ Q′ )+clip(∈,-c,c).

[0034] The actor network update rule in MATD3 is Among them, f i represents the actor network of the i-th UAV, o j represents the observation vector of the jth UAV, and N represents the number of UAVs. Denotes the actor network objective function of UAVi with respect to its parameters θ i The update is approximated using samples from the experience replay buffer, s j is the state observation value of the jth sample. Denotes the Q function of UAVi’s actor network with respect to its parameters θ i gradient.

[0035] In order to better capture the experience gained by the UAV in its interaction with the environment during the inspection process, a two-layer LSTM network is used for better results. The position, speed, and direction of the UAV are used as the input of the first LSTM layer. Then, the output of the first layer and the reward and action obtained by the UAV in the previous time step are used as the input of the second LSTM layer. In addition, the memory function of LSTM enables it to store and utilize previous state information, which is conducive to solving the path planning problem under partially observable conditions. The output of LSTM is represented as h t =LSTM(s t ,h t-1 ). Where s t represents the state at time t, h t Represents the output state of LSTM. Finally, the output of LSTM is combined with the observation vector in MATD3 to obtain the state of UAV as

[0036] Obviously, those skilled in the art should understand that the above-mentioned embodiments of the present invention, the coverage path planning method based on the daily inspection of the DRL multi-UAV dam, each step can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0037] Matters not covered by the present invention are known technologies.

[0038] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A coverage path planning method for daily inspection of dams based on DRL multi-UAV, characterized in that: The following steps are involved: (1) Problem formalization: The multi-UAV coverage path planning problem is defined as an optimization problem, which includes two optimization objectives: the energy consumption of multiple UAVs and the coverage rate; (2) Model establishment: The multi-UAV coverage path planning problem is modeled as a Markov random process. Based on the characteristics of the Markov random process, the EA-MATD model is proposed to solve the multi-UAV coverage path planning problem; EA-MATD3 adopts a centralized training and decentralized execution mechanism based on MATD3, introduces a dual Q network and policy smoothing technology, uses stacked LSTM to process time series data, and reduces repeated paths by analyzing time series data.

2. The coverage path planning method based on DRL multi-UAV daily dam inspection according to claim 1 is characterized in that: The EA-MATD3 model contains a MATD3 network and a stacked LSTM network: In the MATD3 network, Gaussian noise is introduced for action exploration. A noise vector is generated by a deterministic strategy, which is then multiplied by a learnable parameter and added to the selected action. The Gaussian noise is limited to a set range using a clipping function. The stacked LSTM network is a two-layer LSTM network that captures the experience gained by the UAV in interacting with the environment during the inspection process. The position, speed, and direction of the UAV are used as the input of the first LSTM layer. Then, the output of the first layer and the reward and action obtained by the UAV in the previous time step are used as the input of the second LSTM layer. Finally, the output of the stacked LSTM network is combined with the observation vector in the MATD3 network to obtain the state representation of the UAV.

3. The coverage path planning method based on DRL multi-UAV daily dam inspection according to claim 1 or 2 is characterized in that: The Q function update formula in the MATD3 network is: in, represents the loss function of the UAVi critic network, Represents the expected value, tuple (s, a, r i , s′) represents the state, action, reward and state of the UAV at the next moment, represents the experience replay buffer, y i To update the target value of the critic network, Q(s,a;θ) represents the TD target of the MATD3 algorithm. - ) represents the Q value; The MATD3 algorithm uses a target critic network to compute the TD target y t =r t +γQ′(s t+1 ,μ′(s t+1 ∣θ μ′ )|θ Q′ ) Among them, r t represents the reward value at time step t, γ represents the discount factor, Q′(s t+1 ,μ′(s t+1 ∣θ μ′ )|θ Q′ ) represents the target Q value; in order to improve the exploration performance and accelerate the convergence speed, Gaussian noise is introduced for action exploration. Specifically, through the deterministic strategy μ(s|θ μ ) generates a noise vector ∈, which is then multiplied by the learnable parameter σ and added to the selected action, i.e.: a = μ(s|θ μ )+∈,∈~N(0,σ 2 ); therefore, the Q value with noise is expressed as Q(s t ,a t ∣θ Q )+∈, where θ Q represents the parameters of the critic network, s t and a t Represent the state and action at time step t respectively; the noise target y t Expressed as: y t =r t +γQ′(s t+1 ,μ′(s r+1 ∣θ μ′ )|θ Q′ )+clip(∈,-c,c); where clip(∈,-c,c) represents the clipping function, ∈ represents Gaussian noise, and [-c,c] represents the range of Gaussian noise.

4. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the coverage path planning method based on DRL multi-UAV daily dam inspection is implemented as described in any one of claims 1-3.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the coverage path planning method based on DRL multi-UAV daily dam inspection as described in any one of claims 1-3.

Images