A method for autonomous visual navigation and obstacle avoidance of quadrotor drones based on deep reinforcement learning

Abstract

This invention relates to the field of autonomous navigation technology for quadrotor UAVs, specifically to an autonomous visual navigation obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning. The purpose of this invention is to address the problem of low accuracy in existing autonomous visual navigation obstacle avoidance systems for UAVs. The process involves: constructing a simulation environment for autonomous visual navigation obstacle avoidance for quadrotor UAVs; selecting and initializing the dynamic model of the quadrotor UAV; setting the initial position and target position of the UAV; setting the state space, observation space, and action space within the simulation environment; constructing a neural network model of the autonomous visual navigation obstacle avoidance decision-making agent; obtaining the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent; and performing obstacle avoidance control on the quadrotor UAV under control based on the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology for unmanned aerial vehicles (UAVs), specifically to an autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning. Background Technology

[0002] Autonomous visual navigation and obstacle avoidance technology for quadrotor UAVs is a crucial element for performing complex tasks in unknown environments, and it is widely used in urban logistics, pesticide spraying and crop growth monitoring, post-disaster rescue and relief, and battlefield reconnaissance. Based on the task execution process, autonomous navigation and obstacle avoidance technology can be divided into two types: multi-task cascaded and end-to-end. The multi-task cascaded method typically includes tasks such as perception, mapping, planning, decision-making, and control. Each step is performed sequentially, with the results of previous steps serving as input for subsequent steps, resulting in high interpretability. This method, as a traditional UAV navigation algorithm, has been extensively studied. For example, patent CN114355980B proposes an autonomous visual navigation and obstacle avoidance method for UAVs based on ESDF maps and the DDPG deep reinforcement learning algorithm. This method first acquires the UAV's pose through sensors and constructs an ESDF map by combining image information. Subsequently, a path search algorithm is used to generate a series of discrete path points on the map. Then, the UAV's pose vector, the ESDF map matrix, and the discrete path point vectors are input in parallel into the DDPG network, outputting commands to control the motor speed, thus achieving autonomous navigation and obstacle avoidance.

[0003] However, in multi-task cascaded methods, noise and errors from preceding tasks gradually propagate and accumulate in subsequent tasks, ultimately leading to a significant decline in flight performance. Furthermore, mapping tasks require real-time map generation and storage, consuming substantial computational and storage resources, making it difficult for UAVs to efficiently cope with uncertainties in the environment, such as dynamic obstacles and changing lighting conditions. End-to-end deep learning methods, on the other hand, directly map perceived data to control command outputs, avoiding multiple intermediate processing steps and effectively reducing error accumulation. Moreover, deep learning methods have a stronger advantage in processing multimodal perceived data. Through multi-branch feature extraction networks, the system can simultaneously process real-time data from camera images, IMU status data, GPS positioning information, and other sensors such as ultrasound. However, existing end-to-end methods often do not fully consider considerable issues in the actual flight environment, such as incomplete or interfered perceived information. Furthermore, deep reinforcement learning-based algorithms often require extensive trial and error to collect experience, resulting in low experience collection efficiency and slow model convergence. Therefore, there is an urgent need to study an end-to-end autonomous navigation algorithm that can effectively cope with the considerable challenges brought by the environment in UAV navigation missions and achieve safe and efficient autonomous navigation and obstacle avoidance in unknown environments. Summary of the Invention

[0004] The purpose of this invention is to address the problem that existing deep reinforcement learning-based algorithms often require extensive trial and error to collect experience, resulting in low experience collection efficiency, slow model convergence, and low accuracy of autonomous visual navigation and obstacle avoidance for UAVs. Therefore, this invention proposes an autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning.

[0005] The specific process of an autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning is as follows:

[0006] Step 1: Construct an autonomous visual navigation and obstacle avoidance simulation environment for a quadcopter UAV, select and initialize the dynamic model of the quadcopter UAV, set the starting position of the UAV and set the target position;

[0007] Step 2: Set up the state space, observation space, and action space in the simulation environment;

[0008] Step 3: Construct a neural network model for an autonomous visual navigation obstacle avoidance decision-making agent;

[0009] Step 4: Train the neural network model of the autonomous visual navigation obstacle avoidance decision-making agent to obtain the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent.

[0010] Step 5: Perform obstacle avoidance control on the quadcopter drone under control based on the neural network model of the trained autonomous visual navigation obstacle avoidance decision-making agent.

[0011] The beneficial effects of this invention are as follows:

[0012] (1) This invention is the first autonomous navigation obstacle avoidance decision-maker in the field of UAV autonomous navigation to solve some environmental observability problems based on privileged learning. The decision-maker takes depth images and other self-perceived information as input and directly outputs flight control commands for the UAV, thereby completing the autonomous navigation obstacle avoidance task, demonstrating its advanced nature.

[0013] (2) The privileged information-based learning framework and multi-agent experience collection strategy adopted in this invention can effectively handle perceptual noise and accelerate the convergence of deep reinforcement learning models, significantly improving the overall performance of the algorithm.

[0014] (3) Compared with non-end-to-end autonomous navigation obstacle avoidance algorithms, the method of the present invention has significant advantages in terms of algorithm efficiency, environmental adaptability and tracking success rate, and improves the accuracy of UAV autonomous visual navigation obstacle avoidance.

[0015] (4) In addition, the autonomous navigation obstacle avoidance algorithm proposed in this invention is highly compatible with the perception information that the UAV can obtain in actual flight and the input of the underlying controller in terms of state and action space design, thereby enhancing the UAV's ability to migrate from the simulation environment to the real environment and having good transferability. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0017] Figure 2 The following are schematic diagrams of the simulation environment for autonomous visual navigation and obstacle avoidance of UAVs: (a): top view of the training environment; (b): top view of the random environment; (c): schematic diagram of the UAV flying in the environment.

[0018] Figure 3 The images show the effects of adding noise to depth images: (a): Depth image acquired by a depth camera; (b): Image after adding salt and pepper noise to the depth image; (c): Image after adding Gaussian noise to the image in (b); (d): Image after adding motion blur to the image in (c).

[0019] Figure 4 This invention relates to the neural network model DRPL, which is based on deep reinforcement learning for autonomous visual navigation and obstacle avoidance decision-making. In the following diagrams: 1@80×100, the number of channels is before the @ symbol, and the dimension of each channel is after the @ symbol; 8@80×100, the number of channels is before the @ symbol, and the dimension of each channel is after the @ symbol; 8@40×50, the number of channels is before the @ symbol, and the dimension of each channel is after the @ symbol; 16@40×50, the number of channels is before the @ symbol, and the dimension of each channel is after the @ symbol; 16@20×25, the number of channels is before the @ symbol, and the dimension of each channel is after the @ symbol; 25@20×25, the number of channels is before the @ symbol, and the dimension of each channel is after the @ symbol; 25@10×12, the number of channels is before the @ symbol, and the dimension of each channel is after the @ symbol.

[0020] Figure 5 A schematic diagram of parameter updates for the neural network model DRPL of the present invention, based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision-making agent, under POMDP modeling;

[0021] Figure 6 The following are model training curves from a performance comparison experiment of autonomous visual navigation obstacle avoidance algorithms: (a) SR curves of the neural network model DRPL based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision agent of this invention and the ordinary TD3 algorithm during the model training process; (b) AER curves of the neural network model DRPL based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision agent of this invention and the ordinary TD3 algorithm during the model training process.

[0022] Figure 7The flight trajectories for the performance comparison experiment of autonomous visual navigation obstacle avoidance algorithms are shown below: (a) Flight trajectory of the UAV deploying the neural network model DRPL based on the deep reinforcement learning autonomous visual navigation obstacle avoidance decision-making agent of this invention in the training environment; (b) Flight trajectory of the UAV deploying the ordinary TD3 algorithm in the training environment; (c) Flight trajectory of the UAV deploying EGO-Planner-v2 in the training environment; (d) Flight trajectory of the UAV deploying the neural network model DRPL based on the deep reinforcement learning autonomous visual navigation obstacle avoidance decision-making agent of this invention in a random environment; (e) Flight trajectory of the UAV deploying the ordinary TD3 algorithm in a random environment; (f) Flight trajectory of the UAV deploying EGO-Planner-v2 in a random environment.

[0023] Figure 8 The graphs show the model training and evaluation curves of the autonomous visual navigation obstacle avoidance algorithm for ablation experiments targeting privileged information and multi-agent exploration strategies. (a): The SR curves of the neural network model DRPL based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision agent proposed in this invention and the algorithms for removing privileged information and multi-agent exploration strategies respectively during the model training process; (b): The AER curves of the neural network model DRPL based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision agent proposed in this invention and the algorithms for removing privileged information and multi-agent exploration strategies respectively during the model training process. SR represents the success rate, and AER represents the average round reward.

[0024] Figure 9 The graphs show the training and evaluation curves of the autonomous visual navigation obstacle avoidance algorithm for the state space and action space ablation experiments. (a): SR curves during the training process of the neural network model DRPL based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision-making agent with 4-dimensional action space and the neural network model DRPL based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision-making agent with 3-dimensional action space proposed in this invention; (b): AER curves during the training process of the neural network model DRPL based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision-making agent with 4-dimensional action space and the neural network model DRPL based on deep reinforcement learning autonomous visual navigation obstacle avoidance decision-making agent with 3-dimensional action space proposed in this invention. Detailed Implementation

[0025] Specific Implementation Method 1: The specific process of this implementation method for an autonomous visual navigation and obstacle avoidance method for a quadcopter UAV based on deep reinforcement learning is as follows:

[0026] To address the aforementioned issues, this invention proposes an end-to-end autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning, named DPRL (Distributed Privileged Reinforcement Learning). This method employs a policy-based deep reinforcement learning algorithm to design an agent network structure suitable for autonomous navigation and obstacle avoidance tasks of quadrotor UAVs, a continuous state and action space, and an efficient reward function. Simultaneously, based on privileged learning, actor networks and critic networks with different inputs are designed to effectively address some observability issues during flight. Furthermore, a multi-agent exploration strategy is employed to accelerate model convergence, ultimately resulting in a strategy capable of directly generating flight control commands based on noisy visual perception data and the model's own state information.

[0027] This invention proposes an autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning, comprising the following steps:

[0028] Step 1: Construct an autonomous visual navigation and obstacle avoidance simulation environment for a quadcopter UAV, select and initialize the dynamic model of the quadcopter UAV, set the starting position of the UAV and set the target position;

[0029] Step 2: Set up the state space, observation space, and action space in the simulation environment;

[0030] Step 3: Construct a neural network model for an autonomous visual navigation obstacle avoidance decision-making agent;

[0031] Step 4: Train the neural network model of the autonomous visual navigation obstacle avoidance decision-making agent to obtain the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent.

[0032] Step 5: Perform obstacle avoidance control on the quadcopter drone under control based on the neural network model of the trained autonomous visual navigation obstacle avoidance decision-making agent.

[0033] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that, in step 1, a simulation environment for autonomous visual navigation and obstacle avoidance of a quadcopter UAV is constructed, the dynamic model of the quadcopter UAV is selected and initialized, and the starting position and target position of the UAV are set; the specific process is as follows:

[0034] Step 11: Use Unreal Engine 4 to create a training environment containing obstacles (dense obstacles);

[0035] Use the 3D modeling software Unreal Engine 4 to create a training environment containing obstacles (dense obstacles);

[0036] Use the 3D modeling software Unreal Engine 4 to create a model evaluation environment with randomly set obstacle positions. The obstacles in the model evaluation environment are the same as those in the training environment.

[0037] In the training environment, 70 cylindrical obstacles with a radius of 2.5m and a height of 15m are placed inside a circle with the origin as the center and a radius of 60m.

[0038] In the model evaluation environment, 60 obstacles identical to those in the training environment were randomly placed inside a circle with the origin as the center and a radius of 60m.

[0039] Step 12: Integrate the AirSim plugin into the Unreal Engine 4 simulation environment, import the quadcopter drone dynamics model (Multirotor dynamics model) provided by AirSim, and set the dynamic parameters of the quadcopter drone dynamics model;

[0040] The parameters of the UAV dynamics model are the duration of the UAV flight maneuver, the UAV's velocity range along the x-axis, the UAV's velocity range along the y-axis, the UAV's velocity range along the z-axis, and the UAV's yaw rate range.

[0041] The x-axis, y-axis, and z-axis form the coordinate system of the UAV body. The direction of the UAV's head is the x-axis, the z-axis is perpendicular to the x-axis and the y-axis is perpendicular to the xz plane.

[0042] Step 13: Set the initial and target positions of the drone for each training round, define the effective flight space range, and set the conditions for successful navigation and obstacle avoidance flight as well as the failure conditions.

[0043] Reinforcement learning training is conducted in rounds, with each round having a termination condition. When the termination condition is met, the training round ends, the simulation environment is reset, and the next training round begins.

[0044] The other steps and parameters are the same as in Specific Implementation Method 1.

[0045] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that step 2 involves setting up a state space, observation space, and action space within the simulation environment; the specific process is as follows:

[0046] Step 21: Model the autonomous navigation task of the UAV as a partially observable Markov decision process, represented as:<s,o,a,r,p,γ> ;

[0047] Where s is the state, o is the observation, a is the action, r is the reward, p is the state transition probability, and γ is the discount factor;

[0048] Considering the partially observable environmental phenomena caused by factors such as the high-speed movement of the UAV, sensor noise, and environmental wind disturbance, this invention models the autonomous navigation task of the UAV as a partially observable Markov Decision Process (POMDP), denoted as:<s,o,a,r,p,γ> ;

[0049] In the autonomous navigation mission of UAV, state s represents the state information of UAV in the current environment (state information includes two parts: visual perception information and UAV's own state information; visual perception information refers to the depth image without noise, and UAV's own state information refers to the UAV's three-axis position, three-axis velocity, the angle between the current flight direction and the target direction, and the current yaw rate without noise).

[0050] Observation o represents the estimate of state s obtained by the UAV through its onboard sensors;

[0051] Action 'a' is the flight control command executed by the UAV at each time step;

[0052] The reward r is used to evaluate the value of a drone performing a certain action in a certain state to transition to the next state;

[0053] The state transition probability p defines the probability of taking a certain action in the current state to transition to the next specific state. This probability is determined by the UAV dynamics and is affected by factors such as control input, environmental conditions, and system noise.

[0054] The discount factor γ determines the importance of the reward at a future time step relative to the reward at the current time step. The closer the discount factor is to 1, the more important the future reward is to the current reward.

[0055] This invention is based on the idea of ​​privileged learning and adopts the Learning Using Privileged Information (LUPI) framework, which inputs state information as privileged information into the commentator network;

[0056] Step 22: Set up the state space to obtain the current state from the simulation environment. The state space in this invention is a continuous state space; the specific process is as follows:

[0057] Status information includes visual perception information and the drone's own status information;

[0058] Visual perception information is depth images acquired by the drone's onboard depth camera;

[0059] The depth image acquired by the UAV's onboard depth camera is input into the feature extraction network, and the feature extraction network outputs a visual perception information vector S1.

[0060] The feature extraction network consists of a first convolutional block, a second convolutional block, a third convolutional block, and a global average pooling layer.

[0061] Each of the first, second, and third convolutional blocks contains, in sequence, a convolutional layer, a batch normalization (BN) layer, a max pooling layer, and an activation function ReLU.

[0062] The drone's own status information includes the current three-axis distance from the drone to the target point [d] x ,d y ,d z The current three-axis speed of the drone [v] x ,v y ,v z ] Yaw angle deviation between the current flight direction and the target direction and the current yaw rate of drones

[0063] The drone's own state information is an 8-dimensional drone own state information vector S2;

[0064] The current three-axis speed of the drone [v] x ,v y ,v z [This refers to the three-axis velocity in the UAV's body coordinate system;]

[0065] The current three-axis distance from the UAV to the target point [d] x ,d y ,d z [This refers to the distance across the three axes in the world coordinate system.]

[0066] The visual perception information vector S1 and the UAV's own state information vector S2 are concatenated to obtain the state information vector S = [S1, S2].

[0067] Step 23: Set up the motion space to control the flight of the quadcopter drone in the simulation environment. The motion space in this invention is a continuous motion space.

[0068] The action space is the drone's flight control command A, which includes the drone's current three-axis velocities [v]. x ,v y ,v z and yaw rate These continuous actions directly control the movement of the drone in the simulated environment;

[0069] The current three-axis speed of the drone [v]x ,v y ,v z [This refers to the three-axis velocity in the UAV's body coordinate system;]

[0070] The action space is 4-dimensional;

[0071] Step 24: Set up the observation space to obtain the currently observed information from the simulation environment. The observation space in this invention is a continuous observation space.

[0072] The observation information is obtained by adding salt-and-pepper noise, Gaussian noise, and motion blur to the state information to simulate the state that the UAV can observe in some observable environments.

[0073] The observation space has the same dimension as the state space, and can be represented as O∈R. 33 Within the LUPI framework, observational information serves as input to the actor network;

[0074] The specific process is as follows:

[0075] Salt-and-pepper noise, Gaussian noise, and motion blur are added sequentially to the depth image information of the UAV to fully simulate the noise and interference that may be encountered during flight in the real environment.

[0076] In the UAV's own state information, the three-axis distance [d] from the current UAV to the target point is... x ,d y ,d z Add Gaussian noise to simulate positioning noise and other sensor noise;

[0077] In the drone's own state information, the current three-axis velocity [v] of the drone is... x ,v y ,v z Add Gaussian noise to simulate positioning noise and other sensor noise;

[0078] In the UAV's own status information, the yaw angle deviation between the current flight direction and the target direction is... Gaussian noise is added to simulate positioning noise and other sensor noise;

[0079] In the drone's own state information, the current drone yaw rate is... Gaussian noise is added to simulate positioning noise and other sensor noise;

[0080] Adding noise to the drone's state information in a simulation environment to simulate some observable phenomena of the environment (partial observable phenomena of the environment is a technical term in the field of reinforcement learning, used to describe the inability of an agent to obtain complete and accurate state information when interacting with the environment, in order to simulate complex situations in the real environment).

[0081] The drone status information includes visual perception information and the drone's own status information;

[0082] Visual perception information refers to the depth images acquired by the onboard depth camera of the UAV;

[0083] The drone's own status information refers to the current three-axis distance from the drone to the target point [d] x ,d y ,d z The current three-axis speed of the drone [v] x ,v y ,v z ] Yaw angle deviation between the current flight direction and the target direction and the current yaw rate of drones

[0084] Noise includes salt-and-pepper noise, Gaussian noise, and motion blur added to visual perception information, as well as Gaussian noise added to the drone's own state information.

[0085] The specific process is as follows:

[0086] Salt-and-pepper noise, Gaussian noise, and motion blur are added sequentially to the depth image information of the UAV to fully simulate the noise and interference that may be encountered during flight in the real environment.

[0087] The three-axis distance [d] between the current UAV and the target point in the UAV's own state information. x ,d y ,d z Gaussian noise was added to the three-axis position of the drone to simulate positioning noise and other sensor noise;

[0088] The current three-axis velocity [v] of the drone in its own state information. x ,v y ,v z Add Gaussian noise to simulate positioning noise and other sensor noise;

[0089] The yaw angle deviation between the current flight direction and the target direction in the UAV's own status information. Gaussian noise is added to simulate positioning noise and other sensor noise;

[0090] The current yaw rate of the drone in its own state information Gaussian noise is added to simulate positioning noise and other sensor noise.

[0091] Other steps and parameters are the same as in specific implementation method one or two.

[0092] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that step 3 involves constructing a neural network model for an autonomous visual navigation obstacle avoidance decision-making agent; the specific process is as follows:

[0093] The neural network model of the autonomous visual navigation obstacle avoidance decision-making agent consists of a feature extraction network and a decision network.

[0094] The feature extraction network consists of a first convolutional block, a second convolutional block, a third convolutional block, and a global average pooling layer.

[0095] Each of the first, second, and third convolutional blocks sequentially contains a convolutional layer, a batch normalization (BN) layer, a max pooling layer, and an activation function ReLU.

[0096] The decision network consists of a first multilayer perceptron, a second multilayer perceptron, and uses Leaky ReLU as the activation function.

[0097] The activation function uses Leaky ReLU to prevent the gradient vanishing problem from causing the actor network to continuously output boundary values.

[0098] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0099] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that step 4 involves training a neural network model of an autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning, thereby obtaining a trained neural network model of an autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning. The specific process is as follows:

[0100] To overcome noise and interference affecting perceived information, this invention designs an asymmetric input actor and critic network model architecture. During the model training phase, the actor network uses noisy input, while the critic network uses accurate input, thereby enhancing the deep reinforcement learning model's ability to perceive states.

[0101] Define a reinforcement learning reward function, adopt a multi-agent exploration strategy, and train a neural network model based on a deep reinforcement learning autonomous visual navigation obstacle avoidance decision agent. This enables the model to continuously acquire training samples and optimize network parameters through interaction with the environment, thereby improving the success rate of autonomous visual navigation obstacle avoidance.

[0102] Includes the following steps:

[0103] Step 31: Define the reward function. This invention adopts a combination of sparse reward and discrete reward based on the current time step and the current round state.

[0104] Reward functions include sparse rewards and continuous rewards;

[0105] Sparse rewards are as follows: If the current time step round ends, the reward includes a positive reward for successfully reaching the target point, and a negative reward for colliding or exceeding the flight range; the specific process is as follows:

[0106] When the drone enters the target's reach range, it receives a positive reward of +10; if the drone collides with an obstacle or flies out of the environmental boundary, it receives a penalty of -5.

[0107] If the current training round ends due to successfully reaching the target point, the agent receives a positive reward; if the current training round ends due to the drone flying out of the designated flight range or colliding with an obstacle, the agent receives a negative reward. Positive and negative rewards cannot be received simultaneously, as only one termination condition can be triggered.

[0108] If the current time step round has not ended, the reward includes a positive reward for getting close to the target, a negative reward for the target distance error, and a negative reward for the collision probability calculated based on the distance to the obstacle surface;

[0109] Furthermore, the agent receives a series of rewards at each time step, including a positive reward for distance difference, a penalty for distance error, and a penalty for obstacle approach. The positive reward for distance difference is used to evaluate how close the agent is to the target at the current time step compared to the previous time step, encouraging the drone to move closer to the target. The penalty for distance error is calculated based on the distance from the drone to the starting point and the target, ensuring that the drone travels along the shortest straight path and improving the final navigation accuracy. The penalty for obstacle approach is calculated based on the difference between the shortest distance and the safe distance between the drone and the obstacle at each time step, helping to prevent the drone from approaching the obstacle.

[0110] The expression for continuous rewards is as follows:

[0111]

[0112] in,

[0113] r e Indicates a reward item; d g d represents the distance from the starting point to the target. t and d t-1 These represent the distances between the drone and the target at the current and previous time steps, respectively.

[0114] p p Indicates penalty item 1; z and z g Let d represent the z-axis coordinates of the UAV's current time step and the target position, respectively. l This represents the distance from the drone's current position to the straight line connecting the starting point and the target; the clip function is used to limit the data amplitude.

[0115] po Indicates penalty item 2; d o Indicates the shortest distance from the drone to the surface of the obstacle; d c Let d be the collision distance. o Less than d c It was assumed that a collision had occurred; d s It is a safe distance, when d o Less than d s There is a risk of collision for mission-oriented drones;

[0116] r represents the continuous reward value; η r It is r e scaling factor, η p It is p p scaling factor, η o It is p o The scaling factor;

[0117] Step 32: Set the parameters of the experience replay pool and the deep reinforcement learning neural network; the specific process is as follows:

[0118] The experience replay pool parameter is the size parameter of the experience replay pool;

[0119] The hyperparameters of deep reinforcement learning neural networks include network structure, number of neurons, learning rate, discount factor, learning rate, learning start step, experience replay pool size, batch size, training frequency, action noise standard deviation, number of training environments, total number of training steps, and maximum number of steps in each round.

[0120] Step 33: Set an action selection strategy to complete experience collection; the specific process is as follows:

[0121] First, an ε-greedy strategy is used to select either a random action or an action output by the actor network (the action output by the actor network is the observed input to the actor network, and the actor network outputs the action).

[0122] An ε-greedy strategy is adopted, which selects random actions with a low probability or actions output by the actor network with a high probability, thereby balancing exploration and exploitation.

[0123] Then, the drone performs a selected random action or an action output by the actor network and interacts with the environment to generate a new state after the transition, a new observation after the transition, and a reward value returned by the environment.

[0124] The current state, current observation, selected action, new state after transition, new observation after transition, and reward value returned by the environment are considered. t ,o t ,a t ,s t+1 ,o t+1 ,r t ​>Store it into the experience replay pool to complete one experience collection;

[0125] s t Indicates the current state, o t Indicates the current observation, a t Indicates the currently selected action, s t+1 Indicates the new state after the transition, o t+1 Indicates the new observation after the transfer, r t This represents the reward value returned by the environment;

[0126] An action selection strategy is set to collect experience. To balance exploration and utilization, this invention uses an ε-greedy algorithm for action selection. ε is a probability value close to 0. At each time step, a random number is generated. If the random number is greater than or equal to ε, the action with the highest value in the current state is selected according to the reinforcement learning algorithm; otherwise, an action is randomly selected from the action space. The ε-greedy algorithm dynamically adjusts the reinforcement learning's exploration of new strategies and utilization of existing strategies as the training process progresses. Subsequently, the action is executed in the simulation environment, the state at the next time step is collected, and the reward value is calculated. The current state, observation, action, reward, and the state and observation at the next time step are stored in the experience replay pool until the experience replay pool is full or the simulation ends.

[0127] Step 34: Set up a multi-agent exploration strategy to complete experience collection; the specific process is as follows:

[0128] M processes run M AirSim environments in parallel, and place a drone in each environment to perform the experience collection in step 33, thereby completing one experience collection.

[0129] M drones all input observations into the same decision network, obtain corresponding actions, execute actions to interact with the environment, and generate a new state after the transition, a new observation after the transition, and a reward value returned by the environment; the network then stores the current state, current observation, selected action, new state after the transition, new observation after the transition, and reward value returned by the environment. t ,o t ,a t ,s t+1 ,o t+1 ,r t > Store them in the same experience replay pool to complete one experience collection;

[0130] Step 35: Train the neural network model of the autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning, and obtain the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning.

[0131] ​The parameter update mechanism of the neural network model DRPL (Twin Delayed Deep Deterministic Policy Gradient, TD3) of the present invention, based on the deep reinforcement learning-based autonomous visual navigation obstacle avoidance decision-making agent, under the task modeling of POMDP, is as follows: Figure 5 As shown.

[0132] Specifically, the TD3 algorithm comprises six networks: an actor network (with parameter θ), an actor-target network (with parameter θ), and a target network. - Two critic networks (with parameters ω1 and ω2) and two critic target networks (with parameters ω1 and ω2) and ).

[0133] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0134] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One through Five in that, in step 35, a neural network model based on a deep reinforcement learning-based autonomous visual navigation obstacle avoidance decision-making agent is trained to obtain the trained neural network model. The specific process is as follows:

[0135] Step 351: Randomly select 128 experiences from the experience replay pool, and form a batch of 128 experiences;

[0136] Each experience is a 6-tuple consisting of the current state, the current observation, the selected action, the new state after the transition, the new observation after the transition, and the reward value returned by the environment;

[0137] The i-th experience is represented as i ,o i ,a i ,s i+1 ,o i+1 ,r i >, i = 1, 2, ..., 128;

[0138] Among them, s i Let o represent the state in the i-th experience. i Let a represent the observation in the i-th experience. i Let s represent the action in the i-th experience. i+1 Let o represent the new state in the i-th experience after the transition. i+1 Let r represent the new observation in the i-th experience after the transfer. i This represents the reward value returned by the environment;

[0139] ​The experiences in the experience pool are counted individually. For example, my experience pool can store 50,000 experiences. Each experience is a 6-tuple. When sampling, 128 experiences are randomly selected from it to form a batch. i means which number in this batch. This batch is used for one network parameter update.

[0140] Step 352

[0141] The observation o in the i-th experience i Input actor network, actor network outputs motion Outputting actor's movements via the internet Input critic network 1 and critic network 2 respectively;

[0142] The state s in the i-th experience i Input critic network 1 and critic network 2 respectively, and critic network 1 and critic network 2 respectively output the value estimates of state-action pairs. and ω1 represents the parameters of critic network 1, and ω2 represents the parameters of critic network 2;

[0143] The new observation o in the i-th experience after the transfer i+1 Input the actor target network, and the actor target network outputs the selected action. Output action to the actor's target network Input critic target network 1 and critic target network 2 respectively;

[0144] The new state s in the i-th experience after the transition i+1 Input critic target network 1 and critic target network 2 respectively, and critic target network 1 and critic target network 2 respectively output the value estimate of the state-action pair. and For the parameters of the critic's target network 1; For the parameters of the critic target network 2;

[0145] Two critic target networks (with input consistent with the critic network);

[0146] Step 353, based on as well as Calculate timing error

[0147] Indicates taking or

[0148] Indicates taking The smaller of the two values;

[0149] r represents the continuous reward value; γ represents the discount factor.

[0150] Step 354: Take the mean squared error (MSE) of the time series error to obtain the loss function L(ω) of the commentator network;

[0151] Based on the output of the critic network and Construct the loss function L(θ) for the actor network;

[0152] The actor network performs soft updates to the actor target network;

[0153] Critics Network 1 performs a soft update on Critics Target Network 1;

[0154] Critics Network 2 performs a soft update on Critics Target Network 2;

[0155] Step 355: The number of steps in each training round (success or failure is 1 round, and 1 round is 1 step) shall not exceed the maximum number of steps in a round. When the number of steps in this round reaches the maximum number of steps in a round, the drone reaches the target, collides with an obstacle, or goes out of the flight range, the training round ends and the next round begins.

[0156] After the number of training steps exceeds the initial learning step count, the network parameters for the actor (θ) and the target actor (θ) are updated according to the training frequency (every few steps). - , Parameter ω1 of Critics Network 1, Parameter ω2 of Critics Network 2, Parameter of Critics Target Network 1 And critic target network 2 parameters Update;

[0157] Training stops when the reward function converges or reaches the total number of training steps, thus obtaining a trained neural network model of an autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning.

[0158] The experience playback pool of this invention adopts a first-in-first-out strategy to... t ,o t ,a t ,s t+1 ,o t+1 ,r t As a set of samples, a batch of samples of a certain size is drawn during model updates;

[0159] The actor network updates less frequently than the critic network to ensure that the critic network has enough time to fit a more accurate Q function and to prevent the actor network from using unstable Q-value estimates to optimize network parameters.

[0160] Actor Target Network (parameter θ)​- ) and two critic target networks (parameters are) and It is based on a soft update approach, and is updated gradually according to the parameters of the actor and critic network.

[0161] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0162] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One to Six in that the network results of the actor network, actor target network, critic network 1, critic network 2, critic target network 1, and critic target network 2 are the same;

[0163] Each of the actor network, actor target network, critic network 1, critic network 2, critic target network 1, and critic target network 2 sequentially includes a feature extraction network and a decision network;

[0164] The feature extraction network consists of a first convolutional block, a second convolutional block, a third convolutional block, and a global average pooling layer.

[0165] Each of the first, second, and third convolutional blocks sequentially contains a convolutional layer, a batch normalization (BN) layer, a max pooling layer, and an activation function ReLU.

[0166] The decision network consists of a first multilayer perceptron, a second multilayer perceptron, and uses LeakyReLU as the activation function.

[0167] The activation function uses LeakyReLU to prevent the gradient vanishing problem from causing the actor network to continuously output boundary values.

[0168] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0169] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One through Seven in that the mean squared error (MSE) is taken for the time series error to obtain the loss function L(ω) of the commentator network, which is expressed as:

[0170]

[0171] Based on the loss function L(ω) of the critic network, the parameters ω1 of critic network 1 and ω2 of critic network 2 are updated using the gradient descent theorem.

[0172] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0173] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the output based on the critic network... and The loss function L(θ) of the actor network is constructed as follows:

[0174]

[0175] Based on the loss function L(θ), the gradient descent theorem is used to update the parameters θ of the actor network;

[0176] Where N represents the total number of sampled experiences in a batch of the experience replay pool.

[0177] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0178] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One through Nine in that, in step 5, obstacle avoidance control is performed on the quadcopter drone to be controlled based on the neural network model of the trained autonomous visual navigation obstacle avoidance decision-making agent; the specific process is as follows:

[0179] The system acquires the drone's status, inputs the drone's status into the trained neural network model (actor network) of the autonomous visual navigation obstacle avoidance decision-making agent, outputs actions based on the output actions, and performs obstacle avoidance control on the quadcopter drone to be controlled.

[0180] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0181] The working process of the actor network model is as follows:

[0182] The observation information of the UAV agent is input into the feature extraction network in the actor network model. The feature extraction network in the actor network model outputs the features of the depth image. The features of the depth image and the noisy UAV's own state vector are concatenated and input into the decision network in the actor network model. The decision network in the actor network model outputs the action; thus, a trained actor network model is obtained.

[0183] The working process of the critic network model is as follows:

[0184] The state information of the drone agent is input into the feature extraction network of the critic network model. The feature extraction network of the critic network model outputs the features of the depth image. The features of the depth image and the drone's own state vector are concatenated and input into the decision network of the critic network model. The decision network of the critic network model outputs the value assessment; thus, a trained critic network model is obtained.

[0185] The specific structure of the network is shown in Table 3.

[0186] To extract key information from images more effectively, this invention employs a convolutional neural network as the feature extraction network for the actor and critic network models in deep reinforcement learning;

[0187] Taking into account some objective environmental factors, we test and evaluate the performance of an autonomous visual navigation and obstacle avoidance strategy for a quadrotor UAV based on deep reinforcement learning.

[0188] Includes the following steps:

[0189] 1. Determine the evaluation indicators;

[0190] Evaluation metrics should be closely related to the UAV's autonomous navigation and obstacle avoidance mission, so as to determine whether the mission was successfully completed and to quantitatively describe the mission's performance in conjunction with the reward function.

[0191] The present invention uses the following metrics: Average Episode Reward (AER), Average Steps of Successful Episodes (ASSE), and Success Rate (SR).

[0192] 2. Conduct performance comparison experiments of autonomous visual navigation obstacle avoidance algorithms. Comparative experiments were conducted on the proposed DPRL algorithm, the common TD3 algorithm, and the cutting-edge UAV autonomous navigation algorithm EGO-Planner-v2 in both training and random environments. Noise was added to the perception information of all algorithms during the experiments. For each algorithm, 30 rounds of testing were performed in each environment, and empirical data at each time step was saved to calculate various evaluation metrics.

[0193] 3. Ablation experiments were conducted on the autonomous visual navigation obstacle avoidance algorithm. Based on the proposed algorithm, privileged information and the multi-agent experience collection strategy were first removed to verify the impact of these two strategies on the overall performance of the algorithm. Then, the state space and action space of the algorithm were adjusted to evaluate the rationality and effectiveness of the algorithm design proposed in this invention.

[0194] The technical solution of the present invention will be further described below with reference to the embodiments, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0195] like Figure 1 As shown, the specific implementation steps of the autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning of the present invention are as follows:

[0196] Step 1: Construct an autonomous visual navigation and obstacle avoidance simulation environment for the quadcopter UAV. Select and initialize the quadcopter UAV's dynamic model, set the UAV's starting position, and define the target position. The specific steps of Step 1 are as follows:

[0197] Step 11: Use Unreal Engine 4, a 3D modeling software, to design a training environment with dense obstacles, and create a model evaluation environment with randomly set obstacle positions, such as... Figure 2 As shown. In the training environment, 70 cylindrical obstacles with a radius of 2.5m and a height of 15m were placed inside a circle centered at the origin with a radius of 60m. In the model evaluation environment, 60 of the same obstacles as in the training environment were randomly placed inside a circle centered at the origin with a radius of 60m.

[0198] Step 12: Integrate the AirSim plugin into the Unreal Engine 4 simulation environment. Use the Multirotor dynamics model provided by AirSim and set its action execution duration to 0.1s to ensure smooth control commands and avoid placing excessive computational load on the simulation environment, thereby achieving the optimal simulation frame rate. The dynamics model parameter settings for the UAV are shown in Table 1.

[0199] Table 1. Parameter settings for the UAV dynamics model

[0200]

[0201] Step 13: In each training cycle, the UAV takes off from the origin of the environment at a height of 5m. It randomly selects a target point on a circle with a radius of 65m centered at the origin and navigates towards that target point. The training round is considered successful when the UAV enters the target point and reaches its designated distance range. The round fails when the UAV exceeds the permitted flight range of the environment or when the distance to an obstacle is less than the collision distance. The UAV's position and status are then reset, and the next training round begins. The simulation environment parameter settings are shown in Table 2.

[0202] Table 2 Simulation Environment Parameter Settings

[0203]

[0204] Step 2: Model the problem by simulating some observable phenomena in a simulation environment. Combine privileged learning to design a reasonable state space, observation space, and action space, and construct a neural network model for a deep reinforcement learning-based autonomous visual navigation and obstacle avoidance decision-making agent. The specific steps of Step 2 are as follows:

[0205] Step 21: Problem Modeling. Considering the partially observable environmental phenomena caused by factors such as the high-speed movement of the UAV, sensor noise, and environmental wind disturbance, this invention models the autonomous navigation task of the UAV as a partially observable Markov Decision Process (POMDP), which can be represented as follows:<s,o,a,r,p,γ> Let γ represent the state, observation, action, reward, state transition probability, and discount factor, respectively. In an autonomous navigation mission for an unmanned aerial vehicle (UAV), state s represents the accurate state information of the UAV in the current environment; observation o represents the estimate of state s obtained by the UAV through onboard sensors; action a is the flight control command executed by the UAV at each time step; reward r is used to evaluate the value of the UAV performing an action in a certain state to transition to the next state; state transition probability p defines the probability of taking an action in the current state to transition to the next specific state, which is determined by UAV dynamics and affected by factors such as control input, environmental conditions, and system noise; discount factor γ determines the importance of the reward in future time steps compared to the reward in the current time step, with the discount factor closer to 1 indicating that the future reward and the current reward are approximately equally important.

[0206] Step 22: Simulate some observable phenomena in the environment. This invention simulates some observable phenomena that may be encountered in the real world using various types of noise. Specifically, this invention adds a mean value of μ to each dimension of its own state perception information. s The standard deviation is σ s Gaussian noise is used to simulate the sensor's perception error and the estimation error of noise or visual-inertial odometry, and this noise is limited to prevent excessive interference, thereby ensuring the stability of training. Furthermore, the present invention sequentially adds the following types of noise to the depth image perception information:

[0207] 1) Salt and pepper noise: This invention first uses probability p sp Pixels in the depth image are randomly assigned extreme values ​​(0 or 255). Salt-and-pepper noise simulates sudden sensor perception errors caused by abrupt changes in illumination, strong light reflection, or signal loss.

[0208] 2) Gaussian noise: After adding salt-and-pepper noise, this invention adds a mean of μ to the depth image. g The standard deviation is σ g Gaussian noise. Gaussian noise introduces random measurement bias throughout the image, which, when combined with salt and pepper noise, softens some extreme values ​​and simulates a more realistic scenario where multiple noises coexist.

[0209] 3) Motion blurring: After adding Gaussian noise, this invention adds a convolution kernel of size k to the depth image. mbMotion blur. Motion blur simulates the blurring effect caused by the motion of a camera or object, applied after other noise to simulate the aggravating effect of high-speed movement of a drone or shake of an airborne camera on sensor noise in real-world scenarios.

[0210] The effect of adding noise to depth image perceptual information is as follows: Figure 3 As shown.

[0211] Step 23: Setting up the state space. This invention is based on the concept of privileged learning, employing the Learning Using Privileged Information (LUPI) framework, inputting state information as privileged information into the critic network. The state information in this invention is multimodal information containing visual perception information and the UAV's own state perception information. Specifically, the visual perception information is a 240×320 resolution depth image acquired by the UAV's onboard depth camera, which is then adjusted to 80×100 for storage and processing. Subsequently, a feature extraction network extracts features from the depth image and performs dimensionality reduction, transforming the visual perception data D∈R... 80×100 Compressed into S1∈R 25 As part of the system state, another part of the system state is a vector S2∈R containing 8-dimensional information about the UAV's own state. 8 This includes the three-axis distance [d] from the UAV to the target point at the current time step. x ,d y ,d z ]∈R 3 The three-axis speed of the drone [v] x ,v y ,v z ]∈R 3 The yaw angle deviation Δψ∈R between the current flight direction and the target direction 1 and the current yaw rate ψ∈R of the UAV 1 By concatenating the two state vectors, the complete system state vector S = [S1, S2] ∈ R is obtained. 33 .

[0212] Step 24: Set up the observation space. The observation information of this invention is the state information plus the noise from step 22 to simulate the state that the UAV can observe in a partially observable environment. The dimension of the observation space is the same as the dimension of the state space, and can be represented as O∈R. 33 Within the LUPI framework, observational information serves as input to the actor network.

[0213] Step 25: Set the motion space. The motion space of this invention is the flight control command A∈R of the UAV. 4 Includes three-axis speed commands [vx ,v y ,v z ]∈R 3 and yaw rate ψ∈R 1 These continuous actions directly control the movement of the drone in the simulated environment.

[0214] Step 26: Based on the principles and implementation framework of privileged learning, construct a reinforcement learning-based autonomous navigation and obstacle avoidance model using neural networks. This invention combines deep reinforcement learning with privileged learning to construct an asymmetric input actor and critic network model, whose inputs are the observation information and state information of the UAV agent, respectively. The actor and critic networks have the same structure, as shown below. Figure 4 As shown, each network comprises two parts: a feature extraction network and a decision network. The feature extraction network consists of convolutional blocks, each containing a convolutional layer, a batch normalization operation, and a max-pooling layer, with ReLU as the activation function. The decision network is a two-layer multilayer perceptron with 128 neurons per layer, using Leaky ReLU as the activation function to prevent the gradient vanishing problem from causing the network to continuously output boundary values. The specific structure of the network is shown in Table 3.

[0215] Table 3. Deep Reinforcement Learning Network Structure

[0216]

[0217] Step 3: Define a reinforcement learning reward function, employ a multi-agent exploration strategy, and train a deep reinforcement learning-based autonomous visual navigation obstacle avoidance decision-making model. This allows the model to continuously acquire training samples and optimize network parameters through interaction with the environment, thereby improving the success rate of autonomous visual navigation obstacle avoidance. The specific steps of Step 3 are as follows:

[0218] Step 31: Define the reward function. This invention employs a combination of sparse and continuous rewards based on the current time step and the current round state. At the end of each training round, the agent receives a sparse reward including a positive reward for reaching the target, a collision penalty, and a penalty for exceeding the flight range. Specifically, when the drone enters the target's reach range, it receives a positive reward of +10; if the drone collides with an obstacle or flies out of the environment boundary, it receives a penalty of -5. Furthermore, at each time step, the agent receives a continuous reward including a distance difference positive reward, a distance error penalty, and an obstacle approach penalty. The distance difference positive reward is used to evaluate the agent's proximity to the target at the current time step compared to the previous time step, encouraging the drone to move closer to the target. The distance error penalty is calculated based on the distance from the drone to the starting point and the target along a straight line, ensuring the drone travels along the shortest straight path and improving final navigation accuracy. The obstacle approach penalty is calculated based on the difference between the shortest distance and the safe distance between the drone and the obstacle at each time step, helping to prevent the drone from approaching obstacles.

[0219] The expression for continuous rewards is as follows:

[0220]

[0221] in,

[0222] r e Indicates a reward item; d g d represents the distance from the starting point to the target. t and d t-1 These represent the distances between the drone and the target at the current and previous time steps, respectively.

[0223] p p Indicates penalty item 1; z and z g Let d represent the z-axis coordinates of the UAV's current time step and the target position, respectively. l This represents the distance from the drone's current position to the straight line connecting the starting point and the target; the clip function is used to limit the data amplitude.

[0224] p o Indicates penalty item 2; d o Indicates the shortest distance from the drone to the surface of the obstacle; d c Let d be the collision distance. o Less than d c It was assumed that a collision had occurred; d s It is a safe distance, when d o Less than d s There is a risk of collision for mission-oriented drones;

[0225] r represents the continuous reward value; η r It is r escaling factor, η p It is p p scaling factor, η o It is p o The scaling factor;

[0226] Step 32: Set the parameters of the experience replay pool and the hyperparameters of the agent neural network model. The training hyperparameter settings of this invention are shown in Table 4. The number of steps in each training round does not exceed the maximum number of steps in a round. When the number of steps in this round reaches the maximum number of steps in a round, the drone reaches the target, collides with an obstacle, or exceeds the flight range, the training round ends and the next round begins. When the number of training steps exceeds the number of steps at the start of learning, the model is updated according to the training frequency. The experience replay pool of this invention adopts a first-in, first-out strategy to... t ,o t ,a t ,s t+1 ,o t+1 ,r t As a set of samples, a batch size of samples is drawn during model updates.

[0227] Table 4 Training Parameter Settings

[0228]

[0229] Step 33: Set the action selection strategy to complete experience collection. To balance exploration and utilization, this invention uses an ε-greedy algorithm for action selection. ε is a probability value close to 0. At each time step, a random number is generated. If the random number is greater than or equal to ε, the action with the highest value in the current state is selected according to the reinforcement learning algorithm; otherwise, an action is randomly selected from the action space. The ε-greedy algorithm dynamically adjusts the reinforcement learning's exploration of new strategies and utilization of existing strategies as the training process progresses. Subsequently, the action is executed in the simulation environment, the state at the next time step is collected, and the reward value is calculated. The current state, observation, action, reward, and the state and observation at the next time step are stored in the experience replay pool until the experience replay pool is full or the simulation ends.

[0230] Step 34: Define the multi-agent exploration strategy. This invention uses multiple processes to run multiple AirSim environments in parallel. In each environment, a drone is placed, and these drones interact independently with their respective environments, collecting experience as described in Step 33. This experience is then stored in a shared experience replay pool for training a central model. This model provides decision-making action outputs to each drone, ensuring consistent policy updates across all environments. The multi-agent exploration strategy proposed in this invention enables knowledge sharing among drones, improves experience collection efficiency, and thus accelerates model convergence.

[0231] ​Step 35: Train the deep reinforcement learning model. The parameter update of the neural network model DRPL based on deep reinforcement learning for autonomous visual navigation obstacle avoidance decision-making in this invention is based on the TwinDelayed Deep Deterministic Policy Gradient (TD3) algorithm. Under the POMDP task modeling, its parameter update mechanism is as follows: Figure 5 As shown. Specifically, the TD3 algorithm comprises six networks: an actor network (parameter θ), an actor-target network (parameter θ-), two critic networks (parameters ω1 and ω2), and two critic-target networks (parameters ω1 and ω2). and At each time step, The actor network is based on current observations. i The chosen action For the actor target network, observations are based on the next time step. i+1 The selected action, s i and s i+1 Is with o i and o i+1 The corresponding accurate status of the drone. and It consists of two commentator networks' value estimates for a given state-action pair. and These are the value estimates of the next state-action pair by the target network from two critics, from which the temporal error can be calculated. Taking the mean squared error (MSE) of the temporal error of a batch of experience, we can obtain the loss function of the critic network, as shown in Equation 2. Based on the loss function, the parameters of the critic network can be updated using the gradient descent theorem. The loss function of the actor network is constructed based on the Q-value output by the critic network, as shown in Equation 3, where N is the number of batches of playback data used for updating. Based on the loss function, the parameters of the actor network can be updated using the gradient descent theorem. The update frequency of the actor network is lower than that of the critic network to ensure that the critic network has sufficient time to fit a more accurate Q-function and to avoid the actor network using unstable Q-value estimates to optimize its network parameters. All target networks are based on a soft update method, gradually updating the parameters of the actor and critic networks.

[0232]

[0233] Step 4: Considering some observable environmental factors, test and evaluate the performance of the autonomous visual navigation and obstacle avoidance strategy for quadcopter UAVs based on deep reinforcement learning. The specific steps of Step 4 are as follows:

[0234] Step 41: Determine the evaluation metrics. The evaluation metrics should be closely related to the UAV's autonomous navigation and obstacle avoidance task, both determining whether the task was successfully completed and quantitatively describing the task's performance in conjunction with the reward function. This invention selects the following metrics: Average Episode Reward (AER), Average Steps of Successful Episodes (ASSE), and Success Rate (SR). AER is used to evaluate the algorithm's overall performance, including navigation accuracy, obstacle avoidance safety, and algorithm efficiency. ASSE is used to evaluate the algorithm's efficiency; a smaller ASSE indicates that the UAV can complete the task with shorter step lengths, thus achieving higher efficiency. SR measures the algorithm's success rate; a higher SR indicates a safer and more practical algorithm.

[0235] Step 42: Conduct performance comparison experiments of autonomous visual navigation obstacle avoidance algorithms. Comparative experiments were conducted on the proposed DPRL algorithm, the common TD3 algorithm, and the cutting-edge UAV autonomous navigation algorithm EGO-Planner-v2 in both training and random environments. During the experiments, noisy visual and self-state perception information was provided to DPRL and TD3, while noisy odometry information was provided to EGO-Planner-v2, thus interfering with its mapping process. The maximum speed of the UAV using the EGO-Planner-v2 algorithm was kept consistent with the settings of the other two algorithms, and the PD controller was used to output speed and yaw angle control commands to control the UAV's flight. For each algorithm, the differences in their training processes were compared, and the performance of the trained models was tested in both training and random environments. Figure 6 Curves (a) and (b) in the table represent the SR and AER curves of the DPRL and TD3 algorithms trained under four different seeds, respectively. Figure 6 As shown in (a), DPRL converges significantly faster than TD3, achieving an average success rate of 85% after 220,000 steps. Figure 6 The convergence trend of the AER curve shown in (b) is consistent with that of the SR curve. The average reward per round of DPRL increases rapidly in the early to mid-stages of training and stabilizes at a high reward value after 240,000 steps. In contrast, the average reward per round of TD3 increases slowly throughout the training process and does not converge at the end of training.

[0236] In the model performance testing experiments, this invention conducted 30 rounds of testing for each algorithm in two environments, saving the empirical data at each time step to calculate various evaluation metrics. The flight trajectories of the three algorithms in the two environments are visualized as follows: Figure 7As shown in the figure, the blue trajectory represents the trajectory that successfully reaches the target, and the red trajectory represents the trajectory that collides with obstacles. It can be seen that DPRL maintains a high success rate in both training and random environments, demonstrating its robustness to environmental transfer and some observable phenomena, and its strong ability to adapt to new environments. In contrast, the TD3 algorithm has the lowest success rate, performs poorly in both environments, and its trajectory is not as smooth as DPRL's. For the EGO-Planner-v2 algorithm, since both environments are unfamiliar, and the obstacles in the training environment are denser, its performance in the training environment is worse than in the random environment. EGO-Planner-v2 can plan the smoothest trajectory and has the highest arrival accuracy, but its planning efficiency is the lowest, requiring the most steps to reach the target. The evaluation metrics of the three algorithms are compared in Table 5.

[0237] Table 5 Comparison of Model Performance Test Evaluation Indicators

[0238]

[0239] Step 43: Conduct ablation experiments on the autonomous visual navigation obstacle avoidance algorithm. Based on the proposed algorithm, privileged information and the multi-agent experience collection strategy are first removed, and these are named distributed reinforcement learning and privileged reinforcement learning, respectively, to verify the impact of these two strategies on the overall performance of the algorithm. The SR and AER curves of the three algorithms during the training process are shown below. Figure 8 As shown, Figure 8 Figure (a) demonstrates that DPRL achieves faster convergence and a higher final success rate compared to distributed reinforcement learning and privileged reinforcement learning. Specifically, the success rate of DPRL stabilizes after 220,000 steps, while the success rates of distributed reinforcement learning and privileged reinforcement learning converge after 300,000 steps. Figure 8 The trend of the average reward curve in (b) is consistent with the success rate curve. The average reward of DPRL stabilizes above 30 after 240,000 steps, while distributed reinforcement learning and privileged reinforcement learning require more than 300,000 steps to achieve similar rewards. Furthermore, even after training, the average reward of distributed reinforcement learning does not exceed 30. Comparative analysis of the results of distributed reinforcement learning and privileged reinforcement learning shows that privileged learning is more effective than multi-agent exploration strategies. In the early stages of training, distributed reinforcement learning converges faster than privileged reinforcement learning, demonstrating the significant role of multi-agent exploration strategies in accelerating empirical convergence in the early stages. However, after 200,000 steps of training, privileged reinforcement learning outperforms distributed reinforcement learning, ultimately achieving higher SR and AER, highlighting the effectiveness of privileged learning in dealing with considerable environmental phenomena.

[0240] The present invention then conducted ablation experiments on the design of the state and action spaces. The present invention designed another state and action space, modifying the composition of the UAV's own state perception vector in the state space. The original three-axis distance to the target was replaced with distance on the xy-plane and distance on the z-axis, and the original three-axis velocity was replaced with velocity on the xy-plane and velocity on the z-axis. This modification reduced the total state vector dimension from 33 to 31. Simultaneously, the action space was modified, replacing the original x and y-axis velocities with xy-plane velocities, and decomposing the x and y-axis velocities based on the current yaw angle when controlling the UAV's flight. This state and action space setting ensures that the UAV always flies towards the camera's optical axis, keeping obstacles in the field of view. However, this design compresses the action space, reducing the UAV's flight flexibility. The SR and AER curves of DPRL in the two states and action spaces during model training are shown below. Figure 9 As shown, the DPRL designed in the 4D action space of this invention significantly outperforms the algorithm in the 3D action space. Specifically, during training, the DPRL in the 3D action space has almost no successful flight rounds in the early stages, and the model's navigation and obstacle avoidance capabilities learn very slowly, achieving only a 30% success rate by the end of training. The AER curve shows that the average round reward of the DPRL in the 3D action space hardly increases with training, ultimately reaching only 10, demonstrating that the compression of the action space greatly affects the model's learning process, indirectly proving the rationality of the state and action space designed in this invention.

[0241] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for autonomous visual navigation and obstacle avoidance of a quadcopter UAV based on deep reinforcement learning, characterized in that: The specific process of the method is as follows: Step 1: Construct an autonomous visual navigation and obstacle avoidance simulation environment for a quadcopter UAV, select and initialize the dynamic model of the quadcopter UAV, set the starting position of the UAV and set the target position; Step 2: Set up the state space, observation space, and action space in the simulation environment; Step 3: Construct a neural network model for an autonomous visual navigation obstacle avoidance decision-making agent; Step 4: Train the neural network model of the autonomous visual navigation obstacle avoidance decision-making agent to obtain the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent. Step 5: Perform obstacle avoidance control on the quadcopter UAV under control based on the neural network model of the trained autonomous visual navigation obstacle avoidance decision-making agent; In step 4, a neural network model of an autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning is trained to obtain the trained neural network model. The specific process is as follows: Step 31: Define the reward function; Reward functions include sparse rewards and continuous rewards; Sparse rewards are as follows: If the current time step round ends, the reward includes a positive reward for successfully reaching the target point, and a negative reward for colliding or exceeding the flight range; the specific process is as follows: When the drone enters the target's reach range, it receives a positive reward of +10; if the drone collides with an obstacle or flies out of the environmental boundary, it receives a penalty of -5. The expression for continuous rewards is as follows: in, r e Indicates a reward item; d g d represents the distance from the starting point to the target. t and d t-1 These represent the distances between the drone and the target at the current and previous time steps, respectively. p p Indicates penalty item 1; z and z g Let d represent the z-axis coordinates of the UAV's current time step and the target position, respectively. l This represents the distance from the drone's current position to the straight line connecting the starting point and the target; the clip function is used to limit the data amplitude. p o Indicates penalty item 2; d o Indicates the shortest distance from the drone to the surface of the obstacle; d c d represents the collision distance. s It is a safe distance; r represents the continuous reward value; η r It is r e scaling factor, η p It is p p scaling factor, η o It is p o The scaling factor; Step 32: Set the parameters of the experience replay pool and the deep reinforcement learning neural network; the specific process is as follows: The experience replay pool parameter is the size parameter of the experience replay pool; The hyperparameters of deep reinforcement learning neural networks include network structure, number of neurons, learning rate, discount factor, learning rate, learning start step, experience replay pool size, batch size, training frequency, action noise standard deviation, number of training environments, total number of training steps, and maximum number of steps in each round. Step 33: Set an action selection strategy to complete experience collection; the specific process is as follows: First, an ε-greedy strategy is used to select random actions or actions output by the actor network; Then, the drone performs a selected random action or an action output by the actor network and interacts with the environment to generate a new state after the transition, a new observation after the transition, and a reward value returned by the environment. The current state, current observation, selected action, new state after transition, new observation after transition, and reward value returned by the environment are considered. t ,o t ,a t ,s t+1 ,o t+1 ,r t >Store it into the experience replay pool to complete one experience collection;​ s t Indicates the current state, o t Indicates the current observation, a t Indicates the currently selected action, s t+1 Indicates the new state after the transition, o t+1 Indicates the new observation after the transfer, r t This represents the reward value returned by the environment; Step 34: Set up a multi-agent exploration strategy to complete experience collection; the specific process is as follows: M processes run M AirSim environments in parallel, and place a drone in each environment to perform the experience collection in step 33, thereby completing one experience collection. M drones all input observations into the same decision network, obtain corresponding actions, execute actions to interact with the environment, and generate a new state after the transition, a new observation after the transition, and a reward value returned by the environment; the network then stores the current state, current observation, selected action, new state after the transition, new observation after the transition, and reward value returned by the environment. t ,o t ,a t ,s t+1 ,o t+1 ,r t > Store them in the same experience replay pool to complete one experience collection;​ Step 35: Train the neural network model of the autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning, and obtain the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning.

2. The method for autonomous visual navigation and obstacle avoidance of a quadcopter UAV based on deep reinforcement learning according to claim 1, characterized in that: Step 1 involves constructing an autonomous visual navigation and obstacle avoidance simulation environment for a quadcopter UAV, selecting and initializing the quadcopter UAV's dynamic model, setting the UAV's starting position, and defining the target position. The specific process is as follows: Step 11: Use Unreal Engine 4 to create a training environment that includes obstacles; Step 12: Integrate the AirSim plugin into the Unreal Engine 4 simulation environment, import the quadcopter drone dynamics model provided by AirSim, and set the dynamic parameters of the quadcopter drone dynamics model. The parameters of the UAV dynamics model are the duration of the UAV flight maneuver, the UAV's velocity range along the x-axis, the UAV's velocity range along the y-axis, the UAV's velocity range along the z-axis, and the UAV's yaw rate range. The x-axis, y-axis, and z-axis form the coordinate system of the UAV body. The direction of the UAV's head is the x-axis, the z-axis is perpendicular to the x-axis and the y-axis is perpendicular to the xz plane. Step 13: Set the initial and target positions of the drone for each training round, define the effective flight space range, and set the conditions for successful navigation and obstacle avoidance flight as well as the failure conditions.

3. The autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning according to claim 2, characterized in that: In step 2, the state space, observation space, and action space are set up in the simulation environment; the specific process is as follows: Step 21: Model the autonomous navigation task of the UAV as a partially observable Markov decision process, represented as:<s,o,a,r,p,γ> ; Where s is the state, o is the observation, a is the action, r is the reward, p is the state transition probability, and γ is the discount factor; Step 22: Set up the state space, which is a continuous state space; the specific process is as follows: Status information includes visual perception information and the drone's own status information; Visual perception information is depth images acquired by the drone's onboard depth camera; The depth image acquired by the UAV's onboard depth camera is input into the feature extraction network, and the feature extraction network outputs a visual perception information vector S1. The feature extraction network consists of a first convolutional block, a second convolutional block, a third convolutional block, and a global average pooling layer. Each of the first, second, and third convolutional blocks contains, in sequence, a convolutional layer, a batch normalization (BN) layer, a max pooling layer, and an activation function ReLU. The drone's own status information includes the current three-axis distance from the drone to the target point [d] x ,d y ,d z The current three-axis speed of the drone [v] x ,v y ,v z ] Yaw angle deviation between the current flight direction and the target direction and the current yaw rate of drones The drone's own state information is an 8-dimensional drone own state information vector S2; The current three-axis speed of the drone [v] x ,v y ,v z [This refers to the three-axis velocity in the UAV's body coordinate system;] The current three-axis distance from the UAV to the target point [d] x ,d y ,d z [This refers to the distance across the three axes in the world coordinate system.] The visual perception information vector S1 and the UAV's own state information vector S2 are concatenated to obtain the state information vector S = [S1, S2]. Step 23: Set the motion space, which is a continuous motion space; The action space is the drone's flight control command A, which includes the drone's current three-axis velocities [v]. x ,v y ,v z and yaw rate The current three-axis speed of the drone [v] x ,v y ,v z [This refers to the three-axis velocity in the UAV's body coordinate system;] Step 24: Set up the observation space, which is a continuous observation space; The specific process is as follows: Salt-and-pepper noise, Gaussian noise, and motion blur are added sequentially to the depth image information of the UAV; In the UAV's own state information, the three-axis distance [d] from the current UAV to the target point is... x ,d y ,d z Add Gaussian noise; In the drone's own state information, the current three-axis velocity [v] of the drone is... x ,v y ,v z Add Gaussian noise; In the UAV's own status information, the yaw angle deviation between the current flight direction and the target direction is... Add Gaussian noise; In the drone's own state information, the current drone yaw rate is... Add Gaussian noise.

4. The autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning according to claim 3, characterized in that: Step 3 involves constructing a neural network model for an autonomous visual navigation obstacle avoidance decision-making agent; the specific process is as follows: The neural network model of the autonomous visual navigation obstacle avoidance decision-making agent consists of a feature extraction network and a decision network. The feature extraction network consists of a first convolutional block, a second convolutional block, a third convolutional block, and a global average pooling layer. Each of the first, second, and third convolutional blocks sequentially contains a convolutional layer, a batch normalization (BN) layer, a max pooling layer, and an activation function ReLU. The decision network consists of a first multilayer perceptron, a second multilayer perceptron, and uses Leaky ReLU as the activation function.

5. The autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning according to claim 4, characterized in that: In step 35, a neural network model of an autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning is trained to obtain the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning. The specific process is as follows: Step 351: Randomly select 128 experiences from the experience replay pool, and form a batch of 128 experiences; Each experience is a 6-tuple consisting of the current state, the current observation, the selected action, the new state after the transition, the new observation after the transition, and the reward value returned by the environment; The i-th experience is represented as i ,o i ,a i ,s i+1 ,o i+1 ,r i >, i = 1, 2, ..., 128;​ Among them, s i Let o represent the state in the i-th experience. i Let a represent the observation in the i-th experience. i Let s represent the action in the i-th experience. i+1 Let o represent the new state in the i-th experience after the transition. i+1 Let r represent the new observation in the i-th experience after the transfer. i This represents the reward value returned by the environment; Step 352 The observation o in the i-th experience i Input actor network, actor network outputs motion Outputting actor's movements via the internet Input critic network 1 and critic network 2 respectively; The state s in the i-th experience i Input critic network 1 and critic network 2 respectively, and critic network 1 and critic network 2 respectively output the value estimates of state-action pairs. and ω1 represents the parameters of critic network 1, and ω2 represents the parameters of critic network 2; The new observation o in the i-th experience after the transfer i+1 Input the actor target network, and the actor target network outputs the selected action. Output action to the actor's target network Input critic target network 1 and critic target network 2 respectively; The new state s in the i-th experience after the transition i+1 Input critic target network 1 and critic target network 2 respectively, and critic target network 1 and critic target network 2 respectively output the value estimate of the state-action pair. and For the parameters of the critic's target network 1; For the parameters of the critic target network 2; Step 353, based on as well as Calculate timing error Indicates taking or Indicates taking The smaller of the two values; r represents the continuous reward value; γ represents the discount factor. Step 354: Take the mean square error of the time series error to obtain the loss function L(ω) of the commentator network; Based on the output of the critic network and Construct the loss function L(θ) for the actor network; The actor network performs soft updates to the actor target network; Critics Network 1 performs a soft update on Critics Target Network 1; Critics Network 2 performs a soft update on Critics Target Network 2; Step 355: The number of steps in each training round shall not exceed the maximum number of steps in a round. When the number of steps in this round reaches the maximum number of steps in a round, the drone reaches the target, collides with an obstacle, or exceeds the flight range, the training round ends and the next round begins. After the number of training steps exceeds the initial learning step count, the actor network parameters θ and the actor target network parameters are adjusted according to the training frequency. - , Parameter ω1 of Critics Network 1, Parameter ω2 of Critics Network 2, Parameter of Critics Target Network 1 And critic target network 2 parameters Update; Training stops when the reward function converges or reaches the total number of training steps, thus obtaining a trained neural network model of an autonomous visual navigation obstacle avoidance decision-making agent based on deep reinforcement learning.

6. The autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning according to claim 5, characterized in that: The network results for Actor Network, Actor Target Network, Critics Network 1, Critics Network 2, Critics Target Network 1, and Critics Target Network 2 are the same; Each of the actor network, actor target network, critic network 1, critic network 2, critic target network 1, and critic target network 2 sequentially includes a feature extraction network and a decision network; The feature extraction network consists of a first convolutional block, a second convolutional block, a third convolutional block, and a global average pooling layer. Each of the first, second, and third convolutional blocks sequentially contains a convolutional layer, a batch normalization (BN) layer, a max pooling layer, and an activation function ReLU. The decision network consists of a first multilayer perceptron, a second multilayer perceptron, and uses LeakyReLU as the activation function.

7. The autonomous visual navigation and obstacle avoidance method for a quadrotor UAV based on deep reinforcement learning according to claim 6, characterized in that: The mean square error of the time-series error is used to obtain the loss function L(ω) of the commentator network, which is expressed as: Based on the loss function L(ω) of the critic network, the parameters ω1 of critic network 1 and ω2 of critic network 2 are updated using the gradient descent theorem.

8. The autonomous visual navigation and obstacle avoidance method for quadrotor UAVs based on deep reinforcement learning according to claim 7, characterized in that: The commentator network output-based and The loss function L(θ) of the actor network is constructed as follows: Based on the loss function L(θ), the gradient descent theorem is used to update the parameters θ of the actor network; Where N represents the total number of sampled experiences in a batch of the experience replay pool.

9. The autonomous visual navigation and obstacle avoidance method for a quadrotor UAV based on deep reinforcement learning according to claim 8, characterized in that: In step 5, obstacle avoidance control is performed on the quadcopter drone under control based on the neural network model of the trained autonomous visual navigation obstacle avoidance decision-making agent; the specific process is as follows: The system acquires the drone's status, inputs the drone's status into the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent, outputs actions from the trained neural network model of the autonomous visual navigation obstacle avoidance decision-making agent, and performs obstacle avoidance control on the quadcopter drone to be controlled based on the output actions.

Images