Reinforcement learning unmanned aerial vehicle autonomous exploration method combined with pre-training

Abstract

The invention discloses a reinforcement learning unmanned aerial vehicle autonomous exploration method combined with pre-training, and belongs to the field of unmanned aerial vehicle autonomous exploration. According to the method, the target position and direction training of the unmanned aerial vehicle are separated, the pre-training of the target direction is matched, the training task is hierarchically and stably performed, the efficient generation of the training sample required by reinforcement learning is realized, and the overall training speed is improved. And meanwhile, in cooperation with a modular exploration mode, a traditional method is combined with reinforcement learning, a dense reward function is designed, and the overall quality and efficiency of exploration are improved.

Description

Technical Field

[0001] This invention belongs to the field of autonomous exploration of unmanned aerial vehicles (UAVs), specifically involving the use of reinforcement learning to select the next target decision point in exploration tasks. Background Technology

[0002] Autonomous exploration refers to the process by which robots explore unknown environments without any prior knowledge, through autonomous navigation and environmental perception. Quadrotor drones can fly freely in complex environments, requiring rapid adaptation to environmental changes. Only by generating smooth and continuous planned trajectories can they effectively track these trajectories and adapt quickly to dynamic environments. While traditional sampling-based or frontier-based methods possess some autonomous adaptability, in most cases, these constructed optimization problems struggle to find optimal solutions. Current exploration strategies based on deep reinforcement learning can better balance exploration efficiency and computational burden. These algorithms can more effectively utilize environmental information and execute near-optimal decision-making objectives.

[0003] The modular reinforcement learning autonomous exploration method divides the overall exploration process into four parts: mapping, decision-making, planning, and control. These modules work closely together to complete the active exploration task of a quadcopter UAV in an unknown area. When designing the active exploration strategy, the functions and advantages of other modules must be considered to ensure the organic unity and efficient operation of the entire system. Only the decision-making part is completed using reinforcement learning. The decision-making module selects the target point to which the UAV should move based on historical observation information. The module input is the surrounding environment map constructed by the mapping module and the UAV's speed, position, and other state information; the output is the target position and the UAV's orientation.

[0004] Traditional decision-making methods employ frontier-based and sampling-based approaches. Yamuchi pioneered a frontier-based method, which clusters environmental map information to identify frontiers and continuously selects the closest frontier points for exploration. Specifically, the established grid environment map is first divided into three parts: occupied, free, and unknown. Occupied areas are those with obstacles or where the defined exploration boundary has been reached; unknown areas are those not yet explored by sensors; and free areas are those already mapped and without obstacles. Adjacent regions between the free and unknown parts are extracted and segmented into blocks using principal component analysis; these blocks are considered the frontier. A greedy algorithm is then used to traverse these regions, and the center of each frontier region becomes the decision objective. This method builds upon Bircher's RHNBVP, which is inspired by the next best view algorithm. It combines fast exploration with random trees to sample viewpoints in space and designs an exponential penalty function incorporating distance costs. Specifically, the next optimal observation point is selected to efficiently cover unknown areas through random sampling and viewpoint optimization. First, a set of candidate viewpoints is randomly generated around the current robot position (e.g., sampling within a spherical or hemispherical space). For each candidate viewpoint, the information gain (based on coverage area, field of view quality, etc., of unknown voxels) is calculated. Then, using finite-step rolling optimization, the exploration benefits for future steps are evaluated (e.g., RRT* or Monte Carlo tree search). Finally, the viewpoint with the highest information gain and lowest path cost is selected. Most other traditional methods are improvements or fusions of these two approaches. Traditional methods often lack specific decision-making criteria, frequently selecting the optimal value after ranking a few nonlinear combinations of specific state values, which makes it difficult to guarantee optimality.

[0005] Niroui's deep reinforcement learning-based exploration decision-making method combines traditional frontier-based exploration methods with deep reinforcement learning. This learning-based frontier exploration technology can explore more of the environment earlier and is suitable for urban search and rescue exploration tasks. Specifically, map information and robot state are used as inputs, and a weight coefficient is obtained through reinforcement learning. This coefficient is used to calculate the sum of path length and frontier reward, and the current frontier is re-ranked and the corresponding optimal frontier is selected. Li decomposes the exploration process into four parts of modular reinforcement learning, constructing a general exploration framework. Reinforcement learning is used to directly obtain the target grid position for the decision module. Specifically, by inputting the current local map (grid map) and the robot's historical position into the neural network, the next target grid position is obtained. Compared with decision targets such as selecting the frontier, this is more flexible. The target selection is optimized using reinforcement learning algorithms. This method uses LiDAR mapping and does not involve the decision direction required by the visual camera. Zhao Yumin further trained the UAV's target position and target direction using an autoregressive deep neural network model and invalid action masking technology, solving the problems of large action space and many invalid actions. Specifically, the target decision location is first trained using reinforcement learning, while the decision direction is determined using traditional methods. After the reward function for the location training is relatively stable, the decision direction is then trained to obtain the final decision network outputting the target location and direction.

[0006] In summary, UAV autonomous exploration methods can be divided into two categories: traditional methods and reinforcement learning methods. Traditional methods struggle to find optimal solutions to the decision optimization problems they construct. Converting these problems into multi-objective optimization problems and using Shannon entropy to evaluate the selected candidate target points increases the computational burden rapidly with the expansion of the exploration area. Reinforcement learning modular methods select the next target decision point. However, in existing technologies, since the next flight target point in autonomous exploration requires determining both position and orientation, directly training and calculating both results in a low probability of obtaining both optimal position and orientation during initial sampling. This leads to the generation of a large number of inefficient samples, resulting in slow overall effective gradient iteration speed and generally poor training speed and final exploration quality. Summary of the Invention

[0007] To address the issues of slow training speed and unsatisfactory final exploration efficiency in reinforcement learning methods, this paper proposes an improved exploration method for UAVs based on policy gradient reinforcement learning.

[0008] Preparation: A set of deep neural network models was designed, using an autoregressive approach to interconnect the predictions of two actions. The policy's state value function and position / action selection share the same environment state encoding, which is then processed by their respective linear encodings to obtain the final value. Simultaneously, the position / action is passed to another encoder to obtain the environment feature encoding, which is then concatenated with the environment feature encoding and passed through a linear layer to obtain the final target direction. In the reinforcement learning-based UAV autonomous exploration model, the environmental observations at each time step are extracted as model input, divided into map information and UAV current state information, including position, direction, and velocity. The map information input is passed to a convolutional feature encoder. The purpose of feature extraction is to reduce the dimensionality of the original map information and encode the original input information into effective features for use by subsequent linear network layers. A corresponding dataset is constructed using the optimal orientation based on visual features in a grid-based system, using traditional methods, and includes only map information and UAV position information.

[0009] Specifically, the following steps are included:

[0010] S1. Select the optimal direction based on visual features;

[0011] S2. Reinforce learning based on the location of the decision-making target;

[0012] After S2.1 performs convolutional encoding on the map information, the linear layer, together with the current position, orientation, and speed of the UAV, is passed to the policy network and value network respectively.

[0013] S2.2 The value network is used to evaluate the value of the current state, while the policy network only generates positional actions for the target grid; the target direction at this stage is the optimal direction obtained in step S1.

[0014] S3. Pre-training targeting decision-making objectives;

[0015] S3.1 While performing step S2, a directional state mapping dataset for map information and drone position is constructed using the drone exploration platform, and the target direction is trained separately.

[0016] S3.2 also encodes map information; the linear layer only retains the drone's position information and outputs the corresponding […]. The angle and direction;

[0017] S3.3 Using the optimal direction obtained in step S1 as the target, pre-train the policy network for deciding the target direction;

[0018] S4. Intrinsic Entropy-Assisted Rewards and Consistent Selection Processing;

[0019] S4.1 In the early stages of training in step S2, the entropy of the actions can be... As part of the reward function, it is used to explore and update scenarios and fit multiple situations;

[0020] S4.2 sets three termination conditions: the deadline is reached, the step size task is not completed, and the planning task fails. Completing the exploration task is an additional sparse reward, which is part of the overall reward.

[0021] In the mid-to-late stages of S4.3 training, an additional consistency selection process is added to the policy gradient, which uses the angle between the drone's current speed and the target point as part of the penalty evaluation of the overall state value.

[0022] S5. Complete the final training in the full motion space;

[0023] When the training in steps S2 and S3 is stable, the target position obtained in step S2 is used as the input of the position information required in step S3, and reinforcement learning is performed again in the form of an autoregressive model to obtain the target position and target direction.

[0024] Further, the specific process of step S1 is as follows: based on the currently constructed environment map, generate several sampling orientations at the target location; for each orientation, calculate the number of unknown grids or feature points covered by it by simulating the field of view of the camera; arrange all random orientations, and the orientation with the most feature points corresponds to the optimal orientation under the current environment state.

[0025] Furthermore, in step S4, to avoid reciprocating motion, large-scale changes in the direction of motion are avoided; however, the motion extends at most along the velocity direction at the previous position. If there is an existing cell occupying a cell position under a given cell, no operation will be performed on it.

[0026] The beneficial effects of this invention are as follows: This invention provides a hybrid action decision-making method that separates the target position and direction training of a UAV, combined with pre-training on the target direction, enabling hierarchical and stable training tasks. This achieves efficient generation of training samples required for reinforcement learning, improving the overall training speed. Simultaneously, by incorporating a modular exploration approach, it combines traditional methods with reinforcement learning, designing a dense reward function to improve the overall quality and efficiency of exploration. Specifically, it includes:

[0027] 1. Compared with existing methods, the multi-module autonomous exploration framework proposed in this invention not only considers the UAV's position information and map environment when inputting to the decision module, but also considers the current speed direction and speed magnitude. The target output by the module is more suitable for the subsequent trajectory generation module and control module to execute, resulting in a smoother trajectory and more stable overall motion.

[0028] 2. This method performs hierarchical training on the target position and angle within the decision-making module. In the first stage, only the target position is trained using reinforcement learning. Simultaneously with the reinforcement learning task, supervised learning pre-training of the target angle is performed using a dataset constructed based on the selected target direction, obtaining the initial network parameters for the target angle network. Subsequently, the second-stage reinforcement learning task is executed, training both the target position and angle. The processed training exhibits less fluctuation and faster stabilization.

[0029] 3. In this method, to allow the agent to explore and update scenarios as much as possible in the early stages of training and fit multiple situations, entropy is incorporated as a part of the reward function and gradually reduced as training progresses. Different sparse rewards are designed to provide different overall rewards for tasks not completed by the maximum time step, planning task failure, and exploration completion, thereby improving the robustness of the method and increasing the stability and speed of training.

[0030] 4. A dynamic consistency selection extension method is introduced. This method avoids repetitive motion during exploration by incorporating a penalty term into the policy gradient. The penalty function incorporates the angle between the UAV's current speed and the target point as part of the evaluation cost, added midway through training. It also considers cases where the UAV is closer to the obstacle, extending the distance along the speed direction from the current position to the maximum extent. When a grid cell occupies a grid cell position, no penalty is imposed, which shortens the exploration path and improves exploration efficiency. Attached Figure Description

[0031] Figure 1 This is a flowchart of a method for autonomous exploration of drones that combines pre-trained reinforcement learning. Detailed Implementation

[0032] The present invention will be specifically described below through embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention, but are not limited thereto, unless otherwise stated.

[0033] Definitions:

[0034] Pre-training: In reinforcement learning, pre-training refers to initializing and training the agent's model using other methods or data before formally conducting reinforcement learning, in order to improve learning efficiency, accelerate convergence speed, and even improve final performance.

[0035] Occupied grid map: This is a probabilistic 2D / 3D map representation method used for robot navigation and environment modeling. It divides the environment into a regular grid, with each grid storing an occupancy probability, indicating whether the location is occupied by an obstacle. If occupied, it becomes an occupied grid. The occupied grid map is one of the core data structures in SLAM (Simultaneous Localization and Mapping) and path planning, and uses a binary Bayesian filter to dynamically update the occupancy probability of the grid.

[0036] Entropy: Information entropy originates from information theory and is used to quantify the uncertainty of random variables. The higher the entropy value, the greater the uncertainty (e.g., entropy is maximum in a uniform distribution); the lower the entropy value, the stronger the determinism (e.g., entropy is 0 in a one-hot distribution). Policy entropy measures the randomness of action selection and encourages exploration by maximizing policy entropy, avoiding premature convergence to a local optimum.

[0037] Autoregressive models: In neural networks, autoregressive models are models that progressively generate outputs, with subsequent steps relying on the results of previous steps. In the proposed method, the network that ultimately generates the target direction requires the target position as input, and the target position is output by the previous network structure.

[0038] Reward Function: In reinforcement learning, the reward function is the core driving force of the entire learning process, telling the agent what constitutes good behavior. Reinforcement learning itself is an algorithm that learns better strategies by continuously interacting with the environment and receiving rewards. Only by designing a purposeful and appropriate reward function can the agent ultimately complete the required task.

[0039] Rewards: The cumulative sum of rewards from the start to the end of the mission.

[0040] Value network: A neural network used to estimate the value of the current state or state-action pair, i.e., the expected long-term return.

[0041] Policy network: A neural network that directly outputs the probability distribution of an action (in a discrete action space) or the action itself (in a continuous action space).

[0042] Action space: The required target output. In the exploration task, the decision module needs the target's position and direction.

[0043] To address the issues of slow training speed and unsatisfactory final exploration efficiency in reinforcement learning methods, this paper proposes an improvement to the exploration method using policy gradient-based reinforcement learning. Figure 1 The reinforcement learning-based autonomous exploration method for UAVs, as shown, has the following specific steps:

[0044] Preparation: A set of deep neural network models was designed, using an autoregressive approach to interconnect the predictions of two actions. The policy's state value function and position / action selection share the same environment state encoding, which is then processed by their respective linear encodings to obtain the final value. Simultaneously, the position / action is passed to another encoder to obtain the environment feature encoding, which is then concatenated with the environment feature encoding and passed through a linear layer to obtain the final target direction. In the reinforcement learning-based UAV autonomous exploration model, the environmental observations at each time step are extracted as model input, divided into map information and UAV current state information, including position, direction, and velocity. The map information input is passed to a convolutional feature encoder. The purpose of feature extraction is to reduce the dimensionality of the original map information and encode the original input information into effective features for use by subsequent linear network layers. A corresponding dataset is constructed using the optimal orientation based on visual features in a grid-based system, using traditional methods, and includes only map information and UAV position information.

[0045] Step 101: Select the optimal direction based on visual features;

[0046] The specific method involves generating several sampling orientations at the target location based on the currently constructed environment map. For each orientation, the number of unknown grid cells or feature points covered is calculated by simulating the camera's field of view (frustum culling). All random orientations are then arranged, and the orientation with the most feature points corresponds to the optimal value under the current environmental conditions.

[0047] Step 102: Reinforce learning for the location of the decision target;

[0048] As the first stage of reinforcement learning training, after convolutional encoding of the map information, the linear layer, along with the UAV's current position, orientation, and velocity, is passed to the policy network and value network respectively. The value network is used to evaluate the value of the current state, while the policy network only generates position and action information about the target grid. In this stage, the target orientation is obtained through step 101.

[0049] Step 103, pre-training for the decision-making target direction;

[0050] While performing step 102, a directional state mapping dataset for map information and drone position is constructed using a drone exploration platform, and the target direction is trained separately. Similarly, map information is encoded, but the linear layer only retains the drone's position information and outputs the corresponding […]. The angle and direction are determined. Using the value obtained in step 101 as the target, the policy network for deciding the target direction is pre-trained.

[0051] Step 104, handling intrinsic entropy-assisted rewards and consistency selection;

[0052] In step 102, intrinsic entropy rewards and consistency selection processing are added in the early and late stages of training, respectively. Simultaneously, to allow the agent to explore and update scenarios as much as possible and fit multiple situations in the early stages of training, the entropy of actions can be... As part of the reward function, entropy and reward often exhibit a negative correlation; high rewards are often accompanied by low entropy (more deterministic strategies), while low rewards are accompanied by higher entropy (stronger exploratory nature). When the strategy has essentially converged, excessively high entropy can have adverse effects. Three termination conditions are set: reaching the maximum time limit, incomplete step-size task, and failed planning task. Completing the exploration task serves as an additional sparse reward, part of the overall return.

[0053] In the later stages of training (step 102), an additional consistency selection process is added to the policy gradient. The angle between the drone's current speed and the target point is used as part of the penalty evaluation of the overall state value. This is because exploration tasks at intersections and Y-shaped intersections often involve back-and-forth motion. To prevent this, large-scale changes in the direction of movement are avoided. Furthermore, considering that the drone will inevitably need to turn if it is already near an obstacle, the maximum extension along the velocity direction in the forward position is considered. If there is an existing cell occupying a cell position under a given cell, no operation will be performed on it.

[0054] Step 105: Complete the final training within the full motion space;

[0055] Once the training in steps 102 and 103 is stable, the target position obtained in step 102 is used as the input position information required for step 103, and reinforcement learning is performed again in an autoregressive model manner. During this stage, both the target position and target direction are obtained, and the previously trained network parameters are used as the initial values ​​for the current step. Further optimization through reinforcement learning makes the selection of the target direction more flexible.

[0056] This method designs a modular exploration framework that considers trajectory optimization results and overall execution efficiency in the decision-making task. A hybrid action decision-making method is designed to separate the training of the UAV's target position and angle, achieving hierarchical and stable training. While training the target position, a traditional angle selection method is used to pre-train the angle policy network. After these two parts of training stabilize, the pre-trained network parameters are used as initial values ​​for the final overall training, yielding complete target position and orientation outputs. Furthermore, an additional intrinsic entropy reward is introduced, providing additional reward functions for the probability of exploring new areas and different states at the end of exploration, improving the overall training speed. A consistent selection method is also provided to optimize the policy gradient results, reducing the number of large maneuvers performed by the UAV and improving overall exploration efficiency, addressing the problem of repeated back-and-forth movements during exploration.

[0057] The above embodiments are only used to illustrate the present invention. Any equivalent transformations and improvements made on the basis of the technical solutions of the present invention should not be excluded from the protection scope of the present invention.

Claims

1. A method for autonomous exploration of unmanned aerial vehicles (UAVs) combining pre-trained reinforcement learning, characterized in that, Includes the following steps: S1. Select the optimal direction based on visual features; S2. Reinforce learning based on the location of the decision-making target; After S2.1 performs convolutional encoding on the map information, the linear layer, together with the current position, orientation, and speed of the UAV, is passed to the policy network and value network respectively. S2.2 The value network is used to evaluate the value of the current state, while the policy network only generates positional actions for the target grid; the target direction at this stage is the optimal direction obtained in step S1. S3. Pre-training targeting decision-making objectives; S3.1 While performing step S2, a directional state mapping dataset for map information and drone position is constructed using the drone exploration platform, and the target direction is trained separately. S3.2 also encodes map information; the linear layer only retains the drone's position information and outputs the corresponding […]. The angle and direction; S3.3 Using the optimal direction obtained in step S1 as the target, pre-train the policy network for deciding the target direction; S4. Intrinsic Entropy-Assisted Rewards and Consistent Selection Processing; S4.1 In the early stages of training in step S2, the entropy of the actions can be... As part of the reward function, it is used to explore and update scenarios and fit multiple situations; S4.2 sets three termination conditions: the maximum time is reached, the step size task is not completed, and the planning task fails. Completing the exploration task is an additional sparse reward, which is part of the overall reward. In the mid-to-late stages of S4.3 training, an additional consistency selection process is added to the policy gradient, which uses the angle between the drone's current speed and the target point as part of the penalty evaluation of the overall state value. S5. Complete the final training in the full motion space; When the training in steps S2 and S3 is stable, the target position obtained in step S2 is used as the input of the position information required in step S3, and reinforcement learning is performed again in the form of an autoregressive model to obtain the target position and target direction.

2. The autonomous exploration method for unmanned aerial vehicles combining pre-trained reinforcement learning according to claim 1, characterized in that, The specific process of step S1 is as follows: Based on the currently constructed environment map, generate several sampling orientations at the target location. For each orientation, calculate the number of unknown grids or feature points it covers by simulating the field of view of the camera. Arrange all random orientations, and the orientation with the most feature points corresponds to the optimal orientation under the current environmental conditions.

3. The autonomous exploration method for unmanned aerial vehicles combining pre-trained reinforcement learning according to claim 1, characterized in that, In step S4, to avoid reciprocating motion, large-scale changes in the direction of motion are avoided; however, the motion extends most along the velocity direction at the forward position. If there is an existing cell occupying a cell position under a given cell, no operation will be performed on it.

Images