Mobile robot autonomous exploration method based on deep reinforcement learning

Abstract

The invention discloses a mobile robot autonomous exploration method based on deep reinforcement learning, and the method comprises the steps: starting laser radar equipment, a calculation unit and a mobile robot body power supply in an unknown environment, and collecting the point cloud data of the current environment through a laser radar; storing the point cloud data based on the complementary hole structure, constructing a three-dimensional point cloud map, and constructing a two-dimensional grid occupation map after projection; constructing an information graph based on the two-dimensional grid-occupying map, and sparsifying the information graph by using a bidirectional A * algorithm to form a sparse information graph; a reinforcement learning strategy network is constructed, a target function of the reinforcement learning strategy network is designed, the input of the reinforcement learning strategy network is a sparse information graph, and the output of the reinforcement learning strategy network is a target path point at the next moment; a path planner generates a collision-free path according to the target path point, and a local planner generates a linear velocity value and an angular velocity value of the current robot according to the collision-free path; a motion controller of the robot generates a motion command according to the speed value, the mobile robot is controlled to reach the target path point, and the steps are repeated until the robot autonomous exploration task based on deep reinforcement learning is completed; the method has the advantages that a model-free deep reinforcement learning technology is adopted to learn an exploration strategy in a simulation system, the method can be seamlessly transplanted to a real physical environment, higher-efficiency autonomous exploration can be realized, the adaptability to the environment is high, the application scene is wider, and a foundation is laid for completely autonomous operation of the mobile robot.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile robot autonomous exploration and deep reinforcement learning, and specifically relates to a mobile robot autonomous exploration method based on deep reinforcement learning. Background Art

[0002] Autonomous robotic exploration in unknown environments has long attracted attention in the robotics community, with potential applications in urban search and rescue. The benefits of using mobile robots for autonomous exploration include reducing the workload of rescuers and improving the efficiency of victim searches during time-sensitive missions. However, autonomous mobile robot exploration involves localization and navigation under uncertainty, which requires a model of the surrounding environment.

[0003] To address the problem of autonomous exploration in unknown environments, mobile robots need to plan their exploration paths appropriately to build a more complete map of the environment. SLAM methods have been developed to achieve self-localization and mapping during robotic exploration. Autonomous robotic exploration, also known as active SLAM, plans a non-myopic path, balancing the trade-off between exploiting the surrounding environment and exploring unknown areas.

[0004] The most common approach for autonomous exploration by mobile robots is the frontier-based exploration method. This method generates multiple candidate paths, each covering a frontier, and greedily selects the path with the greatest reward. However, as the scope of exploration of the environment increases, the computational requirements for identifying the frontier become high, which makes the overall planning time too long to meet real-time requirements. In recent years, with the development of machine learning technology in the field of robotics, deep reinforcement learning (DRL) methods have been used for autonomous exploration by robots. In these methods, state features are automatically learned, and the optimal strategy is continuously learned through iterative interaction with the environment. Deep reinforcement learning methods do not require a large amount of dataset labeling, thereby reducing the cost of learning and training.

[0005] Patent 115963832A discloses a robot autonomous exploration method based on deep reinforcement learning. This method uses model-free reinforcement learning technology based on Voronoi diagrams to learn exploration strategies during trial-and-error interaction with physical or simulated systems, breaking the dimensionality curse of the action space and observation space without losing the structural properties of the workspace. However, the reinforcement learning reward mechanism in this scheme does not take into account the angular relationship between the current viewpoint and the next viewpoint, making the robot prone to sharp turns and making its path uneven; Patent CN114418126A discloses a deep reinforcement learning exploration method based on information theory. This method uses an information-theoretic approach to measure the exploration ability of a strategy, effectively improving the exploration efficiency of the reinforcement learning algorithm. However, this scheme does not take into account historical state information, which may cause the robot to explore the same area back and forth to a certain extent.

[0006] The journal paper "Deep Reinforcement Learning-based Large-scale Robot Exploration" (IEEE Robotics and Automation Letters, DOI: 10.1109 / LRA.2024.3379804.) proposed a deep reinforcement learning-based reaction planner to solve the problem of large-scale laser-based autonomous robot exploration in a two-dimensional action space, using the basic SAC (Soft Actor-Critic) algorithm and a distance-considered reward function; the conference paper "ARiADNE: A Reinforcement learning approach using Attention-based Deep Networks for Exploration" (2023 IEEE International Conference on Robotics and Automation (ICRA), DOI: 10.1109 / ICRA48891.2023.10160565.) proposed a path planning method based on an attention mechanism and implicitly predicted and explored the potential benefits of unknown areas. A multi-head attention mechanism was used to extract graph features, but this method has not been applied to real mobile robots. Compared with the above papers, the innovation of this invention lies in the different construction of the reinforcement learning network structure and its reward function. Specifically, the original node features are enhanced in the network encoder structure and the historical state of the mobile robot is fully considered. At the same time, a strategic neural network based on the graph self-attention mechanism is proposed, which allows the robot to reason about its environment map at multiple spatial scales. The reward function includes internal and external rewards, taking into account the distance and angle difference between the robot's posture and the next waypoint, as well as curiosity based on explored and unexplored areas. Through these two points, the method of the present invention significantly reduces the situation in which mobile robots are prone to repeatedly exploring known areas and the randomness of target point selection in the original technology, thereby improving the efficiency of mobile robots in autonomous exploration in unknown environments. Summary of the Invention

[0007] The technical problem to be solved by the present invention is that the existing robot autonomous exploration method is prone to repeated exploration of known areas, resulting in low efficiency, and large-scale mapping consumes a lot of computing resources and memory, and relies on environmental models, resulting in poor stability of search strategies in real scenarios. The present invention proposes a mobile robot autonomous exploration method based on deep reinforcement learning, aiming to further improve the exploration efficiency and stability of the autonomous exploration method.

[0008] The present invention solves the above technical problems through the following technical means: a mobile robot autonomous exploration method based on deep reinforcement learning, the method comprising the following steps:

[0009] Step S1: In an unknown environment, turn on the laser radar device, computing unit, and mobile robot body power, and collect point cloud data of the current environment through the laser radar;

[0010] Step S2: storing point cloud data based on the complementary hole structure, constructing a three-dimensional point cloud map and projecting it to construct a two-dimensional grid map;

[0011] Step S3: constructing an information graph based on the two-dimensional grid map, and using a bidirectional A* algorithm to thin out the information graph to form a sparse information graph;

[0012] Step S4: constructing a reinforcement learning strategy network and designing an objective function of the reinforcement learning strategy network, wherein the input of the reinforcement learning network is a sparse information graph and the output is a target path point at the next moment;

[0013] Step S5: The path planner generates a collision-free path according to the target path point, and the local planner generates the linear velocity and angular velocity values ​​of the current robot according to the collision-free path;

[0014] Step S6: The robot's motion controller generates motion commands based on the speed value, controls the mobile robot to reach the target path point, and repeats the above steps until the robot's autonomous exploration task based on deep reinforcement learning is completed.

[0015] Furthermore, the step S2 of storing point cloud data based on the complementary hole structure, constructing a three-dimensional point cloud map, and projecting it to construct a two-dimensional grid map specifically includes the following:

[0016] The original lidar data is input into the point cloud data storage module, and the 3D SLAM generates a 3D point cloud map, which is then projected into a 2D grid map. The grid data is then stored using a complementary hole structure, and a voxel filter is used to reduce the point cloud density. Finally, the local grid based on the complementary hole structure is continuously updated and merged to form a global 2D occupancy map (M g );

[0017] Furthermore, the step S3 described in which the bidirectional A* algorithm is used to thin out the information graph to form a sparse information graph specifically includes the following contents:

[0018] Consider a global occupancy grid map (M g ) represents a bounded unknown environment from which a collision-free graph can be extracted in is a set of nodes and ε=((v1,v2),...,(vm-1 ,v m )) is a collision-free edge set, and the nodes evenly cover the free area. For each node, determine the 8 nearest neighbor nodes and check whether they collide to construct a collision-free boundary set. Expand the collision-free graph and extract an information graph The figure and Share the same set of non-collision boundaries. different, Each node in the has two properties: the utility value of the node (from position v i The number of boundaries that can be observed in the line of sight) and the landmark value (representing v i binary value of whether it has been explored).

[0019] No covering node set from The sparse nodes and the covered node set are initialized; then, nearby nodes are found within the radius threshold; subsequently, the covered node set is updated by checking whether there is a collision between the nodes in the uncovered set and the nearby node set; by using the bidirectional A* algorithm, the search from the current position to path and set the reference node; next, verify whether there are obstacles between each node in the optimal path and the reference node, or evaluate whether the distance between these two types of nodes exceeds a predefined threshold, and then update the sparse node set; finally, construct a sparse information graph as the input of the policy network.

[0020] Furthermore, the step S4 of constructing the reinforcement learning strategy network specifically includes the following:

[0021] The reinforcement deep learning network consists of an actor network and a critic network, both with encoders and decoders. The actor network uses a pointer layer, which outputs attention weights as a policy for neighboring nodes; the actor network only chooses one of the neighboring nodes as the next navigation target. In addition, long short-term memory (LSTM) units are used to ensure that the network also considers the previous state characteristics of the mobile robot to make better decisions.

[0022] The actor network uses two symmetrical value evaluation networks to evaluate the vehicle's decision. Ultimately, two Q values ​​are obtained and the smaller value between them is selected as the final evaluation result to prevent the problem of overestimating the Q value.

[0023] Graph Attention Layer: For the initial input, the self-attention mechanism is first used to calculate the similarity index and establish connections within the graph structure. Unlike the traditional self-attention mechanism, the feature expression of a node considering the adjacent context is obtained by combining and summing. The reason why K is used instead of Q is because the surrounding relationships need to be used to deduce the correct value of the original node. Therefore, after passing it to the graph attention layer, the updated features can be calculated as follows:

[0024]

[0025] in represents the learnable matrix, represents the adjacency matrix as the edge mask, and softmax(·) represents the activation function.

[0026] Encoder: The encoder models the observed environment by learning the dependencies between nodes in the graph. Embedded into node features Then, the node features are normalized (h i ') and input to the graph attention layer. In each graph attention layer, each node in the graph can only access the features of its neighboring nodes. To enable the model to capture dependencies between non-adjacent nodes, we stack four GA layers in the encoder, each layer taking the output of the previous layer as input. In the graph attention layer, an LSTM module is introduced at the end of the encoder to form enhanced features for the current node. The introduction of the LSTM block gives the network a memory function, enabling it to comprehensively utilize historical and current information to make more reasonable decisions. The output of the encoder consists of a set of enhanced node features, each of which represents the dependency between the current node and other nodes.

[0027] Decoder: The input of the decoder is the enhanced current node feature (h ec ) and all node features (h a ). By normalizing the above two features, a new feature (h ec ' and h a '), where h ec ' as query source and h a ' passed to the graph attention layer as a source of keys and values; the decoder output and h ec Connect them to generate a d-dimensional feature (h c '). Then, select its neighbor node features As key source and h c ' is passed to the pointer layer as a query source. After the pointer layer, the final dimension of the strategy corresponds to the number of neighbor nodes. The output of the pointer layer passes through the embedding layer and the squeeze operation, and the attention weight is directly output as the strategy π(a i|z t )=p(v i |z t ),(p t ,v i )∈ε.

[0028] Furthermore, the objective function of the reinforcement learning strategy network described in step S4 specifically includes the following contents:

[0029] The goal of the reinforcement learning policy network is to balance the trade-off between maximizing rewards and policy entropy in the current exploration state to obtain the optimal strategy for generating the next waypoint:

[0030]

[0031] where π * represents the optimal strategy, π is the strategy, z t represents the observation state at time step t, a t is the action at time step t, ρ π represents the trajectory distribution caused by the strategy, T represents the number of time steps, γ t ∈[0,1] is the discount rate at time step t, r(z t ,a t ) is state z t and action a t The reward, α represents the temperature parameter, The entropy of the policy when representing the state.

[0032] This paper proposes an optimized reward that takes into account the distance and angle difference between the vehicle's position and the next waypoint, as well as curiosity based on explored and unexplored areas. The exploration ability of the mobile robot is mainly reflected in the policy, which is expressed by the following reward:

[0033] r(z t ,a t )=w D r D (z t ,a t )+w Λ r Λ (z t ,a t )+w F r F (z t ,a t )+w C r C (z t ,a t )+r E (z t ,a t )

[0034] where w D 、w Λ 、w F and w C is the weight parameter of each sub-reward, and the reward function is divided into five parts, among which the reward r D ,r Λ ,r F and r E can be considered as an external reward, while reward r C Can be considered an intrinsic reward.

[0035] Considering the distance, the reward function with additional distance cost (r D ), which is defined as follows:

[0036]

[0037] in and are the positions of the current state and the next state, respectively, and d(·) represents the Euclidean distance between the two positions.

[0038] In order to make the search path smoother, the angle cost (r Λ ) is defined as follows:

[0039]

[0040] where Λ(·) represents the yaw angle difference between the two positions.

[0041] In order to enable the mobile robot to explore more unknown areas, the reward function (r F ), which is defined as follows:

[0042] r F (z t ,a t )=n F

[0043] where n F Represents the number of observed boundaries associated with the next new state.

[0044] In order to reduce repeated exploration of mobile robots, the intrinsic reward based on curiosity (r C ) is designed as follows:

[0045]

[0046] Where d1 and T0 represent the distance and step thresholds respectively, P history Represents all positions in the historical state set. If and Phistory If the Euclidean distance between them is greater than d1, the mobile robot receives a positive reward. In the early stages of exploration (timestep ≤ T), there are many unexplored areas, and intrinsic rewards are easy to obtain. However, as exploration progresses (timestep > T0), the unknown area will continue to decrease, so the intrinsic reward is doubled to reduce the vehicle's repeated exploration of known areas.

[0047] When the vehicle exploration is completed, the completion reward of the mobile robot’s autonomous exploration task (r E )as follows:

[0048]

[0049] η r (z t ) represents state z t The exploration area rate under .

[0050] Furthermore, the training process of the reinforcement learning network described in step S4 specifically includes the following:

[0051] Training the policy directly from real-world environment information is not feasible because there is a lot of noise in the real environment, so accurate terrain information cannot be obtained. To solve this problem, an actor-critic framework is utilized, where the actor network assists in training the policy network but is not needed after training. This setup enables the robot to use the real state as observations for the critic network. Specifically, the structure of the ordinary critic network will be preserved and will be derived from the s t and z t Building a ground truth map is also an infographic used as input to a policy network, but The node set in Covering the entire free area, the node features (f i ) is a binary value used to represent v' i Whether it is in an area that the mobile robot has explored.

[0052] This paper combines Prioritized Experience Replay (PER) with the actor-critic network. This integration enables the agent to recognize the importance of experience samples and prioritize frequent sampling of high-value samples, thereby accelerating the convergence of the algorithm. In order to reduce overestimation and improve training stability, the actor-critic network uses two Q networks and uses a smaller Q value to calculate the gradient. At the same time, the actor-critic network also contains a policy network, so it is necessary to consider the TD errors of the three networks simultaneously. The TD error of the sampled transition i (δ i ) can be defined as:

[0053] δi =|T e (Q1)|+|T e (Q2)|+αT e (π)|

[0054] Where T e (·) represents the error of each network, and α represents the adjustment parameter used to balance the TD error weights calculated by the policy network and the Q-value network.

[0055] PER utilizes random priorities and generates a probability of sampling transition i, which is given by:

[0056]

[0057] p i =|δ i |+ε

[0058] Among them, p i represents the empirical priority, β∈[0,1] represents the priority control coefficient, and ε usually represents a small positive value to ensure that the empirical samples with TD-error of 0 are included.

[0059] The implementation of the probabilistic sampling mechanism ensures that even experiences with small TD errors can be sampled, thereby maintaining sample diversity during algorithm training. Since priority replay changes the sampling frequency of experiences, it is crucial to introduce importance sampling to adjust the error weight when calculating the gradient. The importance sampling weight (w i ) is calculated as:

[0060]

[0061] Where M represents the capacity of the replay buffer. Indicates the degree of control correction.

[0062] Finally, the proposed network trains two critic networks to estimate the soft action state value, and the loss function used is as follows:

[0063] where l t Represents the length of the training data, and d represents the signal that determines whether the current state is the terminal state. The target state value can be defined as:

[0064]

[0065] Where α represents the temperature parameter.

[0066] Policy Network π θ The policy trained to output the maximum expected state-action value is the following:

[0067]

[0068] The temperature parameters are automatically adjusted during training, and the temperature loss is calculated as follows:

[0069]

[0070] in represents the target entropy.

[0071] Furthermore, the input of the reinforcement learning network is a sparse information graph, and the output is the target path point at the next moment, which specifically includes the following contents:

[0072] The network takes a sparse information graph as input and outputs the next pathpoint, selected from the eight nearest neighbors of the current location. Each network consists of three encoders and a decoder, with the decoder's basic network layer being the graph attention layer. Specifically, it consists of a policy network (actor) and a critic network (critic). The policy network takes the current state as input and outputs the action probability distribution function of the eight nearest neighbors. The critic network does not participate in inference but only participates in model training, evaluating the quality of actions during training.

[0073] The beneficial effects of the present invention are:

[0074] 1. The dense graph sparse algorithm based on complementary hole structure proposed in this paper achieves low computational cost preprocessing;

[0075] 2. The optimized reward function proposed in this paper, including internal and external rewards, makes mobile robot exploration more efficient;

[0076] 3. The proposed graph-based policy neural network with a self-attention mechanism effectively learns features at different scales and has strong generalization capabilities across various environments.

[0077] 4. The autonomous exploration method proposed in this paper is effective and flexible in various scenarios. Even though the model in this method is only trained on the Dungeon dataset, no additional adjustments are required when applied to real scenarios.

[0078] In summary, compared with the existing technology, the present invention can achieve more efficient autonomous exploration, has strong adaptability to the environment, and has a wider range of application scenarios, laying the foundation for realizing fully autonomous operation of mobile robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a flow chart of a mobile robot autonomous exploration method based on deep reinforcement learning of the present invention;

[0080] Figure 2 is a schematic diagram of the actor-critic network structure of the present invention;

[0081] Figure 3 This is a visualization of the attention weights of the critic network decoder on the Dungeon dataset of the present invention;

[0082] Figure 4 is the exploration path result diagram of the present invention in the simulation environment, where Figure 4 (a) is the exploration path result diagram in a large-scale indoor office simulation environment. Figure 4 (b) is the exploration result diagram in a large-scale outdoor campus simulation environment;

[0083] Figure 5 This is a graph of various indicators of the present invention in a large-scale indoor office simulation environment, where Figure 5 (a) is the curve of the detection volume during the exploration process. Figure 5 (b) is the exploration length curve during the exploration process. Figure 5 (c) is the running time curve of each method during the exploration process;

[0084] Figure 6 This is a graph of lidar processing time. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings and embodiments.

[0086] like Figure 1 The method of the present invention is shown as a flow chart, and the method comprises the following steps:

[0087] Step S1: In an unknown environment, turn on the laser radar device, computing unit, and mobile robot body power, and collect point cloud data of the current environment through the laser radar;

[0088] Step S2: storing point cloud data based on the complementary hole structure, constructing a three-dimensional point cloud map and projecting it to construct a two-dimensional grid map;

[0089] Step S3: constructing an information graph based on the two-dimensional grid map, and using a bidirectional A* algorithm to thin out the information graph to form a sparse information graph;

[0090] Step S4: constructing a reinforcement learning strategy network and designing an objective function of the reinforcement learning strategy network, wherein the input of the reinforcement learning network is a sparse information graph and the output is a target path point at the next moment;

[0091] Step S5: The path planner generates a collision-free path according to the target path point, and the local planner generates the linear velocity and angular velocity values ​​of the current robot according to the collision-free path;

[0092] Step S6: The robot's motion controller generates motion commands based on the speed value, controls the mobile robot to reach the target path point, and repeats the above steps until the robot's autonomous exploration task based on deep reinforcement learning is completed.

[0093] Furthermore, the step S2 of storing point cloud data based on the complementary hole structure, constructing a three-dimensional point cloud map, and projecting it to construct a two-dimensional grid map specifically includes the following:

[0094] The original LiDAR data is input into the point cloud data storage module, and the 3D SLAM generates a 3D point cloud map. The 3D point cloud map is then projected into a grid map. The grid data is then stored using a complementary hole structure, and a voxel filter is used to reduce the point cloud density. Finally, the local grid based on the complementary hole structure is continuously updated and merged to form a global 2D occupancy map (M g );

[0095] Furthermore, the step S3 described in which the bidirectional A* algorithm is used to thin out the information graph to form a sparse information graph specifically includes the following contents:

[0096] Consider a global occupancy grid map (M g ) represents a bounded unknown environment from which a collision-free graph can be extracted in is a set of nodes and ε=((v1,v2),...,(v m-1 ,v m )) is a collision-free edge set, and the nodes evenly cover the free area. For each node, determine the 8 nearest neighbor nodes and check whether they collide to construct a collision-free boundary set. Expand the collision-free graph and extract an information graph The figure and Share the same set of non-collision boundaries. different, Each node in the has two properties: the utility value of the node (from position v i The number of boundaries that can be observed in the line of sight) and the landmark value (representing v i binary value of whether explored).

[0097] No covering node set from The sparse nodes and the covered node set are initialized; then, nearby nodes are found within the radius threshold; subsequently, the covered node set is updated by checking whether there is a collision between the nodes in the uncovered set and the nearby node set; by using the bidirectional A* algorithm, the search from the current position to path and set the reference node; next, verify whether there are obstacles between each node in the optimal path and the reference node, or evaluate whether the distance between these two types of nodes exceeds a predefined threshold, and then update the sparse node set; finally, construct a sparse information graph as the input of the policy network.

[0098] Furthermore, the step S4 of constructing the reinforcement learning strategy network specifically includes the following:

[0099] The reinforcement deep learning network consists of an actor network and a critic network, both with encoders and decoders. The actor network uses a pointer layer, which outputs attention weights as a policy for neighboring nodes; the actor network only chooses one of the neighboring nodes as the next navigation target. In addition, long short-term memory (LSTM) units are used to ensure that the network also considers the previous state characteristics of the mobile robot to make better decisions.

[0100] The actor network uses two symmetrical value evaluation networks to evaluate the vehicle's decision. Ultimately, two Q values ​​are obtained and the smaller value between them is selected as the final evaluation result to prevent the problem of overestimating the Q value.

[0101] Graph Attention Layer: For the initial input, the self-attention mechanism is first used to calculate the similarity index and establish connections within the graph structure. Unlike the traditional self-attention mechanism, the feature expression of a node considering the adjacent context is obtained by combining and summing. The reason why K is used instead of Q is because the surrounding relationships need to be used to deduce the correct value of the original node. Therefore, after passing it to the graph attention layer, the updated features can be calculated as follows:

[0102]

[0103] in represents the learnable matrix, represents the adjacency matrix as the edge mask, and softmax(·) represents the activation function.

[0104] Encoder: The encoder models the observed environment by learning the dependencies between nodes in the graph. Embedded into node features Then, the node features are normalized (h i') and input to the graph attention layer. In each graph attention layer, each node in the graph can only access the features of its neighboring nodes. To enable the model to capture dependencies between non-adjacent nodes, we stack four GA layers in the encoder, each layer taking the output of the previous layer as input. In the graph attention layer, an LSTM module is introduced at the end of the encoder to form enhanced features for the current node. The introduction of the LSTM block gives the network a memory function, enabling it to comprehensively utilize historical and current information to make more reasonable decisions. The output of the encoder consists of a set of enhanced node features, each of which represents the dependency between the current node and other nodes.

[0105] Decoder: The input of the decoder is the enhanced current node feature (h ec ) and all node features (h a ). By normalizing the above two features, a new feature (h ec ' and h a '), where h ec ' as query source and h a ' passed to the graph attention layer as a source of keys and values; the decoder output and h ec Connect them to generate a d-dimensional feature (h c '). Then, select its neighbor node features As key source and h c ' is passed to the pointer layer as a query source. After the pointer layer, the final dimension of the strategy corresponds to the number of neighbor nodes. The output of the pointer layer passes through the embedding layer and the squeeze operation, and the attention weight is directly output as the strategy π(a i |z t )=p(v i |z t ),(p t ,v i )∈ε.

[0106] Furthermore, the objective function of the reinforcement learning strategy network described in step S4 specifically includes the following contents:

[0107] The goal of the reinforcement learning policy network is to balance the trade-off between maximizing rewards and policy entropy in the current exploration state to obtain the optimal strategy for generating the next waypoint:

[0108]

[0109] where π * represents the optimal strategy, π is the strategy, z t represents the observation state at time step t, a t is the action at time step t, ρ π represents the trajectory distribution caused by the strategy, T represents the number of time steps, γt ∈[0,1] is the discount rate at time step t, r(z t ,a t ) is state z t and action a t The reward, α represents the temperature parameter, The entropy of the policy when representing the state.

[0110] This paper proposes an optimized reward that takes into account the distance and angle difference between the vehicle's position and the next waypoint, as well as curiosity based on explored and unexplored areas. The exploration ability of the mobile robot is mainly reflected in the policy, which is expressed by the following reward:

[0111] r(z t ,a t )=w D r D (z t ,a t )+w Λ r Λ (z t ,a t )+w F r F (z t ,a t )+w C r C (z t ,a t )+r E (z t ,a t )

[0112] where w D 、w Λ 、w F and w C is the weight parameter of each sub-reward, and the reward function is divided into five parts, among which the reward r D ,r Λ ,r F and r E can be considered as an external reward, while the reward r C Can be considered an intrinsic reward.

[0113] Considering the distance, the reward function with additional distance cost (r D ), which is defined as follows:

[0114]

[0115] in and are the positions of the current state and the next state, respectively, and d(·) represents the Euclidean distance between the two positions.

[0116] In order to make the search path smoother, the angle cost (r Λ ) is defined as follows:

[0117]

[0118] where Λ(·) represents the yaw angle difference between the two positions.

[0119] In order to enable the mobile robot to explore more unknown areas, the reward function (r F ), which is defined as follows:

[0120] r F (z t ,a t )=n F

[0121] where n F Represents the number of observed boundaries associated with the next new state.

[0122] In order to reduce repeated exploration of mobile robots, the intrinsic reward based on curiosity (r C ) is designed as follows:

[0123]

[0124] Where d1 and T0 represent the distance and step thresholds respectively, P history Represents all positions in the historical state set. If and P history If the Euclidean distance between them is greater than d1, the mobile robot receives a positive reward. In the early stages of exploration (timestep ≤ T), there are many unexplored areas, and intrinsic rewards are easy to obtain. However, as exploration progresses (timestep > T0), the unknown area will continue to decrease, so the intrinsic reward is doubled to reduce the vehicle's repeated exploration of known areas.

[0125] When the vehicle exploration is completed, the completion reward of the mobile robot’s autonomous exploration task (r E )as follows:

[0126]

[0127] η r (z t ) represents state z t The exploration area rate under .

[0128] Furthermore, the training process of the reinforcement learning network described in step S4 specifically includes the following:

[0129] Training the policy directly from real-world environment information is not feasible because there is a lot of noise in the real environment, so accurate terrain information cannot be obtained. To solve this problem, an actor-critic framework is utilized, where the actor network assists in training the policy network but is not needed after training. This setup enables the robot to use the real state as observations for the critic network. Specifically, the structure of the ordinary critic network will be preserved and will be derived from the s t and z t Building a ground truth map is also an infographic used as input to a policy network, but The node set in Covering the entire free area, the node features (f i ) is a binary value used to represent v' i Whether it is in an area that the mobile robot has explored.

[0130] This paper combines Prioritized Experience Replay (PER) with the actor-critic network. This integration enables the agent to recognize the importance of experience samples and prioritize frequent sampling of high-value samples, thereby accelerating the convergence of the algorithm. In order to reduce overestimation and improve training stability, the actor-critic network uses two Q networks and uses a smaller Q value to calculate the gradient. At the same time, the actor-critic network also contains a policy network, so it is necessary to consider the TD errors of the three networks simultaneously. The TD error of the sampled transition i (δ i ) can be defined as:

[0131] δ i =|T e (Q1)|+|T e (Q2)|+α|T e (π)|

[0132] Where T e (·) represents the error of each network, and α represents the adjustment parameter used to balance the TD error weights calculated by the policy network and the Q-value network.

[0133] PER utilizes random priorities and generates a probability of sampling transition i, which is given by:

[0134]

[0135] p i =|δ i |+ε

[0136] Among them, p irepresents the empirical priority, β∈[0,1] represents the priority control coefficient, and ε usually represents a small positive value to ensure that the empirical samples with TD-error of 0 are included.

[0137] The implementation of the probabilistic sampling mechanism ensures that even experiences with small TD errors can be sampled, thereby maintaining sample diversity during algorithm training. Since priority replay changes the sampling frequency of experiences, it is crucial to introduce importance sampling to adjust the error weight when calculating the gradient. The importance sampling weight (w i ) is calculated as:

[0138]

[0139] Where M represents the capacity of the replay buffer. Indicates the degree of control correction.

[0140] Finally, the proposed network trains two critic networks to estimate the soft action state value, and the loss function used is as follows:

[0141] where l t Represents the length of the training data, and d represents the signal that determines whether the current state is the terminal state. The target state value can be defined as:

[0142]

[0143] Where α represents the temperature parameter.

[0144] Policy Network π θ The policy trained to output the maximum expected state-action value is the policy loss:

[0145]

[0146] The temperature parameters are automatically adjusted during training, and the temperature loss is calculated as follows:

[0147]

[0148] in represents the target entropy.

[0149] Figure 2This is a schematic diagram of the actor-critic network structure of the present invention. The network input is a sparse information graph, and the output is the next path point, selected from the eight nearest neighbors of the current position. Each network entity includes three encoders and a decoder, and the basic network layer of the decoder is the graph attention layer. Specifically, it is divided into a policy network actor and a criticism network critic. The policy network input is the current state and the output is the action probability distribution function of the eight nearest neighbors. The criticism network does not participate in the inference process, but only participates in model training during training to evaluate the quality of actions during training.

[0150] Figure 3 This is a visualization of the attention weights of the critic network decoder of our invention on the Dungeon dataset. It can be observed that in some attention heads, the model focuses on areas that may lack connections, which shows that the model predicts the structure of unknown areas based on information from known areas. Specifically, when the first and second attention heads notice that two corridors (below) may be connected, they adjust the attention weights. As a result, our model is able to better learn the dependencies between different areas, enabling it to reason about the entire local map and avoid short-sighted decisions.

[0151] Figure 4 is the exploration path result diagram of the present invention in the simulation environment, where Figure 4 (a) shows the exploration path results in a large-scale indoor office simulation environment. DSVP achieved autonomous exploration in 828 seconds, with a total travel distance of 1355 meters; TARE achieved autonomous exploration in 644 seconds, with a total travel distance of 1186 meters; DRL achieved autonomous exploration in 579 seconds, with a total travel distance of 1015 meters; the human-based method achieved autonomous exploration in 494 seconds, with a total travel distance of 857 meters; and our method achieved autonomous exploration in 528 seconds, with a total travel distance of 961 meters. Compared with current state-of-the-art technologies, our method achieved an 8.81% faster exploration time and a 5.32% shorter exploration path than DRL. Compared with the TARE algorithm, our method achieved an 18.01% faster exploration time and an 18.97% shorter exploration path. Furthermore, our method exhibited significant advantages over DSVP in terms of exploration time and trajectory length (36.23% time efficiency and 29.08% distance efficiency). DSVP's exploration strategy develops an RRT, which generates inferior zigzag paths due to surrounding symmetry. This situation will lead to redundant paths, thus reducing the exploration efficiency. Figure 4(b) shows the exploration results in a large-scale outdoor campus simulation environment. DSVP achieved autonomous exploration in 994 seconds, with a total travel distance of 1828 meters; TARE achieved autonomous exploration in 1020 seconds, with a total travel distance of 1873 meters; DRL achieved autonomous exploration in 1280 seconds, with a total travel distance of 1916 meters; the human practice method achieved exploration in 710 seconds, with a total travel distance of 1321 meters; and the method of the present invention achieved autonomous exploration in 747 seconds, with a total travel distance of 1439 meters. Compared with current state-of-the-art technologies, the exploration time of the present invention is 41.64% faster than that of DRL, and the exploration path is 24.90% shorter. Compared with the TARE algorithm, the exploration time of the present invention is 26.76% faster, and the exploration distance is 23.17% shorter. In addition, the method of the present invention has significant advantages over DSVP in terms of exploration time and trajectory length (time efficiency of 24.84% and distance efficiency of 21.28%). The method of the present invention is very close to the human practice method in terms of time and length efficiency, indicating that it is possible to achieve fully autonomous exploration of mobile robots without human control.

[0152] Figure 5 This is a graph of various indicators of the present invention in a large-scale indoor office simulation environment, where Figure 5 (a) is the curve of the detection volume during the exploration process. Figure 5 (b) is the exploration length curve during the exploration process. Figure 5 (c) is a graph showing the running time of each method during the exploration process. As can be seen from the graph, compared with other advanced methods, the method proposed in this invention has lower time cost, shorter exploration path, and shorter time to complete the exploration of the environment, thus achieving efficient autonomous exploration.

[0153] Figure 6 This is a graph of lidar processing time, which includes collecting the raw point cloud, downsampling, and generating the final information map. For the same environment, using a dense graph sparse algorithm combined with a complementary hole structure results in a smoother and lowest processing time. Compared to other methods, our method can reduce time costs by up to 57.4% on average.

[0154] Through the above technical solutions, the present invention enables autonomous exploration of mobile robots in unknown environments. Specifically, a dense graph sparsification algorithm based on complementary holes is proposed to accelerate the processing of point cloud data, making the method applicable to both small and large environments. Compared with other methods, the proposed method can reduce the time cost of lidar processing by up to 57.4% on average. A graph-based policy neural network with a self-attention mechanism is designed to make further non-myopic decisions and avoid redundant actions. This network enables the proposed model to effectively predict the structure of unknown areas from maps of known areas. An optimized reward function is proposed that considers the distance and angle difference between the robot pose and the next waypoint, as well as curiosity based on explored and unexplored areas, to further improve the mobile robot's exploration capability. The maximum average total running time of the proposed method is controlled within 300ms, meeting the real-time requirements of low-speed mobile robot exploration systems. In addition, the proposed method can be seamlessly transferred from simulators to complex real-world environments without specific fine-tuning, demonstrating its strong generalization ability.

[0155] The examples of the present invention are described above in conjunction with the accompanying drawings. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Anyone familiar with this technology can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the definition of the claims.

Claims

1. A mobile robot autonomous exploration method based on deep reinforcement learning, characterized in that: The following steps are involved: Step S1: In an unknown environment, turn on the laser radar device, the computing unit and the power supply of the mobile robot body, and collect point cloud data of the current environment through the laser radar; Step S2: storing point cloud data based on the complementary hole structure, constructing a three-dimensional point cloud map and constructing a two-dimensional grid map after projection; Step S3: constructing an information graph based on the two-dimensional grid map, and using a bidirectional A* algorithm to sparse the information graph to form a sparse information graph; Step S4: construct a reinforcement learning network and design a reward function of the reinforcement learning network, wherein the input of the reinforcement learning network is a sparse information graph and the output is a target path point at the next moment; The reinforcement deep learning network consists of an actor network and a critic network, both of which have encoders and decoders. The actor network uses a pointer layer to output attention weights as a strategy for neighboring nodes and selects one from the neighboring nodes as the next navigation target point. In addition, a long short-term memory (LSTM) unit is used in the encoder to ensure that the network also considers the previous state features of the mobile robot to make better decisions; the encoder and decoder are constructed by three graph attention layers and one graph attention layer, respectively, where the graph attention layer calculates the updated features in the following way: in represents the learnable matrix, represents the adjacency matrix as the edge mask, and softmax(·) represents the activation function; The reward function of the reinforcement learning network is expressed as follows: r(z t ,a t )=w D r D (z t ,a t )+w Λ r Λ (z t ,a t )+w F r F (z t ,a t )+w C r C (z t ,a t )+r E (z t ,a t ) where w D 、w Λ 、w F and w C is the weight parameter of each sub-reward. The reward function is divided into five parts, among which the reward r D ,r Λ ,r F and r E can be considered as an external reward, while the reward r C Can be seen as an intrinsic reward; Step S5: The path planner generates a collision-free path according to the target path point, and the local planner generates the linear velocity and angular velocity values ​​of the current robot according to the collision-free path; Step S6: The robot's motion controller generates a motion command based on the speed value, controls the mobile robot to reach the target path point, and repeats the above steps until the robot's autonomous exploration task based on deep reinforcement learning is completed.

2. The mobile robot autonomous exploration method based on deep reinforcement learning according to claim 1 is characterized in that: The process of step S2 is: The original lidar data is input into the point cloud data storage module, and the three-dimensional SLAM generates a 3D point cloud map, which is then projected into a 2D grid map. Secondly, a complementary hole structure with a resolution of 2 is used to store the grid data, and a voxel filter is used to reduce the point cloud density. Finally, the local grid based on the complementary hole structure is continuously updated and merged to form a global two-dimensional occupancy map.

3. The mobile robot autonomous exploration method based on deep reinforcement learning according to claim 1 is characterized in that: The process of step S3 is as follows: No Covering Node Set From the sparse set of nodes The sparse nodes and the covered node set are initialized; then, nearby nodes are found within the radius threshold; subsequently, the covered node set is updated by checking whether there is a collision between the nodes in the uncovered set and the nearby node set; by using the bidirectional A* algorithm, the search from the current position to , and set the reference node; next, verify whether there are obstacles between each node in the best path and the reference node, or evaluate whether the distance between these two types of nodes exceeds a predefined threshold, and then update the sparse node set; finally, construct a sparse information graph as the input of the policy network.

4. The mobile robot autonomous exploration method based on deep reinforcement learning according to claim 1, characterized in that: The process of constructing the reinforcement learning strategy network described in step S4 is: Encoder: The encoder models the observed environment by learning the dependencies between nodes in the graph. Embedded into node features Then, the node features are normalized (h i ') and input to the graph attention layer. In each graph attention layer, each node in the graph can only access the features of its adjacent nodes. In order to enable the model to capture the dependencies between non-adjacent nodes, four graph attention layers are stacked in the encoder, and each layer takes the output of the previous layer as input; in the graph attention layer, an LSTM module is introduced at the end of the encoder to form enhanced current node features. The introduction of the LSTM block gives the network a memory function, enabling it to make more reasonable decisions by comprehensively utilizing historical and current information; the output of the encoder consists of a set of enhanced node features, each of which represents the dependency between the current node and other nodes; Decoder: The input of the decoder is the enhanced current node feature (h ec ) and all node features (h a ), by normalizing the above two features to form a new feature (h ec ' and h a '), where h ec ' as query source and h a ' passed to the graph attention layer as a source of keys and values; the decoder output and h ec Connect them to generate a d-dimensional feature (h c '); Then, select its neighbor node features As key source and h c ' is passed to the pointer layer as a query source; after the pointer layer, the final dimension of the strategy corresponds to the number of neighbor nodes; the output of the pointer layer passes through the embedding layer and the squeezing operation, and directly outputs the attention weight as the strategy π(a i |z t )=p(v i |z t ),(p t ,v i )∈ε.

5. The mobile robot autonomous exploration method based on deep reinforcement learning according to claim 1, characterized in that: The construction process of the proposed optimized reward function is: Considering the distance, the reward function with additional distance cost (r D ), which is defined as follows: in and are the positions of the current state and the next state, respectively, and d(·) represents the Euclidean distance between the two positions; In order to make the search path smoother, the angle cost (r Λ ) is defined as follows: where Λ(·) represents the yaw angle difference between the two positions; In order to enable the mobile robot to explore more unknown areas, the reward function (r F ), which is defined as follows: r F (z t ,a t )=n F where n F represents the number of observed boundaries associated with the next new state; In order to reduce repeated exploration of mobile robots, the intrinsic reward based on curiosity (r C ) is designed as follows: Where d1 and T0 represent the distance and step count thresholds respectively, P history represents all positions in the historical state set; if and P history If the Euclidean distance between them is greater than d1, the mobile robot receives a positive reward. In the early stages of exploration (time step ≤ T), there are many unexplored areas and intrinsic rewards are easy to obtain. However, as exploration progresses (time step > T0), the unknown areas will continue to decrease, so the intrinsic reward is doubled to reduce the vehicle's repeated exploration of known areas. When the vehicle exploration is completed, the completion reward of the mobile robot's autonomous exploration task (r E )as follows: η r (z t ) represents the state z t The exploration area rate under .

6. The mobile robot autonomous exploration method based on deep reinforcement learning according to claim 1 is characterized in that The process of the training method of the reinforcement learning network described in step S4 is as follows: The structure of the normal critic network will be preserved and will be derived from s t and z t Building a ground truth map is also an infographic used as input to a policy network, but The node set in Covering the entire free area, the node features (f i ) is a binary value used to represent v′ i Whether it is in an area that the mobile robot has already explored; The Prioritized Experience Replay (PER) is combined with the actor-critic network. At the same time, in order to reduce overestimation and improve training stability, the actor-critic network uses two Q networks and uses a smaller Q value to calculate the gradient. In addition, the actor-critic network also contains a policy network, so it is necessary to consider the TD errors of the three networks at the same time. The TD error of the sampled transition i (δ i ) can be defined as: i =|T e (Q1)|+|T e (Q2)|+αT e (π)| Where T e (·) represents the error of each network, and α represents the adjustment parameter used to balance the TD error weight calculated by the policy network and the Q-value network.

7. The mobile robot autonomous exploration method based on deep reinforcement learning according to claim 1 is characterized in that The input of the reinforcement learning network in step S4 is a sparse information graph, and the output is the target path point at the next moment, which specifically includes the following contents: The input of the network is a sparse information graph, and the output is the next path point, which is selected from the 8 nearest neighbor nodes of the current position. Each network body includes three encoders and one decoder. The basic network layer of the decoder is the graph attention layer, which is specifically divided into the policy network actor and the evaluation network critic. The policy network input is the current state, and the output is the action probability distribution parameters of the 8 nearest neighbor nodes; the evaluation network does not participate in the reasoning process, but only participates in model training during the training process, and is used to evaluate the quality of actions during the training process.

Images