A reinforcement learning navigation method for indoor mobile robots

Abstract

The present invention belongs to the field of reinforcement learning, and specifically relates to a reinforcement learning navigation method for indoor mobile robots. In order to solve the problems of low sample utilization and low navigation success rate in intelligent agent navigation under complex environments and complex rewards, the present invention introduces an advantage-after-experience replay algorithm and a hierarchical reinforcement learning structure based on the D3QN algorithm. First, a new target point is selected by comparing the advantage functions of different trajectory points. The trajectory point with the largest advantage function is used as the new target point. According to the new target point, the trajectory samples are re-labeled and stored in the experience pool. Then, under the structure of hierarchical reinforcement learning, the intelligent agent is trained to learn in a complex environment. The top-level intelligent agent is responsible for learning large-scale path planning, the bottom-level intelligent agent is responsible for navigating to sub-goals, and finally achieving navigation to the total target point.

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning, and in particular relates to a reinforcement learning navigation method for an indoor mobile robot. Background Art

[0002] Reinforcement learning technology has advanced rapidly in recent years. Researchers are exploring ways to combine robot navigation with reinforcement learning. This approach has shifted the traditional path planning algorithm's inherent task model of first building a map and then planning. This approach eliminates model constraints and enables robots to fully leverage information from their surroundings for learning and autonomous navigation, achieving an end-to-end decision-making process. For large-scale indoor navigation tasks, it's crucial to increase the robot's learning speed and fully utilize empirical samples to enable it to adapt more quickly to the surrounding environment.

[0003] Traditional hindsight experience replay technology increases the proportion of successful samples in the experience pool by re-marking the target points in the failed samples, thereby improving the utilization rate of failed samples and thus improving the utilization rate of all experience samples. However, the use of traditional hindsight experience replay has too many restrictions and is more suitable for solving the sparse reward problem during agent training. In complex real-world tasks, the reward will be a combination of many complex targets and cannot be directly replayed through certain states. In the case of complex environments and complex rewards, in order to improve the sample utilization rate in agent navigation and improve the navigation success rate of the agent in complex environments, the present invention introduces an advantageous hindsight experience replay algorithm based on the D3QN algorithm, formulates new standards for selecting target points for marking, and allows the agent to learn from the experience of failure. At the same time, hierarchical reinforcement learning is introduced to further improve the robot's adaptability and learning speed to complex environments. Summary of the Invention

[0004] Aiming at the problems of low sample utilization and slow learning speed of robots performing reinforcement learning navigation in complex reward and complex environment, the present invention provides a reinforcement learning navigation method for indoor mobile robots.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A reinforcement learning navigation method for an indoor mobile robot comprises the following steps:

[0007] Step 1: Design different indoor environments for training in the Gazebo simulator in ROS, design the positions of the mobile robot and the target point, and handle the interaction between the mobile robot and the environment;

[0008] Step 2: Design the reward function, action space, and state space for reinforcement learning in mobile robot navigation.

[0009] Step 3: In a simple indoor environment, use the D3QN algorithm with experience replay to train the agent.

[0010] Step 4: In a complex indoor environment, use hierarchical reinforcement learning to establish two reinforcement learning agents, one as the top agent and the other as the bottom agent. The top agent uses the D3QN algorithm to select sub-target points and pass them to the bottom agent, and then plans the path to the overall target point. After obtaining the coordinates of the sub-target points, the bottom agent uses the algorithm model trained in a simple indoor environment to plan the path to the sub-target points.

[0011] Furthermore, in step 1, different indoor environments are designed in the Gazebo simulator in ROS for training, the positions of the mobile robot and the target point are designed, and the interaction between the mobile robot and the environment is processed. The specific steps are as follows:

[0012] Step 1.1: Build similar but different indoor environments in the Gazebo simulator for training and testing, and record the coordinate information in the environment.

[0013] In step 1.2, the initial position of the mobile robot is (0,0), and the position of the target point is randomly generated in the indoor environment;

[0014] In step 1.3, use the 360° lidar sensor on the mobile robot's head to obtain information about the distance to obstacles in its surrounding environment, interact with the surrounding environment, design topics to receive the lidar information, and use the lidar information as input for the reinforcement learning algorithm.

[0015] Furthermore, in step 2, the reward function, action space, and state space of reinforcement learning for mobile robot navigation are designed. The specific steps are as follows:

[0016] Step 2.1: Design a reward function based on goal-directedness: During training, the mobile robot receives a reward for each action decision it makes, and the agent uses this feedback to learn its strategy.

[0017] The reward function formula is set as follows:

[0018]

[0019] Among them, reward_yaw represents the reward or penalty value corresponding to the action, and distance_rate represents the ratio of the current distance between the mobile robot and the target point to the previous distance between the mobile robot and the target point;

[0020] Step 2.2, design the action space: Discretize the action space of the mobile robot into 5 actions, represented by a0, a1, a2, a3, and a4. The linear velocity is fixed at 0.15 m / s, and the angular velocity is different, as shown in the following table:

[0021]

[0022] Among them, a0 means a large right turn, a1 means a small right turn, a2 means going straight, a3 means a small left turn, and a4 means a large left turn;

[0023] Step 2.3, design the state space: set the number of samples of the lidar sensor to 24 dimensions, and the size of the mobile robot's state in the simulation environment to 28 dimensions. The state is represented as follows:

[0024] state=[obs|goal] (2)

[0025] obs=[scan_ranges,heading,current_distance] (3)

[0026] goal=[goal_x,goal_y] (4)

[0027] Among them, scan_ranges is the 24-dimensional data returned by the lidar sensor regarding the distance to the surrounding obstacles, indicating the current distance information of the mobile robot to the surrounding obstacles. heading indicates the direction of the mobile robot relative to the target point. current_distance represents the current distance between the robot and the target point. goal refers to the position coordinates of the target point.

[0028] Furthermore, in step 3, the D3QN algorithm with experience replay after dominance is used to train the agent in a simple indoor environment. The specific steps are as follows:

[0029] Step 3.1: During the interaction between the agent and the environment, the agent will receive feedback from the environment every time it takes a step, and the transition array of each step will be stored in the experience pool as a trajectory sample;

[0030] Step 3.2: After adding advantage to the D3QN algorithm, experience replay is performed. Under the complex reward conditions, a new target point is found. The new target point has the highest reward for the current trajectory. The new target point is selected by comparing the advantage functions of different trajectory points. The trajectory point with the largest advantage function is used as the new target point. Based on the new target point, the state, next state, and reward in the transition array of each step before the new target point in the trajectory are modified. The relabeled transition array is placed in the experience pool.

[0031] The calculation method of the advantage function of the target point is shown in formulas (5) and (6):

[0032] A(τ,v j )=R(τ|v j )-V π (S0,v j ) (5)

[0033]

[0034] Among them, A(τ,v j ) represents the point v in the trajectory τ j As the advantage function of the trajectory when it is the target point, R(τ|v j ) represents point v j As the total reward value of the trajectory when it is the target point, V π (S0,v j ) represents the value function of the initial state S0 under the strategy π, R(v i |v j ) point v j As the target point v i The reward value;

[0035] In step 3.3, the agent selects a batch of samples from the experience pool each time and inputs them into the neural network for training, updating the neural network parameters so that the agent can output better actions.

[0036] Furthermore, in step 4, the top-level agent uses the D3QN algorithm to select sub-target points and pass them to the bottom-level agent, which then plans the path to the overall target point. After obtaining the coordinates of the sub-target points, the bottom-level agent uses the D3QN algorithm trained in a simple indoor environment and adding advantage experience replay to plan the path to the sub-target points. The specific steps are as follows:

[0037] Step 4.1: The top-level agent inputs actions into the environment based on the initial state, and the surrounding environment returns rewards, penalties, and the next state based on the reward function.

[0038] Step 4.2: Define a finite set of sub-target points in the complex environment and train the top-level agent to find the next sub-target point within the set of sub-target points.

[0039] Step 4.3, pass the coordinates of the selected sub-goal point to the underlying agent;

[0040] Step 4.4: Define the reward function, action space, and state space of the top-level agent.

[0041] The top-level agent uses the D3QN algorithm, and the reward function is designed as follows:

[0042]

[0043] Among them, goal_distance is the distance between the current sub-goal point and the overall goal point, and ex_distance is the distance between the previous sub-goal point and the overall goal point;

[0044] The action space is divided into eight directions: southeast, northwest, northeast 45 degrees, southeast 45 degrees, northwest 45 degrees, and southwest, with a step length of 1m;

[0045] The state space consists of the x, y coordinates of the target point, as shown below:

[0046] state=[goal_x,goal_y] (8)

[0047] In step 4.5, the top-level agent and the bottom-level agent are trained simultaneously. The top-level agent returns the coordinates of a sub-target point to the bottom-level agent in each round. After receiving the coordinates of the sub-target point, the bottom-level agent uses the algorithm model trained in a simple indoor environment to navigate to the sub-target point.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] (1) Under complex reward conditions, the traditional hindsight experience replay target selection is improved. On the agent's experience trajectory, the target point that allows the agent to obtain the maximum reward is found, and the new target point is marked to improve the sample utilization rate in the experience pool and the generalization ability of the agent.

[0050] (2) The structure of the hierarchical reinforcement learning algorithm is complex. To achieve convergence between the top and bottom layers, the algorithm training process is long. The top-level agent uses the hierarchical idea to select sub-target points and train the bottom-level agent in advance, giving it a certain navigation ability and shortening the learning time of the top and bottom-level agents when training simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Advantages after seeing the empirical method diagram;

[0052] Figure 2 Hierarchical reinforcement learning graphs;

[0053] Figure 3 This is a simulation diagram of environment 1 in Example 1;

[0054] Figure 4 This is a simulation diagram of environment 2 in Example 1;

[0055] Figure 5 This is a simulation diagram of environment 3 in Example 1. DETAILED DESCRIPTION

[0056] Example 1

[0057] A reinforcement learning navigation method for an indoor mobile robot comprises the following steps:

[0058] Step 1: Design different indoor environments for training in the Gazebo simulator in ROS, design the positions of the mobile robot and the target point, and handle the interaction between the mobile robot and the environment;

[0059] The specific steps are:

[0060] Step 1.1: Build similar but different indoor environments in the Gazebo simulator for training and testing, and record the coordinate information in the environment.

[0061] In step 1.2, the initial position of the mobile robot is (0,0), and the position of the target point is randomly generated in the indoor environment;

[0062] In step 1.3, use the 360° lidar sensor on the mobile robot's head to obtain information about the distance to obstacles in its surrounding environment, interact with the surrounding environment, design topics to receive the lidar information, and use the lidar information as input for the reinforcement learning algorithm.

[0063] Step 2: Design the reward function, action space, and state space for reinforcement learning in mobile robot navigation.

[0064] The specific steps are:

[0065] Step 2.1: Design a reward function based on goal-directedness: During training, the mobile robot receives a reward for each action decision it makes, and the agent uses this feedback to learn its strategy.

[0066] The reward function formula is set as follows:

[0067]

[0068] Among them, reward_yaw represents the reward or penalty value corresponding to the action, and distance_rate represents the ratio of the current distance between the mobile robot and the target point to the previous distance between the mobile robot and the target point;

[0069] Step 2.2, design the action space: Discretize the action space of the mobile robot into five actions, represented by a0, a1, a2, a3, and a4. The linear velocity is fixed at 0.15 m / s, and the angular velocity is different, as shown in Table 1:

[0070] Table 1 Action space design

[0071]

[0072] Among them, a0 means a large right turn, a1 means a small right turn, a2 means going straight, a3 means a small left turn, and a4 means a large left turn;

[0073] Step 2.3, design the state space: set the number of samples of the lidar sensor to 24 dimensions, and the size of the mobile robot's state in the simulation environment to 28 dimensions. The state is represented as follows:

[0074] state=[obs|goal] (2)

[0075] obs=[scan_ranges,heading,current_distance] (3)

[0076] goal=[goal_x,goal_y] (4)

[0077] Among them, scan_ranges is the 24-dimensional data returned by the lidar sensor regarding the distance to the surrounding obstacles, indicating the current distance information of the mobile robot to the surrounding obstacles. heading indicates the direction of the mobile robot relative to the target point. current_distance represents the current distance between the robot and the target point. goal refers to the position coordinates of the target point.

[0078] Step 3: In a simple indoor environment, use the D3QN algorithm with experience replay to train the agent, so that the agent can output more optimal actions.

[0079] like Figure 1 As shown, the specific steps are:

[0080] Step 3.1: During the interaction between the agent and the environment, the agent will receive feedback from the environment every time it takes a step, and the transition array of each step will be stored in the experience pool as a trajectory sample;

[0081] Step 3.2: After adding advantage to the D3QN algorithm, experience replay is performed. Under the complex reward conditions, a new target point is found. The new target point has the highest reward for the current trajectory. The new target point is selected by comparing the advantage functions of different trajectory points. The trajectory point with the largest advantage function is used as the new target point. Based on the new target point, the state, next state, and reward in the transition array of each step before the new target point in the trajectory are modified. The relabeled transition array is placed in the experience pool.

[0082] The calculation method of the advantage function of the target point is shown in formulas (5) and (6):

[0083] A(τ,v j )=R(τ|v j )-Vπ (S0,v j ) (5)

[0084]

[0085] Among them, A(τ,v j ) represents the point v in the trajectory τ j As the advantage function of the trajectory when it is the target point, R(τ|v j ) represents point v j As the total reward value of the trajectory when it is the target point, V π (S0,v j ) represents the value function of the initial state S0 under the strategy π, R(v i |v j ) point v j As the target point v i The reward value;

[0086] In step 3.3, the agent selects a batch of samples from the experience pool each time and inputs them into the neural network for training, thereby updating the neural network parameters.

[0087] Step 4: In complex indoor environments, simple-structured agents have limited ability to learn about the environment and find it difficult to quickly learn navigation strategies. Hierarchical reinforcement learning, on the other hand, demonstrates its advantages in dealing with complex environments. The top layer learns sub-goals within a large range, while the bottom layer is responsible for reaching the sub-goals. Using hierarchical reinforcement learning, two reinforcement learning agents are established: one as the top agent and the other as the bottom agent. The top agent uses the D3QN algorithm to select sub-goal points and pass them to the bottom agent, which then plans the path to the overall goal. After obtaining the coordinates of the sub-goal points, the bottom agent uses the algorithm model trained in a simple indoor environment to plan the path to the sub-goal points.

[0088] The specific steps are:

[0089] Step 4.1, as Figure 2 As shown, the top-level agent inputs actions into the environment based on the initial state, and the surrounding environment returns rewards, penalties, and the next state based on the reward function;

[0090] Step 4.2: Define a finite set of sub-target points in the complex environment and train the top-level agent to find the next sub-target point within the set of sub-target points.

[0091] Step 4.3, pass the coordinates of the selected sub-goal point to the underlying agent;

[0092] Step 4.4: Define the reward function, action space, and state space of the top-level agent.

[0093] The top-level agent uses the D3QN algorithm, and the reward function is designed as follows:

[0094]

[0095] Among them, goal_distance is the distance between the current sub-goal point and the overall goal point, and ex_distance is the distance between the previous sub-goal point and the overall goal point;

[0096] The action space is divided into eight directions: southeast, northwest, northeast 45 degrees, southeast 45 degrees, northwest 45 degrees, and southwest, with a step length of 1m;

[0097] The state space consists of the x, y coordinates of the target point, as shown below:

[0098] state=[goal_x,goal_y] (8)

[0099] In step 4.5, the top-level agent and the bottom-level agent are trained simultaneously. The top-level agent returns the coordinates of a sub-target point to the bottom-level agent in each round. After receiving the coordinates of the sub-target point, the bottom-level agent performs path planning using the D3QN algorithm with experience replay after adding advantages. If the top and bottom levels are trained from scratch at the same time, the convergence time will be too long. Choosing to transfer the mature bottom-level path planning agent can speed up learning and better navigate in complex environments.

[0100] In order to show that the present invention can significantly improve the utilization rate of trajectory samples by the agent and adapt to the navigation environment more quickly, in an environment with a structure similar to the training environment 1( Figure 3 ), Environment 2( Figure 4 ), Environment 3( Figure 5 ) were tested separately, and the current mainstream DRL algorithm D3QN, the D3QN algorithm based on priority experience replay (D3QN_PER), and the D3QN algorithm based on hindsight experience replay (future mode) (D3QN_HER) were selected for experimental comparison. The results are shown in Table 2.

[0101] Table 2 Navigation test results in multiple environments

[0102]

[0103] The results show that the present invention can effectively solve the problem of low sample utilization in the robot reinforcement learning process, accelerate the robot's learning rate and has good generalization ability.

Claims

1. A reinforcement learning navigation method for an indoor mobile robot, characterized in that: The following steps are involved: Step 1: Design different indoor environments for training in the Gazebo simulator in ROS, design the positions of the mobile robot and the target point, and handle the interaction between the mobile robot and the environment; Step 2: Design the reward function, action space, and state space for reinforcement learning in mobile robot navigation. Step 3: In a simple indoor environment, use the D3QN algorithm with experience replay to train the agent. Step 4: In a complex indoor environment, use hierarchical reinforcement learning to establish two reinforcement learning agents, one as the top agent and the other as the bottom agent. The top agent uses the D3QN algorithm to select sub-target points and pass them to the bottom agent, which then plans the path to the overall target point. After obtaining the coordinates of the sub-target points, the bottom agent uses the D3QN algorithm trained in a simple indoor environment and adds advantage experience replay to plan the path to the sub-target points.

2. The method for indoor mobile robot reinforcement learning navigation according to claim 1, characterized in that: In step 1, different indoor environments are designed in the Gazebo simulator in ROS for training, the positions of the mobile robot and the target point are designed, and the interaction between the mobile robot and the environment is processed. The specific steps are as follows: Step 1.1: Build similar but different indoor environments in the Gazebo simulator for training and testing, and record the coordinate information in the environment. In step 1.2, the initial position of the mobile robot is (0,0), and the position of the target point is randomly generated in the indoor environment; In step 1.3, use the 360° lidar sensor on the mobile robot's head to obtain information about the distance to obstacles in its surrounding environment, interact with the surrounding environment, design topics to receive the lidar information, and use the lidar information as input for the reinforcement learning algorithm.

3. The method for indoor mobile robot reinforcement learning navigation according to claim 1, characterized in that: In step 2, the reward function, action space, and state space of reinforcement learning for mobile robot navigation are designed. The specific steps are as follows: Step 2.1: Design a reward function based on goal-directedness: During training, the mobile robot receives a reward for each action decision it makes, and the agent uses this feedback to learn its strategy. The reward function formula is set as follows: Among them, reward_yaw represents the reward or penalty value corresponding to the action, and distance_rate represents the ratio of the current distance between the mobile robot and the target point to the previous distance between the mobile robot and the target point; Step 2.2, design the action space: Discretize the action space of the mobile robot into 5 actions, represented by a0, a1, a2, a3, and a4. The linear velocity is fixed at 0.15 m / s, and the angular velocity is different, as shown in the following table: Among them, a0 means a large right turn, a1 means a small right turn, a2 means going straight, a3 means a small left turn, and a4 means a large left turn; Step 2.3, design the state space: set the number of samples of the lidar sensor to 24 dimensions, and the size of the mobile robot's state in the simulation environment to 28 dimensions. The state is represented as follows: state=[obs|goal] (2) obs=[scan_ranges,heading,current_distance] (3) goal=[goal_x,goal_y] (4) Among them, scan_ranges is the 24-dimensional data returned by the lidar sensor regarding the distance to the surrounding obstacles, indicating the current distance information of the mobile robot to the surrounding obstacles. heading indicates the direction of the mobile robot relative to the target point. current_distance represents the current distance between the robot and the target point. goal refers to the position coordinates of the target point.

4. The method for indoor mobile robot navigation by reinforcement learning according to claim 1, characterized in that: In step 3, the D3QN algorithm with experience replay after dominance is used to train the agent in a simple indoor environment. The specific steps are as follows: Step 3.1: During the interaction between the agent and the environment, the agent will receive feedback from the environment every time it takes a step, and the transition array of each step will be stored in the experience pool as a trajectory sample; Step 3.2: After adding advantage to the D3QN algorithm, experience replay is performed. Under the complex reward conditions, a new target point is found. The new target point has the highest reward for the current trajectory. The new target point is selected by comparing the advantage functions of different trajectory points. The trajectory point with the largest advantage function is used as the new target point. Based on the new target point, the state, next state, and reward in the transition array of each step before the new target point in the trajectory are modified. The relabeled transition array is placed in the experience pool. The calculation method of the advantage function of the target point is shown in formulas (5) and (6): A(τ,v j )=R(τ|v j )-V π (S0,v j ) (5) Among them, A(τ,v j ) represents the point v in the trajectory τ j As the advantage function of the trajectory when it is the target point, R(τ|v j ) represents point v j As the total reward value of the trajectory when it is the target point, V π (S0,v j ) represents the value function of the initial state S0 under the strategy π, R(v i |v j ) point v j As the target point v i The reward value; In step 3.3, the agent selects a batch of samples from the experience pool each time and inputs them into the neural network for training, updating the neural network parameters so that the agent can output better actions.

5. The method for indoor mobile robot reinforcement learning navigation according to claim 1, characterized in that: In step 4, the top-level agent uses the D3QN algorithm to select sub-target points and pass them to the bottom-level agent, which then plans the path to the overall target point. After obtaining the coordinates of the sub-target points, the bottom-level agent uses the D3QN algorithm trained in a simple indoor environment and adding advantage experience replay to plan the path to the sub-target points. The specific steps are as follows: Step 4.1: The top-level agent inputs actions into the environment based on the initial state, and the surrounding environment returns rewards, penalties, and the next state based on the reward function. Step 4.2: Define a finite set of sub-target points in the complex environment and train the top-level agent to find the next sub-target point within the set of sub-target points. Step 4.3, pass the coordinates of the selected sub-goal point to the underlying agent; Step 4.4: Define the reward function, action space, and state space of the top-level agent. The top-level agent uses the D3QN algorithm, and the reward function is designed as follows: Among them, goal_distance is the distance between the current sub-goal point and the overall goal point, and ex_distance is the distance between the previous sub-goal point and the overall goal point; The action space is divided into eight directions: southeast, northwest, northeast 45 degrees, southeast 45 degrees, northwest 45 degrees, and southwest, with a step length of 1m; The state space consists of the x, y coordinates of the target point, as shown below: state=[goal_x,goal_y] (8) In step 4.5, the top-level agent and the bottom-level agent are trained simultaneously. The top-level agent returns the coordinates of a sub-target point to the bottom-level agent in each round. After receiving the coordinates of the sub-target point, the bottom-level agent uses the D3QN algorithm with experience replay after taking advantage of training in a simple indoor environment to plan a path to the sub-target point.

Images