A robot obstacle avoidance method based on model-based and model-free reinforcement learning

Abstract

The application discloses a robot obstacle avoidance method based on model-based and model-free reinforcement learning, belongs to the field of robot navigation, and adds model-based reinforcement learning as a forward-looking module into the framework of model-free reinforcement learning, uses the model-based reinforcement learning to improve the sample data utilization efficiency of the model-free reinforcement learning, and avoids the robot from selecting actions that may cause the robot to be in a dangerous situation by deducing the interaction between the robot and the surrounding environment in a future period of time, uses the model-free reinforcement learning as a main decision algorithm, and reduces the influence caused by an inaccurate environment model; the obstacle avoidance strategy is trained in a two-dimensional simulation environment, the perception module is designed and built offline on a data set, the perception module with the output information in the form of pseudo laser radar data is obtained, and then the obstacle avoidance strategy network obtained in the two-dimensional simulation environment can be combined to realize obstacle avoidance in a real environment. The application improves the performance of the obstacle avoidance algorithm.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot navigation and relates to a robot obstacle avoidance method. Background Technology

[0002] Autonomous navigation is a fundamental and crucial capability for mobile robots across various fields. Based on the application scenario and navigation objective, it can be mainly divided into four problems: local obstacle avoidance, indoor navigation, multi-robot navigation, and social navigation. Local obstacle avoidance refers to the navigation process of avoiding obstacles and reaching the target point without collision in an unknown environment. As the foundation for the other three problems, it has gradually become a research hotspot.

[0003] There are two common obstacle avoidance methods: map-based and mapless obstacle avoidance. Map-based obstacle avoidance algorithms are relatively mature, but they require a global prior map and often need to be readjusted when moving from one environment to another, making them difficult to adapt to environments with high pedestrian traffic and complex, ever-changing conditions. Mapless obstacle avoidance navigation is a data-driven, end-to-end obstacle avoidance strategy. It uses neural networks to fit a mapping relationship from raw sensor information input to actions, using sensor information as the input to the planner and action information as the planner's output to achieve obstacle avoidance. Deep reinforcement learning effectively combines the representational capabilities of deep learning with the decision-making capabilities of reinforcement learning. It can automatically extract effective features from sensor data and learn obstacle avoidance strategies during interaction with the environment, achieving end-to-end obstacle avoidance. This offers advantages such as not requiring a global prior map, simple deployment, and no need for parameter adjustment, and has therefore gradually gained attention.

[0004] Deep reinforcement learning algorithms are mainly divided into model-based reinforcement learning algorithms and model-free reinforcement learning algorithms. Model-based reinforcement learning algorithms refer to training an agent by interacting with the environment model after establishing one. Model-free reinforcement learning algorithms refer to training by directly interacting with the real environment without establishing an environment model. Each method has its advantages and disadvantages. Model-based reinforcement learning algorithms require maintaining a model of the environment, so the accuracy of the modeling greatly affects the actual performance of the algorithm. Model-free reinforcement learning algorithms do not require an environment model, but they need to obtain the optimal policy through multiple sets of interaction data, resulting in lower efficiency in utilizing sample data. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies and solve the problems existing in the obstacle avoidance strategies of current model-free reinforcement learning, this invention provides a robot obstacle avoidance method based on model-based and model-free reinforcement learning. The model-based reinforcement learning is added as a look-ahead module to the model-free reinforcement learning framework. By extrapolating the trajectory over a period of time in the future, the composition of the reward is modified to reduce the frequency of selecting high-risk trajectories.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] A robot obstacle avoidance method based on model-based and model-free reinforcement learning includes the following steps:

[0008] Step 1: Complete the establishment of the simulation environment. The simulation environment is built using a two-dimensional environment and will be used for subsequent training of robot obstacle avoidance strategies.

[0009] Step 2: Complete the establishment of the mobile robot decision-making module based on model-free reinforcement learning. Utilize deep reinforcement learning algorithms to establish the corresponding neural network model. Employ the Proximal Policy Optimization (PPO) algorithm to calculate the equivalent advantage function based on the reward, thereby directly adjusting the probability of the chosen action. Use an Actor-Critic architecture, where the Critic network is responsible for estimating the state value function, and the Actor network is responsible for outputting action probabilities. The output actions adopt a discrete action space. The PPO algorithm uses importance sampling, using the gap between the current optimized policy and the policy of the exploration environment to correct the updated loss function, and restricting it through preset parameters to avoid the updated new policy deviating too far from the old policy.

[0010] Step 3: Complete the establishment of the mobile robot look-ahead module based on model-based reinforcement learning. Use a deep neural network with a Transformer structure to fit the environment model. After entering a new environment, the original model is still used for training to obtain the environment model network. Input the environment feature vector and the robot motion vector into the environment model network to obtain the predicted value of the environment feature vector at the next moment and the predicted value of the immediate reward given to the robot by the environment. Then, use the obtained environment model to perform forward interactive extrapolation for 6 time steps to obtain multiple interaction records containing state, time step and environment reward. After processing them through the trajectory encoding network, they are used as the output of the look-ahead module. After the mobile robot interacts with the real environment for the next time, the real interaction state is stored in the trajectory interaction dataset. After completing a set number of navigation processes, the environment model in the look-ahead module is trained with gradient descent correction of the error term to achieve correction.

[0011] Step 4: Equip the decision-making module and look-ahead module to be trained, and place the simulation car in different simulation environments for training. Take full advantage of the parallel training capability of the simulation environment, set up various types of environments, treat the simulation car as an intelligent agent, randomly initialize the intelligent agent's position and navigation target point, and then the intelligent agent performs corresponding actions based on the acquired information, obtains corresponding rewards and new sensor information from the environment, thereby completing an interaction with the environment. Store the interaction data in the experience replay pool, repeat the above process until the target point is reached or a collision with an obstacle occurs, the current round of interaction ends, return to the starting point, and start the next round of interaction.

[0012] Once the data in the experience replay pool reaches a set quantity, complete sequence data from each navigation process is retrieved. First, the state value and advantage function are calculated to maximize the advantage of the Actor network's action output. The Actor network's loss function is then calculated, and its parameters are updated using gradient ascent. Next, to minimize the gap between the Critic network's output and the discounted reward, the Critic network's loss function is calculated, and its parameters are updated using gradient descent. The new network parameters are then used to output actions, continuing interaction with the environment until training is complete. After training, a trained decision module is obtained. During training, the rewards provided by the environment are manually set.

[0013] Step 5: Perform obstacle avoidance in a real environment. Install a LiDAR and an RGB-D vision sensor on the vehicle. Both the LiDAR and vision sensor are installed at the front of the vehicle. Use a multi-sensor fusion module to process the information from multiple sensors and obtain information in the form of pseudo-LiDAR as the output of the perception module. Place the robot in a real scene and apply navigation commands to the robot for target points. The robot's target point information processing module processes the commands to obtain the target point vector. Concatenate the vector output by the perception module with the target point vector to obtain the environmental feature vector.

[0014] Step 6: Input the environmental feature vector obtained in Step 5 into the trained decision module. The PPO algorithm will output the robot's action command at that moment based on the input feature vector until the robot reaches the destination and completes the navigation task.

[0015] Furthermore, in step 3, the look-ahead module trained in the simulation environment is used as a pre-trained network for the look-ahead module used when the robot is placed in a new environment for obstacle avoidance. After each interaction with the real environment during obstacle avoidance, the interaction data is stored. After reaching a set number, the real interaction data is used to train the currently used look-ahead module using gradient descent, thereby improving the accuracy of the environment model.

[0016] Furthermore, in step 1, training is performed in a two-dimensional simulation environment to obtain a decision module for transforming LiDAR information into obstacle avoidance strategies and a look-ahead module for fitting the environment model and performing forward interactive inference. In step 5, obstacle avoidance is performed in a three-dimensional real environment. Single-modal information is used to reduce the impact of the gap between the virtual and real environments on sensor information, and the complementarity of multi-modal information is used to obtain effective information. The robustness of LiDAR is fully utilized to improve the transfer problem of obstacle avoidance strategies based on reinforcement learning.

[0017] Furthermore, in step 2, in order to improve the performance of the obstacle avoidance policy network based on model-free reinforcement learning, improvements are made to the network structure by using an LSTM network and an attention mechanism to enhance the network's feature extraction capabilities and improve obstacle avoidance performance.

[0018] In step 3, in order to enhance the fitting effect of the neural network on the environment model, a network structure such as Transformer, which can process temporal and contextual information, is used instead of a simple fully connected network. The environment state at the next moment can be estimated based on the past environment state and agent actions in the interaction sequence. Subsequently, the accuracy of the environment model can be further improved by introducing structures such as graph neural networks that can model interaction relationships, thereby improving the obstacle avoidance effect.

[0019] In step 3, the two-dimensional grid cost map obtained from the lidar data is used as a description of the environmental state, thereby making full use of the spatial information description of the presence of obstacles in the environment in the lidar data and improving the accuracy of the environmental model.

[0020] In step 4, a reward function is designed to accelerate the convergence of the obstacle avoidance strategy network. The reward function includes a safety performance index, an efficiency index, and a trajectory optimization performance index. The safety performance index includes negative rewards for colliding with obstacles and negative rewards for the distance to obstacles. The efficiency index includes positive rewards for reaching the endpoint and positive rewards for the difference in distance from the endpoint at adjacent time steps. The trajectory optimization performance index includes negative rewards for the magnitude of acceleration and negative rewards for the difference in angular velocity direction between adjacent time steps, in order to avoid the problem of reward sparsity and improve the training effect.

[0021] In step 4, the advantages of a two-dimensional simulation environment, such as no need for physical calculations, low computational resource requirements, and parallel training, are utilized to build various obstacle avoidance environments with different levels of complexity and different obstacle characteristics, thereby improving the obstacle avoidance performance and robustness of the trained obstacle avoidance strategy.

[0022] This invention integrates two algorithms to maintain an internal environmental model for future prediction and pre-planning. This can avoid some trial-and-error processes and unavoidable dangerous situations in real environments, thereby improving the efficiency of sample data utilization. At the same time, it uses model-free reinforcement learning as the main source of action strategies to avoid the impact of inaccurate environmental models on the robot's obstacle avoidance strategy, thereby improving the robot's obstacle avoidance performance.

[0023] Although the look-ahead module's results are inaccurate in the new environment and need to be retrained, since the network parameters of the look-ahead module were trained in the simulation environment, it can still provide the robot with some prediction effect and look-ahead awareness even in the worst case. Furthermore, the look-ahead module can be further trained using the interaction data obtained during the execution of different tasks in the same environment, thereby further improving the robot's obstacle avoidance performance.

[0024] This invention combines model-based and model-free reinforcement learning by incorporating a look-ahead module into a model-free obstacle avoidance algorithm. Model-based reinforcement learning improves the efficiency of sample data utilization in model-free reinforcement learning, and by predicting the robot's interactions with its environment over a future period, it avoids actions that could lead to dangerous situations. Model-free reinforcement learning is used as the primary decision-making algorithm to reduce the impact of inaccurate environmental models. Model-based reinforcement learning maintains a neural network that fits the environmental model. After receiving the output of the perception module, it enumerates all actions in the action space and interacts with the environmental model sequentially, using the current action policy for subsequent interactions. This results in an interaction trajectory sequence containing the environmental state, immediate reward, and current action. An encoding network encodes this sequence to obtain the final look-ahead module output. Model-free reinforcement learning is used as the primary decision-making algorithm, taking the look-ahead module output and the perception module output as inputs. It outputs the action vector at the current moment and records real interaction data for training the look-ahead module.

[0025] The beneficial effects of this invention are mainly reflected in:

[0026] 1) By incorporating model-based reinforcement learning into the decision-making module based on model-free reinforcement learning in the form of a look-ahead module, the fusion of model-based and model-free reinforcement learning is achieved. Model-based reinforcement learning is used to model past interaction data, which improves the problem of low data utilization efficiency of agents based on model-free reinforcement learning. The look-ahead module is used to infer the strategy and environmental state during the next six decision steps, which avoids the robot from making actions that may put itself in dangerous situations and improves the performance of obstacle avoidance algorithms.

[0027] 2) The predicted interaction trajectory obtained by the look-ahead module is processed by the encoding network and used as an auxiliary input for action decision-making based on model-free reinforcement learning in the next moment, so as to improve the performance degradation of obstacle avoidance strategy caused by model error based on model reinforcement learning.

[0028] 3) The environment model fitting part in the forward-looking module is completed using neural networks. In the future, the interaction between the robot and dynamic obstacles such as pedestrians in the environment can be modeled using graph neural networks and other methods, thereby further improving the model fitting effect and leaving room for improvement in the environment model fitting method and further improving the obstacle avoidance effect. Attached Figure Description

[0029] Figure 1 This is a structural diagram of robot obstacle avoidance methods based on model-based and model-free reinforcement learning.

[0030] Figure 2 This is the network structure diagram of the action strategy of the obstacle avoidance algorithm for a car. Detailed Implementation

[0031] The present invention will now be further described with reference to the accompanying drawings.

[0032] Reference Figure 1 and Figure 2 A robot obstacle avoidance method based on model-based and model-free reinforcement learning includes the following steps:

[0033] Step 1: Complete the establishment of the simulation environment. Training the obstacle avoidance strategy based on reinforcement learning requires a large amount of interaction data between the agent and the environment. Considering the difficulty of setting up the environment, the ease of obtaining information, and the difficulty of deploying the model on a real vehicle, the simulation environment can be built using a two-dimensional environment for subsequent training of the robot's obstacle avoidance strategy.

[0034] Step 2: Establish the mobile robot decision-making module based on model-free reinforcement learning. Utilizing deep reinforcement learning algorithms, a corresponding neural network model is built. The Proximal Policy Optimization (PPO) algorithm is employed, an on-policy policy gradient algorithm applicable to high-dimensional continuous state spaces and continuous and discrete action spaces. It calculates an equivalent advantage function based on rewards, directly adjusting the probability of chosen actions. An Actor-Critic architecture is used, with the Critic network responsible for estimating the state value function. In this method, the output action uses a discrete action space, so the Actor network is responsible for outputting action probabilities. To improve the utilization of sampled data, the PPO algorithm employs importance sampling, using the gap between the currently optimized policy and the policy of the exploration environment to correct the updated loss function, and limiting it with preset parameters to prevent the updated policy from deviating excessively from the old policy.

[0035] Step 3: Establish the look-ahead module for the mobile robot based on model-based reinforcement learning. Use a deep neural network with Transformer-like structures capable of handling temporal and contextual information to fit the environment model. After entering a new environment, the original model is used for training, resulting in an environment model network. Inputting the environment state vector and robot motion vector into the environment model network yields the predicted environment state and the immediate reward given to the robot at the next moment. Then, the obtained environment model is used for a forward interactive simulation over six time steps, generating multiple interaction records containing state, time steps, and environmental rewards. These records are processed by a trajectory encoding network and used as the output of the look-ahead module. After the next interaction between the mobile robot and the real environment, the actual interaction state is stored in the trajectory interaction dataset. After completing a certain number of navigation processes, gradient descent correction training is performed on the environment model in the look-ahead module to correct errors.

[0036] Step 4: Equip the decision-making module and look-ahead module to be trained, and place the simulated car in different simulation environments for training. Take full advantage of the parallel training capability of simulation environments, and set up various types of environments (including open environments, dense environments, and narrow environments, etc.). Treat the simulated car as an intelligent agent, randomly initialize the agent's position and navigation target point, and then the intelligent agent performs corresponding actions based on the acquired information, obtains corresponding rewards from the environment and new sensor information, thereby completing an interaction with the environment. Store the interaction data in the experience replay pool, and repeat the above process until the target point is reached or a collision with an obstacle occurs. After that, the current round of interaction ends, and the car returns to the starting point to start the next round of interaction. Once the data in the experience replay pool reaches a certain quantity, complete sequence data from each navigation process is retrieved. First, the state value and advantage function are calculated. To maximize the advantage of the Actor network's action output, the Actor network's loss function is calculated, and gradient descent is used to update the network parameters. Then, to minimize the gap between the Critic network's output and the discounted reward, the Critic network's loss function is calculated, and gradient descent is used to update the network parameters. The new network parameters are then used to output actions, continuing to interact with the environment until training is complete. After training, the trained decision module is obtained. During training, the rewards provided by the environment are manually set, primarily considering safety (negative rewards for collisions with obstacles), efficiency (positive rewards for reaching the target point and positive rewards for distance differences between adjacent time steps), and trajectory smoothness (negative rewards for excessively large differences in angular velocity between adjacent time steps).

[0037] Step 5: Perform obstacle avoidance in a real environment. Install a LiDAR and an RGB-D vision sensor on the vehicle. Both the LiDAR and the vision sensor are installed at the front of the vehicle. Use a multi-sensor fusion module to process the information from multiple sensors and obtain information in the form of pseudo-LiDAR as the output of the perception module. Place the robot in a real scene and apply navigation commands to the robot for target points. The robot's target point information processing module processes the commands to obtain the target point vector. Concatenate the vector output by the perception module with the target point vector to obtain the environmental feature vector at the current moment.

[0038] Step 6: Input the environmental feature vector obtained in Step 5 into the trained decision module. The PPO algorithm will output the robot's action command at that moment based on the input feature vector until the robot reaches the destination and completes the navigation task.

[0039] The look-ahead module trained in the simulation environment is used as a pre-trained network for the look-ahead module when the robot is placed in a new environment for obstacle avoidance. After each interaction with the real environment during obstacle avoidance, the interaction data is stored. After a certain amount of data is collected, the look-ahead module currently in use is trained with gradient descent using the real interaction data to improve the accuracy of the environment model.

[0040] In step 1, training is performed in a two-dimensional simulation environment to obtain a decision module for obstacle avoidance strategy from LiDAR information and a look-ahead module for fitting the environment model and performing forward interactive inference. In step 5, obstacle avoidance is performed in a three-dimensional real environment. Single-modal information is used to reduce the impact of the gap between the virtual environment and the real environment on the sensor information. The complementarity of multi-modal information is used to obtain effective information and fully utilize the robustness of LiDAR to improve the transfer problem of obstacle avoidance strategy based on reinforcement learning.

[0041] In step 2, in order to improve the performance of the obstacle avoidance policy network based on model-free reinforcement learning, the network structure is improved by using an LSTM network and an attention mechanism to enhance the feature extraction capability of the network and improve the obstacle avoidance performance.

[0042] In step 2, a reward function is designed to accelerate the convergence of the obstacle avoidance strategy network. The reward function includes safety performance indicators (negative reward for colliding with obstacles, negative reward for distance from obstacles), efficiency indicators (positive reward for reaching the endpoint, positive reward for the difference in distance from the endpoint at adjacent time points), and trajectory optimization performance indicators (negative reward for the magnitude of acceleration, negative reward for the difference in angular velocity direction between adjacent time points) to avoid the problem of reward sparsity and improve the training effect.

[0043] In step 3, to enhance the fitting effect of the neural network on the environment model, a network structure such as Transformer, which can handle temporal and contextual information, is used instead of a simple fully connected network. This allows the next environmental state to be estimated based on past environmental states and agent actions in the interaction sequence. Furthermore, the accuracy of the environment model can be further improved by introducing structures such as graph neural networks that can model interaction relationships, thereby enhancing obstacle avoidance performance.

[0044] In step 3, the two-dimensional grid cost map obtained from the lidar data is used as a description of the environmental state, thereby making full use of the spatial information description of the presence of obstacles in the environment in the lidar data and improving the accuracy of the environmental model.

[0045] In step 4, the advantages of a two-dimensional simulation environment, such as no need for physical calculations, low computational resource requirements, and parallel training, are utilized to build various obstacle avoidance environments with different levels of complexity and different obstacle characteristics, thereby improving the obstacle avoidance performance and robustness of the trained obstacle avoidance strategy.

[0046] The method for training and deploying obstacle avoidance strategies for indoor robots based on model-free and model-based reinforcement learning in this embodiment includes the following steps:

[0047] Step 1: Establish a two-dimensional simulation environment based on OpenCV, determine whether the robot collides based on whether the pixel is occupied, obtain the distance array between the robot and obstacles within a 360-degree range based on the Bressenham linear algorithm as the planar LiDAR data, set the driving mode of the circle representing the car to differential drive mode, and the kinematic equation is shown in Equation (1):

[0048]

[0049] Where (x',y',θ') represents the position and orientation of the car at the next moment, (x,y,θ) represents the position and orientation of the car at the current moment, (v,ω) represents the current linear velocity and angular velocity of the car, respectively, in m / s and rad / s, and Δt represents the speed control period, that is, the time interval between the current moment and the next moment.

[0050] Because it doesn't require complex physics calculations like those on platforms such as Gazebo, this environment has low computational load, high speed, and low resource consumption on experimental equipment during the vehicle's movement. Furthermore, images can be used for map creation when building the environment. The resulting 2D environment is an abstraction of the 3D environment; the circles and rectangles in the image correspond to cylinders and cubes in the 3D environment, but do not contain irregular obstacles from the 3D world.

[0051] Step 2: Complete as follows Figure 2 The mobile robot decision-making module shown is established by using deep reinforcement learning algorithm to build a corresponding neural network model and adopting the proximal policy optimization algorithm (PPO). This is an on-policy policy gradient algorithm suitable for high-dimensional continuous action space. The equivalent advantage function is calculated based on the reward, thereby directly adjusting the probability of the selected behavior. The advantage function calculation formula is shown in equation (2):

[0052] A t =δ t +(γλ)δ t+1 +…+…+(γλ) T-t+1 δ T-1 (2)

[0053] Where, δ t =r t +γV(s t+1 )-V(s t), where λ is the decay factor, ranging from 0 to 1, and is set to 0.95 here; γ represents the decay factor of environmental reward over time, ranging from 0 to 1, and is set to 0.99 here; r t V(s) represents the environmental reward at time t. t ) and V(s t+1 ) represent the environmental state values ​​at the current moment and the next moment, respectively.

[0054] PPO employs an Actor-Critic architecture, where the Critic network is responsible for estimating the state-value function to ensure greater accuracy. In this method, the output actions utilize a continuous action space, so the Actor network is responsible for outputting action distribution parameters. The goal is to maximize the advantage function value of the output actions. During training, actions are sampled from this distribution to enhance exploration. During testing, the action values ​​are directly output based on the mean of the output action distribution parameters. To improve the utilization of sampled data, the PPO algorithm uses importance sampling, adjusting the updated loss function based on the gap between the current optimized policy and the policy in the exploration environment. Preset parameters constrain this adjustment, preventing the updated policy from deviating excessively from the old policy.

[0055] This invention makes the following improvements to the PPO algorithm for indoor scenes containing dynamic obstacles:

[0056] 1) Reward function reconstruction: The reward function is an important signal that drives the learning strategy of the agent in reinforcement learning. It includes safety performance indicators (negative reward for colliding with obstacles, positive reward for reaching the destination, and negative reward for the distance from the obstacle) and trajectory optimization performance indicators (negative reward for the magnitude of linear velocity acceleration and negative reward for the magnitude of angular velocity change).

[0057] 2) Adding an LSTM network layer: The PPO actor and critic networks are themselves multilayer perceptron (MLP) structures, using three fully connected layers. They do not have the ability to model the long and short term of sequence data separately. This invention modifies the multilayer perceptron by replacing the middle fully connected layer with an LSTM layer. The cell state of the LSTM network is used to fit the hidden state in a partially observable Markov decision process, thereby achieving the solution.

[0058] 3) Add an attention mechanism layer: Add an attention mechanism between the convolutional layers that process local maps. Use a mask to implement the attention mechanism. The principle of the mask is to identify the key features in the image data through another layer of weights. The mask generation network is trained through the forward propagation process. The weight matrix is ​​learned using the feature map obtained from the local map to generate the parameters of the network. Then, the trained weight matrix is ​​applied to the original feature map, so that the decision network can extract the feature information of the more important areas in each local map, thus forming the attention.

[0059] The state space consists of three parts: a two-dimensional grid cost map generated from the corrected LiDAR information, the relative position of the target point and the robot, and the robot's actions. The corrected LiDAR information and target information are generated by the perception module. To fully utilize the description of the two-dimensional space occupancy in the environment from the LiDAR information and improve the robustness of the obstacle avoidance strategy, the LiDAR information is appropriately abstracted. A two-dimensional grid map is used to describe the robot's environment, and a cost value is assigned to each grid in the grid map based on the LiDAR information, thus obtaining the cost map, which includes three regions: obstacle region, unexplored region, and freely movable region. The robot's actions represent the linear velocity and angular velocity output by the robot at the previous moment. Since the three most recent frames of perception information are more important for the decision at the current moment, the current moment and the two previous frames, a total of three frames of perception information, are selected as the state input.

[0060] The action space adopts a discrete action space. Considering that the reinforcement learning algorithm PPO is applicable to both continuous state spaces and discrete action spaces, this invention uses the PPO algorithm as the model-free reinforcement learning algorithm. The combinations of linear velocity v and angular velocity ω are divided into 9 types, specifically:

[0061] (1m / s,-0.5rad / s),(1m / s,0rad / s),(1m / s,0.5rad / s),(0.5m / s,-1rad / s),

[0062] (0.5m / s,0rad / s),(0.5m / s,1rad / s),(0m / s,-1rad / s),(0m / s,0.1rad / s),

[0063] (0 m / s, 1 rad / s)

[0064] Step 3: Complete the establishment of the mobile robot's look-ahead module based on model-based reinforcement learning. Use a deep neural network with structures such as Transformer that can handle temporal and contextual information to fit the environment model. For the environment model, a two-dimensional grid cost map is used as the input and output form of the neural network in the look-ahead model for the environment state. Upon entering a new environment, the original model is still used for training to obtain the environment model network. The environmental state vector and robot action vector are input into the environmental model network to obtain the estimated value of the environmental state vector and the estimated value of the immediate reward given to the robot by the environment at the next moment. Then, the obtained environmental model is used to perform a forward interactive extrapolation for 6 time steps. That is, after inputting the real environmental state s0 into the decision module, all actions a0 in the action space are input into the look-ahead module to obtain the environmental state s1 and reward r0 at the next moment. Then, the new environmental state is sampled through the current action policy to obtain the corresponding action a1, and then input into the look-ahead module to obtain the environmental state and reward at the next moment again. This process is repeated for 6 time steps to obtain multiple interaction records τ={s0,a0,r0,,s6,a6,r6} containing environmental state, agent action, and environmental reward. This means that at the current moment, the next action is taken by taking a certain action in the action space, and the subsequent interaction with the environment can be obtained by continuously using the current obstacle avoidance strategy. This is encoded through the interaction trajectory encoding network and mapped to a vector space of the same dimension. The mapped vector is used as the output of the look-ahead module. Furthermore, after the next interaction between the mobile robot and the real environment, the interaction process is stored in the dataset corresponding to the interaction trajectory. After completing 100 navigation processes, random samples are taken from this dataset, and gradient descent correction training is performed on the environment model in the look-ahead module based on the actual environment transition state. This achieves error correction of the neural network corresponding to the environment model estimation in the look-ahead module, thereby improving the prediction effect of the environment model.

[0065] Step 4: Equip the decision-making module to be trained and place the simulated car in different simulation environments for training. To avoid slow network convergence and low exploration success rates due to overly complex environments in the early stages of training, a design approach is adopted that gradually increases the complexity of the environment, starting with the simplest static obstacles and gradually increasing the environmental area and complexity to complete the training of the reinforcement learning algorithm model for the decision-making module. In the later stages of model training, the parallel training advantage of the simulation environment is fully utilized by setting up various types of environments (including open environments, dense environments, and narrow environments). The simulated car is treated as an intelligent agent, and an initial experience replay pool of 4000 is used to store interaction data for updating the reinforcement learning model parameters. The experience replay pool can store information about the interaction process between the agent and the environment, improving sample utilization efficiency and avoiding excessive hardware performance requirements for online interaction. Furthermore, the replay pool can store and retrieve complete interaction data, thus obtaining complete interaction sequences for training the PPO network containing LSTM layers. The PPO algorithm divides the same batch of samples in the experience pool into multiple mini-batches and reuses them multiple times. It also limits the parameter update magnitude when calculating the policy gradient, thus avoiding training instability. In this invention, interaction data from the same complete navigation process within a batch of samples are divided into the same mini-batch, thereby maintaining the temporal correlation of the corresponding data. The agent's position and navigation target point are randomly initialized. The agent then performs corresponding actions based on the acquired information, obtaining corresponding rewards from the environment and new sensor information, thus completing one interaction with the environment. The interaction data is stored in the experience replay pool. This process is repeated until the target point is reached or a collision with an obstacle occurs. The current round of interaction ends, and the agent returns to the starting point to begin the next round of interaction.

[0066] The specific training process is as follows: First, initialize the Critic network Q. θ and Actor Network The corresponding parameters are θ, Initialize the experience replay pool. Concatenate the interaction trajectory sequence vector output by the look-ahead module, the environmental feature vector output by the perception module, the target point information vector, and the linear and angular velocity vectors planned by the vehicle in the previous moment, and use this as the state input s. t After passing through the Actor network, the action distribution parameters of the agent are obtained. In this invention, the indices of the nine action combinations in the action space are assigned equal probability weights and transformed into a discrete probability distribution for subsequent sampling operations. Specifically, the probability distribution is constructed using the torch.distributions.Categorical() interface provided by torch, and the distribution is sampled to obtain the action instruction vector indices, corresponding to the action instructions denoted as a.t That is, linear velocity and angular velocity. Then, the agent performs actions according to action instructions and obtains the corresponding reward r given by the environment. t With the new sensor information, the map processing module can obtain the state input s for the next moment. t+1 This completes an interaction with the environment, sending the interaction data and the end marker (s) to the environment. t ,a t ,r t ,s t+1 Once the data in the experience replay pool is stored ("done"), the process is repeated until the target point is reached or a collision with an obstacle occurs. At this point, the current round of interaction ends, and the user returns to the starting point to begin the next round. After the experience replay pool has accumulated a certain amount of data, the data is retrieved to update the agent's network parameters. All interaction data from eight complete navigation sequences are randomly selected, and then the network is updated.

[0067] The Actor network is updated so that the output action advantage is maximized. After extracting the interaction data, the state value of each time step is first calculated using the Critic network, and then the advantage function of each state is calculated using equation (2). Using equation (3) as the loss function, the gradient descent method is used to update the multi-step gradient.

[0068] L CLIP (θ)=E t [min(r t (θ)A t ,clip(r t (θ), 1-, 1+)A t (3)

[0069] in, Indicates the degree of difference between the old and new strategies, a t Indicates the selection of an action, s t π represents the state, and π represents the policy mapping function.

[0070] The Critic network is updated using the multi-step TD method, employing equation (4). Starting from the current state, the reward returned by the environment at each step is multiplied by the discount factor and then summed to obtain the discounted reward G. t Using G t The current state value v(s) predicted by the Critic network t The mean squared loss function between the two sides is used as the loss function for gradient descent training.

[0071] G t =r t+1 +γr t+2 ++γ T-t r T+1+γ T+1-t v(s T+1 (4)

[0072] Among them, G t v(s) represents the discount reward, γ represents the reward discount parameter, and v(s) represents the discount reward. T+1 ) represents the state value at time T+1 output by the Critic network, r t+1 r t+2 and r T+1 These represent the environmental rewards at the corresponding time points.

[0073] The cache pool data is then cleared, and new network parameters are used to output actions, continuing to interact with the environment until training is complete. After training, a decision module trained in a two-dimensional simulation environment is obtained.

[0074] Step 5: The pseudo-LiDAR information obtained from visual and LiDAR information is used as an auxiliary task for separate training. The specific steps are as follows: Visual information includes color image information and depth image information. The lightweight MobileNet V2 network is selected as the backbone network of the semantic segmentation network. The semantic segmentation model is trained on the classic semantic segmentation dataset ADE20k indoor objects dataset from MIT. This model labels different pixels in the color image to represent categories. Then, mask layers for passable and impassable areas and mask layers for human and non-human areas are used to process the segmented images sequentially to obtain three labels indicating categories: passable areas, impassable areas, and human areas. A test set of 100 images taken from the real environment is used. The segmentation results of the semantic segmentation network with the more complex and accurate ResNet network as the backbone network are used as ground values ​​for testing. After obtaining the semantic segmentation map, the depth image and color image at the same time are registered. Since the field of view of the depth camera is smaller than that of the RGB image, the registration process involves attaching the corresponding RGB data to the pixels of the depth image to obtain the registered image. The segmented color image is then used to perform dimensionality reduction on the depth image, considering only the distance values ​​of impassable areas, and applying a dilation layer to the distance values ​​of human areas to obtain pseudo-LiDAR data. For LiDAR sensors, the point cloud map acquired by the LiDAR sensor can provide depth information, but the computational load is large. Therefore, it needs to be projected. In common forward-view projection and bird's-eye view projection, epipolar grid mapping based on the forward view can retain points in the point cloud map. Therefore, the epipolar grid mapping method is used to process the point cloud map data to obtain an omnidirectional distance array centered on the vehicle. Then, the pseudo-LiDAR data is used to correct the LiDAR data at similar times to obtain the corrected LiDAR data. After the auxiliary task is completed, the sensor information processing module is obtained. Then, a LiDAR and an RGB-D vision sensor are installed on the automated guided vehicle (AGV) (this invention uses a Turtlebot 2 vehicle as the experimental vehicle). Both the LiDAR and RGB-D vision sensor are mounted at the front of the vehicle, with sufficient space reserved for the LiDAR (360° horizontal range) to avoid obstruction. The vision sensor is pointed in the direction of travel; this invention uses a KinectV2 as the vision sensor. The robot is placed in a real-world scene, which may differ from the virtual environment, to test the generalization performance of the navigation method. Navigation commands for the target point, i.e., the target position vector relative to the robot, are applied to the robot. The robot's target point information processing module processes the commands to obtain the target point vector (x...). t ,y t ).

[0075] Step 6: Perform obstacle avoidance tasks in a real environment. At each moment, the robot acquires RGB-D image information and LiDAR point cloud information from the installed vision and LiDAR sensors. This information, along with the navigation target point information, is input into the trained perception module. The perception information processing module obtained in Step 4 processes the information to obtain environmental feature vectors. The environmental feature vectors are then concatenated with the robot's current linear and angular velocity vectors, resulting in two types of information: a local grid map centered on the robot and feature vectors. These are input into the decision module and look-ahead module obtained from the obstacle avoidance task in the simulated environment in Step 3, through two pathways. The PPO algorithm outputs the robot's action commands at that moment based on the input feature vectors and the interaction trajectory sequence obtained by the look-ahead module through forward inference, until the robot reaches the destination and completes the navigation task.

[0076] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A robot obstacle avoidance method based on model-based and model-free reinforcement learning, characterized in that, The method includes the following steps: Step 1: Complete the establishment of the simulation environment. The simulation environment is built using a two-dimensional environment and will be used for subsequent training of robot obstacle avoidance strategies. Step 2: Complete the establishment of the mobile robot decision-making module based on model-free reinforcement learning. Utilize deep reinforcement learning algorithms to establish the corresponding neural network model. Employ the Proximal Policy Optimization (PPO) algorithm to calculate the equivalent advantage function based on the reward, thereby directly adjusting the probability of the chosen action. Use an Actor-Critic architecture, where the Critic network is responsible for estimating the state value function, and the Actor network is responsible for outputting action probabilities. The output actions adopt a discrete action space. The PPO algorithm uses importance sampling, using the gap between the current optimized policy and the policy of the exploration environment to correct the updated loss function, and restricting it through preset parameters to avoid the updated new policy deviating too far from the old policy. Step 3: Complete the establishment of the mobile robot look-ahead module based on model-based reinforcement learning. Use a deep neural network with a Transformer structure to fit the environment model. After entering a new environment, the original model is still used for training to obtain the environment model network. Input the environment state vector and robot motion vector into the environment model network to obtain the predicted value of the environment state vector at the next moment and the predicted value of the immediate reward given to the robot by the environment. Then, use the obtained environment model to perform forward interactive extrapolation for 6 time steps to obtain multiple interaction records containing state, time steps and environment rewards. After processing them through the trajectory encoding network, they are used as the output of the look-ahead module. After the mobile robot interacts with the real environment for the next time, the real interaction state is stored in the trajectory interaction dataset. After completing a set number of navigation processes, the environment model in the look-ahead module is trained with gradient descent correction for the error term to achieve correction. Step 4: Equip the decision-making module and look-ahead module to be trained, and place the simulation car in different simulation environments for training. Take full advantage of the parallel training capability of the simulation environment, set up various types of environments, treat the simulation car as an intelligent agent, randomly initialize the intelligent agent's position and navigation target point, and then the intelligent agent performs corresponding actions based on the acquired information, obtains corresponding rewards and new sensor information from the environment, thereby completing an interaction with the environment. Store the interaction data in the experience replay pool, repeat the above process until the target point is reached or a collision with an obstacle occurs, the current round of interaction ends, return to the starting point, and start the next round of interaction. The training process is as follows: First, the Critic network of the decision-making module is initialized. and Actor Network The corresponding parameters are as follows: Initialize the experience replay pool by concatenating the interaction trajectory sequence vector output by the look-ahead module, the environmental feature vector output by the perception module, the target point information vector, and the linear and angular velocity vectors planned by the simulation vehicle at the previous moment, and use them as the state input. After passing through the Actor network, the action distribution parameters of the agent are obtained; Once the data in the experience replay pool reaches a set quantity, complete sequence data from each navigation process is retrieved. First, the state value and advantage function are calculated to maximize the advantage of the Actor network's action output. The Actor network's loss function is then calculated, and gradient descent is used to update the network parameters. Next, to minimize the gap between the Critic network's output and the discounted reward, the Critic network's loss function is calculated, and gradient descent is used to update the network parameters. The new network parameters are then used to output actions, continuing interaction with the environment until training is complete. After training, the trained decision module is obtained. During training, the rewards provided by the environment are manually set. Step 5: Perform obstacle avoidance in a real environment. Install a LiDAR and an RGB-D vision sensor on the vehicle. Both the LiDAR and vision sensor are installed at the front of the vehicle. Use a multi-sensor fusion module to process the information from multiple sensors and obtain information in the form of pseudo-LiDAR as the output of the perception module. Place the robot in a real scene and apply navigation commands to the robot for target points. The robot's target point information processing module processes the commands to obtain the target point vector. Concatenate the vector output by the perception module with the target point vector to obtain the environmental feature vector. Step 6: Input the environmental feature vector obtained in Step 5 into the trained decision module. The PPO algorithm will output the robot's action command at that moment based on the input feature vector until the robot reaches the destination and completes the navigation task.

2. The robot obstacle avoidance method based on model-based and model-free reinforcement learning as described in claim 1, characterized in that, In step 3, the look-ahead module trained in the simulation environment is used as the pre-trained network of the look-ahead module used when the robot is placed in the new environment for obstacle avoidance. After each interaction with the real environment during the obstacle avoidance process, the interaction data is stored. After reaching a set number, the real interaction data is used to train the look-ahead module currently in use using gradient descent.

3. A robot obstacle avoidance method based on model-based and model-free reinforcement learning as described in claim 1 or 2, characterized in that, In step 1, the decision-making module for obstacle avoidance strategy based on LiDAR information and the look-ahead module for fitting the environment model and performing forward interactive inference are trained in a two-dimensional simulation environment. In step 5, obstacle avoidance is performed in a three-dimensional real environment. Single-modal information is used to reduce the impact of the gap between the virtual environment and the real environment on the sensor information. The complementarity of multi-modal information is used to obtain effective information and fully utilize the robustness of LiDAR to improve the transfer problem of obstacle avoidance strategy based on reinforcement learning.

4. A robot obstacle avoidance method based on model-based and model-free reinforcement learning as described in claim 1 or 2, characterized in that, In step 2, improvements are made to the network structure by using an LSTM network and an attention mechanism to enhance the network's feature extraction capabilities.

5. A robot obstacle avoidance method based on model-based and model-free reinforcement learning as described in claim 1 or 2, characterized in that, In step 3, a Transformer network structure that can process temporal and contextual information is used instead of a simple fully connected network. The environment state at the next moment can be estimated based on the past environment states and agent actions in the interaction sequence. Subsequently, the accuracy of the environment model can be further improved by introducing a graph neural network structure that can model the interaction relationship.

6. A robot obstacle avoidance method based on model-based and model-free reinforcement learning as described in claim 1 or 2, characterized in that, In step 3, the two-dimensional grid cost map obtained from the lidar data is used as a description of the environmental state, thereby making full use of the spatial information description in the lidar data when there are obstacles in the environment.

7. A robot obstacle avoidance method based on model-based and model-free reinforcement learning as described in claim 1 or 2, characterized in that, In step 4, a reward function is designed to accelerate the convergence of the obstacle avoidance strategy network. The reward function includes a safety performance index, an efficiency index, and a trajectory optimization performance index. The safety performance index includes negative rewards for colliding with obstacles and negative rewards for the distance to obstacles. The efficiency index includes positive rewards for reaching the endpoint and positive rewards for the difference in distance to the endpoint between adjacent time steps. The trajectory optimization performance index includes negative rewards for the magnitude of acceleration and negative rewards for the difference in angular velocity direction between adjacent time steps, in order to avoid the problem of reward sparsity.

8. A robot obstacle avoidance method based on model-based and model-free reinforcement learning as described in claim 1 or 2, characterized in that, In step 4, the two-dimensional simulation environment is used to build various obstacle avoidance environments with different levels of complexity and different obstacle characteristics, taking advantage of the characteristics that the two-dimensional simulation environment does not require physical calculation, has low computational resource requirements, and can be trained in parallel.

Images