A reinforcement learning-based obstacle avoidance navigation method for mobile robots in dynamic and complex environments

Abstract

The present invention discloses a dynamic complex environment obstacle avoidance navigation method for a mobile robot based on deep reinforcement learning, including an obstacle avoidance module, a target approach module, a DRL strategy training module, and a simulated reality task. The beneficial effects of the present invention are as follows: the method enables the mobile robot to rely on the observation information of the sensor in the absence of global map information, process the observation results through a multi-obstacle recognition network, combine the relative position information of the target, and train through a deep reinforcement learning model to learn the obstacle avoidance navigation strategy, avoid multiple dynamic obstacles, and reach the designated target point; no longer rely on the global map, no need to consume a lot of computing power to build a high-precision map, and no need to consider the maintenance and update of the global map information; input sensor observation information, output robot control actions, and the end-to-end navigation strategy mode responds quickly and can adapt to complex dynamic multi-obstacle environments.

Description

Technical Field

[0001] The present invention relates to a robot obstacle avoidance navigation method, specifically to a mobile robot dynamic complex environment obstacle avoidance navigation method based on deep reinforcement learning, belonging to the technical field of robot automation control and human-computer interaction. Background Art

[0002] Navigation based on map information has achieved remarkable results in the past few years, thanks to the development of simultaneous localization and mapping (SLAM) technology, which uses sensors such as lidar or cameras to build a global map, locate the robot's position, and build a local cost map based on this, and cooperate with the path planner for navigation. However, this technology is completely dependent on pre-established maps. Changes in the environment require the map to be updated, otherwise the robot cannot locate in the original map, let alone navigate. Secondly, in the map-based navigation method, building maps and planning navigation are actually regarded as two independent tasks. This split relationship makes the entire navigation system only work in simple and relatively static environments, and cannot be applied to unknown environments. Finally, when dealing with complex and multi-dynamic obstacle scenes, the map-based navigation method relies on the timely update of the local cost map and the reasonable and effective prior obstacle avoidance rules. The update of the local cost map is often not timely enough, and the artificially determined obstacle avoidance rules cannot guarantee the global optimality. Both of these also limit the application of map-based navigation methods in complex and multi-dynamic obstacle environments.

[0003] The main features of the obstacle avoidance and navigation method for mobile robots based on deep reinforcement learning are as follows: the robot obtains sensor observation information and the relative position of the target point in the local coordinate system, sets a reward function through a deep reinforcement learning model, performs training in a simulated environment, inputs sensor observation information, outputs robot control actions, and learns strategies for avoiding obstacles and approaching target points. Finally, the model is transplanted into a real environment through adjustments.

[0004] Main disadvantages: Deep reinforcement learning has low sampling efficiency and requires a lot of training; it is trained in a simulated environment and has difficulty migrating to reality; existing DRL learning methods are not suitable for dynamic environments; and local optimal decisions are prone to occur. Summary of the invention

[0005] The purpose of the present invention is to provide a mobile robot obstacle avoidance navigation method in a dynamic complex environment based on deep reinforcement learning in order to solve at least one of the above technical problems.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions: A mobile robot dynamic complex environment obstacle avoidance navigation method based on deep reinforcement learning, comprising the following steps

[0007] Step 1, obstacle avoidance module: the depth camera collects image information, extracts obstacle features through the obstacle feature recognition network, combines the depth ranging information, and inputs the combined information into the DRL strategy training module for training;

[0008] Step 2: Target approach module: The laser radar collects environmental information, combines the position of the target point in the local coordinate system, and inputs it into the DRL strategy training module for training;

[0009] Step 3: DRL strategy training module: Combine the two subtasks and train them in a virtual environment through a deep hierarchical reinforcement learning model;

[0010] Step 4: Simulation-Real-World Task: The trained model is transplanted into the real-world environment.

[0011] As a further solution of the present invention: the step 1 specifically includes:

[0012] The robot collects environmental information through a depth camera and inputs it into a multi-obstacle recognition network to identify obstacle boundary features. It also locates and identifies obstacles in the local environmental information in combination with the depth ranging information, outputs the position information and boundary feature information of the obstacles in the local coordinate system, and inputs it into the DRL strategy training module for training.

[0013] As a further solution of the present invention: the step two, specifically comprising;

[0014] The lidar collects environmental information, combines it with the position information of the target point in the local coordinate system, and inputs it into the DRL strategy training module and the obstacle avoidance module for joint training.

[0015] As a further solution of the present invention: the step three specifically includes:

[0016] A deep hierarchical reinforcement learning model is used to jointly train the obstacle position information and boundary feature information input by the obstacle avoidance module, the lidar information and the relative position information of the target point input by the target approach module, learn the obstacle avoidance navigation strategy, and output the robot control strategy in the local coordinate system.

[0017] As a further solution of the present invention: the step 4 specifically includes:

[0018] The training process of deep reinforcement learning models can only be performed in a simulated environment. After the training is completed, the model is deployed in an actual robot and needs to be fine-tuned to adapt to the task requirements.

[0019] As a further solution of the present invention:

[0020] The beneficial effects of the present invention are as follows: through this method, a mobile robot can rely on sensor observation information in the absence of global map information, process the observation results through a multi-obstacle recognition network, combine the relative position information of the target, and train through a deep reinforcement learning model to learn obstacle avoidance navigation strategies, avoid multiple dynamic obstacles, and reach a designated target point; it no longer relies on the global map, does not need to consume a lot of computing power to establish a high-precision map, and does not need to consider the maintenance and update of global map information; it inputs sensor observation information and outputs robot control actions, and the end-to-end navigation strategy mode responds quickly and can adapt to complex dynamic environments with multiple obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the overall framework structure of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Embodiment 1

[0024] like Figure 1 As shown, a mobile robot dynamic complex environment obstacle avoidance navigation method based on deep reinforcement learning includes the following steps

[0025] Step 1, obstacle avoidance module: the depth camera collects image information, extracts obstacle features through the obstacle feature recognition network, combines the depth ranging information, and inputs the combined information into the DRL strategy training module for training;

[0026] Step 2: Target approach module: The laser radar collects environmental information, combines the position of the target point in the local coordinate system, and inputs it into the DRL strategy training module for training;

[0027] Step 3: DRL strategy training module: Combine the two subtasks and train them in a virtual environment through a deep hierarchical reinforcement learning model;

[0028] Step 4: Simulation-Real-World Task: The trained model is transplanted into the real-world environment.

[0029] In the embodiment of the present invention, the step 1 specifically includes:

[0030] The robot collects environmental information through a depth camera and inputs it into a multi-obstacle recognition network to identify obstacle boundary features. It also locates and identifies obstacles in the local environmental information in combination with the depth ranging information, outputs the position information and boundary feature information of the obstacles in the local coordinate system, and inputs it into the DRL strategy training module for training.

[0031] In an embodiment of the present invention, the step two specifically includes;

[0032] The lidar collects environmental information, combines it with the position information of the target point in the local coordinate system, and inputs it into the DRL strategy training module and the obstacle avoidance module for joint training.

[0033] Embodiment 2

[0034] like Figure 1 As shown, a mobile robot dynamic complex environment obstacle avoidance navigation method based on deep reinforcement learning includes the following steps

[0035] Step 1, obstacle avoidance module: the depth camera collects image information, extracts obstacle features through the obstacle feature recognition network, combines the depth ranging information, and inputs the combined information into the DRL strategy training module for training;

[0036] Step 2: Target approach module: The laser radar collects environmental information, combines the position of the target point in the local coordinate system, and inputs it into the DRL strategy training module for training;

[0037] Step 3: DRL strategy training module: Combine the two subtasks and train them in a virtual environment through a deep hierarchical reinforcement learning model;

[0038] Step 4: Simulation-Real-World Task: The trained model is transplanted into the real-world environment.

[0039] In the embodiment of the present invention, the step three specifically includes:

[0040] A deep hierarchical reinforcement learning model is used to jointly train the obstacle position information and boundary feature information input by the obstacle avoidance module, the lidar information and the relative position information of the target point input by the target approach module, learn the obstacle avoidance navigation strategy, and output the robot control strategy in the local coordinate system.

[0041] In the embodiment of the present invention, the step 4 specifically includes:

[0042] The training process of deep reinforcement learning models can only be performed in a simulated environment. After the training is completed, the model is deployed in an actual robot and needs to be fine-tuned to adapt to the task requirements.

[0043] Working principle: Environmental information is collected through depth cameras and lidars, processed by deep reinforcement learning models, and output robot control actions. The end-to-end control mode enables the robot to navigate in complex, dynamic, and multi-obstacle environments.

[0044] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0045] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A mobile robot obstacle avoidance navigation method in a dynamic complex environment based on deep reinforcement learning, characterized by: The following steps are included Step 1: Obstacle avoidance module: The depth camera collects image information and extracts obstacle features through the obstacle feature recognition network. Combined with the depth ranging information, the combined input is fed into the DRL strategy training module for training; The step 1 specifically includes: The robot collects environmental information through a depth camera, inputs it into a multi-obstacle recognition network to identify obstacle boundary features, and combines the depth ranging information to locate and identify obstacles in the local environmental information. It outputs the position information and boundary feature information of the obstacles in the local coordinate system, and inputs it into the DRL strategy training module for training; Step 2: Target approach module: The laser radar collects environmental information, combines the position of the target point in the local coordinate system, and inputs it into the DRL strategy training module for training; Step 3: DRL strategy training module: Combine the two subtasks and train them in a virtual environment through a deep hierarchical reinforcement learning model; Step 4: Simulation-Real-World Task: The trained model is transplanted into the real-world environment.

2. According to claim 1, a method for mobile robot to avoid obstacles in a dynamic complex environment based on deep reinforcement learning, characterized in that: The step two specifically includes; The lidar collects environmental information, combines it with the position information of the target point in the local coordinate system, and inputs it into the DRL strategy training module and the obstacle avoidance module for joint training.

3. According to claim 1, a method for mobile robot to avoid obstacles in a dynamic complex environment based on deep reinforcement learning, characterized in that: The step three specifically includes: A deep hierarchical reinforcement learning model is used to jointly train the obstacle position information and boundary feature information input by the obstacle avoidance module, the lidar information and the relative position information of the target point input by the target approach module, learn the obstacle avoidance navigation strategy, and output the robot control strategy in the local coordinate system.

4. According to claim 1, a method for mobile robot to avoid obstacles in a dynamic complex environment based on deep reinforcement learning, characterized in that: The step 4 specifically includes: The training process of deep reinforcement learning models can only be performed in a simulated environment. After the training is completed, the model is deployed in an actual robot and needs to be fine-tuned to adapt to the task requirements.

Images