A mobile robot navigation method and system based on deep reinforcement learning

Abstract

The application discloses a kind of mobile robot navigation method and system based on deep reinforcement learning, and high efficiency, stable navigation of robot in complex static and dynamic environment is realized by structured laser radar state representation, obstacle time series perception network and asymmetric actor-critic strategy.The navigation method of the application comprises the following steps: step 1, obtain the laser radar point cloud data of the environment around the robot, and map the point cloud into angle-distance feature vector of fixed dimension by angle partition nearest point method;Step 2, the feature vector is input into obstacle perception network for time series feature extraction, and the short-term motion trend information of obstacle is obtained;Step 3, the extracted environmental features, robot body state and target position information are input into asymmetric actor-critic strategy network, and continuous speed action is output;Step 4, the speed action is input into the robot bottom controller, and navigation movement is realized.

Description

Technical Field

[0001] This invention belongs to the field of mobile robot navigation and obstacle avoidance technology, specifically relating to a mobile robot navigation method and system based on deep reinforcement learning. Background Technology

[0002] Existing mobile robot navigation methods mainly include traditional path planning algorithms and vision / depth perception-driven reinforcement learning strategies. Traditional path planning methods, such as A*, Dijkstra's algorithm, or TEB algorithm, often suffer from insufficient real-time adaptability in complex dynamic environments, making it difficult to handle unknown obstacles or partially observable scenes. Vision-based end-to-end reinforcement learning strategies rely on high-dimensional dense features, resulting in low training efficiency, susceptibility to illumination or noise, and limited transfer performance in real-world environments. While LiDAR provides robust geometric information, its raw point clouds are high-dimensional, unstructured, and highly temporally correlated, making direct use as reinforcement learning input prone to policy instability and convergence difficulties. Therefore, there is an urgent need for a deep reinforcement learning navigation method that can effectively fuse perceptual information, handle dynamic environments, and generate smooth, continuous actions. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, the present invention aims to provide a mobile robot navigation method and system based on deep reinforcement learning. By using structured LiDAR state representation, obstacle temporal perception network and asymmetric actor-critic strategy, the robot can achieve efficient and stable navigation in complex static and dynamic environments.

[0004] To achieve the above objectives, the following specific technical solutions may be adopted in this application: On the one hand, this application provides a mobile robot navigation method based on deep reinforcement learning, including the following steps: Step 1: Obtain the LiDAR point cloud data of the robot's surrounding environment, and map the point cloud into a fixed-dimensional angle-distance feature vector using the angle partitioning nearest point method; Step 2: Input the feature vector into the obstacle perception network for temporal feature extraction to obtain short-term obstacle motion trend information; Step 3: Input the extracted environmental features, robot body state and target position information into the asymmetric actor-critic policy network, and output continuous linear velocity and angular velocity; Step 4: Input the speed action into the robot's underlying controller to achieve navigation movement.

[0005] Furthermore, the obstacle perception network includes a multi-layer perceptron (MLP) encoder and a GRU (Gated Recurrent Unit). The GRU recurrent neural network is used to fuse radar observation features at continuous time intervals to model the dynamic movement trend of obstacles.

[0006] Furthermore, during the training of the asymmetric actor-critic policy, the critic network incorporates robot-privileged observation information to reduce the variance of the value function estimation, while the actor network in the execution phase relies solely on observable states to implement policy deployment.

[0007] Furthermore, a course-based environment difficulty scheduling mechanism is adopted in the simulation environment to gradually increase the complexity of static and dynamic obstacles, so as to improve the efficiency of strategy training and navigation generalization ability.

[0008] On the other hand, the present invention provides a mobile robot navigation system based on deep reinforcement learning for executing the aforementioned mobile robot navigation method based on deep reinforcement learning, comprising: The environmental perception module is used to acquire raw sensor data of the robot's surrounding environment; The state representation building module is used to structure and encode raw perceptual data to construct state inputs that can be directly used by reinforcement learning strategies. The navigation decision module, based on a deep reinforcement learning policy network, outputs robot motion control commands according to the current state. The motion execution module is used to convert the control commands output by the navigation decision module into low-level execution signals to drive the robot's movement; The training and update module is used to train the navigation strategy offline in a simulation environment and deploy the trained model to a real robot platform.

[0009] Furthermore, the environment perception module uses a 3D LiDAR as a sensor to acquire point cloud information of the robot's surrounding environment; the state representation construction module processes the LiDAR data as follows: ① Using the robot's own coordinate system as a reference, the horizontal field of view of the lidar is divided into several fixed angle intervals; ② Within each angle interval, extract the closest distance from the obstacle to the robot as the observation value for that interval; ③ The distance observations of all angle intervals are spliced ​​together in angular order to form a structured lidar state vector.

[0010] Furthermore, the navigation decision module is built on an actor-critic architecture, where the actor network is used to output continuous actions based on the current state. The inputs to the actor network are the structured LiDAR state, the target relative pose information, and historical state features. After being extracted by a multi-layer neural network, the outputs the robot's linear velocity and angular velocity commands. The critic network is used to evaluate the value function of the current state-action pair. The critic network additionally introduces the robot's linear velocity during the training phase to construct an asymmetric actor-critic structure.

[0011] Furthermore, the motion execution module adopts a motion strategy that relies solely on the robot's body state as input, including the body's angular velocity, the projection of gravity in the body coordinate system, the current velocity command, joint state information, and the action output of the previous moment, without introducing any environmental perception information.

[0012] Furthermore, the training and update module trains the navigation strategy based on the near-end policy optimization algorithm. The training process includes the following steps: ① Randomly generate the starting point, target point, and obstacle distribution in the simulation environment; ② The robot interacts with the environment according to the current strategy, collecting status, action, and reward data; ③ Construct a reward function based on the navigation task objective; the reward function includes objective achievement reward, collision penalty, path efficiency reward, motion smoothness reward, and dynamic obstacle avoidance reward; ④ The policy network parameters are iteratively updated using a near-end policy optimization method; Repeat the above process until the strategy converges.

[0013] Compared with the prior art, the present invention has the following advantages: This invention reduces the state dimension and explicitly represents the environmental geometry through structured LiDAR perception; enhances the ability to model the motion trends of dynamic obstacles through a temporal sensing network; achieves end-to-end navigation decision optimization through deep reinforcement learning; and improves the stability and generalization ability of policy learning through an asymmetric training mechanism. This invention enables stable, safe, and efficient autonomous navigation in complex dynamic environments, demonstrating good engineering feasibility and widespread application value. Attached Figure Description

[0014] Figure 1 This is a flowchart of the reinforcement learning training process of the present invention; Figure 2 This is a schematic diagram of the point cloud angle partitioning representation of the lidar of the present invention, illustrating the mapping method from the original point cloud to a fixed-dimensional angle-distance vector; Figure 3 This is a schematic diagram of the hierarchical navigation system framework of the present invention, illustrating the data flow and interaction between the perception, strategy, and motion execution modules; Figure 4 This is a schematic diagram of an obstacle perception network structure. Figure 5 This is a schematic diagram of the navigation decision module architecture of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] Example 1 like Figure 1 As shown, this embodiment provides a mobile robot navigation method based on deep reinforcement learning, including the following steps: Step 1: Obtain LiDAR point cloud data of the robot's surrounding environment, and map the point cloud into a fixed-dimensional angle-distance feature vector using the angle partitioning nearest point method.

[0017] Step 2: Input the feature vector into the obstacle perception network to extract temporal features and obtain short-term motion trend information of the obstacle.

[0018] The obstacle perception network includes a multi-layer MLP encoder and a GRU recurrent neural network. The GRU recurrent neural network is used to fuse radar observation features at continuous time intervals to model the dynamic movement trend of obstacles.

[0019] Step 3: Input the extracted environmental features, robot body state and target position information into the asymmetric actor-critic policy network, and output continuous linear velocity and angular velocity.

[0020] When training the asymmetric actor-critic policy, the critic network incorporates robot-privileged observation information to reduce the variance of the value function estimation, while the actor network in the execution phase relies solely on observable states to implement the policy deployment.

[0021] Step 4: Input the speed and motion into the robot's underlying controller to achieve navigation and movement.

[0022] In the simulation environment, a course-based environment difficulty scheduling mechanism is adopted to gradually increase the complexity of static and dynamic obstacles in order to improve the efficiency of strategy training and navigation generalization ability.

[0023] Example 2 This embodiment provides a mobile robot navigation system based on deep reinforcement learning, used to execute the mobile robot navigation method based on deep reinforcement learning described in Embodiment 1, including an environment perception module, a state representation construction module, a navigation decision module, a motion execution module, and a training and update module.

[0024] (1) Environmental perception module The environment perception module uses a 3D LiDAR as the primary sensor to acquire point cloud information of the robot's surrounding environment. To reduce the dimensionality of the perception data and enhance its geometric representation capabilities, the state representation construction module processes the LiDAR data as follows: ① Using the robot's own coordinate system as a reference, the horizontal field of view of the lidar is divided into several fixed angle intervals; ② Within each angle interval, extract the closest distance from the obstacle to the robot as the observation value for that interval; ③ The distance observations of all angle intervals are spliced ​​together in angular order to form a structured lidar state vector.

[0025] Specifically, with the robot's body coordinate system as a reference, a 360-degree field of view is covered in the horizontal direction and divided into 120 angular partitions at 3-degree intervals; in the vertical direction, within the effective field of view of the sensor, it is also evenly divided into 20 partitions at 3 degrees, thereby discretizing and mapping the original point cloud into 2400 fixed angular partitions.

[0026] For each angular partition, only the obstacle point closest to the robot is retained, and its distance value is used to characterize the minimum safe distance in that direction. To suppress long-range noise and improve perception stability, the maximum effective range of the LiDAR is truncated to 3 m. When no effective echo is detected in a certain angular partition, or the distance to the nearest point exceeds this threshold, its feature value is uniformly set to 3 m to represent the free space in that direction. After the above processing, the irregular and high-dimensional original point cloud is mapped into a 2400-dimensional angle-distance feature vector. This representation, with the robot itself as the reference frame, explicitly encodes the orientation distribution and nearest distance information of surrounding obstacles. While significantly reducing the state dimension and computational complexity, it still retains the spatial structure information that is crucial for local obstacle avoidance decisions.

[0027] To address the temporal changes of obstacles in dynamic environments, this paper performs sequential modeling of historical state vectors within continuous time steps, inputting multi-frame features into the obstacle perception network. Specifically, each frame's structured state vector is first mapped to a low-dimensional frame-level feature representation by an encoder, and then historical information is aggregated in the temporal dimension through recursive units (such as GRUs) to form dynamic obstacle latent state features. These latent state features reflect both the current spatial distribution of obstacles and encode their short-term motion trends, enabling navigation strategies to predict the movement direction and speed changes of obstacles. This allows for more timely and smooth obstacle avoidance decisions in dynamic scenes, reducing the collision risk caused by single-frame observation lag.

[0028] Finally, the obtained time-series perception features are used as state inputs and fed into the navigation policy network to generate continuous and executable speed commands, thereby achieving efficient target-oriented navigation in dynamic environments.

[0029] (2) State representation construction module The state representation building module is used to structure and encode raw perceptual data, constructing state inputs that can be directly used by reinforcement learning strategies.

[0030] (3) Navigation Decision Module The navigation decision module, based on a deep reinforcement learning policy network, outputs robot motion control commands according to the current state.

[0031] The navigation decision module is built on an actor-critic architecture, where the actor network is used to output continuous actions based on the current state, and the critic network is used to evaluate the value function of the current state-action pair.

[0032] The actor network input consists of structured LiDAR status, target relative pose information, and historical state features. After extraction by a multi-layer neural network, it outputs the robot's linear velocity and angular velocity commands. The motion space is a continuous space that can directly correspond to the robot's underlying motion control interface, thereby ensuring the smoothness and executability of motion commands.

[0033] The critic network introduces an additional robot linear velocity during the training phase to construct an asymmetric actor-critic structure, thereby improving the stability of policy training and sample utilization efficiency; during the actual deployment phase, only the actor network is retained for real-time decision-making.

[0034] (4) Motion execution module The motion execution module is responsible for mapping the desired robot velocity command output by the navigation decision module into joint-level control signals, completing the conversion from the robot motion layer to the execution layer, thereby driving the robot to achieve actual movement. To ensure the stability and accuracy of velocity tracking, the adopted motion strategy relies solely on the robot's body state as input, including the body angular velocity, the projection of gravity in the body coordinate system, the current velocity command, joint state information, and the previous action output, without incorporating any environmental perception information. A low-level execution module is provided for the navigation strategy.

[0035] (5) Training and Update Module The training and update module is used to train the navigation strategy offline in a simulation environment and deploy the trained model to a real robot platform.

[0036] The training and update module trains the navigation strategy based on the near-end policy optimization algorithm. The training process includes the following steps: 1) Randomly generate the starting point, target point, and obstacle distribution in the simulation environment.

[0037] 2) The robot interacts with the environment according to the current strategy and collects status, action and reward data.

[0038] 3) Construct a reward function based on the navigation task objective; reward function These include goal achievement rewards, collision penalties, path efficiency rewards, motion smoothness rewards, and dynamic obstacle avoidance rewards.

[0039] ① Rewards for achieving the goal : When the robot successfully reaches the target location (distance is less than the set threshold) When this occurs, a fixed positive reward will be given: ; When the goal is not achieved .

[0040] ② Collision Penalty : When the robot collides with any obstacle, a negative reward is given: ; If there is no collision, then .

[0041] ③ Path efficiency reward Calculated based on the robot's current step length and the shortest path length to the target: in, and These represent the distances from the previous step and the current step to the target, respectively. This is used as a weight. This factor encourages the robot to move closer to the goal with each step.

[0042] ④ Rewards for smooth movement Punishment for continuous changes in actions in As the weight, v x v y These represent the forward and lateral linear velocity commands in the robot's body coordinate system, respectively, while ω represents the yaw rate command. The superscripts t and t−1 represent the current and previous moments, respectively. This option encourages smooth changes in velocity and angular velocity, avoiding abrupt changes.

[0043] ⑤ Dynamic obstacle avoidance reward When a robot successfully avoids moving obstacles (predicts the obstacle's movement and maintains a safe distance) When this occurs, a small positive reward will be given: If you fail to avoid it in advance or the distance is less than This item can be 0 or a negative value.

[0044] The final comprehensive reward function is: .

[0045] 4) The policy network parameters are iteratively updated using a near-end policy optimization method.

[0046] Repeat the above process until the strategy converges.

[0047] To further enhance the generalization ability of the strategy, a course learning mechanism is introduced during the training process, gradually transitioning from simple, sparse obstacle scenarios to complex, dense dynamic environments.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mobile robot navigation method based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain the LiDAR point cloud data of the robot's surrounding environment, and map the point cloud into a fixed-dimensional angle-distance feature vector using the angle partitioning nearest point method; Step 2: Input the feature vector into the obstacle perception network for temporal feature extraction to obtain short-term obstacle motion trend information; Step 3: Input the extracted environmental features, robot body state and target position information into the asymmetric actor-critic policy network, and output continuous linear velocity and angular velocity; Step 4: Input the speed action into the robot's underlying controller to achieve navigation movement.

2. The mobile robot navigation method based on deep reinforcement learning according to claim 1, characterized in that, The obstacle perception network includes a multi-layer MLP encoder and a GRU recurrent neural network. The GRU recurrent neural network is used to fuse radar observation features at continuous time intervals to model the dynamic movement trend of obstacles.

3. The mobile robot navigation method based on deep reinforcement learning according to claim 1, characterized in that, When training the asymmetric actor-critic policy, the critic network incorporates robot-privileged observation information to reduce the variance of the value function estimation, while the actor network in the execution phase relies solely on observable states to implement the policy deployment.

4. A mobile robot navigation method based on deep reinforcement learning according to any one of claims 1-3, characterized in that, In the simulation environment, a course-based environment difficulty scheduling mechanism is adopted to gradually increase the complexity of static and dynamic obstacles in order to improve the efficiency of strategy training and navigation generalization ability.

5. A mobile robot navigation system based on deep reinforcement learning, used to execute the mobile robot navigation method based on deep reinforcement learning as described in claim 4, characterized in that, include: The environmental perception module is used to acquire raw sensor data of the robot's surrounding environment; The state representation building module is used to structure and encode raw perceptual data to construct state inputs that can be directly used by reinforcement learning strategies. The navigation decision module, based on a deep reinforcement learning policy network, outputs robot motion control commands according to the current state. The motion execution module is used to convert the control commands output by the navigation decision module into low-level execution signals to drive the robot's movement; The training and update module is used to train the navigation strategy offline in a simulation environment and deploy the trained model to a real robot platform.

6. A mobile robot navigation system based on deep reinforcement learning according to claim 5, characterized in that, The environmental perception module uses a 3D LiDAR as a sensor to acquire point cloud information of the robot's surrounding environment; the state representation construction module processes the LiDAR data as follows: ① Using the robot's own coordinate system as a reference, the horizontal field of view of the lidar is divided into several fixed angle intervals; ② Within each angle interval, extract the closest distance from the obstacle to the robot as the observation value for that interval; ③ The distance observations of all angle intervals are spliced ​​together in angular order to form a structured lidar state vector.

7. A mobile robot navigation system based on deep reinforcement learning according to claim 5, characterized in that, The navigation decision module is built on an actor-critic architecture. The actor network is used to output continuous actions based on the current state. The input of the actor network is the structured LiDAR state, the target relative pose information, and historical state features. After being extracted by a multi-layer neural network, the output is the robot's linear velocity and angular velocity commands. The critic network is used to evaluate the value function of the current state-action pair. The critic network additionally introduces the robot's linear velocity during the training phase to construct an asymmetric actor-critic structure.

8. A mobile robot navigation system based on deep reinforcement learning according to claim 5, characterized in that, The motion execution module uses a motion strategy that relies solely on the robot's body state as input, including the body's angular velocity, the projection of gravity in the body coordinate system, the current velocity command, joint state information, and the action output from the previous moment, without introducing any environmental perception information.

9. A mobile robot navigation system based on deep reinforcement learning according to claim 5, characterized in that, The training and update module trains the navigation strategy based on the near-end policy optimization algorithm. The training process includes the following steps: ① Randomly generate the starting point, target point, and obstacle distribution in the simulation environment; ② The robot interacts with the environment according to the current strategy, collecting status, action, and reward data; ③ Construct a reward function based on the navigation task objective; the reward function includes objective achievement reward, collision penalty, path efficiency reward, motion smoothness reward, and dynamic obstacle avoidance reward; ④ The policy network parameters are iteratively updated using a near-end policy optimization method; Repeat the above process until the strategy converges.

Images