Robotic local navigation method based on multi-scale temporal residual reinforcement learning architecture

Abstract

This invention discloses a robot local navigation method based on a multi-scale temporal residual reinforcement learning architecture, comprising the following steps: 1) data input and preprocessing; 2) multi-scale path encoding construction; 3) temporal frame difference perception and residual fusion; 4) end-to-end inference and deployment control. Addressing the industry challenge of robots easily "freezing" due to unsolvable planning in confined environments, this invention resolves the conflict between algorithm stability and environmental adaptability by fusing global features using multi-scale Transformer, perceiving dynamic trends using LiDAR temporal difference, and combining a reinforcement learning residual adjustment strategy. Although average performance parameters may fluctuate normally under some conventional kits, this invention provides a solution to overcome extreme deadlock phenomena in highly challenging scenarios such as U-shaped walls and narrow doorways, improving stability, robustness, and deployment safety.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and deep reinforcement learning for mobile robots, specifically a robot local navigation method based on a multi-scale temporal residual proximal policy optimization architecture (MSTR-PPO). Background Technology

[0002] 1. The pain point of "robot freezing" in classic local navigation control

[0003] Existing hierarchical navigation systems perform well in static environments, but in narrow, enclosed indoor geometries (such as U-shaped dead ends, narrow corridors, or S-shaped passages), due to the spatial expansion mechanism of the local cost map, traditional controllers (such as those based on Dynamic Window (DWB), Model Predicted Path Integral (MPPI), or Adjusted Pure Chase (RPP)) easily determine all sampled probe trajectories as physically infeasible. In this situation, the behavior tree is forced to trigger a series of recovery behaviors, such as spinning in place and backing away. Once recovery fails, the robot enters a chain reaction of deadlock and stops moving forward, forming the well-known "frozen robot problem." If pedestrians also cross in the aforementioned narrow environment, classic single-frame algorithms cannot distinguish between pedestrians and newly appearing solid walls, further exacerbating the failure rate in such terrains.

[0004] 2. Limitations of existing end-to-end deep RL strategies

[0005] a) Short line-of-sight limitation: Standard reinforcement learning control networks often only select a few path anchor points within a 2-meter range in front of the robot as the observation state, lacking far-end topology foresight, making it difficult for them to handle U-shaped structures where the exit is located in the direction of the start.

[0006] b) Lack of temporal awareness: If multiple frames of LiDAR point clouds are directly stitched together, the input dimension will increase dramatically; if a recurrent neural network (RNN / LSTM) is introduced, the system memory overhead and the difficulty of maintaining hidden states during online inference will be greatly increased.

[0007] c) Controlling jitter and insecurity: Pure deep RL networks output raw actions without absolute physical constraints, which often produce sudden braking reversals or high-frequency oscillations, and can easily destroy the execution stability of the underlying ROS 2 Nav2 protocol stack.

[0008] 3. Technical problems that urgently need to be solved

[0009] a) How to endow local control algorithms with multi-scale long-distance path prediction capabilities and dynamic obstacle trend recognition capabilities without consuming huge computing power;

[0010] b) How to use a traditional pure geometric controller as the ballast of the algorithm while gaining the advantage of high mobility of the strategy, so as to eliminate the risk of chassis loss of control caused by the mutation output of the neural network, and safely and smoothly deploy it in the real ROS environment. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of existing local navigation schemes, such as path freezing and obstacle avoidance deadlock, as well as the difficulty in stable deployment of deep reinforcement learning schemes. This invention proposes a robot local navigation method based on a multi-scale temporal residual reinforcement learning architecture. This method is lightweight and efficient, providing the robot with both temporal and global geometric perception without the need for complex recursive networks. It significantly reduces the stalling phenomenon caused by narrow terrain and directly replaces traditional local actuators in the ROS 2 stack.

[0012] The technical solution to achieve the objective of this invention is:

[0013] A robot local navigation method based on a multi-scale temporal residual reinforcement learning architecture includes the following steps:

[0014] 1) Data input and preprocessing: Obtain the reference path generated by the global planner, the latest LiDAR scan data, and the robot's odometry status;

[0015] 2) Multi-scale path encoding backbone construction: Sample the reference path generated by the global planner at three different look-ahead distances (near, medium, and far) to obtain multi-scale preview points, extract and fuse the geometric features of the multi-scale preview points, and generate a multi-scale path representation;

[0016] 3) Temporal frame difference perception and residual fusion: Calculate the beam-by-beam difference between continuous LiDAR scan data to capture dynamic motion trends. The dynamic motion feature representation converted from the beam-by-beam difference, the static scene feature representation of the latest scan data, and the multi-scale path representation are fused and input into the PPO policy network, and the action residual correction amount is output.

[0017] 4) End-to-end inference and deployment control: The Pure Pursuit control algorithm is used as the base controller to generate the base speed command. The action residual correction amount output by the PPO strategy network is subjected to bounded truncation processing, and the processed bounded residual correction amount is superimposed with the base speed command to obtain the initial control command. Before the final command is issued, a hard safety guard mechanism is introduced. The local airborne controller calculates the minimum safe clearance distance of the current environment in real time based on the latest LiDAR scan data. When the minimum safe clearance distance is lower than the set safety threshold, the initial control command is forcibly corrected to an emergency stop command. Otherwise, the initial control command is issued as the final speed command to achieve end-to-end closed-loop local control.

[0018] The specific process for generating the multi-scale path representation in step 2) is as follows:

[0019] 2.1) On the reference path in step 1), collect multiple preview points along the set near, middle and far look-ahead distances, where near, middle and far look-ahead distances correspond to 0.75m, 2.25m and 4.75m respectively;

[0020] 2.2) Represent each preview point as a 5-dimensional feature vector (χ²) in the robot's body coordinate system. i ,y i ,d i ,θ i ,к i ), χ i ,y i The Cartesian coordinates of the preview point, d i Represents the Euclidean distance to the preview point, θ i The heading angle represented by к at the preview point. i This represents the local path curvature calculated using neighboring discrete points at the preview point;

[0021] 2.3) The feature vectors of the sampled preview points are mapped to a high-dimensional space, and learnable position and scale embeddings are added. Then, they are fed into the Transformer encoder for self-attention fusion and feature output.

[0022] 2.4) Perform mean pooling on the features output by the Transformer to output the final multi-scale path representation.

[0023] The specific process of step 3) temporal frame difference sensing and residual fusion is as follows:

[0024] 3.1) Calculate the beam-by-beam difference of two consecutive frames of normalized LiDAR scan data. This low-computing-cost differential signal characterizes the motion state of the environment, where... Representing the present Time of the first The normalized scanning distance value of the laser beam. Represents the previous control cycle Time of the first The normalized scanning distance value of the laser beam. This is a truncation function used to constrain the difference within the interval [-1, 1]; when the beam-by-beam difference... When it is negative, it is physically reflected in the first... The relative distance between the obstacle and the robot in the direction of the laser beam is decreasing, indicating that the obstacle is approaching the robot.

[0025] 3.2) The scan data of the current frame and the calculated beam-by-beam difference are independently input into different multilayer perceptron (MLP) branches to obtain static scene feature representation and dynamic motion feature representation;

[0026] 3.3) Static scene feature representation and dynamic motion feature representation are spliced ​​and fused through a linear layer, and combined with the multi-scale path representation output in step 2) to form the current frame observation feature. Then, the current frame observation feature is combined with the previous time-lapse hidden state feature transmitted by the recurrent neural network layer in the PPO policy network to form a complete action residual correction amount and output it.

[0027] The specific process of end-to-end inference and deployment control in step 4) is as follows:

[0028] 4.1) Set a bounded truncation constraint for the PPO strategy network output to control the upper limit of the absolute influence of its action residuals. The formula is expressed as follows: ,in, The normalized control action command represents the final output; This represents the pure chasing basic controller based on the current observation state. The generated basic control actions; The action residual represents the PPO policy network's response based on the current observation state. Output motion residual correction amount; The residual weighting coefficient represents the set value, used to scale the upper limit of the absolute influence of the action residual on the basic controller; This is a truncation function used to rigidly constrain the total action command after superimposed residuals to a safe range of [-1, 1].

[0029] 4.2) Set residual weighting coefficients = 0.3, to ensure that most of the control weights are dominated by the pure pursuit base controller, and to use geometric tracking inertia to resist the disturbances caused by the sudden changes in the output of the reinforcement learning network, so that it can play a "stabilizing" role in the control stability during navigation.

[0030] 4.3) Using a linear mapping relationship, the normalized control action commands obtained after superposition and truncation in step 4.1) are... Mapping to the actual physical velocity space yields the actual physical velocity command, and boundary range constraints are applied: linear velocity is constrained to the interval [-0.25, 1.0] m / s, and angular velocity is constrained to the interval [-1.5, 1.5] rad / s;

[0031] 4.4) Add a hard safety guard mechanism before the final command is issued. When the minimum clearance safety distance detected by LiDAR is less than 0.18 m, the emergency stop command will be forcibly triggered.

[0032] Step 4) end-to-end inference and deployment control also includes:

[0033] The convergent PPO policy network, along with the pure pursuit control algorithm, is packaged and fused to form a single ONNX computation graph model. This model is then directly loaded as a local C++ controller plugin within the ROS 2 Nav2 navigation stack. During the control cycle, feature assembly and ONNX Runtime inference operations are performed online, achieving high-speed loop response under low-latency hardware conditions.

[0034] The advantages and beneficial effects of this technical solution are:

[0035] (1) Solving specific frozen geometry problems: In the strongly restricted narrow area with dynamic pedestrians (such as the test set of partition room and U-shaped wall), the simulation environment can achieve a 100% success rate and 0 recovery behavior trigger events; Although the residual structure based on the basic controller limit does not perform best on some special loops with continuous rotation (such as when the average success rate is 91.1%), the core requirement of this design is to overcome the "frozen and stuck" problem, which is the only method that does not produce chain failures in the easily frozen area.

[0036] (2) High stability and ultra-low collision rate verification: The classic "pure pursuit" is retained as the "stabilizing force" in the whole action network, maximizing the absorption of reinforcement learning surges; the collision rate remains at an extremely low level of 0.1% in multiple sub-verifications, providing an extremely solid engineering safety guard.

[0037] (3) Millisecond-level inference and mass production deployment: There is no need to deal with lengthy time stacking and circular memory. With the ONNXRuntime engine and the directly packaged C++ navigation plugin, the single-cycle extremely low computing power overhead of 5~10 ms can be achieved in a single CPU core, which is perfectly adapted to the high-frequency control cycle of the service and inspection robot. Attached Figure Description

[0038] Figure 1 The overall structure and information flow diagram of the embodiment;

[0039] Figure 2 The following is a scene diagram for the simulation environment evaluation of the embodiment, where each line represents the movement trajectory of a dynamic obstacle of the corresponding color;

[0040] Figure 3This example demonstrates experimental scenarios of "corridor obstacle avoidance, right-angle turns, and doorway passage" on a real machine (Unitree Go2). Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0042] Example: This example uses the Unitree Go2 EDU quadruped robot as the hardware deployment platform to demonstrate the robot's empowerment control and low-level mapping method using a planar velocity interface mode. Figure 1 As shown:

[0043] 1. Hardware and communication mapping settings

[0044] 1.1 Sensor Data Intervention: Read airborne 2D or 3D flattened LiDAR data (resolution reset to 240°, 720 scan lines); global planning continues using existing A... * Pathfinding algorithms and AMCL adaptive Monte Carlo localization.

[0045] 1.2 Differential Interface Encapsulation: Although the quadruped robot has higher-dimensional control, at the Nav2 navigation protocol layer, its control interface is constrained to the (linear.x, angular.z) planar two-degree-of-freedom torque instruction set, ignoring the lateral displacement component, in order to maintain the same mapping logic as the differential drive vehicle in the training phase.

[0046] 2. Model Preset and Operational Structure

[0047] 2.1 Multi-view sampling parameters (Transformer): d near =0.75m, d mid =2.25m and d far A three-level sampling strategy with unequal spacing is set at a distance of 4.75m to generate a total of 10 structured representation pre-aiming point vectors with curvature values.

[0048] 2.2 Frame Difference Processing: By subtracting the current scan from the previous control cycle scan, By combining MLP, the original-size data is directly converted into 128-dimensional (static) and 64-dimensional (dynamic) data and then stitched together, which greatly reduces the overhead of the traditional LSTM structure.

[0049] 2.3 Residual Weighted Hyperparameters: The coefficients of the pure pursuit basic control account for 70% of the absolute weight by default, and the PPO motion correction network is clamped within a = 0.3; this threshold not only preserves the excellent tracking inertia of pure pursuit in narrow spaces (the stabilizing effect), but also avoids the output avalanche that is easy to occur in pure end-to-end architecture.

[0050] 2.4 Emergency Stop Limit: The minimum clearance safety threshold is set at 0.18m.

[0051] 3. Implementation Process: Real-time Closed-Loop Operation Mechanism

[0052] 3.1 Feature Acquisition and Assembly: During the control cycle, the sensor stream is received synchronously, the inter-frame changes are calculated, and a super-large state observation vector matrix with 1512 dimensions is assembled in the C++ plugin node.

[0053] 3.2 Graph Fusion Inference: The optimized ONNX Runtime engine is invoked to perform forward propagation inference; even with high observation dimensions, the total latency (p95 level) for completing inference and dual-path policy synthesis is less than 0.6 ms.

[0054] 3.3 Distribution Control: The processed speed parameters are converted into ROS Twist message types and periodically distributed to the Unitree underlying driver logic; for example... Figure 2 As shown, the simulation evaluation environment constructs nine Gazebo verification worlds containing a highly dynamic intersection scenario with a high density of non-cooperative moving pedestrians. Figure 2 Each line and colored cylinder represents the reciprocating motion trajectory of a corresponding dynamic obstacle (pedestrian) over multiple cycles. These obstacles do not actively avoid the robot during testing, simulating challenging non-cooperative pedestrian flows. Experimental results show that in core high-difficulty scenarios such as partitioned rooms and U-shaped walls with high-frequency interference from dynamic pedestrians, which easily cause traditional planners to "freeze and get stuck," this example, relying on the beam-by-beam difference of two consecutive frames of LiDAR normalized ranging calculated in step 2.2), enables the policy network to keenly capture the instantaneous movement trend of dynamic obstacles without relying on complex networks, achieving a 100% obstacle avoidance success rate and extremely stable performance with zero recovery behavior triggers. In the first seven comprehensive daily indoor test kits, the overall average test success rate reached 91.1%, with no control crashes or logic deadlocks occurring.

[0055] like Figure 3As shown, to verify the algorithm's general deployment capability, the trained and converged MSTR-PPO hybrid control model was directly loaded onto the Unitree Go2 EDU quadruped robot platform via a C++ plugin. In three typical real-world scenarios—corridor obstacle avoidance, right-angle turns, and doorway passage—this example achieved a total escape success rate of 27 / 30 (up to 90.0%) when facing untrained, non-cooperative pedestrians with random walking paths.

Claims

1. A robot local navigation method based on a multi-scale temporal residual reinforcement learning architecture, characterized in that, Includes the following steps: 1) Data input and preprocessing: Obtain the reference path generated by the global planner, the latest LiDAR scan data, and the robot's odometry status; 2) Multi-scale path encoding backbone construction: Sample the reference path generated by the global planner at three different look-ahead distances (near, medium, and far) to obtain multi-scale preview points, extract and fuse the geometric features of the multi-scale preview points, and generate a multi-scale path representation; 3) Temporal frame difference perception and residual fusion: Calculate the beam-by-beam difference between continuous LiDAR scan data to capture dynamic motion trends. The dynamic motion feature representation converted from the beam-by-beam difference, the static scene feature representation of the latest scan data, and the multi-scale path representation are fused and input into the PPO policy network, and the action residual correction amount is output. 4) End-to-end inference and deployment control: The pure pursuit control algorithm is used as the basic controller to generate the basic speed command. The action residual correction amount output by the PPO policy network is subjected to bounded truncation processing, and the processed bounded residual correction amount is superimposed with the basic speed command to obtain the initial control command. Before the final command is issued, a hard safety guard mechanism is introduced. The local airborne controller calculates the minimum safe clearance distance of the current environment in real time based on the latest LiDAR scan data. When the minimum safe clearance distance is lower than the set safety threshold, the initial control command is forcibly modified into an emergency stop command. Otherwise, the initial control command is issued as the final speed command, thereby realizing end-to-end closed-loop local control.

2. The robot local navigation method based on a multi-scale temporal residual reinforcement learning architecture according to claim 1, characterized in that, The specific process for generating the multi-scale path representation in step 2) is as follows: 2.1) On the reference path in step 1), collect multiple preview points along the set near, middle and far look-ahead distances, where near, middle and far look-ahead distances correspond to 0.75m, 2.25m and 4.75m respectively; 2.2) Represent each preview point as a 5-dimensional feature vector (χ²) in the robot's body coordinate system. i ,y i ,d i ,θ i ,к i ), χ i ,y i The Cartesian coordinates of the preview point, d i Represents the Euclidean distance to the preview point, θ i The heading angle represented by к at the preview point. i This represents the local path curvature calculated using neighboring discrete points at the preview point; 2.3) The feature vectors of the sampled preview points are mapped to a high-dimensional space, and learnable position and scale embeddings are added. Then, they are fed into the Transformer encoder for self-attention fusion and feature output. 2.4) Perform mean pooling on the features output by the Transformer to output the final multi-scale path representation.

3. The robot local navigation method based on a multi-scale temporal residual reinforcement learning architecture according to claim 1, characterized in that, The specific process of step 3) temporal frame difference sensing and residual fusion is as follows: 3.1) Calculate the beam-by-beam difference of two consecutive frames of normalized LiDAR scan data. This low-computing-cost differential signal characterizes the motion state of the environment, where... Representing the present Time of the first The normalized scanning distance value of the laser beam. Represents the previous control cycle Time of the first The normalized scanning distance value of the laser beam. This is a truncation function used to constrain the difference within the interval [-1, 1]; when the beam-by-beam difference... When it is negative, it is physically reflected in the first... The relative distance between the obstacle and the robot in the direction of the laser beam is decreasing, indicating that the obstacle is approaching the robot. 3.2) The scan data of the current frame and the calculated beam-by-beam difference are independently input into different branches of the multilayer perceptron to obtain static scene feature representation and dynamic motion feature representation; 3.3) Static scene feature representation and dynamic motion feature representation are spliced ​​and fused through a linear layer, and combined with the multi-scale path representation output in step 2) to form the current frame observation feature. Then, the current frame observation feature is combined with the previous time-lapse hidden state feature transmitted by the recurrent neural network layer in the PPO policy network to form a complete action residual correction amount and output it.

4. The robot local navigation method based on a multi-scale temporal residual reinforcement learning architecture according to claim 1, characterized in that, The specific process of end-to-end inference and deployment control in step 4) is as follows: 4.1) Set a bounded truncation constraint for the PPO strategy network output to control the upper limit of the absolute influence of its action residuals. The formula is expressed as follows: ,in, The normalized control action command represents the final output; This represents the pure chasing basic controller based on the current observation state. The generated basic control actions; The action residual represents the PPO policy network's response based on the current observation state. Output motion residual correction amount; The residual weighting coefficient represents the set value, used to scale the upper limit of the absolute influence of the action residual on the basic controller; This is a truncation function used to rigidly constrain the total action command after superimposed residuals to a safe range of [-1, 1]. 4.2) Set residual weighting coefficients = 0.3, to ensure that most of the control weights are dominated by the pure chasing basic controller, and to use geometric tracking inertia to resist the disturbances caused by the sudden changes in the output of the reinforcement learning network; 4.3) Using a linear mapping relationship, the normalized control action commands obtained after superposition and truncation in step 4.1) are... Mapping to the actual physical velocity space yields the actual physical velocity command, and boundary range constraints are applied: linear velocity is constrained to the interval [-0.25, 1.0] m / s, and angular velocity is constrained to the interval [-1.5, 1.5] rad / s; 4.4) Add a hard safety guard mechanism before the final command is issued, and force the emergency stop command to be triggered when the minimum clearance safety distance detected by LiDAR is less than 0.18 m.

5. The robot local navigation method based on a multi-scale temporal residual reinforcement learning architecture according to claim 1, characterized in that, Step 4) end-to-end inference and deployment control also includes: The convergent PPO policy network, along with the pure pursuit control algorithm, is packaged and fused to form a single ONNX computation graph model. This model is then directly loaded as a local C++ controller plugin within the ROS 2 Nav2 navigation stack. During the control cycle, feature assembly and ONNX Runtime inference operations are performed online, achieving high-speed loop response under low-latency hardware conditions.

Images