Robot motion planning method fusing GRU-Attention and TD3-SAC

By introducing GRU and attention modules into the Actor-Critic network and combining TD3 and SAC algorithms, the robot's exploration and decision-making capabilities in unknown environments are enhanced, the problem of unstable robot navigation in existing technologies is solved, and more robust motion planning is achieved.

CN120821265APending Publication Date: 2025-10-21NANJING UNIV OF SCI & TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202510774364.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-11
Publication Date
2025-10-21

AI Technical Summary

Technical Problem

Existing deep reinforcement learning algorithms have difficulty maintaining exploration capabilities and robustness in unknown environments, especially in dynamic and complex environments, and are prone to falling into suboptimal behavior patterns or overfitting, resulting in unstable robot navigation.

Method used

The gated recurrent unit (GRU) and attention module are introduced into the Actor-Critic network, combined with the double-delayed deep deterministic policy gradient (TD3) and the soft actor-critic (SAC) algorithm, and the ability to represent temporal features and explore the action space is enhanced by introducing a noise mechanism and the maximum entropy principle.

Benefits of technology

It improves the robot's autonomous navigation ability in unknown environments, avoids falling into local optimality, and achieves more robust decision-making and more efficient motion planning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120821265A_ABST
    Figure CN120821265A_ABST
Patent Text Reader

Abstract

The invention discloses a robot motion planning method fusing GRU-Attention and TD3-SAC, and aims to solve the problem that motion planning based on a traditional reinforcement learning method is relatively weak in strategy diversity, stability and exploration capability balance. Specifically, it is proposed that (1) a gated loop unit (GRU) and an attention module (Attention Module) are introduced into an actor-commentator network to enhance the representation ability of time sequence features; and (2) on the basis of an Actor-Critic network in which GRU-Attention is introduced, a dual-delay depth deterministic strategy gradient (TD3) and a soft actor-commentator (SAC) algorithm are fused, and through fusion of a promotion exploration mechanism of the TD3 and the SAC algorithm, the exploration ability of the robot to a complex unknown environment is enhanced while the training stability and robustness are kept.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a robot motion planning method based on reinforcement deep learning in an unknown environment, and in particular to a robot motion planning method integrating GRU-Attention and TD3-SAC. Background Art

[0002] Achieving diversity and robustness in the actions generated by learned policies is crucial for robots in tasks such as autonomous navigation and motion control in unknown environments. This is primarily due to the inherent uncertainty and complexity of real-world environments, where robots often need to operate under unforeseen conditions, partially observable states, and dynamic changes in the environment. In such situations, strategies that utilize only known policies may quickly reach suboptimal performance or even fail when transferred to unfamiliar situations. Therefore, a well-designed reinforcement learning (RL) algorithm should not only strive to maximize cumulative rewards but also maintain a strong exploration capability, enabling the robot to perform a wide range of actions to seek the optimal solution.

[0003] Classic deep reinforcement learning (DRL) algorithms, such as Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO), embody the challenge of balancing these objectives. While DDPG's deterministic policies achieve sample efficiency, they often suffer from mode collapse, becoming trapped in suboptimal behavior due to insufficient exploration. Conversely, PPO's stochastic policies often explore the action space inefficiently, requiring excessive environmental interactions to discover high-reward policies. These limitations become particularly problematic in dynamic navigation tasks, as neither approach can reliably maintain stability in avoiding obstacles or flexibility in adapting to new scenarios.

[0004] In double-delayed deep deterministic policy gradient (TD3), although deterministic behaviors improve development efficiency, they are susceptible to overfitting and instability, especially in the continuous behavior space, where small deviations can lead to large differences in results. In addition, during value updates, function approximation errors in the critic network can significantly distort the learning objective, leading to overestimation bias and performance degradation. To alleviate these problems, TD3 introduces a target policy smoothing regularization technique that uses random Gaussian noise during the exploration phase and during target value calculation. Perturb the actions output by the policy. This injected noise helps smooth the policy, making learning more robust to estimation errors and encouraging the agent to explore neighboring actions in the continuous action space. By adding regularization noise to the actions, we not only smooth the value estimates and reduce estimation errors, but also improve the diversity, stability, and exploration capabilities of the policy.

[0005] In the Soft Actor-Critic (SAC) algorithm, the maximum entropy method is introduced to encourage policy optimization not only in the direction of the maximum Q-value estimated by the critic network, but also to fully consider policy diversity. This method aims to prevent the policy from falling into local optimality by generating multiple feasible solutions within the action space. Summary of the Invention

[0006] The purpose of this paper is to propose a robot motion planning method that integrates GRU-Attention and TD3-SAC. The specific steps are as follows:

[0007] 1. The gated recurrent unit (GRU) and attention module are introduced into the actor-critic network to enhance the representation ability of temporal features.

[0008] 2. Based on the Actor-Critic network with GRU-Attention, the dual-delayed deep deterministic policy gradient (TD3) and the soft actor-critic (SAC) algorithm are integrated. By integrating their exploration-promoting mechanisms, the robot's exploration capability in complex unknown environments is enhanced while maintaining training stability and robustness.

[0009] Step 1 introduces GRU and Attention Module into the Actor-Critic network to enhance its ability to represent temporal features. First, a GRU module specifically designed to process sequential information is incorporated. As a variant of RNN, the GRU is able to process state sequences by maintaining an internal memory that updates over time. This internal memory allows the network to remember and utilize important information from previous time steps, effectively resolving the problem of temporal dependencies. This enables more accurate and robust decisions in dynamic and ever-changing environments. Furthermore, since the GRU receives an input sequence at each time step, not all elements in the sequence are valuable for decision-making. Therefore, an attention mechanism for the GRU output is introduced. The attention mechanism allows the model to assign different weights to different elements in the sequence based on their relevance to the current decision.

[0010] In step 2, based on the Actor-Critic network with GRU-Attention, the dual-delayed deep deterministic policy gradient (TD3) is combined with the soft actor-critic (SAC) algorithm to add noise to the actions during target value estimation (inherited from TD3) and the maximum entropy principle (borrowed from SAC), which enhances the reward function with an entropy term to promote more diverse behaviors. By combining these two strategies, the hybrid algorithm balances the use of known high-reward actions with extensive exploration of the action space. This results in a more robust policy that is less likely to fall into local optima and is better suited for the complex, high-dimensional control tasks often encountered in robotics.

[0011] Compared with the prior art, the present invention has the following significant advantages:

[0012] (1) The gated recurrent unit (GRU) and attention module are introduced into the traditional actor-critic network to enhance the representation ability of temporal features, which is conducive to making more accurate and robust decisions in dynamic and changing environments.

[0013] (2) Combining TD3’s mechanism of adding noise to actions during target value estimation with SAC’s maximum entropy principle, we achieve a balance between exploiting known high-reward actions and exploring the action space extensively. This results in a more robust search strategy that is less prone to falling into local optima and is more suitable for autonomous navigation tasks in unknown and complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic diagram of the system architecture;

[0015] Figure 2 A schematic diagram of the Actor-Critic network that introduces GRU-Attention;

[0016] Figure 3 Schematic diagram of the training environment;

[0017] Figure 4 Schematic diagram of the test environment;

[0018] Figure 5 Schematic diagram of the average reward curve for different methods in the training environment;

[0019] Figure 6 is the average reward experimental result of the test environment;

[0020] Figure 7 This is the experimental result of the collision rate of the test environment; DETAILED DESCRIPTION

[0021] The present invention will be further described below with reference to the accompanying drawings.

[0022] 1. Introducing GRU-attention Network into the implementation of Actor-Critic Network

[0023] To overcome the limitations of traditional actor-critic networks, we first incorporate a GRU module specifically designed to process sequential information. As a variant of an RNN, the GRU is able to process state sequences by maintaining an internal memory that updates over time. This internal memory allows the network to remember and utilize important information from previous time steps, effectively resolving temporal dependencies. This enables more accurate and robust decision-making in constantly changing environments.

[0024] In addition, an attention mechanism is introduced for the GRU output. The attention mechanism allows the model to assign different weights to different elements in the sequence based on their relevance to the current decision. Essentially, it helps the network focus on the most important states in the sequence and ignore those with less relevance. The input sequence of the GRU is represented as {x1,x2,...x n}, the process first normalizes the sequence to ensure that the values ​​are in a comparable range, reducing the impact of large or small values ​​that could distort the attention mechanism:

[0025] u i =tanh(ωGRU(x i )+b ω ) (1)

[0026] where ω is the learnable matrix.

[0027] The attention score of each element in the sequence i It is calculated using the softmax function, which can be expressed as:

[0028]

[0029] where μ is a learnable matrix.

[0030] The attention score of each input element is calculated using formula (2) and multiplied with the original input to obtain more accurate temporal information reflecting the importance of each element. The attention score is used to calculate the weighted sum of the states in the sequence:

[0031] o i =score i GRU(x i ) (3)

[0032] where oi represents the attention output at time step i.

[0033] The weighted output is passed through the GRU to update the internal memory and make more intelligent decisions based on the sequence context. Finally, the network generates an action based on the weighted sum of the states, enabling the agent to select the most appropriate action given the current state and the entire past state sequence. Figure 2 shown.

[0034] 2. Implementation of TD3-SAC hybrid method based on improved Actor-Critic Network

[0035] like Figure 1 The system architecture diagram shown in Figure 2 integrates the GRU attention mechanism into the TD3-SAC hybrid method, allowing the actor-critic to receive not only the current state as input, but also a maintained historical state buffer. Where n is the predefined history length, s n Represents the current state. This enables the policy to capture temporal context information.

[0036] First, we initialize two critic networks with random parameters and a policy network π φ , and their corresponding target networks and π φ′ . Initialize a replay buffer to store experience tuples, and a separate history buffer To maintain the latest n states.

[0037] At each training step t, if the current time step is less than n, it means that the length of the historical state sequence has not yet reached the set standard, so the current state s t Appended to the history buffer Without discarding any state. Otherwise, The oldest state in will be deleted and replaced by the new state s t Make room to ensure a fixed-size sliding window of historical observations. Based on the current history buffer The policy network selects actions following the following strategy:

[0038]

[0039] Perform action a in the environment t After that, the robot receives the reward r t and transition to the new state s t+1 . Convert tuple Stored in the replay buffer In order to calculate the target Q value, by adding s t+1 (if the original history has fewer than n elements) or remove the oldest state and append s t+1 Using target policy network Generate target actions.

[0040] The target Q-value is calculated using the minimum of the two target critics:

[0041]

[0042] Where γ is the discount factor. The critic network is updated by minimizing the mean squared Bellman error:

[0043]

[0044] Use a small batch N from Sampling in.

[0045] The policy network is updated every two steps by minimizing the policy loss defined as follows:

[0046]

[0047] It combines the maximum entropy objective. Finally, all target networks are soft-updated using the Polyak averaging scheme:

[0048] θ i ′←τθ i +(1―τ)θ i ′(i=1,2) (8)

[0049] φ′←τφ+(1―τ)φ′

[0050] where τ is the target update rate. This temporal expansion enables the policy to learn from sequences of observations rather than single time-step snapshots, improving performance, especially in partially observable and dynamic environments.

[0051] 3. Task Description

[0052] In robotic obstacle avoidance and navigation tasks, comprehensive and precise task and environment design is crucial for successful policy learning. We first construct a simulation environment using tools like Gazebo. The robot operates on a predefined map populated with static and dynamic obstacles. The robot is assigned a starting point and a goal location. The goal is to enable the robot to autonomously reach the goal in the shortest possible time while avoiding collisions with any obstacles along the way.

[0053] A key aspect of developing reinforcement learning tasks lies in correctly formulating the state and action spaces, as these spaces determine what the agent perceives and how it interacts with the environment. In our design, the robot is equipped with a 2D lidar sensor that provides 360-degree environmental perception. However, because full 360-degree data may introduce unnecessary noise and computational complexity, we restrict the field of view to the forward 180 degrees, which is more relevant to the direction of motion. This angular range is evenly divided into 20 parts, each of which provides the minimum distance to an obstacle within that angular window. These 20 laser scan distances form the first 20 dimensions of the state space.

[0054] In addition to obstacle perception, effective navigation requires the robot to know its relative position and orientation with respect to the goal. Therefore, we add two state dimensions: (1) the Euclidean distance between the robot and the goal point, and (2) the relative angle between the robot's current heading and the goal direction. This together results in a 22-dimensional state vector at each time step.

[0055] Regarding the action space, the robot operates under differential drive, which means that its motion can be controlled by two continuous control variables: the forward linear velocity v and the rotational angular velocity ω. These two variables constitute a two-dimensional continuous action space.

[0056] Once the state and action spaces are defined, the next key component is the design of a reward function, which guides the agent towards the desired behavior during the learning process. The reward function must be carefully designed to balance various competing objectives such as speed, safety, and goal completion. The robot is rewarded when it reaches a position close to the goal - defined by the Euclidean distance d less than a threshold d. threshold ——It will receive a large positive terminal reward r final , encouraging goal-oriented behavior.

[0057] Collisions are heavily penalized because they represent a faulty state in navigation. When colliding with any obstacle, the robot receives a significant negative reward r collision , the current event will terminate. This teaches the robot to avoid unsafe trajectories.

[0058] For time steps where the robot neither reaches the goal nor collides, a shaped reward is applied to encourage fast and smooth movement. Specifically, the robot receives a reward of v ― |ω|, which incentivizes higher forward speeds while penalizing excessive rotations. This helps balance exploration and motion efficiency, ultimately leading to safer and faster navigation behavior.

[0059] Based on the above design, the reward in the robot obstacle avoidance navigation task can be defined as:

[0060]

[0061] 4. Experimental Setup

[0062] A custom simulation environment was built using the Gazebo simulation platform. Figure 3 For the training environment, Figure 4 The simulation environment consists of various obstacles placed within a bounded area, designed to simulate real-world navigation scenarios where the robot must move towards a goal location without collision.

[0063] In this virtual environment, the robot is allowed to autonomously explore and interact with its surroundings to collect empirical data, which is used to train the actor and critic networks of the DRL model. The training process is divided into 400 epochs. Each epoch consists of 10,000 time steps, during which the robot performs a series of actions, receives observations from the environment, and receives rewards based on the designed reward function. These time steps are further divided into multiple segments to provide clear boundaries for individual navigation attempts.

[0064] 5. Comparative Experiment

[0065] In order to verify the advanced nature of the present invention, it is compared with several baseline methods, including the traditional TD3 algorithm, the traditional SAC algorithm, the GRU-TD3 algorithm without the attention mechanism, and the GRU-TD3 algorithm with the attention mechanism.

[0066] All models were trained and evaluated under the same experimental setup, including the same initial conditions, environment dynamics, reward structure, and obstacle reset strategy. This ensures that performance differences can be fully attributed to the internal architecture and learning capabilities of each algorithm.

[0067] Given that collisions in navigation tasks can produce significant negative rewards, their impact on the overall average reward metric is substantial—often masking more subtle differences in the robot's movement quality (such as smoother speed and stable turns). Therefore, to more objectively evaluate the performance of each model, we introduce two separate quantitative metrics:

[0068] (1) The average reward does not take collision penalties into account, which isolates the model’s performance in terms of motion efficiency and control smoothness.

[0069] (2) Average collision rate, which directly measures safety and obstacle avoidance capabilities.

[0070] Figure 5The average test reward over 400 training epochs is shown. Each data point in the curve represents the average reward across 10 evaluation episodes using that model. To reflect the consistency of the model's performance in collision-free scenarios, a light-colored region is drawn around the reward curve, corresponding to the standard deviation of the reward in epochs that did not result in a collision. This provides additional insight into the stability and reliability of each algorithm.

[0071] Figure 6 and Figure 7 The average rewards and collision rates in test environment 1 and test environment 2 are given. The experimental results show that the proposed method outperforms the baseline method in terms of both evaluation indicators.

Claims

1. A robot motion planning method integrating GRU-Attention and TD3-SAC, characterized in that: The following steps are involved: 1) A gated recurrent unit (GRU) and an attention module are introduced into the actor-critic network to enhance the representation of temporal features. 2) Based on the actor-critic network with the introduction of GRU-Attention, the double-delayed deep deterministic policy gradient (TD3) and the soft actor-critic (SAC) algorithm are integrated. By integrating their exploration-promoting mechanisms, the robot's exploration capability in complex unknown environments is enhanced while maintaining training stability and robustness.

2. A robot motion planning method integrating GRU-attention and TD3-SAC according to claim 1, characterized in that Step 1 introduces GRU and Attention Module into the Actor-Critic network to enhance the representation ability of temporal features: First, a GRU module specifically designed to process sequential information is incorporated to process state sequences by maintaining an internal memory updated over time, allowing the network to remember and utilize important information from previous time steps, effectively solving the problem of temporal dependency and helping to make more accurate and robust decisions in dynamic and changing environments; in addition, since the GRU receives an input sequence at each time step, and not all elements in the sequence are valuable for decision-making, an attention mechanism for the GRU output is introduced, allowing the model to assign different weights to different elements in the sequence based on their relevance to the current decision.

3. A robot motion planning method integrating GRU-attention and TD3-SAC according to claim 1, characterized in that Step 2 is based on the Actor-Critic network with GRU-Attention, integrating the double-delayed deep deterministic policy gradient (TD3) and the soft actor-critic (SAC) algorithm. It adds noise (inherited from TD3) and the maximum entropy principle (borrowed from SAC) to the actions during the target value estimation process to promote more diverse behaviors, balance the use of known high-reward actions and the extensive exploration of the action space, and form a more robust strategy that is not prone to falling into local optimality and is more suitable for the robot's autonomous navigation tasks in unknown and complex environments.

Citation Information

Cited By

  • Unmanned aerial vehicle self-adaptive route planning and dynamic obstacle avoidance method and system for distribution line inspection

    CN121657730A