Robot navigation method and system based on deep reinforcement learning and agent simulation

Abstract

This invention discloses a robot navigation method and system based on deep reinforcement learning and surrogate simulation. The method first defines the state, observation, and action spaces; it constructs a surrogate simulation environment based on a simplified kinematic model, using geometry to describe obstacles and calculate ideal LiDAR ranging values, superimposing sensor spatiotemporal noise to simulate network communication uncertainties, and combining actuator dynamic constraints to complete state transitions; it constructs a policy network based on the TD3 algorithm, pre-training it in the surrogate environment using a multi-constraint reward function and a hybrid target generation strategy; it constructs a ROS2 Gazebo high-fidelity dynamic simulation environment, loading the pre-trained policy into the high-fidelity environment for dynamic fine-tuning, and deploying it to the robot to perform online navigation tasks after convergence. This invention improves sample efficiency and policy robustness through two-stage training, and ensures the stability of the ROS2 system through differentiated QoS policies and multi-threaded actuators.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and intelligent control technology for mobile robots, specifically to a robot navigation method and system based on deep reinforcement learning and surrogate simulation. Background Technology

[0002] With the widespread application of mobile robots in warehousing and logistics, indoor delivery, and security inspection, their autonomous navigation capability in complex environments has become a key factor affecting system performance and deployment reliability. Traditional mobile robot navigation methods are typically based on accurate environmental mapping, path planning, and model-driven control strategies, which are highly dependent on the accuracy of robot dynamics modeling, sensor stability, and communication reliability. When dynamic obstacles, sensor noise, or communication uncertainties exist in the environment, the robustness and adaptability of these methods decrease significantly.

[0003] In recent years, end-to-end navigation methods based on deep reinforcement learning have gradually emerged. These methods directly map sensor observations to control commands through deep neural networks, reducing reliance on precise models and manual rules to some extent. However, existing deep reinforcement learning-based navigation methods still face the following technical challenges in the ROS2 distributed simulation and deployment environment:

[0004] In high-fidelity physics simulation environments such as Gazebo, rigid body dynamics, collision detection, sensor simulation, and middleware communication overhead must be considered simultaneously, which limits the simulation speed. Deep reinforcement learning algorithms typically require a large number of interactive samples, have long training cycles, high resource consumption, and low sample efficiency.

[0005] When the reward function and exploration strategy are poorly designed, the agent is prone to converge to local optimal behaviors such as rotating in place and remaining still for a long time. Although collisions can be avoided, navigation efficiency is low and it is difficult to complete the expected task.

[0006] Existing simulation training environments often simplify or ignore sensor noise, network communication uncertainties, and actuator dynamic constraints, resulting in a significant performance degradation of the trained strategies when transferred to high-fidelity simulation environments or real robot platforms, with a large gap between simulation and reality.

[0007] Many existing works are still based on the process-oriented node structure of ROS1. When directly migrating to ROS2's DDS middleware, multi-threaded executor and asynchronous communication mechanism, system stability issues such as mismatch between reinforcement learning synchronous interface and ROS2 asynchronous callback, service call blocking, and node freezing during long training periods are likely to occur. Summary of the Invention

[0008] The purpose of this invention is to provide a robot navigation method and system based on deep reinforcement learning and surrogate simulation, so as to solve the problems of low training efficiency, easy getting trapped in local optima, large gap between simulation and reality, and insufficient system stability of existing navigation technologies in ROS2 system.

[0009] To achieve the above objectives, the technical solution provided by this invention is: a robot navigation method based on deep reinforcement learning and agent simulation, comprising the following steps:

[0010] S1: Define the state space, observation space, and motion space for robot navigation; the state space includes the robot's position, attitude angle, linear velocity, and angular velocity; the observation space includes lidar ranging information and the robot's relative pose to the target point; the motion space includes velocity control commands.

[0011] S2: Based on the parameters of the state space, observation space, and action space, a proxy simulation environment based on a simplified kinematic model is constructed; obstacles are described geometrically, and ideal lidar ranging values ​​are calculated using the ray casting method as the raw observation data; sensor spatiotemporal noise is superimposed on the raw observation data to generate noisy observation data; random delays and random packet loss simulating network communication uncertainties are introduced into the noisy observation data to generate the final observation data; the robot's state transition is completed according to the speed control command and the preset actuator dynamics constraints.

[0012] S3: Construct a deep reinforcement learning network structure based on the TD3 algorithm, which includes a policy network and an evaluation network; the policy network takes the final observation data as input and outputs normalized speed control commands; the evaluation network evaluates the value of the robot's state and actions;

[0013] S4: In the proxy simulation environment constructed in step S2, the deep reinforcement learning network constructed in step S3 is pre-trained by combining the multi-constraint reward function and the hybrid target generation strategy to obtain the pre-trained policy parameters.

[0014] S5: Construct a high-fidelity dynamics simulation environment for ROS2 Gazebo, which includes a robot rigid body dynamics model and sensor plugins; load the pre-trained policy parameters into the ROS2 Gazebo high-fidelity simulation environment, fine-tune the policy network parameters under real dynamics and communication conditions, and deploy the policy network onto the robot to perform online navigation tasks after the fine-tuning converges.

[0015] To optimize the above technical solution, the specific measures also include:

[0016] In step S2, the robot state transition is completed by combining the speed control command and the preset actuator dynamics constraints. Specifically, the process is as follows: based on the linear velocity control command and angular velocity control command in the speed control command, and combined with the preset maximum acceleration and actuator response time constant, the robot's actual linear velocity and actual angular velocity are updated. The robot's new pose is then calculated using the actual linear velocity and actual angular velocity to complete the state transition. The calculation formula is:

[0017] ;

[0018] ;

[0019] in, Indicates at time step The robot's actual linear velocity at that time; Indicates at time step The robot's actual angular velocity at that time; Indicates at time step The robot's actual linear velocity at that time; ; Indicates at time step The desired linear velocity command for the robot; Indicates at time step The robot's desired angular velocity command; This represents the response time constant of the robot chassis actuator; This indicates the maximum linear acceleration allowed by the robot chassis. Indicates the time step of the simulation environment; This represents the truncation function.

[0020] Furthermore, in step S3, the target value calculation of the TD3 algorithm introduces a target policy smoothing mechanism, expressed as:

[0021] ; ;

[0022] in, Indicates the target action; The parameters representing the policy network; It indicates the state at the next time step, including the robot's lidar observations and its own state at the next moment; The target policy is to smooth out noise; This indicates that the mean is 0 and the standard deviation is 0. The normal distribution; This represents the truncation function; The standard deviation of noise; This represents the noise boundary constant.

[0023] In step S4, the multi-constraint reward function This includes target approach rewards, coupled forward progress rewards, rotation penalties, and graded obstacle avoidance penalties, specifically:

[0024] ;

[0025] Target approach reward :

[0026] ;

[0027] Furthermore, coupling forward rewards :

[0028] ;

[0029] Spin penalty :

[0030] ;

[0031] Tiered obstacle avoidance penalties :

[0032] ;

[0033] Single-step time penalty :

[0034] ;

[0035] in, A coefficient representing the percentage reward for approaching the target; This represents the Euclidean distance between the robot's position and the target point at the previous time step t-1; This represents the Euclidean distance between the robot's current position and the target point at the current time step t. This represents the weighting coefficient of the coupling forward reward; Indicates the robot at time step linear velocity; Indicates the robot at time step angular velocity; Indicates the rotation penalty coefficient; Indicates the linear velocity threshold; This indicates that the collision penalty will be terminated. Indicates the distance to the nearest obstacle; Indicates the physical collision threshold; Represents the obstacle avoidance potential field coefficient; Indicates the safe distance threshold; This represents a fixed single-step penalty constant.

[0036] Furthermore, in step S2, constructing a proxy simulation environment based on a simplified kinematic model also includes: applying random displacement to the position of static obstacles or applying random perturbation to the speed of dynamic obstacles in some simulation steps.

[0037] In step S2, when generating the final observation data, outlier handling is also included: when packet loss occurs, the most recent valid observation is used as a replacement; when the lidar ranging value exceeds the sensor's effective range or is lower than the minimum detection distance, the outlier value is replaced with a preset boundary value.

[0038] Furthermore, in step S5, the ROS2 Gazebo high-fidelity simulation environment configures differentiated communication quality strategies for different data types: the Best Effort strategy is used for LiDAR sensor data streams, which means that no reception acknowledgment is required and no retransmission is allowed; the Reliable strategy is used for speed control command streams, which means that reception acknowledgment and timeout retransmission are forcibly enabled.

[0039] As another important technical solution, this invention also provides a robot navigation system based on deep reinforcement learning and agent simulation, comprising:

[0040] The space definition module is used to define the state space, observation space, and motion space of robot navigation; the state space includes the robot's position, attitude angle, linear velocity, and angular velocity; the observation space includes lidar ranging information and the robot's relative pose to the target point; the motion space includes velocity control commands.

[0041] The proxy simulation module is used to construct a proxy simulation environment based on a simplified kinematic model according to the parameters of the state space, observation space, and action space. It describes obstacles geometrically and calculates ideal lidar ranging values ​​as raw observation data using the ray casting method. Sensor spatiotemporal noise is superimposed on the raw observation data to generate noisy observation data. Random delays and random packet loss to simulate network communication uncertainties are introduced into the noisy observation data to generate the final observation data. The robot's state transition is completed according to speed control commands and preset actuator dynamic constraints.

[0042] The TD3 network module is used to construct a deep reinforcement learning network structure based on the TD3 algorithm, which includes a policy network and an evaluation network. The policy network takes the final observation data as input and outputs normalized linear velocity and angular velocity control commands. The evaluation network evaluates the value of the robot's state and actions.

[0043] The pre-training module is used to pre-train the deep reinforcement learning network constructed in step S3 in the proxy simulation environment constructed in step S2, combining the multi-constraint reward function and the hybrid target generation strategy, to obtain the pre-trained policy parameters.

[0044] The fine-tuning deployment module is used to build a high-fidelity dynamics simulation environment for ROS2 Gazebo, which includes a robot rigid body dynamics model and sensor plugins. It loads pre-trained policy parameters into the ROS2 Gazebo high-fidelity simulation environment, fine-tunes the policy network parameters under real dynamics and communication conditions, and deploys the policy network onto the robot to perform online navigation tasks after the fine-tuning converges.

[0045] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the robot navigation method based on deep reinforcement learning and agent simulation as described above.

[0046] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute the robot navigation method based on deep reinforcement learning and agent simulation as described above.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] This invention constructs a proxy simulation environment based on a simplified kinematic model, avoiding the computational overhead of high-fidelity physical simulation during pre-training. This significantly reduces the complexity of single-step simulation, increases the training interaction frequency, and effectively shortens the training cycle while ensuring the learning effect of navigation strategies. It also reduces the consumption of computational resources and solves the problems of low training sample efficiency and long training time in the prior art.

[0049] This invention designs a multi-constraint reward function that couples linear velocity and angular velocity to encourage stable forward movement and penalizes low-speed, large-amplitude rotation. At the same time, a non-zero minimum linear velocity limit is set during the action mapping stage, which effectively suppresses local optimal behaviors such as stationary rotation and long-term stillness that are common in reinforcement learning navigation. This guides the robot to complete the navigation task in a more efficient manner, improving navigation efficiency and path quality.

[0050] This invention introduces key uncertainty factors such as sensor spatiotemporal noise, communication delay and packet loss, and actuator dynamic constraints in advance in the proxy simulation environment, enabling the strategy to adapt to multi-source uncertainties during the pre-training stage. Subsequently, it is fine-tuned in a high-fidelity simulation environment, which significantly reduces the performance gap between simplified simulation, high-fidelity simulation and real system, and enhances the robustness and reliability of the strategy in actual deployment.

[0051] This invention encapsulates the reinforcement learning training environment as a ROS2 node and uses a multi-threaded executor to handle sensor callbacks, control command issuance, and service calls. This effectively decouples the synchronous training process of reinforcement learning from the asynchronous communication mechanism of ROS2, avoiding training interruptions and node freezes caused by callback blocking or abnormal service calls, and improving the stability of the system during long-term training and operation in a distributed environment.

[0052] This invention introduces a resource management and adaptive perception processing mechanism to configure the number of CPU threads and the size of the experience replay buffer in the deep learning framework, thereby limiting the computational resource consumption of the training process, avoiding resource competition with the simulation system and communication middleware, and enabling the navigation method to run stably on platforms with limited computational resources. This improves the engineering deployability and application value of the method. Attached Figure Description

[0053] Figure 1 This invention presents a schematic diagram of the overall structure of a ROS2 mobile robot navigation system based on two-stage deep reinforcement learning and surrogate simulation.

[0054] Figure 2 : A schematic diagram of the kinematic agent simulation environment (Dummy environment) of this invention.

[0055] Figure 3 The system architecture and synchronous-asynchronous bridging structure of this invention in ROS2 are illustrated in the following diagram.

[0056] Figure 4 : A schematic diagram of the two-stage training process of this invention.

[0057] Figure 5 : A logical diagram of the triple-constraint reward function and hybrid objective generation strategy of this invention.

[0058] Figure 6 : A schematic diagram of the dummy pre-training state Q-value training state in an embodiment of the present invention.

[0059] Figure 7 : A schematic diagram of the training state of the average Q value of the Gazebo simulation in this embodiment of the invention.

[0060] Figure 8 : A schematic diagram of the training state of the Gazebo simulation algorithm loss in this embodiment of the invention.

[0061] Figure 9 : A schematic diagram of the maximum Q-value training state in the Gazebo simulation of this invention embodiment. Detailed Implementation

[0062] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0063] In some implementations, such as Figure 1 As shown, this invention provides a robot navigation method based on deep reinforcement learning and agent simulation, comprising the following steps:

[0064] S1: Define the state space, observation space, and action space for robot navigation; the state space includes the robot's position, attitude angle, linear velocity, and angular velocity; the observation space includes lidar ranging information and the robot's relative pose to the target point; the action space includes velocity control commands.

[0065] S2: Based on the parameters of the state space, observation space, and action space, a proxy simulation environment based on a simplified kinematic model is constructed; obstacles are described geometrically, and ideal lidar ranging values ​​are calculated using the ray casting method as the original observation data; sensor spatiotemporal noise is superimposed on the original observation data to generate noisy observation data; random delays and random packet loss to simulate network communication uncertainties are introduced into the noisy observation data to generate the final observation data; the robot state transition is completed according to the speed control command and the preset actuator dynamics constraints.

[0066] In some implementations, such as Figure 3 As shown, this agent environment is based on a simplified kinematic model, avoiding the computational overhead of high-fidelity physical simulation. At the same time, it proactively models key uncertainties that affect policy migration, thereby learning a robust basic navigation policy under low-cost conditions.

[0067] When describing obstacles using a geometric approach, random displacements are applied to the positions of static obstacles or random perturbations are applied to the movement speeds of dynamic obstacles in some simulation steps. This gives the environment dynamic perturbation characteristics and prevents the strategy from overfitting to the static scene.

[0068] In some implementations, the sensor spatiotemporal noise includes an angle factor related to the laser incident angle, an amplitude factor related to the ranging distance, and time-related noise described using an autoregressive model, specifically:

[0069] An angle factor related to the laser incident angle is used to simulate the increased measurement variance caused by diffuse reflection when a laser beam illuminates an obstacle surface at an angle. This factor forces navigation strategies to reduce their reliance on ranging data at large incident angles (such as wall edges), improving the robot's decision-making safety at corners or narrow passages.

[0070] An amplitude factor related to the ranging distance is used to simulate the amplification effect of long-distance ranging errors caused by laser beam divergence and reduced signal-to-noise ratio at the receiver as the detection distance increases.

[0071] Time-related noise described by an autoregressive model is easily filtered out by neural networks, while conventional Gaussian white noise is easily smoothed out. However, in real robot sensors, due to chassis vibration or internal hardware drift during continuous sampling, the measurement error is strongly correlated between adjacent time steps.

[0072] The proxy simulation environment of this invention breaks the assumptions of ideal sensors or memoryless white noise in conventional reinforcement learning. Without increasing the huge computational overhead of rigid body collisions in the physics engine, it covers the key physical defects of real sensors in advance with analytical mathematical models. This allows the policy network to learn robust feature representations to cope with complex noise during the first stage of pre-training, thereby effectively narrowing the gap between simulation and reality and greatly improving the success rate and stability of transferring the policy to high-fidelity environments and real robot platforms.

[0073] In some implementations, the generation of final observation data also includes handling outliers: when packet loss occurs, the most recent valid observation is used as a replacement; when the lidar ranging value exceeds the sensor's effective range or falls below the minimum detection distance, the outlier is replaced with a preset boundary value to prevent navigation mission interruption due to a single data source anomaly or short-term data loss.

[0074] The robot's state transition is achieved by combining speed control commands and preset actuator dynamic constraints. Specifically, based on the linear velocity and angular velocity control commands in the speed control instructions, and combined with the preset maximum acceleration and actuator response time constant, the robot's actual linear velocity and actual angular velocity are updated. The robot's new pose is then calculated using these actual linear velocity and angular velocity, completing the state transition. The calculation formula is as follows:

[0075] ;

[0076] ;

[0077] in, Indicates at time step The actual linear velocity of the robot at that time Indicates at time step The robot's actual angular velocity at that time; Indicates at time step The robot's actual linear velocity at that time; Indicates at time step The robot's actual angular velocity at that time; Indicates at time step The robot's desired linear velocity command. Indicates at time step The robot's desired angular velocity command; This represents the response time constant of the robot chassis actuator, used to characterize the time scale required for the motor system to reach steady state, and reflects the hysteresis characteristics of the system; This represents the maximum linear acceleration allowed by the robot chassis, used to simulate the dynamic saturation characteristics of the physical system; Indicates the time step of the simulation environment; This represents the cutoff function, which is taken if the theoretically required acceleration is less than the minimum positive acceleration. or If the theoretically required acceleration is greater than the maximum positive acceleration, then take... or Otherwise, the required acceleration is taken; this cutoff function is used to ensure that the calculated acceleration does not exceed the physical limit.

[0078] S3: Construct a deep reinforcement learning network structure based on the TD3 algorithm, which includes a policy network and an evaluation network; the policy network takes the final observation data as input and outputs normalized linear velocity and angular velocity control commands; the evaluation network evaluates the value of the robot's state and actions.

[0079] In some implementations, two parallel evaluation networks independently assess the expected value (Q-value) of the robot in its current state performing a specific action. When subsequently calculating the target action value, by minimizing the Q-values ​​output by the two evaluation networks, the overestimation of action value, a common problem in traditional deep reinforcement learning algorithms, can be effectively suppressed, thus providing a more robust and conservative optimization direction for the policy network.

[0080] A deep reinforcement learning network structure based on the TD3 algorithm is constructed, which mainly consists of a policy network and an evaluation network. The policy network, acting as an actor, receives the processed features of the concatenated LiDAR observations and relative target information at its input layer and outputs normalized linear and angular velocity control commands.

[0081] The processed lidar observation refers to the one-dimensional tensor obtained by sequentially performing outlier removal, downsampling, boundary truncation, and normalization mapping operations on the final observation data sequence. In other words, the original high-dimensional ray data is downsampled to a fixed dimension, and the ranging values ​​that exceed the maximum effective range are truncated to the maximum diagonal distance of the environment. Finally, the data is uniformly normalized to the interval of [-1,1] or [0,1].

[0082] Relative target information refers to the pose vector of the target point relative to the robot in the robot's current local coordinate system (i.e., a coordinate system with the robot's own center as the origin and the current direction of the vehicle as the front). Specifically, it includes the relative distance between the robot and the target point and the relative azimuth angle of the target point relative to the robot's direction of the vehicle. By using relative target information in the local coordinate system instead of global absolute coordinates, it is possible to ensure that the deep reinforcement learning network has strict translation invariance and rotation invariance, so that the trained navigation strategy can be generalized to unknown and complex map scenes with zero samples.

[0083] In some implementations, an experience replay buffer and a target network are configured, and training hyperparameters such as learning rate and discount factor are set. The experience replay buffer refers to the data generated by the robot when interacting with the agent environment or high-fidelity environment, which has a strong temporal correlation. The experience replay buffer breaks the temporal coupling between data by storing historical interaction data and adopting a random mini-batch sampling mechanism. The experience replay buffer allows historical high-value samples to be reused, which greatly improves the sample utilization rate in the first and second stages of training of this invention and effectively alleviates the problem of high interaction cost in high-fidelity simulation environment.

[0084] The target network refers to the network used when calculating the TD target value. If the main network, which is being updated rapidly, is used frequently, it is easy to cause training oscillations and overestimation of action value. This invention configures a target network with the same structure as the main network but with extremely slow parameter updates. A soft update mechanism is used to provide it with a stable and smooth supervision signal, thereby ensuring the absolute convergence stability of the robot's navigation strategy learning in complex dynamic obstacle environments.

[0085] In some implementations, a resource management mechanism for multi-process parallel systems is introduced to address the underlying computational power contention between deep learning training and high-fidelity physics simulation. The specific mechanism is as follows:

[0086] By setting environment variables at the system level to strictly limit the maximum number of CPU threads that the deep learning framework can call, the physics engine operation, sensor callback mechanism, and ROS2 node asynchronous communication always have a guaranteed CPU time slice, thus maintaining the frequency stability of the reinforcement learning control loop from the bottom layer.

[0087] The TD3 algorithm introduces a target policy smoothing mechanism in its target value calculation, expressed as follows:

[0088] ; ;

[0089] in, This represents the target action, which is determined by the target policy network based on the hypothetical next-moment state. The reference control instructions output by forward inference are necessary pre-variables for subsequent evaluation of expected future value; The parameters representing the policy network; It indicates the state at the next time step, including the robot's lidar observations and its own state at the next moment; This represents the target policy smoothing noise introduced to stabilize the training process of the main policy network by delaying the update of the copy. Mathematically, it is a truncated normal noise (i.e., noise with a mean of 0 and a standard deviation of 0). By sampling from a normal distribution and strictly truncating it within the boundary range of [-c, c], this restricted noise forces the evaluation network to perform average evaluation within a small neighborhood of the target action, thereby regularizing the value estimate, effectively smoothing the local space of the value function, and further suppressing the Q-value overestimation phenomenon commonly found in deep reinforcement learning. This indicates that the mean is 0 and the standard deviation is 0. The normal distribution This represents a cutoff function used to limit noise values ​​to the interval [-c, c] to prevent excessive noise from deviating from the actual motion distribution. Represents the noise boundary constant; Indicates the target Q value at the next moment; The parameter is The i A target evaluation network, in which i =1, 2, TD3 algorithm effectively alleviates the overvaluation problem that often occurs in the evaluation network in reinforcement learning by introducing two independent target evaluation networks and taking the minimum value of their output evaluation value; Indicates the first i Evaluation parameters for each network; This indicates that the robot receives a reward for each step of its actions in response to environmental feedback. This represents the reward discount factor, which weighs the current reward against the future reward.

[0090] Evaluation of the network loss function (Critic Loss): Minimizing the mean squared error of the two Critic networks:

[0091] ;

[0092] in, Indicates the first The loss function of a Critic network. This indicates the number of the Critic network, with a value of 1 or 2, corresponding to the two parallel Critic networks in the algorithm, respectively. Indicates the first The weight parameters of the Critic network, i.e., the variables that need to be updated through gradient descent; This represents the mathematical expectation operation, which in actual computation refers to the operation on the experience replay buffer. Calculate the mean of small-batch empirical data obtained from random sampling. This represents the experience playback buffer, which stores historical interaction data between the robot and its environment; Indicates from the buffer An empirical tuple sampled from the middle, wherein The current state is from the experience replay buffer. Samples sampled from the middle; The action currently being performed.

[0093] Indicates the first The current Q-value prediction of the Critic network, i.e., the evaluation network based on the input state. and actions The output is a value estimate.

[0094] In some implementations, the policy network loss function uses only the first Critic network. The evaluation network is the core evaluation component in the deep reinforcement learning Actor-Critic architecture of this invention. The evaluation network takes the robot's current environmental state features and the specific action commands generated by the policy network as joint inputs, passes them through a multi-layer feedforward neural network with nonlinear mapping, and outputs a scalar value of state and action value to update the policy network.

[0095] ;

[0096] in, This represents the loss function of the policy network, a negative value used to measure the performance of the current policy; The weight parameters of the policy network are the variables that need to be optimized through gradient descent. This represents the mathematical expectation operation, which in actual computation refers to the operation on the experience replay buffer. Calculate the mean of a small batch of randomly sampled state data. This represents the prediction of actions, i.e., the policy network predicts actions based on the current input state. The output is a deterministic action.

[0097] S4: In the proxy simulation environment constructed in step S2, the deep reinforcement learning network constructed in step S3 is pre-trained by combining the multi-constraint reward function and the hybrid target generation strategy to obtain the pre-trained policy parameters.

[0098] The multi-constraint reward function includes target approach reward, coupled forward reward, rotation penalty, and hierarchical obstacle avoidance penalty, specifically:

[0099] ;

[0100] In some implementations, the goal approaches the reward. This means that continuous positive guidance rewards are applied based on the Euclidean distance difference between adjacent time steps as the robot approaches the target point. When the distance falls below a set tolerance threshold, navigation is considered successful, the loop is terminated, and a high level completion reward is given. The expression is:

[0101] ;

[0102] in, The proportionality coefficient representing the reward for approaching the target is an adjustable hyperparameter used to amplify or reduce this reward item, thereby controlling the proportion of the action of moving towards the target in the total reward; This indicates that the robot took a step in the previous time period. t When -1, the Euclidean distance between its own position and the target point (i.e., the absolute straight-line distance between the two points). Indicates the robot's current time step t At that time, the Euclidean distance between its own position and the target point; if This means the robot is now closer to the goal than it was in the previous step, and the robot receives a positive reward. If This indicates that the robot is moving away from the target, and the robot receives a negative penalty.

[0103] Coupling forward reward This involves coupling the absolute values ​​of linear velocity and angular velocity, and providing a larger positive reward when moving approximately in a straight line towards the target. The expression is:

[0104] ;

[0105] in, This represents the weighting coefficient of the coupling forward reward; Indicates the robot at time step linear velocity; Indicates the robot at time step angular velocity; when angular velocity When the value is large, the reward will decrease or even become negative. Do not guide the robot to turn too quickly; only accelerate when it is going straight.

[0106] Spin penalty This indicates that when the linear velocity is below a preset threshold and the angular velocity is large, an additional penalty is applied to suppress low-speed, large-amplitude rotational behavior. The expression is:

[0107] ;

[0108] in, This represents the linear velocity threshold, used to determine whether the robot is in a near-stationary or low-speed state.

[0109] Tiered obstacle avoidance penalties This indicates that different levels of penalties are applied based on the distance range to the nearest obstacle. When the distance falls below the minimum safe threshold, the round ends and a heavier penalty is applied. The expression is:

[0110] ;

[0111] in, This indicates that the collision penalty will be terminated. Indicates the distance to the nearest obstacle; Indicates the physical collision threshold; Represents the obstacle avoidance potential field coefficient; This indicates the safe distance threshold.

[0112] In some implementations, single-step time penalty This means that a small negative penalty is applied at each time step of environmental interaction. By continuously introducing time costs, the robot is forced to complete navigation using the shortest path and in the fastest time, preventing it from wandering aimlessly for extended periods within the safe area. The expression is:

[0113] ;

[0114] in, This represents a fixed single-step penalty constant (usually a very small positive value, such as 0.01), which imposes a slight penalty on the robot's exploration time, guiding the agent to complete the navigation task with the shortest path and the fewest time steps, effectively preventing it from wandering aimlessly for a long time in an unobstructed and safe area.

[0115] Hybrid target generation strategy: During the round initialization phase, targets are generated in a local area with a certain probability, and targets are sampled evenly in the global area with another probability, taking into account both course learning and global exploration.

[0116] In some implementations, such as Figure 5 As shown, the multi-constraint reward function, the non-zero minimum linear velocity action constraint, and the hybrid target generation strategy together constitute a complete reinforcement learning pre-training closed-loop architecture in the proxy simulation environment. Their specific collaborative working mechanism is as follows:

[0117] During the initialization of each navigation round, the system dynamically switches between local generation and global sampling branches by determining random numbers and thresholds, thereby setting the robot's specific target point and providing guidance for the current round.

[0118] The TD3 policy network outputs the initial action based on the current state observation. This initial action, after passing through the action mapping constraint module, is forcibly mapped into an actual control command with a non-zero minimum linear velocity. This effectively blocks the robot's attempt to avoid collision penalties by remaining stationary for extended periods, thus ensuring exploration efficiency.

[0119] In some implementations, after the environment receives the speed command and completes the state transition, it sends the latest state input in parallel to three evaluation branches: a coupled forward branch (to encourage straight-line acceleration), a rotation penalty branch (to suppress spinning in place), and a graded obstacle avoidance branch (to protect the safety boundary). The three branches combine the target approach reward and the time penalty to jointly calculate the total reward of the current time step as a supervision feedback signal, which is then transmitted back to the TD3 policy network. The network uses this as a basis for gradient backpropagation and parameter updates.

[0120] In some implementations, such as Figure 4 As shown, S5: Construct a high-fidelity dynamics simulation environment for ROS2 Gazebo, which includes a robot rigid body dynamics model and sensor plugins; load the pre-trained policy parameters into the ROS2 Gazebo high-fidelity simulation environment, fine-tune the policy network parameters under real dynamics and communication conditions, and deploy the policy network onto the robot to perform online navigation tasks after the fine-tuning converges.

[0121] The ROS2 Gazebo simulation environment analyzes the robot's URDF / XACRO model through a low-level physics engine, introducing complex physical properties that are ignored by proxy environments, including but not limited to: the actual mass and inertia tensor of the robot chassis, the nonlinear sliding friction and rolling resistance between the wheels and the ground, the peak torque output limit of the actuator motor, and the rigid body forces and rebound feedback when a collision occurs.

[0122] The ROS2 Gazebo high-fidelity dynamics simulation environment is encapsulated into an environment node class that inherits from ROS2 nodes. This class is used to uniformly subscribe to sensor topics, publish speed control commands, and call the service interfaces for resetting the simulation and pausing / resuming the physics engine.

[0123] In some implementations, the ROS2 Gazebo high-fidelity simulation environment configures differentiated communication quality strategies for different data types: for lidar sensor data streams, the Best Effort strategy (i.e., a mechanism that does not require reception acknowledgment and does not retransmit, in order to reduce communication latency and prevent congestion) is adopted; for speed control command streams, the Reliable strategy (i.e., a reliable transmission mechanism that forces reception acknowledgment and timeout retransmission to ensure the reliability of command execution and prevent security incidents caused by packet loss) is adopted to ensure the reliability of command execution and prevent security incidents caused by packet loss.

[0124] By using a multi-threaded executor to run sensor callbacks and service processing in separate threads, and by enabling a service call rollback mechanism when simulation reset fails, the continuity and stability of the training process are ensured.

[0125] In some implementations, such as Figure 2 As shown in the figure, this diagram illustrates the system architecture and synchronous-asynchronous bridging mechanism for the interaction between reinforcement learning and the ROS2 simulation environment proposed in this invention. The architecture mainly comprises three core modules: reinforcement learning, ROS2 environment nodes, and the simulation environment.

[0126] As the underlying physics and sensor simulation engine, the simulation environment module is responsible for running the robot's rigid body dynamics model in real time. On one hand, it continuously publishes the robot's current state, position information, radar information, etc.; on the other hand, it receives chassis speed control commands to drive the robot's movement and initializes the simulation scene upon receiving a reset signal.

[0127] The ROS2 environment node module serves as the core bridging layer, acting as a hub connecting synchronous algorithms and asynchronous environments, and is scheduled by a multi-threaded executor. Internally, the node integrates sensor subscription, state caching, velocity publishing, and simulation control services. The sensor subscription module asynchronously receives state data from the simulation environment and updates the state cache in real time. The velocity publishing module is responsible for translating actions into standard velocity control commands for distribution, while the simulation control service sends reset requests to the simulation environment at the end of a round or upon triggering a collision.

[0128] The reinforcement learning module runs in an independent synchronous main loop, sequentially executing state reading, policy reasoning, and action issuance, and collecting state transition data during the interaction process to complete the training and update of the deep reinforcement learning network.

[0129] Through the aforementioned state caching mechanism and multi-threaded executor design, this system effectively decouples the synchronous training process of reinforcement learning from the asynchronous communication mechanism of ROS2, avoiding the problem of node freezing caused by sensor callback blocking or service call anomalies, and greatly improving the system's operational stability during long-term distributed training.

[0130] In some implementations, such as Figure 6 The figure shows a schematic diagram illustrating the change in the average Q-value as a function of the number of running steps during the agent simulation pre-training phase in this invention example. As can be seen from the figure, in the initial training stage, the agent is exploring the unknown environment, frequently triggering tiered obstacle avoidance penalties, and the average Q-value shows a decreasing trend. Subsequently, with the accumulation of interaction data and policy optimization, the agent gradually learns effective behaviors towards the target, and the average Q-value begins to steadily increase and eventually converges smoothly. This training result indicates that in the first stage of pre-training, the policy network successfully learns basic obstacle avoidance and approach navigation capabilities.

[0131] like Figure 7 The diagram shows the detailed training state during the high-fidelity simulation fine-tuning stage of this invention, including curves illustrating the changes in average Q-value, evaluation network loss, and maximum Q-value. The curves show that after initial fluctuations, the evaluation network loss rapidly decreases and tends to converge, indicating that the evaluation network can accurately fit the value distribution. Simultaneously, both the average Q-value and maximum Q-value maintain a smooth and consistent upward trend and eventually stabilize. This demonstrates that the TD3 dual-evaluation network structure and target policy smoothing mechanism employed in this invention effectively suppress the common Q-value overestimation problem in reinforcement learning, ensuring numerical stability throughout the training process.

[0132] like Figure 8 The diagram shows the variation curves of single-round reward and single-round steps during the fine-tuning training phase in this embodiment of the invention. As the training rounds progress, the total reward per round rapidly increases and stabilizes in the highest range above 150; correspondingly, the number of steps required per round decreases rapidly in the early stage and eventually stabilizes at a lower level (around 180 steps). This fully demonstrates that under the effective guidance of the multi-constraint reward function designed in this invention, the robot not only learns to safely reach the goal, but also actively plans a more efficient and shorter travel trajectory, successfully avoiding local optima traps such as lingering in place and prolonged stillness.

[0133] like Figure 9 The figure shows a schematic diagram of the final evaluation index status during the fine-tuning stage in an embodiment of the present invention, including evaluation curves for average reward, success rate, and collision rate. The data in the figure shows that as fine-tuning progresses, the navigation success rate steadily increases and eventually converges to a high success rate level close to 1.0 (above 97%). More importantly, the collision rate index remains at 0 throughout the entire evaluation period, directly confirming that the two-stage training paradigm of proxy simulation pre-training and high-fidelity dynamics fine-tuning proposed in this invention results in a policy network that not only possesses extremely high navigation efficiency in dynamic environments but also exhibits excellent safety and robustness, fully meeting the online safe deployment requirements of actual robots.

[0134] In another embodiment, the present invention proposes a robot navigation system based on deep reinforcement learning and agent simulation, comprising:

[0135] The space definition module is used to define the state space, observation space, and motion space of robot navigation; the state space includes the robot's position, attitude angle, linear velocity, and angular velocity; the observation space includes lidar ranging information and the robot's relative pose to the target point; the motion space includes velocity control commands.

[0136] The proxy simulation module is used to construct a proxy simulation environment based on a simplified kinematic model according to the parameters of the state space, observation space, and action space. It describes obstacles geometrically and calculates ideal lidar ranging values ​​as raw observation data using the ray casting method. Sensor spatiotemporal noise is superimposed on the raw observation data to generate noisy observation data. Random delays and random packet loss to simulate network communication uncertainties are introduced into the noisy observation data to generate the final observation data. The robot's state transition is completed according to speed control commands and preset actuator dynamic constraints.

[0137] The TD3 network module is used to construct a deep reinforcement learning network structure based on the TD3 algorithm, which includes a policy network and an evaluation network. The policy network takes the final observation data as input and outputs normalized speed control commands. The evaluation network evaluates the value of the robot's state and actions.

[0138] The pre-training module is used to pre-train the deep reinforcement learning network constructed in step S3 in the proxy simulation environment constructed in step S2, combining the multi-constraint reward function and the hybrid target generation strategy, to obtain the pre-trained policy parameters.

[0139] The fine-tuning deployment module is used to build a high-fidelity dynamics simulation environment for ROS2 Gazebo, which includes a robot rigid body dynamics model and sensor plugins. It loads pre-trained policy parameters into the ROS2 Gazebo high-fidelity simulation environment, fine-tunes the policy network parameters under real dynamics and communication conditions, and deploys the policy network onto the robot to perform online navigation tasks after the fine-tuning converges.

[0140] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a robot navigation method based on deep reinforcement learning and agent simulation as described above.

[0141] In another embodiment of the present invention, a computer-readable storage medium is provided storing a computer program that causes a computer to execute a robot navigation method based on deep reinforcement learning and agent simulation as described above.

[0142] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A robot navigation method based on deep reinforcement learning and surrogate simulation, characterized in that, Includes the following steps: S1: Define the state space, observation space, and motion space for robot navigation; the state space includes the robot's position, attitude angle, linear velocity, and angular velocity; the observation space includes lidar ranging information and the robot's relative pose to the target point; the motion space includes velocity control commands. S2: Based on the parameters of the state space, observation space, and action space, a proxy simulation environment based on a simplified kinematic model is constructed; obstacles are described geometrically, and ideal lidar ranging values ​​are calculated using the ray casting method as the raw observation data; sensor spatiotemporal noise is superimposed on the raw observation data to generate noisy observation data; random delays and random packet loss simulating network communication uncertainties are introduced into the noisy observation data to generate the final observation data; the robot's state transition is completed according to the speed control command and the preset actuator dynamics constraints. S3: Construct a deep reinforcement learning network structure based on the TD3 algorithm, which includes a policy network and an evaluation network; the policy network takes the final observation data as input and outputs normalized speed control commands; the evaluation network evaluates the value of the robot's state and actions; S4: In the proxy simulation environment constructed in step S2, the deep reinforcement learning network constructed in step S3 is pre-trained by combining the multi-constraint reward function and the hybrid target generation strategy to obtain the pre-trained policy parameters. S5: Construct a high-fidelity dynamics simulation environment for ROS2 Gazebo, which includes a robot rigid body dynamics model and sensor plugins; load the pre-trained policy parameters into the ROS2 Gazebo high-fidelity simulation environment, fine-tune the policy network parameters under real dynamics and communication conditions, and deploy the policy network onto the robot to perform online navigation tasks after the fine-tuning converges.

2. The robot navigation method based on deep reinforcement learning and surrogate simulation according to claim 1, characterized in that: In step S2, the robot state transition is completed by combining the speed control command and the preset actuator dynamics constraints. Specifically, the process is as follows: based on the linear velocity control command and angular velocity control command in the speed control command, and combined with the preset maximum acceleration and actuator response time constant, the robot's actual linear velocity and actual angular velocity are updated. The robot's new pose is then calculated using the actual linear velocity and actual angular velocity to complete the state transition. The calculation formula is: ； ； in, Indicates at time step The robot's actual linear velocity at that time; Indicates at time step The robot's actual angular velocity at that time; Indicates at time step The robot's actual linear velocity at that time; This represents the robot's actual angular velocity at time step t-1; Indicates at time step The desired linear velocity command for the robot; Indicates at time step The robot's desired angular velocity command; This represents the response time constant of the robot chassis actuator; This indicates the maximum linear acceleration allowed by the robot chassis. Indicates the time step of the simulation environment; This represents the truncation function.

3. The robot navigation method based on deep reinforcement learning and surrogate simulation according to claim 1, characterized in that: In step S3, the TD3 algorithm introduces a target policy smoothing mechanism in the calculation of the target value, expressed as: ； ； in, Indicates the target action; The parameters representing the policy network; It indicates the state at the next time step, including the robot's lidar observations and its own state at the next moment; This indicates that the target policy smooths out noise, and the target policy refers to the expected benchmark for outputting the next state action; This indicates that the mean is 0 and the standard deviation is 0. The normal distribution The standard deviation represents the normal distribution. This represents the truncation function; This represents the noise boundary constant.

4. The robot navigation method based on deep reinforcement learning and surrogate simulation according to claim 1, characterized in that: In step S4, the multi-constraint reward function This includes target approach rewards, coupled forward progress rewards, rotation penalties, and graded obstacle avoidance penalties, specifically: ； Target approach reward : ； Coupling forward reward : ； Spin penalty : ； Tiered obstacle avoidance penalties : ； Single-step time penalty : ； in, A coefficient representing the percentage reward for approaching the target; Let represent the Euclidean distance between the robot's position and the target point at the previous time step t-1; This represents the Euclidean distance between the robot's current position and the target point at the current time step t. The weighting coefficients represent the coupling forward reward; Indicates the robot at time step linear velocity; Indicates the robot at time step angular velocity; Indicates the rotation penalty coefficient; Indicates the linear velocity threshold; This indicates that the collision penalty will be terminated. Indicates the distance to the nearest obstacle; Indicates the physical collision threshold; Represents the obstacle avoidance potential field coefficient; Indicates the safe distance threshold; This represents a fixed single-step penalty constant.

5. The robot navigation method based on deep reinforcement learning and surrogate simulation according to claim 1, characterized in that: In step S2, constructing a proxy simulation environment based on a simplified kinematic model also includes: applying random displacements to the position of static obstacles or applying random perturbations to the motion speed of dynamic obstacles in some simulation steps.

6. The robot navigation method based on deep reinforcement learning and surrogate simulation according to claim 1, characterized in that: In step S2, when generating the final observation data, outlier handling is also included: when packet loss occurs, the most recent valid observation is used as a replacement; when the lidar ranging value exceeds the sensor's effective range or is lower than the minimum detection distance, the outlier value is replaced with a preset boundary value.

7. The robot navigation method based on deep reinforcement learning and surrogate simulation according to claim 1, characterized in that: In step S5, the ROS2 Gazebo high-fidelity simulation environment configures differentiated communication quality strategies for different data types: the Best Effort strategy is used for LiDAR sensor data streams, which means that no reception acknowledgment is required and no retransmission is allowed; the Reliable strategy is used for speed control command streams, which means that reception acknowledgment and timeout retransmission are forcibly enabled.

8. A robot navigation system based on deep reinforcement learning and agent simulation, characterized in that, include: The space definition module is used to define the state space, observation space, and motion space of robot navigation; the state space includes the robot's position, attitude angle, linear velocity, and angular velocity; the observation space includes lidar ranging information and the robot's relative pose to the target point; the motion space includes velocity control commands. The proxy simulation module is used to construct a proxy simulation environment based on a simplified kinematic model according to the parameters of the state space, observation space, and action space. It describes obstacles geometrically and calculates ideal lidar ranging values ​​as raw observation data using the ray casting method. Sensor spatiotemporal noise is superimposed on the raw observation data to generate noisy observation data. Random delays and random packet loss to simulate network communication uncertainties are introduced into the noisy observation data to generate the final observation data. The robot's state transition is completed according to speed control commands and preset actuator dynamic constraints. The TD3 network module is used to construct a deep reinforcement learning network structure based on the TD3 algorithm, which includes a policy network and an evaluation network. The policy network takes the final observation data as input and outputs normalized speed control commands. The evaluation network evaluates the value of the robot's state and actions. The pre-training module is used to pre-train the deep reinforcement learning network constructed in step S3 in the proxy simulation environment constructed in step S2, combining the multi-constraint reward function and the hybrid target generation strategy, to obtain the pre-trained policy parameters. The fine-tuning deployment module is used to build a high-fidelity dynamics simulation environment for ROS2 Gazebo, which includes a robot rigid body dynamics model and sensor plugins. It loads pre-trained policy parameters into the ROS2 Gazebo high-fidelity simulation environment, fine-tunes the policy network parameters under real dynamics and communication conditions, and deploys the policy network onto the robot to perform online navigation tasks after the fine-tuning converges.

9. An electronic device, characterized in that, include: The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a robot navigation method based on deep reinforcement learning and agent simulation as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute a robot navigation method based on deep reinforcement learning and agent simulation as described in any one of claims 1 to 7.

Images