A mobile robot navigation method based on deep reinforcement learning

Abstract

The application provides a kind of mobile robot navigation method based on deep reinforcement learning, realizes guiding to accelerate convergence before the training of deep reinforcement learning strategy, so that mobile robot can complete safe and efficient autonomous navigation only relying on on-board sensor in complex dynamic environment.The application comprises the following steps: S1, collecting data online, comprising the following steps: S11, training navigation model in simulation environment;Define the state space of mobile robot in simulation environment, and form state vector;S12, send state vector into graph attention network, obtain state representation;S13, policy network takes state representation as input, and outputs control action;The reference action of linear velocity and angular velocity is calculated by artificial potential field method;S2, during the training phase of navigation model, train critic network, policy network and graph attention network;S3, after the training of navigation model is completed, deploy navigation model on mobile robot.

Description

Technical Field

[0001] This invention belongs to the field of autonomous navigation and intelligent control technology for mobile robots, specifically relating to a mobile robot navigation method based on deep reinforcement learning, which can be applied to scenarios requiring autonomous navigation of mobile robots, such as warehousing and logistics, park inspection, indoor service robots, and outdoor inspection. Background Technology

[0002] In numerous real-world scenarios such as warehousing and logistics, industrial park inspection, manufacturing, and healthcare, autonomous mobile robots need to efficiently and safely complete navigation and obstacle avoidance tasks. Traditional navigation technologies mostly rely on prior maps and use methods such as grid search and sampling planning to generate paths. These methods can achieve good results when the map is complete and the environment is relatively simple, but their navigation performance will significantly decrease when faced with unknown or frequently changing environments, making it difficult to meet the needs of practical applications.

[0003] To reduce reliance on prior maps, existing technologies have proposed reactive methods based on sensor feedback, such as the artificial potential field method and the dynamic window method. These methods utilize lidar to acquire obstacle information and directly calculate control actions by designing a potential field function or cost function, offering advantages such as good real-time performance and relatively simple implementation. However, these methods are highly dependent on human experience and parameter tuning, and in complex scenarios with dense obstacles or narrow passages, they are prone to getting trapped in local optima, failing to achieve effective navigation and obstacle avoidance.

[0004] In recent years, deep reinforcement learning has been gradually applied to the field of mobile robot navigation. By allowing robots to interact with simulated environments and automatically learn continuous control strategies, the decision-making ability of robots in unknown environments has been improved to some extent. Existing methods often simply discretize LiDAR measurement results into a one-dimensional distance vector as the state input, or use demonstration data to guide the strategy; other studies utilize global observation information to construct an environmental graph structure and use graph neural networks to model global spatial relationships. These methods have improved navigation and obstacle avoidance performance under certain conditions, but still face many challenges in practical applications. On the one hand, most methods that use graph structures to characterize environmental states rely on global observations to construct environmental graphs, making it difficult to accurately depict the spatial relationships between obstacles when relying solely on local observation information. This limits their application in real-world scenarios where only local observations can be obtained. On the other hand, in terms of policy learning and guidance, some studies rely on demonstration data to provide prior knowledge. This not only requires large-scale, high-quality offline datasets for support, but also is prone to hindering the effective exploration of optimal solutions by the policy due to the excessive duration of imitation constraints, thus affecting the overall performance of the navigation policy.

[0005] Therefore, there is an urgent need for a technology that enables robots to perform efficient and safe navigation and obstacle avoidance in complex and dynamic environments by relying solely on onboard sensors. Summary of the Invention

[0006] The purpose of this invention is to overcome the above-mentioned shortcomings in the prior art and provide a mobile robot navigation method based on deep reinforcement learning. By constructing the observation data collected by lidar into a graph structure and feeding it into a graph attention network to generate a more effective state representation, and introducing reference actions generated by the artificial potential field method, guidance is achieved in the early stage of deep reinforcement learning strategy training to accelerate convergence. This enables the mobile robot to complete safe and efficient autonomous navigation in complex dynamic environments by relying only on airborne sensors.

[0007] The technical solution adopted by this invention to solve the above problems is: a mobile robot navigation method based on deep reinforcement learning, characterized by including the following steps: S1. Collect data online, including the following steps: S11. Train the navigation model in a simulation environment. The robot moves in the simulation environment and uses LiDAR to acquire surrounding obstacles. Define the state space of the mobile robot in the simulation environment. Divide the field of view of the LiDAR into M sectors evenly, and take the distance of the nearest obstacle in each sector to form the environment vector d. t =[d1,d2,…,d M ]; Combining the distance ρ between the robot and the target point t Target azimuth angle Δθ t Current linear velocity v t and angular velocity ω t , forming the state vector s t =[d1,…,d M ,ρ t , Δθ t v t ω t ]; S12, Transfer the state vector s t The data is fed into a graph attention network to obtain the enhanced state representation S. GAT,t ; S13, The policy network represents S using state representation. GAT,t The input is used to control the output action; at the same time, the reference action of linear velocity and angular velocity is calculated by the artificial potential field method. S2. During the training phase of the navigation model, the critic network, policy network, and graph attention network are trained. S3. After the navigation model is trained, deploy the navigation model on the mobile robot.

[0008] Step S12 of the present invention includes the following steps: S121. Graph Construction: Transform the state vector s t Construct it into a graph structure to obtain graph data; S122. Graph Attention Feature Extraction: Input the graph data into the graph attention network to obtain the updated node features; S123. Pooling and Concatenation: Perform mean pooling on the node features to obtain the environment feature vector. V L Represents the set of LiDAR sector nodes; S124. Transfer the environmental feature vector z env,t Initial characteristics of robot nodes By piecing them together, the final state representation is formed: .

[0009] Step S13 of the present invention includes the following steps: S131, The policy network represents S using state representation. GAT,t The input and output linear velocity and angular velocity control action a t =[v t ,ω t ]; S132, Robot executes control action a t Then, a reward r is obtained based on environmental feedback. t and the state vector s at the next time step t+1 Meanwhile, the artificial potential field method calculates the reference motion (v) of linear velocity and angular velocity based on the current observation data. apf,t ,ω apf,t ); S133, change the current state vector s t Control action a t Rewards r t The state s at the next moment t+1 Termination flag D t and reference action composition experience tuple (s t ,a t ,r t ,s t+1 D t ,v apf,t ,ω apf,t Then store it.

[0010] In step S2 of this invention, the critic network adopts a dual critic structure, denoted as Q1 and Q2, with network parameters as follows: and The corresponding target commentator network parameters are: and The target policy network parameter is θ'.

[0011] In step S2 of this invention, the training of the critic network includes the following steps: S1. Generate noisy target actions using a target policy network. The expression for this action is: , Where ε is the noise term, σ is the noise standard deviation, and c is the noise clipping threshold; S2. Input the target action into the dual-commenter network, and construct the temporal difference objective by taking the minimum value of its output. The temporal difference objective function is: , i∈{1,2}, Where γ is the discount factor; S3. The dual-commentator network uses the aforementioned temporal difference objective as the regression objective and employs mean squared error as the regression loss, where the gradient of the i-th commentator network is: , Where N is the batch sample size; S4. Using the gradient described above, update the commentator parameters via stochastic gradient descent: , where η Q The learning rate of the commentator network.

[0012] In step S2 of this invention, the training of the policy network includes the following steps: S1. Introduce an artificial potential field consistency aid commentator Q. APF Let the control action output by the current policy network be a. t =[v t ,ω t Define the artificial potential field consistency auxiliary commentator as: , where k is the gain coefficient; S2. Align the output of the critic network Q1 with the artificial potential field to assist critic Q. APF Combining and constructing a hybrid loss function for the policy network: , λ t ∈[0,1], Where λ t These are weights that monotonically decrease as the training progresses. Taking the derivative of the mixed loss function with respect to the policy parameter θ, we obtain the policy gradient: ; S3. Update the policy network parameters using stochastic gradient descent, utilizing the policy gradient: , where η π is the learning rate of the policy network.

[0013] In step S2 of this invention, the training of the graph attention network includes the following steps: S1, the parameters of the graph attention network are denoted as... The corresponding state mapping is: ; S2, Critics' Loss on Graph Attention Network Parameters The gradient is: ,i∈{1,2}; S3. Combine the gradients from the two commentators and update the parameters of the graph attention network: , where η GAT To calculate the learning rate of the attention network.

[0014] In step S1 of this invention, the reward r t The construction includes the following steps: Set a reward R for getting closer to the target based on changes in target distance. C Safety penalty term R based on the distance to the nearest obstacle O Smoothing penalty term R based on angular velocity change V Acceleration penalty term R based on linear velocity change a And the penalty term R representing the time cost. t The reward r is obtained by weighted summation. t Its expression is: , Where w c w o w v w a w t These are the weighting coefficients for each item.

[0015] This invention deploys the trained navigation model onto a mobile robot. The mobile robot's onboard control unit periodically collects environmental perception information, robot pose information, and target point information, and combines them to construct the current state vector s. t Input the graph into an attention network and a policy network to obtain the control action [v] t ,ω t The command is then sent to the control module for execution, enabling the robot to navigate autonomously in unknown environments.

[0016] In step S1 of this invention, the 180° field of view of the lidar is uniformly divided into 20 sectors in the simulation environment.

[0017] Compared with the prior art, the present invention has the following advantages and effects: 1. This invention uses a graph attention mechanism to represent the state of local observation data, which can accurately depict the spatial relationship between obstacles based solely on local observation information, without relying on global observation and prior maps, thus reducing the dependence on environmental information.

[0018] 2. By introducing an artificial potential field consensus-assisted commentator, without relying on large-scale demonstration data, the reference actions generated by the artificial potential field method guide the policy in the early stage of training, which accelerates the convergence speed and ensures the safety of the policy. In the later stage of training, the policy update is gradually transitioned to be dominated by value estimation, which takes into account the optimality of navigation.

[0019] 3. Finally, the navigation model is deployed on the self-developed robot platform. Without the need for parameter retuning, it can complete multi-scenario navigation and obstacle avoidance experiments in unknown outdoor environments, including regular obstacles, irregular obstacles, random disturbance obstacles, and moving pedestrians. Collision-free round-trip navigation can be achieved in each scenario, which has good practicality and robustness and can adapt to a variety of complex and dynamic real-world application scenarios.

[0020] 4. The present invention can achieve stable convergence in about 1000 rounds during simulation training, with a final navigation success rate of about 0.95. Compared with the comparison algorithms DDPG, SAC and TD3, it has a faster convergence speed and a higher success rate, while having fewer average planning steps, which can significantly improve training efficiency and navigation performance.

[0021] 5. In tests on the publicly available BARN dataset, the navigation model trained by this invention achieved an average success rate of 0.88 and an average planning step count of approximately 166 steps across 300 test scenarios. It also maintained a superior level in terms of time cost and path length. Compared with comparative algorithms such as DDPG, SAC, and TD3, it showed significant improvements in both success rate and efficiency, verifying the effectiveness and superiority of the navigation model in complex scenarios. Attached Figure Description

[0022] Figure 1 This is a general flowchart of an embodiment of the present invention; Figure 2 This is a schematic diagram of the state space of a mobile robot according to an embodiment of the present invention; Figure 3 This is a simulation training environment diagram of an embodiment of the present invention; Figure 4 This is a comparison curve of the success rate of different methods during training in a static random obstacle environment. Figure 5 This is a graph showing the comparison of reward values ​​for different methods during training in a static random obstacle environment. Figure 6This is a graph showing the comparison of the number of steps per round for different methods during training in a static random obstacle environment. Figure 7 A comparative curve of success rate of different methods during training under dynamic pedestrian interference environment; Figure 8 This is a comparison curve of reward values ​​for different methods during training under dynamic pedestrian interference. Figure 9 This is a comparison curve of the number of steps per round for different methods during training in a dynamic pedestrian interference environment; Figure 10 This is a comparison chart of the distribution of planning steps in fifty test environments according to an embodiment of the present invention; Figure 11 This is a diagram showing the motion trajectory of the robot controlled by the DWA method in a dense BARN dataset scenario. Figure 12 This is a graph showing the motion trajectory of the robot controlled by the DRL_VO method in a dense BARN dataset scenario. Figure 13 This is a graph showing the motion trajectory of the robot controlled by the SAC method in a dense BARN dataset scenario. Figure 14 This is a graph showing the motion trajectory of the robot controlled by the TD3 method in a dense BARN dataset scenario. Figure 15 This is a graph showing the motion trajectory of the robot controlled by the APF-TD3 method in a dense BARN dataset scenario. Figure 16 This is a motion trajectory diagram of the robot controlled by the method of this invention in a dense BARN dataset scenario. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0024] I. The present invention, GAVEL-TD3, includes the following steps: S1. Collect data online, including the following steps: S11, such as Figure 3 As shown, in this embodiment, the navigation model is trained in the Gazebo simulation environment. The robot moves in the simulation environment according to the current control action and uses LiDAR to obtain surrounding obstacles.

[0025] Define the state space of the mobile robot in the simulation environment. In the simulation environment, the 180° field of view of the LiDAR is uniformly divided into 20 sectors, and the distance to the nearest obstacle in each sector is taken to form the environment vector d. t =[d1,d2,…,d 20 ]; Combining the distance ρ between the robot and the target point t Target azimuth angle Δθ t Current linear velocity v t and angular velocity ω t This forms a 24-dimensional state vector, whose expression is: s t =[d1,…,d 20 ,ρ t , Δθ t v t ω t ].

[0026] This state space fully describes the robot's local observation data, including environmental information, target information, and its own motion state at the current moment.

[0027] S12, Reward r for constructing the navigation model t To establish a controllable trade-off between proximity to the target, collision safety, smooth motion, and time cost, the following steps are included: Set a reward R for getting closer to the target based on changes in target distance. C Safety penalty term R based on the distance to the nearest obstacle O Smoothing penalty term R based on angular velocity change V Acceleration penalty term R based on linear velocity change a And the penalty term R representing the time cost. t The reward r is obtained by weighted summation. t Its expression is: , Where w c w o w v w a w t These are the weighting coefficients for each item, which can be adjusted according to the needs of the actual application scenario.

[0028] S13, convert the above state vector s t The data is fed into a graph attention network, which is then used to process the state vector s. t State representation is performed to obtain the enhanced state representation S. GAT,t This includes the following steps: S131. Graph Construction: Convert the state vector s obtained in step S11 above into a graph. tThe data is constructed as a graph structure, resulting in graph data containing LiDAR sector nodes and robot body nodes.

[0029] S132. Graph Attention Feature Extraction: Input the graph data into the graph attention network to obtain the updated node features.

[0030] S133. Pooling and Stitching: Mean pooling is performed only on the features of the LiDAR sector nodes to obtain the environmental feature vector, whose expression is: V L This represents the set of LiDAR sector nodes.

[0031] S134. Transfer the environmental feature vector z env,t Initial characteristics of robot nodes By splicing together the data, a reinforced state representation is formed, the expression of which is: .

[0032] S14, The policy network represents S using state representation. GAT,t Input, output control action a t Simultaneously, the artificial potential field guidance module calculates reference actions from the original observation data using the artificial potential field method, which are used for policy guidance in subsequent training phases; finally, the current state vector s is... t Control action a t Rewards r t The state vector s at the next time step t+1 And reference actions are stored for use during the training phase. Specifically, this includes the following steps: S141, Policy Network Represents S as a State GAT,t The input and output linear velocity and angular velocity control action a t =[v t ,ω t ], used to drive the robot's movement.

[0033] S142, Robot executes control action a t Then, a reward r is obtained based on environmental feedback. t and the state vector s at the next time step t+1 Meanwhile, the artificial potential field guidance module uses the artificial potential field method based on the current observation data. t The reference motion (v) for calculating linear velocity and angular velocity. apf,t ,ω apf,t ).

[0034] S143. Finally, the current state vector s t Control action a t Rewards r t The state s at the next moment t+1 Termination flag Dt and reference action composition experience tuple (s t ,a t ,r t ,s t+1 D t ,v apf,t ,ω apf,t ), where the termination flag D t This is used to identify whether the current state is a terminated state; then it is stored in the experience replay pool for subsequent network training.

[0035] S2. During the training phase of the navigation model, the critic network, policy network, and graph attention network are trained. Each training iteration samples empirical tuples (s) from the empirical replay pool. t ,a t ,r t ,s t+1 D t ,v apf,t ,ω apf,t ), and transform it into (S) through a graph attention network. GAT,t ,a t ,r t ,S GAT,t+1 D t ,v apf,t ,ω apf,t Specifically, it includes the following steps: S21. Critics Network Training, including the following steps: A dual-commentator structure is adopted, denoted as Q1 and Q2, with network parameters as follows: and The corresponding target commentator network parameters are: and The target policy network parameters are θ'. In each training step: S211. First, generate noisy target actions using the target policy network, the expression of which is: , Where ε is the noise term, used to increase exploratory power, σ is the noise standard deviation, and c is the noise clipping threshold.

[0036] S212. Next, the target action is input into two target critic networks to obtain the corresponding Q-value estimates. The minimum value of their outputs is taken to construct the temporal difference objective. The temporal difference objective function is: , i∈{1,2}, Here, γ is the discount factor, used to balance the weights of current rewards and future rewards.

[0037] S213. Next, the two commentator networks use the aforementioned temporal difference objective as the regression objective and employ mean squared error as the regression loss, where the gradient of the i-th commentator network is: , i∈{1,2}, Where N is the number of samples in the batch.

[0038] S214. Finally, using the gradients described above, update the commentator parameters via stochastic gradient descent: ,i∈{1,2}, Where η Q The learning rate of the commentator network.

[0039] S22. Policy network training includes the following steps: S221. Introduce an artificial potential field consistency aid commentator Q. APF Let the control action output by the current policy network be a. t =[v t ,ω t Define the artificial potential field consistency auxiliary commentator as: , Where k is the gain coefficient, the above formula reaches its minimum value when the current speed is consistent with the reference speed, and the greater the deviation, the greater the penalty. This function is used to measure the consistency between the policy and the prior action.

[0040] S222. In strategy optimization, the output of the critic network Q1 is aligned with the artificial potential field to assist critic Q. APF Combining and constructing a hybrid loss function for the policy network: , λ t ∈[0,1], Where λ t For weights that monotonically decrease as training progresses, λ is used in the early stages of training. t The λ value is relatively large, and the updates are mainly dominated by the artificial potential field constraint, which makes the policy converge quickly within the safe region near the reference action. In the later stages of training, λ... t Gradually reduce and smoothly transition the focus of updates to a dual-commentator-led approach, thereby achieving a balance between safety and optimality.

[0041] Taking the derivative of the above equation with respect to the policy network parameters θ, we obtain the policy gradient: .

[0042] S223. Using the gradients described above, update the network parameters using a stochastic gradient descent strategy: , where η π is the learning rate of the policy network.

[0043] S23. Training the graph attention network, including the following steps: S231, the parameters of the graph attention network are denoted as... The corresponding state mapping is: ; S232. During training, to stabilize the optimization process, the policy branch adjusts the S... GAT,t The "stop gradient" operation is used, meaning that the graph attention network parameters are not updated directly through the policy loss. The graph attention network is jointly trained using only the commentator regression loss, ensuring that the learned state representations are consistent with the value estimates.

[0044] Due to the commentator's output Q i (S GAT,t ,a t Depends on state representation S GAT,t Critics' loss on graph attention network parameters The gradient is: ,i∈{1,2}; S233. Combining the gradients from the two commentators, update the parameters of the graph attention network: , where η GAT To calculate the learning rate of the attention network.

[0045] S3. After the navigation model is trained, deploy the navigation model on the mobile robot, including the following steps: The trained navigation model is deployed on a mobile robot. In practical applications, the mobile robot's onboard control unit periodically collects environmental perception information, robot pose information, and target point information, and combines them in the aforementioned manner to construct the current state vector s. t Input the graph into an attention network and a policy network to obtain the control action [v] t ,ω t The command is then sent to the control module for execution, thereby enabling the robot to navigate autonomously in unknown environments.

[0046] II. Experiment.

[0047] like Figures 4-9As shown, the training process of this invention's GAVEL-TD3 was compared with algorithms such as DDPG, TD3, SAC, and APF-TD3 in static random obstacle environments and dynamic environments with pedestrian interference. APF-TD3 is an ablation algorithm that adds an artificial potential field consensus-assisted commentator to TD3 without using graph attention state representation. The success rate curves show that GAVEL-TD3 rises the fastest and stabilizes at a high level close to 1 earlier, followed by APF-TD3, while TD3, DDPG, and SAC have slower convergence speeds and lower stability values. In the reward curves, GAVEL-TD3 has the highest steady-state reward and the smallest fluctuation. In the step count curves, GAVEL-TD3 requires the fewest steps and has the smallest variance, while APF-TD3 falls between TD3 and GAVEL-TD3. This indicates that introducing an artificial potential field consensus-assisted commentator can significantly improve convergence speed and safety, and further combining it with graph attention state representation maximizes the overall decision-making ability.

[0048] To verify the comprehensive performance of the algorithm in complex and unknown environments, this embodiment randomly selected fifty test environments from the BARN dataset to evaluate the GAVEL-TD3, APF-TD3, TD3, and SAC of this invention.

[0049] like Figure 10 The decision-making step statistics shown (only successful samples are plotted) indicate that TD3 and APF-TD3 have moderate step counts in some simple scenarios, but experience a sharp increase in step counts in some complex environments, with a relatively scattered distribution, indicating unstable decision-making. SAC's step count distribution is relatively concentrated but generally high, suggesting that while it can complete the task, each planning step requires a large number of steps, resulting in low decision-making efficiency. In contrast, the scatter points corresponding to the present invention GAVEL-TD3 are almost entirely concentrated in the lower region of the graph, with minimal fluctuations across different scenarios, achieving both a high success rate and higher decision-making efficiency. Figures 11-16 The typical scenario trajectory further reveals its performance difference: when facing dense obstacles and narrow passages, other algorithms all collide because they cannot effectively avoid obstacles, while only the GAVEL-TD3 of this invention can plan a safe and smooth trajectory to reach the target smoothly.

[0050] Furthermore, it should be noted that the specific embodiments described in this specification may differ in the shape and name of their components, and the above description is merely illustrative of the structure of the present invention. All equivalent or simple variations made based on the structure, features, and principles described in this patent concept are included within the scope of protection of this patent.

Claims

1. A mobile robot navigation method based on deep reinforcement learning, characterized in that: Includes the following steps: S1. Collect data online, including the following steps: S11. Train the navigation model in a simulation environment. The robot moves in the simulation environment and uses LiDAR to acquire surrounding obstacles. Define the state space of the mobile robot in the simulation environment. Divide the field of view of the LiDAR into M sectors evenly, and take the distance of the nearest obstacle in each sector to form the environment vector d. t =[d1,d2,…,d M ]; Combining the distance ρ between the robot and the target point t Target azimuth angle Δθ t Current linear velocity v t and angular velocity ω t , forming the state vector s t =[d1,…,d M ,ρ t , Δθ t v t ω t ]; S12, Transfer the state vector s t The data is fed into a graph attention network to obtain the enhanced state representation S. GAT,t Step S12 includes the following steps: S121. Graph Construction: Transform the state vector s t Construct it into a graph structure to obtain graph data; S122. Graph Attention Feature Extraction: Input the graph data into the graph attention network to obtain the updated node features; S123. Pooling and Concatenation: Perform mean pooling on the node features to obtain the environment feature vector. V L Represents the set of LiDAR sector nodes; S124. Transfer the environmental feature vector z env,t Initial characteristics of robot nodes By piecing them together, the final state representation is formed: ; S13, The policy network represents S using state representation. GAT,t The input is used to control the output action; at the same time, the reference action of linear velocity and angular velocity is calculated by the artificial potential field method. S2. During the training phase of the navigation model, the critic network, policy network, and graph attention network are trained. S3. After the navigation model is trained, deploy the navigation model on the mobile robot.

2. The mobile robot navigation method based on deep reinforcement learning according to claim 1, characterized in that: Step S13 includes the following steps: S131, The policy network represents S using state representation. GAT,t The input and output linear velocity and angular velocity control action a t =[v t ,ω t ]; S132, Robot executes control action a t Then, a reward r is obtained based on environmental feedback. t and the state vector s at the next time step t+1 Meanwhile, the artificial potential field method calculates the reference motion (v) of linear velocity and angular velocity based on the current observation data. apf,t ,ω apf,t ); S133, change the current state vector s t Control action a t Rewards r t The state s at the next moment t+1 Termination flag D t and reference action composition experience tuple (s t ,a t ,r t ,s t+1 D t ,v apf,t ,ω apf,t Then store it.

3. The mobile robot navigation method based on deep reinforcement learning according to claim 2, characterized in that: In step S2, the critic network adopts a dual critic structure, denoted as Q1 and Q2, with network parameters as follows: and The corresponding target commentator network parameters are: and The target policy network parameter is θ'.

4. The mobile robot navigation method based on deep reinforcement learning according to claim 3, characterized in that: In step S2, the training of the critic network includes the following steps: S1. Generate noisy target actions using a target policy network. The expression for this action is: ， Where ε is the noise term, σ is the noise standard deviation, and c is the noise clipping threshold; S2. Input the target action into the dual-commenter network, and construct the temporal difference objective by taking the minimum value of its output. The temporal difference objective function is: , i∈{1,2}, Where γ is the discount factor; S3. The dual-commentator network uses the aforementioned temporal difference objective as the regression objective and employs mean squared error as the regression loss, where the gradient of the i-th commentator network is: , Where N is the batch sample size; S4. Using the gradient described above, update the commentator parameters via stochastic gradient descent: , where η Q The learning rate of the commentator network.

5. The mobile robot navigation method based on deep reinforcement learning according to claim 3, characterized in that: In step S2, the training of the policy network includes the following steps: S1. Introduce an artificial potential field consistency aid commentator Q. APF Let the control action output by the current policy network be a. t =[v t ,ω t Define the artificial potential field consistency auxiliary commentator as: , where k is the gain coefficient; S2. Align the output of the critic network Q1 with the artificial potential field to assist critic Q. APF Combining and constructing a hybrid loss function for the policy network: ,l t ∈[0,1], Where λ t These are weights that monotonically decrease as the training progresses. Taking the derivative of the mixed loss function with respect to the policy parameter θ, we obtain the policy gradient: ； S3. Update the policy network parameters using stochastic gradient descent, utilizing the policy gradient: , where η π is the learning rate of the policy network.

6. The mobile robot navigation method based on deep reinforcement learning according to claim 3, characterized in that: In step S2, the training of the graph attention network includes the following steps: S1, the parameters of the graph attention network are denoted as... The corresponding state mapping is: ; S2, Critics' Loss on Graph Attention Network Parameters The gradient is: ,i∈{1,2}； S3. Combine the gradients from the two commentators and update the parameters of the graph attention network: , where η GAT To calculate the learning rate of the attention network.

7. The mobile robot navigation method based on deep reinforcement learning according to claim 2, characterized in that: In step S1, the reward r t The construction includes the following steps: Set a reward R for getting closer to the target based on changes in target distance. C Safety penalty term R based on the distance to the nearest obstacle O Smoothing penalty term R based on angular velocity change V Acceleration penalty term R based on linear velocity change a And the penalty term R representing the time cost. t The reward r is obtained by weighted summation. t Its expression is: , Where w c w o w v w a w t These are the weighting coefficients for each item.

8. The mobile robot navigation method based on deep reinforcement learning according to claim 1, characterized in that: After the trained navigation model is deployed on the mobile robot, the onboard control unit of the mobile robot periodically collects environmental perception information, robot pose information, and target point information, and combines them to construct the current state vector s. t Input the graph into an attention network and a policy network to obtain the control action [v] t ,ω t The command is then sent to the control module for execution, enabling the robot to navigate autonomously in unknown environments.

9. The mobile robot navigation method based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the 180° field of view of the lidar is uniformly divided into 20 sectors in the simulation environment.

Images