A method for autonomous navigation of robots in dynamic environments based on deep reinforcement learning

Abstract

The present invention discloses a method for autonomous navigation of a robot in a dynamic environment based on deep reinforcement learning, including: setting the observation space and action space of a mobile robot; obtaining and preprocessing environmental data, extracting features from laser radar and depth camera data, and generating a feature map of the observation space of the mobile robot; obtaining the target position of the robot, using the A* algorithm to generate a path from the current position to the global target, and generating sub-target points according to the current position and the global path; constructing a PPO deep reinforcement learning network, using the observation space feature map and sub-target points as inputs of the network, and outputting the state value, forward speed and turning speed of the mobile robot to achieve autonomous navigation; designing a reward function, establishing a simulation environment, and training a deep reinforcement learning network based on the reward function for autonomous navigation in a real-time complex dynamic environment. The mobile robot is able to actively avoid collisions with pedestrians and obstacles while keeping moving forward.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile robot autonomous navigation, and in particular relates to a robot autonomous navigation method in a dynamic environment based on deep reinforcement learning. Background Art

[0002] With the growing demand for industrial automation and intelligent service robots, mobile robots play an important role in many fields, such as logistics and distribution, environmental monitoring, cleaning services, and personal assistance.

[0003] Navigation is a basic task for mobile robots. In practical applications, robots need to navigate safely and effectively in complex and dynamically changing environments, including static obstacles and dense crowds. These factors require robots to be able to perceive changes in the surrounding environment in real time, process large amounts of data, and make quick decisions. This not only places high demands on the robot's perception system and computing power, but also poses severe challenges to the robustness and adaptability of navigation algorithms. Summary of the invention

[0004] In response to the above technical problems, the present invention provides a robot autonomous navigation method in a dynamic environment based on deep reinforcement learning.

[0005] The technical solution adopted by the present invention to solve the technical problem is:

[0006] A method for autonomous navigation of a robot in a dynamic environment based on deep reinforcement learning, the method comprising the following steps:

[0007] S100: Setting the observation space and action space of the mobile robot;

[0008] S200: Acquire environmental data, pre-process the environmental data, extract features from the laser radar and depth camera data, obtain a radar data feature map and a pedestrian data feature map, and then generate a mobile robot observation space feature map;

[0009] S300: Obtain the target position of the robot, use the A* algorithm to generate a path from the current position to the global target, and generate sub-target points according to the current position and the global path;

[0010] S400: construct a PPO deep reinforcement learning network, take the observation space feature map and sub-target points as the input of the PPO deep reinforcement learning network, and output the state value, forward speed and turning speed of the mobile robot;

[0011] S500: Design a reward function, establish a simulation environment, train the PPO deep reinforcement learning network based on the state value and reward function of the mobile robot, and obtain a trained PPO deep reinforcement learning network after training for a set total number of time steps. Based on the trained PPO deep reinforcement learning network, obtain the real-time forward speed and turning speed of the mobile robot to achieve real-time autonomous navigation in complex dynamic environments.

[0012] Preferably, S100 includes:

[0013] S110: The observation space of the mobile robot includes the laser radar historical data, the kinematic information of pedestrians within a preset range centered on the mobile robot, and the position of the sub-target point, specifically:

[0014] (1)

[0015] in, Represents the historical data of LiDAR. represents the kinematic information of pedestrians within a preset range centered on the mobile robot. Indicates the location of the sub-target point; Indicates the moment;

[0016] S120: The motion space of the mobile robot includes the robot forward speed and the robot turning speed, specifically:

[0017] (2)

[0018] in, represents the forward speed of the mobile robot, represents the turning angular velocity of the mobile robot, represents the time, where m / s, rad / s.

[0019] Preferably, S200 includes:

[0020] S210: Use a laser radar sensor to obtain laser scanning data of the surrounding environment in real time, perform minimum and average pooling on the laser radar data of the past ten scans, and generate historical distance measurement data, that is, a (1,80,80)-dimensional radar data feature map;

[0021] S220: Use a depth camera and the YOLO target detection algorithm to detect pedestrians in the environment and record their relative positions. Use a multi-hypothesis tracking algorithm to track the movement of pedestrians, obtain their position and speed information, map the position and speed information of pedestrians into a grid map, and generate a (2,80,80)-dimensional pedestrian kinematic feature map.

[0022] S230: The radar data feature map and the pedestrian kinematic feature map together constitute an observation space feature map of the mobile robot.

[0023] Preferably, S300 includes:

[0024] S310: Determine the final destination that the robot needs to reach, and use the A* algorithm to generate a path from the current position to the global goal;

[0025] S320: Calculate the path point closest to the robot on the path;

[0026] S330: Determine a sub-target point according to the current position and the nearest point;

[0027] S340: Convert the global target point to a target point in the robot local coordinate system.

[0028] Preferably, S320 is specifically:

[0029] S321: Define the path and current position, assuming the path consists of a sequence of line segments , ,…, Composition, of which is a point on the path, and the current robot position is ;

[0030] S322: Traverse each segment on the path, and for the first Segment, its starting point ,end ;

[0031] S323: Calculate the direction vector, the unit direction vector, and the vector from the starting point of the line segment to the robot position. The specific formula is as follows:

[0032] (3)

[0033] In formula (3), is the direction vector of the line segment;

[0034] (4)

[0035] In formula (4), is the unit direction vector of the line segment;

[0036] (5)

[0037] In formula (5), is the vector from the starting point of the line segment to the robot position;

[0038] S324: Calculate the projection distance based on the vector from the starting point of the line segment to the robot position and the direction vector of the line segment , the specific formula is as follows:

[0039] (6)

[0040] S325: Determine the nearest point based on the projection distance and the direction vector of the line segment. The specific formula is as follows:

[0041] (7)

[0042] S326: Calculation based on projection distance and closest point The vertical distance to the line segment is as follows:

[0043] (8)

[0044] S327: Repeat S323 to S326 for all line segments, and record the minimum vertical distance and the corresponding nearest point.

[0045] Preferably, S330 specifically includes:

[0046] (9)

[0047] In formula (9), is the set look-ahead distance, where .

[0048] Preferably, the PPO deep reinforcement learning network in S400 includes a feature extraction network, a fully connected layer, an Actor network and a Critic network; the feature extraction network processes the input observation space feature map, and extracts the intermediate feature map through a series of convolutional layers, pooling layers and three embedded efficient attention mechanism bottleneck residual blocks; wherein, the output of the first pooling layer of the feature extraction network is feature concatenated with the output of the third bottleneck residual block and passed to the second pooling layer, and the output of the first bottleneck residual block of the feature extraction network is also feature concatenated with the output of the second bottleneck residual block, and the result is input to the third bottleneck residual block; each bottleneck residual block consists of six convolutional layers and two channel attention mechanisms. The force module is divided into two groups, each of which contains three consecutive convolutional layers followed by a channel attention module, where the input of the first convolutional layer in each group is concatenated with the output of the channel attention module to form the input of the next group; the fully connected layer receives the intermediate feature map and sub-target point information output by the feature extraction network, and further processes it to generate a high-level feature map; the Actor network is composed of two fully connected layers, which takes the high-level feature map as input, and outputs the turning speed and forward speed of the mobile robot; the Critic network also contains two fully connected layers, which takes the same high-level feature map as input, outputs the value function of the mobile robot's action, and provides feedback to the Actor network to optimize the strategy.

[0049] Preferably, S500 includes:

[0050] S510: Design multi-goal rewards, including goal achievement rewards, obstacle avoidance rewards, path smoothness rewards, and active heading rewards;

[0051] S520: Establish a simulation environment, which is the Gazebo platform in the Noetic version of ROS. Use the pedsim_ros model to simulate complex dynamic environment scenes. During the training process, randomly set target points. Perform simulation training and testing according to the network strategy based on the state value and reward function of the mobile robot.

[0052] S530: After training for a set total number of time steps, a trained PPO deep reinforcement learning network is obtained. Based on the trained PPO deep reinforcement learning network, the real-time forward speed and turning speed of the mobile robot are obtained to achieve real-time autonomous navigation in a complex dynamic environment.

[0053] Preferably, the multi-target reward in S510 is specifically:

[0054] (10)

[0055] In formula (9), , , They are goal achievement reward, obstacle avoidance reward and active heading reward. The specific formulas are as follows:

[0056] (11)

[0057] In formula (11), yes The target's position at the moment, , , , is the training parameter;

[0058] (12)

[0059] In formula (12), yes The position of the obstacle at any moment, , , , is the training parameter, is the angle discount factor, is the angle between the mobile robot's orientation and the line connecting the mobile robot and the nearest obstacle;

[0060] (13)

[0061] In formula (13), yes The desired orientation in the robot's local coordinate system at that moment, , is the training parameter.

[0062] Preferably, in S520, simulation training is performed according to a network strategy based on the state value and reward function of the mobile robot, specifically:

[0063] S521: Actor selects actions according to the strategy and obtains immediate rewards;

[0064] S522: Critic updates the status value;

[0065] S523: Use the immediate reward and state value to calculate the advantage function and evaluate the pros and cons of the current action, specifically:

[0066] (14)

[0067] in, is the advantage function, For instant rewards, is the discount factor, is the state value of the next state, is the state value of the current state;

[0068] S524: Update the Actor using the PPO clipping loss function; wherein the loss function calculation formula is:

[0069] (15)

[0070] in, is the strategy probability ratio, is a hyperparameter that controls the update step.

[0071] The present invention preprocesses environmental data, extracts features from lidar and depth camera data, generates a feature map of the mobile robot's observation space, extracts key features and reduces noise, thereby improving the effectiveness of the navigation strategy. At the same time, sub-target points are used to simplify path planning and enhance adaptability to dynamic environments. Deep reinforcement learning technology is used to further enhance the model's ability to recognize and represent key features, promote the model's adaptability and generalization ability in complex dynamic environments, and design a multi-objective reward function to guide the robot's autonomous learning, thereby improving the robustness, rationality and generalization ability of its navigation strategy, enabling the robot to make more efficient and safe decisions in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1This is a flow chart of a method for autonomous navigation of a robot in a dynamic environment based on deep reinforcement learning in one embodiment of the present invention;

[0073] Figure 2 This is a diagram of a PPO-based deep reinforcement learning network structure in one embodiment of the present invention;

[0074] Figure 3 This is a bottleneck residual block structure diagram based on introducing efficient channel attention in one embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0076] With the growing demand for industrial automation and intelligent service robots, mobile robots play an important role in many fields, such as logistics and distribution, environmental monitoring, cleaning services, and personal assistance.

[0077] Navigation is a basic task for mobile robots. In practical applications, robots need to navigate safely and effectively in complex and dynamically changing environments, including static obstacles and dense crowds. These factors require robots to be able to perceive changes in the surrounding environment in real time, process large amounts of data, and make quick decisions. This not only places high demands on the robot's perception system and computing power, but also poses severe challenges to the robustness and adaptability of navigation algorithms.

[0078] In order to solve the above problems, the present application provides a method for autonomous navigation of a mobile robot in a complex dynamic environment based on deep reinforcement learning, so as to realize safe and efficient autonomous navigation of the mobile robot in a dynamic environment full of static obstacles and dense crowds.

[0079] In one embodiment, Figure 1 As shown, a method for autonomous navigation of a robot in a dynamic environment based on deep reinforcement learning, the method comprising the following steps:

[0080] S100: Setting the observation space and action space of the mobile robot; Setting the observation space and action space is the basis of the reinforcement learning strategy;

[0081] S200: Acquire environmental data, pre-process the environmental data, extract features from the laser radar and depth camera data, obtain radar data feature maps and pedestrian data feature maps, and then generate a mobile robot observation space feature map; further, this helps to reduce data dimension, reduce noise, and enhance feature expression, so that the network can focus on learning effective navigation strategies;

[0082] S300: Get the robot's target position, use the A* algorithm to generate a path from the current position to the global target, and generate sub-target points based on the current position and the global path; the introduction of sub-target points can simplify continuous navigation tasks in complex environments, improve the robot's ability to adapt to dynamic changes, and realize active obstacle avoidance and dynamic path planning through real-time analysis of environmental data, while maintaining a balance between forward speed and safety;

[0083] S400: construct a PPO deep reinforcement learning network, take the observation space feature map and sub-target points as the input of the PPO deep reinforcement learning network, and output the state value, forward speed and turning speed of the mobile robot;

[0084] S500: Design a reward function, establish a simulation environment, train the PPO deep reinforcement learning network based on the state value and reward function of the mobile robot, and obtain a trained PPO deep reinforcement learning network after training for a set total number of time steps. Based on the trained PPO deep reinforcement learning network, obtain the real-time forward speed and turning speed of the mobile robot to achieve real-time autonomous navigation in complex dynamic environments.

[0085] In one embodiment, S100 includes:

[0086] S110: The observation space of the mobile robot includes the laser radar historical data, the kinematic information of pedestrians within a preset range centered on the mobile robot, and the position of the sub-target point, specifically:

[0087] (1)

[0088] in, Represents the historical data of LiDAR. represents the kinematic information of pedestrians within a preset range centered on the mobile robot. Indicates the location of the sub-target point; Indicates the moment;

[0089] S120: The motion space of the mobile robot includes the robot's forward speed and the robot's turning speed, specifically:

[0090] (2)

[0091] in, represents the forward speed of the mobile robot, represents the turning angular velocity of the mobile robot, represents the time, where m / s, rad / s.

[0092] In one embodiment, S200 includes:

[0093] S210: Use a laser radar sensor to obtain laser scanning data of the surrounding environment in real time, perform minimum and average pooling on the laser radar data of the past ten scans, and generate historical distance measurement data, that is, a (1,80,80)-dimensional radar data feature map;

[0094] S220: Use a depth camera and the YOLO target detection algorithm to detect pedestrians in the environment and record their relative positions. Use a multi-hypothesis tracking algorithm to track the movement of pedestrians, obtain their position and speed information, map the position and speed information of pedestrians into a grid map, and generate a (2,80,80)-dimensional pedestrian kinematic feature map.

[0095] S230: The radar data feature map and the pedestrian kinematic feature map together constitute an observation space feature map of the mobile robot.

[0096] In one embodiment, S300 includes:

[0097] S310: Determine the final destination that the robot needs to reach, and use the A* algorithm to generate a path from the current position to the global goal;

[0098] S320: Calculate the path point closest to the robot on the path;

[0099] S330: Determine a sub-target point according to the current position and the nearest point;

[0100] S340: Convert the global target point to a target point in the robot local coordinate system.

[0101] In one embodiment, S320 specifically includes:

[0102] S321: Define the path and current position, assuming the path consists of a sequence of line segments , ,…, Composition, of which is a point on the path, and the current robot position is ;

[0103] S322: Traverse each segment on the path, and for the first Segment, its starting point ,end ;

[0104] S323: Calculate the direction vector, the unit direction vector, and the vector from the starting point of the line segment to the robot position. The specific formula is as follows:

[0105] (3)

[0106] In formula (3), is the direction vector of the line segment;

[0107] (4)

[0108] In formula (4), is the unit direction vector of the line segment;

[0109] (5)

[0110] In formula (5), is the vector from the starting point of the line segment to the robot position;

[0111] S324: Calculate the projection distance based on the vector from the starting point of the line segment to the robot position and the direction vector of the line segment , the specific formula is as follows:

[0112] (6)

[0113] S325: Determine the nearest point based on the projection distance and the direction vector of the line segment. The specific formula is as follows:

[0114] (7)

[0115] S326: Calculation based on projection distance and closest point The vertical distance to the line segment is as follows:

[0116] (8)

[0117] S327: Repeat S323 to S326 for all line segments, and record the minimum vertical distance and the corresponding nearest point.

[0118] In one embodiment, S330 specifically includes:

[0119] (9)

[0120] In formula (9), is the set look-ahead distance, where .

[0121] In one embodiment, the PPO deep reinforcement learning network in S400 includes a feature extraction network, a fully connected layer, an Actor network, and a Critic network; the feature extraction network processes the input observation space feature map through a series of convolutional layers, pooling layers, and three embedded efficient attention mechanism bottleneck residual blocks to extract the intermediate feature map; wherein, the output of the first pooling layer of the feature extraction network is feature concatenated with the output of the third bottleneck residual block and passed to the second pooling layer, and the output of the first bottleneck residual block of the feature extraction network is also feature concatenated with the output of the second bottleneck residual block, and the result is input to the third bottleneck residual block; each bottleneck residual block consists of six convolutional layers and two channels The attention module is divided into two groups. Each group contains three consecutive convolutional layers followed by a channel attention module. The input of the first convolutional layer in each group is concatenated with the output of the channel attention module to form the input of the next group. The fully connected layer receives the intermediate feature map and sub-target point information output by the feature extraction network, and further processes it to generate a high-level feature map. The Actor network consists of two fully connected layers, which takes the high-level feature map as input, and outputs the turning speed and forward speed of the mobile robot. The Critic network also contains two fully connected layers, which takes the same high-level feature map as input, outputs the value function of the mobile robot's action, and provides feedback to the Actor network to optimize the strategy. Further, the structural diagram of the three embedded efficient attention mechanism bottleneck residual blocks is shown in the figure below. Figure 3 shown.

[0122] In one embodiment, S500 includes:

[0123] S510: Design multi-goal rewards, including goal achievement rewards, obstacle avoidance rewards, path smoothness rewards, and active heading rewards; further, goal achievement rewards encourage the robot to move towards the goal; obstacle avoidance rewards enable the robot to avoid collisions with obstacles; path smoothness rewards make the robot's movements smoother and avoid sudden turns; active heading rewards are based on the speed barrier theory and guide the robot to actively avoid pedestrians and move towards the goal, thereby achieving a better balance between obstacle avoidance and speed;

[0124] S520: Establish a simulation environment, which is the Gazebo platform in the Noetic version of ROS. Use the pedsim_ros model to simulate complex dynamic environment scenes. During the training process, randomly set target points. Perform simulation training and testing according to the network strategy based on the state value and reward function of the mobile robot.

[0125] S530: After training for a set total number of time steps, a trained PPO deep reinforcement learning network is obtained. Based on the trained PPO deep reinforcement learning network, the real-time forward speed and turning speed of the mobile robot are obtained to achieve real-time autonomous navigation in a complex dynamic environment.

[0126] In one embodiment, the multi-target reward in S510 is specifically:

[0127] (10)

[0128] In formula (9), , , They are goal achievement reward, obstacle avoidance reward and active heading reward. The specific formulas are as follows:

[0129] (11)

[0130] In formula (11), yes The target's position at the moment, , , , is the training parameter;

[0131] (12)

[0132] In formula (12), yes The position of the obstacle at any moment, , , , is the training parameter, is the angle discount factor, is the angle between the mobile robot's orientation and the line connecting the mobile robot and the nearest obstacle;

[0133] (13)

[0134] In formula (13), yes The desired orientation in the robot's local coordinate system at that moment, , is the training parameter.

[0135] In one embodiment, in S520, simulation training is performed according to the network strategy based on the state value and reward function of the mobile robot, specifically:

[0136] S521: Actor selects actions according to the strategy and obtains immediate rewards;

[0137] S522: Critic updates the status value;

[0138] S523: Use the immediate reward and state value to calculate the advantage function and evaluate the pros and cons of the current action, specifically:

[0139] (14)

[0140] in, is the advantage function, For instant rewards, is the discount factor, is the state value of the next state, is the state value of the current state;

[0141] S524: Update the Actor using the PPO clipping loss function; wherein the loss function calculation formula is:

[0142] (15)

[0143] in, is the strategy probability ratio, is a hyperparameter that controls the update step.

[0144] Beneficial effects of the present invention:

[0145] 1. The present invention generates a feature map of the mobile robot's observation space by preprocessing pedestrian and radar data, extracts key features of environmental information, reduces noise, and improves the effectiveness of navigation strategies. At the same time, sub-target points are used to simplify path planning and enhance adaptability to dynamic environments.

[0146] 2. The present invention adopts deep reinforcement learning technology to guide the mobile robot to learn autonomously by designing a multi-objective reward function, thereby improving the robustness, rationality and generalization ability of the robot's navigation strategy, and enabling it to make more efficient and safe decisions in complex environments. The feature extraction network introduces an efficient channel attention mechanism, which dynamically emphasizes important channels and suppresses unimportant channels, thereby improving the model's ability to recognize and represent key features, and promoting the model's adaptability and generalization ability in complex dynamic environments.

[0147] In one embodiment, a robot dynamic environment autonomous navigation system based on deep reinforcement learning includes an observation space and action space setting module, an observation space feature map generation module, a sub-target point generation module, a PPO deep reinforcement learning network construction module, and a PPO deep reinforcement learning network training and autonomous navigation module;

[0148] The observation space and action space setting module is used to set the observation space and action space of the mobile robot;

[0149] The observation space feature map generation module is used to obtain environmental data, pre-process the environmental data, extract features from the laser radar and depth camera data, obtain radar data feature maps and pedestrian data feature maps, and then generate the mobile robot observation space feature map;

[0150] The sub-target point generation module is used to obtain the target position of the robot, use the A* algorithm to generate a path from the current position to the global target, and generate sub-target points based on the current position and the global path;

[0151] The PPO deep reinforcement learning network construction module is used to construct the PPO deep reinforcement learning network. The observation space feature map and sub-target points are used as the input of the PPO deep reinforcement learning network, and the state value, forward speed and turning speed of the mobile robot are obtained as output;

[0152] The PPO deep reinforcement learning network training and autonomous navigation module is used to design the reward function, establish the simulation environment, and train the PPO deep reinforcement learning network based on the state value and reward function of the mobile robot. After training with a set total number of time steps, a trained PPO deep reinforcement learning network is obtained. Based on the trained PPO deep reinforcement learning network, the real-time forward speed and turning speed of the mobile robot are obtained, realizing real-time autonomous navigation in complex dynamic environments.

[0153] For the specific definition of a robot dynamic environment autonomous navigation system based on deep reinforcement learning, please refer to the definition of a robot dynamic environment autonomous navigation method based on deep reinforcement learning above, which will not be repeated here. Each module in the above-mentioned robot dynamic environment autonomous navigation system based on deep reinforcement learning can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0154] The above is a detailed introduction to a method for autonomous navigation of a robot in a dynamic environment based on deep reinforcement learning provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A robot autonomous navigation method in a dynamic environment based on deep reinforcement learning, characterized in that: The method comprises the following steps: S100: Setting the observation space and action space of the mobile robot; S200: Acquire environmental data, pre-process the environmental data, extract features from the laser radar and depth camera data, obtain a radar data feature map and a pedestrian data feature map, and then generate a mobile robot observation space feature map; S300: Obtain the target position of the robot, use the A* algorithm to generate a path from the current position to the global target, and generate sub-target points according to the current position and the global path; S400: construct a PPO deep reinforcement learning network, take the observation space feature map and sub-target points as the input of the PPO deep reinforcement learning network, and output the state value, forward speed and turning speed of the mobile robot; wherein, the PPO deep reinforcement learning network includes a feature extraction network, a fully connected layer, an Actor network and a Critic network; the feature extraction network processes the input observation space feature map, and extracts the intermediate feature map through a series of convolutional layers, pooling layers and three embedded efficient attention mechanism bottleneck residual blocks; wherein, the output of the first pooling layer of the feature extraction network is feature spliced ​​with the output of the third bottleneck residual block, and is passed to the second pooling layer, and the output of the first bottleneck residual block of the feature extraction network is also feature spliced ​​with the output of the second bottleneck residual block, and the result is input to the third bottleneck residual block; S500: Design a reward function, establish a simulation environment, train the PPO deep reinforcement learning network based on the state value and reward function of the mobile robot, and obtain a trained PPO deep reinforcement learning network after training for a set total number of time steps. Based on the trained PPO deep reinforcement learning network, obtain the real-time forward speed and turning speed of the mobile robot to achieve real-time autonomous navigation in complex dynamic environments.

2. The method according to claim 1, characterized in that S100 includes: S110: The observation space of the mobile robot includes the laser radar historical data, the kinematic information of pedestrians within a preset range centered on the mobile robot, and the position of the sub-target point, specifically: (1) in, Represents the historical data of LiDAR. represents the kinematic information of pedestrians within a preset range centered on the mobile robot. Indicates the location of the sub-target point; Indicates the moment; S120: The motion space of the mobile robot includes the robot forward speed and the robot turning speed, specifically: (2) in, represents the forward speed of the mobile robot, represents the turning angular velocity of the mobile robot, represents the time, where m / s, rad / s.

3. The method according to claim 2, characterized in that S200 includes: S210: Use a laser radar sensor to obtain laser scanning data of the surrounding environment in real time, perform minimum and average pooling on the laser radar data of the past ten scans, and generate historical distance measurement data, that is, a (1,80,80)-dimensional radar data feature map; S220: Use a depth camera and the YOLO target detection algorithm to detect pedestrians in the environment and record their relative positions. Use a multi-hypothesis tracking algorithm to track the movement of pedestrians, obtain their position and speed information, map the position and speed information of pedestrians into a grid map, and generate a (2,80,80)-dimensional pedestrian kinematic feature map. S230: The radar data feature map and the pedestrian kinematic feature map together constitute an observation space feature map of the mobile robot.

4. The method according to claim 3, characterized in that S300 includes: S310: Determine the final destination that the robot needs to reach, and use the A* algorithm to generate a path from the current position to the global goal; S320: Calculate the path point closest to the robot on the path; S330: Determine a sub-target point according to the current position and the nearest point; S340: Convert the global target point to a target point in the robot local coordinate system.

5. The method according to claim 4, characterized in that S320 is specifically: S321: Define the path and current position, assuming the path consists of a sequence of line segments , ,…, Composition, of which is a point on the path, and the current robot position is ; S322: Traverse each segment on the path, and for the first Segment, its starting point ,end ; S323: Calculate the direction vector, the unit direction vector, and the vector from the starting point of the line segment to the robot position. The specific formula is as follows: (3) In formula (3), is the direction vector of the line segment; (4) In formula (4), is the unit direction vector of the line segment; (5) In formula (5), is the vector from the starting point of the line segment to the robot position; S324: Calculate the projection distance based on the vector from the starting point of the line segment to the robot position and the direction vector of the line segment , the specific formula is as follows: (6) S325: Determine the nearest point based on the projection distance and the direction vector of the line segment. The specific formula is as follows: (7) S326: Calculation based on projection distance and closest point The vertical distance to the line segment is as follows: (8) S327: Repeat S323 to S326 for all line segments, and record the minimum vertical distance and the corresponding nearest point.

6. The method according to claim 5, characterized in that S330 is specifically: (9) In formula (9), is the set look-ahead distance, where .

7. The method according to claim 6, characterized in that Each bottleneck residual block in S400 consists of six convolutional layers and two channel attention modules, which are divided into two groups. Each group contains three consecutive convolutional layers followed by a channel attention module, where the input of the first convolutional layer in each group is concatenated with the output of the channel attention module to form the input of the next group; the fully connected layer receives the intermediate feature map and sub-target point information output by the feature extraction network, and further processes it to produce a high-level feature map; the Actor network consists of two fully connected layers, which takes the high-level feature map as input, and outputs the turning speed and forward speed of the mobile robot; the Critic network also contains two fully connected layers, which takes the same high-level feature map as input, outputs the value function of the mobile robot's action, and provides feedback to the Actor network to optimize the strategy.

8. The method according to claim 7, characterized in that S500 includes: S510: Design multi-goal rewards, including goal achievement rewards, obstacle avoidance rewards, path smoothness rewards, and active heading rewards; S520: Establish a simulation environment, which is the Gazebo platform in the Noetic version of ROS. Use the pedsim_ros model to simulate complex dynamic environment scenes. During the training process, randomly set target points. Perform simulation training and testing according to the network strategy based on the state value and reward function of the mobile robot. S530: After training for a set total number of time steps, a trained PPO deep reinforcement learning network is obtained. Based on the trained PPO deep reinforcement learning network, the real-time forward speed and turning speed of the mobile robot are obtained to achieve real-time autonomous navigation in a complex dynamic environment.

9. The method according to claim 8, characterized in that The specific multi-target rewards in S510 are: (10) In formula (10), , , They are goal achievement reward, obstacle avoidance reward and active heading reward. The specific formulas are as follows: （11） In formula (11), yes The target's position at the moment, , , , is the training parameter; （12） In formula (12), yes The position of the obstacle at any moment, , , , is the training parameter, is the angle discount factor, is the angle between the mobile robot's orientation and the line connecting the mobile robot and the nearest obstacle; （13） In formula (13), yes The desired orientation in the robot's local coordinate system at that moment, , is the training parameter.

10. The method according to claim 9, characterized in that In S520, simulation training is performed according to the network strategy based on the state value and reward function of the mobile robot, specifically: S521: Actor selects actions according to the strategy and obtains immediate rewards; S522: Critic updates the status value; S523: Use the immediate reward and state value to calculate the advantage function and evaluate the pros and cons of the current action, specifically: （14） in, is the advantage function, For instant rewards, is the discount factor, is the state value of the next state, is the state value of the current state; S524: Update the Actor using the PPO clipping loss function; wherein the loss function calculation formula is: （15） in, is the strategy probability ratio, is a hyperparameter that controls the update step.

Images