Deep reinforcement learning path planning method and device for complex indoor environments

Abstract

The present invention discloses a deep reinforcement learning path planning method and device for complex indoor environments, belonging to the technical fields of artificial intelligence, deep learning and path planning. The method includes: addressing the problem that robots have difficulty in fully perceiving the environment in complex dynamic environments, using a deep convolutional neural network to extract feature information, thereby enhancing their ability to perceive the environment; addressing the poor adaptability of traditional path planning in the face of dynamic obstacles, designing a reinforcement learning path planning method based on a proximal strategy optimization algorithm, and designing a heuristic reward function based on speed obstacles, aiming to guide the robot to find a path that can dynamically avoid obstacles and quickly advance toward the target while maintaining the target direction. Therefore, the present invention enables the robot to have efficient planning and decision-making capabilities in complex scenarios, so as to achieve autonomous, safe and efficient navigation to the target point.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, deep learning, and path planning technology, and in particular to a deep reinforcement learning-based path planning method and device for complex indoor environments. The entire path planning method and device are intended to enable a robot to navigate autonomously in complex indoor environments. Background Art

[0002] With the rapid development of technology, robots have been widely used in many fields, especially in indoor scenarios such as homes, offices, and shopping malls. In these complex indoor environments, path planning has become a core challenge for robots to achieve efficient and safe autonomous navigation.

[0003] Traditional path planning algorithms are primarily based on graph search, such as the A* algorithm and the Dijkstra algorithm. These algorithms perform reasonably well in simple environments with static obstacles. However, when faced with dynamic obstacles such as constantly moving pedestrians and suddenly appearing transport equipment, the limitations of these traditional algorithms become apparent. On the one hand, they struggle to respond to the rapid changes in obstacle positions and states in real time. On the other hand, most traditional algorithms are based on pre-defined rules and models, lacking the ability to adaptively learn in complex environments and struggling to dynamically adjust their strategies to suit different scenarios.

[0004] At the same time, sensor technology has made significant progress in recent years. Radar and camera sensors, for example, can provide robots with a wealth of environmental perception data. However, how to fully and efficiently utilize this massive amount of perception data to assist in path planning has become a pressing issue. The rise of deep reinforcement learning technology has brought new opportunities for robot path planning. It enables robots to repeatedly explore and learn in simulated environments, continuously optimizing their behavioral strategies based on the reward feedback they receive, thereby enabling them to make autonomous decisions in complex and dynamic environments. Compared with traditional methods, deep reinforcement learning can better adapt to environmental uncertainty, dynamically adjusting path planning to avoid obstacles and efficiently approach the target. Developing a method and device that fully utilizes advanced sensor technology, combined with the advantages of deep reinforcement learning, to effectively solve the problem of robot path planning in complex indoor environments has extremely high practical significance and value. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a deep reinforcement learning-based path planning method and device for complex indoor environments. Addressing the poor adaptability of traditional algorithms to dynamic obstacles, deep reinforcement learning is introduced to enable robots to perceive and avoid obstacles in real time, eliminating the need for frequent path replanning, reducing computational burdens, and ensuring smooth and safe operation in dynamic scenarios. This method enables robots to autonomously and efficiently navigate to their destinations in complex indoor environments.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a deep reinforcement learning-based path planning method for complex indoor environments, comprising the following steps:

[0007] S1. Obtain environmental perception data through radar and camera sensors, estimate and identify the robot's position in the coordinate system, the spatial distribution of static obstacles, and the kinematic parameters of surrounding pedestrians, and generate a LiDAR information data map and a pedestrian position data map through modal processing.

[0008] S2. Use a deep convolutional neural network to extract multimodal features from the lidar information data graph and the pedestrian posture data graph to generate feature information including environmental structure features, dynamic pedestrian motion features, and feasible area constraints;

[0009] S3. Build a deep reinforcement learning training environment model and initialize the network of the deep reinforcement learning training environment model based on the proximal policy optimization PPO algorithm;

[0010] S4. The extracted feature information is used as the input of the observation state of the reinforcement learning part, and the optimal control strategy is trained based on the deep reinforcement learning proximal policy optimization algorithm using a multi-objective reward function;

[0011] S5. Through the learned optimal control strategy, the optimal action of the robot's current observation state is obtained, and dynamic obstacle avoidance and navigation tasks are completed without stopping, so as to achieve path planning and autonomous navigation in complex indoor environments containing static and multiple dynamic obstacles.

[0012] As a preferred technical solution of the present invention, the specific steps of sensor data extraction and processing in step S1 are as follows:

[0013] S1.1. LiDAR data is processed using multi-resolution rasterization to convert the original point cloud data into a two-dimensional occupancy grid map, i.e., the LiDAR information data map;

[0014] S1.2. Convert the dynamic pedestrian motion parameters extracted by the depth camera into a two-dimensional position and velocity feature map in a local Cartesian coordinate system with the robot itself as the origin, that is, the pedestrian's posture data map.

[0015] As a preferred technical solution of the present invention, the specific steps of feature extraction of local observation information in step S2 are as follows:

[0016] S2.1. The lidar information data graph and the pedestrian posture data graph are input into the convolutional neural network to further extract environmental features and obtain the current partial observation results. .

[0017] As a preferred technical solution of the present invention, the specific steps of initializing the network of the deep reinforcement learning training environment model based on the proximal policy optimization algorithm in step S3 are as follows:

[0018] S3.1. The robot planning and navigation process is regarded as a partially observable Markov decision process. The POMDP model can be represented as a seven-tuple ,in Represents the state space, which contains all possible environmental states . Represents the observation space, which contains all possible observations

[0019] . Represents the action space, which contains all possible actions . State transition probability, from the observed state Execute an action Then transfer to observation state probability. Represents the reward function, that is, in the observed state Next action and transfer to the observation state Instant rewards . represents the observation probability distribution. represents the discount factor used to determine the current value of future rewards. The goal of the deep reinforcement learning network of the present invention is to maximize the expected discounted return. The objective function is defined as , the specific formula is:

[0020]

[0021] in, Expressed as the discount factor, Indicates that in the strategy The expected value below.

[0022] S3.2. Define the observation space is the relative position and velocity of the obstacle in the robot’s local coordinate system;

[0023]

[0024] in, Represents the training steps The relative position diagram of the sub-targets at the same time. Represents the training steps Data graph of 10 scans of the lidar. Represents the training steps Pose data of crowd with dynamic obstacles.

[0025] S3.3. Define the action space The speed control signal of the robot includes the forward linear velocity and the turning angular velocity in the robot coordinate system;

[0026]

[0027] in, Represents the training steps The forward linear speed, , the unit is . Represents the training steps The turning angular velocity, , the unit is .

[0028] S3.4. Establish the Actor network of the PPO algorithm, which is a two-layer fully connected neural network. The activation function is a linear activation function. The Actor strategy network is based on the current observation state. As input, the output dimension is 2, the output action The probability distribution of strategy ,in Represents the parameters of the Actor network, which can select the best action based on the current observation state. The random sampling process satisfies the diagonal Gaussian probability distribution and is completed by the following formula:

[0029]

[0030] in, represents the probability density function, which gives the probability density function at the position The probability density at . represents a random variable The dimension of , which is the dimension of the action space. Indicates an action The sampling results. represents the mean vector of the normal distribution, It is The mean of the dimensions, which determines the expected position of the action in that dimension. It is The standard deviation of a dimension, which indicates the uncertainty or dispersion of the action in that dimension. , is the square of the Mahalanobis distance. The smaller this term is, the better. The closer to the mean, the greater the probability density is, according to the properties of the exponential function. , which is a normalization term that ensures that the integral of the probability density function over the entire space is equal to 1.

[0031] parameter Update by maximizing the expected return, the formula is:

[0032]

[0033] in, is the learning rate. Expressed as The error is the same as in the Critic network, that is, the difference between the actual reward plus the discounted value estimate of the next observed state and the value estimate of the current observed state.

[0034] S3.5. Build the Critic network of the PPO algorithm, which is also a two-layer fully connected neural network. The Critic network structure is basically the same as the overall structure of the Actor network, with an output dimension of 1, i.e., the value function

[0035] , is the parameter of the Critic network. The activation function is a linear activation function. The input of the Critic network is the observation state information extracted from the experience pool. In each navigation process, the experience pool of PPO can store up to The experience information of the step length is then taken from the experience pool The extracted sample observation state information is input into the Critic network to evaluate the value function of the current strategy. , calculate the advantage function through the value function , thereby updating the Actor network in reverse. Set to 576, Set to 64.

[0036] Advantage function

[0037] The calculation formula is:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] in, It is expressed as an advantage function. represents the discount factor, Represents the attenuation coefficient of GAE, with a value of . is the time series difference error, The calculation formula is:

[0044]

[0045] in, It’s an instant reward. is the discount factor set here to . is the value estimate of the next observed state. is the value function estimate of the current observation state.

[0046] Observation state value function Weight The update is done by gradient ascent method, and the strategy improvement formula is:

[0047]

[0048] in, is the weight of the observation state value function. is the learning rate, which controls the magnitude of the update. Expressed as error. is the observed state value function with respect to weight gradient.

[0049] S4. The extracted feature information is used as the input of the observation state of the reinforcement learning part, and the optimal control strategy is trained based on the deep reinforcement learning proximal policy optimization algorithm using a multi-objective reward function.

[0050] As a preferred technical solution of the present invention, the multi-objective reward function of the proximal policy optimization PPO algorithm defined in step 4 is implemented through the following sub-steps:

[0051] The multi-objective reward function defined in S4.1 consists of four parts: a reward for reaching the target, a reward for avoiding collisions, a reward for preventing robot oscillations, and a reward for active avoidance based on speed and steering control; namely:

[0052]

[0053] in, represents the reward for reaching the goal, represents the collision avoidance reward, Indicates the prevention of robot shock rewards, It represents active obstacle avoidance reward based on improved speed obstacle theory speed control and lateral steering control.

[0054] S4.2. Define the reward for reaching the goal for:

[0055]

[0056] in, It represents the reward coefficient for reaching the target point, which is set to 18 here; Indicated as in the training step When , the distance from the current robot position to the target point; Expressed as the path point coefficient, here it is set to 3; It is expressed as the range tolerance of reaching the target point, which is set as a circle with a radius of 0.25m; Represents the current number of training steps; The maximum number of steps to complete a navigation.

[0057] S4.3. Define collision avoidance reward for:

[0058]

[0059] in, Expressed as the collision penalty coefficient, here it is set to -15;

[0060] is the minimum non-zero distance in the LiDAR scan, representing the distance between the robot and the nearest obstacle. is the radius of the robot. is the penalty coefficient for scanning distance, which is set to -0.2 here.

[0061] S4.4. Define collision avoidance rewards for:

[0062] r o =r smooth ×|θ t-1 -θ t |

[0063] in Represented as the robot in step The azimuth at time. It is expressed as the oscillation penalty coefficient, which is set to -0.02 here.

[0064] S4.5. Define a heuristic active avoidance reward function based on speed plus steering control according to the speed barrier theory .

[0065]

[0066] in and Represents the weight coefficient, which is set here as , is 0.7. Represented as a training step The optimal heading angle expected by the robot. It is expressed as the maximum allowable deviation coefficient of the heading direction angle, which is set here as . It is expressed as the angle reward coefficient, which is set to 0.6 here. is the robot training step The optimal speed when is the maximum deviation speed, which is set to 0.1 here. Expressed as a speed bonus coefficient, here it is set to 0.8.

[0067] As the preferred technical solution of the present invention, the step S5 deep reinforcement learning algorithm trains the optimal control strategy , which further includes:

[0068] The deep reinforcement learning algorithm trains the control strategy model in the simulation environment GAZEBO under Ubuntu 20.04 system The training method starts with starting the training and first determines whether the maximum number of training steps has been reached. If it has been reached, the training environment is initialized and the initial observation results are obtained. Post-execution policy actions , and then get the current observation results , and calculate the reward. Then, it checks in sequence whether the maximum number of iterations allowed for the current round, the maximum number of collisions in the round, and the target point have been reached. If the corresponding conditions are met, the environment and update counters are reset and the next training round begins. If the target point is not reached, the action steps are repeated.

[0069] After the training is completed, the performance and effect of the trained strategy are tested in the environment. The test indicators include success rate, collision rate, average time, and average road length. Success rate: The success rate refers to the proportion of samples in which the robot can reach the target point from the starting point without collision among all test samples. Collision rate: The collision rate refers to the proportion of samples in which the robot collides with pedestrians among all test samples. Average time: The average time refers to the average time the robot takes to travel from the starting point to the end point among all test samples that successfully reach the target point. Average road length: The average road length refers to the average length of the path actually traveled by the robot among all test samples that successfully reach the target point.

[0070] The technical solution of the present invention also relates to a planning and execution device for a robot in a complex indoor environment, wherein the device includes a computer system, and the computer device includes the above-mentioned computer-readable storage medium and a deep reinforcement learning module.

[0071] Compared with the existing technology, the present invention provides a deep reinforcement learning-based path planning method and device for complex indoor environments, which has the following beneficial effects:

[0072] This invention addresses the poor adaptability of traditional algorithms to dynamic obstacles by introducing deep reinforcement learning, enabling robots to achieve real-time obstacle avoidance, eliminating the need for frequent path replanning, reducing computational burdens, and ensuring smooth and safe operation in dynamic scenarios. This method primarily relies on speed control, supplemented by steering control, aiming to guide the robot to find a path that avoids obstacles while rapidly advancing toward its target, while maintaining its direction. This approach enables robots to autonomously and efficiently navigate to their target in complex indoor environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a flow chart of a deep reinforcement learning path planning method for complex indoor environments according to the present invention;

[0074] Figure 2 This is a general model diagram of a deep reinforcement learning path planning method for complex indoor environments in the present invention;

[0075] Figure 3 This is a flow chart of the training steps of a deep reinforcement learning path planning method for complex indoor environments according to the present invention;

[0076] Figure 4 This is a model diagram of a deep reinforcement learning path planning device for complex indoor environments according to the present invention. DETAILED DESCRIPTION

[0077] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0078] Reference Figure 1 、 one A deep reinforcement learning-based path planning method for complex indoor environments includes the following steps:

[0079] S1. Obtain environmental perception data through radar and camera sensors, estimate and identify the robot's position in the coordinate system, the spatial distribution of static obstacles, and the kinematic parameters of surrounding pedestrians, and generate a LiDAR information data map and a pedestrian position data map through modal processing.

[0080] S2. Use a deep convolutional neural network to extract multimodal features from the lidar information data graph and the pedestrian posture data graph to generate feature information including environmental structure features, dynamic pedestrian motion features, and feasible area constraints;

[0081] S3. Build a deep reinforcement learning training environment model and initialize the network of the deep reinforcement learning training environment model based on the proximal policy optimization PPO algorithm;

[0082] S4. The extracted feature information is used as the input of the observation state of the reinforcement learning part, and the optimal control strategy is trained based on the deep reinforcement learning proximal policy optimization algorithm using a multi-objective reward function;

[0083] S5. Through the learned optimal control strategy, the optimal action of the robot's current observation state is obtained, and dynamic obstacle avoidance and navigation tasks are completed without stopping, so as to achieve path planning and autonomous navigation in complex indoor environments containing static and multiple dynamic obstacles.

[0084] Reference Figure 2 , build a deep reinforcement learning training environment model, and initialize the network of the deep reinforcement learning training environment model based on the proximal policy optimization algorithm, which is achieved through the following sub-steps:

[0085] The robot obtains environmental perception data through radar and camera sensors, estimates and identifies the position information of the robot itself, static obstacles and surrounding pedestrians, and converts the depth camera and radar perception data into a network grid with the same shape to represent the position and motion state of the surrounding environment and pedestrian obstacles in the local Cartesian coordinate system with the robot itself as the origin; then the lidar information data map and the pedestrian posture data map are input into the convolutional neural network to further extract environmental features and obtain the current partial observation results. .

[0086] Furthermore, considering the robot's planning and navigation process as a partially observable Markov decision process, the POMDP model can be represented as a seven-tuple ,in Represents the state space, which contains all possible environmental states . Represents the observation space, which contains all possible observations . Represents the action space, which contains all possible actions . State transition probability, from the observed state Execute an action Then transfer to observation state probability. Represents the reward function, that is, in the observed state Next action and transfer to the observation state Instant rewards . represents the observation probability distribution. represents the discount factor used to determine the current value of future rewards. The goal of the deep reinforcement learning network of the present invention is to maximize the expected discounted return. The objective function is defined as , the specific formula is:

[0087]

[0088] in, Expressed as a discount factor, here it is set to 0.98, Indicates that in the strategy The expected value below.

[0089] S3.1. Define the observation space is the relative position and velocity of the obstacle in the robot’s local coordinate system;

[0090]

[0091] in, Represents the training steps The relative position diagram of the sub-targets at the same time. Represents the training steps Data graph of 10 scans of the lidar. Represents the training steps Pose data of crowd with dynamic obstacles.

[0092] S3.2. Define the action space The speed control signal of the robot includes the forward linear velocity and the turning angular velocity in the robot coordinate system;

[0093]

[0094] in, Represents the training steps The forward linear speed, , the unit is . Represents the training steps The turning angular velocity, , the unit is .

[0095] S3.3. Establish the Actor network of the PPO algorithm, which is a two-layer fully connected neural network. The activation function is a linear activation function. The Actor strategy network is based on the current observation state. As input, the output dimension is 2, the output action The probability distribution of strategy ,in Represents the parameters of the Actor network, which can select the best action based on the current observation state. It satisfies the diagonal Gaussian probability distribution and the specific formula is as follows:

[0096]

[0097] in, represents the probability density function, which gives the probability density function at the position The probability density at . represents a random variable The dimension of , that is, the dimension of the action space, is set to 2 here. Indicates an action The sampling results. represents the mean vector of the normal distribution, It is The mean of the dimensions, which determines the expected position of the action in that dimension. It is The standard deviation of a dimension, which indicates the uncertainty or dispersion of the action in that dimension. , is the square of the Mahalanobis distance. The smaller this term is, the better. The closer to the mean, the greater the probability density is, according to the properties of the exponential function.

[0098] , which is a normalization term that ensures that the integral of the probability density function over the entire space is equal to 1.

[0099] parameter Update by maximizing the expected return, the formula is:

[0100]

[0101] in, is the learning rate. Expressed as The error is the same as in the Critic network, that is, the difference between the actual reward plus the discounted value estimate of the next observed state and the value estimate of the current observed state.

[0102] S3.4. Build the Critic network of the PPO algorithm, which is also a two-layer fully connected neural network. The Critic network structure is basically the same as the overall structure of the Actor network, with an output dimension of 1, i.e., the value function

[0103] , is the parameter of the Critic network. The activation function is a linear activation function. The input of the Critic network is the observation state information extracted from the experience pool. In each navigation process, the experience pool of PPO can store up to The experience information of the step length is then taken from the experience pool The extracted sample observation state information is input into the Critic network to evaluate the value function of the current strategy. , calculate the advantage function through the value function , thereby updating the Actor network in reverse. Set to 576, Set to 64.

[0104] Advantage function The calculation formula is:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] in, It is expressed as an advantage function. represents the discount factor, Represents the attenuation coefficient of GAE, with a value of . is the time series difference error, The calculation formula is:

[0111]

[0112] in, It’s an instant reward. is the discount factor set here to . is the value estimate of the next observed state. is the value function estimate of the current observation state.

[0113] Observation state value function Weight The update is done by gradient ascent method, and the strategy improvement formula is:

[0114]

[0115] in, is the weight of the observation state value function. is the learning rate, which controls the magnitude of the update. Expressed as error. is the observed state value function with respect to weight gradient.

[0116] For the multi-objective reward function of the proximal policy optimization PPO algorithm, it is implemented through the following sub-steps:

[0117] The multi-objective reward function defined in S4.1 consists of four parts: a reward for reaching the target, a reward for avoiding collisions, a reward for preventing robot oscillations, and a reward for active avoidance based on speed and steering control; namely:

[0118]

[0119] in, represents the reward for reaching the goal, represents the collision avoidance reward, Indicates the prevention of robot shock rewards, It represents active obstacle avoidance reward based on improved speed obstacle theory speed control and lateral steering control.

[0120] S4.2. Define the reward for reaching the goal for:

[0121]

[0122] in, It represents the reward coefficient for reaching the target point, which is set to 18 here; Indicated as the training step When , the distance from the current robot position to the target point; Expressed as the path point coefficient, here it is set to 3; It is expressed as the range tolerance of reaching the target point, which is set as a circle with a radius of 0.25m; Represents the current number of training steps; The maximum number of steps to complete a navigation.

[0123] S4.3. Define collision avoidance reward for:

[0124]

[0125] in, Expressed as the collision penalty coefficient, here it is set to -15; is the minimum non-zero distance in the LiDAR scan, representing the distance between the robot and the nearest obstacle. is the radius of the robot. is the penalty coefficient for scanning distance, which is set to -0.2 here.

[0126] S4.4. Define anti-shock rewards for:

[0127] r o =r smooth ×|θ t-1 -θ t |

[0128] in Represented as the robot in step The azimuth at time. It is expressed as the oscillation penalty coefficient, which is set to -0.02 here.

[0129] S4.5. Define an active avoidance reward function based on speed plus steering control according to the speed barrier theory .

[0130]

[0131] in and Represents the weight coefficient, which is set here as , is 0.7. Represented as a training step The optimal heading angle expected by the robot. It is expressed as the maximum allowable deviation coefficient of the heading direction angle, which is set here as . It is expressed as the angle reward coefficient, which is set to 0.6 here. is the robot training step The optimal speed when is the maximum deviation speed, which is set to 0.1 here. Expressed as a speed bonus coefficient, here it is set to 0.8.

[0132] Reference Figure 3 This paper proposes a training flowchart of a deep reinforcement learning path planning method for complex indoor environments. The simulation is carried out in the GAZEBO simulation environment under Ubuntu 20.04 system to train the control strategy model. , starting from the start of training, first determine whether the maximum number of training steps has been reached. If it has been reached, evaluate and save the best model before ending; if it has not been reached, initialize the training environment and obtain the initial observation results Post-execution policy actions , and then get the current observation results , and calculate the reward. Then, it checks in sequence whether the maximum number of iterations allowed for the current round, the maximum number of collisions in the round, and the target point have been reached. If the corresponding conditions are met, the environment and update counters are reset and the next training round begins. If the target point is not reached, the action steps are repeated.

[0133] When the reward value tends to be flat, the training ends, the model with the highest reward is extracted, and the performance and effect of the trained policy model are tested in the environment. In the configured simulation scenario, a rule-based method and three learning-based navigation methods DRL, DRL_VO and the navigation method used in the present invention are simulated, and each is randomly tested 500 times. The specific hardware configuration is 13th Gen Intel(R) Core(TM) i5-13490F 2.50 GHz with 16GBRAM. The simulation experiment is carried out on The test is conducted in a simulated indoor area, and the comparison indicators include success rate, collision rate, average driving time, and average road length. Success rate: The success rate refers to the proportion of samples in which the robot can reach the target point from the starting point without collision among all test samples. Collision rate: The collision rate refers to the proportion of samples in which the robot collides with pedestrians among all test samples. Average driving time: The average driving time refers to the average driving time of the robot from the starting point to the end point among all test samples that successfully reach the target point, and the unit is Average path length: The average path length refers to the average value of the path length actually traveled by the robot in all test samples that successfully reach the target point, and the unit is .

[0134] The specific simulation experiment results are shown in the following table. OURS indicates that the path planning method proposed in the present invention is used. The downward arrow indicates that the lower the value, the higher the model planning and navigation efficiency. The upward arrow indicates that the higher the value, the higher the model planning and navigation efficiency. For a rule-based method and three learning-based navigation methods, the following conclusions can be drawn: In complex indoor environments with different numbers of pedestrians, the methods proposed in the present invention all show excellent performance, with relatively higher success rates, lowest collision rates and shortest average road lengths in more complex environments, while maintaining a lower average driving time. This shows that the method proposed in the present invention can not only effectively avoid collisions with pedestrians in terms of path planning, but also optimize navigation paths and improve navigation efficiency. Especially when the environment is more complex, the advantages of the method proposed in the present invention are more significant, and it can better cope with the challenges of complex environments and achieve efficient and safe navigation.

[0135] Table 1 Simulation results

[0136]

[0137] Reference Figure 4 The present invention proposes a deep reinforcement learning-based path planning device for complex indoor environments. The device acquires environmental perception data through at least one sensor device, then processes this data using a deep neural network in a computer system device to generate observation state space vectors, providing the robot with an understanding and representation of the environment. Next, the trained deep reinforcement learning module outputs a control strategy based on these vectors, enabling the robot to make optimal decisions in complex environments. Finally, the chassis control actuator controls the robot's steering angular velocity and linear velocity based on the output strategy, ensuring that the robot can move safely and efficiently along the intended planned path, thereby achieving autonomous navigation in complex indoor environments.

[0138] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A deep reinforcement learning-based path planning method for complex indoor environments, characterized by: The following steps are involved: S1. Obtain environmental perception data through radar and camera sensors, estimate and identify the robot's position in the coordinate system, the spatial distribution of static obstacles, and the kinematic parameters of surrounding pedestrians, and generate a LiDAR information data map and a pedestrian position data map through modal processing. S2. Use a deep convolutional neural network to extract multimodal features from the lidar information data graph and the pedestrian posture data graph to generate feature information including environmental structure features, dynamic pedestrian motion features, and feasible area constraints; S3. Build a deep reinforcement learning training environment model and initialize the network of the deep reinforcement learning training environment model based on the proximal policy optimization PPO algorithm; S4. The extracted feature information is used as the input of the observation state of the reinforcement learning part, and the optimal control strategy is trained based on the deep reinforcement learning proximal policy optimization algorithm using a multi-objective reward function; The multi-objective reward function in the proximal policy optimization PPO algorithm in step S4 is implemented through the following sub-steps: The multi-objective reward function defined in S4.1 consists of four parts: a reward for reaching the target, a reward for avoiding collisions, a reward for preventing robot oscillations, and a reward for active avoidance based on speed and steering control; namely: r t ＝r a +r c +rx+r v_θ Among them, r a represents the reward for reaching the goal, r c represents the collision avoidance reward, r o Indicates the reward for preventing robot shock, r θ,v It represents active obstacle avoidance reward based on improved speed obstacle theory speed control and lateral steering control; S4.

2. Define the reward r for reaching the goal a for: Among them, r arrival Expressed as the reward coefficient for reaching the target point; It is expressed as the distance from the current robot position to the target point at training step t; r waypoint Expressed as path point coefficient; g r Represents the range tolerance for reaching the target point; i t Represents the current number of training steps; i m The maximum number of steps for a complete navigation; S4.

3. Define collision avoidance reward r c for: Among them, r collision Expressed as collision penalty coefficient; d min is the minimum non-zero distance in the lidar scan, representing the distance between the robot and the nearest obstacle, r robot is the radius of the robot, r s is the penalty coefficient of scanning distance; S4.

4. Define the anti-oscillation collision reward r o for: r o =r smooth ×|θ t-1 -θ t | where θ t Expressed as the robot's azimuth at step t, r smooth Expressed as the shock penalty coefficient; S4.

5. Based on the speed barrier theory, define an active avoidance reward function r based on speed plus steering control θ,v ; Where ω1 and ω2 represent weight coefficients, It is expressed as the optimal heading angle expected by the robot at training step t, θ m Expressed as the maximum allowable deviation coefficient of the heading direction angle, r θ Expressed as the angle reward coefficient, is the optimal speed of the robot at training step t, V m is the maximum deviation speed, r v Expressed as a speed bonus coefficient; S5. Through the learned optimal control strategy, the optimal action of the robot's current observation state is obtained, and dynamic obstacle avoidance and navigation tasks are completed without stopping, so as to achieve path planning and autonomous navigation in complex indoor environments containing static and multiple dynamic obstacles.

2. The path planning method according to claim 1, characterized in that: The camera sensor is a depth camera, the radar is a single-line laser radar, and S1 specifically includes: S1.

1. LiDAR data is processed using multi-resolution rasterization to convert the original point cloud data into a two-dimensional occupancy grid map, i.e., the LiDAR information data map; S1.

2. Convert the dynamic pedestrian motion parameters extracted by the depth camera into a two-dimensional position and velocity feature map in a local Cartesian coordinate system with the robot itself as the origin, that is, the pedestrian's posture data map.

3. The path planning method according to claim 1, wherein: The S2 specifically includes: S2.

1. The lidar information data graph and the pedestrian posture data graph are input into the convolutional neural network to extract environmental features and obtain the current partial observation result Z t .

4. The path planning method according to claim 1, wherein: The S3 specifically includes: S3.

1. The robot planning and navigation process is regarded as a partially observable Markov decision process. The POMDP model is used as the deep reinforcement learning training environment model. The model is represented as a seven-tuple {S, Z, A, P, R, O, γ}, where S represents the state space, which contains all possible environment states S t , Z represents the observation space, which contains all possible observations Z t , A represents the action space, which contains all possible actions a t , P state transition probability, from the observed state Z t Execute action a t Then transfer to observation state Z t+1 The probability of R represents the reward function, that is, in the observation state Z t Next, perform action a t And transfer to observation state Z t+1 The instant reward r t+1 , O represents the observation probability distribution, γ represents the discount factor used to determine the current value of future rewards; The goal of the deep reinforcement learning network is to maximize the expected discounted return, and its objective function is defined as The specific formula is: Among them, γ represents the discount coefficient, Indicates that in the strategy Expected value under S3.

2. Define the observation space Z t is the relative position and velocity of the obstacle in the robot’s local coordinate system; Z t ={g t ,l t ,h t } Among them, g t represents the relative position coordinates of the sub-target at training step t, l t Represents the data graph of 10 scans of the lidar at training step t, h t The pose data graph of the dynamic obstacle crowd at training step t, including the position of the pedestrian in the Cartesian coordinate system with the robot as the origin and the pedestrian's speed; S3.

3. Define the action space A t It is the speed control signal of the robot, including the forward linear speed and turning angular speed in the robot coordinate system; A t ={V t ,oh t } Among them, V t represents the forward linear velocity at training step t, and ω represents the turning angular velocity at training step t; S3.

4. Establish the Actor network of the PPO algorithm, which is a two-layer fully connected neural network with a linear activation function. The Actor strategy network is based on the current observation state Z t As input, the output dimension is 2, and the output action is A t The probability distribution of the forward speed V x and angular velocity w z ,Strategy in Represents the parameters of the Actor network, which can select the best action based on the current observation state; action A t The random sampling process satisfies the diagonal Gaussian probability distribution and is completed by the following formula: P(x) represents the probability density function, which gives the probability density at position X, n represents the dimension of the random variable X, that is, the dimension of the action space, and x represents an action A t The sampling result of μ represents the mean vector of the normal distribution, μ i is the mean of the i-th dimension, which determines the expected position of the action in that dimension, σ i is the standard deviation of the i-th dimension, which represents the uncertainty or dispersion of the action in this dimension, is the square of the Mahalanobis distance, is a normalization term used to ensure that the integral of the probability density function over the entire space is equal to 1; parameter Update by maximizing the expected return, the formula is: Where α is the learning rate and δ is the TD error, which is the same as in the Critic network. S3.

5. Establish the Critic network of the PPO algorithm, which is also a two-layer fully connected neural network. The Critic network structure is the same as the overall structure of the Actor network. The output dimension is 1, that is, the value function V(Z|ω), ω is the parameter of the Critic network, and the activation function is a linear activation function. The input of the Critic network is the observation state information extracted from the experience pool. During each navigation process, the experience pool of PPO will store the experience information of the previous L steps at most, and then take B samples from the experience pool, and input the extracted observation state information of these samples into the Critic network to evaluate the value function V(Z|ω) of the current strategy, and calculate the advantage function through the value function Thereby updating the Actor network in reverse; Advantage function The calculation formula is: in, It is expressed as the advantage function, γ represents the discount factor, μ represents the attenuation coefficient of GAE, is the timing difference TD error, The calculation formula is: Among them, r t is the immediate reward, γ is the discount factor, V(Z t+1 |ω) is the value estimate of the next observed state, V(Z t |ω) is the value function estimate of the current observation state; The update of the weight ω of the observation state value function V(Z|ω) is done by the gradient ascent method, and the strategy improvement formula is: Among them, ω is the weight of the observation state value function, β is the learning rate used to control the amplitude of the update, and δ is expressed as the TD error. is the gradient of the observed state value function with respect to the weight ω.

5. The path planning method according to claim 1, wherein: The deep reinforcement learning algorithm in S5 trains the optimal control strategy Specifically include: The deep reinforcement learning algorithm trains the control strategy model in the simulation environment GAZEBO under Ubuntu 20.04 system The training method starts with starting the training, first determining whether the maximum number of training steps has been reached. If so, the training environment is initialized, the initial observation result Z0 is obtained, and the strategy action is executed. Then get the current observation result Z t , and calculate the reward; then check in turn whether the maximum number of iterations allowed in the current round, the maximum number of collisions in the round, and the target point have been reached. If the corresponding conditions are met, reset the environment and update the counter and enter the next training round. If the target point is not reached, repeat the action steps.

6. A deep reinforcement learning path planning device for complex indoor environments, characterized by: The system comprises at least one sensor device for acquiring sensory data of the environment; a deep neural network for processing the sensory data and generating an observation state space; a deep reinforcement learning module for training the control strategy; and a chassis control actuator for controlling the steering angle and speed of the robot according to the trained strategy. The path planning device is used to implement the path planning method described in any one of claims 1-5.

Images