Robot path planning method fusing dynamic window and PPO algorithm

Abstract

The invention relates to the technical field of robot path planning, in particular to a robot path planning method and device fusing a dynamic window and a PPO algorithm and a computer readable storage medium, and the method comprises the steps: constructing a robot motion model based on a dynamic window method, and limiting a speed window of the robot motion model, establishing an evaluation function of the robot motion model; in a training environment, weight parameters in an evaluation function of the robot motion model are used as learnable parameters, the range of the learnable parameters is given, a near-end strategy optimization algorithm is used for learning the learnable parameters, target weight parameters are output, and a target evaluation function of the robot motion model is obtained. According to the method, the robot has a smooth autonomous learning capability when performing path planning, the adaptive capacity of the robot in an environment with dynamic obstacles is improved, and the quality of the path planned by the robot in a complex and changeable environment is also improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot path planning, and in particular to a robot path planning method, device and computer-readable storage medium integrating a dynamic window and a PPO algorithm. Background Art

[0002] In recent years, with the rapid increase in demand for mobile robots, the development of robot path planning, obstacle recognition and obstacle avoidance technology is being actively carried out. When autonomously navigating, the robot must safely reach the destination according to the set route. Path planning can be roughly divided into global path planning and local path planning. Global path planning assumes that the mobile robot has prior knowledge of the environment and generates an optimal path that does not conflict with the terrain and obstacles marked on the map; local path planning is to avoid conflicts in a dynamically changing environment. The internal sensors of the mobile robot must collect environmental map information in real time, determine the distribution of the map and obstacles, and then choose the correct action for the next step.

[0003] Dynamic Window Approach (DWA) is a local path planning method that obtains the optimal linear speed and angular velocity based on the robot's speed, direction, and the distance to obstacles obtained from sensor information, thereby avoiding obstacles and reaching the destination.

[0004] The dynamic window method shows good obstacle avoidance performance in the actual environment, shows a good avoidance rate for static and low-speed obstacles, and executes faster with a smaller amount of computation. However, the existing dynamic window method does not consider the following of the path line between the destinations. The robot may change direction and speed frequently, resulting in an increase in path length and a decrease in path quality. In addition, the performance of the dynamic window method depends on multiple parameters in the evaluation function. In different environments, these parameters usually need to be adjusted manually based on experience. It cannot adapt to dynamically changing environments and obstacles and lacks real-time performance. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem that the robot path planning method based on the dynamic window method in the prior art cannot adapt to the dynamic environment and has poor path quality.

[0006] In order to solve the above technical problems, the present invention provides a robot path planning method integrating a dynamic window and a PPO algorithm, comprising:

[0007] The robot motion model is constructed based on the dynamic window method, and the speed window of the robot motion model is limited, and the evaluation function of the robot motion model is established;

[0008] In the training environment, the weight parameters in the evaluation function of the robot motion model are used as learnable parameters. Given the range of the learnable parameters, the learnable parameters are learned using the proximal strategy optimization algorithm, and the target weight parameters are output to obtain the target evaluation function of the robot motion model.

[0009] In each time step, the robot motion model samples several speed groups in the speed window, calculates the target evaluation function value of each speed group, and selects the speed group with the largest target evaluation function value as the robot's movement strategy.

[0010] Preferably, the robot motion model is constructed based on the dynamic window method, including:

[0011] Define the current position of the robot as (x, y), the angle between the robot and the x-axis as θ, and the angular velocity and linear velocity as ω and v respectively;

[0012] If the angular velocity is zero, after a time step of Δt, the robot's position is updated to (x', y') = (x + vΔtcosθ, y + vΔtsinθ);

[0013] If the angular velocity is not zero, after a time step Δt, the robot's position is updated to:

[0014]

[0015] Where (x', y') is the position of the robot after time step Δt, is the angle between the robot and the x-axis after the robot time step Δt.

[0016] Preferably, the speed window of the robot motion model is limited according to the robot's speed range, maximum acceleration, maximum deceleration and minimum distance between the robot and an obstacle.

[0017] Preferably, the speed window of the robot motion model is limited according to the speed range of the robot, and the formula is expressed as:

[0018] V m ={v∈[v min ,v max ],ω∈[ω min ,ω max ]}

[0019] Among them, V m is the first velocity window, v min and v max are the minimum and maximum linear speeds of the robot, ω min and ω max are the minimum angular velocity and maximum angular velocity of the robot respectively;

[0020] The speed window of the robot motion model is limited according to the maximum acceleration and maximum deceleration. The formula is expressed as:

[0021]

[0022] Among them, V d is the second speed window, v c and ω c are the current linear velocity and angular velocity of the robot, respectively, b and ω b are the maximum deceleration of the robot’s linear velocity and angular velocity, v a and ω a are the maximum acceleration of the robot’s linear velocity and angular velocity, respectively;

[0023] The speed window of the robot motion model is limited according to the minimum distance between the robot and the obstacle. The formula is expressed as:

[0024]

[0025] Among them, V a is the third speed window, dist min (v,ω) represents the minimum distance between the robot and the obstacle;

[0026] The target speed window is determined based on the first speed window, the second speed window and the third speed window:

[0027] V=V m ∩V d ∩V a

[0028] Where V is the target speed window.

[0029] Preferably, an evaluation function of the robot motion model is established, and the formula is:

[0030] G0(v,ω)=α0·heading(v,ω)+β0·dist(v,ω)+γ0·velocity(v,ω)

[0031] Among them, G0(v,ω) is the evaluation function of the robot motion model, heading(v,ω) represents the angle evaluation function, which is the angle difference between the robot's heading and the target when it reaches the predicted position, dist(v,ω) represents the distance evaluation function, velocity(v,ω) represents the speed evaluation function, and α0, β0 and γ0 are all weight parameters.

[0032] Preferably, a path evaluation function is introduced to improve the evaluation function of the robot motion model, and the improved evaluation function is:

[0033] G1(v,ω)=α1·heading(v,ω)+k·β1·dist(v,ω)+γ1·velocity(v,ω)+(1-k)·δ1·d line (v,ω)

[0034] Among them, G1(v,ω) is the first improved evaluation function, d line (v,ω) represents the path evaluation function; k represents the obstacle detection coefficient; α1, β1, γ1 and δ1 are all weight parameters.

[0035] Preferably, the obstacle detection coefficient is changed to a collision avoidance coefficient, and the evaluation function of the robot motion model is further optimized as follows:

[0036] G p (v,ω)=α p ·heading(v,ω)+λ·β p ·dist(v,ω)+γ p velocity(v,ω)+|1-λ|·δ p ·d line (v,ω)

[0037] Among them, G p (v,ω) is the second improved evaluation function, λ is the collision avoidance coefficient, α p , β p , γ p and δ p They are all weight parameters;

[0038] The collision avoidance coefficient is defined as:

[0039]

[0040] Among them, λ is the collision avoidance coefficient, (x',y') is the position of the robot at the next time node, (x obs ,y obs ) is the position of the obstacle, d collide Denotes the collision distance, d avoid Indicates a safe distance.

[0041] Preferably, the reward of the proximal policy optimization algorithm is set to:

[0042] r t =r g t +r c t +r m t

[0043] Among them, r t represents the total reward of the robot at time t, rg t represents the robot's positive reward at time t, r c t represents the robot's penalty at time t, r m t represents the step size penalty of the robot at time t.

[0044] The present invention also provides a robot path planning device integrating a dynamic window and a PPO algorithm, comprising:

[0045] A motion model building module is used to build a robot motion model based on a dynamic window method, limit the speed window of the robot motion model, and establish an evaluation function of the robot motion model;

[0046] A parameter training module is used to use the weight parameters in the evaluation function of the robot motion model as learnable parameters in the training environment, give the range of the learnable parameters, use the proximal strategy optimization algorithm to learn the learnable parameters, output the target weight parameters, and obtain the target evaluation function of the robot motion model;

[0047] The path planning module is used to sample several speed groups in the speed window of the robot motion model in each time step, calculate the target evaluation function value of each speed group, and select the speed group with the largest target evaluation function value as the robot's movement strategy.

[0048] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the robot path planning method that integrates the dynamic window and the PPO algorithm are implemented.

[0049] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0050] The present invention discloses a robot path planning method integrating dynamic window and PPO algorithm, constructs a robot motion model based on the dynamic window method, establishes an evaluation function of the robot motion model, uses weight parameters in the evaluation function as learnable parameters, and uses a proximal strategy optimization algorithm to learn the learnable parameters, so that the robot has smooth autonomous learning ability when performing path planning, improves the robot's adaptability in an environment with dynamic obstacles, and also improves the quality of the path planned by the robot in a complex and changeable environment.

[0051] Furthermore, the present invention optimizes the evaluation function of the robot motion model, adds the evaluation of the path on the basis of the standard evaluation function, so that the robot can still follow the path after avoiding obstacles. And introduces the collision avoidance coefficient into the evaluation function, which improves the robot's obstacle avoidance in a complex environment with multiple dynamic obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 It is a flow chart of a robot path planning method integrating dynamic window and PPO algorithm of the present invention;

[0054] Figure 2 It is a schematic diagram of path tracking of the dynamic window method;

[0055] Figure 3 It is the basic framework diagram of the proximal strategy optimization algorithm;

[0056] Figure 4 is a schematic diagram of an obstacle in Embodiment 1;

[0057] Figure 5 This is the path planning result diagram under simple obstacles in the simulation experiment, where: Figure 5 a in the figure is the path planning result of the dynamic window method in a static obstacle environment. Figure 5 b in the figure is the path planning result graph of the traditional dynamic window method. Figure 5 c in the figure is the path planning result diagram of the improved dynamic window method with limited maximum speed and acceleration. Figure 5 d in is the path planning result graph of the fusion algorithm;

[0058] Figure 6 This is a schematic diagram of the path planning process under complex obstacles in the simulation experiment, where: Figure 6 (a) is a schematic diagram of the first stage path planning. Figure 6 (b) is a schematic diagram of the second stage path planning. Figure 6 (c) is a schematic diagram of the path planning in the third stage. Figure 6 (d) is a schematic diagram of the path planning in the fourth stage;

[0059] Figure 7 This is the path planning result diagram under complex obstacles in the simulation experiment, where: Figure 7 Column (a) in the figure is the path planning result of the dynamic window method. Figure 7 Column (b) in is the path planning result diagram of the dynamic window method using the first improved evaluation function. Figure 7 Column (c) in is the path planning result graph of the fusion algorithm;

[0060] Figure 8 It is a simulation diagram of the physical map of the second embodiment;

[0061] Fig. 9This is a diagram of the path planning results of the physical experiment of Example 2. DETAILED DESCRIPTION

[0062] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0063] Embodiment 1

[0064] Reference Figure 1 As shown, the present invention provides a robot path planning method integrating a dynamic window and a PPO algorithm, comprising:

[0065] The robot motion model is constructed based on the dynamic window method, and the speed window of the robot motion model is limited, and the evaluation function of the robot motion model is established;

[0066] In the training environment, the weight parameters in the evaluation function of the robot motion model are used as learnable parameters. Given the range of the learnable parameters, the proximal policy optimization algorithm (PPO) is used to learn the learnable parameters, and the target weight parameters are output to obtain the target evaluation function of the robot motion model.

[0067] In each time step, the robot motion model samples several speed groups in the speed window, calculates the target evaluation function value of each speed group, and selects the speed group with the largest target evaluation function value as the robot's movement strategy.

[0068] The robot motion model is constructed based on the dynamic window method, including:

[0069] Define the current position of the robot as (x, y), the angle between the robot and the x-axis as θ, the angular velocity and linear velocity as ω and v respectively. Set the robot to not move in all directions, but only forward and turn. In the adjacent time, the robot's trajectory is an arc, and the radius of the arc is

[0070] If the angular velocity is zero, after a time step Δt, the robot's position is updated to (x', y') = (x + vΔtcosθ, y + vΔtsinθ).

[0071] If the angular velocity is not zero, after a time step Δt, the robot's position is updated to:

[0072]

[0073] Where (x', y') is the position of the robot after time step Δt, is the angle between the robot and the x-axis after the robot time step Δt.

[0074] Since the robot's speed (v, ω) is theoretically infinite, it is necessary to limit the speed window of the robot's motion model based on the robot's speed range, maximum acceleration, maximum deceleration, and the minimum distance between the robot and the obstacle, including:

[0075] The speed window of the robot motion model is limited according to the robot's speed range. The formula is expressed as:

[0076] V m ={v∈[v min ,v max ],ω∈[ω min ,ω max ]}

[0077] Among them, V m is the first velocity window, v min and v max are the minimum and maximum linear speeds of the robot, ω min and ω max are the minimum and maximum angular velocities of the robot respectively.

[0078] Since the torque of the motor is limited, the performance of the robot will be affected by the motor drive module, and there are limits on acceleration and deceleration. The actual speed that the robot can achieve is expressed as:

[0079]

[0080] Among them, V d is the second speed window, v c and ω c are the current linear velocity and angular velocity of the robot, respectively, d and ω b are the maximum deceleration of the robot’s linear velocity and angular velocity, v a and ω a are the maximum acceleration of the robot’s linear velocity and angular velocity respectively.

[0081] For safety reasons, in order to ensure that the robot can stop before encountering an obstacle, the speed should be within a range with maximum deceleration conditions, expressed as:

[0082]

[0083] Among them, V a is the third speed window, dist min (v,ω) represents the minimum distance between the robot and the obstacle.

[0084] The target speed window is determined based on the first speed window, the second speed window and the third speed window:

[0085] V=V m ∩V d ∩V a

[0086] Where V is the target speed window.

[0087] The dynamic window method samples several speed groups, that is, several feasible trajectories, from the speed window in each time step, and then evaluates each trajectory according to the evaluation function to obtain the optimal value of the robot in the speed vector space at the next time point.

[0088] The standard evaluation function of the dynamic window method is:

[0089] G0(v,ω)=α0·heading(v,ω)+β0·dist(v,ω)+γ0·velocity(v,ω)

[0090] Among them, G0(v,ω) is the standard evaluation function, heading(v,ω) represents the angle evaluation function, which is used to evaluate the angle difference between the robot's heading and the target when it reaches the predicted position under the currently set sampling speed, dist(v,ω) represents the distance evaluation function, which is used to evaluate the distance between the robot and the obstacle, velocity(v,ω) represents the speed evaluation function, which is used to evaluate the speed of the current trajectory, and α0, β0 and γ0 are all weight parameters.

[0091] The present invention improves the evaluation function of the dynamic window method. In order to enable the robot to still follow the path after avoiding collision, the function and weight for evaluating the robot position and the distance from the path are added. The improved evaluation function is:

[0092] G1(v,ω)=α1·heading(v,ω)+k·β1·dist(v,ω)+γ1·velocity(v,ω)+(1-k)·δ1·d line (v,ω)

[0093] Among them, G1(v,ω) is the first improved evaluation function, d line (v,ω) represents the path evaluation function, which is used to evaluate the robot position and the distance from the path; k represents the obstacle detection coefficient, which is 1 when there is obstacle detection and 0 when there is no obstacle detection; α1, β1, γ1 and δ1 are all weight parameters.

[0094] Reference Figure 2As shown in the figure, when there are many obstacles and the movable area is narrow, the robot can follow the path line according to the improved evaluation function, but it will still produce a large detour in the area. In addition, the weight parameters of the added path evaluation function still need to be set through experiments, so it will also fall into the regional optimum.

[0095] In order to make up for the problem that the improved evaluation function cannot simply reflect the path following function according to whether an obstacle is detected, the present invention further changes the obstacle detection coefficient to a collision avoidance coefficient, which is defined as:

[0096]

[0097] Among them, λ is the collision avoidance coefficient, (x',y') is the position of the robot at the next time node, (x obs ,y obs ) is the position of the obstacle, d collide Denotes the collision distance, d avoid Indicates a safe distance.

[0098] The formula for collision distance is:

[0099]

[0100] Where a represents the acceleration of the robot.

[0101] Therefore, the further optimized evaluation function is:

[0102] G p (v,ω)=α p ·heading(v,ω)+λ·β p ·dist(v,ω)+γ p velocity(v,ω)+|1-λ|·δ p ·d line (v,ω)

[0103] Among them, G p (v,ω) is the second improved evaluation function, α p , β p , β p and δ p are all weight parameters.

[0104] In order to be applicable to the reinforcement learning proximal policy optimization algorithm, the weight parameter α in the further optimized evaluation function is p , β p , γ p , δ p and d avoid as a learnable parameter.

[0105] The robot must not collide with obstacles under any circumstances and must follow the given path line in an obstacle-free environment, so dist(v,ω) and d line The value of the weight parameter of (v,ω) must have a higher value than the weight parameter of heading(v,ω) or velocity(v,ω).

[0106] Specifically, the proximal policy optimization algorithm is a method of learning the optimal strategy to maximize the reward in a known or unknown environment based on the actions selected by the robot to identify the state of the environment. The reward is a parameter that determines the state value based on the effect of the Q-value function. The following is a detailed introduction.

[0107] Reinforcement Learning (RL) is a Markov Decision Process (MDP). In this framework, the agent learns how to make the best decision by interacting with the environment. The Markov decision process is used to describe decision-making problems in discrete time steps. It includes the following:

[0108] (1) A finite set of states S (initial state s0);

[0109] (2) Action set A;

[0110] (3) Strategy π θ (s t+1 |s t ,a t ), using parameter θ to control the transition from the state and action pair at time t to the state at time t+1;

[0111] (4) Reward function R(s) t ,a t ,s t+1 ) is used to evaluate actions;

[0112] (5) A discount factor γ∈[0,1], which is used to evaluate the importance of immediate rewards to future rewards (smaller values ​​of γ place more emphasis on immediate rewards). The goal of RL is to determine the optimal policy π * , so that the maximum expected reward is obtained in all states:

[0113] By applying the Markov decision process in reinforcement learning, the agent continuously optimizes its strategy through trial and error, learning, and adjustment to obtain the maximum cumulative reward.

[0114] As a specific implementation method of reinforcement learning, the proximal policy optimization algorithm is a gradient optimization algorithm proposed and released by researchers at OpenAI. The main feature of the proximal policy optimization algorithm is multiple small batches of updated gradients. The standard policy gradient samples data through interaction with the environment and uses stochastic gradient ascent to optimize the objective function. The policy gradient method performs gradient updates on each data sample. Figure 3 It is the basic framework diagram of the proximal strategy optimization algorithm.

[0115] There are many variants of the PPO algorithm, the most commonly used are PPO-Clip and PPO-Penalty. In order to further strengthen the constraints on policy updates, the PPO-Penalty algorithm introduces a penalty term to limit the amplitude of policy changes, thereby achieving more precise control of the step size of policy updates and improving the stability and reliability of the algorithm. This variant is similar to Trust Region Policy Optimization (TRPO) and has better sample complexity. However, the amount of computation required for each update step is very large. Therefore, in PPO-Clip, the penalty term is changed to a hard constraint, and a clipping term is introduced to limit the amplitude of policy updates, thereby achieving a more stable optimization process. This method solves the problem of the original PPO algorithm in that the update step size is too small.

[0116] On the basis of reinforcement learning, two other concepts are introduced to describe the RL model; the state value function V π (s) and the action-value function Q π (s,a). The state value function measures the value of a state by evaluating the expected return of an agent starting from a specific state and following a certain policy. The action value function measures the goodness of an action by evaluating the expected return of taking a specific action in a specific state. The state value function and action value can be defined by the following two expressions:

[0117]

[0118] The PPO algorithm uses the following advantage function to update parameters:

[0119] Ad π (s,a)=Q π (s,a)-V π (s)

[0120] Secondly, in order to avoid the situation where greedy decisions cause parameter updates to deviate from good approximations, the PPO algorithm updates parameters within a trust region. This modification can be applied to the following formula:

[0121]

[0122] This leads to the following formula:

[0123]

[0124] Among them, θ old is the current parameter in the network, and ∈ is used to control the size of the trust region. The role of g(∈,Ad) is to ensure that the parameter update is not too large and to adjust the degree of parameter update. With these two improvements, the PPO algorithm provides stable parameter updates, which is very beneficial for handling large environments. In addition, the PPO algorithm supports discrete and continuous actions and observations, which has greater flexibility in practical problem design.

[0125] This embodiment optimizes the reward of the proximal strategy optimization algorithm and sets the reward to:

[0126] r t =r g t +r c t +r m t

[0127] Among them, r t represents the total reward of the robot at time t, r g t represents the robot's positive reward at time t, r c t represents the robot's penalty at time t, r m t represents the step size penalty of the robot at time t.

[0128] The robot’s positive reward r at time t g t It can make the robot reach the target position. The calculation formula is:

[0129]

[0130] Among them, r arrival represents the arrival reward value, d represents the distance between the target position and the robot position, and d goal Indicates the preset target distance. The condition for the robot to reach the target position is that the distance d between the target position and the robot position is less than the preset target distance d goal .

[0131] When the robot collides with an obstacle, the robot is penalized. The robot’s penalty at time t is r c t for:

[0132]

[0133] Among them, r collision represents the collision penalty value, l n Indicates the distance from the robot to the obstacle, d r Indicates the robot radius.

[0134] The robot radius is defined as a circle equal to the size of the robot model as the radius of the robot's center. n At least one of them is smaller than the robot radius d r In the case of satisfying the penalty r c t The collision condition is met and the event ends.

[0135] In order to prevent local minima, this embodiment sets a small penalty at each time step, as the robot's step penalty r at time t m t , the formula is:

[0136] r m t =r time

[0137] Among them, r time Represents the time step penalty value.

[0138] This embodiment uses the proximal strategy optimization algorithm to learn the weight parameters in the second improved evaluation function, establishes a fusion algorithm, and outputs target weight parameters, including: α p The value of is 0.2, β p The value of is 1, γ p The value of is 0.3, δ p The value of d is 1, avoid The value of is 2.

[0139] The target evaluation function of the robot motion model is obtained based on the target weight parameters.

[0140] In each time step, the robot motion model samples several speed groups in the speed window, calculates the target evaluation function value of each speed group, and selects the speed group with the largest target evaluation function value as the robot's movement strategy.

[0141] In summary, the robot path planning method that integrates dynamic window and PPO algorithm described in the present invention constructs a robot motion model based on the dynamic window method, establishes an evaluation function of the robot motion model, uses the weight parameters in the evaluation function as learnable parameters, and uses the proximal strategy optimization algorithm to learn the learnable parameters, so that the robot has smooth autonomous learning ability when performing path planning, so as to adapt to environments with dynamic obstacles and also has robustness in complex and changeable environments.

[0142] Furthermore, the present invention optimizes the evaluation function of the robot motion model, adds the evaluation of the path on the basis of the standard evaluation function, so that the robot can still follow the path after avoiding obstacles. And introduces the collision avoidance coefficient into the evaluation function, which improves the robot's obstacle avoidance in a complex environment with multiple dynamic obstacles.

[0143] Embodiment 2

[0144] In order to verify the effect of the robot path planning method integrating the dynamic window and the PPO algorithm described in Example 1, this example conducted simulation experiments and result analysis.

[0145] This embodiment sets a 30×30 map as the experimental environment for robot path planning. In the simulation, a complex multi-obstacle (including dynamic obstacles) environment is established. Figure 4 As shown in the figure, the red hollow circle is a static obstacle, the black hollow circle is a dynamic obstacle, and the white part is the robot's traversable area. The action space is three possible actions (turn left, turn right, and move forward), where the specific turning angle depends on the robot's maximum turning speed and acceleration limit, as well as the position and speed of obstacles in the current environment. The algorithm generates multiple combinations of candidate speeds and turning angles, and then evaluates the score of each candidate combination, and selects the combination with the highest score as the robot's next move strategy. The robot's starting coordinates are (0, 0) and the target point coordinates are (28, 28).

[0146] In order to obtain a better contrast effect, this embodiment constructs simple, general and complex obstacle environments respectively. Figure 5 This is the path planning result diagram under simple obstacles in the simulation experiment, where Figure 5 a in the figure is the path planning result diagram of the dynamic window method in a static obstacle environment. After adding dynamic obstacles, the path smoothness of the algorithm is poor and there are more turning points due to the movement of dynamic obstacles. Figure 5 b in the figure is the path planning result graph of the traditional dynamic window method. Figure 5 c in the figure is the path planning result diagram of the improved dynamic window method with limited maximum speed and acceleration. Figure 5 The d in is the path planning result diagram of the fusion algorithm. By comparison, it is found that in an environment with only static simple obstacles, the dynamic window method has a better effect, but in a simple environment with dynamic obstacles, the algorithm of the present invention can still maintain a good planning efficiency.

[0147] Figure 6 This is a schematic diagram of the path planning process under complex obstacles in the simulation experiment, where: Figure 6 (a) is a schematic diagram of the first stage path planning. Figure 6(b) is a schematic diagram of the second stage path planning. Figure 6 (c) is a schematic diagram of the path planning in the third stage. Figure 6 (d) in the figure is a schematic diagram of the path planning in the fourth stage.

[0148] Figure 7 This is the path planning result diagram under complex obstacles in the simulation experiment, where: Figure 7 Column (a) in the figure is the path planning result of the dynamic window method. Figure 7 Column (b) in is the path planning result diagram of the dynamic window method using the first improved evaluation function. Figure 7 Column (c) in the figure is the path planning result graph of the fusion algorithm.

[0149] Depend on Figure 7 It can be seen that in an environment with 30 static obstacles and 8 dynamic obstacles, and in a complex environment with 50 static obstacles and 10 dynamic obstacles, the path planning method proposed in the present invention does not require any prior knowledge of the map environment, and can also quickly and safely find the target point and reasonably plan a smooth path.

[0150] Table 1 shows the three simulation results of the dynamic window method, the dynamic window method using the first improved evaluation function, and the fusion algorithm in two environments. It can be seen that compared with the traditional dynamic window method, the planning time of the fusion algorithm is reduced by 45.12%, the path is shortened by 23.04%, and the turning points in the path are reduced by 47.37%. Although the traditional algorithm can eventually find the end point, it is easy to fall into the local optimum.

[0151] Table 1 Comparison of simulation results

[0152]

[0153] In the experiment in the real environment, due to the limitation of the experimental conditions, this embodiment only established a simple grid-type obstacle and performed simulation in ROS.

[0154] The robot platform and environment are built in Ubuntu 16.04. In addition, the software code is based on ROS Kinetic to integrate learning and control. The robot used in this embodiment is a differential robot with a maximum linear speed of 0.5m / s and a maximum rotation speed of 1.2rad / s. The robot is equipped with a 180° laser range radar sensor. Its scanning frequency is 15Hz and the angular resolution is 0.18°. The robot detects obstacles in front through the laser radar and returns data to the robot.

[0155] In the above map environment, in addition to the wall, two other robots were used as obstacles on the path to confirm the avoidance performance and the ability to escape from the self-damage situation. In order to reduce the learning time to reach the completion condition, the safety distance d avoid Set to 0.28, which is the maximum radius that satisfies the target position to completely avoid dynamic obstacles.

[0156] In order to fully compensate for the collision between the sensor-detected obstacles and the robot, the robot radius is set to 0.3, which is larger than the radius of the actual robot model (0.225m). In order to obtain map information about the experimental environment, the navigation stack is used in the ROS navigation package to drive the dynamic window method. The physical map simulation diagram is shown in the figure Figure 8 As shown in Figure 2, the path planning results of the physical experiment are as follows: Fig. 9 shown.

[0157] Through simulation experiments and physical experiments, it is found that when the robot interacts with an unknown environment, the random probability of choosing an action is relatively large. This randomness is due to the robot's lack of prior knowledge and experience of the unknown environment, as well as the need to adapt to uncertain factors. When the robot first encounters an unknown environment, it may take a variety of actions in order to explore and obtain more information about the environment. This results in the action performed by the robot in the current state not necessarily being the action with the largest evaluation function value. Therefore, the success rate fluctuates greatly during the early training process. As the robot continues to explore the environment, it gradually learns sufficient prior knowledge, and the robot can choose the optimal action in the current state to execute. Therefore, using this fusion algorithm, the robot has a smooth learning ability when performing path planning, and is also robust in complex and changing environments.

[0158] Embodiment 3

[0159] Based on the robot path planning method integrating dynamic window and PPO algorithm described in the first embodiment, the present invention further provides a robot path planning device integrating dynamic window and PPO algorithm, comprising:

[0160] A motion model building module is used to build a robot motion model based on a dynamic window method, limit the speed window of the robot motion model, and establish an evaluation function of the robot motion model;

[0161] A parameter training module is used to use the weight parameters in the evaluation function of the robot motion model as learnable parameters in the training environment, give the range of the learnable parameters, use the proximal strategy optimization algorithm to learn the learnable parameters, output the target weight parameters, and obtain the target evaluation function of the robot motion model;

[0162] The path planning module is used to sample several speed groups in the speed window of the robot motion model in each time step, calculate the target evaluation function value of each speed group, and select the speed group with the largest target evaluation function value as the robot's movement strategy.

[0163] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the robot path planning method that integrates the dynamic window and the PPO algorithm are implemented.

[0164] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0165] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0168] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A robot path planning method integrating dynamic window and PPO algorithm, characterized in that: include: The robot motion model is constructed based on the dynamic window method, and the speed window of the robot motion model is limited. Establish the evaluation function of the robot motion model; In the training environment, the weight parameters in the evaluation function of the robot motion model are used as learnable parameters. Given the range of the learnable parameters, the learnable parameters are learned using the proximal strategy optimization algorithm, and the target weight parameters are output to obtain the target evaluation function of the robot motion model. In each time step, the robot motion model samples several speed groups in the speed window, calculates the target evaluation function value of each speed group, and selects the speed group with the largest target evaluation function value as the robot's movement strategy.

2. A robot path planning method integrating dynamic window and PPO algorithm according to claim 1, characterized in that: The robot motion model is constructed based on the dynamic window method, including: Define the current position of the robot as (x, y), the angle between the robot and the x-axis as θ, and the angular velocity and linear velocity as ω and v respectively; If the angular velocity is zero, after a time step of Δt, the robot's position is updated to (x', y') = (x + vΔtcosθ, y + vΔtsinθ); If the angular velocity is not zero, after a time step Δt, the robot's position is updated to: Where (x', y') is the position of the robot after time step Δt, is the angle between the robot and the x-axis after the robot time step Δt.

3. The robot path planning method integrating dynamic window and PPO algorithm according to claim 1, characterized in that: The velocity window of the robot motion model is limited according to the robot's velocity range, maximum acceleration, maximum deceleration, and the minimum distance between the robot and obstacles.

4. A robot path planning method integrating dynamic window and PPO algorithm according to claim 3, characterized in that: The speed window of the robot motion model is limited according to the robot's speed range. The formula is expressed as: V m ={v∈[v min ,v max ],ω∈[ω min ,ω max ]} Among them, V m is the first velocity window, v min and v max are the minimum and maximum linear speeds of the robot, ω min and ω max are the minimum angular velocity and maximum angular velocity of the robot respectively; The speed window of the robot motion model is limited according to the maximum acceleration and maximum deceleration. The formula is expressed as: Among them, V d is the second speed window, v c and ω c are the current linear velocity and angular velocity of the robot, respectively, b and ω b are the maximum deceleration of the robot’s linear velocity and angular velocity, v a and ω a are the maximum acceleration of the robot’s linear velocity and angular velocity, respectively; The speed window of the robot motion model is limited according to the minimum distance between the robot and the obstacle. The formula is expressed as: Among them, V a is the third speed window, dist min (v,ω) represents the minimum distance between the robot and the obstacle; The target speed window is determined based on the first speed window, the second speed window and the third speed window: V=V m ∩V d ∩V a Where V is the target speed window.

5. The robot path planning method integrating dynamic window and PPO algorithm according to claim 1, characterized in that: The evaluation function of the robot motion model is established, and the formula is: G0(v,ω)=α0·heading(v,ω)+β0·dist(v,ω)+γ0·velocity(v,ω) Among them, G0(v,ω) is the evaluation function of the robot motion model, heading(v,ω) represents the angle evaluation function, which is the angle difference between the robot's heading and the target when it reaches the predicted position, dist(v,ω) represents the distance evaluation function, velocity(v,ω) represents the speed evaluation function, and α0, β0 and γ0 are all weight parameters.

6. A robot path planning method integrating dynamic window and PPO algorithm according to claim 5, characterized in that: The path evaluation function is introduced to improve the evaluation function of the robot motion model. The improved evaluation function is: G1(v,ω)=α1·heading(v,ω)+k·β1·dist(v,ω)+γ1·velocity(v,ω)+(1-k)·δ1·d line (v,ω) Among them, G1(v,ω) is the first improved evaluation function, d line (v,ω) represents the path evaluation function; k represents the obstacle detection coefficient; α1, β1, γ1 and δ1 are all weight parameters.

7. The robot path planning method integrating dynamic window and PPO algorithm according to claim 6, characterized in that: The obstacle detection coefficient is changed to the collision avoidance coefficient, and the evaluation function of the robot motion model is further optimized as follows: G p (v,ω)=a p ·heading(v,ω)+λ·β p ·dist(v,ω)+γ p ·velocity(v,ω)+|1-λ|·δ p ·d line (v,ω) Among them, G p (v,ω) is the second improved evaluation function, λ is the collision avoidance coefficient, α p , β p , γ p and δ p They are all weight parameters; The collision avoidance coefficient is defined as: Among them, λ is the collision avoidance coefficient, (x',y') is the position of the robot at the next time node, (x obs ,y obs ) is the position of the obstacle, d collide Denotes the collision distance, d avoid Indicates a safe distance.

8. The robot path planning method integrating dynamic window and PPO algorithm according to claim 1, characterized in that: The reward setting for the proximal policy optimization algorithm is: r t ＝r g t +r c t +r m t Among them, r t represents the total reward of the robot at time t, r g t represents the robot's positive reward at time t, r c t represents the robot's penalty at time t, r m t represents the step penalty of the robot at time t.

9. A robot path planning device integrating dynamic window and PPO algorithm, characterized in that: include: The motion model building module is used to build the robot motion model based on the dynamic window method, limit the speed window of the robot motion model, and establish the evaluation function of the robot motion model; A parameter training module is used to use the weight parameters in the evaluation function of the robot motion model as learnable parameters in the training environment, give the range of the learnable parameters, use the proximal strategy optimization algorithm to learn the learnable parameters, output the target weight parameters, and obtain the target evaluation function of the robot motion model; The path planning module is used to sample several speed groups in the speed window of the robot motion model in each time step, calculate the target evaluation function value of each speed group, and select the speed group with the largest target evaluation function value as the robot's movement strategy.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a robot path planning method integrating a dynamic window and a PPO algorithm as described in any one of claims 1 to 8 are implemented.

Images