Mobile Robot Path Planning Method Based on Improved Q-learning Algorithm

Abstract

The present invention is a path planning method for a mobile robot based on an improved Q-learning algorithm. The mobile robot first uses the traditional Q-learning algorithm to perform initial path exploration. Once the mobile robot reaches the target point, it stops exploring, initializes the Q-table, and uses the initialized Q-table for subsequent path planning. Then, the greedy strategy and action selection strategy of the traditional Q-learning algorithm are improved. The mobile robot uses the improved Q-learning algorithm to perform path exploration. The improved greedy strategy constrains the exploration probability based on the position of the mobile robot and the current iteration number. The improved action selection strategy uses the initialized Q-table and the distance between the next state and the target point to select actions, reducing ineffective exploration and solving problems such as the slow convergence speed and large blindness in the early stage of exploration of the traditional Q-learning algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of robot path planning, and particularly to a path planning method for a mobile robot based on an improved Q-learning algorithm. Background Art

[0002] With the development of robot technology, mobile robots are widely used in various fields. Autonomous navigation is the prerequisite for mobile robots to complete various tasks, and path planning is the key content for mobile robots to achieve autonomous navigation. Path planning mainly enables a mobile robot to find a collision-free optimal path from a starting point to an ending point within a specified area, which can effectively reduce redundant paths and improve work efficiency. Currently, path planning algorithms are mainly divided into two categories: path planning based on traditional algorithms and path planning based on swarm intelligence algorithms. Traditional algorithms include the A* algorithm, artificial potential field method, etc., and swarm intelligence algorithms include particle swarm algorithm, ant colony algorithm, etc. Both traditional algorithms and swarm intelligence algorithms have problems such as large computational complexity, long convergence time, and poor adaptability to complex environments.

[0003] Reinforcement learning is an important branch of machine learning. An agent continuously optimizes the state-action correspondence by interacting with the environment to obtain feedback from the environment. The path planning method based on reinforcement learning enables a mobile robot to have the ability of autonomous learning, providing a new idea for solving the path planning of mobile robots in complex environments. The Q-learning algorithm in reinforcement learning algorithms is widely used in the path planning of mobile robots. For a given state-action pair, a corresponding action value will be generated. The environment gives corresponding rewards according to the actions taken by the mobile robot and updates the Q value. The mobile robot selects the best action according to the Q value, and then plans to obtain the optimal path. However, the Q-learning algorithm has problems such as large blindness in the early exploration stage, slow convergence speed, and being easily trapped in local optima. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the technical problem to be solved by the present invention is to provide a path planning method for a mobile robot based on an improved Q-learning algorithm.

[0005] The technical solution adopted by the present invention to solve the above technical problem is as follows:

[0006] A path planning method for a mobile robot based on an improved Q-learning algorithm, characterized in that the method includes the following steps:

[0007] Step 1: Construct a grid map, and obtain the starting point, target point, and obstacle positions;

[0008] Step 2: The mobile robot uses the traditional Q-learning algorithm for initial path exploration and stops exploring once it reaches the target point. During the exploration process, the action value is calculated according to the action value function in Equation (1), and the Q-table is initialized.

[0009]

[0010] In the formula, Q(s t ,a t ) is the action value function at time t, indicating the value generated by the mobile robot executing action a t in state s t . α is the learning rate, R(s t ,a t ) represents the reward obtained by executing action a t , Q(s t+1 ,a t+1 ) represents the prediction of the action value at time t + 1, s t+1 is the state at time t + 1, a t+1 is the action at time t + 1, A represents the action set, and γ represents the reward decay factor.

[0011] Step 3: Improve the greedy strategy and action selection strategy of the traditional Q-learning algorithm, and the mobile robot uses the improved Q-learning algorithm for path exploration.

[0012] The exploration probability ε of the improved greedy strategy is expressed as:

[0013] ε = ε 1 *exp(b * i)(3)

[0014]

[0015] In the formula, ε 0 is the initial exploration probability, i is the current iteration number, k 1 , k 2 and b are all hyperparameters, d 1 is the distance between the mobile robot and the starting point, d 2 is the distance between the mobile robot and the target point, (x t ,y t ) represents the position of the mobile robot at time t, U 1 represents the area around the starting point, U 2 represents the area around the target point, U 3 represents the area in the grid map except U 1 , U 2 ;

[0016] The improved action selection strategy is as follows: Generate a random number between 0 and 1. If the random number is less than or equal to the exploration probability, compare the maximum action value of the predicted next state with the action values of each action corresponding to the state in the initialized Q-table, and select the action with the maximum action value as the execution action of the mobile robot. If the random number is greater than the exploration probability, calculate the distances between the eight possible states of the next state and the target point respectively through Equation (5), and select the state with the shortest distance as the next state of the mobile robot.

[0017]

[0018] Among them, (x goal , y goal ) represents the coordinates of the target point, represents the coordinate point of the next state of the mobile robot;

[0019] Step 4: According to the reward obtained at the current moment, judge whether the mobile robot collides with an obstacle. If so, end this iteration and return to Step 3; if not, execute Step 5;

[0020] Step 5: According to the reward obtained at the current moment, judge whether the mobile robot reaches the target point. If so, go to Step 6; otherwise, return to Step 3;

[0021] Step 6: Judge whether the maximum number of iterations is reached. If so, output the optimal path; otherwise, return to Step 3 until the optimal path is obtained.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. Based on the idea of transfer learning, before exploring the shortest path, the mobile robot conducts initial path exploration through the traditional Q-learning algorithm. Once the mobile robot reaches the target point, the exploration stops. The Q-table is initialized using the Q-values obtained during the initial path exploration, and the initialized Q-table is transferred to the subsequent path optimization as prior knowledge for finding the shortest path, reducing the ineffective exploration during the path optimization process and solving the problems of large blindness and slow convergence speed existing in the traditional Q-learning algorithm in the early stage of exploration.

[0024] 2. Improve the greedy strategy and action selection strategy of the traditional Q-learning algorithm, use the robot's position and the number of iterations to constrain the exploration probability, introduce the distance between the next state and the target point for action selection, exclude useless actions, reduce ineffective exploration, reduce the dimension (number of rows) of the Q-table, and further shorten the convergence time. The simulation results show that the improved Q-learning algorithm for path planning in a complex environment can effectively reduce the number of path steps, shorten the convergence time, and improve the path planning efficiency. Brief Description of the Drawings

[0025] Figure 1 is the overall flowchart of the present invention;

[0026] Figure 2 is the convergence curve graph of the traditional Q-learning algorithm in Experiment 1;

[0027] Figure 3 is the convergence curve graph of the improved Q-learning algorithm of the present invention in Experiment 1;

[0028] Figure 4 is the path planning trajectory graph of the traditional Q-learning algorithm in Experiment 1;

[0029] Figure 5 is the path planning trajectory graph of the improved Q-learning algorithm of the present invention in Experiment 1;

[0030] Figure 6 is the convergence curve graph of the traditional Q-learning algorithm in Experiment 2;

[0031] Figure 7 is the convergence curve graph of the improved Q-learning algorithm of the present invention in Experiment 2;

[0032] Figure 8 is the path planning trajectory graph of the traditional Q-learning algorithm in Experiment 2;

[0033] Figure 9 is the path planning trajectory graph of the improved Q-learning algorithm of the present invention in Experiment 2. Detailed Implementation Manner

[0034] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to describe the technical solutions of the present invention in detail and are not used to limit the protection scope of this application.

[0035] The present invention provides a mobile robot path planning method based on an improved Q-learning algorithm, and the method includes the following steps:

[0036] Step 1: Construct a grid map, and obtain the starting point, target point, and obstacle positions; in order to avoid the planned path being too close to the obstacles, causing the mobile robot to collide with the obstacles, each obstacle is subjected to dilation processing, and the dilation size is half of the grid length;

[0037] Step 2: The initial values of the Q-table in the traditional Q-learning algorithm are all 0, resulting in strong randomness, great blindness, and long convergence time in the early exploration of the mobile robot. Therefore, the mobile robot is used to perform path exploration through the traditional Q-learning algorithm. Once the mobile robot reaches the target point, the exploration stops. During the exploration process, the Q value is calculated to initialize the Q-table of the Q-learning algorithm, and the initialized Q-table is used for subsequent path planning to play a guiding role in path planning.

[0038] The process of path exploration is as follows: In each state, an action is selected from the action set according to the action selection strategy, and the action value, that is, the Q value, is calculated through the action value function to initialize the Q-table.

[0039] The action set includes a total of eight actions: moving up, moving down, moving left, moving right, moving up-left, moving down-left, moving up-right, and moving down-right. After each action is executed, the mobile robot moves one grid in the corresponding direction.

[0040] The action selection strategy is as follows: Generate a random number between 0 and 1, compare the random number with the exploration probability. If the random number is less than or equal to the exploration probability, execute the action with the maximum value of the next state; if the random number is greater than the exploration probability, execute the action randomly. The exploration probability of the traditional Q-learning algorithm is a fixed value.

[0041] The action value function is:

[0042]

[0043] In the formula, Q(s t ,a t ) is the action value function at time t, indicating the value generated by the mobile robot executing action a t in state s t ; α is the learning rate, which affects the update speed of the Q value; R(s t ,a t ) represents the reward obtained by executing action a t , Q(s t+1 ,a t+1 ) represents the prediction of the action value at time t + 1, s t+1 is the state at time t + 1, a t+1 is the action at time t + 1, and A represents the action set; γ represents the reward decay factor, reflecting the influence of future reward values on the Q value of the current state.

[0044] The reward function is:

[0045]

[0046] Step 3: Improve the traditional Q-learning algorithm, including the improvement of the greedy policy and the action selection policy; the mobile robot uses the improved Q-learning algorithm for path exploration;

[0047] The greedy policy of the traditional Q-learning algorithm selects actions with strong randomness, the exploration probability is a given value, and the convergence speed is slow. Therefore, the present invention improves the greedy policy and dynamically adjusts the exploration probability ε from two perspectives; on the one hand, the grid map is divided into three regions, region U 1 is a square region located around the starting point with a side length of 4 to 6 grid lengths, region U 2 is a square region located around the target point with a side length of 4 to 6 grid lengths, and the remaining region is region U 3 , and the exploration probability ε is adjusted according to the region where the mobile robot is located at the current moment; on the other hand, the exploration probability is constrained by the number of iterations, and the update formula of the exploration probability ε is:

[0048] ε = ε 1 *exp(b * i)(3)

[0049]

[0050] In the formula, ε 0 is the initial exploration probability, i is the current number of iterations, k 1 , k 2 and b are all hyperparameters, d 1 is the distance between the mobile robot and the starting point, d 2 is the distance between the mobile robot and the target point, (x t , y t ) represents the position of the mobile robot at time t;

[0051] Use the distance between the position of the mobile robot and the target point to guide action selection. The improved action selection policy is: generate a random number between 0 and 1. If the random number is less than or equal to the exploration probability, compare the maximum action value of the predicted next state with the action values of each action in the corresponding state of the initialized Q-table, and select the action with the largest action value as the execution action of the mobile robot, reducing ineffective exploration; for example, if the maximum action value of the predicted next state is greater than the action values of each action in the corresponding state of the initialized Q-table, the mobile robot executes the action with the largest value of the predicted next state; if the maximum action value of the predicted next state is less than the action value of a certain action in the corresponding state of the initialized Q-table, the mobile robot executes the action corresponding to this action value;

[0052] If the random number is greater than the exploration probability, calculate the distances between the eight possible states of the next state and the target point respectively through Equation (5), and select the state with the shortest distance as the next state of the mobile robot; in the current state, there are eight possible next states of the mobile robot, corresponding to eight actions respectively;

[0053] d = |x goal - x st+1 | + |y goal - y st+1 | (5)

[0054] where represents the coordinates of the target point, represents the coordinate point of the next state of the mobile robot;

[0055] The mobile robot uses the improved Q - learning algorithm for path exploration, that is, selects an action from the action set according to the improved action selection strategy, calculates the reward value through the reward function; calculates the action value using the action value function of Equation (1), and updates the Q - table;

[0056] Step 4: According to the reward obtained at the current moment, judge whether the mobile robot collides with an obstacle. If so, end this iteration and return to Step 3; if not, execute Step 5;

[0057] Step 5: According to the reward obtained at the current moment, judge whether the mobile robot reaches the target point. If so, go to Step 6; otherwise, return to Step 3;

[0058] Step 6: Judge whether the maximum number of iterations is reached. If so, output the optimal path; otherwise, return to Step 3 until the optimal path is obtained.

[0059] In the above steps, the maximum number of iterations is 5000 times, the learning rate α = 0.01, the reward decay factor γ = 0.9, and the initial exploration probability ε 0 = 0.9.

[0060] Verify the effectiveness of the mobile robot path planning method based on the improved Q - learning algorithm through simulation.

[0061] Experiment 1: The size of the grid map is 25 * 25, the length of each grid is 20, the starting point of the mobile robot is (0, 0), the target point is (20, 20), a total of eight obstacles are set, and the target point is set in the concave area formed by the obstacles, and the map environment is complex. Figure 2 、 3They are respectively the convergence curve graphs of the traditional Q-learning algorithm and the improved Q-learning algorithm of the present invention. It can be seen from the graphs that the traditional Q-learning algorithm converges after 2,800 iterations, and the improved Q-learning algorithm converges after 1,200 iterations. Compared with the traditional Q-learning algorithm, the convergence time of the improved Q-learning algorithm of the present invention is shortened by 57.1%, significantly improving the path planning efficiency. Figure 4 、 5 They are respectively the path planning trajectory graphs obtained by the traditional Q-learning algorithm and the improved Q-learning algorithm of the present invention. In the graphs, the small black circles represent the starting points, the small gray squares represent the target points, and the black polygons represent the obstacles; it can be seen from the graphs that the optimal path planned by the traditional Q-learning algorithm contains 30 steps; the optimal path planned by the improved Q-learning algorithm contains 26 steps. The optimal path of the present invention is shorter and has fewer turning points.

[0062] Experiment 2: The size of the grid map is 30*30, the starting point of the mobile robot is (0,0), the target point is (25,25), and a total of thirteen obstacles are set. The target point is set in the concave area formed by the obstacles; compared with the map environment of Experiment 1, the environment of Experiment 2 is more complex, with more given obstacles, increasing the difficulty of path planning. Figure 6 、 7 They are respectively the convergence curve graphs of the traditional Q-learning algorithm and the improved Q-learning algorithm of the present invention. It can be seen from the graphs that the traditional Q-learning algorithm converges after 4,000 iterations, and the improved Q-learning algorithm converges after 1,900 iterations. Compared with the traditional Q-learning algorithm, the convergence time of the improved Q-learning algorithm of the present invention is shortened by 52.5%, accelerating the algorithm convergence speed. Figure 8 、 9 They are respectively the path planning trajectory graphs obtained by the traditional Q-learning algorithm and the improved Q-learning algorithm of the present invention. It can be seen from the graphs that the optimal path planned by the traditional Q-learning algorithm contains 37 steps; the optimal path planned by the improved Q-learning algorithm contains 32 steps. The optimal path of the present invention is shorter and has fewer turning points.

[0063] Based on the above two experiments, the improved Q-learning algorithm of the present invention has a short path planning convergence time and fewer steps in the optimal path in a complex environment, significantly improving the convergence speed.

[0064] Matters not described in the present invention are applicable to the prior art.

Claims

1. A path planning method for a mobile robot based on an improved Q-learning algorithm, characterized in that, the method comprises the following steps: Step 1: Construct a grid map, and obtain the starting point, the target point and the positions of obstacles; Step 2: The mobile robot uses the traditional Q-learning algorithm to perform initial path exploration. Once the mobile robot reaches the target point, the exploration stops; during the exploration process, the action value is calculated according to the action value function of formula (1), and the Q-table is initialized; where Q(s t , a t ) is the action-value function at time t, representing the value generated by the mobile robot when executing action a t in state s t ; α is the learning rate, R(s t , a t ) represents the reward obtained by executing action a t , Q(s t+1 , a t+1 ) represents the prediction of the action value at time t + 1, s t+1 is the state at time t + 1, a t+1 is the action at time t + 1, A represents the action set, and γ represents the reward decay factor; Step 3: Improve the greedy strategy and the action selection strategy of the traditional Q-learning algorithm, and the mobile robot uses the improved Q-learning algorithm to perform path exploration; The exploration probability ε of the improved greedy strategy is expressed as: ε = ε 1 *exp(b*i) (3) where ε 0 is the initial exploration probability, i is the current iteration number, k 1 , k 2 and b are all hyperparameters, d 1 is the distance between the mobile robot and the starting point, d 2 is the distance between the mobile robot and the target point, (x t , y t ) represents the position of the mobile robot at time t, U 1 represents the area around the starting point, U 2 represents the area around the target point, U 3 represents the area in the grid map except U 1 , U 2 ; The improved action selection strategy is: generate a random number between 0 and 1. If the random number is less than or equal to the exploration probability, compare the maximum action value of the predicted next state with the action values of each action in the corresponding state in the initialized Q-table, and select the action with the largest action value as the execution action of the mobile robot; if the random number is greater than the exploration probability, calculate the distances between the eight possible states of the next state and the target point respectively through formula (5), and select the state with the shortest distance as the next state of the mobile robot; Among them, (x goal , y goal ) represents the coordinates of the target point, represents the coordinate point of the next state of the mobile robot; Step 4: According to the reward obtained at the current moment, judge whether the mobile robot collides with an obstacle. If so, end this iteration and return to Step 3; if not, execute Step 5; Step 5: According to the reward obtained at the current moment, judge whether the mobile robot reaches the target point. If so, go to Step 6; otherwise, return to Step 3; Step 6: Judge whether the maximum number of iterations is reached. If so, output the optimal path; otherwise, return to Step 3 until the optimal path is obtained.

2. The path planning method for a mobile robot based on an improved Q-learning algorithm according to claim 1, characterized in that, the action set of the mobile robot includes eight actions: moving up, moving down, moving left, moving right, moving up-left, moving down-left, moving up-right and moving down-right.

3. The path planning method for a mobile robot based on an improved Q-learning algorithm according to claim 1, characterized in that, Region U 1 is a square region located around the starting point with a length of 4 to 6 grid lengths. Region U 2 is a square region located around the target point with a length of 4 to 6 grid lengths.

4. The path planning method for a mobile robot based on an improved Q-learning algorithm according to any one of claims 1 to 3, characterized in that, the reward function of the mobile robot is:

Images