A reinforcement learning path planning method integrating historical information

Abstract

The present invention relates to the technical field of robot path planning, and in particular to a reinforcement learning path planning method integrating historical information, comprising the following steps: S1: acquiring an environment image and establishing a grid map; S2: defining a reinforcement learning state space and an action space; S3: initializing algorithm parameters; S4: dynamically adjusting the action space; S5: selecting an action in the adjusted action space using an ε-greedy strategy; S6: executing the action and updating the Q value; S7: repeatedly executing the fourth step, the fifth step, and the sixth step until a certain number of steps or a certain convergence condition is reached; S8: selecting an action with the largest Q value in each step to obtain an optimal path; and S9: sending the optimal path to a controller of a mobile robot to control the mobile robot to walk according to the optimal path.

Description

Technical Field

[0001] The present invention relates to the technical field of robot path planning, and in particular to a reinforcement learning path planning method integrating historical information. Background Art

[0002] In recent years, mobile robots have been widely used in various industries, and the requirements for the performance of mobile robots are also increasing. The path planning problem in mobile robots is a key technology to realize the functions of mobile robots. Path planning is to obtain the optimal motion path of the robot from the initial position to the target position based on a certain performance indicator defined by humans. Most traditional path planning algorithms such as artificial potential field method, ant colony algorithm, and visibility graph method have disadvantages such as low real-time performance and easy to fall into local optimality. Because reinforcement learning algorithms do not require environmental models, they can continuously interact with the environment in unknown environments to find the optimal path, which makes reinforcement learning algorithms more and more used in the field of mobile robot path planning.

[0003] The most widely used reinforcement learning algorithm in the field of mobile robot path planning is the Q-learning algorithm. The Q-learning algorithm is a value-based algorithm in reinforcement learning. For a given state-action pair (s, a), there will be a corresponding value function Q(s, a) corresponding to it. The environment will give a reward r according to the action taken by the agent to update the Q value. The main idea of ​​the algorithm is to construct a Q table of states and actions to store Q values, and then select the action that can obtain the maximum benefit based on the Q value. The iteration of the Q-learning algorithm is a process of trial and error and exploration. The condition for its convergence is to ensure that the agent makes enough attempts for each state-action pair so that the agent can finally learn the optimal strategy. When the Q-learning algorithm is applied to mobile robot path planning, how to reduce invalid exploration, speed up the convergence of the agent, and improve the efficiency of the algorithm is a hot topic in current research. Summary of the invention

[0004] In order to improve the learning speed of the Q-learning algorithm when applied to mobile robot path planning, the present invention proposes a reinforcement learning path planning method that integrates historical information. The previous historical state of the agent is introduced into the state space to reflect the characteristics of environmental changes. Before the agent selects an action, the action space is dynamically adjusted to prohibit the agent from repeatedly returning to the previous position, thereby reducing invalid exploration, improving the exploration efficiency of the algorithm, and shortening the path planning time.

[0005] The present invention provides a reinforcement learning path planning method integrating historical information, comprising the following steps:

[0006] S1: Acquire the environment image and build a grid map;

[0007] S2: Define reinforcement learning state space and action space;

[0008] S3: Initialize algorithm parameters;

[0009] S4: Dynamically adjust the action space;

[0010] S5: In the adjusted action space, the ε-greedy strategy is used to select actions;

[0011] S6: Execute the action and update the Q value;

[0012] S7: Repeat the fourth step, the fifth step, and the sixth step until a certain number of steps or a certain convergence condition is reached;

[0013] S8: Select the action with the largest Q value at each step to obtain the optimal path;

[0014] S9: Send the optimal path to the controller of the mobile robot, and control the mobile robot to walk according to the optimal path.

[0015] Furthermore, the specific operation of step S1 is as follows: an environmental image is obtained based on a camera carried by the mobile robot, and the image is divided into a 20×20 grid, and an environmental model is established using a grid method. If an obstacle is found in the grid, the grid is defined as an obstacle position and the robot cannot pass through it; if a target point is found in the grid, the grid is defined as the target position, which is the position that the mobile robot will eventually reach; other grids are defined as obstacle-free grids and the robot can pass through them.

[0016] Furthermore, the specific operation of step S2 is as follows: define the state space of reinforcement learning as the current position coordinates and the previous position coordinates of the agent, and the action space as actions in the four directions of up, down, left, and right. After each action is executed, the agent moves one grid in the corresponding direction.

[0017] Furthermore, the algorithm parameters in step S3 include a learning rate ɑ∈(0,1), a discount factor γ∈(0,1), a greed factor ε∈(0,1), a maximum number of iterations, and a reward function r; all Q values ​​are initialized to 0, and an action is randomly given, and the action is executed to reach the next state.

[0018] Furthermore, the dynamic adjustment of the action space in step S4 is that if the previous action is up, the action space is adjusted to {up, left, right}; if the previous action is down, the action space is adjusted to {down, left, right}; if the previous action is left, the action space is adjusted to {up, down, left}; if the previous action is right, the action space is adjusted to {up, down, right}.

[0019] Furthermore, the specific operation of step S6 is as follows: execute the action a selected in step S5, arrive at s, obtain the immediate reward R(s, a), update the Q value function, and update the rule as shown in formula (1):

[0020] (1)

[0021] Among them, (s, a) is the current state-action pair, (s ， , a ， ) is the state-action pair at the next moment, and R(s, a) is the immediate reward for performing action a in state s.

[0022] Beneficial effects of the present invention:

[0023] In view of the problems of slow convergence speed and low operating efficiency of the traditional Q-learning algorithm when applied to the path planning problem of mobile robots, the present invention proposes an improved Q-learning algorithm that integrates historical information. The state space is defined as the current position coordinates and the previous position coordinates of the agent to reflect the changing characteristics of the position. Before the agent selects an action, the action space is dynamically adjusted to prohibit the agent from repeatedly returning to the previous position, thereby reducing invalid exploration. The improved algorithm is applied to the path planning problem of mobile robots, and the results prove the feasibility of the algorithm. The simulation results based on the grid map show that when the improved Q-learning algorithm is applied to the path planning problem of mobile robots, the planning time is shortened by 25.93%, and the number of running steps before convergence is reduced by 26.29%, thereby improving the efficiency of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 The figure is a schematic diagram of the overall process of the method of the present invention.

[0026] Figure 2 A grid map is run for the mobile robot according to an embodiment of the present invention.

[0027] Figure 3 This is a diagram of the convergence of traditional Q-learning.

[0028] Figure 4 This is a diagram showing the convergence of improved Q-learning according to an embodiment of the present invention.

[0029] Figure 5This is the optimal path diagram planned by the improved Q-learning according to the embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0031] See also Figure 1 The present invention provides a reinforcement learning path planning method integrating historical information, and the method steps are as follows:

[0032] Step 1: Obtain an environmental image based on the camera carried by the mobile robot, and divide the image into a 20×20 grid. Use the grid method to establish an environmental model. The grid map in this embodiment is as follows: Figure 2 As shown in the figure, if an obstacle is found in the grid, the grid is defined as the obstacle position and the robot cannot pass through it; if a target point is found in the grid, the grid is defined as the target position, which is the position that the mobile robot will eventually reach; other grids are defined as grids without obstacles and the robot can pass through them.

[0033] Step 2: Define the state space of reinforcement learning as the current position coordinates and the previous position coordinates of the agent, and the action space as the actions in the four directions of up, down, left, and right. After each action is performed, the agent moves one grid in the corresponding direction.

[0034] Step 3: Initialize algorithm parameters

[0035] Including: learning rate ɑ∈(0,1), discount factor γ∈(0,1), greed factor ε∈(0,1), maximum number of iterations, reward function r. Initialize all Q values ​​to 0, and randomly give an action to execute to reach the next state.

[0036] In this example, the learning rate ɑ = 0.01, the discount factor γ = 0.9, the greed factor ε = 0.2, the maximum number of iterations is set to 3000, and the reward function is set to:

[0037]

[0038] Step 4: Dynamically adjust the action space

[0039] If the previous action was up, the action space is adjusted to {up, left, right}; if the previous action was down, the action space is adjusted to {down, left, right}; if the previous action was left, the action space is adjusted to {up, down, left}; if the previous action was right, the action space is adjusted to {up, down, right}.

[0040] Step 5: In the adjusted action space, use the ε-greedy strategy to select actions to ensure that the agent does not repeatedly return to the previous position and reduce invalid exploration.

[0041] Step 6: Execute the action a selected in step 3 ， Arrivals ， , get the immediate reward R(s, a), update the Q value function, and the update rule is as shown in formula (1).

[0042] (1)

[0043] Among them, (s, a) is the current state-action pair, (s ， , a ， ) is the state-action pair at the next moment, and R(s, a) is the immediate reward for performing action a in state s.

[0044] Repeat steps 4, 5, and 6 until a certain number of steps or a certain convergence condition is reached.

[0045] Step 7: Select the action with the largest Q value at each step to get the optimal path.

[0046] Step 8: Send the optimal path to the controller of the mobile robot and control the mobile robot to walk according to the optimal path.

[0047] In this embodiment, we use the above method and the above parameter settings to obtain the optimal path as follows: Figure 5 shown.

[0048] Figure 4 This is a diagram showing the convergence of the improved Q-learning of the embodiment of the present invention. Figure 3 Compared with the traditional Q-learning convergence diagram, the reinforcement learning path planning method integrating historical information in this embodiment shortens the convergence time by 25.93%, reduces the number of iterations by 26.29%, and improves the efficiency of the algorithm.

[0049] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

Claims

1. A reinforcement learning path planning method integrating historical information, characterized in that: The following steps are involved: S1: Acquire the environment image and build a grid map; S2: Define reinforcement learning state space and action space; S3: Initialize algorithm parameters; S4: Dynamically adjust the action space; S5: In the adjusted action space, the ε-greedy strategy is used to select actions; S6: Execute the action and update the Q value; S7: Repeat the fourth step, the fifth step, and the sixth step until a certain number of steps or a certain convergence condition is reached; S8: Select the action with the largest Q value at each step to obtain the optimal path; S9: Send the optimal path to the controller of the mobile robot, and control the mobile robot to walk according to the optimal path; The specific operation of step S1 is as follows: based on the camera carried by the mobile robot, an environmental image is obtained, and the image is divided into a 20×20 grid, and an environmental model is established using a grid method. If an obstacle is found in the grid, the grid is defined as an obstacle position and the robot cannot pass through it; if a target point is found in the grid, the grid is defined as the target position, which is the position that the mobile robot will eventually reach; other grids are defined as grids without obstacles and the robot can pass through them; The specific operation of step S2 is as follows: define the state space of reinforcement learning as the current position coordinates and the previous position coordinates of the agent, and the action space as the actions in the four directions of up, down, left, and right. After each action is performed, the agent moves one grid in the corresponding direction; The dynamic adjustment of the action space in step S4 is: if the previous action is up, the action space is adjusted to {up, left, right}; if the previous action is down, the action space is adjusted to {down, left, right}; if the previous action is left, the action space is adjusted to {up, down, left}; if the previous action is right, the action space is adjusted to {up, down, right}.

2. The reinforcement learning path planning method integrating historical information according to claim 1 is characterized in that: The algorithm parameters in step S3 include learning rate ɑ∈(0,1), discount factor γ∈(0,1), greed factor ε∈(0,1), maximum number of iterations, and reward function r; Initialize all Q values ​​to 0, randomly assign an action, and execute the action to reach the next state.

3. The reinforcement learning path planning method integrating historical information according to claim 1 is characterized in that: The specific operation of step S6 is as follows: execute the action a selected in step S5, arrive at s, obtain the immediate reward R(s, a), update the Q value function, and update the rule as formula (1): （1） Among them, (s, a) is the current state-action pair, (s ， , a ， ) is the state-action pair at the next moment, and R(s, a) is the immediate reward for performing action a in state s.

4. The reinforcement learning path planning method integrating historical information according to claim 1 is characterized in that: The learning rate ɑ = 0.01, the discount factor γ = 0.9, the greed factor ε = 0.2, the maximum number of iterations is set to 3000, and the reward function is set to: .

Images