Novel intelligent path planning method for mobile robot based on Q-Learning

Abstract

The invention discloses a Q-Learning-based novel intelligent path planning method for a mobile robot, and the method comprises the following steps: building a grid map according to a known static two-dimensional space, setting a grid sequence number, and selecting a starting point S and a target point E; the grid map comprises a feasible space and an obstacle; a Q-learning algorithm is adopted to obtain global initial path key nodes and a global initial planning path of the mobile robot, the mobile robot detects position obstacle information according to a sensor module, calculates an obstacle movement track and a robot movement track, and adopts an improved artificial potential field method to perform local dynamic obstacle avoidance. Taking the current position of the robot as a starting point, and taking the nearest neighbor point on the global planning path as a temporary target point for dynamic obstacle avoidance; the robot travels along the planned path and safely reaches the target point, and the real-time obstacle avoidance capability of the mobile robot is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot path planning, and in particular to a novel intelligent path planning method for a mobile robot based on Q-Learning. Background Art

[0002] With the rapid development of new generation information technologies such as artificial intelligence and the Internet of Things, mobile robot technology has shown many new features, and its application areas are becoming more and more extensive. Autonomous path planning technology of mobile robots is the key technology for their autonomous driving and obstacle avoidance, and is also one of the core technologies that enable mobile robots to be widely used.

[0003] In the field of robot path planning, for static obstacle maps, traditional global path planning methods such as the A* algorithm and the Dijkstra algorithm have low computational complexity and are suitable for path planning in simple environments. However, these methods may not meet the task requirements when the automation tasks are more demanding and the environment is more complex. Reinforcement learning has received increasing attention in robot path planning due to its powerful learning ability, fast response speed and high adaptability.

[0004] In the actual working environment, unknown dynamic obstacles greatly increase the difficulty of robot path planning. Faced with dynamic and complex working environments, the common local path planning algorithm, the dynamic window method (DWA algorithm), has natural limitations in foresight, dynamic obstacle avoidance effect, and global optimal path selection, and cannot obtain the global path planning path in a complex environment. The artificial potential field method uses a virtual potential field to simulate the changes in the robot's surrounding environment, and guides the robot to perform path planning and obstacle avoidance through the gradient information of the potential field. It can better adapt to the changes of unknown obstacles in a dynamic environment, improve the robustness and reliability of path planning, and thus show better performance and effects in complex environments. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a new intelligent path planning method for a mobile robot based on Q-Learning. The training speed of the algorithm is improved by improving the Q-learning algorithm, improving the action selection strategy, and adding a reward and punishment function. In the obstacle avoidance path planning process, dynamic obstacles are detected by sensors, and the improved artificial potential field method is used to effectively avoid unknown obstacles, thereby realizing the intelligent path planning of the mobile robot.

[0006] In order to solve the above technical problems, the present invention proposes a new intelligent path planning method for mobile robots based on Q-Learning. The technical solutions adopted to solve the technical problems are as follows:

[0007] A grid map is established according to a known static two-dimensional space, a grid sequence number is set, and a starting point S and a target point E are selected; the grid map includes feasible space and obstacles;

[0008] The Q-learning algorithm is used to obtain the key nodes of the global initial path of the mobile robot and the global initial planned path.

[0009] Furthermore, each grid in the grid map represents a state, denoted as s t ;

[0010] The total number of rows in the raster map is recorded as cols and the total number of columns is recorded as rows;

[0011] Then the mobile robot has a total of rows*cols states, and the state set S composed of all states is S = {s t |s t =j(i,j),i∈mod(t,cols),j∈floor(t,cols};

[0012] The action set A of the mobile robot is denoted as .

[0013] A={a i ,i=1~8} (15)

[0014] where a 1 、a 2 、a 3 、a 4 、a 5 、a 6 、a 7 、a 8 They represent upward, right, down, left, upper-right, lower-right, upper-left, and lower-left movements respectively.

[0015] Furthermore, in the Q-learning algorithm, the action selection strategy adopted is as follows:

[0016] At each decision point, first generate a random number p, p∈(0,1). If p<ε, then choose the ε-greedy strategy, randomly select an action with probability ε, and select the action with the highest Q value with probability 1-ε;

[0017] If p ≥ ε, the ε-greedy strategy is chosen, and actions are selected according to the Boltzmann strategy probability distribution, where the temperature parameter τ controls the width of the probability

[0018] Furthermore, the reward and penalty function of the Q-learning algorithm is defined as follows:

[0019]

[0020] Where: motion1 means that the mobile robot collides with an obstacle or exceeds the boundary of the environment, and the reward value is -100; motion2 means that the robot moves one grid in the diagonal direction, and the reward value is motion3 means the robot moves one grid, and the reward value is 1; motion4 means the robot visits the state visited in the current round, and the reward value is -5; motion5 means the robot reaches the target point, and the reward value is 20.

[0021] Furthermore, the novel intelligent path planning method for a mobile robot based on Q-Learning of the present invention further includes local path planning, where the local path refers to the path from the current position of the mobile robot to the next key node; the local path planning is specifically:

[0022] Based on the global initial path key nodes, the mobile robot screens out moving obstacles that affect local path planning when moving to the next key node;

[0023] Determine whether the moving obstacle that affects the path planning collides with the moving robot, and if the result of the determination is that a collision occurs, use deceleration, acceleration, braking or an artificial potential field method to avoid the obstacle for the mobile robot;

[0024] The local path planning process is repeated until the mobile robot reaches the target point E.

[0025] Furthermore, based on the global initial path key nodes, the mobile robot screens out the moving obstacles that affect the local path planning in the process of moving to the next key node. The specific steps are as follows:

[0026] Establish a virtual straight line from the current position of the moving robot to the next key node, denoted as L ST ;

[0027] The distance from the moving obstacle to the moving robot is d obs-robot , the distance from the moving obstacle to the virtual straight line is d obs-ST ; When the distance d from the moving obstacle to the moving robot obs-robot Or the distance d from the moving obstacle to the virtual straight line obs-ST When any one of them is less than the respective safety threshold, the obstacle is determined to be a moving obstacle that affects path planning.

[0028] Virtual straight line L ST It is expressed as:

[0029] L ST : y=k st x+b st , x∈(x robot ,xtarget ) (17)

[0030]

[0031] Where: k ST is the slope of the straight line; b ST The straight line L ST Intercept; (x robot ,y robot )、(x target ,y target ) are the position coordinates of the self robot and the target point respectively;

[0032] The distance from the moving obstacle to the moving robot is d obs-robot , the distance from the moving obstacle to the virtual straight line is d obs-ST Respectively expressed as:

[0033]

[0034] Further, it is determined whether the obstacle affecting the path planning collides with the moving robot. If the result of the determination is that a collision occurs, the mobile robot is decelerated, accelerated, braked or uses an artificial potential field method to avoid the obstacle, specifically:

[0035] Establish a ray starting from the current position coordinates of the moving obstacle and pointing to the direction of the obstacle's movement, denoted by L obs ; Let ray L obs With virtual straight line L ST The virtual intersection point is P(x p ,y p ), the virtual intersection point is P(x p ,y p ) is also a virtual collision point;

[0036] Calculate the distance between the moving robot and the moving obstacle from the current position to the virtual collision point P(x p ,y p ) The time required t robot-p ,t obs-p ;

[0037] When t obs-p <t robot-p , indicating that the moving obstacle will reach the virtual intersection point before the mobile robot; calculate the time it takes for the moving obstacle to reach the virtual collision point P(x p ,y p ) when the distance between the moving robot and the moving robot is greater than the safety distance, the result of the judgment is that no collision occurs, and the moving robot will maintain the original state of driving; if the distance is less than the safety distance, the result of the judgment is that a collision occurs, and the moving robot is decelerated, braked or the artificial potential field method is used to avoid obstacles;

[0038] When t obs-p =t robot-p , the result is that a collision occurs, and the mobile robot is decelerated, braked or uses an artificial potential field method to avoid the obstacle;

[0039] If t obs-p >t robot-p , indicating that the moving robot will reach the virtual intersection point before the obstacle. Calculate the time it takes for the moving robot to reach the virtual collision point P(x p ,y p ) when the distance to the obstacle is greater than the safety distance, the result is that no collision occurs, and the mobile robot will maintain its original state; if the distance is less than the safety distance, the result is that a collision occurs, and the mobile robot adopts acceleration or artificial potential field method to avoid obstacles.

[0040] Ray L obs The expression is:

[0041]

[0042] Where: θ obs is the angle between the obstacle’s moving direction and the horizontal axis, (x obs ,y obs ) is the position coordinate of the obstacle.

[0043] Assume that ray L obs With virtual straight line L ST The virtual intersection point is P(x p ,y p ), its horizontal and vertical coordinate expressions are:

[0044]

[0045] Where: (x obs ,y obs ) is the obstacle position coordinate; θ obs is the angle between the obstacle’s moving direction and the horizontal direction;

[0046] The moving robot and the moving obstacle reach the virtual collision point P(x p ,y p ) The time required t robot-p ,t obs-p The expression is:

[0047]

[0048] Where: S robot-P is the distance from the current position of the ego robot to the virtual intersection point P; S obs-Pis the distance from the current position of the obstacle robot to the virtual intersection point P.

[0049] Furthermore, in the artificial potential field method, the constructed gravitational function F att With the repulsion function F rep , the specific form is shown in the following formula.

[0050] F att =-εr 0 n

[0051]

[0052] Where: ε is the positive proportionality coefficient of the gravitational potential field, r 0 is the distance between the current position of the mobile robot and the target point, r ob The distance between the current position of the moving obstacle and the target point, r is the influence range of the moving obstacle, n depends on the number of moving obstacles, n=i-2, i is the number of moving obstacles that affect path planning, and m is an integer.

[0053] Beneficial effects: The present invention is based on the global path planning of the improved Q-learning algorithm, adopts the improved artificial potential field method to avoid unknown obstacles, and improves the real-time obstacle avoidance capability of the mobile robot.

[0054] The present invention comprehensively considers the practical problems of static obstacles and dynamic obstacles in the map environment, improves the action strategy selection mechanism of the Q-learning algorithm, adds a reward and punishment function for global path planning, adopts an improved artificial potential field method to avoid obstacles during the robot's travel process, and completes local path planning. The robot has relatively high practicality and research value when running on actual maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention is further described in detail below with reference to the accompanying drawings.

[0056] Figure 1 It is a block diagram of a new intelligent path planning method for a mobile robot based on Q-Learning provided in an example of the present invention.

[0057] Figure 2 It is a flow chart of a new intelligent path planning method for a mobile robot based on Q-Learning provided in an example of the present invention.

[0058] Figure 3 This is a comparison chart of the global path planning results of the improved Q-learning algorithm and the traditional Q-learning algorithm in the static 20*20 grid map of the present invention.

[0059] Figure 4It is a schematic diagram of lateral dynamic obstacles in the local path planning method described in the present invention.

[0060] Figure 5 It is a schematic diagram of local obstacle avoidance simulation using a local artificial potential field method when the mobile robot of the present invention runs to a certain node and detects an unknown static obstacle on the expected path.

[0061] Figure 6 It is a schematic diagram of local obstacle avoidance simulation using a local artificial potential field method when the mobile robot of the present invention runs to a certain node and detects an unknown dynamic obstacle on the expected path.

[0062] Figure 7 It is a schematic diagram of the action space in the global path of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the accompanying drawings of the present invention to clearly and completely describe and discuss the technical solutions in the embodiments of the present invention. Obviously, what is described here is only a part of the examples of the present invention, not all examples. A new intelligent path planning method for a mobile robot based on Q-Learning of the present invention, such as Figure 1 and 2 As shown, the following steps are included:

[0064] (1): Create a raster map

[0065] A 20*20 two-dimensional view of the real environment is simulated based on the real environment, where obstacles are black squares, feasible spaces are white spaces, the starting point is yellow, and the target point is magenta. The environment in which the mobile robot is located can be abstracted as a tuple (O, E), where O represents the obstacle grid set (o∈O) and E represents the empty space grid set (e∈E). The mobile robot is regarded as a particle, and any given current or next action of the robot can uniquely determine the position l t It belongs to the obstacle grid set or the blank space grid set, as shown in formula (1):

[0066]

[0067] s t represents the state of the mobile robot at time t, a t represents the mobile robot at time t, s t The action taken in the state.

[0068] (2): Global path planning based on improved Q-learning algorithm

[0069] A. Improved Q-learning algorithm

[0070] The present invention uses an improved Q-learning algorithm as a search algorithm to complete the planning task. The Q-learning algorithm uses a Q-value function to predict the long-term reward of taking a specific action in a specific state. The Q value is stored in a Q table. In a grid map environment, the mobile robot has several executable operations in each state. Executing these operations will cause the mobile robot to enter a new state, thereby updating the Q table. Assuming there are m states and n actions, the size of the Q table is m*n.

[0071] The algorithm finds the optimal action that the mobile robot should take in different states by querying the maximum value of the Q value of each row in the Q table. t ,a t ) indicates that the mobile robot is in state s t Take action a t The estimated value of the reward obtained. The Q value update formula is shown in formula (2):

[0072]

[0073] where s t represents the state of the mobile robot at time t, a t Indicates that the mobile robot is at s t The action taken at this time in the state, Q(s t ,a t ) represents the current Q value; R(s t ,a t ) indicates that the robot is in s t Execute a in the state t The reward obtained; γ represents the discount factor; Q(s t+1 ,a t+1 ) represents the Q value in the new state s t+1 Take action a t+1 Estimated value of rewards received, Take action on behalf of the robot t Arrival state s t+1 Possibly the greatest potential future return.

[0074] It should be noted that a is the learning rate, a∈[0,1], which indicates the importance of the experience learned by the mobile robot at the new Q value. When a is 0, it means that the mobile robot has not learned anything, the information comes from the new Q value, and only the old information is assigned; when a is 1, it means that the mobile robot does not care about the old experience, but only pays attention to the newly learned information. If the expression Q(s t ,a t ) combined with a, the Q value update formula can be expressed as formula (3):

[0075]

[0076] (1-a)Q(s t ,a t ) represents the ratio of the old Q value to the new Q value, represents the reward the mobile robot learns from this action.

[0077] The improved Q-learning algorithm introduces a new reward and punishment function based on the Q-learning algorithm, optimizes the action strategy and state action, and accelerates the convergence speed, so that the mobile robot can find a collision-free path faster. The pseudo code of the improved Q-learning algorithm is shown in Table 1.

[0078] Table 1 Q-learning algorithm flow

[0079]

[0080] B. State-Action Representation

[0081] Based on the operating environment, the present invention arbitrarily sets the starting point and target point of the mobile robot. In theory, the mobile robot is regarded as a particle. Each grid of the mobile robot in the operating environment represents a state, which is recorded as s. t , the coordinates are marked as According to the total number of rows (cols) and columns (rows) of the grid map we set, we can get a total of rows*cols states. The state set S composed of all states is Assume that the mobile robot can move freely in the surrounding grids. In most studies, researchers divide the exploration of the mobile robot into four basic discrete actions: up, down, left, and right. In order to improve the exploration ability of the Q-learning algorithm, the present invention adds exploration behavior in the diagonal direction. Therefore, the mobile robot has eight discrete actions: up, right, down, left, upper right, lower right, upper left, and lower left. The unit length of a horizontal movement of one grid is 1, and the step length of a diagonal movement of one grid is about 1.4. Figure 7 As shown. The action set A is denoted as.

[0082] A={a i ,i=1~8} (31)

[0083] where a 1 、a 2 、a 3 、a 4 、a 5 、a 6 、a 7 、a 8 They represent upward, right, down, left, upper-right, lower-right, upper-left, and lower-left movements respectively.

[0084] According to the different actions selected by the mobile robot, the robot's state will change differently. The state changes corresponding to the execution of 8 different actions are shown in Table 2.

[0085] Table 2 State-Action Relationship Table CQ table initialization

[0086] In the initial learning stage, the agent knows very little about the state of the environment and blindly selects actions from the action set. Therefore, using prior knowledge to initialize the Q table is an effective way to reduce the convergence time of the Q-learning algorithm. From the first part, we can see that the size of the Q table is (number of state spaces * actions). A matrix is ​​established to store Q values. The rows represent each state s, and the columns represent the actions a corresponding to each state. The Q value is the reward obtained by performing a certain action in a certain state. The representation of the Q table is shown in formula (5).

[0087]

[0088] The present invention can initialize and optimize the Q table based on known environmental information, assign values ​​to important nodes, and obtain the final Q table for selecting the optimal path through continuous iteration and updating through training.

[0089] D. Action strategy selection

[0090] Reinforcement learning faces the dilemma of exploration-exploitation, so the performance of the Q-learning algorithm is closely related to the selected exploration strategy. The two most common exploration strategies are the ε-greedy strategy and the Boltzmann strategy.

[0091] This paper adopts a new action selection strategy in the improved Q-learning, which combines the ε-greedy and Boltzmann strategies and has the advantages of avoiding local optimality and accelerating convergence. In the initial training stage, this strategy makes the mobile robot more inclined to explore the location environment. When the number of training sets reaches a certain threshold, this strategy makes the robot tend to use available prior knowledge.

[0092] Initially, the mobile robot does not know the state of the environment and must randomly select actions from the action set. Due to this randomness, executing an action may result in the same state as the previous episode, resulting in unnecessary state repetition. The hybrid strategy is to control the trade-off between the ε-greedy strategy and the Boltzmann strategy through the parameter ε. At each decision point, the ε-greedy strategy is selected with probability ε and the Boltzmann strategy is selected with probability 1-ε. Depending on the selected strategy, at each state, the optimal action is selected with a certain probability or the action is selected based on the probability distribution of the Q value. Table 3 gives the pseudo code for combining the ε-greedy strategy and the Boltzmann strategy.

[0093] Table 3 Action selection strategy

[0094]

[0095] E. Reward and Penalty Function

[0096] In the improved Q-learning algorithm, the goal of the mobile robot is to obtain the maximum cumulative reward through interaction with the environment (from the initial state to the target state). In this study, the reward function is designed as shown in formula (6):

[0097]

[0098] Where: motion1 means that the mobile robot collides with an obstacle or exceeds the boundary of the environment, and the reward value is -100; motion2 means that the robot moves one grid in the diagonal direction, and the reward value is motion3 means the robot moves one grid, and the reward value is 1; motion4 means the robot visits the state visited in the current round, and the reward value is -5; motion5 means the robot reaches the target point, and the reward value is 20.

[0099] This paper determines this set of reward function values ​​through integer search and optimizes the performance of the Q-learning algorithm. It is reasonable to set motion1 and motion4 as penalties because they have a negative impact on the completion of the final task. However, considering the scenario that the robot may encounter a dead end and return, the penalty set for motion4 is slightly lower. Motion5 represents a strong positive incentive, indicating the completion of the final task, and therefore obtains the highest reward value, such as Figure 3 As shown in the figure, it is a comparison of the global path planning results of the improved Q-learning algorithm and the traditional Q-learning algorithm in a static 20*20 grid map. Figure 3 The left figure in the middle shows the global path planning result of the traditional Q-learning algorithm. Figure 3 The right figure in the middle shows the global path planning result of the improved Q-learning algorithm.

[0100] (3) Sensor-based detection of unknown environmental obstacles

[0101] A. Obstacle screening

[0102] There may be multiple unknown moving obstacles in an unknown environment. Therefore, it is necessary to detect the obstacles in the environment in advance, filter out the moving obstacles that have no effect on path planning, and then evaluate the risk of the remaining moving obstacles on path planning before proceeding to the next step of planning.

[0103] First, establish a virtual straight line from the ego robot to the path node, denoted as L ST , whose expression is:

[0104] L ST : y=k st x+b st , x∈(x robot ,x target ) (34)

[0105]

[0106] Where: k ST is the slope of the straight line; b ST The straight line L ST Intercept; (x robot ,y robot )、(x target ,y target ) are the position coordinates of the ego robot and the target point respectively.

[0107] Assume the distance from the obstacle to the ego robot is d obs-robot , the distance from the obstacle to the virtual straight line is d obs-ST , when d obs-robot or obs-ST When any item is less than the respective safety threshold, it is determined that the corresponding obstacle will affect the subsequent path planning of the self-robot, thereby screening out these moving obstacles that may affect the path planning. This moving obstacle pre-screening method ignores the impact of moving obstacles at a distance on the desired trajectory of the robot, greatly improving the efficiency of path planning. obs-robot and d obs-ST The expression is:

[0108]

[0109] The corresponding safety thresholds are and Through reasonable settings and Autonomous robots can plan effective obstacle avoidance paths in the shortest possible time. Or When , the corresponding moving obstacle will participate in the subsequent planning algorithm calculation.

[0110] B. Dynamic Obstacle Planning

[0111] According to the moving direction of the moving obstacle, establish a ray starting from the current position coordinates of the moving obstacle and pointing to the moving direction of the moving obstacle, denoted by L obs , according to the movement intention of the moving obstacle, determine whether it is likely to collide with the self-robot. obs The expression is:

[0112]

[0113] Where: θ obs is the angle between the moving direction of the moving obstacle and the horizontal axis, which is a dynamic variable; (x obs ,y obs ) is the position coordinate of the moving obstacle, which is updated at all times as the moving obstacle moves.

[0114] Assume that ray L obs With virtual straight line L st The virtual intersection point is P(x p ,y p ), its horizontal and vertical coordinate expressions are:

[0115]

[0116] Where: k ST and b ST Same as formula (11) (12); (x obs ,y obs ) is the obstacle position coordinate; θ obs is the angle between the obstacle's moving direction and the horizontal direction.

[0117] Based on the virtual collision point P, the time required for the ego robot and the moving obstacle robot to reach the virtual collision point P from the current position is calculated respectively. Assume that the current speed of the ego robot is v robot , the acceleration is a robot , the current speed of the moving obstacle robot is v obs , the acceleration is a obs The time taken for the two to reach point P in their current motion state is t robot-p ,t obs-p , whose expression is:

[0118]

[0119] Where: S robot-P is the distance from the current position of the ego robot to the virtual intersection point P; S obs-P is the distance from the current position of the moving obstacle robot to the virtual intersection point P.

[0120] Compare the time it takes for the ego robot and the obstacle robot to move to point P. If t obs-p <t robot-p , indicating that the obstacle reaches the virtual intersection before the robot. When the moving obstacle robot approaches point P and the distance to the self-robot is greater than the safe distance, the self-robot will maintain the original state. If the distance from the moving obstacle to the self-robot is less than the safe distance, the self-robot needs to slow down or even brake in advance to avoid collision. When the moving obstacle robot leaves point P, the self-robot will resume the original speed and drive towards the target point. If t obs-p =t robot-p , it is necessary to determine what measures should be taken to avoid collision based on the kinematic and dynamic constraints of the self-robot. Set the collision decision priority on the premise of ensuring driving safety, and the deceleration and braking conditions have higher priority than the acceleration and steering conditions. When there is a risk of collision, the self-robot will first take deceleration measures while ensuring a safe distance. If the collision risk is still not reduced after deceleration, braking measures will be taken to avoid moving obstacles. After the obstacle gradually moves away from the self-robot, the self-robot begins to slowly accelerate until it returns to the given speed. If t obs-p >t robot-p , indicating that the self-robot will reach the virtual intersection point faster. If the distance from the obstacle to the self-robot is less than the safe distance, the self-robot will deviate from the straight track under the repulsive force of the obstacle to avoid the dynamic obstacle; secondly, when the self-robot senses the increased risk of collision, it will increase the speed while avoiding it to quickly escape the influence range of the obstacle, further ensuring the safety of path planning. If the distance from the obstacle to the self-robot is greater than the safe distance, the self-robot will maintain the original state until it reaches the target point.

[0121] (4) Local path planning based on improved artificial potential field method

[0122] A. Artificial Potential Field Algorithm

[0123] The artificial potential field method is a commonly used obstacle avoidance path planning method, which was first proposed by Khatib (1986). It is inspired by the concept of potential field in physics. The artificial potential field method has been widely used in the field of obstacle avoidance path planning due to its simplicity, high efficiency and easy implementation. Its core idea is to abstract the robot's environment into a gravitational field and a repulsive field. The target point generates gravitational force on the robot, and the obstacle generates repulsive force on the robot, so that the robot stays away from the obstacle. The resultant force on any point of the robot in the working environment is determined by the vector sum of the repulsive force generated by the obstacle and the gravitational force generated by the target point.

[0124] The gravitational field can attract the robot to the target. The basic principle is that when the robot moves away from the target, the attractive field will gradually increase. The attractive potential function can have many forms, but the most commonly used is the quadratic form. Since the quadratic function has the problem of discontinuity in the target position, the zero point must be considered in the calculation method. Therefore, this paper describes the quadratic potential of the attractive potential function as:

[0125]

[0126] Where: ε is the positive proportionality coefficient of the gravitational potential field; r 0 is the distance between the robot's current position and the target point, a vector whose direction is from the robot to the target point. The gravitational function F can be obtained by differentiating the gravitational potential field function. att , whose expression is

[0127]

[0128] The repulsive force generated by the repulsive field keeps the robot away from obstacles. However, if there is a certain distance threshold between the robot and the obstacle, its movement must be guaranteed not to be affected in any way. The repulsive force function is:

[0129]

[0130] If there are n obstacles in the scene, the total repulsion function is:

[0131]

[0132] Where U repi is the repulsive force field value of each obstacle i, μ is the positive proportional coefficient of the repulsive force function; r ob is a vector of the distance between the robot's current position and the obstacle, with the direction pointing from the robot to the opposite direction of the obstacle; r is the obstacle's influence range. Beyond this distance, the repulsive force of the obstacle has no effect on the robot. When it is less than this threshold, the repulsive force of the obstacle on the robot is affected by the distance between the two. Therefore, the magnitude of the repulsive force generated by the obstacle is the negative gradient of the repulsive function:

[0133]

[0134] The design of the repulsive field takes into account the distance threshold r between the robot and the obstacle. 0 The closer the current position is, the greater the gradient is, and the greater the repulsive force on the robot is. When the distance between the robot’s current position and the obstacle is greater than r 0 When , the robot is no longer repelled by obstacles. So the reasonable formula is:

[0135]

[0136] B. Improved Artificial Potential Field Method

[0137] The present invention proposes an improved artificial potential field method. In view of the problem that the artificial potential field method is prone to fall into trap areas and local minima in local obstacle avoidance planning, the present invention constructs a new type of attraction and repulsion function. The specific form is shown in formula (25).

[0138] F att =-εr 0 n

[0139]

[0140] Where: n depends on the number of obstacles that affect the path planning, n = i-2, i is the number of obstacles that affect the path planning, m is an integer and can be taken arbitrarily; when r ob ≤r, the repulsion function F rep F rep1 and F rep2 The combined force.

[0141] During the movement of the mobile robot, if it encounters an unknown obstacle that hinders its movement, the improved artificial potential field method will be used for local path planning to avoid the temporary obstacles. After completing the obstacle avoidance behavior, the mobile robot will continue to move towards the sub-target point. Repeat steps 3-4 until the mobile robot reaches the target point. When the target point is reached, the algorithm ends.

[0142] (5) Mobile robot path planning is completed

[0143] Through the above steps, the present invention designs a mobile robot intelligent path planning method, such as Figure 6 As shown in the figure, a static two-dimensional grid map is established, and the improved Q-learning algorithm is used for global path planning; the mobile robot detects the position obstacle information according to the sensor module, calculates the obstacle motion trajectory and the robot motion trajectory, and uses the improved artificial potential field method to perform local dynamic obstacle avoidance. The robot's current position is used as the starting point, and the nearest point on the global planning path is used as the temporary target point for dynamic obstacle avoidance; the robot drives along the planned path and reaches the target point safely; as shown in the figure Figure 5 The figure shows a schematic diagram of local obstacle avoidance simulation using a local artificial potential field method when the mobile robot of the present invention runs to a certain node and detects an unknown static obstacle on the expected path.

[0144] In the experimental map, 22 known obstacles, 3 temporary static obstacles and 2 dynamic obstacles are set to prove the practicality of this method. The planning results are shown in Figure 4 shown.

[0145] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A mobile robot path planning method based on Q-Learning, characterized in that: The steps include: A grid map is established according to a known static two-dimensional space, a grid sequence number is set, and a starting point S and a target point E are selected; the grid map includes feasible space and obstacles; The Q-learning algorithm is used to obtain the key nodes of the global initial path of the mobile robot and the global initial planned path.

2. According to claim 1, a mobile robot path planning method based on Q-Learning is characterized in that: Each grid in the grid map represents a state, denoted as s t ; The total number of rows in the raster map is recorded as cols and the total number of columns is recorded as rows; Then the mobile robot has a total of rows*cols states, and the state set S composed of all states is The action set A of the mobile robot is denoted as; A={a i ,i=1~8}(1) Among them, a1, a2, a3, a4, a5, a6, a7, and a8 represent upward, right, down, left, upper-right, lower-right, upper-left, and lower-left actions respectively.

3. According to claim 2, a mobile robot path planning method based on Q-Learning is characterized in that: In the Q-learning algorithm, the action selection strategy adopted is as follows: First, generate a random number p, p∈(0,1). If p<ε, select the ε-greedy strategy, randomly select an action with probability ε, and select the action with the highest Q value with probability 1-ε. If p ≥ ε, the ε-greedy strategy is chosen, then actions are selected according to the Boltzmann strategy probability distribution, where the temperature parameter τ controls the width of the probabilities.

4. According to claim 2, a mobile robot path planning method based on Q-Learning is characterized in that: The reward and penalty function of the Q-learning algorithm is defined as follows: Where: motion1 means that the mobile robot collides with an obstacle or exceeds the boundary of the environment, and the reward value is -100; motion2 means that the robot moves one grid in the diagonal direction, and the reward value is motion3 means the robot moves one grid, and the reward value is 1; motion4 means the robot visits the state visited in the current round, and the reward value is -5; motion5 means the robot reaches the target point, and the reward value is 20.

5. The mobile robot path planning method based on Q-Learning according to claim 1, characterized in that: It also includes local path planning, where the local path refers to the path from the current position of the mobile robot to the next key node; the local path planning is specifically as follows: Based on the global initial path key nodes, the mobile robot screens out moving obstacles that affect local path planning when moving to the next key node; Determine whether the moving obstacle that affects the path planning collides with the moving robot, and if the result of the determination is that a collision occurs, use deceleration, acceleration, braking or an artificial potential field method to avoid the obstacle for the mobile robot; The local path planning process is repeated until the mobile robot reaches the target point E.

6. A mobile robot path planning method based on Q-Learning according to claim 5, characterized in that: Based on the global initial path key nodes, the mobile robot screens out the moving obstacles that affect the local path planning when moving to the next key node. The specific steps are as follows: Establish a virtual straight line from the current position of the moving robot to the next key node, denoted as L ST ; The distance from the moving obstacle to the moving robot is d obs-robot , the distance from the moving obstacle to the virtual straight line is d obs-ST ; When the distance d from the moving obstacle to the moving robot obs-robot Or the distance d from the moving obstacle to the virtual straight line obs-ST When any one of them is less than the respective safety threshold, the obstacle is determined to be a moving obstacle that affects path planning.

7. A mobile robot path planning method based on Q-Learning according to claim 6, characterized in that: Virtual straight line L ST It is expressed as: L ST :y=k ST x+b ST ,x∈(x robot ,x target )(3) Where: k ST is the slope of the straight line; b ST The straight line L ST Intercept; (x robot ,y robot )、(x target ,y target ) are the position coordinates of the self robot and the target point respectively; The distance from the moving obstacle to the moving robot is d obs-robot , the distance from the moving obstacle to the virtual straight line is d obs-ST Respectively expressed as:

8. A mobile robot path planning method based on Q-Learning according to claim 7, characterized in that: Determine whether the obstacle that affects the path planning collides with the moving robot. If the result of the determination is that a collision occurs, the mobile robot is decelerated, accelerated, braked or uses an artificial potential field method to avoid the obstacle, specifically: Establish a ray starting from the current position coordinates of the moving obstacle and pointing to the direction of the obstacle's movement, denoted by L obs ; Let ray L obs With virtual straight line L ST The virtual intersection point is P(x p ,y p ), the virtual intersection point is P(x p ,y p ) is also a virtual collision point; Calculate the movement of the robot and the obstacle from the current position to the virtual collision point P(x p ,y p ) The time required t robot-p ,t obs-p ; When t obs-p <t robot-p , indicating that the moving obstacle will reach the virtual intersection point before the mobile robot; Calculate the moving obstacle to reach the virtual collision point P(x p ,y p ) when the distance between the moving robot and the moving robot is greater than the safety distance, the result of the judgment is that no collision occurs, and the moving robot will maintain the original state of driving; if the distance is less than or equal to the safety distance, the result of the judgment is that a collision occurs, and the mobile robot is decelerated, braked or uses an artificial potential field method to avoid obstacles; When t obs-p =t robot-p , the result is that a collision occurs, and the mobile robot is decelerated, braked or uses an artificial potential field method to avoid the obstacle; If obs-p >t robot-p , indicating that the moving robot will reach the virtual intersection point before the obstacle. Calculate the time it takes for the moving robot to reach the virtual collision point P(x p ,y p ) when the distance to the obstacle is greater than the safety distance, the result of the judgment is that no collision occurs, and the mobile robot will maintain the original state; if the distance is less than or equal to the safety distance, the result of the judgment is that a collision occurs, and the mobile robot adopts acceleration or artificial potential field method to avoid obstacles.

9. A mobile robot path planning method based on Q-Learning according to claim 8, characterized in that: Ray L obs The expression is: Where: θ obs is the angle between the obstacle’s moving direction and the horizontal axis, (x obs ,y obs ) is the position coordinate of the obstacle; Assume that ray L obs With virtual straight line L ST The virtual intersection point is P(x p ,y p ), its horizontal and vertical coordinate expressions are: Where: (x obs ,y obs ) is the obstacle position coordinate; θ obs is the angle between the obstacle’s moving direction and the horizontal direction; The moving robot and the moving obstacle reach the virtual collision point P(x p ,y p ) The time required t robot-p ,t obs-p The expression is: Where: S robot-P is the distance from the current position of the ego robot to the virtual intersection point P; S obs-P is the distance from the current position of the obstacle robot to the virtual intersection point P.

10. A mobile robot path planning method based on Q-Learning according to claim 5 or 7, characterized in that: In the artificial potential field method, the constructed gravitational function F att and the repulsion function F rep , the specific form is shown in formula (25); Where: ε is the proportional coefficient of the gravitational potential field, r0 is the distance between the current position of the mobile robot and the target point, r ob The distance between the current position of the moving obstacle and the target point, r is the range of influence of the moving obstacle, n depends on the number of moving obstacles, n = i-2, i is the number of moving obstacles that affect the path planning, m is an integer, when r ob ≤r, the repulsive force function F rep F rep1 and F rep2 The combined force.

Images