An Automatic Path Planning Method Based on Improved Q-learning Algorithm

Abstract

The present invention provides an automatic path planning method based on an improved Q - learning algorithm. The method includes: obtaining a grid map of a target area, specifying a starting node and an ending node in the grid map, and calculating an iterative critical value of the grid map according to the scale of the grid map and the proportion of obstacle nodes in the grid map; for the current node that is neither an obstacle nor an ending node, initializing the Q - value table corresponding to the grid map according to the distance relationship between the current node and the ending node; the mobile robot selects and executes an action using the ε - acc - increasing strategy according to the current state; dynamically updating the Q - value table using the Q - value update rule to reduce the Q - value of the nodes that are far from the ending node; repeating the above two steps to train the mobile robot until the stopping condition is reached; analyzing the current Q - value table, and for each state, selecting the action with the highest Q - value as the optimal action, thereby forming the final path planning strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to an automatic path planning method based on an improved Q-learning algorithm. Background Art

[0002] The application of intelligent robots has a wide range of application prospects. Therefore, the algorithms that serve as the "brains" of intelligent robots are particularly important, and among them, path planning algorithms are of utmost importance.

[0003] Currently, path planning algorithms are divided into three categories: traditional path planning algorithms, sampling-based path planning algorithms, and intelligent bionic algorithms. Traditional path planning algorithms mainly include the A* algorithm, Dijkstra algorithm, D* algorithm, and artificial potential field method. Traditional path planning algorithms are not applicable to high-dimensional spaces and dynamic environments. In the current complex and changing application environments, their inadaptability becomes prominent; Sampling-based path planning algorithms include the PRM algorithm and RRT algorithm. This type of algorithm can well adapt to multi-dimensional spaces, but the smoothness of the optimal path planned is insufficient and it still cannot be used in dynamic environments, with insufficient practicality; Intelligent bionic path planning algorithms include neural network algorithms, ant colony algorithms, genetic algorithms, etc. In comparison, the reinforcement learning algorithm in this type of algorithm makes full use of the perception ability of deep learning and the decision-making ability of reinforcement learning, and continuously iterates through the interaction process between the robot and the environment. Through the evaluative feedback of the environment, it realizes more intelligent decision-making control of the system. However, the path planning efficiency of this type of algorithm in unknown environments is low. Summary of the Invention

[0004] In the case of relatively complex and changeable application environments, in order to improve the path planning efficiency of intelligent robots, the present invention provides an automatic path planning method based on an improved Q-learning algorithm.

[0005] An automatic path planning method based on an improved Q-learnming algorithm provided by the present invention includes:

[0006] Step 1: Obtain the grid map of the target area, specify the starting node and the ending node in the grid map, and calculate the iterative critical value of the grid map according to the scale of the grid map and the proportion of obstacle nodes in the grid map;

[0007] Step 2: For the current node that is neither an obstacle nor an ending node, initialize the Q-value table corresponding to the grid map according to the distance relationship between the current node and the ending node; the Q-value table is used to store the Q-values of state-action pairs; among them, a grid node in the grid map corresponds to a state of the mobile robot, and the movement from one grid node to another grid node corresponds to an action of the mobile robot;

[0008] Step 3: The mobile robot selects and executes an action according to its current state using the ε-acc-increasing strategy; the ε-acc-increasing strategy is used to increase the exploration factor ε as the number of iterations increases when the current number of iterations is less than the iteration critical value, and control the change rate of the exploration factor ε to be fast first and then slow;

[0009] Step 4: Dynamically update the Q-value table using the Q-value update rule to reduce the Q-values of nodes farther from the end node and increase the Q-values of nodes closer to the end node;

[0010] Step 5: Repeat Steps 3 to 4 to train the mobile robot until the stop condition is reached;

[0011] Step 6: Analyze the current Q-value table, and for each state, select the action with the highest Q-value as the optimal action, thereby forming the final path planning strategy.

[0012] Further, calculate the iteration critical value T of the grid map according to the following formula:

[0013] T = 1.0179 * s 1.411 + 39.85 * r 0.614 + 30.144

[0014] where s represents the size of the grid map, and r represents the proportion of obstacle nodes in the grid map.

[0015] Further, in Step 2, initialize the Q-value table corresponding to the grid map according to the distance relationship between the current node and the end node, specifically including:

[0016]

[0017] where C1 represents a preset constant, q i represents the reward value of the current node i, D i represents the distance between the current node i and the end node, d i represents the distance between the current node i and the diagonal, and s represents the size of the grid map.

[0018] Further, in Step 2, the ε-acc-increasing strategy is expressed as:

[0019]

[0020] where t represents the current number of iterations, and T represents the iteration critical value of the grid map.

[0021] Further, in Step 4, the Q-value update rule specifically includes:

[0022]

[0023] Among them, both C2 and C3 represent preset constants, and D represents the distance between the starting node and the ending node.

[0024] Furthermore, in step 2, before initializing the Q-value table corresponding to the grid map, it further includes:

[0025] For obstacle nodes, non-obstacle nodes with a distance less than the inflation distance from the obstacle nodes are set as obstacle nodes; among them, the inflation distance E(V i ) is calculated according to the following formula:

[0026]

[0027] Among them, r represents the radius when the mobile robot is equivalent to a cylinder, V r represents the cruising speed, V i is the current speed of the mobile robot, and V max is the maximum traveling speed.

[0028] Advantages of the present invention:

[0029] (1) By initializing the Q-value table corresponding to the grid map according to the distance relationship between the nodes and the ending node, when the path planning method is in the learning stage, each grid node already has a relatively reasonable initial Q value. In this way, during the learning process, it is no longer completely random, but can move forward along both the direction with the maximum reward value and the direction closest to the target node at the same time, enhancing the directivity of the Q-value table and solving the problem of random paths.

[0030] (2) According to the scale of the grid map and the proportion of obstacle nodes in the grid map, calculate the iterative critical value of the grid map to achieve adaptability to different maps, which has universality; at the same time, design the ε-acc-increasing strategy. When the current iteration number is less than the iterative critical value, make the exploration factor ε increase with the increase of the iteration number, and control the change rate of the exploration factor ε to be fast first and then slow, so that it is fully random in the exploration stage, fully update the Q-value table, and fully utilize it in the stage close to exploitation, no longer randomly select, speed up the time of each iteration, and improve the efficiency of the path planning method.

[0031] (3) During the iterative training process, a dynamic Q-value update rule is designed to increase the Q-values of the nodes closer to the end node and decrease the Q-values of the nodes farther from the end node, thereby guiding the next action of the mobile robot to always move towards the end node, that is, planning the shortest and most reasonable path; at the same time, an 8-direction omnidirectional search for path nodes is realized, which not only solves the right-angle turning problem of path planning by 4-direction search, but also solves the problems of disorder and jumping points in 8-direction search.

[0032] (4) The concept of expansion distance is introduced, and a "collision buffer zone" is preset between the obstacle and the planned path of the mobile robot to enhance the obstacle avoidance performance of the mobile robot. Brief Description of the Drawings

[0033] Figure 1 It is a flowchart of the automatic path planning method based on the improved Q-learnming algorithm provided by the embodiment of the present invention;

[0034] Figure 2 It is a diagram of "jumping points" and "turning back points" provided by the embodiment of the present invention;

[0035] Figure 3 It is a block diagram of the optimized greedy strategy provided by the embodiment of the present invention;

[0036] Figure 4 It is a diagram of the reward function assignment and path selection before and after optimization provided by the embodiment of the present invention;

[0037] Figure 5 It is a schematic diagram of the expansion distance provided by the embodiment of the present invention. Detailed Embodiment

[0038] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0039] Embodiment 1

[0040] As Figure 1 shown, the embodiment of the present invention provides an automatic path planning method based on an improved Q-learnming algorithm, including the following steps:

[0041] S101: Obtain the grid map of the target area, specify the starting node and the ending node in the grid map, and calculate the iterative critical value of the grid map according to the scale of the grid map and the proportion of the obstacle nodes in the grid map;

[0042] Specifically, by automatically obtaining the critical value of the reinforcement learning iteration times, adaptability to different maps is achieved. Thus, when selecting actions according to the ε-acc-increasing strategy subsequently, the Q-value table can be fully updated, while the time for each iteration is accelerated and the efficiency is improved.

[0043] S102: For the current node that is neither an obstacle nor an end node, initialize the Q-value table corresponding to the grid map according to the distance relationship between the current node and the end node; the Q-value table is used to store the Q-values of state-action pairs; wherein, a grid node in the grid map corresponds to a state of the mobile robot, and the movement from one grid node to another corresponds to an action of the mobile robot.

[0044] Specifically, in the reward function of the traditional Q-learning algorithm (as shown in formula (1)), when reaching the end point, a reward value C1 greater than zero is set, and when encountering an obstacle, a reward value C1 less than zero is set. For other nodes, regardless of the distance relationship between the node and the end point, the reward value is set to 0. From this reward function, it can be seen that the reward function of the traditional Q-learning algorithm has two problems: 1. The reward value has no directivity, which will present problems such as unregulated search of the entire map and jumping points in the final path selection. 2. It can only perform 4-direction search on the map, and the search effect is not good. The planned path shows problems such as multiple right-angle turns. If 8-direction search is carried out, due to the lack of directivity of the reward function, more obvious randomness will occur when updating the Q table according to the reward value during the learning process, and thus an effective path cannot be planned.

[0045]

[0046] To solve the above problems, the embodiment of the present invention improves the strategy of only assigning values to obstacles and target nodes in the reward function of the traditional Q-learning algorithm, introduces the distance relationship between the current node of the path and the target node, and initializes and assigns the reward value of the current node according to the distance relationship between the current node and the end node (the closer the distance, the greater the reward value). In this way, during the learning stage, each grid node already has a relatively reasonable initial Q value, so that during the learning process, it is no longer completely random, but can move forward along the direction with the largest reward value and the direction closest to the target node at the same time. Therefore, it already has a certain degree of directivity initially.

[0047] As an implementable manner, initializing the Q-value table corresponding to the grid map according to the distance relationship between the current node and the end node specifically includes:

[0048]

[0049] Among them, C1 represents a preset constant, and q i represents the reward value of the current node i, and D i represents the distance between the current node i and the end node, and d i represents the distance between the current node i and the diagonal line, and s represents the scale size of the grid map.

[0050] Specifically, in this embodiment, the Euclidean distance is used to calculate the distance between nodes. For example, assume that the coordinates of the current node are (x i , y i ), the coordinates of the end point are (x a , y a ), then Furthermore, according to the Pythagorean theorem, the distance between the current node and the diagonal line can be obtained

[0051] S103: The mobile robot selects and executes an action according to the current state using the ε-acc-increasing strategy. The ε-acc-increasing strategy is used to make the exploration factor ε increase with the increase of the current iteration number when the current iteration number is less than the iteration critical value, and control the change rate of the exploration factor ε to be fast first and then slow;

[0052] Specifically, the "exploration-exploitation" mechanism adopted by the traditional Q-learning algorithm is the ε-greedy strategy. The principle of the ε-greedy strategy is as follows: In the initialization process of the Q-learning algorithm, a random exploration factor ε is set, where 0 < ε ≤ 1. When the algorithm selects an action in the exploration stage, a random number between 0 and 1 is randomly generated and judged. If the random number is greater than ε, a random action is selected. If the random number is less than ε, the action that can currently obtain the maximum reward is selected; after multiple iterations, ε is set to 1, and it is in the full exploitation stage, that is, always select the action that can currently obtain the maximum reward until the iteration ends. The ε-greedy strategy can be expressed as formula (3):

[0053]

[0054] Among them, s represents the exploration factor, t represents the iteration number, and T represents the iteration critical value preset according to experience or subjective will.

[0055] From the above description, we can see that the ε-greedy strategy of the traditional Q-learning algorithm has the following two problems: First, ε is completely randomly generated during the initialization process, which is not necessarily a good choice for the algorithm itself. If the value selected at the beginning is very large, the exploration stage loses its meaning, the Q table cannot be updated, and the final learning effect is not obvious, resulting in a certain degree of uncertainty in the final generated Q table, and then "jump points" and "turnaround points" appear in the path planning process. "Jump points" and "turnaround points" are like Figure 2 As shown. Second, T is set based on experience or subjective will. The selection of this value has great limitations and cannot be effectively applied to unfamiliar physical environments and different environments. The selection of this value is not universal. In real path planning task scenarios, the physical environment is ever-changing, which leads to unsatisfactory application of the algorithm in different environments.

[0056] In view of the above problems, the embodiment of the present invention improves the ε-greedy strategy and proposes a new greedy strategy: the ε-acc-increasing greedy strategy. The purpose of the ε-acc-increasing greedy strategy is to increase the exploration factor ε as the number of iterations increases during the exploration phase (i.e., when the current number of iterations is less than the iteration critical value), and to control the change rate of the exploration factor ε to be first fast and then slow. In the exploration phase, the exploration factor ε is made to increase slowly from small to large, so that it is fully random in the exploration phase, the Q value table is fully updated, and it is fully utilized in the phase close to utilization, no longer randomly selected, thereby improving efficiency and speeding up each iteration time.

[0057] Furthermore, in the embodiment of the present invention, the iteration critical value is no longer determined by relying on experience or subjective will. Instead, the iteration critical value for the greedy strategy to reach the utilization stage in different map environments is determined according to the size of the grid map and the proportion of obstacles. In this way, before reaching the iteration critical value, the greedy factor will be dynamically adjusted to enable the mobile robot to fully explore and learn. After reaching the iteration critical value, the greedy factor will be set to 1 and the utilization stage will be entered. Random exploration will no longer waste time, which is more scientific and efficient.

[0058] S104: dynamically updating the Q value table using a Q value update rule to reduce the Q value of a node that is far from the end node and increase the Q value of a node that is close to the end node;

[0059] Specifically, "dynamic adjustment" aims to adjust the Q value according to the next action of the mobile robot, that is, the distance relationship between the node to be visited and the terminal node. If it is close to the terminal node, the reward value is assigned a positive reward value based on the initialization Q value or the last assignment, and the closer the distance, the greater the reward value; otherwise, the reward value is negative, and the farther the distance, the smaller the reward value.

[0060] S105: Repeat steps S103 to S104 to train the mobile robot until the stop condition is reached;

[0061] S106: Analyze the current Q-value table. For each state, select the action with the highest Q-value as the optimal action, thereby forming the final path planning strategy.

[0062] Embodiment 2

[0063] Based on the above Embodiment 1, the embodiments of the present invention adopt methods such as experiments, controlling variables, and curve fitting. As Figure 3 shown, through the engineering mode of "experiment → fitting → verification", an ε-acc-increasing greedy strategy is designed as follows:

[0064] S201: Obtain the iterative critical value of a specific map.

[0065] Through experiments, collect the time of each iteration of the algorithm in a specific map scale scenario (here, a 10*10 map scale and 40% obstacle ratio are adopted). Perform curve fitting on the collected data to obtain the functional relationship between the algorithm iteration time and the iteration number as (4):

[0066]

[0067] Its derivative function is (5):

[0068]

[0069] Finally, calculate that f`(x) is equal to 0 to obtain the iterative critical value of the specific map.

[0070] S202: Fix the same proportion of obstacles in different maps, change the map scale, and obtain the relationship between the map scale and the critical value. Use the same method to obtain the iteration number when the algorithm iteration time reaches stability in other scale map scenarios through experiments. A total of 9 groups of data are collected. Perform curve fitting on the collected data to obtain the functional relationship (3-13) between the map size s and the iterative critical value:

[0071] f(s) = 1.131*s1411 + 25.15(6)

[0072] S203: Fix the map scale, change the proportion of obstacles in the obstacle map, and obtain the relationship between the obstacle ratio and the critical value. Use the same method to obtain the iteration number when the algorithm iteration time reaches stability in other obstacle ratio scenarios through experiments. A total of 9 groups of data are collected. Perform curve fitting on the collected data to obtain the functional relationship (7) between the map obstacle ratio r and the iterative critical value:

[0073] f(r) = 398.5 * r 0.614 + 75.09 (7)

[0074] S204: As introduced above, the critical value of the number of iterations is related to two factors, namely the map scale and the proportion of obstacles in the map. Therefore, it is necessary to weight formulas (6) and (7) to obtain an automated formula for calculating the critical value. During the experiment, it was found that the influence of the map scale on the critical value is much greater than that of the proportion of obstacles. Therefore, a greater weight should be assigned to formula (6). Considering all the above factors, the formula for calculating the critical value is (8):

[0075] T = n * f(s) + (1 - n) * f(r) (8)

[0076] Substituting the function into it, we get formula (9):

[0077] T = 1.0179 * s 1.411 + 39.85 * r 0.614 + 30.144 (9)

[0078] S205: Obtain the ε-acc-increasing "exploration-exploitation" mechanism function. After obtaining the critical value of any map, we need to obtain a functional relationship in which the exploration factor ε changes as the number of iterations t increases. Considering the purpose that the functional relationship needs to achieve as mentioned in Embodiment 1, the following two restrictions that the functional relationship should have are given in this embodiment:

[0079] First, the change range of the exploration factor ε is between 0.5 and 1. This value range is selected according to the characteristics of the Q-learning algorithm. The Q-learning algorithm is more greedy and bold during implementation, and each action selection aims at maximizing the reward value. Therefore, setting the change range of ε between 0.5 and 1 can not only take into account the random probability effect in the initial exploration stage but also fully reflect the bold and greedy characteristics of the Q-learning algorithm.

[0080] Second, the change rate of the exploration factor ε when it changes from 0.5 to 1 is fast first and then slow. This is because in the initial exploration stage, since the environment is completely unfamiliar, relatively rich experience is obtained after one iteration, so the change rate of ε is fast. However, as the number of iterations increases, the learned experience gradually stabilizes and gradually approaches 1, so the change rate of ε is slow. Finally, when reaching the iteration critical value, the exploration factor is set to 1.

[0081] According to the above conditional restrictions, this embodiment intends to use a logarithmic function for fitting and uses the maple software for fitting. Finally, the function of the exploration factor ε changing with t is (10):

[0082]

[0083] In summary, the formula for the ε-acc-increasing greedy strategy proposed in this embodiment is (11):

[0084]

[0085] S206: To verify the correctness of the critical value theory, verification is carried out in a map scenario where experimental data has not been collected. The theoretical value of the critical value is calculated through theory, and then the true critical value is obtained through experiments in this map. By comparing the two, the correctness of the theory can be verified. The results show that the theory for automatic acquisition of the critical value designed in this embodiment is correct and effective.

[0086] Embodiment 3

[0087] Based on the above embodiments, an embodiment of the present invention provides a Q-value update rule, and "dynamic adjustment" is started for each node during each iterative learning process. The principle of "dynamic adjustment" is as follows: In each iterative learning process of the Q-learning algorithm, each node is re-assigned according to the assignment strategy, and this process is repeated. During the learning process of the algorithm, the reward value of each node is dynamically assigned. Each assignment is classified according to the different situations of the state and action of the current node, and the reward value is adjusted. The dynamic assignment strategy in this embodiment is as follows:

[0088] S301: If the action of the mobile robot in the next step makes it closer to the end node and there is no collision, then increase the reward value Q of the current node according to formula (12) i :

[0089]

[0090] S302: If the action of the mobile robot in the next step makes it farther from the end node and there is no collision, then decrease the reward value Q of the current node according to formula (13) i :

[0091]

[0092] S303: In summary, the reward function obtained in the "dynamic adjustment" stage is as shown in (14):

[0093]

[0094] Among them, both C2 and C3 represent preset constants, D represents the distance between the start node and the end node. Here, the Euclidean distance is still used. Assuming the start coordinate is (x 0 , y 0 ), and the end coordinate is (y a,y a ), the distance between the starting point and the ending point is obtained as

[0095] In the above-mentioned Embodiment 1, when initializing the Q-value table, it can be regarded as a reward function with static assignment, while the reward function proposed in the embodiment of the present invention can be regarded as a reward function with dynamic assignment. Through the dual design of the "static" + "dynamic" reward function, a distance factor is introduced into the reward function, making the reward function more reasonable and efficient. The advantages of this reward function are mainly reflected in three aspects: 1. Guide the next action of the mobile robot to always move towards the end node, that is, plan the shortest and most reasonable path; 2. The reward function designed through the distance relationship assigns different reward values to the nodes, thereby guiding the direction of the path and solving the problem of no directivity in the traditional reward mechanism; 3. Realize the 8-direction omnidirectional search of the path nodes, which not only solves the right-angle turning problem of planning the path by 4-direction search, but also solves the problems of disorder and jumping points in 8-direction search. In this embodiment, for a map with a size of 5*5, the assignment and path selection of the reward function before optimization and the optimized reward function are as Figure 4 shown.

[0096] Embodiment 4

[0097] The traditional Q-learning algorithm does not have the feature of obstacle avoidance. The path planned by the algorithm may be close to obstacles. In a complex environment with dense obstacles, the risk of collision increases significantly, posing a huge challenge to the normal movement of the mobile robot. On the basis of the above-mentioned embodiments, in order to help the mobile robot avoid obstacles better, in the automatic path planning method proposed in the embodiment of the present invention, an inflated distance is introduced. The definition of the inflated distance is: set an appropriate distance around the obstacle, and this distance is treated as an obstacle during path planning. All nodes within this distance are not used as nodes for planning the path, and this distance is called the inflated distance.

[0098] The inflated distance serves as a "collision buffer zone" between the robot and the obstacle during the movement of the mobile robot. Through this distance, the risk of collision during the movement of the mobile robot can be effectively reduced, and the obstacle avoidance ability of the path planning method of the present invention is enhanced. The schematic diagram of the inflated distance is as Figure 5 shown.

[0099] The advantages of the inflated distance are not limited to enhancing the obstacle avoidance ability of the path planning method, but also can improve the efficiency of the path planning method. The nodes of the inflated distance, like the obstacle nodes, will not be searched and traversed during the path planning process, that is, in a certain sense, it reduces the scale of the nodes traversed by the algorithm in the map, thereby improving the efficiency of the algorithm, and the more complex the obstacle scenario is, the more obvious the improvement of the algorithm efficiency is.

[0100] The mobile robot realizes the grid-based modeling of the map by adopting SLAM technology. The inflation distance expands outward around the obstacle with the grid as the basic unit, which is the shortest distance allowing the path to approach the obstacle.

[0101] Preferably, for the value of the inflation distance, a grid node can be default selected, that is, the non-obstacle nodes adjacent to the obstacle nodes are also set as obstacle nodes; in addition, the value of the inflation distance can also be automatically adjusted according to different environments.

[0102] When automatically adjusting the value of the inflation distance according to different environments, it is necessary to model the robot. For example, the robot is modeled as a cylinder or a sphere. In this case, it is more appropriate to default the inflation distance to the radius of the cylinder or the sphere, that is, half of the size of the robot. Because when there are obstacles on both sides, it is in line with the best use status of the physical scenario that there is a buffer zone of half of the vehicle body on both the left and right sides of the robot. In this case, neither the effective space is wasted nor the obstacle avoidance effect can be guaranteed.

[0103] Furthermore, in order to make the inflation distance more adaptable to different grid map environments, through analysis, it is known that the value of the inflation distance is related to factors such as the speed of the robot, the size of the robot, and the number of grids. In order to simplify the model, the following assumptions are made for the robot and the obstacle in this embodiment: (1) The robot model is equivalent to a cylinder, and its radius is set as r, and the cruising speed is V r , satisfying V r ≤V max , where V max is the maximum driving speed, which is determined by the performance of the robot, V r is the speed threshold, and V i is the current speed of the robot. When V i ≤V r , the inflation distance is r. If the mapping rule between the robot model and the map is that the radius of the robot is equal to the side length of the grid, at this time, only 1 grid node is expanded, that is, the obstacle node expands one grid outward. If V i ≥V r , the risk of the robot colliding with the obstacle increases. In this case, the inflation distance should increase accordingly, that is, the number of expanded nodes should increase accordingly. (2) The obstacle is equivalent to one or more square grids. The principle of taking the value of the inflation distance is shown in formula (15). The number of expanded nodes is the ratio of the current speed to the speed threshold:

[0104]

[0105] Formula (3-19) shows the relationship between the value of the inflation distance and factors such as the size of the robot and the driving speed. When V i ≤V r, the expansion distance is defaulted to the radius of the cylinder as the expansion distance. When V r <V i , E(V i ) should be increased accordingly, otherwise the risk of robot collision will increase. The value of the expansion distance has a linear relationship with the driving speed V i .

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automatic path planning method based on an improved Q - learning algorithm, characterized in that, it includes: Step 1: Obtain the grid map of the target area, specify the starting node and the ending node in the grid map, and calculate the iterative critical value of the grid map according to the scale of the grid map and the proportion of obstacle nodes in the grid map; Step 2: For the current node that is neither an obstacle nor an ending node, initialize the Q - value table corresponding to the grid map according to the distance relationship between the current node and the ending node; the Q - value table is used to store the Q - values of state - action pairs; among them, a grid node in the grid map corresponds to a state of the mobile robot, and the movement from one grid node to another corresponds to an action of the mobile robot; Step 3: The mobile robot selects an action according to the current state using the ε - acc - increasing strategy and executes it; the ε - acc - increasing strategy is used to make the exploration factor ε increase with the increase of the iteration number when the current iteration number is less than the iterative critical value, and control the change rate of the exploration factor ε to be fast first and then slow; Step 4: Dynamically update the Q - value table using the Q - value update rule to reduce the Q - value of the nodes far from the ending node and increase the Q - value of the nodes close to the ending node; Step 5: Repeat Step 3 to Step 4 to train the mobile robot until the stop condition is reached; Step 6: Analyze the current Q - value table, and for each state, select the action with the highest Q - value as the optimal action, thereby forming the final path planning strategy.

2. The automatic path planning method based on an improved Q - learning algorithm according to claim 1, characterized in that, calculate the iterative critical value T of the grid map according to the following formula: T = 1.0179 * s 1.411 + 39.85 * r 0.614 + 30.144 where s represents the scale of the grid map, and r represents the proportion of obstacle nodes in the grid map.

3. The automatic path planning method based on an improved Q - learning algorithm according to claim 1, characterized in that, in Step 2, initializing the Q - value table corresponding to the grid map according to the distance relationship between the current node and the ending node specifically includes: Among them, C1 represents a preset constant, and q i represents the reward value of the current node i, and D i represents the distance between the current node i and the end node, and d i represents the distance between the current node i and the diagonal line, and s represents the scale of the grid map.

4. The automatic path planning method based on an improved Q - learning algorithm according to claim 1 or 2, characterized in that, in Step 3, the ε - acc - increasing strategy is expressed as: where t represents the current iteration number, and T represents the iterative critical value of the grid map.

5. The automatic path planning method based on an improved Q - learning algorithm according to claim 3, characterized in that, in Step 4, the Q - value update rule specifically includes: where C2 and C3 both represent preset constants, and D represents the distance between the starting node and the ending node.

6. The automatic path planning method based on an improved Q - learning algorithm according to claim 3, characterized in that, in Step 2, before initializing the Q - value table corresponding to the grid map, it further includes: For the obstacle nodes, set the non-obstacle nodes whose distances from the obstacle nodes are less than the dilation distance as obstacle nodes; where the dilation distance E(V i ) is calculated according to the following formula: Among them, r represents the radius when the mobile robot is equivalent to a cylinder, V r represents the cruising speed, V i is the current speed of the mobile robot, V max is the maximum traveling speed.

Images