Unknown dynamic environment mobile robot path planning method based on reinforcement learning

Abstract

The invention relates to an unknown dynamic environment mobile robot path planning method based on reinforcement learning, and belongs to the technical field of path planning. According to the method, a two-layer reinforcement learning structure is adopted, a bottom layer module learns global path planning, and an ellipse specific and precise strategy and a GliEF-Dyna-Q algorithm are provided in the early stage; the high-level module learns an obstacle avoidance rule to complete local dynamic obstacle avoidance; according to the GliEF-Dyna-Q algorithm, elliptic geometric knowledge is introduced to design an elliptic knowledge special and precise strategy to narrow the sampling range, a reward mechanism and path search are constrained and optimized through global and local perspectives, and an improved SA-epsilon greedy strategy is fused into Dyna-Q to achieve efficient global path planning. The high-level module provides a brand-new obstacle avoidance behavior reward mechanism and a more adaptive value function updating method so as to carry out comprehensive local obstacle avoidance activities. According to the invention, the robot can more efficiently realize obstacle avoidance and complete optimal path driving in a dynamic environment.

Description

Technical Field

[0001] The invention belongs to the technical field of path planning and relates to a path planning method for a mobile robot in an unknown dynamic environment based on reinforcement learning. Background Art

[0002] Path planning is a research hotspot in the field of mobile robots. Its purpose is to search for an optimal or near-optimal safe and collision-free path from the current point to the destination in an unknown or known environment according to technical requirements (such as shortest distance, highest safety, strong real-time performance, shortest time, etc.). At present, local path planning algorithms mainly include artificial potential field method and dynamic window method, and global path planning algorithms include Dijkstra algorithm, A* algorithm, ant colony algorithm, etc. The above methods have a common constraint: the path planning strategy needs to be manually pre-programmed by the designer, which is difficult and time-consuming. In addition, when facing complex, unstructured or dynamic environments, mobile robots are not adaptive and robust in uncertainty. Reinforcement learning (RL) is a branch of machine learning. Modern RL is goal-oriented learning, which maximizes numerical reward signals through interactive learning with uncertain environments. RL is mainly divided into model-based learning and model-free learning. Mobile robots without prior knowledge of the environment can learn from delayed rewards through trial and error using model-free reinforcement learning algorithms (such as Q-learning and its variants) to obtain the optimal path for planning. However, it is difficult for a number of model-free learning methods such as Q-learning to efficiently achieve optimal path planning in large and complex environments, and the sampling utilization rate of large environments is very low. Model-based reinforcement learning algorithms can improve the sample utilization rate in the environment. By building a virtual environment model, the intelligent agent can conduct trial and error learning in the virtual environment, greatly reducing the sampling of useless data. The Dyna-Q method combines the advantages of the two, which not only improves the sampling efficiency, but also enhances the expected return after convergence. It interacts with the real environment while learning, uses experience storage as the basis for model building, and consolidates learning in the model. However, the Dyna-Q algorithm still has many shortcomings. First, the experience data for building the model comes directly from Q-learning, and Q-learning with excellent strategies is required to quickly build a good model. However, Q-learning has poor search capabilities in complex environments. Secondly, the action selection of indirect learning is too random, and it is difficult to search the model efficiently, which is very time-consuming. These problems are unacceptable for efficient path planning of mobile robots in complex dynamic environments. When planning paths in dynamic environments, some studies still directly run static reinforcement learning algorithms, but the convergence speed is slow. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method for mobile robot path planning in an unknown dynamic environment based on reinforcement learning. The method has two modules. The bottom module solves the problems that the traditional Dyna-Q algorithm is applied to the dynamic path planning of mobile robots in complex scenes, such as large initial search space, high sampling rate of a large number of invalid samples, poor search ability, slow convergence speed, and easy to fall into local optimality when there is a deadlock environment. The Dyna framework that combines the model-based reinforcement learning algorithm and the model-independent reinforcement learning algorithm integrates the Q-learning algorithm and is applied to the path planning of mobile robots according to the hierarchical reinforcement learning idea, so as to find the optimal path and improve the efficiency of path planning. The high-level module provides a new obstacle avoidance behavior reward mechanism, and makes forward-looking improvements to the Q value update method to carry out comprehensive local obstacle avoidance activities.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A reinforcement learning-based path planning method for mobile robots in unknown dynamic environments adopts a two-layer reinforcement learning structure. The bottom module learns global path planning, provides an elliptical specialization strategy in the early stage, and involves an improved Dyna-Q algorithm that optimizes the strategy from both global and local perspectives, referred to as the GliEF-Dyna-Q (Dyna-Q with globaland local perspectives in the elliptical field) algorithm; the high-level module learns obstacle avoidance rules to complete local dynamic obstacle avoidance; the bottom module static global algorithm GliEF-Dyna-Q algorithm introduces shallow rules and elliptical geometry knowledge to complete the initialization of the values ​​in the Q table, and constrains the optimization reward mechanism and path search from both global and local perspectives, and integrates the improved SA-ε greedy strategy into Dyna-Q to achieve efficient global path planning;

[0006] The method of learning global path planning in the bottom-level module specifically includes: first, the intelligent agent with the shallow trial rules is put into the map to find the path, and a rough path is explored according to the similar idea of ​​human exploration of mazes to quickly find a feasible path. While finding the path, the Q value information along the way is updated to speed up the convergence speed of learning; after completing the shallow trial, the feasible path is used as the major axis length of the ellipse, the starting point and the end point are used as the two foci, and the focal length is used as the straight-line distance between the starting point and the end point to construct the ellipse specialization range, so as to narrow the search range, reduce invalid sampling, improve the convergence speed, and gradually narrow the range in subsequent learning; then use the improved SA -ε greedy strategy is used to select actions, which can prevent the occurrence of local optimality; secondly, a reward function that can express state information in both global and local dimensions is constructed, and more useful environmental reward information is fed back to the robot according to the corresponding progress; finally, in the indirect learning link based on the virtual model, the model range is directly narrowed to the elliptical specialization range to start building the environmental model for learning, which greatly accelerates the convergence speed, and quotes the idea of ​​​​global perspective to heuristically control the algorithm to use environmental information more efficiently in the virtual model; the above improvements can enable the robot to plan paths more efficiently in a complex global environment and ensure that the robot finds the global optimal path.

[0007] In a dynamic environment, the robot drives along the global path planned by the underlying Dyna framework. When the sensor detects moving obstacles around, the high-level module is activated. At this time, the robot's behavior is divided into two steps: obstacle avoidance and returning to the optimal path. A new obstacle avoidance behavior reward function is established to enable the robot to quickly return to the optimal path after avoiding obstacles. At the same time, in order to make the strategy more forward-looking, the Q value update method in dynamic situations is improved. The above improvements enable the robot to more efficiently avoid obstacles and complete optimal path driving in a dynamic environment.

[0008] Furthermore, the underlying global path planning method specifically includes the following steps:

[0009] S101: Input environmental map information to obtain initial state and target point information;

[0010] S102: Introduce the tasting rule to explore feasible paths and complete the Q value update;

[0011] S103: Use the feasible path obtained after shallow attempts to build an elliptical specialization range;

[0012] S104: After completing the construction of the specialization range, start the underlying Dyna-Q learning on the initialized Q table;

[0013] S105: After the shallow trial is completed, the improved SA-ε greedy strategy is integrated into Dyna-Q, and the learning of Dyna-Q is started on the updated Q table;

[0014] S106: Setting the end point reward, obstacles and boundaries. The reward values ​​in other states are comprehensively evaluated and scored from global and local perspectives. The goal of obtaining the global optimal path is to accumulate the highest reward, that is, the fewer paths taken, the optimal path.

[0015] S107: Q value update:

[0016]

[0017] Among them, α represents the learning rate of the algorithm, γ is the decay rate, Q(S t ,A t ) is the Q value of the test action in the Q table, R t+1 is the reward value obtained by adjusting the reward function by integrating the global progress and local progress, Q(S t+1 ,a) are all possible Q values ​​in the next state after executing the action;

[0018] S108: Model learning;

[0019] S109: Model learning updates the Q value. After selecting the action according to the global perspective strategy, the Q value of the current state is updated by correcting the formula of S107. Here, the model learning step K is set;

[0020] S110: Determine the end of learning: If the model reaches the end point after learning, determine whether it has cycled to the maximum number of iterations. If it has, it ends; otherwise, continue learning under the Dyna framework. If it has not reached the end point, continue iterative learning.

[0021] Further, in step S102, the tasting rules and rewards are set as follows:

[0022] 1) Give a small positive reward rr1 for each small step. When there are no obstacles around, determine the distance between the agent state s and the target point and choose an action;

[0023] 2) When you have to go back, give the state a large negative reward rr2;

[0024] 3) When reaching the target point, a large positive reward rr3 is given;

[0025] 4) Finally, the state-action value function table is updated based on this.

[0026] Further, in step S103, an ellipse specialization range is constructed according to the ellipse specialization strategy, specifically including: the focal length of the ellipse is 2c, the major axis length is 2a, and the minor axis length is 2b; the focus s is defined as the starting point, the focus g is defined as the end point, and the straight-line distance from the starting point to the end point is P min =2c; at the same time, the major axis of the ellipse is defined as the average path length learned before the current episode, that is, Pnow =2a; the distance from any point on the ellipse to the two foci is equal to the length of the major axis; in connection with path planning, according to the relevant parameters of the ellipse, the following formula is obtained

[0027] d 1 +d 2 =2a=P now

[0028] Among them, d 1 and d 2 are the distances from a point on the ellipse to the two foci respectively; its practical significance is that a path from the starting point, passing through a point on the ellipse to the end point, the length of this path is equal to the average path length P learned before the current episode now ; Therefore, the optimal path must be less than P now , which means that the points on the optimal path must be inside the ellipse;

[0029] P now After finding a feasible path in S102, we start to build the elliptical range. At this time, P now =P 1 , P 1 That is the first path length; after that, when the number of successful paths reaches a certain threshold, the ellipse range will be updated, and the length of the major axis is equal to the average path length learned before the current episode. The threshold θ under this strategy is set as follows

[0030] θ=η k

[0031] Among them, η is the threshold base value, the initial value of k is 0, and k will increase by 1 when the threshold is reached once.

[0032] Further, in step S105, the improved SA-ε greedy strategy is:

[0033]

[0034] Among them, ε k , ε 0 and ε f They represent the ε value, initial ε value and final ε value of the kth scene, respectively, the latter two are set values; β is the proportional factor, N tar is an intermediate set number; the calculation formula of ρ is as follows:

[0035]

[0036] Where N is the total number of episodes.

[0037] Further, in step S106, the reward function formula is as follows:

[0038]

[0039] Among them, r GL (s,a) represents the reward function; r 0 is a large positive value, so that the agent can obtain a good positive feedback when it reaches the target point; r 1 is a negative value with a large absolute value, so that the agent will receive a larger penalty when encountering an obstacle; r gl is the dual-view reward function in general, where χ and λ are the scaling factors of global progress and local progress, and τ is the local view scaling factor; d now represents the Euclidean distance between the current state s and the target state target, d t Represents the next state s′ to the local history state s h The Euclidean distance of

[0040] d now =||s-goal|| 2

[0041] d t =||s′-s h || 2

[0042] Among them, s h represents the hth state before s′, d t It is a local perspective.

[0043] Further, in step S108, model learning specifically includes: after completing the calculation and update of the Q value, storing the state grid and the score of the state in the model table, and when the Q-learning part has learned n steps, the model learning will be carried out; using the global perspective in the Dyna architecture to guide the action taken in the state;

[0044] The evaluation of each state is only related to the global progress function, that is, the Euclidean distance between the next state and the target state in the planning update process: H(s,a)=||s′-goal|| 2 , the algorithm finds the heuristic action h by minimizing the heuristic function H a (s,H):

[0045] Furthermore, the high-level local dynamic obstacle avoidance method specifically includes the following steps:

[0046] S201: The robot drives along the global optimal path planned by the bottom-level module. When the robot drives along the optimal path and the sensor detects an unknown obstacle nearby, the high-level module is called to avoid the obstacle.

[0047] S202: Local dynamic obstacle avoidance mainly consists of two steps. One is to immediately respond to unknown obstacles, and the other is to quickly return to the optimal path line after avoiding the obstacles. For the first step, the robot makes a choice according to the global perspective, that is, the obstacle avoidance rule is selected based on the Euclidean distance from the next state to the target point.

[0048] For the second step, an expert reward function is adopted, which can constrain it to quickly return to the original path after local dynamic obstacle avoidance.

[0049]

[0050] Take actions to avoid obstacles. Give a small negative reward r1 for each new state (a state not on the optimal path) reached, and a larger negative reward r1 + r2 when hitting an obstacle, where r2 < r1 < 0; when the robot reaches a cell on the global guiding path, it will obtain a large positive reward r1 + Nc × r3, r3 > |r1| > 0; N c is the number of cells the robot moves on the global guiding path between the point where it first leaves the path and the point where it rejoins the path.

[0051] S203: Q - value update: Improve the update strategy of the Q - table so that it can learn deeper information. The improved update rule is:

[0052] Q(s,a) = Q(s,a)+α[r + γ(n 1 maxQ′+n 2 maxQ”)-Q]

[0053] where Q” represents the maximum Q - value obtained when the next state S' reaches S” after executing possible actions; Q′ represents the maximum Q - value that can be obtained at the next moment in the current state; where, n 1 +n 2 = 1 and n 1 >n 2 > 0; α represents the learning rate of the algorithm, γ is the decay rate, and r represents the reward value.

[0054] S204: The robot avoids unknown obstacles according to the high - level module and returns to the global optimal path, and continues to drive along this path.

[0055] The beneficial effect of the present invention is that before the underlying module Dyna-Q learning is performed, the present invention first performs a shallow attempt to roughly explore the path and update the Q value at the same time, which can effectively reduce the blind search in the early stage. At the same time, an elliptical specialization range is built according to the explored optional path to initialize the Q table and narrow the search range, so as to reduce useless trials and accelerate the convergence speed. At the same time, this range will gradually narrow as the learning progress increases. In the Q-learning of the Dyna framework, the ε strategy is first replaced with the improved SA-ε strategy, and the exploration intensity of the algorithm is appropriately increased in the early stage of the algorithm to avoid local optimality, and better balance the utilization efficiency between exploration and development. By constructing a dual-perspective reward function that can comprehensively examine the real-time global and local two-dimensional state information to constrain the reward information of the environment feedback, evaluate its current stage behavior, and make it jump out of the current complex environment faster. In the indirect learning process of the Dyna framework, in order to avoid the efficiency loss caused by blind search, the global progress perspective is introduced to select actions, which can more efficiently utilize the model information of the environment. The method has a faster convergence speed, a stronger global search capability and stronger robustness. In the high-level module, a new local obstacle avoidance behavior reward function is established, which enables the robot to complete better obstacle avoidance actions while efficiently returning to the global optimal path. Finally, the Q-value update method is improved to make the strategy more forward-looking.

[0056] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0058] Figure 1 It is an overall scheme diagram of the path planning method of a mobile robot in an unknown dynamic environment based on reinforcement learning of the present invention;

[0059] Figure 2 It is the algorithm flow chart of the bottom layer module of the present invention;

[0060] Figure 3 It is a schematic diagram of the ellipse specialization strategy of the present invention;

[0061] Figure 4 It is a partial schematic diagram of the local progress function of the present invention;

[0062] Figure 5 It is a flow chart of the algorithm of the high-level local obstacle avoidance module of the present invention;

[0063] Figure 6 This is an example diagram of the high-level module reward function of the present invention;

[0064] Figure 7 This is a comparison chart of the results of the bottom-level module ablation experiment path;

[0065] Figure 8 This is a comparison chart of the convergence curves of the bottom-level module ablation experiment results;

[0066] Fig. 9 This is a comparison chart of the path effect of the underlying module and other algorithms on a 30*30 map;

[0067] Fig.10 This is a comparison chart of the path convergence curves of the underlying module and other algorithms on a 30*30 map;

[0068] Fig.11 This is a comparison chart of the reward value convergence of the underlying module and other algorithms on a 30*30 map;

[0069] Fig.12 This is a comparison chart of the path results of the underlying module and other algorithms on a 60*60 deadlock map;

[0070] Fig.13 This is a comparison chart of the convergence curves of the underlying module and other algorithms on a 60*60 deadlock map;

[0071] Fig.14 This is the path planning effect diagram in the dynamic environment of the present invention. DETAILED DESCRIPTION

[0072] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0073] See also Figure 1 to Figure 14The present invention provides a method for path planning of a mobile robot in an unknown dynamic environment based on reinforcement learning. First, a feasible path is found and the Q value is updated by using the shallow trial rules formulated. Then, the initial Q value of the elliptical specialization range is built according to the path and the map exploration range is limited. After completing the initialization of the Q table, the underlying global module part is entered. The real experience is obtained by direct learning with the environment by GLiEF-Q-learning. At this time, the reward is set as a reward function that comprehensively considers the state information of the global and local dimensions. The action is executed and the Q value is updated according to the improved SA-ε strategy. Then, a virtual model is constructed with the sampled real experience to conduct indirect learning for N steps, and a global perspective is introduced to consider action selection during model learning. The purpose of the underlying global module is to provide a reinforcement learning path planning method and system that relies on both global and local perspectives, which can effectively overcome the problems of low search efficiency and sampling rate, accelerate the convergence speed of reinforcement learning, and thus plan the path more quickly and accurately. After finding the global path, the robot starts to drive in a dynamic scene according to the path, and this path is the optimal path by default. When the sensor detects a moving obstacle nearby, the high-level local dynamic obstacle avoidance module is started. The reward rule scores the action according to the formulated special dynamic obstacle avoidance reward function, and updates the state value function according to the more forward-looking Q value update formula. After completing the obstacle avoidance, the robot returns to the optimal path to continue driving. The purpose of the high-level module of this invention is to provide a professional obstacle avoidance and scoring mechanism for reinforcement learning dynamic obstacle avoidance, so that the robot can quickly return to the optimal path to continue to perform the task after completing efficient obstacle avoidance. The overall solution process is shown in Figure 1 .

[0074] like Figure 2 As shown, the underlying global path planning method steps are as follows:

[0075] Step 1: Input the environment map information to obtain the initial state and target point information;

[0076] Step 2: Introduce the tasting rule to explore feasible paths and complete the Q value update;

[0077] Formulate a shallow trial rule for the scanned map information. In the early stage of the algorithm, the lack of Q table will reduce the search efficiency of the algorithm. In order to improve the search efficiency of the early stage of the algorithm, first refer to human experience to formulate exploration behavior rules, and use the behavior rules to conduct shallow trials to quickly find a feasible path, initialize the state-action value function table, and reduce unnecessary calculation costs. The shallow trial rules and reward settings are as follows:

[0078] 1) Give a small positive reward rr1 for each small step. When there are no obstacles around, determine the distance between the agent state s and the target point and choose an action;

[0079] 2) When you have to go back, give the state a large negative reward rr2;

[0080] 3) When reaching the target point, a large positive reward rr3 is given;

[0081] 4) Finally, the state-action value function table is updated based on this.

[0082] Step 3: Use the feasible path obtained after shallow attempts to build an elliptical specialization range. Specialization range combination Figure 3 Explain. Figure 3 In the figure, the focal length of the ellipse is 2c, the major axis length is 2a, and the minor axis length is 2b. Define the focus s as the starting point and the focus g as the end point, then the straight-line distance from the starting point to the end point is P min =2c. At the same time, the major axis of the ellipse is defined as the average path length learned before the current episode, that is, P now =2a. The distance from any point on the ellipse to the two foci is equal to the length of the major axis, that is, Figure 3 The length of the two yellow segments is equal to the major axis length. In connection with path planning, according to the ellipse-related parameters designed according to the present invention, the following formula is obtained:

[0083] d 1 +d 2 =2a=P now (1)

[0084] Among them, d 1 and d 2 are the distances from a point on the ellipse to the two foci. Its practical significance is that the length of a path from the starting point to the end point after passing through a point on the ellipse is equal to the average path length P learned before the current episode. now Therefore, the optimal path must be less than P now , which means that the points on the optimal path must be within the ellipse. Limiting the sampling range of the agent to the ellipse can greatly improve the sampling efficiency and accelerate the convergence speed.

[0085] Next, for P now Since the agent is completely unknown to the environment at the beginning, after finding a feasible path in Step 2, it starts to build the elliptical range. At this time, P now =P 1 , P 1 This is the first path length. After that, when the number of successful paths reaches a certain threshold, the ellipse range will be updated, and the length of the major axis is equal to the average path length learned before the current episode. The threshold θ under this strategy is set as follows

[0086] θ=η k (2)

[0087] Among them, η is the threshold base value, the initial value of k is 0, and k will increase by 1 when the threshold is reached once.

[0088] Step 4: After completing the construction of the specialized range, start the underlying Dyna-Q learning on the initialized Q table.

[0089] Step 5: After the shallow trial is completed, Dyna-Q learning starts on the updated Q table. First, the improved SA-ε selects exploration or development:

[0090]

[0091] Among them, ε k , ε 0 and ε f They represent the ε value, initial ε value and final ε value of the kth scene respectively, and the latter two are set values. β is the proportional factor, N tar For a set of intermediate sets, the calculation formula of ρ is as follows:

[0092]

[0093] Where N is the total number of episodes. As the number of episodes required to complete the task increases, this change can make the ε value change more smoothly with the number of iterations. This strategy guides the agent to exploit the experience it has experienced to maximize the cumulative reward on the one hand, and explore new states to get better experience on the other.

[0094] Step 6: The present invention sets the end point reward to 1000, the obstacles and boundaries to -1000, and the reward values ​​in other states are comprehensively evaluated and scored from the global and local perspectives. The goal of obtaining the global optimal path is to accumulate the highest reward, that is, the fewer paths taken, the optimal path. The global progress is represented by the distance from the current state point to the target point divided by the distance from the starting point to the target point. The local progress function is partially combined with Figure 4 Explain. Figure 4 As shown in the figure, it is assumed that each state on the path is a red star, and each table is a square with a side length of 1 (hidden in the figure). The idea of ​​local progress is to evaluate the behavior of the agent in the current part of the time period by whether it can jump out of the local range within k steps. In this regard, the present invention stipulates that the radius is The circular range is the current local range, and the center of the circle is the state k steps ago, such as Figure 4 If the state jumps out of the circle range after k steps, it will be given extra points, otherwise it will be punished. Finally, a comprehensive evaluation is made by integrating the global progress and the local progress. The global progress guides the robot to get as close to the target as possible, and the local progress encourages the robot to escape from the local trap. In particular, when there are no traps around, the local perspective can also make the robot get closer to the target faster. The GL-reward formula is as follows:

[0095]

[0096] Among them, r 0 is a large positive value (1000) so that the agent gets a good positive feedback when it reaches the target point. 1 is a large negative value (-1000) so that the agent will receive a larger penalty when it encounters an obstacle. gl This is the dual-view reward function in general, where χ and λ are the scaling factors of global progress and local progress, and τ is the local view scaling factor. now represents the Euclidean distance between the current state s and the target state target, d t Represents the next state s′ to the local history state s h The Euclidean distance of , which are expressed as:

[0097] d now =||s-goal|| 2 (6)

[0098] d t =||s′-s h || 2 (7)

[0099] Among them, s h represents the hth state before s′, d t for Figure 4 The local perspective in .

[0100] Step 7: Q value update:

[0101]

[0102] Among them, α represents the learning rate of the algorithm, γ is the decay rate, Q(S t ,A t ) is the Q value of the test action in the Q table, R t+1 is the reward value obtained by adjusting the reward function by integrating the global progress and local progress, Q(S t+1 ,a) are all possible Q values ​​in the next state after executing the action.

[0103] Step 8: Model learning:

[0104] After the calculation and update of the Q value is completed, the state grid and the score of the state are stored in the model table. When the Q-learning part has learned n steps, the model learning will be carried out. The traditional indirect reinforcement learning algorithm randomly extracts states and actions from the model for simulated experience; therefore, there is a certain blindness in the model search process. Considering that in actual operation, the robot can obtain the distance between its surrounding area and the target within a certain range, the present invention uses a global perspective in the Dyna architecture to guide the actions taken in the state to improve planning efficiency. The evaluation of each state is only related to the global progress function, that is, the Euclidean distance between the next state and the target state during the planning update process:

[0105] H(s,a)=||s′-goal|| 2 (9)

[0106] The algorithm finds the heuristic action h by minimizing the heuristic function H a (s,H):

[0107]

[0108] Step 9: Model learning updates the Q value. After selecting the action according to the global perspective strategy, the Q value of the current state is also updated through formula (8). Here, the model learning step K is set.

[0109] Step 10: Determine the end of learning: If the model reaches the end point after learning, determine whether it has cycled to the maximum number of iterations. If it has, it ends; otherwise, continue learning under the Dyna framework. If it has not reached the end point, continue iterative learning.

[0110] High-level local dynamic obstacle avoidance method:

[0111] Step 11: The robot drives along the global optimal path planned by the bottom-level module. When the robot drives on the optimal path and the sensor detects unknown obstacles nearby, it starts to call the high-level module to avoid obstacles.

[0112] Step 12: Local dynamic obstacle avoidance is mainly divided into two steps: one is to respond immediately to unknown obstacles, and the other is to quickly return to the optimal path after avoiding obstacles. For step 1, the robot selects according to the global perspective, that is, the obstacle avoidance rule according to the Euclidean distance from the next state to the target point.

[0113] Step 13: For step 2, the present invention designs an expert reward function that can constrain the vehicle to quickly return to the original path after local dynamic obstacle avoidance:

[0114]

[0115] Take actions to avoid obstacles. A small negative reward r1 is given each time a new state (a state not on the optimal path) is reached. A larger negative reward r1 + r2 is given when an obstacle is encountered, where r2 < r1 < 0. When the robot reaches a cell on the global guiding path, a large positive reward r1 + Nc × r3 is given, where r3 > |r1| > 0. Nc is the number of cells the robot moves on the global guiding path between the point where it first leaves the path and the point where it rejoins the path. Combined with Figure 6 for explanation, Figure 6 In Figure 6 , the green squares are static known obstacles, the yellow ones are detected unknown obstacles, the red square is the robot's position, and the gray path is the default global optimal path. As can be seen from Figure 2, the robot first takes actions to avoid obstacles but leaves the optimal path. At this time, it chooses to continue moving left until it returns to the optimal path in Figure 4. Then the reward function is calculated as R1 = R2 =... = R6 = r1, and R7 = r1 + 7 * r3. This approach does not require the robot to return along the original path after avoiding obstacles, but can continue to explore and return to the optimal path along a shorter path.

[0116] Step14: Q-value update. The traditional Q-learning algorithm has the problem of slow convergence speed. One reason analyzed according to the Q-value update formula is that when updating the Q-value, only the information of the next state is used, that is, only one-step exploration is carried out. When the scale of the environment is large, the number of states formed after processing the environment is extremely large. The learning efficiency of only learning one-step information each time is extremely low, and the number of iterations required to explore the optimal path will also increase, which leads to the slow convergence speed of the algorithm. Therefore, the update strategy of the Q-table is improved to enable it to learn deeper information. The improved update rule is:

[0117] Q(s,a) = Q(s,a) + α[r + γ(n 1 maxQ′ + n 2 maxQ”) - Q] (14)

[0118] where Q” represents the maximum Q-value obtained when the next state S' reaches S” after executing possible actions. Among them, n 1 +n 2 = 1 and n 1 > n 2 > 0.

[0119] Step15: The robot avoids unknown obstacles according to the high-level module and returns to the global optimal path, and continues to drive along this path.

[0120] Figure 7The purple path is the result of the algorithm of the present invention. Obviously, the complete GLiEF-Dyna-Q algorithm has the best convergence effect, and the original Dyna-Q algorithm has the worst effect. Figure 8 From the convergence curve analysis, it can be seen that each improvement of the underlying module algorithm of the present invention has an excellent guiding role in path planning, and the complete GLiEF-Dyna-Q algorithm has the best convergence effect.

[0121] Fig. 9 The path results of the GLiEF-Dyna-Q algorithm, SA-Dyna-Q, APF-Dyna-Q and Dyna_HDP algorithms in a 30*30 ordinary table map are shown in Figure 1. The purple path is the calculated path of the present invention. Intuitively, the purple path is superior to other paths in terms of distance and direction. Furthermore, the step number convergence curves of each algorithm are shown in Figure 2. Fig.10 As shown, they more effectively reveal the differences in algorithm performance. Fig.10 The blue curve in the middle is the convergence curve of the GLiEF-Dyna-Q algorithm. In terms of path length convergence, the GLiEF-Dyna-Q algorithm completes convergence the fastest. Similarly, Fig.11 This is a comparison chart of the reward value convergence. The GLiEF-Dyna-Q algorithm also approaches convergence the fastest. The early oscillation is due to the use of an elliptical field to narrow the sampling range and reduce the sampling rate of useless states. At the same time, due to the drive of the dual-view reward function, it can obtain high rewards faster.

[0122] In order to further highlight the effective response of the underlying module of the present invention to the trap environment, the GLiEF-Dyna-Q algorithm is compared with SA-Dyna-Q and APF-Dyna-Q in a 60*60 trap-containing map. Fig.12 As shown in Figure 1, under the condition of 2000 training times, only the GLiEF-Dyna-Q algorithm of the present invention can find a feasible path in the complex trap map, while the APF-Dyna-Q algorithm cannot find it at all, and the SA-Dyna-Q algorithm falls into the trap and cannot escape. Furthermore, the relevant convergence curve is shown in Figure 1. Fig.13 As shown, it is clearly and intuitively demonstrated that only the GLiEF-Dyna-Q algorithm of the bottom module of the present invention can complete normal convergence and complete the path planning task.

[0123] Finally, experiments were conducted in a dynamic environment to prove that Fig.14As shown in the figure, three unknown static obstacles are on the path of the original path, and their specific locations are in the area framed by the red dotted line in the figure; at the same time, five dynamic obstacles moving along the black arrows are added to the map, represented by orange squares. The green curve is the static planning path of the bottom module, and the purple line is the path after dynamic planning of the high-level module. It can be seen from the figure that the present invention can effectively avoid obstacles. When obstacles approach, it can effectively avoid obstacles and continue to drive according to the global path of the bottom module to finally reach the target point.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for mobile robot path planning in unknown dynamic environment based on reinforcement learning, characterized in that: It adopts a two-layer reinforcement learning structure. The bottom module learns global path planning. In the early stage, it provides an elliptical specialization strategy and an improved Dyna-Q algorithm that optimizes the strategy from both global and local perspectives, referred to as the GliEF-Dyna-Q algorithm. The high-level module learns obstacle avoidance rules to complete local dynamic obstacle avoidance; the low-level module static global algorithm GliEF-Dyna-Q introduces shallow rules and elliptical geometry knowledge to narrow the search range of the algorithm, and constrains the optimization reward mechanism and path search through global and local dimensions, and integrates the improved SA-ε greedy strategy into Dyna-Q; The method of learning global path planning in the bottom-level module specifically includes: firstly, putting the agent with the shallow taste rule into the map to roughly explore the path, quickly find a feasible path, and update the Q value information along the way while finding the path; After completing the shallow trial, the feasible path found is used as the major axis length of the ellipse, the starting point and the end point are used as the two foci, and the focal length is used as the straight-line distance between the starting point and the end point to construct an ellipse specialization range, so as to narrow the search range, reduce invalid sampling and improve the convergence speed, and gradually narrow the range in subsequent learning; then use the improved SA-ε greedy strategy to select actions to prevent the occurrence of local optimality; secondly, construct a reward function that can express both global and local state information, and feedback more useful environmental reward information to the robot according to the corresponding perspective; finally, in the indirect learning link based on the virtual model, directly narrow the model range to the ellipse specialization range to start building the environmental model for learning, and quote the idea of ​​global perspective to heuristically control the algorithm to efficiently use environmental information in the virtual model; In a dynamic environment, the robot drives along the global path planned by the underlying Dyna framework. When the sensor detects moving obstacles around, the high-level module is activated. At this time, the robot's behavior is divided into two steps: obstacle avoidance and returning to the optimal path. A new reward function is designed based on this, so that the robot can quickly return to the optimal path after avoiding obstacles. At the same time, the way to update the Q value in dynamic conditions is improved.

2. The mobile robot path planning method according to claim 1, characterized in that: The underlying global path planning method specifically includes the following steps: S101: Input environmental map information to obtain initial state and target point information; S102: Introduce the tasting rule to explore feasible paths and complete the Q value update; S103: Use the feasible path obtained after a brief trial to build an elliptical specialization range; S104: After completing the construction of the elliptical specialization range, start the underlying Dyna-Q learning on the initialized Q table; S105: After the shallow trial is completed, the improved SA-ε greedy strategy is integrated into Dyna-Q, and the learning of Dyna-Q is started on the updated Q table; S106: Setting the end point reward, obstacles and boundaries. The reward values ​​in other states are comprehensively evaluated and scored by the dual-view reward function. The goal of obtaining the global optimal path is to accumulate the highest reward, that is, the fewer paths taken, the optimal path. S107: Q value update: Among them, α represents the learning rate of the algorithm, γ is the decay rate, Q(S t ,A t ) is the Q value of the test action in the Q table, R t+1 is the reward value obtained by adjusting the reward function by integrating the global progress and local progress, Q(S t+1 ,a) are all possible Q values ​​in the next state after executing the action; S108: Model learning; S109: Model learning updates the Q value. After selecting the action according to the global perspective strategy, the Q value of the current state is updated by correcting the formula of S107. Here, the model learning step K is set; S110: Determine the end of learning: If the model reaches the end point after learning, determine whether it has cycled to the maximum number of iterations. If it has, it ends; otherwise, continue learning under the Dyna framework. If it has not reached the end point, continue iterative learning.

3. The mobile robot path planning method according to claim 2, characterized in that: In step S102, the tasting rules and rewards are set as follows: 1) Give a small positive reward rr1 for each small step. When there are no obstacles around, determine the distance between the agent state s and the target point and choose an action; 2) When you have to go back, give the state a large negative reward rr2; 3) When reaching the target point, a large positive reward rr3 is given; 4) Finally, the state-action value function table is updated based on this.

4. The mobile robot path planning method according to claim 2, characterized in that: In step S103, an ellipse refinement range is constructed according to the ellipse refinement strategy, specifically including: the focal length of the ellipse is 2c, the major axis length is 2a, and the minor axis length is 2b; the focus s is defined as the starting point, the focus g is defined as the end point, and the straight-line distance from the starting point to the end point is P min =2c; at the same time, the major axis of the ellipse is defined as the average path length learned before the current episode, that is, P now =2a; the distance from any point on the ellipse to the two foci is equal to the length of the major axis; in connection with path planning, according to the relevant parameters of the ellipse, the following formula is obtained d1+d2=2a=P now Among them, d1 and d2 are the distances from a point on the ellipse to the two foci respectively; its practical significance is that a path from the starting point, passing through a point on the ellipse to the end point, the length of this path is equal to the average path length P learned before the current episode now ; Therefore, the optimal path must be less than P now , which means that the points on the optimal path must be inside the ellipse; P now After finding a feasible path in S102, we start to build the elliptical range. At this time, P now =P1, where P1 is the first path length; after that, when the number of successful paths reaches a certain threshold, the ellipse will be updated, and the length of the major axis is equal to the average path length learned before the current episode. The threshold θ of this strategy is set as follows θ=η k Among them, η is the threshold base value, the initial value of k is 0, and k will increase by 1 when the threshold is reached once.

5. The mobile robot path planning method according to claim 4, characterized in that: In step S105, the improved SA-ε greedy strategy is: Among them, ε k , ε0 and ε f They represent the ε value, initial ε value and final ε value of the kth scene, respectively, the latter two are set values; β is the proportional factor, N tar is an intermediate set number; the calculation formula of ρ is as follows: Where N is the total number of episodes.

6. The mobile robot path planning method according to claim 2, characterized in that: In step S106, the reward function formula is as follows: Among them, r GL (s, a) represents the reward function; r0 is a large positive value so that the agent gets a good positive feedback when it reaches the target point; r1 is a large negative value so that the agent gets a large penalty when it encounters an obstacle; r gl is the dual-view reward function in general, where χ and λ are the scaling factors of global progress and local progress, and τ is the local view scaling factor; d now represents the Euclidean distance between the current state s and the target state target, d t Represents the next state s′ to the local history state s h The Euclidean distance of d now =||s-goal||2 d t =||s′-s h ||2 Among them, s h represents the hth state before s′, d t It is a local perspective.

7. The mobile robot path planning method according to claim 6, characterized in that: In step S108, model learning specifically includes: after completing the calculation and update of the Q value, storing the state grid and the score of the state in the model table, and when the Q-learning part has learned n steps, model learning will be performed; using the global perspective in the Dyna architecture to guide the action taken in the state; The evaluation of each state is only related to the global progress function, that is, the Euclidean distance between the next state and the target state during the planning update process: H(s,a)=||s′-goal||2, and the heuristic action h is found through the minimum heuristic function H a (s,H):

8. The mobile robot path planning method according to claim 1, characterized in that: The high-level local dynamic obstacle avoidance method specifically includes the following steps: S201: The robot drives along the global optimal path planned by the bottom-level module. When the robot drives along the optimal path and the sensor detects an unknown obstacle nearby, the high-level module is called to avoid the obstacle. S202: Local dynamic obstacle avoidance is divided into two steps: one is to respond immediately to unknown obstacles, and the other is to quickly return to the optimal path line after avoiding the obstacles. For the first step, the robot selects according to the global perspective, that is, the obstacle avoidance rule according to the Euclidean distance from the next state to the target point; For the second step, an expert reward function is used to constrain the vehicle to quickly return to the original path after local dynamic obstacle avoidance: Take actions to avoid obstacles, giving a small negative reward r1 every time a new state is reached, and a large negative reward r1 + r2 when hitting an obstacle, where r2 < r1 < 0; when the robot reaches a cell on the global guidance path, it will receive a large positive reward r1 + Nc × r3, r3 > |r1| > 0; N c is the number of cells moved on the global guidance path between the point where the robot first leaves the path and the point where it rejoins the path; S203: Q value update: Improve the Q table update strategy so that it can learn deeper information; the improved update rule is: Q(s,a)=Q(s,a)+α[r+γ(n1maxQ′+n2maxQ”)-Q] Where Q” represents the maximum Q value obtained when the next state S’ reaches S” after executing possible actions; Q′ represents the maximum next Q value that can be obtained in the current state; where n1+n2=1 and n1>n2>0; α represents the learning rate of the algorithm, γ is the decay rate, and r represents the reward value; S204: The robot avoids unknown obstacles according to the high-level module and returns to the global optimal path, and continues to drive along the path.

Images