Mobile robot path planning method based on improved Q-learning

Abstract

The invention provides a mobile robot path planning method based on improved Q-learning. The mobile robot path planning method comprises the following steps: modeling an environment by adopting a grid method; modeling a mobile robot path planning problem by adopting a Markov decision process; and solving an optimal path based on an improved Q-learning algorithm. According to the invention, the navigation performance and decision-making intelligence of the mobile robot in a complex environment are improved, and a more adaptive path planning solution is provided for scenes such as intelligent storage and outdoor inspection.

Description

Technical Field

[0001] The present invention relates to the technical field of robot path planning, and particularly to a mobile robot path planning method based on improved Q-learning. Background Art

[0002] With the rapid development of mobile robot technology, path planning, as the core link of autonomous navigation, its optimization efficiency and path quality directly affect the operation efficiency of the robot. As a classic method of reinforcement learning, the Q-learning algorithm is widely used in path planning problems in complex environments due to its model-free characteristics. However, the existing mobile robot path planning methods based on Q-learning still have the following technical bottlenecks:

[0003] 1. The path quality is limited by the single-objective reward function

[0004] Most of the existing methods use a single path length as the reward function, resulting in the paths generated by the algorithm being mostly zigzag paths. The frequent change of turning angles in the planned path not only reduces the movement efficiency of the robot but also exacerbates the wear of mechanical components.

[0005] 2. The lack of environmental prior knowledge leads to low convergence efficiency

[0006] The traditional Q-learning algorithm adopts a strategy of randomly initializing the Q-table and does not effectively utilize environmental prior information. This "zero-knowledge start" method makes the intelligent agent need to explore the environment through a large number of trials and errors, significantly prolonging the training time.

[0007] 3. The action selection strategy lacks an initial guidance mechanism

[0008] The traditional ε-greedy strategy has a problem of misleading cold start of Q values in the initial stage of training. Due to the lack of integration of heuristic information such as target approach degree and turning smoothness, the intelligent agent lacks effective guidance when selecting actions, resulting in a slow convergence speed. Summary of the Invention

[0009] Aiming at the existing technical bottlenecks, the present invention proposes a mobile robot path planning method based on improved Q-learning, aiming to realize the collaborative optimization of path economy and motion coherence by constructing a multi-objective composite reward function, designing a Q-table initialization strategy based on environmental prior knowledge, and improving the action selection strategy, so as to provide an efficient and robust navigation solution for mobile robots in complex environments.

[0010] To achieve the above object, the present invention provides a mobile robot path planning method based on improved Q-learning, including the following steps:

[0011] Model the environment using the grid method;

[0012] The Markov decision process is used to model the path planning problem of mobile robots;

[0013] Solve the optimal path based on the improved Q-learning algorithm.

[0014] Furthermore, the grid method is used to model the environment, including the following steps:

[0015] Environment rasterization: discretize the two-dimensional plane environment in which the robot is located into a number of square grid units of equal size;

[0016] Attribute tag: Assign a status tag to each grid, usually represented by binary: 0 represents a free grid with no obstacles, and 1 represents an impassable grid with obstacles.

[0017] Furthermore, the specific steps of modeling the mobile robot path planning problem using the Markov decision process are:

[0018] The state space, action space, and reward function in the Markov decision process model of the mobile robot path planning problem are specifically defined as follows:

[0019] State space: state s t Defined as s t ={(x t ,y t ),a t-1}, where: (x t ,y t ) represents the position coordinates of the robot in the grid map; a t-1 is the movement direction of the robot in the previous step;

[0020] Action space: action a t Defined as the direction of movement of the robot; the robot can move from its current position to the adjacent grid, and there are 8 possible movement directions: up, down, left, right, upper left, upper right, lower left, and lower right, such as Figure 3 When you select the 4 moving directions of up, down, left, and right, the moving distance is 1. When you select the 4 moving directions of upper left, upper right, lower left, and lower right, the moving distance is When selecting action a t When the robot moves outside the grid map boundary or collides with obstacles, it is necessary to determine the feasibility of the action to ensure that the robot does not move outside the grid map boundary or collide with obstacles.

[0021] Reward function: A multi-objective reward function is used, and path length and path smoothness are integrated into the reward function design.

[0022] Furthermore, the reward function is specifically defined as follows:

[0023] r(s t ,a t ) = R goal +R step +R smooth

[0024] Each sub - item is defined as follows:

[0025] Target arrival reward item R goal , which is used to encourage the robot to reach the target position as soon as possible, and a large positive reward is given when the robot reaches the target position:

[0026]

[0027] Where K g > 0 is the target arrival reward constant;

[0028] Moving distance penalty item R step , which is used to penalize the distance the robot moves and encourage the robot to choose a shorter path:

[0029] R step = - ||a t ||

[0030] Where: ||a t || represents the moving distance of action a t . When moving in a straight line, ||a t || = 1; when moving diagonally,

[0031] Steering angle penalty item R smooth , which is used to penalize the change of the steering angle in the robot's path and encourage the robot to choose a smooth path:

[0032] R smooth = cos(Δθ)-1

[0033] Where: Δθ is the angle between the current moving direction and the previous moving direction.

[0034] Furthermore, the specific steps of solving the optimal path based on the improved Q - learning algorithm include:

[0035] Construct a comprehensive heuristic function that fuses the target approach degree and steering smoothness;

[0036] Design a fusion mechanism for Q - value and heuristic information;

[0037] Improve the ε - greedy strategy.

[0038] Furthermore, the construction of the comprehensive heuristic function that fuses the target approach degree and steering smoothness:

[0039] H(a t ) = ω d ·Δd(a t ) + ω θ ·cos(Δθ(a t ))

[0040] Where: Δd(a t ) is the target approach heuristic, and cos(Δθ(a t )) is the steering smoothness heuristic; ω d and ω θ are the balance target approach weight coefficient and the steering smoothness weight coefficient, respectively.

[0041] Furthermore, the specific steps for fusing the designed Q value with the heuristic information are as follows:

[0042] Perform weighted fusion of the Q value and the heuristic information:

[0043] Q enh (s t , a t ) = Q(s t , a t ) + β·H(a t )

[0044] Where β is the heuristic information weight, which decays exponentially:

[0045]

[0046] Where β base is the initial heuristic information weight, λ is the decay coefficient, and N episode is the number of training episodes.

[0047] Furthermore, the specific steps for improving the ε-greedy strategy are as follows: Randomly select an action uniformly from the set of legal actions of the current state s t with probability ε, and select the action with the maximum Q enh (s t , a) value from the set of legal actions with probability 1 - ε, that is:

[0048]

[0049] Adjust the exploration rate ε in the following way:

[0050] ε = max(ε min , ε init ·γ t )

[0051] Where: ε init is the initial exploration rate, γ < 1 is the decay factor, and t is the total number of training steps; εmin is the minimum exploration rate to prevent the strategy from becoming rigid.

[0052] According to one aspect of the present invention, there is provided a storage medium in which instructions are stored. When a computer reads the instructions, the computer is caused to execute the mobile robot path planning method based on improved Q-learning described in any one of the above.

[0053] According to another aspect of the present invention, there is provided an electronic device including a processor and the above storage medium, and the processor executes the instructions in the storage medium.

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] The mobile robot path planning method based on improved Q-learning proposed by the present invention realizes collaborative optimization of core indicators such as path economy and motion coherence compared with traditional methods.

[0056] First, a Markov decision process model for mobile robot path planning is constructed, and a multi-objective composite reward function that combines path length and smoothness is designed to improve trajectory continuity while ensuring the global path optimality.

[0057] Secondly, the traditional Q-learning algorithm is improved: a Q-table initialization strategy guided by environmental prior knowledge is adopted, and the initial value of the Q-table is generated by combining action legality determination and target approach degree evaluation; an improved ε-greedy strategy is designed to incorporate heuristic information of target approach degree and turning smoothness at the initial stage of training to accelerate convergence.

[0058] The present invention improves the navigation performance and decision-making intelligence of mobile robots in complex environments, and provides a more adaptable path planning solution for scenarios such as intelligent warehousing and outdoor inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0060] Unless otherwise defined, the technical terms or scientific terms used herein shall have the ordinary meanings understood by those of ordinary skill in the art to which the present invention pertains.

[0061] Figure 1 is the flowchart of the method in a preferred embodiment of the present invention;

[0062] Figure 2 is the environmental grid map in a preferred embodiment of the present invention;

[0063] Figure 3 It is a schematic diagram of the action space in a preferred embodiment of the present invention;

[0064] Figure 4 It is a schematic diagram of the steering angle in a preferred embodiment of the present invention;

[0065] Figure 5 It is a schematic diagram for calculating the improved amount of the standardized distance in a preferred embodiment of the present invention. Detailed implementation manners

[0066] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] Embodiment 1: As Figures 1-5 shown, the present invention provides a mobile robot path planning method based on improved Q-learning, including the following steps:

[0068] Step S0, modeling the environment using the grid method;

[0069] Step S1, modeling the mobile robot path planning problem using the Markov decision process;

[0070] Step S2, solving the optimal path based on the improved Q-learning algorithm.

[0071] Furthermore, the modeling of the environment using the grid method includes the following steps:

[0072] (1) Environment gridification: Discretize the two-dimensional plane environment where the robot is located into a number of square grid cells of equal size.

[0073] (2) Attribute marking: Assign a state identifier to each grid, usually represented in binary: 0 represents a free grid without obstacles, and 1 represents an impassable grid with obstacles.

[0074] Furthermore, in the Markov decision process model of the mobile robot path planning problem, the state space, action space, and reward function are specifically defined as follows:

[0075] (1) State space: State s t is defined as s t = {(x t , y t ), a t-1}), where: (x t , y t ) represents the position coordinates of the robot in the grid map; a t-1 is the moving direction of the robot in the previous step.

[0076] (2) Action Space: Action a t is defined as the moving direction of the robot. The robot can move from the current position to adjacent grids, and there are 8 moving directions: up, down, left, right, upper left, upper right, lower left, lower right. When choosing the 4 moving directions of up, down, left, and right, the moving distance is 1. When choosing the 4 moving directions of upper left, upper right, lower left, and lower right, the moving distance is

[0077] (3) Reward Function: A multi-objective reward function is adopted, integrating the path length and path smoothness into the reward function design. The specific definitions are as follows:

[0078] r(s t ,a t ) = R goal +R step +R smooth

[0079] The respective sub-items are defined as follows:

[0080] 1) Target Arrival Reward Item R goal :

[0081]

[0082] where K g > 0 is the target arrival reward constant.

[0083] 2) Moving Distance Penalty Item R step :

[0084] R step = -||a t ||

[0085] where: ||a t || represents the moving distance of action a t . When moving in a straight line, ||a t || = 1; when moving diagonally,

[0086] 3) Turning Angle Penalty Item R smooth :

[0087] R smooth = cos(Δθ) - 1

[0088] where: Δθ is the angle between the current moving direction and the previous moving direction.

[0089] Furthermore, the improved Q-learning algorithm optimizes the initialization of the Q-table and incorporates heuristic information into the action selection strategy.

[0090] Aiming at the problem of low convergence efficiency caused by the random initialization of the Q-table in the traditional Q-learning algorithm, the present invention proposes a method for initializing the Q-table based on prior environmental knowledge, and the specific steps are as follows:

[0091] (1) Action legality determination

[0092] Perform a feasibility test on each state-action pair (s, a), which needs to meet the following conditions: the coordinates after movement do not exceed the boundary of the grid map; the target grid has no obstacle mark.

[0093] (2) Initialization of Q-value for illegal actions

[0094] For illegal actions, initialize their corresponding Q-values to a relatively large negative constant:

[0095] Q init (s,a) = K illegal

[0096] where K illegal is a relatively large negative constant, which is used to guide the algorithm to avoid illegal actions.

[0097] (3) Initialization of Q-value for legal actions

[0098] 1) Calculate the Euclidean distance from the current position (x, y) to the target position g = (x g , y g ):

[0099]

[0100] 2) Determine the next state s' after executing the action a, and calculate the Euclidean distance from the new position (x', y') to the target position:(0)

[0101]

[0102] 3) Calculate the normalized distance improvement:

[0103]

[0104] 4) Set the initial Q-value:

[0105] Q init (s,a) = Δd

[0106] The improved Q-learning algorithm of the present invention optimizes the traditional ε-greedy strategy: First, construct a comprehensive heuristic function that integrates the target approach degree and the turning smoothness; Second, design a fusion mechanism for Q-values and heuristic information. This improvement achieves: strengthening heuristic guidance in the initial stage of training to accelerate convergence, and emphasizing Q-value utilization in the later stage of training to improve policy stability. The specific implementation steps are as follows:

[0107] (1) Design of Comprehensive Heuristic Function

[0108] Construct a comprehensive heuristic function that combines the target approach degree and steering smoothness:

[0109] H(a t ) = ω d ·Δd(a t ) + ω θ ·cos(Δθ(a t ))

[0110] Where: Δd(a t ) is the heuristic of the target approach degree, and cos(Δθ(a t )) is the heuristic of steering smoothness. The weight coefficients ω d and ω θ are used to balance the priorities of the target approach degree and steering smoothness.

[0111] (2) Fusion of Q-Value and Heuristic Information

[0112] Perform weighted fusion of the Q-value and heuristic information:

[0113] Q enh (s t , a t ) = Q(s t , a t ) + β·H(a t )

[0114] Where β is the weight of the heuristic information, which decays exponentially:

[0115] <>

[0116] Where β base is the initial weight of the heuristic information, λ is the decay coefficient, and N episode is the number of training episodes.

[0117] (3) Improved ε-greedy Strategy

[0118] The improved ε-greedy strategy randomly selects an action uniformly from the set of legal actions of the current state s t with probability ε, and selects the action with the maximum Q enh (s t , a) value from the set of legal actions with probability 1 - ε, that is:

[0119]

[0120] Adjust the exploration rate ε in the following way:

[0121] ε = max(ε min, ε init ·γ t )

[0122] Where: ε init is the initial exploration rate, γ < 1 is the decay factor, and t is the total number of training steps; ε min is the minimum exploration rate to prevent the policy from becoming rigid.

[0123] Example 2:

[0124] The computer-readable storage medium of this embodiment stores a computer program, and when the program is executed by a processor, it implements the steps in the mobile robot path planning method based on improved Q-learning in Example 1.

[0125] The computer-readable storage medium of this embodiment can be an internal storage unit of the terminal, such as the hard disk or memory of the terminal; the computer-readable storage medium of this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash card, etc. equipped on the terminal; further, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the terminal.

[0126] The computer-readable storage medium of this embodiment is used to store the computer program and other programs and data required by the terminal, and the computer-readable storage medium can also be used to temporarily store the data that has been output or will be output.

[0127] Example 3:

[0128] The computer device of this embodiment includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the mobile robot path planning method based on improved Q-learning in Example 1.

[0129] In this embodiment, the processor can be a central processing unit, or it can also be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc.; the memory can include a read-only memory and a random access memory, and provides instructions and data to the processor. A part of the memory can also include a non-volatile random access memory. For example, the memory can also store information about the device type.

[0130] Those skilled in the art should understand that the content disclosed in the embodiments can be provided as a method, a system, or a computer program product. Therefore, the present solution can be implemented in the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present solution can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0131] The present solution is described with reference to the flowcharts and / or block diagrams of methods and computer program products according to the embodiments of the present solution. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions; these computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0132] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, so that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0134] Those of ordinary skill in the art can understand that all or part of the processes of implementing the above-described embodiment methods can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-described method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0135] The examples described in the present invention are only descriptions of the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. Without departing from the design idea of the present invention, various deformations and improvements made by those skilled in the art to the technical solutions of the present invention shall fall within the protection scope of the present invention.

Claims

1. A path planning method for a mobile robot based on improved Q-learning, characterized in that, It includes the following steps: Model the environment using the grid method; Model the mobile robot path planning problem using the Markov decision process; Solve the optimal path based on the improved Q-learning algorithm.

2. The method according to claim 1, wherein The step of modeling the environment using the grid method includes the following steps: Environment gridification: Discretize the two-dimensional plane environment where the robot is located into a number of square grid cells of equal size; Attribute marking: Assign a state identifier to each grid.

3. The method according to claim 1, wherein The specific steps of modeling the mobile robot path planning problem using the Markov decision process are as follows: In the Markov decision process model of the mobile robot path planning problem, the state space, action space, and reward function are specifically defined as follows: State space: state s t is defined as s t ={(x t , y t ), a t-1}, where: (x t , y t ) represents the position coordinates of the robot in the grid map; a t-1 is the moving direction of the robot in the previous step; Action space: Action a t is defined as the moving direction of the robot; Reward function: Adopt a multi-objective reward function and incorporate the path length and path smoothness into the design of the reward function.

4. The method according to claim 3, wherein The specific definition of the reward function is as follows: r(s t ,a t ) = R goal +R step +R smooth Each sub-item is defined as follows: Target arrival reward item R goal , which is used to encourage the robot to reach the target position as soon as possible and gives a large positive reward when the robot reaches the target position: where K g > 0 is the target arrival reward constant; Moving distance penalty term R step , which is used to penalize the distance the robot moves and encourage the robot to choose a shorter path: R step = -||a t || Wherein: ||a t || represents the moving distance of action a t When moving in a straight line, ||a t || = 1; when moving diagonally Steering angle penalty term R smooth , which is used to penalize the change of steering angle in the robot path and encourage the robot to choose a smooth path: R smooth = cos(Δθ) - 1 Where: Δθ is the angle between the current moving direction and the previous moving direction.

5. The method according to claim 1, wherein The specific steps of solving the optimal path based on the improved Q-learning algorithm include: Construct a comprehensive heuristic function that combines the target approach degree and turning smoothness; Design a fusion mechanism for Q value and heuristic information; Improve the ε-greedy strategy.

6. The method according to claim 5, characterized in that, The construction of the comprehensive heuristic function that combines the target approach degree and turning smoothness: H(a t ) = ω d ·Δd(a t ) + ω θ ·cos(Δθ(a t )) where: Δd(a t ) is the target approach heuristic, and cos(Δθ(a t )) is the steering smoothness heuristic; ω d and ω θ are the weight coefficients for balancing the target approach and the steering smoothness, respectively.

7. The method according to claim 5, wherein The specific steps of designing the fusion of Q value and heuristic information are as follows: Perform weighted fusion of the Q value and heuristic information: Q enh (s t ,a t ) = Q(s t ,a t ) + β·H(a t ) Where β is the weight of the heuristic information, which decays exponentially: Among which β base is the initial weight of heuristic information, λ is the decay coefficient, and N episode is the number of training rounds.

8. The method according to claim 5, characterized in that The specific steps of the improved ε-greedy strategy are as follows: With probability ε, randomly select an action uniformly from the set of legal actions of the current state s t , and with probability 1 - ε, select the action with the maximum Q enh (s t , a) value from the set of legal actions, that is: Adjust the exploration rate ε in the following manner: where: ε init is the initial exploration rate, γ < 1 is the decay factor, and t is the total number of training steps; ε min is the minimum exploration rate to prevent the policy from becoming rigid.

9. A storage medium, characterized in that, The storage medium stores instructions, and when the computer reads the instructions, it causes the computer to execute the mobile robot path planning method based on the improved Q-learning as described in any one of claims 1-8.

10. An electronic device, characterized in that, It includes a processor and the storage medium described in claim 9, and the processor executes the instructions in the storage medium.

Images