Mobile robot path planning method based on improved DQN algorithm

Abstract

The present invention discloses a mobile robot path planning method based on an improved DQN algorithm, wherein the method steps include: constructing a two-dimensional grid environment map for path planning; constructing a DQN path planning model based on the two-dimensional grid environment map; optimizing the DQN path planning model to obtain a final model; and using the final model to complete the path planning of the mobile robot. Based on the DQN path algorithm, the present invention designs an artificial potential field to guide action selection method, combined with a time-varying dynamic greedy strategy to guide action selection, which can effectively reduce useless exploration and improve exploration efficiency; at the same time, the present invention also designs a high-value priority experience replay system based on classification, improves the replay method from the two perspectives of experience storage and experience sampling, improves the sampling utilization rate, and reduces the impact of the traditional priority experience replay system on the overall algorithm.

Description

Technical Field

[0001] The present invention relates to the field of robot path planning, and in particular to a mobile robot path planning method based on an improved DQN algorithm. Background Art

[0002] As a key part of mobile robot technology, path planning has become a hot topic in this field. Mobile robot path planning is based on certain optimization principles, such as the shortest walking distance and the shortest walking time, to find the optimal path to reach the target point in an unknown environment. Classic path planning algorithms mainly include A* algorithm, artificial potential field method, genetic algorithm, ant colony algorithm, etc. The above algorithms need to model the environment before use, and the accuracy and completeness of the modeling affect the quality of path planning. However, in practical applications, the variability and complexity of environmental information require mobile machines to be able to achieve autonomous learning, perceive the surrounding environment and make decisions.

[0003] With the continuous development of the field of artificial intelligence, methods such as reinforcement learning and deep learning have been introduced into path planning, enabling mobile robots to achieve perception and decision-making in complex environments. The DQN algorithm is a deep reinforcement learning method that combines the decision-making ability of reinforcement learning with the perception ability of deep learning, solving the problem of dimensionality disaster caused by reinforcement learning in complex environments. It has achieved good results in path planning and has been widely used, but it still has problems such as slow early convergence speed and low sampling utilization when facing different environments. Summary of the invention

[0004] In order to solve the technical problems in the above background, the present invention aims to provide an improved DQN path planning algorithm and system with guided action selection and classified high-value priority experience replay. In order to solve the problem of slow convergence speed of the algorithm in the early stage, the present invention designs an artificial potential field to guide action selection mechanism, calculates the optimal action of the robot in the current state through the potential field gravity value, and guides action selection in combination with a time-varying greedy strategy, reduces invalid actions on the basis of ensuring the "exploration-utilization" balance, and accelerates the convergence speed of the algorithm in the early stage. In view of the problem of low algorithm sampling utilization, the present invention designs a classified high-value priority experience replay mechanism. In the experience sample storage stage, a double experience pool is set, and the experience samples are classified and stored according to the immediate reward value. In the experience sample sampling stage, a high-value priority experience sampling method is proposed, which combines the state-action value function of the sample with the TD error as the measurement standard of priority sampling, and uses the high-value priority experience sampling strategy and the random uniform sampling strategy to extract experience samples from the two experience pools according to the proportion, which effectively improves the sampling utilization rate, and reduces the impact of the traditional priority experience replay system on the overall algorithm.

[0005] To achieve the above object, the present invention provides a mobile robot path planning method based on an improved DQN algorithm, the steps comprising:

[0006] Construct a two-dimensional grid environment map for path planning;

[0007] Based on the two-dimensional grid environment map, a DQN path planning model is constructed;

[0008] Optimizing the DQN path planning model to obtain a final model;

[0009] The final model is used to complete the path planning of the mobile robot.

[0010] Preferably, in the constructed two-dimensional grid environment map, each square has the same size and is evenly distributed; at the same time, it contains squares with four attributes, namely the initial starting point of the mobile robot, the feasible area, the obstacle and the end point; the mobile robot and the target point are connected.

[0011] Preferably, the method for constructing the DQN path planning model includes: combining a deep neural network with a Q-learning algorithm, using a neural network as a function approximation to replace the Q value table, and calculating the value function, using the neural network as a carrier of the state-action value function, and approximating the state-action value function by continuously iteratively updating the f parameter θ of the neural network, which is defined as:

[0012] f(s,a,θ)≈Q(s,a)

[0013] Among them, f(s,a,θ) represents the approximate substitution function, which replaces the Q value with the output of the neural network; s represents the current state; and a represents the action.

[0014] Preferably, the method for optimizing the DQN path planning model includes:

[0015] Design an artificial potential field to guide action selection method to replace the traditional decision-making method in the DQN path planning model;

[0016] A classified high-value priority experience replay method is designed to replace the traditional experience replay method in the DQN path planning model.

[0017] Preferably, the artificial potential field guided action selection method introduces the artificial potential field into the DQN algorithm and combines it with a time-varying dynamic greedy strategy to reduce useless exploration and accelerate the convergence speed of early training, specifically including:

[0018] The gravity value is selected as a guide for action selection, and the gravity is defined as F(p):

[0019]

[0020] Among them, k f Represents the gravitational field gain coefficient, which controls the size of the gravitational field; d size Represents the diagonal distance of the map; d(p,p goal ) represents the distance between the robot and the target point; d t Indicates the shortest distance threshold.

[0021] The optimal action of the current action is calculated according to the gravity value of the artificial potential field, and introduced into the 4 random action selections. The selection probabilities of the 5 actions are as follows:

[0022]

[0023] Among them, k represents the gravity value discount factor, which is used to balance the gravity value and ensure that the probability of selecting the optimal action in the artificial potential field is lower than the probability of random action, avoiding the situation where the gravity value is too large and affects the selection of random actions.

[0024] Preferably, the classified high-value priority experience playback method consists of two stages: experience storage and experience sampling. In the experience storage stage, positive and negative dual experience pools are set up, and experience samples are classified and stored according to the immediate reward value. In the experience sampling stage, a high-value priority experience sampling method is proposed in the positive experience pool, and the Q value and TD error are combined as the evaluation criteria for sample priority; the random uniform sampling method is still used in the negative experience pool. The priority of experience e in the positive experience pool is expressed as:

[0025]

[0026] Among them, P Q (e) represents the Q value priority of experience e; P TD (e) represents the TD error priority of experience e; β represents the time-varying weight factor, β = 1 when t = 0.

[0027] The present invention also provides a mobile robot path planning system based on an improved DQN algorithm, the system is used to implement the above method, including: a map building module, a model building module, an optimization module and a path planning module;

[0028] The map construction module is used to construct a two-dimensional grid environment map for path planning;

[0029] The model building module is used to build a DQN path planning model based on the two-dimensional grid environment map;

[0030] The optimization module is used to optimize the DQN path planning model to obtain a final model;

[0031] The path planning module is used to complete the path planning of the mobile robot using the final model.

[0032] The beneficial effects of the present invention are as follows:

[0033] Based on the DQN path algorithm, the present invention designs an artificial potential field to guide action selection method, which is combined with a time-varying dynamic greedy strategy to guide action selection, which can effectively reduce useless exploration and improve exploration efficiency. At the same time, the present invention also designs a high-value priority experience replay system based on classification, improves the replay method from the two perspectives of experience storage and experience sampling, improves the sampling utilization rate, and reduces the impact of the traditional priority experience replay system on the overall algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of a 20×20 grid map according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the principle framework of the DQN algorithm according to an embodiment of the present invention;

[0038] Figure 4 This is a flow chart of artificial potential field guided action selection according to an embodiment of the present invention;

[0039] Figure 5 Schematic diagram of a classified high-value priority experience playback method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Embodiment 1

[0043] like Figure 1FIG. 1 is a schematic diagram of the method flow of this embodiment, and the steps include:

[0044] S1. Construct a two-dimensional grid environment map for path planning.

[0045] like Figure 2 As shown, in the constructed two-dimensional grid environment map, each square has the same size and is evenly distributed; at the same time, the two-dimensional grid environment map also contains squares with four attributes, namely the initial starting point of the mobile robot, the feasible area, the obstacle and the end point, and the mobile robot is connected to the target point.

[0046] S2. Build a DQN path planning model based on a two-dimensional grid environment map.

[0047] The principle framework of the DQN algorithm is as follows Figure 3 As shown in the figure, the deep neural network is combined with the Q-learning algorithm, and the neural network is used as a function approximation to replace the Q value table, and the value function is calculated. The neural network is used as the carrier of the state-action value function, and the f parameter of the neural network is continuously iterated to approximate the state-action value function, which is defined as:

[0048] f(s,a,θ)≈Q(s,a)

[0049] Among them, f(s,a,θ) represents the approximate substitution function, which replaces the Q value with the output of the neural network; s represents the current state; and a represents the action.

[0050] Initialize the state function, action function, and reward function of the DQN path planning model. Each cell in the grid corresponds to a state. The mobile robot interacts with the environment to obtain the current state s. Then, the time-varying dynamic greedy algorithm is used to select the action a. The optimal action is selected with a probability of ε, and the action is randomly selected with a probability of 1-ε. After execution, the new state s′ and reward value r are obtained. The state behavior sequence (s t ,a t ,r t ,s t+1 ) is stored in the experience pool. Action a contains 4 different actions, namely up, down, left, and right, and each unit step only moves one unit distance, that is, one square. Reward value r t , as follows:

[0051]

[0052] Among them, C1 is the positive reward for reaching the end point, C2 is the negative reward for collision or exceeding the boundary, and C3 is the reward value for the mobile robot to operate normally in the feasible area.

[0053] The DQN model uses a time-varying dynamic greedy strategy to select actions, and the time-varying exploration factor ε can be expressed as:

[0054]

[0055] Where δ represents the exploration decay rate.

[0056] During training, a batch of samples are randomly and uniformly extracted from the experience pool as training data, and then the TD error is calculated to update the network parameters. The TD error is expressed as:

[0057] L t (θ t )=E[(y t -Q(s,a,θ t )) 2 ]

[0058] Among them, y t represents the optimization objective, which is defined as:

[0059]

[0060] Among them, θ t and θ t - They respectively represent the parameters of the estimated network and the target network, are the actions with the maximum Q value corresponding to a certain state, and are the discount factor coefficients.

[0061] The deep neural network in the DQN model includes an estimation network and a target network, which are identical except for the differences in network parameters. It consists of a 5-layer fully connected neuron structure, namely the input layer, hidden layer and output layer. Each layer uses the relu function as the activation function. The output layer outputs a 4-dimensional vector table, which represents the Q value of 4 different actions in the state.

[0062] In the DQN model, the main network uses the stochastic gradient descent (SGD) method to back-propagate the loss and update the estimated network parameters. The parameters of the target network are synchronized with the estimated network once every C steps, making the target network more stable than the estimated network.

[0063] S3. Optimize the DQN path planning model to obtain the final model.

[0064] S301. Design an artificial potential field to guide action selection method to replace the traditional decision-making method in the DQN path planning model.

[0065] After the design of the deep reinforcement learning DQN path planning model based on the grid graph environment is completed, the traditional action decision method is improved. This embodiment proposes an artificial potential field to guide action selection method, which introduces the idea of ​​artificial potential field into the DQN algorithm and combines it with the time-varying dynamic greedy strategy to reduce useless exploration and effectively speed up the convergence speed of early training. Figure 4 shown.

[0066] The core of the artificial potential field method is to transform the environment into a virtual force field, and add gravity and repulsion to the path points in the environment according to the positions of the starting point, the target point, and the obstacles, so that the robot in the environment moves according to the combined force of these two forces until it reaches the target point. This embodiment only uses the gravity value as a guide for action selection, and defines gravity F(p) as:

[0067]

[0068] Among them, k f Represents the gravitational field gain coefficient, which controls the size of the gravitational field; d size Represents the diagonal distance of the map; d(p,p goal ) represents the distance between the robot and the target point; d t Indicates the shortest distance threshold.

[0069] The optimal action of the current action is calculated according to the gravity value of the artificial potential field, and introduced into the 4 random action selections. The selection probabilities of the 5 actions are as follows:

[0070]

[0071] Among them, k represents the gravity value discount factor, which is used to balance the gravity value and ensure that the probability of selecting the optimal action in the artificial potential field is lower than the probability of random action, avoiding the situation where the gravity value is too large and affects the selection of random actions.

[0072] S302. Design a classified high-value priority experience replay method to replace the traditional experience replay method in the DQN path planning model.

[0073] In view of the problem that the traditional DQN experience replay method is prone to low sampling utilization, this embodiment proposes a classification high-value priority experience replay method, which improves the algorithm from the two perspectives of experience storage and sampling. The schematic diagram of the classification high-value priority experience replay method is shown in Figure 5 shown.

[0074] In the traditional DQN algorithm, the robot interacts with the environment to generate different experience samples, which are stored in a first-in, first-out experience pool. Based on this, in the experience storage stage, this embodiment stores the experience samples in the positive and negative experience pools according to the size of the immediate reward value. This achieves efficient use of some experience samples that have positive significance for network training.

[0075] In the traditional DQN algorithm, random uniform sampling is used to select experience samples from the experience pool for training, and this method has a low sampling utilization rate. Then the researchers used the sampling method of priority experience playback to effectively improve the sampling utilization rate, but there are problems such as long time consumption. Based on this, in the experience sampling stage, this embodiment adopts different sampling methods according to the actual needs of the two experience pools, and selects different numbers of experience samples from each experience buffer pool according to the proportion to update the network parameters.

[0076] This embodiment proposes a high-value priority experience sampling method in the positive experience pool. This method combines the Q value and the TD error as the evaluation criteria for sample priority. The priority of the experience e in the positive experience pool can be expressed as:

[0077]

[0078] Among them, P Q (e) represents the Q value priority of experience e; P TD (e) represents the TD error priority of experience e; β represents the time-varying weight factor, β = 1 when t = 0.

[0079] Most of the experiences in the negative experience pool are low-value experiences, so the random uniform sampling method is still used.

[0080] As the training progresses, the ratio of selected experience is adjusted. In the early stage of training, the ratio of selected experience is 5:5. In the later stage of training, the ratio of selected experience is 10:1.

[0081] Table 1

[0082]

[0083] The model parameter settings of this embodiment are shown in Table 1.

[0084] S4. Use the final model to complete the path planning of the mobile robot.

[0085] After the above steps, the trained final model is used to complete the path planning of the mobile robot.

[0086] Embodiment 2

[0087] This embodiment also provides a mobile robot path planning system based on an improved DQN algorithm, including: a map construction module, a model construction module, an optimization module and a path planning module; the map construction module is used to construct a two-dimensional grid environment map for path planning; the model construction module is used to construct a DQN path planning model based on the two-dimensional grid environment map; the optimization module is used to optimize the DQN path planning model to obtain a final model; the path planning module is used to use the final model to complete the path planning of the mobile robot.

[0088] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A mobile robot path planning method based on an improved DQN algorithm, characterized in that the steps include: Construct a two-dimensional grid environment map for path planning; Based on the two-dimensional grid environment map, a DQN path planning model is constructed; Optimizing the DQN path planning model to obtain a final model; The method for optimizing the DQN path planning model includes: designing an artificial potential field to guide action selection method to replace the traditional decision-making method in the DQN path planning model; designing a classified high-value priority experience replay method to replace the traditional experience replay method in the DQN path planning model; the artificial potential field to guide action selection method introduces an artificial potential field into the DQN algorithm and combines it with a time-varying dynamic greedy strategy to reduce useless exploration and accelerate the convergence speed of early training, specifically including: The gravity value is selected as a guide for action selection, and the gravity is defined as F(p): Among them, k f Represents the gravitational field gain coefficient, which controls the size of the gravitational field; d size Represents the diagonal distance of the map; d(p,p goal ) represents the distance between the robot and the target point; d t Indicates the shortest distance threshold; The optimal action of the current action is calculated according to the gravity value of the artificial potential field, and introduced into the 4 random action selections. The selection probabilities of the 5 actions are as follows: Among them, k represents the gravity value discount factor, which is used to balance the gravity value, ensure that the probability of selecting the optimal action of the artificial potential field is lower than the probability of random action, and avoid the situation where the gravity value is too large to affect the random action selection; The final model is used to complete the path planning of the mobile robot.

2. The mobile robot path planning method based on the improved DQN algorithm according to claim 1 is characterized in that: In the constructed two-dimensional grid environment map, each square has the same size and is evenly distributed; at the same time, it contains squares with four attributes, namely the initial starting point of the mobile robot, the feasible area, the obstacle and the end point; the mobile robot and the target point are connected.

3. The mobile robot path planning method based on the improved DQN algorithm according to claim 1 is characterized in that: The method for constructing the DQN path planning model includes: combining a deep neural network with a Q-learning algorithm, using a neural network as a function approximation to replace the Q value table, and calculating the value function, using the neural network as a carrier of the state-action value function, and approximating the state-action value function by continuously iteratively updating the f parameter θ of the neural network, which is defined as: f(s,a,θ)≈Q(s,a) Among them, f(s,a,θ) represents the approximate substitution function, which replaces the Q value with the output of the neural network; s represents the current state; and a represents the action.

4. The mobile robot path planning method based on the improved DQN algorithm according to claim 1 is characterized in that: The classified high-value priority experience playback method consists of two stages: experience storage and experience sampling. In the experience storage stage, positive and negative dual experience pools are set up to classify and store experience samples according to the immediate reward value. In the experience sampling stage, a high-value priority experience sampling method is proposed in the positive experience pool, and the Q value and TD error are combined as the evaluation criteria for sample priority. The random uniform sampling method is still used in the negative experience pool. The priority of experience e in the positive experience pool is expressed as: Among them, P Q (e) represents the Q value priority of experience e; P TD (e) represents the TD error priority of experience e; β represents the time-varying weight factor, β = 1 when t = 0.

Images