A DM-DQN-based mobile robot path planning method

Abstract

The application relates to the technical field of DQN algorithm, in particular to a mobile robot path planning method based on DM-DQN, which comprises the following steps: a mobile robot path planning model based on DM-DQN is established; a state space, an action space, a DM-DQN network model and a reward function of the DM-DQN algorithm are designed; the DM-DQN algorithm is trained to obtain an experienced reward value, and collision-free path planning of a robot is completed. The application introduces a competitive network structure, decomposes the network structure into a value function and an advantage function, thereby decoupling action selection and action evaluation, so that the state is no longer completely dependent on the value of the action for judgment, individual value prediction can be carried out, and the problem of slow convergence speed is solved; and through the design of a reward function based on an artificial potential field, the problem that the robot is too close to the edge of an obstacle is solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of DQN algorithms, and in particular to a mobile robot path planning method based on DM-DQN. BACKGROUND

[0002] With the development trend of artificial intelligence, the robot industry is also developing in the direction of intelligent self-learning and self-exploration, and path planning of a mobile robot is a core problem in robot movement, which aims to find an optimal or suboptimal path without collision from a starting point to an ending point; with the continuous development of science and technology, the environment faced by the robot is becoming more and more complex, and in an unknown environment, the information of the entire environment cannot be obtained, so the traditional path planning algorithm cannot meet the needs of people, for example, artificial potential field algorithm, ant colony algorithm, genetic algorithm and particle swarm algorithm. In view of this situation, deep reinforcement learning is proposed, which combines deep learning and reinforcement learning, wherein deep learning mainly extracts features of an unknown environment state input by a neural network to realize fitting of an environment state to an action value function; reinforcement learning completes decision-making according to the output of the deep neural network and an exploration strategy, so as to realize mapping of a state to an action. The combination of deep learning and reinforcement learning solves the dimension disaster problem caused by the mapping of a state to an action, and can better meet the robot movement demand in a complex environment. SUMMARY

[0003] In view of the shortcomings of the existing algorithm, the application introduces a competitive network structure, decomposes the network structure into a value function and an advantage function, thereby decoupling action selection and action evaluation, so that the state is no longer completely dependent on the value of the action for judgment, and can be independently predicted, solving the problem of slow convergence speed; and by designing a reward function based on an artificial potential field, the problem of the robot being too close to the edge of an obstacle is solved.

[0004] The technical scheme adopted by the application is: a mobile robot path planning method based on DM-DQN comprises the following steps:

[0005] Step one, establishing a mobile robot path planning model based on DM-DQN;

[0006] Step two, designing a state space, an action space, a DM-DQN network model and a reward function of the DM-DQN algorithm;

[0007] Further, the structure of the DM-DQN network model is divided into a value function V(s, omega, alpha) and an advantage function A(s, a, omega, beta), and the output of the DM-DQN network model is represented as:

[0008] Q(s, a, omega, alpha, beta) = V(s, omega, alpha) + A(s, a, omega, beta) (4)

[0009] where s represents state, a represents action, ω is a common parameter of V and A, α and β are parameters of V and A respectively, V value can be regarded as the average of Q value in s state, A value is limited by the average of 0, and the sum of V value and A value is the original Q value.

[0010] Further, the advantage function is centralized, and the output of the DM-DQN network model is represented as:

[0011]

[0012] where s represents state, a represents action, a' represents next action, A is the action available for selection, ω is a common parameter of V and A, α and β are parameters of V and A respectively.

[0013] Further, the reward function is divided into a position reward function and a direction reward function, and the total reward function is calculated according to the position reward function and the direction reward function.

[0014] Further, in the position reward function, first, a target guidance reward function is constructed using a gravitational potential field function:

[0015]

[0016] where ζ represents a gravitational reward function constant, d goal represents the distance between the current position and the target point;

[0017] Secondly, a repulsive potential field function is used to construct an obstacle avoidance reward function, which is a negative reward that decreases as the distance between the robot and the obstacle decreases:

[0018]

[0019] where η represents a repulsive reward function constant, d obs represents the distance between the current position and the obstacle, and d max represents the maximum influence distance of the obstacle.

[0020] Further, the direction reward function is represented according to the angle difference between the expected direction of the robot and the actual direction, and the formula is:

[0021]

[0022] where F q represents the expected direction, F a represents the actual direction, represents the included angle between the expected direction and the actual direction;

[0023] The direction reward function can be represented as:

[0024]

[0025] Further, the total reward function of the mobile robot is represented as:

[0026]

[0027] Wherein, r goal represents the target area radius centered on the target point, r obs represents the collision area radius centered on the obstacle;

[0028] Step three, the DM-DQN algorithm is trained, and the experienced reward value is obtained, and the collision-free path planning of the robot is completed.

[0029] The beneficial effects of the present application are:

[0030] 1. By introducing the competitive network structure, the network structure is decomposed into the value function and the advantage function, so that the action selection and the action evaluation are decoupled, the state is no longer completely dependent on the value of the action for judgment, the value prediction can be carried out alone, the problem of slow convergence speed is solved, and the generalization performance is better.

[0031] 2. By designing the reward function based on the artificial potential field, the problem of the robot being too close to the edge of the obstacle is solved; the learning efficiency in the dynamic unknown environment is higher, the convergence speed is faster, and the collision-free path far away from the obstacle can be planned. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is the DM-DQN network structure diagram of the present application;

[0033] Figure 2 (a) and (b) are the static environment diagram and the dynamic and static environment diagram of the present application, respectively;

[0034] Figure 3 (a) and (b) are the reward value diagrams of the static environment and the dynamic environment of the DM-DQN algorithm of the present application;

[0035] Figure 4 (a) and (b) are the static environment generated path diagram and the dynamic and static environment generated path diagram of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described below in conjunction with the drawings and examples, which are simplified schematic diagrams and only schematically show the basic structure of the present application, and therefore only show the structures related to the present application.

[0037] To solve the problem of slow convergence of M-DQN, a competitive network structure is introduced, which decomposes the network structure into value function and advantage function. To solve the problem of too close trajectory of robot to the edge of obstacle, an artificial potential field method is designed to make the trajectory of robot far away from the edge of obstacle.

[0038] As shown in Figure 1 , a DM-DQN-based mobile robot path planning method includes the following steps:

[0039] Step 1: Establish a DM-DQN-based mobile robot path planning model, and describe the mobile robot path planning problem as a Markov decision process.

[0040] First, estimate the Q value by online dueling Q-network with weight θ, and copy the weight θ to the target network with weight every C steps;

[0041] Second, interact with the environment by using the ε-greedy strategy, and obtain the reward and the next state according to the designed reward function based on artificial potential field. Finally, store the transitions (s t ,a t ,r t ,s t+1 ) in a fixed-size FIFO replay buffer, and every F steps, DM-DQN randomly extracts a batch D t from the replay buffer D, and according to the following formula:

[0042]

[0043] Regression target, minimize the loss;

[0044] In the formula, s represents the state, a represents the action, r represents the reward value, and γ represents the discount factor. Satisfies τ is a hyperparameter for controlling the weight of entropy, a' represents the action at t+1, α is a hyperparameter set to 1, represents the selected strategy in this state, is the action available for selection.

[0045] Step 2: Design the state space, action space, DM-DQN network model and reward function of DM-DQN algorithm.

[0046] The state space includes: laser radar data, current control instruction of mobile robot, last control instruction of mobile robot, orientation and distance of target point.

[0047] The action space includes angular velocity and linear velocity of the robot;

[0048] The action space of the robot is discretized into 5 actions, the linear velocity v is fixed as 0.15 m / s, the angular velocity value is given, the output of the control quantity selects the angular velocity instead of directly giving the rotation angle, which is more in line with the kinematic characteristics of the mobile robot, and the angular velocity given is according to the following formula:

[0049]

[0050] Wherein, action_size represents that the action space is discretized into 5 actions, action[5] represents that the value of the action is 0-4, max_angel vel The maximum angular velocity value of the robot turning is 1.5 rad / s, and the 5 actions are calculated according to formula (2), as shown in formula (3), wherein the linear velocity v is in unit of m / s, and the angular velocity ω is in unit of rad / s.

[0051]

[0052] Further, the DM-DQN network model divides the network structure into two parts, as shown in Figure 1 The first part is only related to the state S, called the value function, represented as V(s, ω, α); the other part is related to the state S and the action A, called the advantage function, represented as A(s, a, ω, β), therefore, the output of the network can be represented as:

[0053] Q(s, a, ω, α, β) = V(s, ω, α) + A(s, a, ω, β) (4)

[0054] Wherein s represents the state, a represents the action, ω is the common parameter of V and A, α and β are the parameters of V and A respectively, the value of V can be regarded as the average number of Q value under the state s, the value of A is limited to have an average number of 0, and the sum of the values of V and A is the original Q value;

[0055] Since the sum of A values must be 0, the network will preferentially update the V value, the V value is the average number of Q value, the adjustment of the average number is equivalent to updating all Q values under the state at a time, so when the network is updated, not only the Q value of a certain action is updated, but also all Q values of all actions under this state are adjusted at a time.

[0056] Further, in the robot path planning, the value function is to learn the situation that the robot does not detect obstacles, and the advantage function is to learn the situation that the robot detects obstacles, in order to solve the problem of identifiability, the advantage function is centralized:

[0057]

[0058] where s denotes state, a denotes action, a' denotes next action, A is the set of available actions, ω is a common parameter for V and A, and α and β are parameters for V and A, respectively.

[0059] Further, the reward function is designed according to the artificial potential field method, and the reward function is divided into two parts: the first part is the position reward function, including the target guidance reward function and the obstacle avoidance reward function, the target reward function is to guide the robot to quickly reach the target point, and the obstacle avoidance reward function is to keep the robot away from the obstacle at a certain distance;

[0060] The second part is the direction reward function, the current direction of the robot plays a key role in reasonable navigation, and the direction of the resultant force of the robot in the artificial potential field can well match the motion direction of the robot, so the direction reward function is designed to guide the robot to move towards the target point.

[0061] Further, in the position reward function, first, the target guidance reward function is constructed using the gravitational potential field function:

[0062]

[0063] where ζ represents the gravitational reward function constant, d goal represents the distance between the current position and the target point;

[0064] Secondly, the obstacle avoidance reward function is constructed using the repulsive potential field function, which is a negative reward, and decreases as the distance between the robot and the obstacle decreases:

[0065]

[0066] where η represents the repulsive reward function constant, d obs represents the distance between the current position and the obstacle, and d max represents the maximum influence distance of the obstacle.

[0067] Further, in the direction reward function, the angle difference between the expected direction of the robot and the actual direction is represented as:

[0068]

[0069] where F q represents the expected direction, F a represents the actual direction, represents the included angle between the expected direction and the actual direction;

[0070] Therefore, the direction reward function can be represented as:

[0071]

[0072] Furthermore, the overall reward function can be expressed as:

[0073]

[0074] The overall reward function for the mobile robot is expressed as:

[0075]

[0076] Where, r goal r represents the radius of the target area centered at the target point. obs This represents the radius of the collision zone centered on the obstacle.

[0077] Design a simulation environment where the mobile robot interacts with the environment to acquire training data. Sample the training data to perform simulation training on the mobile robot and complete collision-free path planning.

[0078] Step 3: Train the DM-DQN algorithm to obtain experience reward values ​​and complete the collision-free path planning for the robot.

[0079] The specific experimental steps are as follows:

[0080] A virtual simulation environment is created using the Gazebo simulator, and a robot model is built using Gazebo to perform path planning tasks. The simulation environment includes static and dynamic environments, such as... Figure 2 As shown, Figure 2 (a) For static environment, Figure 2 (b) is a dynamic environment;

[0081] The path planning algorithm is implemented using Python, and the built-in Gazebo simulator is used to control the robot's movement and acquire the robot's perception information.

[0082] The DM-DQN algorithm, after 320 simulation training iterations, obtained experience reward values, such as... Figure 3 As shown, Figure 3 (a) and (b) represent the cumulative reward and average reward of the agent in each round recorded by the DM-DQN algorithm in static and dynamic-static environments, respectively. Each point represents a round, and the black curve represents the average reward. This shows that DM-DQN adopts a competitive network structure, which decouples action selection and action evaluation, resulting in a faster learning rate. Therefore, it can make fuller use of the experience gained from exploring the environment in the early stages, thereby obtaining a larger reward.

[0083] The robot was navigated to seven designated points. In an unknown environment, the robot autonomously and without collision, progressed from position 1 to positions 2 through 7 sequentially and then returned to position 1, achieving collision-free path planning. Figure 4 As shown.

[0084] As shown in Table 1, under the same training condition, compared with the existing algorithm, the average number of movements to reach a target point and the number of successful target point reaching in 300 rounds are compared respectively. It can be found from the table that the average number of movements of DM-DQN is the least, the success number is increased by 50% compared with DQN algorithm, increased by 23.6% compared with Dueling DQN, and increased by 19.3% compared with M-DQN.

[0085] Table 1

[0086]

[0087] Based on the above ideal embodiments according to the present application, through the above description, relevant personnel can make various changes and modifications without deviating from the technical idea of the present application. The technical scope of the present application is not limited to the contents of the specification, and the technical scope must be determined according to the scope of claims.

Claims

1. A DM-DQN based path planning method for mobile robots, characterized in that, The method comprises the following steps: Step one, establishing a mobile robot path planning model based on DM-DQN; Step two, designing the state space, action space, DM-DQN network model and reward function of the DM-DQN algorithm; Discretize the action space of the robot, fix the linear velocity, and the formula of the given angular velocity is: （2） wherein, action_size action[5] represents the value of the action, represents the maximum angular velocity value of the robot turning. The direction reward function of the reward function is expressed as: （9） wherein is the angle difference; Step three, training the DM-DQN algorithm to obtain the experience reward value and complete the collision-free path planning of the robot; The structure of the DM-DQN network model is divided into a value function and an advantage function , and the output of the DM-DQN network model is represented as: （4） wherein s represents a state, a represents an action, is V and A a common parameter of and are respectively parameters of V and A , and V the average of the values of s in the state Q the average of the values.

2. The DM-DQN-based mobile robot path planning method of claim 1, wherein, The advantage function is centralized, and the output of the DM-DQN network model is expressed as: ) （5） wherein s represents a state, a represents an action, a' represents a next action, A is an alternative action, is V and A a common parameter of and parameters of V and A respectively. 3.The DM-DQN based path planning method for mobile robots according to claim 1, wherein: The reward function is divided into a position reward function and a direction reward function, and the total reward function is calculated according to the position reward function and the direction reward function.

4. The DM-DQN-based mobile robot path planning method of claim 3, wherein, In the position reward function, first, a target guidance reward function is constructed using a gravitational potential field function: （6） wherein, represents a gravitational reward function constant, represents a distance between the current position and the target point; Secondly, a repulsive potential field function is used to construct an obstacle avoidance reward function: （7） wherein, represents a repulsive reward function constant, represents a distance between the current position and the obstacle, represents a maximum influence distance of the obstacle.

5. The DM-DQN-based mobile robot path planning method of claim 3, wherein, The direction reward function is expressed according to the angle difference between the expected direction of the robot and the actual direction, and the angle difference formula is: （8） wherein, represents the intended direction, represents the actual direction, represents the angle between the intended direction and the actual direction.

6. The DM-DQN-based mobile robot path planning method of claim 4, wherein, The total reward function is expressed as: wherein, represents a target region radius centered on the target point, represents a collision region radius centered on the obstacle, represents an included angle between the intended direction and the actual direction.

Images