A mobile robot path planning based on improved deep Q-network algorithm

Abstract

The present invention discloses a mobile robot path planning based on an improved deep Q network algorithm. In view of the problems existing in the prior art, the path planning of the present invention introduces an improved artificial potential field gravity function and a target guidance action function into the action guidance strategy of the deep Q network algorithm, which effectively reduces the number of collisions in the algorithm training process and improves the convergence speed of the algorithm; the path planning of the present invention designs a piecewise reward function, which assigns a discounted reward value to the neighboring point closest to the target point at the current position, so as to encourage the mobile robot to explore towards the target point, thereby making the planned path better; the path planning of the present invention improves the sampling strategy, and through the priorities of all samples stored in the sample experience pool, the probability of the samples in the experience pool being sampled decreases as the number of sampling times increases, thereby improving the sample utilization rate and effectively solving the problems of loss value divergence and neural network overfitting in the algorithm training process.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a mobile robot path planning based on an improved deep Q network algorithm. Background Art

[0002] Robots are the "crown jewel of the manufacturing industry". Their research and development, manufacturing and application are important indicators of a country's scientific and technological innovation and high-end manufacturing level. The development of robot technology and industry has become a major strategic need for our country. Mobile robots are widely used in service, rescue, surveying and other fields. "Made in China 2025" and the "Outline of the New Generation Artificial Intelligence Development Plan" particularly emphasize the promotion of mobile robot development. Mobile robots have become an important direction of artificial intelligence research.

[0003] Path planning technology is one of the important contents in the field of mobile robot research. Effective path planning technology can enable mobile robots to complete operations and tasks efficiently. According to the characteristics of the algorithm, path planning algorithms are divided into traditional algorithm path planning and intelligent algorithm path planning. Most traditional algorithm path planning requires a priori environmental models and cannot effectively explore and plan in unknown environments; intelligent algorithm path planning imitates the experience and characteristics of biological groups in nature, thus having the ability to explore unknown environments, and can be better used in path planning in real environments. Deep reinforcement learning algorithm is one of them.

[0004] The deep reinforcement learning algorithm combines the perception ability of deep learning with the decision-making ability of reinforcement learning, so that when facing an unknown environment, the mobile robot can interact with the environment through the reinforcement learning algorithm, establish a mapping relationship between the learning state and the behavior, maximize the numerical return, obtain the optimal action sequence, and thus plan the optimal path; and then use the deep neural network to process high-dimensional state space tasks to solve the dimensionality disaster problem that occurs when reinforcement learning processes high-dimensional data. Among them, the Deep Q-Network (DQN) algorithm is a classic deep reinforcement learning algorithm based on value functions, which combines the convolutional neural network with the Q-Learning algorithm in reinforcement learning, thereby replacing the Q table in the Q-Learning algorithm with the convolutional neural network.

[0005] Google's DeepMind team proposed the DQN algorithm in 2013. The basic framework of the algorithm is the Q-Learning algorithm. Its action guidance strategy adopts a greedy strategy. When the mobile robot obtains the current action value, it needs to traverse the current state and the reward value of the next state after executing the action. Therefore, the mobile robot needs to traverse more invalid environmental states, which consumes a lot of computing time and causes the algorithm to converge slowly.

[0006] Whether the path planned by the DQN algorithm is optimal depends on whether the action guidance strategy of the Q-Learning algorithm is optimal. In the Q-Learning algorithm, after the mobile robot interacts with the environment and obtains rewards, it will be more inclined to perform actions with higher reward values. However, in traditional reward functions, except for the target point and the collision point with the obstacle, the reward value of other points is 0, which will cause the mobile robot to form redundant paths and a large number of turning points, thus failing to prompt the mobile robot to plan the optimal path.

[0007] The DQN algorithm uses an experience replay mechanism to process samples during network model training. Each time, a small batch of samples is extracted with equal probability from the stored experience sample pool for model training. Therefore, it is unable to effectively distinguish the importance of different samples, resulting in a low sample utilization rate, which leads to loss value divergence and neural network overfitting during algorithm training. Summary of the invention

[0008] The purpose of the present invention is to provide a mobile robot path planning based on an improved deep Q network algorithm. The path planning solves the problems of slow convergence speed of the deep Q network algorithm, non-optimal planned path and low sample utilization by improving the traditional DQN algorithm, effectively reducing the number of collisions during algorithm training, improving the convergence speed of the algorithm, and prompting the mobile robot to explore towards the target point, thereby making the planned path better.

[0009] The purpose of the present invention can be achieved through the following technical solutions:

[0010] A mobile robot path planning based on an improved deep Q network algorithm, the path planning comprising the following steps:

[0011] Step 1: Determine the position of the mobile robot and the target point, and initialize relevant parameters.

[0012] Step 2: Initialize the prediction network parameters ω and the target network parameters ω', and determine the current state S of the mobile robot.

[0013] Step 3: The mobile robot selects action a to execute with a probability of ε by using the improved artificial potential field gravity function as the action strategy, and selects action a with the maximum Q value to execute with a probability of 1-ε.

[0014] Step 4: Since the gravitational force of the artificial potential field is relatively large in the later stage of the algorithm, the mobile robot is prone to fall into the local minimum point. The problem of the mobile robot falling into the local minimum point is solved by guiding the action function with the goal.

[0015] Step 5: After the mobile robot executes the action a selected in steps 3 to 4, it obtains the current state reward value r, the next state S' and the next state reward value r' according to the Q value function and the improved reward function, and stores (S, a, r, r', S') as an experience sample in the corresponding experience pool.

[0016] Step 6: Samples are sampled from the experience pool D1 with probability ρ, and samples are sampled from the experience pool D2 with probability 1-ρ. Then, the priority p of all samples stored in the sample experience pools D1 and D2 is calculated using the sample priority formula. Then, the probability of each sample being sampled is calculated using the probability formula for sample sampling. Finally, samples are sampled from the experience pool according to probability.

[0017] Step 7: If the next state of the sampled sample is the target point state, the target Q value is the reward value of the target point state, otherwise the target Q value is the reward value of the target point state plus the discounted maximum Q value of the next state.

[0018] Step 8: Steps 6 to 7 are executed continuously, and the sampling sequence is repeated n times to obtain the final target Q value.

[0019] Step 9: Calculate the loss value according to the mean square error loss function formula, and use the mini-batch semi-gradient descent method to update the network parameters w of the prediction network.

[0020] Step 10: Assign the network parameters w of the prediction network to the network parameters w' of the target network at regular training steps, and plan the path from the starting point to the end point based on the action selected by the mobile robot at each step.

[0021] Step 11: If the position state of the mobile robot in the tth step does not reach the target point state, repeat steps 2 to 10. Otherwise, determine whether the number of training episodes reaches the total number of network training episodes. If it does not reach the total number of network training episodes, repeat steps 1 to 10. If it reaches the total number of network training episodes, end the training, and plan the path from the starting point to the end point based on the maximum Q value action selected by the mobile robot, and the algorithm ends.

[0022] Furthermore, the relevant parameters initialized in step 1 are as follows:

[0023] Determine the current position X of the mobile robot t =[x t ,y t ], the number of steps explored by the mobile robot is t, and the horizontal and vertical coordinates x of the mobile robot's position at the tth step t ,y t , the target point position is X tar =[x tar ,y tar ], the horizontal and vertical coordinates of the target point are xtar ,y tar Initialize the capacity of the positive and negative value sample experience pools D1 and D2, the greedy sampling strategy ε, the total number of episodes N for network training, the sample priority p, and the number of sampling sequences n.

[0024] Furthermore, the action a selected in step 3 for the mobile robot to perform next is specifically as follows:

[0025] Improved artificial potential field gravity function F att The size of (X) is:

[0026]

[0027] Where m is the threshold, L is the diagonal distance of the map environment, the direction of gravity is the direction of the mobile robot pointing to the target point, dis(X tar -X t ) is the current position of the mobile robot X t =[x t ,y t ] and the target point X tar =[x tar ,y tar ], the formula is as follows:

[0028]

[0029] From the above gravitational function F att (X) formula shows that the current position of the mobile robot is X t =[x t ,y t ] and the target point X tar =[x tar ,y tar ], the larger the distance between them, the smaller the gravitational force value. t =[x t ,y t ] and the target point X tar =[x tar ,y tar ]When the distance is less than the threshold m, the gravity value remains unchanged.

[0030] The selection of the maximum Q-value action is determined by the Q-value function, and the Q-value function update iteration is as follows:

[0031]

[0032] Among them, Q(S,a) is the Q value of the current state, S is the current state, a is the execution action, α(0≤α≤1) is the learning rate, r' is the reward value given by the environment to the next state, γ(0≤γ≤1) is the discount coefficient, S' is the next state, and a' is the action to be performed next.

[0033] Furthermore, the improved reward function in step 5 is as follows:

[0034] The improved reward function formula is as follows:

[0035]

[0036] Among them, λ(0<λ<1) is the discount factor of the reward function, X ne =[x ne ,y ne ] is the nearest neighboring point to the target point at the current position, x ne ,y ne It is the horizontal and vertical coordinates of the nearest neighboring point to the target point at the current position. is the set of points adjacent to the current position, is the distance set between the neighboring points and the target point, and i is the direction of the neighboring points at the current position. It is obtained from the following Euclidean distance formula:

[0037]

[0038] Furthermore, the sampling process in step 6 is specifically as follows:

[0039] The priority p of sample j j The formula is:

[0040]

[0041] In the formula, υ j is the number of times each sample is sampled. From the formula, we can get the priority p of the sample. j It will gradually decrease as the number of samples increases. Therefore, according to the priority of the sample, the probability P(j) of each sample being sampled is defined as:

[0042]

[0043] In the formula, size(D1 or D2) is the sample experience pool capacity. From the formula, it can be seen that the algorithm extracts samples from D1 and D2 according to the sample sampling probability P(j), which reduces the probability of high-value samples being repeatedly sampled and increases the probability of low-value samples being collected, thereby effectively alleviating the problem of loss value divergence and neural network overfitting.

[0044] Furthermore, the formula of the square error loss function and the formula of the small batch semi-gradient descent method in step 9 are as follows:

[0045] The mean square error loss function formula is as follows:

[0046]

[0047] Where, Ε(·) is the mean square error function, w' is the target network parameter, w is the predicted network parameter, Q(S,a,w) is the predicted network output, and Q(S',a',w') is the target network output. is the maximum Q value of the next state, is the target Q value.

[0048] The formula of the small batch semi-gradient descent method is as follows:

[0049]

[0050] in To find the gradient of Q(S,a,w).

[0051] Furthermore, the assignment formula of the network parameter w' of the target network in step 10 is as follows:

[0052] w'←τw+(1-τ)w'

[0053] Among them, τ (τ≤1), through the assignment formula of w', can effectively control the amplitude of each target Q value update and improve the stability of the mobile robot during learning.

[0054] Beneficial effects of the present invention:

[0055] 1. The path planning of the present invention introduces an improved artificial potential field gravity function and a target-guided action function into the action guidance strategy of the deep Q network algorithm, which effectively reduces the number of collisions during the algorithm training process and improves the convergence speed of the algorithm;

[0056] 2. The path planning of the present invention designs a piecewise reward function, which assigns a discounted reward value to the neighboring point closest to the target point at the current position, prompting the mobile robot to explore towards the target point, thereby making the planned path better;

[0057] 3. The path planning of the present invention improves the sampling strategy. By storing the priorities of all samples in the sample experience pool, the probability of samples in the experience pool being sampled decreases as the number of sampling times increases, thereby improving sample utilization and effectively solving the problems of loss value divergence and neural network overfitting during algorithm training. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will be further described below in conjunction with the accompanying drawings.

[0059] Figure 1 is a flow chart of the path planning method of the present invention;

[0060] Figure 2 These are the 8 directions in which the robot can move;

[0061] Figure 3 is the path planning result diagram of the present invention;

[0062] Figure 4 It is a performance diagram of the path planning algorithm training process of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] A mobile robot path planning based on an improved deep Q-network algorithm, such as Figure 1 As shown, path planning includes the following steps:

[0065] Step 1: Determine the position of the mobile robot and the target point, and initialize relevant parameters.

[0066] First determine the current position X of the mobile robot t =[x t ,y t ], the number of steps explored by the mobile robot is t, and the horizontal and vertical coordinates x of the mobile robot's position at the tth step t ,y t , the target point position is X tar =[x tar ,y tar ], the horizontal and vertical coordinates of the target point are x tar ,y tar Initialize the capacity of the positive and negative value sample experience pools D1 and D2, the greedy sampling strategy ε, the total number of episodes N for network training, the sample priority p, and the number of sampling sequences n.

[0067] Step 2: Initialize the prediction network parameters ω and the target network parameters ω', and determine the current state S of the mobile robot.

[0068] Step 3: The mobile robot selects action a to execute with a probability of ε by using the improved artificial potential field gravity function as the action strategy, and selects action a with the maximum Q value to execute with a probability of 1-ε; the specific method includes:

[0069] Improved artificial potential field gravity function F att The size of (X) is:

[0070]

[0071] Where m is the threshold, L is the diagonal distance of the map environment, the direction of gravity is the direction of the mobile robot pointing to the target point, dis(X tar -X t ) is the current position of the mobile robot X t =[x t ,y t ] and the target point X tar =[x tar ,y tar ], the formula is as follows:

[0072]

[0073] From the above gravitational function F att (X) formula shows that the current position of the mobile robot is X t =[x t ,y t ] and the target point X tar =[x tar ,y tar ], the larger the distance between them, the smaller the gravitational force value. t =[x t ,y t ] and the target point X tar =[x tar ,y tar ]When the distance is less than the threshold m, the gravity value remains unchanged.

[0074] The selection of the maximum Q-value action is determined by the Q-value function, and the Q-value function update iteration is as follows:

[0075]

[0076] Among them, Q(S,a) is the Q value of the current state, S is the current state, a is the execution action, α(0≤α≤1) is the learning rate, r' is the reward value given by the environment to the next state, γ(0≤γ≤1) is the discount coefficient, S' is the next state, and a' is the action to be performed next.

[0077] Step 4: Since the gravitational force of the artificial potential field is relatively large in the later stage of the algorithm, the mobile robot is prone to fall into the local minimum point. The problem of the mobile robot falling into the local minimum point is solved by guiding the action function with the goal.

[0078] If the mobile robot falls into a local minimum point during the exploration of the environment, the target-guided action function is used as the action-guiding strategy to perform the action.

[0079] like Figure 2 As shown, the movement direction of the mobile robot is divided into eight directions: upward, upper right, right, lower right, downward, lower left, left, and upper left, and is represented by 1, 2, 3, 4, 5, 6, 7, and 8. The target guidance action function is:

[0080]

[0081] When the mobile robot is trapped in a local minimum point, an action set A is selected. In the action set A, the mobile robot performs random actions to explore the environment, avoiding repeated execution of gravity-guided collision obstacle actions, so that the mobile robot escapes the local minimum point.

[0082] Step 5: After the mobile robot executes the action a selected in steps 3 to 4, it obtains the current state reward value r, the next state S' and the next state reward value r' according to the Q value function and the improved reward function, and stores (S, a, r, r', S') as an experience sample in the corresponding experience pool.

[0083] The improved reward function formula is as follows:

[0084]

[0085] Among them, λ(0<λ<1) is the discount factor of the reward function, X ne =[x ne ,y ne ] is the nearest neighboring point to the target point at the current position, x ne ,y ne It is the horizontal and vertical coordinates of the nearest neighboring point to the target point at the current position. is the set of points adjacent to the current position, is the distance set between the neighboring points and the target point, and i is the direction of the neighboring points at the current position. It is obtained from the following Euclidean distance formula:

[0086]

[0087] Since the improved reward function gives X ne =[x ne ,y ne ] A discounted reward value that makes Xne =[x ne ,y ne ] has a much larger Q value than other nearby points. The mobile robot executes the action with the maximum Q value with a probability of 1-ε. When the policy expectation of obtaining the maximum cumulative reward is obtained, the mobile robot will be guided to prioritize the action of moving toward the target point. It can be seen that this reward function will guide the mobile robot to find the shortest path with the least turning angle and redundant points.

[0088] If the current state reward value r in the experience sample is ≥ 0, the sample is stored in the positive value sample experience pool, otherwise the sample is stored in the negative value sample pool.

[0089] Step 6: Use the greedy sampling strategy to sample samples. Samples are sampled from the experience pool D1 with probability ρ, and samples are sampled from the experience pool D2 with probability 1-ρ. Then, the priority p of all samples stored in the sample experience pools D1 and D2 is calculated using the sample priority formula. Then, the probability of each sample being sampled is calculated using the probability formula for sample sampling. Finally, samples are sampled in the experience pool according to probability.

[0090] The priority p of sample j j The formula is:

[0091]

[0092] In the formula, υ j is the number of times each sample is sampled. From the formula, we can get the priority p of the sample. j It will gradually decrease as the number of samples increases. Therefore, according to the priority of the sample, the probability P(j) of each sample being sampled is defined as:

[0093]

[0094] In the formula, size(D1 or D2) is the sample experience pool capacity. From the formula, it can be seen that the algorithm extracts samples from D1 and D2 according to the sample sampling probability P(j), which reduces the probability of high-value samples being repeatedly sampled and increases the probability of low-value samples being collected, thereby effectively alleviating the problem of loss value divergence and neural network overfitting.

[0095] Step 7: If the next state of the sampled sample is the target point state, the target Q value is the reward value of the target point state, otherwise the target Q value is the reward value of the target point state plus the discounted maximum Q value of the next state.

[0096] Step 8: Steps 6 to 7 are executed continuously, and the sampling sequence is repeated n times to obtain the final target Q value.

[0097] Step 9: Calculate the loss value according to the mean square error loss function formula, and use the mini-batch semi-gradient descent method to update the network parameters w of the prediction network.

[0098] The mean square error loss function formula is as follows:

[0099]

[0100] Where, Ε(·) is the mean square error function, w' is the target network parameter, w is the predicted network parameter, Q(S,a,w) is the predicted network output, and Q(S',a',w') is the target network output. is the maximum Q value of the next state, is the target Q value.

[0101] The formula for the mini-batch semi-gradient descent method is as follows:

[0102]

[0103] in To find the gradient of Q(S,a,w).

[0104] Step 10: Assign the network parameters w of the prediction network to the network parameters w' of the target network at regular training steps, and plan the path from the starting point to the end point based on the action selected by the mobile robot at each step.

[0105] The assignment formula of the network parameter w' of the target network is as follows:

[0106] w'←τw+(1-τ)w'

[0107] Among them, τ (τ≤1), through the assignment formula of w', can effectively control the amplitude of each target Q value update and improve the stability of the mobile robot during learning.

[0108] Step 11: If the position state of the mobile robot in the tth step does not reach the target point state, repeat steps 2 to 10. Otherwise, determine whether the number of training episodes reaches the total number of network training episodes. If it does not reach the total number of network training episodes, repeat steps 1 to 10. If it reaches the total number of network training episodes, end the training, and plan the path from the starting point to the end point based on the maximum Q value action selected by the mobile robot, and the algorithm ends.

[0109] like Figure 3 As shown in the figure, the path trajectory planned by the mobile robot, the line is the path from the starting point to the end point planned by the mobile robot, the path length is short, the path smoothness is high, the turning angle during the robot movement is reduced, and a better path is obtained, so that the mobile robot path planning can be better realized in the real environment.

[0110] like Figure 4 As shown in the figure, during the training process of the heuristic deep Q-network algorithm for mobile robot path planning, the peak number of training steps did not reach 2000 steps in the first 100 sections. After 100 sections, the algorithm converged quickly and tended to be flat, with almost no network model oscillation, which effectively alleviated the problems of loss value divergence and neural network overfitting.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0112] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A mobile robot path planning based on an improved deep Q network algorithm, characterized in that: The path planning comprises the following steps: Step 1: Determine the position of the mobile robot and the target point, and initialize relevant parameters; Step 2: Initialize the prediction network parameters w and target network parameters w', and determine the current state S of the mobile robot; Step 3: The mobile robot selects action a to execute with a probability of ε by using the improved artificial potential field gravity function as the action strategy, and selects action a with the maximum Q value to execute with a probability of 1-ε, where ε represents the greedy sampling strategy; Step 4: Since the gravitational force of the artificial potential field is relatively large in the later stage of the algorithm, the mobile robot is prone to fall into the local minimum point. The problem of the mobile robot falling into the local minimum point is solved by guiding the action function with the goal: Step 5: After the mobile robot executes the action a selected in steps 3 to 4, it obtains the current state reward value r, the next state S' and the next state reward value r' according to the Q value function and the improved reward function, and stores (S, a, r, r', S') as an experience sample in the corresponding experience pool; Step 6: Samples are sampled from the experience pool D1 with probability ρ, and samples are sampled from the experience pool D2 with probability 1-ρ. Then, the priority p of all samples stored in the sample experience pools D1 and D2 is calculated using the sample priority formula. Then, the probability of each sample being sampled is calculated using the sample sampling probability formula. Finally, samples are sampled from the experience pool according to the probability. Step 7: If the next state of the sampled sample is the target point state, the target Q value is the reward value of the target point state, otherwise the target Q value is the reward value of the target point state plus the discounted maximum Q value of the next state; Step 8: Steps 6 to 7 are executed continuously, and the sampling sequence is repeated n times to obtain the final target Q value; Step 9: Calculate the loss value according to the mean square error loss function formula, and use the mini-batch semi-gradient descent method to update the network parameters w of the prediction network; Step 10: Assign the network parameters w of the prediction network to the network parameters w' of the target network at regular training steps, and plan the path from the starting point to the end point according to the action selected by the mobile robot at each step; Step 11: If the position state of the mobile robot in the tth step does not reach the target point state, repeat steps 2 to 10. Otherwise, determine whether the number of training episodes reaches the total number of network training episodes. If it does not reach the total number of network training episodes, repeat steps 1 to 10. If it reaches the total number of network training episodes, end the training, and plan the path from the starting point to the end point according to the maximum Q value action selected by the mobile robot, and the algorithm ends. The improved reward function in step 5 is as follows: The improved reward function formula is as follows: Where λ is the discount factor of the reward function, where 0<λ<1, X ne =[x ne ,y ne ] is the nearest neighboring point to the target point at the current position, x ne ,y ne It is the horizontal and vertical coordinates of the nearest neighboring point to the target point at the current position. is the set of points adjacent to the current position, is the distance set between the neighboring points and the target point, i is the direction of the neighboring point at the current position; It is obtained from the following Euclidean distance formula:

2. The mobile robot path planning based on the improved deep Q network algorithm according to claim 1 is characterized in that: The relevant parameters initialized in step 1 are as follows: Determine the current position X of the mobile robot t =[x t ,y t ], the number of steps explored by the mobile robot is t, and the horizontal and vertical coordinates x of the mobile robot's position at the tth step t ,y t , the target point position is X tar =[x tar ,y tar ], the horizontal and vertical coordinates of the target point are x tar ,y tar ; Initialize the capacity of the positive and negative value sample experience pools D1 and D2, the greedy sampling strategy ε, the total number of episodes N for network training, the sample priority p, and the number of sampling sequences n.

3. The mobile robot path planning based on the improved deep Q network algorithm according to claim 1 is characterized in that: The action a selected in step 3 for the next step of the mobile robot is as follows: Improved artificial potential field gravity function F att The size of (X) is: Where m is the threshold, L is the diagonal distance of the map environment, the direction of gravity is the direction of the mobile robot pointing to the target point, dis(X tar -X t ) is the current position of the mobile robot X t =[x t ,y t ] and the target point X tar =[x tar ,y tar ], the formula is as follows: From the above gravitational function F att (X) formula shows that the current position of the mobile robot is X t =[x t ,y t ] and the target point X tar =[x tar ,y tar ], the larger the distance between them, the smaller the gravitational force value. t =[x t ,y t ] and the target point X tar =[x tar ,y tar ]When the distance is less than the threshold m, the gravity value remains unchanged; The selection of the maximum Q-value action is determined by the Q-value function, and the Q-value function update iteration is as follows: Among them, Q(S,a) is the Q value of the current state, S is the current state, a is the execution action, α is the learning rate, where 0≤α≤1, r' is the reward value given by the environment to the next state, γ is the discount coefficient, where 0≤γ≤1, S' is the next state, and a' is the action to be performed next.

4. The mobile robot path planning based on the improved deep Q network algorithm according to claim 1 is characterized in that: The sampling process in step 6 is as follows: The priority p of sample j j The formula is: In the formula, υ j is the number of times each sample is sampled; from the formula, we can get the priority p of the sample j It will gradually decrease as the number of samples increases. Therefore, according to the priority of the sample, the probability P(j) of each sample being sampled is defined as: In the formula, size (D1 or D2) is the capacity of the sample experience pool; From the formula, we can see that the algorithm extracts samples from D1 and D2 according to the sample sampling probability P(j), which reduces the probability of high-value samples being repeatedly sampled and increases the probability of low-value samples being collected, thereby effectively alleviating the problems of loss value divergence and neural network overfitting.

5. The mobile robot path planning based on the improved deep Q network algorithm according to claim 1 is characterized in that: The formula of the square error loss function and the formula of the small batch semi-gradient descent method in step 9 are as follows: The mean square error loss function formula is as follows: Where, Ε(·) is the mean square error function, w' is the target network parameter, w is the predicted network parameter, Q(S,a,w) is the predicted network output, and Q(S',a',w') is the target network output. is the maximum Q value of the next state, is the target Q value; The formula of the small batch semi-gradient descent method is as follows: in To find the gradient of Q(S,a,w).

6. The mobile robot path planning based on the improved deep Q network algorithm according to claim 1 is characterized in that: The assignment formula of the network parameter w' of the target network in step 10 is as follows: w'←τw+(1-τ)w' Among them, τ≤1, the assignment formula of w' can effectively control the amplitude of each target Q value update, and improve the stability of the mobile robot during learning.

Images