Mobile robot path planning method based on reinforcement learning

Abstract

The invention discloses a mobile robot path planning method based on reinforcement learning. The method comprises the following steps: firstly, designing a double-experience pool to store experiences in positive and negative aspects respectively, and setting a sampling proportion of the two experience pools to balance a relationship between exploration so as to accelerate a convergence rate; secondly, the generation direction of the repulsive force is optimized, an artificial potential field method is enhanced, an improved APF method is used for replacing a Double DQN random exploration strategy, and the convergence speed is further increased; and finally, eliminating redundant nodes by analyzing the spatial relationship between the path and the obstacle, and smoothing the path by using a Bezier curve, thereby improving the quality of the path. According to the invention, the learning network training process is optimized, so that the robot can make the optimal path planning action.

Description

Technical Field

[0001] The present invention relates to the fields of automation and mechanical control, and in particular to a mobile robot path method based on reinforcement learning. Background Art

[0002] Mobile robot path planning is a key research area in robotics. It involves finding the optimal or feasible path from a starting point to a destination in a given environment. Path planning technology is widely used in fields such as autonomous driving, drone navigation, and industrial robotics. With the continuous advancement of artificial intelligence and computer technology, path planning methods have also experienced significant progress, primarily in the following areas: The development and advancement of traditional methods: The earliest path planning methods were primarily based on graph search and optimization techniques, such as the A* algorithm and the Dijkstra algorithm. These methods are suitable for known environments and can effectively plan in environments with static obstacles, but perform poorly in dynamic environments. Real-time path planning: With the increasing complexity of environments, path planning in real-time dynamic environments has become a research hotspot. Methods such as the dynamic window algorithm (DWA) and rapidly expanding random trees (RRT) have been proposed, capable of responding to and adjusting paths in real time in constantly changing environments. Applications of artificial intelligence and machine learning: In recent years, artificial intelligence technologies such as deep learning and reinforcement learning have begun to be applied to path planning. They can learn environmental characteristics from large amounts of data and adaptively optimize planning strategies. For example, path planning methods based on deep reinforcement learning can automatically select paths in complex environments by adjusting robot behavior through trial and error. Multi-robot collaborative planning: With the development of multi-robot systems, coordinating multiple robots to complete tasks has become a new challenge in path planning. In the future, with the improvement of computing power and the continuous optimization of algorithms, mobile robot path planning technology will become more intelligent, real-time, and able to cope with more complex and dynamic environments, providing robots with stronger autonomy in practical applications.

[0003] Patent publication number CN113159432A, "A Multi-Agent Path Planning Method Based on Deep Reinforcement Learning," is a distributed path planning method that inputs local observation information of a single agent into a neural network. A graph neural network is used to transmit information between agents, and the neural network is trained to approximate a policy function, thereby outputting a movement policy. Patent publication number CN112325897B, "A Path Planning Method Based on Heuristic Deep Reinforcement Learning," utilizes the Actor-Critic reinforcement learning algorithm for training and employs the classic heuristic algorithm A* as the reinforcement learning heuristic function to construct heuristic information. Summary of the Invention

[0004] The present invention proposes a robot path planning method based on an improved dual-depth Q-learning algorithm, aiming to solve the problems of slow convergence and poor path quality in traditional path planning. This method designs a dual experience pool to store experiences of positive rewards and negative rewards respectively, optimizes the experience replay mechanism, and improves training efficiency. At the same time, the improved artificial potential field method accelerates the convergence speed of the algorithm by optimizing the direction of repulsive force generation and using the distance factor to replace random exploration. Finally, the path smoothing optimization analyzes the spatial relationship between the path and obstacles, eliminates redundant nodes, and uses Bezier curves for path smoothing, thereby improving the path quality. Compared with the traditional dual-depth Q-learning algorithm, the algorithm of the present invention has been significantly improved in both convergence speed and path quality.

[0005] The technical solution adopted by the present invention is: a mobile robot path planning method based on reinforcement learning, comprising the following steps:

[0006] Step 1: Build a simulation platform to simulate a grid environment, where the grid includes robots, obstacles, and traversable areas;

[0007] Step 2: Establish the robot's four movements in the grid environment: up, down, left, and right;

[0008] Step 3: Establish two neural network models, a training network and a target network; the training network and the target network have the same structure but different weight parameters;

[0009] Step 4: Use the current grid image and the robot's current coordinate position as the input of the convolutional neural network, and use the pixel Q values ​​of the four action predictions as the output, and filter out the action with the maximum Q value at that moment;

[0010] Step 5: Optimize the repulsive force potential field by introducing the distance from the target point, and calculate the total potential field and the direction of the resultant force;

[0011] Step 6: The robot selects a behavior based on the direction of the force of the potential field with probability ε, and selects the best behavior currently known based on the neural network with probability 1-ε;

[0012] Step 7: When the robot successfully reaches the goal, a positive reward is given; when the robot collides with an obstacle, a negative reward is given;

[0013] Step 8: Create two experience pools and store the positive reward experience and negative reward experience into the positive and negative experience pools respectively according to the sign of r in (s,a,s′,r);

[0014] Among them, s represents the current state, a represents the action taken in the current state, s' is the next state after taking the action, and r is the reward obtained from the action;

[0015] Step 9: Based on reinforcement learning, the mobile robot path planning autonomous Q value prediction is performed, and the neural network weights are updated through the dual experience pool data. The weight parameters of the training network are copied to the target network at regular intervals.

[0016] Step 10: After training, output the planned route.

[0017] Furthermore, the step 1 specifically includes the following steps:

[0018] Step 1-1: Use MATLAB software to build a simulation environment to simulate a mobile robot in a real environment;

[0019] Step 1-2: Set a 10×10 grid area in the simulation environment as the working area. Randomly generate black obstacles of different proportions in this area. Obstacles in the grid are represented by black pixels, and passable areas are represented by white pixels.

[0020] Step 1-3: Define the action space A = {′right′,′up′,′left′,′down′}, provide the state information of the current environment through the map matrix, and track the current position by updating the state information.

[0021] Furthermore, the step 3 specifically includes the following steps:

[0022] Step 3-1: Create an input layer that accepts a 10x10 matrix as input and converts it into an image with a shape of [10, 10, 1].

[0023] Step 3-2: Create convolutional layers: The neural network contains three convolutional layers, each followed by a batch normalization layer and a ReLU activation function:

[0024] Convolutional layer 1: 3x3 convolution kernel, 16 output channels;

[0025] Convolutional layer 2: 3x3 convolution kernel, 32 output channels;

[0026] Convolutional layer 3: 3x3 convolution kernel, 64 output channels;

[0027] Step 3-3: Building fully connected layers: The final part of the network consists of two fully connected layers:

[0028] First layer: 64 neurons;

[0029] Second layer: 4 neurons, outputting the Q value of each action;

[0030] Step 3-4: Create a regression layer: The output layer is a regression layer that generates the Q value for each action.

[0031] Furthermore, the step 4 specifically includes the following steps:

[0032] Step 4-1: Use a convolutional neural network to extract a 10×10 grid image and the robot’s row and column indices within the grid, enabling the neural network to better understand the spatial layout of the environment and the robot’s current position.

[0033] Step 4-2: Use the value function approximation method, and use the optimal action value function θ as the parameter θ * (s,a) is the goal so that the action value function Q(s,a,θ) approaches it, as shown in the following formula,

[0034] Q(s,a,θ)≈Q * (s,a);

[0035] Step 4-3: Output a vector of length four, corresponding to the Q-value predictions of the four actions; each Q-value represents the expected reward of the robot taking the corresponding action in the current state.

[0036] Furthermore, the step 5 specifically includes the following steps:

[0037] Step 5-1: Construct a gravitational potential field; determine the gravitational potential field by calculating the distance between the current position and the target point, and attract the robot to move toward the target point:

[0038]

[0039] Where: represents K att represents the gravitational gain, d 2 (X,X T ) represents the current point X and the target point X T The square of the Euclidean distance between them;

[0040] Step 5-2: Construct a repulsive potential field to prevent the robot from colliding with obstacles:

[0041]

[0042] Among them: K rep represents the repulsion gain coefficient, d(X,X o ) represents the distance between the current point X and the obstacle X o The Euclidean distance between them, d0 represents the repulsion threshold;

[0043] Step 5-3: Improve the repulsive force potential field and introduce the distance from the target point to optimize the repulsive force field:

[0044]

[0045] Where: λ is any positive real number;

[0046] Step 5-4: Calculate the total potential field and the magnitude of the resultant force direction; the global potential field function is the vector superposition of the gravitational field function and the repulsive field function, defined as:

[0047] U(X)=U att (X)+U rep (X);

[0048] Step 5-5: Calculate the negative gradient of the total potential field to obtain the magnitude and direction of the resultant force acting on the robot;

[0049] Step 5-6: Discretize the resultant force into four directions A = {'right', 'up', 'left', 'down'} according to reasonable directions.

[0050] Furthermore, the step 5-5 specifically includes the following steps:

[0051] Step 5-5-1: Take the negative gradient of the gravitational potential field to get the gravitational force:

[0052] F aat (X) = -K att d(X,X T );

[0053] Step 5-5-2: Find the negative gradient of the improved repulsive potential field to obtain the repulsive force: o ) Take the negative gradient and get

[0054]

[0055] For d(X,X o )Find the negative gradient and we get:

[0056]

[0057] Then the net force on the robot in the improved repulsive field is:

[0058]

[0059] Step 5-5-3: The net force acting on the robot is the vector superposition of the gravitational force and the repulsive force:

[0060] F(X)=F att (X)+F rep1 +F rep2 .

[0061] Furthermore, the action reward strategy in step 7 includes a primary reward and a secondary reward, wherein the primary reward is:

[0062] When the robot successfully reaches the goal, a positive reward of +10 is given;

[0063] When the robot collides with an obstacle, a negative reward of -1 is given;

[0064] Give the robot a negative reward of -0.1 for each step;

[0065] The main reward formula is:

[0066]

[0067] The secondary rewards are:

[0068] When the distance between the robot's current position and the goal decreases, a positive reward of +0.5 is given;

[0069] When the distance between the robot's current position and the goal increases, a negative reward of -0.5 is given;

[0070] The secondary reward formula is:

[0071]

[0072] Where ΔD is the difference between the Euclidean distance between the current position and the target point and the Euclidean distance between the next position and the target point.

[0073] Furthermore, the step 8 specifically includes the following steps:

[0074] Step 8-1: Establish positive and negative experience pools;

[0075] Step 8-2: In the current state s, the robot selects an action a according to the current strategy. After executing action a, the environment returns a new state and immediate reward r, which will serve as a reference for subsequent learning;

[0076] Step 8-3: Store the experience points into two experience pools based on the positive or negative value of the reward at each step;

[0077] Step 8-4: Check the experience pool capacity and remove the oldest experience; if the positive experience pool or negative experience pool reaches the upper limit, remove the oldest experience;

[0078] Step 8-5: Calculate the number of samples from the positive experience pool and the negative experience pool, and set the ratio by the hyperparameter ρ; the specific steps are:

[0079] Step 8-5-1: Sampling data of the positive experience pool: Calculate the number of positive experiences sampled according to the ratio ρ:

[0080] n p =n×ρ

[0081] Among them, n is the total number of samples, ρ is the sampling ratio parameter, n p The number of samples of positive reward experience for each training process;

[0082] Step 8-5-2: Sampling data of negative experience pool: The number of negative reward experience can be obtained by subtracting the number of positive reward experience from the total sample data n experience pool:

[0083] n n =nn p

[0084] Among them, n is the total number of samples, n n The number of negative reward experiences sampled during each training process;

[0085] Step 8-5-3: Send the experience sampled from the two experience pools into the network for training.

[0086] Furthermore, the step 9 specifically includes the following steps:

[0087] Step 9-1: Update the neural network weights using the dual experience pool data. The experience (s, a, r, s′) during the robot's interaction with the environment is stored in the positive and negative experience pools. The specific steps are as follows:

[0088] Step 9-1-1: Sampling from the experience pool; Each time the network is updated, a batch of experience samples are randomly taken from the two experience pools. Each sample contains the current state s, the action a taken in the current state, the reward r of the action, and the next state s′ after taking the action;

[0089] Step 9-1-2: Calculate the target Q value; use the current network to select the action:

[0090]

[0091] Use the target network to calculate the target Q value:

[0092]

[0093] Where γ is the discount factor; r t+1 represents the expected reward at time t+1; Represents the objective function value, which is obtained by the reward value of the current action and the Q value of the best action corresponding to the next state; θ t Represents the parameters of the current neural network, θ t ′ represents the parameters of the target neural network;

[0094] Step 9-2: Neural network weight update, using the difference between the Q value calculated by the current network and the Q value evaluated by the target network to update the parameters of the current network;

[0095] Use the mean square error loss function to calculate the error between the predicted Q value and the target Q value to train the neural network:

[0096]

[0097] In the formula, is the target Q value, Is the current network according to the state s t and action a t Predicted Q value;

[0098] Optimize the parameters θ of the current network by gradient descent t , in order to minimize, the weight update is completed through the following formula:

[0099]

[0100] in, Indicates that the partial derivative of the network layer is obtained by chain derivation rule; θ t Represents the parameters of the current neural network.

[0101] Furthermore, the step 10 specifically includes the following steps:

[0102] Step 10-1: Let the path node set be {P k |k=1,2,3,…,n}, P1 represents the starting point, P n Represents the target point; the specific steps are:

[0103] Step 10-1-1: Starting from P2, solve the vector and If the angle between the two vectors is 0, it means that P n-1 ,P n ,P n+1 If the three path nodes are on the same line, remove the redundant collinear node P n , repeat the process until there are no sequence points to be removed;

[0104] Step 10-1-2: In the remaining nodes, calculate the vector and If the angle between these two vectors is not 0, it indicates that the path node P a-1 、P a 、P a+1 are not on the same straight line. At this time, solve the line segment L(P a-1 P a+1 The equation of , and judge whether the line segment intersects with the obstacle. If the line segment does not intersect with the obstacle, then P a is a redundant turning point and should be removed; if the line segment intersects an obstacle, then P a It is a necessary turning point and should be retained; repeat the above process until there are no sequence points that need to be removed;

[0105] Step 10-2: After removing redundant nodes, use a third-order Bezier curve to further smooth the path:

[0106] B(t)=(1-t) 3 P0+3(1-t) 2 tP1+3(1-t)t 2 P2+t 3 P3

[0107] Among them, P0 and P3 are the starting point and end point, P1 and P2 are intermediate control points, and t∈[0,1].

[0108] The beneficial effects of the present invention are:

[0109] 1. Dual experience pool design: By designing a dual experience pool to store positive reward and negative reward experience separately, the experience replay mechanism is optimized, thereby accelerating exploration and improving training efficiency.

[0110] 2. Improved artificial potential field method: By optimizing the direction of repulsive force and using the distance factor instead of random exploration, the convergence speed of the algorithm is accelerated.

[0111] 3. Path smoothing optimization: By analyzing the spatial relationship between the path and obstacles, eliminating redundant nodes, and using Bezier curves for path smoothing, the path quality is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 A flow chart of the invented method;

[0113] Figure 2 A simulation map constructed for the present invention;

[0114] Figure 3 This is a diagram of the convolutional neural network structure used in the present invention;

[0115] Figure 4 Schematic diagram of the artificial potential field method before and after improvement in the present invention;

[0116] Figure 5 This is a schematic diagram of the path smoothing process of the present invention;

[0117] Figure 6 The simulation results of the present invention are Figure 1 ;

[0118] Figure 7 The simulation results of the present invention are Figure 2 . DETAILED DESCRIPTION

[0119] The present invention will be further described below with reference to the accompanying drawings.

[0120] like Figure 1 As shown, the present invention is a mobile robot path planning method based on reinforcement learning, comprising the following steps:

[0121] Step 1: Establish a grid map simulation platform for robot path planning. Use MATLAB software to build a simulation platform to simulate a grid environment. The grid includes the robot, obstacles, and traversable areas. The specific steps are as follows:

[0122] Step 1-1: Use MATLAB software to build a map matrix and simulate the movement of the robot in a real environment to control the robot's movement.

[0123] Step 1-2: Convert the map matrix into image format and create a 10×10 grid area as the working area. Obstacles can be randomly generated in this area and the obstacle ratio can be adjusted by yourself. Obstacles in the grid are represented by black pixels and passable areas are represented by white pixels. Figure 2 As shown, set the robot's starting point to the blue lower left corner and the end point to the red upper right corner.

[0124] Step 1-3: Define the action space A = {′right′,′up′,′left′,′down′}, provide the state information of the current environment through the map matrix, and track the current position by updating the state information.

[0125] Step 2: Establish the robot's four movements in the grid environment: up, down, left, and right.

[0126] Step 3: Establish two convolutional neural network models, namely the training network and the target network. The structures of the above two network models are exactly the same, but the weight parameters are different. The specific steps are:

[0127] Step 3-1: Create an input layer. The input layer accepts a 10x10 matrix as input and converts it into an image with a shape of [10,10,1], that is, a 10x10 single-channel image.

[0128] Step 3-2: Create a convolutional layer. The neural network contains three convolutional layers, each followed by a batch normalization layer and a ReLU activation function:

[0129] Convolutional layer 1: 3x3 convolution kernel, 16 output channels;

[0130] Convolutional layer 2: 3x3 convolution kernel, 32 output channels;

[0131] Convolutional layer 3: 3x3 convolution kernel, 64 output channels.

[0132] Step 3-3: Building fully connected layers: The final part of the network consists of two fully connected layers:

[0133] First layer: 64 neurons;

[0134] The second layer has 4 neurons, which output the Q value of each action.

[0135] Step 3-4: Establish a regression layer. The output layer is a regression layer that generates the Q value of each action.

[0136] like Figure 3 As shown in the figure, the CNN network inputs a 10×10 grid map and the current position status of the robot, derives environmental features through three 3×3 layers, integrates the features through a fully connected layer, and finally outputs Q values ​​for the four corresponding directions of up, down, left, and right. The robot can choose the best moving direction based on the maximum Q value.

[0137] Step 4: Improve the Double DQN algorithm, using a dual-network architecture to perform action selection and policy evaluation separately to address the problem of overestimation of Q values ​​in traditional deep Q networks (DQNs), thereby improving the accuracy and efficiency of path planning. Q value prediction is achieved through two independent neural networks: the current network is used to select actions and the target network is used to evaluate actions. The convolutional neural network uses the current grid image and the current coordinate position of the robot as inputs, and the pixel Q values ​​of the four action predictions as outputs, screening out the behavior with the maximum Q value at that moment. The specific steps are as follows:

[0138] Step 4-1: Input the two-dimensional matrix image of the environment grid and the coordinates of the robot's current position, that is, the row and column index of the robot in the grid; extract the features of these inputs through the convolutional neural network, so that the network can better understand the spatial layout of the environment and the current position information of the robot.

[0139] Step 4-2: The value function approximation method is as follows:

[0140] Q(s,a,θ)≈Q * (s,a)

[0141] A neural network with parameters θ is used to find the optimal action value function θ * (s,a) is the target so that the action-value function Q(s,a,θ) approaches it.

[0142] Step 4-3: Output a vector of length four, corresponding to the Q-value predictions of the four actions. Each Q-value represents the expected reward of the robot taking the corresponding action in the current state.

[0143] Step 5: The artificial potential field method optimizes the repulsive force by introducing the distance from the target point, making it less likely for the robot to fall into a local minimum when approaching the target point, thereby reaching the target point more efficiently; the force diagrams of the artificial potential field method before and after the improvement are shown in the figure below. Figure 4 The specific steps are as follows:

[0144] Step 5-1: Construct a gravitational potential field to attract the robot to move toward the target point. The target point generates a gravitational force that attracts the robot toward it. The gravitational potential field is determined by calculating the distance between the current position and the target point, as shown in the following formula:

[0145]

[0146] Among them, K att represents the gravitational gain, d 2 (X,X T ) represents the square of the Euclidean distance between the current point and the target point.

[0147] Step 5-2: Construct a repulsive force potential field. The repulsive force potential field is used to prevent the robot from colliding with obstacles, as shown in the following formula:

[0148]

[0149] Among them: K rep represents the repulsion gain coefficient, d(X,X o ) represents the distance between the current point X and the obstacle X o d0 represents the repulsion threshold. When the threshold is greater than the threshold, obstacles within the threshold distance will not generate repulsion, thereby affecting the intelligent system.

[0150] The obstacle exerts a repulsive force on the robot. When the robot approaches the obstacle, the repulsive force becomes stronger, pushing the robot away from the obstacle.

[0151] Step 5-3: Improve the repulsive force potential field as shown below:

[0152]

[0153] Here, λ is any positive real number.

[0154] Traditional artificial potential fields are prone to falling into local minima when approaching the target point, causing the target to be unreachable; therefore, the present invention improves the artificial potential field method by optimizing the repulsive field by introducing the distance from the target point, so that the robot can successfully descend the gradient when approaching the target point, thereby reaching the target point more efficiently.

[0155] Step 5-4: Calculate the total potential field and the magnitude of the resultant force direction. The global potential field function is the vector superposition of the gravitational field function and the repulsive field function, and is defined as:

[0156] U(X)=U att (X)+U rep (X)

[0157] Step 5-5: Calculate the negative gradient of the total potential field to obtain the magnitude and direction of the resultant force acting on the robot. The specific steps are:

[0158] Step 5-5-1: Take the negative gradient of the gravitational potential field to get the gravitational force:

[0159] F aat (X) = -K att d(X,X T );

[0160] Step 5-5-2: Find the negative gradient of the improved repulsive potential field to obtain the repulsive force: o )Find the negative gradient and we get:

[0161]

[0162] For d(X,X o )Find the negative gradient and we get:

[0163]

[0164] Then the net force on the robot in the improved repulsive field is:

[0165]

[0166] Step 5-5-3: The net force acting on the robot is the vector superposition of the gravitational force and the repulsive force:

[0167] F(X)=F att (X)+F rep1 +F rep2

[0168] Step 5-6: Discretize the four directions A = {'right', 'up', 'left', 'down'} according to the direction of the resultant force.

[0169] Step 6: The action selection strategy uses the ε-greedy strategy combined with heuristic search and the artificial potential field; the robot selects the behavior according to the direction of the force in the potential field with a certain probability ε, and selects the best behavior currently known according to the neural network with a probability of 1-ε.

[0170] Step 7: Establish an action reward strategy. The robot chooses four directions to move. When the robot successfully reaches the goal, it is given a larger positive reward of +10 to ensure that the robot has sufficient motivation to pursue the goal. When the robot collides with an obstacle or goes out of bounds, it is given a negative reward of -1 to prevent the robot from colliding with objects. To encourage the robot to choose the shortest path, a small penalty of -0.1 can be applied to each step to ensure that the robot does not make meaningless detours, allowing the robot to walk to reduce the path length and improve the efficiency of path planning. The reward formula is as follows:

[0171]

[0172] In addition, the present invention also designs a secondary reward, the formula is:

[0173]

[0174] Where ΔD is the difference between the Euclidean distance between the current position and the target point and the Euclidean distance between the next position and the target point.

[0175] The reward is adjusted based on the distance between the robot's current position and the target. When the distance to the target decreases, the reward is a positive value of +0.5; when it moves away from the target, the reward is a negative value of -0.5, which encourages the robot to move closer to the target.

[0176] Establish two experience pools. Based on the sign of r in (s, a, s′, r), positive reward experiences and negative reward experiences are stored in the positive and negative experience pools respectively. This facilitates the neural network to update the neural network through the experience pool data. Where s is the current state, a is the action taken, r is the reward obtained, and the next state s′. By sampling from the experience pool, the neural network training is made more stable. The following steps are included:

[0177] Step 8-1: Create positive and negative experience pools. The size of the experience pools can be set by yourself.

[0178] Step 8-2: In the current state s, the robot selects an action a according to the current strategy. After executing action a, the environment returns a new state and immediate reward r, which will serve as a reference for subsequent learning;

[0179] Step 8-3: Store the experience points into two experience pools based on the positive or negative value of the reward at each step;

[0180] Step 8-4: Check the experience pool capacity and remove the oldest experience; if the positive experience pool or negative experience pool reaches the upper limit, remove the oldest experience;

[0181] Step 8-5: Calculate the number of samples from the positive experience pool and the negative experience pool, and set the ratio by the hyperparameter ρ; the specific steps are:

[0182] Step 8-5-1: Sampling data of the positive experience pool: Calculate the number of positive experiences sampled according to the ratio ρ:

[0183] n p =n×ρ

[0184] Among them, n is the total number of samples, ρ is the sampling ratio parameter, n p The number of samples of positive reward experience for each training process;

[0185] Step 8-5-2: Sampling data of negative experience pool: The number of negative reward experience can be obtained by subtracting the number of positive reward experience from the total sample data n experience pool:

[0186] n n =nn p

[0187] Among them, n is the total number of samples, n n The number of negative reward experiences sampled during each training process;

[0188] Step 8-5-3: Send the experience sampled from the two experience pools into the network for training.

[0189] Step 9: Autonomous Q-value prediction for mobile robot path planning based on reinforcement learning, and updating of neural network weights using dual experience pool data, including the following steps:

[0190] Step 9-1: Update the neural network weights using the dual experience pool data. The experience (s, a, r, s′) during the robot's interaction with the environment is stored in the positive and negative experience pools. The specific steps are as follows:

[0191] Step 9-1-1: Sampling from the experience pool. Each time the network is updated, a batch of experience samples will be randomly sampled from the two experience pools according to step 8. Each sample contains the current state s, the action a taken, the reward r obtained, and the next state s′.

[0192] Step 9-1-2: Calculate the target Q value and select the action using the current network:

[0193]

[0194] Use the target network to calculate the target Q value:

[0195]

[0196] Where γ is the discount factor; r t+1 represents the expected reward at time t+1; Represents the objective function value, which is obtained by the reward value of the current action and the Q value of the best action corresponding to the next state; θ t Represents the parameters of the current neural network, θ t ′ represents the parameters of the target neural network.

[0197] Step 9-2: Neural network weights are updated. The difference between the Q value calculated by the current network and the Q value evaluated by the target network is used to update the parameters of the current network.

[0198] Use the mean square error loss function to calculate the error between the predicted Q value and the target Q value to train the neural network:

[0199]

[0200] In the formula, is the target Q value, Is the current network according to the state s t and action a t Predicted Q value;

[0201] Optimize the parameters θ of the current network by gradient descent t , in order to minimize, the weight update is completed through the following formula:

[0202]

[0203] in, Indicates that the partial derivative of the network layer is obtained by chain derivation rule; θ t Represents the parameters of the current neural network.

[0204] Step 10: After the training is completed, use the trained model to output the planned route; the path smoothing optimization uses the Bezier curve method to smooth the planned path to reduce the corners and length of the path and improve the smoothness and operability of the path; the path smoothing process is as follows Figure 5 As shown, the specific steps include:

[0205] Step 10-1: Let the path node set be {P k |k=1,2,3,…,n}, P1 represents the starting point, P n Represent the target point. The specific steps include:

[0206] Step 10-1-1: Starting from P2, solve the vector and If the angle between the two vectors is 0, it means that P n-1 ,P n ,P n+1 If the three path nodes are on the same line, remove the redundant collinear node P n , repeating the process until there are no more sequence points to remove.

[0207] Step 10-1-2: In the remaining nodes, calculate the vector and If the angle between these two vectors is not 0, it indicates that the path node P a-1 、P a 、P a+1 are not on the same straight line. At this time, solve the line segment L(P a-1 P a+1 ) and determine whether the line segment intersects with the obstacle; if the line segment does not intersect with the obstacle, then Pa is a redundant turning point and should be removed; if the line segment intersects an obstacle, then P a It is a necessary turning point and should be retained. Repeat the above process until there are no sequence points that need to be removed.

[0208] Step 10-2: After removing redundant nodes, use a third-order Bezier curve to further smooth the path. The formula is as follows:

[0209] B(t)=(1-t) 3 P0+3(1-t) 2 tP1+3(1-t)t 2 P2+t 3 P3

[0210] Among them, P0 and P3 are the starting point and end point; P1 and P2 are intermediate control points; t∈[0,1].

[0211] The system of the present invention is compared with the traditional DDQN system in the same simulation environment. The training results are shown in Figure 2. Figure 6 and Figure 7 As shown in Figure 2, two metrics are selected for evaluation: the planned path length and the change in the number of robot steps during training. The change in the number of robot steps during training can reflect the speed of convergence. The training batch size is 500.

[0212] A comparison of the two indicators of the present invention and the DDQN system is shown in Table 1. Both the planned path length and the training convergence speed are higher than those of the traditional DDQN system. This shows that the system designed by the present invention has better task completion results.

[0213] Table 1 Comparison of indicators

[0214]

[0215] Simulation results in the MATLAB environment show that compared with the original DDQN algorithm, the method of the present invention has a 52.01% faster convergence speed and a 27.39% reduction in path length. The method of the present invention has a faster convergence speed and generates smoother and shorter paths.

[0216] In summary, the present invention uses MATLAB to build a 10x10 map matrix, and then converts it into a picture form to represent the map of the maze, where 0 represents a passable path and 1 represents an obstacle; the obstacle ratio and the starting and ending points of the maze environment can be customized. The present invention further calculates the resultant force exerted on the robot under the improved artificial potential field, and then discretizes the resultant force into four directions: up, down, left, and right. The improved artificial potential field method is used to replace the random exploration strategy, which provides an important reference for the robot's action decision-making. Subsequently, the robot's current position and map moment are obtained as input to the convolutional neural network. Finally, the network outputs the predicted Q value of the action in each direction. The present invention selects the most suitable action by combining the action with the maximum Q value at the current moment and the artificial potential field method. The training process stops when the robot reaches the target point.

[0217] The present invention can optimize the learning network training process, enabling the robot to make the most favorable movement action decisions based on real-time training results. At the same time, the design of dual experience pools makes the actions more efficient, greatly improving training efficiency.

[0218] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or replacements can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A mobile robot path planning method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Build a simulation platform to simulate a grid environment, where the grid includes robots, obstacles, and traversable areas; Step 2: Establish the robot's four movements in the grid environment: up, down, left, and right; Step 3: Establish two neural network models, a training network and a target network; the training network and the target network have the same structure but different weight parameters; Step 4: Use the current grid image and the robot's current coordinate position as the input of the convolutional neural network, and use the pixel Q values ​​of the four action predictions as the output, and filter out the action with the maximum Q value at that moment; Step 5: Optimize the repulsive force potential field by introducing the distance from the target point, and calculate the total potential field and the direction of the resultant force; Step 6: The robot selects a behavior based on the direction of the force of the potential field with probability ε, and selects the best behavior currently known based on the neural network with probability 1-ε; Step 7: When the robot successfully reaches the goal, a positive reward is given; when the robot collides with an obstacle, a negative reward is given; Step 8: Create two experience pools according to (s, a, s ′ ,r,) the positive or negative of r will store the positive reward experience and negative reward experience into the positive and negative experience pools respectively; Among them, s represents the current state, a represents the action taken in the current state, s' is the next state after taking the action, and r is the reward obtained from the action; Step 9: Based on reinforcement learning, the mobile robot path planning autonomous Q value prediction is performed, and the neural network weights are updated through the dual experience pool data. The weight parameters of the training network are copied to the target network at regular intervals. Step 10: After training, output the planned route.

2. A mobile robot path planning method based on reinforcement learning according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1-1: Use MATLAB software to build a simulation environment to simulate a mobile robot in a real environment; Step 1-2: Set a 10×10 grid area in the simulation environment as the working area. Randomly generate black obstacles of different proportions in this area. Obstacles in the grid are represented by black pixels, and passable areas are represented by white pixels. Step 1-3: Define the action space A = {′right′,′up′,′left′,′down′}, provide the state information of the current environment through the map matrix, and track the current position by updating the state information.

3. The mobile robot path planning method based on reinforcement learning according to claim 1, characterized in that: The step 3 specifically includes the following steps: Step 3-1: Create an input layer that accepts a 10x10 matrix as input and converts it into an image with a shape of [10, 10, 1]. Step 3-2: Create convolutional layers: The neural network contains three convolutional layers, each followed by a batch normalization layer and a ReLU activation function: Convolutional layer 1: 3x3 convolution kernel, 16 output channels; Convolutional layer 2: 3x3 convolution kernel, 32 output channels; Convolutional layer 3: 3x3 convolution kernel, 64 output channels; Step 3-3: Building fully connected layers: The final part of the network consists of two fully connected layers: First layer: 64 neurons; Second layer: 4 neurons, outputting the Q value of each action; Step 3-4: Create a regression layer: The output layer is a regression layer that generates the Q value for each action.

4. A mobile robot path planning method based on reinforcement learning according to claim 1, characterized in that: The step 4 specifically includes the following steps: Step 4-1: Use a convolutional neural network to extract a 10×10 grid image and the robot’s row and column indices within the grid, enabling the neural network to better understand the spatial layout of the environment and the robot’s current position. Step 4-2: Use the value function approximation method, and use the optimal action value function θ as the parameter θ * (s,a) is the goal so that the action value function Q(s,a,θ) approaches it, as shown in the following formula, Q(s,a,θ)≈Q * (s,a); Step 4-3: Output a vector of length four, corresponding to the Q-value predictions of the four actions; each Q-value represents the expected reward of the robot taking the corresponding action in the current state.

5. The mobile robot path planning method based on reinforcement learning according to claim 1, characterized in that: The step 5 specifically includes the following steps: Step 5-1: Construct a gravitational potential field; determine the gravitational potential field by calculating the distance between the current position and the target point, and attract the robot to move toward the target point: Where: represents K att represents the gravitational gain, d 2 (X,X T ) represents the current point X and the target point X T The square of the Euclidean distance between them; Step 5-2: Construct a repulsive potential field to prevent the robot from colliding with obstacles: Among them: K rep represents the repulsion gain coefficient, d(X,X o ) represents the distance between the current point X and the obstacle X o The Euclidean distance between them, d0 represents the repulsion threshold; Step 5-3: Improve the repulsive force potential field and introduce the distance from the target point to optimize the repulsive force field: Where: λ is any positive real number; Step 5-4: Calculate the total potential field and the magnitude of the resultant force direction; the global potential field function is the vector superposition of the gravitational field function and the repulsive field function, defined as: U(X)=U att (X)+U rep (X); Step 5-5: Calculate the negative gradient of the total potential field to obtain the magnitude and direction of the resultant force acting on the robot; Step 5-6: Discretize the resultant force into four directions A = {'right', 'up', 'left', 'down'} according to reasonable directions.

6. A mobile robot path planning method based on reinforcement learning according to claim 5, characterized in that: The step 5-5 specifically includes the following steps: Step 5-5-1: Take the negative gradient of the gravitational potential field to get the gravitational force: F aat (X)=-K att d(X,X T ); Step 5-5-2: Find the negative gradient of the improved repulsive potential field to obtain the repulsive force: o ) Take the negative gradient and get For d(X,X o )Find the negative gradient and we get: Then the net force on the robot in the improved repulsive field is: Step 5-5-3: The net force acting on the robot is the vector superposition of the gravitational force and the repulsive force: F(X)=F att (X)+F rep1 +F rep2 。 7. The mobile robot path planning method based on reinforcement learning according to claim 1, characterized in that: The action reward strategy in step 7 includes a primary reward and a secondary reward, where the primary reward is: When the robot successfully reaches the goal, a positive reward of +10 is given; When the robot collides with an obstacle, a negative reward of -1 is given; Give the robot a negative reward of -0.1 for each step; The main reward formula is: The secondary rewards are: When the distance between the robot's current position and the target decreases, a positive reward of +0.5 is given; When the distance between the robot's current position and the goal increases, a negative reward of -0.5 is given; The secondary reward formula is: Where ΔD is the difference between the Euclidean distance between the current position and the target point and the Euclidean distance between the next position and the target point.

8. The mobile robot path planning method based on reinforcement learning according to claim 1, characterized in that: The step 8 specifically includes the following steps: Step 8-1: Establish positive and negative experience pools; Step 8-2: In the current state s, the robot selects an action a according to the current strategy. After executing action a, the environment returns a new state and immediate reward r, which will serve as a reference for subsequent learning; Step 8-3: Store the experience points into two experience pools based on the positive or negative value of the reward at each step; Step 8-4: Check the experience pool capacity and remove the oldest experience; if the positive experience pool or negative experience pool reaches the upper limit, remove the oldest experience; Step 8-5: Calculate the number of samples from the positive experience pool and the negative experience pool, and set the ratio by the hyperparameter ρ; the specific steps are: Step 8-5-1: Sampling data of the positive experience pool: Calculate the number of positive experiences sampled according to the ratio ρ: n p =n×ρ Among them, n is the total number of samples, ρ is the sampling ratio parameter, n p The number of samples of positive reward experience for each training process; Step 8-5-2: Sampling data of negative experience pool: The number of negative reward experience can be obtained by subtracting the number of positive reward experience from the total sample data n experience pool: n n =n-n p Among them, n is the total number of samples, n n The number of negative reward experiences sampled during each training process; Step 8-5-3: Send the experience sampled from the two experience pools into the network for training.

9. The mobile robot path planning method based on reinforcement learning according to claim 1, characterized in that: The step 9 specifically includes the following steps: Step 9-1: Update the neural network weights through the dual experience pool data; the experience (s, a, r, s) during the interaction between the robot and the environment ′ ) is stored in the positive and negative experience pools. The specific steps are: Step 9-1-1: Sampling from the experience pool; Each time the network is updated, a batch of experience samples will be randomly taken from the two experience pools. Each sample contains the current state s, the action a taken in the current state, the reward r for the action, and the next state s after taking the action. ′ ; Step 9-1-2: Calculate the target Q value; use the current network to select the action: Use the target network to calculate the target Q value: Where γ is the discount factor; r t+1 represents the expected reward at time t+1; Represents the objective function value, which is obtained by the reward value of the current action and the Q value of the best action corresponding to the next state; θ t Represents the parameters of the current neural network, θ t ′ Represents the parameters of the target neural network; Step 9-2: Neural network weight update, using the difference between the Q value calculated by the current network and the Q value evaluated by the target network to update the parameters of the current network; Use the mean square error loss function to calculate the error between the predicted Q value and the target Q value to train the neural network: In the formula, is the target Q value, Is the current network according to the state s t and action a t Predicted Q value; Optimize the parameters θ of the current network by gradient descent t , in order to minimize, the weight update is completed through the following formula: in, Indicates that the partial derivative of the network layer is obtained by chain derivation rule; θ t Represents the parameters of the current neural network.

10. The mobile robot path planning method based on reinforcement learning according to claim 1, characterized in that: The step 10 specifically includes the following steps: Step 10-1: Let the path node set be {P k |k=1,2,3,…,n}, P1 represents the starting point, P n Represents the target point; the specific steps are: Step 10-1-1: Starting from P2, solve the vector and If the angle between the two vectors is 0, it means that P n-1 ,P n ,P n+1 If the three path nodes are on the same line, remove the redundant collinear node P n , repeat the process until there are no sequence points to be removed; Step 10-1-2: In the remaining nodes, calculate the vector and If the angle between these two vectors is not 0, it indicates that the path node P a-1 、P a 、P a+1 are not on the same straight line. At this time, solve the line segment L(P a-1 P a+1 The equation of , and judge whether the line segment intersects with the obstacle. If the line segment does not intersect with the obstacle, then P a is a redundant turning point and should be removed; if the line segment intersects an obstacle, then P a It is a necessary turning point and should be retained; repeat the above process until there are no sequence points that need to be removed; Step 10-2: After removing redundant nodes, use a third-order Bezier curve to further smooth the path: B(t)=(1-t) 3 P0+3(1-t) 2 tP1+3(1-t)t 2 P2+t 3 P3 Among them, P0 and P3 are the starting point and end point, P1 and P2 are intermediate control points, and t∈[0,1].

Images